User:Asasaki:MobileMaster

mobile-master.mv.mozilla.com is the buildbot master for the Nokia N810's running unit tests and Talos performance tests that show up on [the Mobile tinderbox page].

Network Setup

mobile-master uses a wired lan connection for general use, and Mozilla Guest wifi for n810 connections. Currently these are set to 10.250.1.48 and 10.250.16.48

The buildbot ports are 8010/9010 and 8011/9011 for production and staging, respectively.

Buildbot Setup

This is pretty similar to our other masters. Running as cltbld, with /tools/buildbotcustom and /builds/buildbot/mobile-production and /builds/buildbot/mobile-staging. The configs softlink to files in /builds/buildbot/configs, which is currently cloned from http://hg.mozilla.org/users/asasaki_mozilla.com/mobile-configs until the unit test configs are fully merged back into build/buildbot-configs.

Files

I have some files in /Users/cltbld that are useful for reimaging the N810s.

buildbot-src contains our buildbot source; it can be recloned easily.

pages contains the Tp3 pages; it can be copied from any Tp3 talos machine. tp4 contains the Tp4 pages.

talos-maemo contains the Maemo talos source; it can be recloned easily from http://hg.mozilla.org/users/asasaki_mozilla.com/talos-maemo.

maemkit contains jmaher's Maemkit, which splits unit tests for speed and non-crashiness. It can be recloned from http://hg.mozilla.org/qa/maemkit, but it contains a number of local patches from both jmaher and aki, so it might be easiest to copy from another N810 should the need arise.

maemo-ssh contains the .ssh directory for each n810.

maemo-ping contains n810 monitoring scripts:

maemo-ping

There are two main files in ~cltbld/maemo-ping.

maemo_daily_reboot.sh, which is run from cron every minute:

#!/bin/sh

BASEDIR=`dirname $0`
LOGDIR="$BASEDIR/logs"

SSH_TIMEOUT=15
SSH_COMMAND="/etc/cron.daily/daily_reboot.sh"
#SSH_COMMAND="uptime"
ALL_MAEMO="$BASEDIR/all_maemo.txt"

TIMESTAMP_NOW=`date +%s`
NOTIFY_WAIT=1200
RECIPIENT_LIST="aki@mozilla.com,jford@mozilla.com"

#--------------------------------------------------------------------------
# Get random order of hosts
#--------------------------------------------------------------------------
sorted_list=`grep -v \# $ALL_MAEMO`
random_list=""
tmp_list=""
for i in $sorted_list; do
    tmp_list="$RANDOM-$i\n$tmp_list"
done

random_list=`echo $tmp_list | sort | sed -e 's/[0-9]*-//g'`

#--------------------------------------------------------------------------
#
#--------------------------------------------------------------------------
for i in $random_list ; do
    host=`printf "maemo-n810-%02d" $i`
    logdir="$LOGDIR/$host"
    if [ ! -d $logdir ] ; then
        mkdir -p $logdir
    fi

    STATUSFILE="$logdir/status"
    LASTHEARDFILE="$logdir/lastheard"
    LOGFILE="$logdir/log"

    lastheard=0
    if [ -e $LASTHEARDFILE ] ; then
        lastheard=`cat $LASTHEARDFILE`
    fi
    laststatus="unknown"
    if [ -e $STATUSFILE ] ; then
        laststatus=`cat $STATUSFILE`
    fi

    output=`ssh -o ConnectTimeout=$SSH_TIMEOUT root@$host "$SSH_COMMAND" 2>&1`
    status=$?

    if [ $status -eq 0 ] ; then
        # We're up
        if [ $laststatus != "up" ] ; then
            echo "up" > $STATUSFILE
            if [ $laststatus != "up" ] ; then
                echo "$host up at `date`" >> $LOGFILE
                if [ "a$output" != "a" ] ; then
                    echo "\n$output\n" >> $LOGFILE
                fi
                if [ $laststatus == "down" ] ; then
                    echo "$host up!\n\n$output" | mail -s "$host up" "$RECIPIENT_LIST"
                fi
            fi
        fi
        echo $TIMESTAMP_NOW > $LASTHEARDFILE
    elif [ $laststatus != "down" ] ; then
        if [ `expr $TIMESTAMP_NOW - $lastheard` -gt $NOTIFY_WAIT ] ; then
            echo "down" > $STATUSFILE
            echo "$host down at `date`" >> $LOGFILE
            if [ "a$output" != "a" ] ; then
                echo "\n$output\n" >> $LOGFILE
            fi
            if [ $laststatus == "up" ] ; then
                echo "$host down since `date -r $lastheard`!\n\n$output" | mail -s "$host down" "$RECIPIENT_LIST"
            else
                echo "$host down since epoch!\n\n$output" | mail -s "$host down" "$RECIPIENT_LIST"
            fi
        fi
    fi
done

and all_maemo.txt:

1
2
3
4
5
6
7     # alice tp4/graphs-new
8     # alice tp4/graphs-new
9
10
11
12
13    # master image
14
15
16
17

Nokia Jumphost

You can't get to Mozilla Guest from outside the office, even if you're using Mozilla-MV vpn.

However, you can ssh to mobile-master (as cltbld), then ssh to the N810 in question (as root).

Alternately, if you *need* to see the screen, you can VNC to mobile-master, then launch Chicken of the VNC on mobile-master and VNC to the N810 in question.

This should no longer be necessary since the new Nokia wifi lan should be VPN routable.