From MozillaWiki
Jump to: navigation, search

Upcoming vacation/PTO:

  • alin - aug31-sep11
  • coop - aug 3, aug 17-28
  • kmoir - aug 3, aug 24
  • otilia - aug10-21, half day on July31
  • vlad - jul31 ; aug14-aug27
  • Monday Aug 3 - Holiday in Canada


   @Callek: Kim reviewed the patch and said it was ok. If you have time, please land it and merge it to the production branch. - done --> thanks! -->backed out, see bug


1. took care of several common tasks: --> - talos-linux32-ix-022

   Van Le ran several memory and disk tests, all of them passed
   the slave has been re-imaged, so I enabled it in slavealloc
   waiting to see if it takes jobs/how they run

UPDATE: burned another job, disabled --> - t-xp32-ix-041

   burned the latest several jobs, disabled
   re-imaged, waiting to see if it works now

UPDATE: started taking jobs, looks fine

2. started looking over the additional bugs to work on

   tried to run "" script from my local machine
   I am using Cygwin and Windows 8.1
   copied the script in /home/alin.selagea, installed pip and then fabric

$ pip freeze | grep Fabric Fabric==1.10.2

   spent a lot of time debugging, but I get: 

File "", line 13, in <module>

   import util.fabric.actions

ImportError: No module named util.fabric.actions

   downloaded this module from github and copied the folder "fabric" under /lib/python/util

Q: do you know of any additional settings that should be made? Did you copy the entire tools dir to /home/alin.selagea or just the one script? You need to copy the entire repo, not just the one script. \o/ NOW IT WORKS \o/ :)


   patch to disable freshclam failed on t-snow and t-yosemite machines and has been rolled back
   talked to Amy and it seems that the "unless" condition is not correct
   currently working to change that line

onlyif => " /bin/test -f /etc/freshclam.conf && /bin/test "$(/usr/bin/grep 'Checks 0' /etc/freshclam.conf)" != "Checks 0"" ;

   the second test condition needs to be adjusted, as it doesn't work fine yet
   Amy also mentioned that "org.clamav.freshclam-init.plist has RunAtLoad true", so maybe we should add that to the disabled services in order to fix the bug
   Kim will find pointers to change plists
   [ ~]# launchctl list | grep freshclam
   -       0       org.clamav.freshclam-init

4. in the morning I was asked by Nigel if there were some changes to the linux machines as noticed some failures:

   errors like: 'mach build' did not run successfully. Please check log for errors.
   on the #developers channel, it was suggested that the issue was not related to infrastructure but to clobber. 
   I don't know exactly what "clobber" refers to in this case

--> in the case of clobberring a tree -> refers to the process of erasing the build directory from a machine Q1: where do we find the logs to see if a clobber has been run? Q2: (I know this was briefly explained during the first week) some tips&tricks for working with Treeherder

You can see that each build clobbers the properties and scripts dirs - see clobber_scripts and clobber_properties here for a particular job and machine. During the run script stage, it checks to see if a clobber needs to be run, for example You can see 01:46:04 INFO - ##### 01:46:04 INFO - ##### Running clobber step. 01:46:04 INFO - ##### 01:46:04 INFO - Running pre-action listener: influxdb_recording_pre_action 01:46:04 INFO - Running main action method: clobber 01:46:04 INFO - retry: Calling run_command with args: [['/builds/slave/m-in-m64-d-0000000000000000000/scripts/external_tools/', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-s', 'oauth.txt', '-t', '168', '', 'mozilla-inbound', 'OS X 10.7 64-bit mozilla-inbound leak test build', 'm-in-m64-d-0000000000000000000', 'bld-lion-r5-083', '']], kwargs: {'error_list': [{'substr': 'Error contacting server', 'explanation': 'Error contacting server for clobberer information.', 'level': 'error'}], 'cwd': '/builds/slave'}, attempt #1 01:46:04 INFO - Running command: ['/builds/slave/m-in-m64-d-0000000000000000000/scripts/external_tools/', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-s', 'oauth.txt', '-t', '168', '', 'mozilla-inbound', 'OS X 10.7 64-bit mozilla-inbound leak test build', 'm-in-m64-d-0000000000000000000', 'bld-lion-r5-083', ''] in /builds/slave 01:46:04 INFO - Copy/paste: /builds/slave/m-in-m64-d-0000000000000000000/scripts/external_tools/ -s scripts -s logs -s buildprops.json -s token -s oauth.txt -t 168 mozilla-inbound "OS X 10.7 64-bit mozilla-inbound leak test build" m-in-m64-d-0000000000000000000 bld-lion-r5-083 01:46:05 INFO - Checking clobber URL: 01:46:05 INFO - m-in-m64-d-0000000000000000000:Our last clobber date: 2015-08-18 10:12:57 01:46:05 INFO - m-in-m64-d-0000000000000000000:Server clobber date: 2015-08-10 15:40:31 01:46:05 INFO - Return code: 0 01:46:05 INFO - Running command: ['/builds/slave/m-in-m64-d-0000000000000000000/scripts/external_tools/', '-s', '10', '--max-age', '14', '--not', 'info', '--not', 'rel-*:10d', '--not', 'tb-rel-*:10d', '/builds/slave'] 01:46:05 INFO - Copy/paste: /builds/slave/m-in-m64-d-0000000000000000000/scripts/external_tools/ -s 10 --max-age 14 --not info --not rel-*:10d --not tb-rel-*:10d /builds/slave 01:46:05 INFO - Using env: {'HG_SHARE_BASE_DIR': '/builds/hg-shared', 01:46:05 INFO - 'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin'} 01:46:05 INFO - Ignored directory 'rel-m-beta-m64_beta_u_v_3-0000' exceeds cutoff time 01:46:05 INFO - Ignored directory 'rel-m-beta-m64_rpk_1-000000000' exceeds cutoff time 01:46:05 INFO - Ignored directory 'rel-m-beta-m64_rpk_3-000000000' exceeds cutoff time 01:46:05 INFO - Ignored directory 'rel-m-rel-m64_rpk_5-0000000000' exceeds cutoff time 01:46:05 INFO - Ignored directory 'tb-rel-c-esr38-m64_bld-0000000' exceeds cutoff time 01:46:05 INFO - Ignored directory 'tb-rel-c-esr38-m64_rpk_1-00000' exceeds cutoff time 01:46:05 INFO - 519.60 GB of space available 01:46:05 INFO - Return code: 0


Will look for logs for clobber -- ^^ see above


1. common tasks:

   re-imaged the machines that returned from loan and enabled them in slavealloc
   loaned one win 8 and one win xp machines to Kim :) 

2. - disable freshclam

   tried both with the example suggested by Kim and Amy's patch
   received errors on t-snow and t-yosemite slaves, debugging 


1. common tasks:

   loaned two AWS slaves to Ted Mielczarek:
   --> tst-linux32-ec2-ted  
   --> tst-linux64-ec2-ted - t-snow-r4-0089
 --> this slave has been disabled due to GC crashes
 --> re-imaged and returned it to the pool, waiting to see how it goes

UPDATE: already completed 10+ jobs without issues --> marked the bug as Resolved

2. received 5 consecutive alerts like: [sns alert] Thu 03:08:45 PDT t-snow-r4-0032 puppet-agent: /usr/local/bin/screenresolution set 1600x1200x32 returned 1 instead of one of [0]

   connected to the slave --> "/usr/local/bin/screenresolution get" 

"2015-08-27 03:42:38.042 screenresolution[8382:903] Error: failed to get list of active displays" tried the same command on t-snow-r4-0089: "2015-08-27 03:42:07.742 screenresolution[1109:903] Display 0: 1600x1200x32" restarted the slave, ran the command again: "2015-08-27 03:46:53.403 screenresolution[655:903] Display 0: 1600x1200x32" -- seems ok now Yes, this alert usually means that we need to reboot the machine. We should add a check in runner to check for this alert and reboot, perhaps open a bug to investigate.

3. also received several alerts from relengbot:

   <relengbot> [sns alert] Thu 05:08:02 PDT Count: 633 | First instance: 2015-08-27 04:00:03-0700 | Most recent instance: 2015-08-27 04:29:00-0700 | Twistd exception: twisted.cred.error.UnauthorizedLogin - unknown
   investigated, I don't know where those IPs are coming from
   asked in #releng channel for suggestions
   This is an machine trying to connect to the buildbot master that has either 1) wrong password 2) wrong username 3) is not listed in the range that can connect ot the master
   Looks like this machine is try-linux64-spot-407 (I looked in the AWS console)

[Broker,24354,] BuildSlave.detached(try-linux64-spot-349) 2015-08-27 04:26:08-0700 [Broker,25204,] Peer will receive following PB traceback: 2015-08-27 04:26:08-0700 [Broker,25204,] Unhandled Error

4. - retry_dead_queue should run periodically

   I am now able to run the script from my local machine using Cygwin
   tried to find a way to implement a cronjob to run the command on all the masters
   looked at bug - Automate monthly graceful restart of buildbot masters for a clue
   my guess is that we can use the options -D, -M, -H to select the masters

Q: which machine should be used to set the cronjob? Do a needinfo on the bug for rail to ask about that

5. - disable freshclam on OSX builders

   loaned two slaves: t-snow-r4-0156 and t-yosemite-r5-0090
   modified slave.pp file to test Amy's patch
   added t-snow-r4-0156 and t-yosemite-r5-0090 in moco-nodes.pp configuration file
   working fine on bld-lion-r5-078
   on t-snow and t-yosemite I get some ugly errors:

Could not set 'present' on ensure: No such file or directory - at 38:/etc/puppet/environments/aselagea/modules/packages/manifests/pkgdmg.pp

   kim will look at yosemite - didn't get a chance to do that today because we had a lot of issues today, will look at it first thing tomorrow

Build changeset is here f8086bd3c84f

search for build changeset on this page f8086bd3c84f

changeset referred to in bug is 8205877b3b30


Yesterday we had some buildduty issues, thought you might want to read them for interest

In the morning SETA stopped working for 6 hrs on Aug 27, then started working again, why? buildbot-master81 in the twistd.log

which caused Very high test backlog

In the afternoon, - 301 /(.*) to$1 Summary: IT made a change to change to The ip address changed and this meant we didn't have the ssh key to scp content there

ftp max clients is WARNING: Using 2400 out of 2400 Clients ftp server in load balancing cluster failed, one had to be put in passive mode. Unclear if this is the result of extra traffic from bug 1198296 above

We can discuss tomorrow. Also, thanks for checking the state of the AWS masters starting instances, that was very helpful.


[alin] 1. common tasks:

   bugs 1198820, 1198821 - terminated instances, removed records from inventory, revoked VPN access
   re-imaged several slaves, monitoring:
       --> b-2008-ix-0025: 0 jobs passed | 0 failed
       --> b-2008-ix-0016: 0 jobs passed | 0 failed
       --> t-w732-ix-001:   4 jobs passed | 0 failed


   t-snow-r4-0093 has been decommissioned by the DCOps team
   opened bug to implement the required changes on our side: 1199586
   attached patch to remove slave from configs --> please review it when you have time :)
   also removed the slave from slavealloc

3. - retry_dead_queue should run periodically

   did a needinfo on the bug for Rail, waiting for reply

4. - disable freshclam on OSX builders

   the issue does not seem to be related to the patch
   when you have time, please take a look
   loaned slaves: t-snow-r4-0156 and t-yosemite-r5-0090

[vlad] 1. Requested access for wiki page 2. Helped Alin to figure it why the puppet is not working on t-yosemite-r5 slaves and t-snow-r4 slaves

Additional bugs to work on 1) retry_dead_queue should run periodically Way to stop all these alerts :-) 2) Add a runner task to check resolution on Windows testers before starting buildbot 3) Add T testing to the trychooser UI Code is in and trychooser dir

FYI Friday afternoon we had this problem