ReleaseEngineering/Buildduty/SVMeetings/Oct17-Oct21

Meeting Details

• Room Link: https://v.mozilla.com/flex.html?roomdirect.html&key=RSAkZHawz3ED

• Meetings every Wednesday
• Meeting notes archive:
  • https://wiki.mozilla.org/ReleaseEngineering/Buildduty/SVMeetings

Upcoming vacation/PTO:

• Alin: Oct 21
• Andrei:
• Florin: November 1-10
• Coop:
• Ihsiao:
• Ashiue: Oct 24
• Kmoir:

Holidays:

• Canada:
• Romania:
  • November 30
  • December 1 (National Day)
• Taiwan:
• Retrocession Day, Oct 25
• Birthday of President Chiang Kai-shek, Oct 31

Possible bugs

• Write a tool to update bugzilla bugs for outstanding machine loans and ask if the loaner is still needed on say, a weekly basis: https://bugzilla.mozilla.org/show_bug.cgi?id=1171165
• Long Inter-Job Timing on t-yosemite-r7 https://bugzilla.mozilla.org/show_bug.cgi?id=1258480
• Create new script to setup a buildbot master for development: https://bugzil.la/1275135

From coop:

• Incident response (from hg.m.o post-mortem)
  • https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Dealing_With_Outages
• Server alerts: finishing up work on fwunit and ssl cert changes
• https://bugzilla.mozilla.org/show_bug.cgi?id=1307757
• Prep for Hawaii

From kmoir:

• Revert changes to https://bugzilla.mozilla.org/show_bug.cgi?id=1302578 g2.2xlarge price spike and add more capacity to the pool in other regions https://bugzilla.mozilla.org/show_bug.cgi?id=1302549. (Done and I think ondemand instances were also added by rail/catlee)
• How are preparations for the mercurial upgrades going for build/test machines? (to support bug 1298976)
• Other possible bugs:
  • Schedule win10vm test jobs on try https://bugzilla.mozilla.org/show_bug.cgi?id=1304065
  • watch_pending should send an alert when we're outbid for spot instances https://bugzilla.mozilla.org/show_bug.cgi?id=1302567
  • Create alerts for t-w732 and g-w732 pending https://bugzilla.mozilla.org/show_bug.cgi?id=1304158

Taipei: 2016/10/17 - 2016/10/21

• Check failures on trees and mark failures/file new bugs
• Backout changesets if any failures or bustage: bug 1302787
• Triage perf alerts
• Investigate perf regression issues: bug 1310113, bug 1310119, bug 1189901, bug 837961
• Investigate bug 1309956 - sessionrestore_no_auto_restore generating NaNs

SV: 2016/10/17 - 2016/10/21

• Bug 1304065 - Schedule win10vm test jobs on try
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1304065
  • Creating the patch for puppet and build-cloud-tools
• Bug 1299421 - Switch to running restart_masters.sh on the buildduty-tools machine
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1299421
  • I will add the keys to that machine,also I will keep the sed part in order to delete lines 4-8 from /home/buildduty/.ssh/config because otherwise the script will fail when trying to connect by password to the buildbot-masters
• Bug 1308544 - Enable automation jobs on Cedar twig
  • Linux x64 opt tests are running in both Buildbot and Taskcluster (tier2)
• Bug 1307757 - Deploy python 2.7.12 to releng servers
  • Looked for a machine class attribute to use for those two machines => I don’t think we have one in our puppet repo
  • Stick to manually specifying those two servers and add a comment for that?
  • Manually running puppet agent cronjob (as listed in /etc/cron.d/puppetcheck) will result in some errors on buildduty-tools machine:

E.g. “Wed Oct 19 06:18:09 -0700 2016 Puppet (err): Could not get latest version: Execution of '/usr/local/bin/python /usr/lib/ruby/site_ruby/1.8/puppet/provider/package/yumhelper.py' returned 1: _pkg mozilla-python27.x86_64 0 releng-public-CentOS6 x86_64 2.7.7-1.el6”

• • I don’t see any e-mails for that when it runs automatically
  • Also, it runs fine on cruncher