Firefox

Release build 1

• (RelMan) Ship the l10n milestone (build1 only)
• (RelMan) Submit to Ship It - abandoned - Media Playback team has received reports of A/V sync problems (multiple seconds) with some YouTube content (bug 1245696. This is a regression in FF 44 from bug 1229605. Turning off "media.mediasource.webm.audio.enabled" will revert from Opus audio to AAC audio, which is a well-tested code path. Opus has slight better sound quality and/or lower bandwidth requirements than AAC. Building build2 with that pref turned off
• Push to mirrors
• Apply Throttling
• Verify throttling
• Publish in Balrog
• Post-release tasks

Release build 2

• (RelMan) Ship the l10n milestone (build1 only)
• (RelMan) Submit to Ship It
• Push to mirrors
• Apply Throttling
• Verify throttling
• Publish in Balrog
• Post-release tasks

Issues

We were a bit naughty and didn't wait for 'ready for release' email before pushing updates. Turns out the bouncer check was stalled

2016-02-08 14:25:43-0800 [HTTPPageGetter,client] TriggerBouncerCheck: uptake is 0

which was from the request

https://bounceradmin.mozilla.com/api/uptake/?product=Firefox-44.0.1-Partial-41.0.2&os=win64

not having any uptake. Win64 started at 42.0, so a partial from 41.0.2 makes no sense. Would not be surprised if this is another case of 'ship-it makes bad suggestions for partial updates' (bug 1146863).

To fix this up nthomas dropped a dummy text file at pub/firefox/releases/44.0.1/update/win64/zh-TW/firefox-41.0.2-44.0.1.partial.mar, but buildbot failed to continue once it had uptake for all requests:

2016-02-08 14:30:43-0800 [HTTPPageGetter,client] TriggerBouncerCheck: uptake is 2000100
2016-02-08 14:30:43-0800 [HTTPPageGetter,client] TriggerBouncerCheck: Stopping uptake monitoring: Reached required uptake: 2000100
2016-02-08 14:30:43-0800 [HTTPPageGetter,client] TriggerBouncerCheck failed:
        Traceback (most recent call last):
          File "/builds/buildbot/build1/lib/python2.7/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/build1/lib/python2.7/site-packages/twisted/internet/defer.py", line 664, in _cbDeferred
            self.callback(self.resultList)
          File "/builds/buildbot/build1/lib/python2.7/site-packages/twisted/internet/defer.py", line 318, in callback
            self._startRunCallbacks(result)
          File "/builds/buildbot/build1/lib/python2.7/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/builds/buildbot/build1/lib/python2.7/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbotcustom/scheduler.py", line 327, in checkUptake
            Triggerable.trigger(self, self.ss, self.set_props)
          File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_41f0fbee10f4_production_0.8-py2.7.egg/buildbot/schedulers/triggerable.py", line 66, in trigger
            d = self.parent.db.runInteraction(self._trigger, ss, props)
        exceptions.AttributeError: 'NoneType' object has no attribute 'db'

There was a reconfig at 11:00 PST (ie after the initial start of TriggerBouncerCheck and it succeeding) which is a known to cause failures.

To work around this the release-mozilla-release-firefox_release_start_uptake_monitoring builder was forced (setting script_repo_revision and release_config properties), which fired off jobs and emails as expected, culminating in the 'ready to release' email.