Release Duty

• FF 46 cycle: kmoir

Misc

Shipped

Firefox 46.0.b9 (kmoir/rail)

• https://hg.mozilla.org/releases/mozilla-beta/rev/b007110e90053e58946b59765605bfca766c30d1
• Task graph: https://tools.taskcluster.net/task-graph-inspector/#g40tKtc8Sc2KgrYA7KU5NA/
• had to rerun some update verify jobs again via tctalker that not scheduled/cancelled

Fennec 45.0.2 (nthomas/kmoir)

• https://hg.mozilla.org/releases/mozilla-release/ac2c157c4bd7
• build1 - Fennec 45.0.2 build1: build step failed on android-api-9, problems downloading packages, restarted

Firefox-46.0b10 (rail/kmoir)

• https://hg.mozilla.org/releases/mozilla-beta/rev/9ea83990839bd513869018e57bcbedb3454b63bb
• Task graph: https://tools.taskcluster.net/task-graph-inspector/#ohNsYaogRtuE2kNFh_oEFw/
• Firefox build1 - some update verify and repack jobs were not scheduled, reran them vi tctalker

Firefox 45.0.2 (kmoir/rail/nthomas)

• https://hg.mozilla.org/releases/mozilla-release/ac2c157c4bd7
• build1 - Firefox partner repacks broken, filed bug 1263082. Repacks being broken are due to a bad (new version) of git being deployed on our build machines. I reran the repacks for all platforms from the command line by hand.
• had to run checksums, final verification jobs manually due to repack failure
• because the checksums job ran after the push to mirrors job, the checksums jobs are missing. I tried to copy it manually but it didn't work. Turns out it works with boto, not with s3cmd. bug 1263831

Firefox 45.0.2esr (nthomas/kmoir)

• http://hg.mozilla.org/releases/mozilla-esr45/GECKO4501esr_2016031618_RELBRANCH
• build 1 - all update verify jobs failed because we built off GECKO4501esr_2016031618_RELBRANCH and bug 1255245 part 2 was only landed on default. The update files had the firefox-mozilla-release tag on them and 45.0.1 expected firefox-mozilla-esr, so the updater refused the updates. Landed on part 2 fix the relbranch,
• build2 - many repack failures win64 4 and 6, win32 3 and 7(submitting to balrog).
• build2 - many update verify jobs failed see bug 1263721
• build3 - was needed because some patches did not land on the relbranch
• build3 - win64 repack 8/10 had a hang in make upload, reran. 3x mac repacks took 5+ hours to fail when mac-v2-signing7 was overloaded, reran. Many update verify failures.
• build 3 - after release added win64 bouncer config (bug 1264102. This is more fallout from using a relbranch as bug 1254784 hadn't landed there

Fennec-46.0b10

• https://hg.mozilla.org/releases/mozilla-beta/rev/9ea83990839b
• Fennec build1 - release-mozilla-beta-android-api-15_repack_8 failed due to problems downloading packages from mirrors

Ongoing

Thunderbird 45.0 (nthomas/kmoir)

• https://hg.mozilla.org/releases/comm-esr45/983d1ccaea8f
• * win32 build step failed for Thunderbird 45.0 build1, bash crashed during the generation of a partial update, which is an intermittent issue. Rerunning the job. nthomas reran
• release-comm-esr45-win32_repack_8/10 hit a crash in bash while creating an update, reran.
• release-comm-esr45-win32_repack_8/10 and 5/10 hit a hang while creating an update, reran both.
• lots or retries on windows repacks, as slaves disconnected from masters
• repack_5/10 failed for Thunderbird 45.0 build1 on win32 failed many times with bash error, rerunning now
• release-comm-esr45-thunderbird_antivirus failed connecting to s3. Maybe we should add a retry there? Will have to go look at code.

9:51:51     INFO - retry: Calling <bound method Key.get_contents_to_filename of <Key: net-mozaws-prod-delivery-archive,pub/thunderbird/candidates/45.0-candidates/build1/update/linux-x86_64/ro/thunderbird-45.0.complete.mar>> with args: (u'cache/update/linux-x86_64/ro/thunderbird-45.0.complete.mar',), kwargs: {}, attempt #5
09:51:51     INFO - retry: Giving up on <bound method Key.get_contents_to_filename of <Key: net-mozaws-prod-delivery-archive,pub/thunderbird/candidates/45.0-candidates/build1/update/linux-x86_64/ro/thunderbird-45.0.complete.mar>>
09:51:51    FATAL - Uncaught exception: Traceback (most recent call last):
09:51:51    FATAL -   File "/builds/slave/tb-rel-c-esr45-av-000000000000/scripts/mozharness/base/script.py", line 1756, in run
09:51:51    FATAL -     self.run_action(action)
09:51:51    FATAL -   File "/builds/slave/tb-rel-c-esr45-av-000000000000/scripts/mozharness/base/script.py", line 1698, in run_action
09:51:51    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
09:51:51    FATAL -   File "/builds/slave/tb-rel-c-esr45-av-000000000000/scripts/mozharness/base/script.py", line 1639, in _possibly_run_method
09:51:51    FATAL -     return getattr(self, method_name)()
09:51:51    FATAL -   File "scripts/scripts/release/antivirus.py", line 177, in get_files
09:51:51    FATAL -     pool.map(worker, find_release_files())
09:51:51    FATAL -   File "/tools/python27/lib/python2.7/multiprocessing/pool.py", line 227, in map
09:51:51    FATAL -     return self.map_async(func, iterable, chunksize).get()
09:51:51    FATAL -   File "/tools/python27/lib/python2.7/multiprocessing/pool.py", line 528, in get
09:51:51    FATAL -     raise self._value
09:51:51    FATAL - IncompleteRead: IncompleteRead(0 bytes read, 32737227 more expected)

Second run of this job succeeded.

• release updates failed because patcher config was missing, hg cp mozEsr38-thunderbird-branch-patcher2.cfg mozEsr45-thunderbird-branch-patcher2.cfg in tools and reran job. Then it failed submitting to balrog because of missing rules. Fixed in bug 1263279 When I ran the job again it failed since the it couldn't commit configs with the same tag, will have to run balrog job manually and retrigger downstream jobs.
• I changed the cdntest rule (id 334) to remove the (Version: >45.0) restriction and set the rate to 10 as per bug bug 1263279
• ran checksums and antivirus builders by hand since upstream jobs had failed
• final verification failed. When the version bump part of the release updates job ran

http://buildbot-master74.bb.releng.usw2.mozilla.com:8001/builders/release-comm-esr45-thunderbird_release_updates/builds/4

It didn't update the update configs, only the patcher configs. Looking at the log for ff esr45 it appears the job changes update configs too.

changeset: 6700:64c64b628a31 tag: THUNDERBIRD_45_0_BUILD1_RUNTIME tag: THUNDERBIRD_45_0_RELEASE_RUNTIME user: tbirdbld date: Mon Apr 11 15:09:18 2016 -0400 files: release/patcher-configs/mozEsr45-thunderbird-branch-patcher2.cfg description: Automated configuration bump: update configs for Thunderbird 45.0 build 1 on release

4:55 PM <rail> python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozEsr45-thunderbird-branch-patcher2.cfg --platform macosx64 --output tools/release/updates/mozEsr45-thunderbird-mac64.cfg --release-config-file mozilla/release-thunderbird-comm-esr45.py -b https://hg.mozilla.org/build/buildbot-configs --channel release-localtest -t
4:55 PM <rail> THUNDERBIRD_45_0_RELEASE
4:55 PM <rail> with tools and buildbot-config cloned in the $PWD
4:55 PM <kmoir> okay
4:56 PM <rail> repeat it for linux/linux64/win32 (need to change the cmd in 2 places) and you are done
4:56 PM <rail> you will need to commit and tag it with the something_RUNTIME tag

Roundtable

• notes from nthomas: where possible we should look for the 'ready for release' email to come in before shipping. GTK3 gets in the way of that by breaking update verify, but for ESR45 we could remove the ESR38 and older stuff until 45.2 comes along. Not sure there's a good way of handling partner_repack failures

Action items