release tracking bugs for post-mortem:

• Bug 1198816 - tracking bug for build and release of firefox 40.0.3
• Bug 1199390 - tracking bug for build and release of Thunderbird 41.0
• Bug 1188493 - tracking bug for build and release of Firefox and Fennec 41.0

Firefox 40.0.3

• Win32 repack 13/20 failed with what looks like sh crashing during configure
  • Rebuilt the chunk
• Updates step failed with:

release.updates.patcher.PatcherConfigError: Found multiple past-updates with duplicate to/from versions: ['40.0', '40.0.2', ['betatest', 'releasetest', 'release']]

• • Fixed with https://bugzilla.mozilla.org/attachment.cgi?id=8652919&action=edit and a move of the FIREFOX_40_0_3_{RELEASE,BUILD1} tags to the new rev. A rebuild then succeeded.

Firefox 41.0b2

• failed at firefox_beta_updates on update_verify
  • resolution: patch patcher configs: https://bugzilla.mozilla.org/show_bug.cgi?id=1188493#c19
• failed at repack_9/10 on win64
  • resolution: was a slow slave and timed out in configure step. nick re-ran this builder
• [fennec only] bouncer submission and post-release steps failed
  • resolution: ignore
  • further action: solve https://bugzil.la/1193613
• update_verify_beta_3/6 failed for Firefox 41.0b2 build1 on win32 <- hung for 10 hours
  • resolution: rebuild because it slave was in bad condition (couldn't delete previous file artifacts)
  • further action: always sanity check that there is a "ready to ship to the beta channel" email after relman asks for 'push to beta channel'

Firefox 41.0b3

• win32 build failed because of slow win slave trying to remove old buildirs (retriggered)
• win{32,64}_xul failed because of https://bugzil.la/1170212 (win32_xul didn't even make it that far because it too timed out trying to remove old builddirs)
• win64 repack_5/10 failed due to the en-US firefox.zip being only ~9MB instead of 50MB. nthomas reran
• Firefox 41.0b3 build1: failed at update_verify_beta_4/6 on win32. jlund re-ran. https://bugzilla.mozilla.org/show_bug.cgi?id=1188493#c23

Firefox 41.0b4

• win{32,64} xulrunner failed to build/post-release
• win64 needed a buildspace bump of s/12/14
• bouncer single locale submission still broken. should be fixed in beta5 via https://bugzil.la/1193613

Firefox 41.0b5

• bouncer submit (again) bc single locale: tracking https://bugzilla.mozilla.org/show_bug.cgi?id=1193613#c27

Thunderbird 41.0b1

• all the repacks failed because they needed 'export' in make config target
  • resolved with https://bugzilla.mozilla.org/show_bug.cgi?id=1193620#c5
• repacks are now failing because of a build sys error. something to do with lightning. tb folks are debugging
  • looks like it's been resolved: https://bugzilla.mozilla.org/show_bug.cgi?id=1199390#c6
  • expect a build2 soon

Roundtable

• [jlund] TCW - we accidentally kicked off a bunch of release jobs. anything we can do about blocking that in the future?
  • https://dxr.mozilla.org/build_buildbotcustom/source/scheduler.py#446
  • action: jlund to file bug
  • action: file bug to chmod -R a-w candidates / release directories (excepting contrib)
• ftp -> archive.m.o (s3) seemed to not cause any issues
  • anything to look out for in future?