Releases:Release Post Mortem:2016-03-09

From MozillaWiki
Jump to: navigation, search

Meeting Details

« previous week | index | next week »
< most recent | upcoming >

Release Duty

  • FF 45 cycle: mtabara
  • FF 46 cycle: kmoir (welcome Kim!)



Firefox 45.0 (nthomas/bhearsum/mtabara/kmoir)

  • dev release notes here:
  • one repack timeout and the usual GTK3 update_verify_beta errors
  • one mac update verify failure due to complete mar d/l failure, rerun was OK
  • beta_final_verification didn't run because the uptake check was blocked trying to find partials. This is the same as we had for 44.0.1. Unblocked by dropping files on archive.m.o

and forcing the start_uptake_check builder again. We didn't get the 'ready to realease on beta' email because of the gtk3 update failures

  • initially throttled to 25%, now 0% (March 9th)

Fennec 45.0 (nthomas/bhearsum/mtabara/kmoir)

release build 1

  • failed at repack_7/10 on android-api-9 - no space left on device while signtool, automatic retry.
  • we filed bug 1253114 to increase the disk space requirement for Android repacks
  • interrupted because there was a failed branding for Fennec 45, RC build is branded as Beta - tracking bug 1253173

Explanation is incorrect

release build 2

  • all good with the build but a follow-up build is on the way to address bug 1246956

release build 3

  • release runner got restarted - This was a buildbot problem - a change in the SETA data which wasn't handled in the buildbot code, so jmaher backed out the SETA change. The releases have been restarted, with a tweaked list of partials for desktop
  • Looking at docs, I saw the link to post-release-tasks was listed but not push to mirrors for fennec. So I ran post-release without running push to mirrors which caused bug 1255057. Have updated doc references to clarify.


Firefox 45.0esr (mtabara/nthomas/kmoir)

Build 1

  • Intermittent 'invalid language code(s)' error from bounceradmin. We've had this issue in the past, there's a tracking bug for this bug 1248490
  • build step failed on macosx64 with complains on rustc.tar.bz2 - nthomas stated this as a "file in new tooltool is not in old tooltool" sort of error. We checked the 45.0 tooltool logs as well as we were curious on how did the same error did not occur for 45.0 as well but it turns out it was retrieved from cache. Other job must've put it there. nthomas uploaded it in the tooltool
  • failed at firefox_esr_updates - mozEsr45-branch-patcher2.cfg doesn't exist in, but I think we normally disable updates for the first release anyway. Then do updates from 45.0 when we do 45.0.1/45.1.0.

Build 2

  • push to mirrors failed because of bug 1254395 (push to cdn for non release promotion case doesn't use credentials) and bug 1254422 (Do not try to set S3 ACLs). So I retagged mozilla-esr45 with FIREFOX_45_0esr_RELEASE and FIREFOX_45_0esr_BUILD to and reran the push to mirrors builder

Firefox 38.7.0esr (mtabara/nthomas/kmoir)

build 1

  • antivirus failed due to intermittent downloading issue - retriggered it.
  • antivirus failed three times in a row - I followed up with a comment in bug 1248299
  • update_verify_esr_6/6 on linux64/linux/macosx failed yet again with "binary files found in diff" as in bug 1241951 back in 44.0-build2 - retriggered the builder, will follow-up with more information. Filed bug 1253185 to track these.
  • Softvision had some issues with esr-cdntest, attempted fix at described here
  • had to update release rules so that Firefox-38.6.1esr-build1-whatsnew points to 38.7.0.

Firefox 46.0b1 (rail/jlund)

(notes may be missing some details, there were so many patches and fixes landed by superheroes rail and jlund) Build 1

  • Fennec failed on push to mirrors due to AWS creds issue 17:06:29 FATAL - S3ResponseError: S3ResponseError: 403 Forbidden bug 1254395
  • push to cdn for non release promotion case doesn't use credentials bug 1254395
  • beetmover can't use set_canned_acl on candidate artifacts bug 1254422

Build 2

Build 3

  • bug 1254602 buildbot-bridge - investigate reason for green buildbot jobs being reported as failed and cancelled by taskcluster

Build 4

Fennec 46.0b1


Action items

  • nthomas to look at balrog rules for ESRs, may need some cleanup