Releases:Release Post Mortem:2016-01-13

Meeting Details

• 3:30pm ET
• Vidyo - Release Engineering room
• Dial-in:
  • 650-903-0800 or 650-215-1282 x92 Conf# 98225 (US/INTL)
  • 1-800-707-2533 (pin 369) Conf# 98225 (US)
• #releaseduty on irc.mozilla.org
• https://trello.com/b/MXHaVRcP/release-promotion-meeting

« previous week | index | next week »
< most recent | upcoming >

Release Duty

• FF 44 cycle: jlund/rail (mtabara shadowing)
• FF 45 cycle: mtabara

Misc

• Firefox_44.0b3 and Firefox_44.0b5 skipped for holiday reasons
• there will be NO "What's New Page" for FF 44 release

Shipped

Firefox 43.0.4 (jlund/rail/mtabara)

• "Because of a high volume crash on Windows on 43.0.3 we disabled updates to 43.0.3 and are about to build, test, and I hope release 43.0.4"
• Several failures along the way
  • build1: failed at repack_9/10 on win32 - timed out during cloning stuff, rerunning
  • build1: abandoned, starting build2 with "Backed out gdata patch, added sha-1 deprecation pref flip"
  • build2: abandoned as well. release automation pushed artifacts to build1 url location instead of build2
  • build2: there was an automation race condition where our masters tried pulling from hg.mozilla.org/build/buildbot-config and did not get served a recent push that included an incremental buildnum bump from 1 -> 2. This was likely due to the hg.m.o vcsreplicator issues that happened today and overlapped with the release and reconfig.
  • build2: we had an infrastructure failure with build2. Added a small ridealong patch to support download directories on multi-user machines
  • build3: started, in progress
  • build3: failed at firefox_antivirus, re-triggered; also failed in 4 out of 6 update_verify_release steps on win64, all retriggered

Firefox/Fennec_44.0b6 (jlund/rail)

• post-release step still undone for Fennec 44.0b6. still pending QE signoff?
• dragged from last week, had to run the post-release step after QE sign-off
• No failures of any type, some intermittent stuff:
  • failed at repack_9/10 on win64, automatic retry
  • all of the update verify linux failures were gtk related, hence ignored

Thunderbird 38.5.1 (jlund/rail/mtabara)

• watershed release, has part2 patch
• release notes here
• last status from 2015: release-localtest channel looks good, awaiting request for release-cdntest. It's being hold off until the various upgrade paths using the waterfall are fully tested.
• Some intermittent failures came up:
  • build1 - bug 1235071 ssh: symbolpush.mozilla.org: no address associated with name, invalid bug - wrong cset
  • build2 - bug 1226276 l10n taggging fails when running 'hg branches' against clean checkout of certain locale repos, rerunning

Firefox/Fennec_44.0b7 (jlund/mtabara/rail)

• No failures of any type, some intermittent stuff:
  • failed in some repacks, automatic retry
  • run failed, update status was not successful, retriggered
  • could not download source firefox-44.0b7.tar.bz2, download_builds returned non-zero, retriggered
  • no complete update found, download_mars returned non-zero, retriggered
  • linux/linux64 update verification errors GTK3-related, ignored

Roundtable

• jlund/rail/mtabara: discussion over the Releaseduty FAQ page. Pointed urls for it here and here.
• bhearsum: RelMan access to Balrog?