Releases:Release Post Mortem:2016-02-03

From MozillaWiki
Jump to: navigation, search

Meeting Details

« previous week | index | next week »
< most recent | upcoming >

Release Duty

  • FF 45 cycle: mtabara



Firefox/Fennec_44.0 (jlund/rail/mtabara)

  • Fennec 44.0 build2 went smoothly
  • pushed desktop Firefox 44.0 build1 to beta channel.
  • Some intermittent failures came up on buil1:
    • repack_10/10 complete for Firefox 44.0 build1 on linux, re-triggered
    • Firefox 44.0 build1: failed at repack_7/10 on win32, retry
    • merge mozconfig issue with mozilla-release and tools whitelist not being updated
      • resolved: Bug 1233262 - add ENABLE_RELEASE_PROMOTION variable to sanity whitelist for mozilla-release
  • There's a follow-up build2. It might be a 43 m-r based release with selectively uplifted sec fixes, not yet decided - awaiting more data on crash-rates IIUC
  • Some intermittent failures came up on buil2:
    • firefox_source failed for Firefox 44.0 build2 - failed to checkout mozilla-release. jlund suspected this is related to scl3 network issues mentioned in #moc, retried
    • Firefox 44.0 build2: failed at update_verify_beta_2/6 on linux64 - gtk3 errors that can be ignored
    • all update verify tasks in beta for linux/linux64/macosx64 failed mostly because of binary files found in diff - could be that we built off of m-b. Tracking bug for this is bug 1241951
    • desktop ff - no antivirus? no email. not in pending/recent/running.
      • there is reason to believe antivirus was never ran because of infra issues/downtime bug 1241334 that overlapped with the timing of 44.0 gtb
      • manually re-triggered to force build outside of the release graph
      • initially manual re-triggering failed for a downloading issue, retriggered that once again
  • Firefox 44.0 build3. usual update verify gtk errors but flawless victory
    • bhearsum manually synced the SDKs to the releases dir due to bug 1243740 with local patch applied:
  • Firefox 38.6.0esr build1 is too on fire and going well in the pipeline
    • shipped. flawless victory

Firefox/Fennec_45.0b1 (jlund/rail/callek/nick)

  • TODO sum-up, where do we stand with the GTK3 known issue?
  • gtk3 not supported via buildbot based builds
    • solution: Bug 1242641 - GTK+3 still not working for buildbot builds on beta
  • bhearsum manually synced the SDKs to the releases dir due to bug 1243740 with the local patch applied:
  • nthomas manually ran checksums/push-to-mirrors because source builders failed
  • jlund forced antivirus for the same ^ reason
  • Bug 1243448 - beta release source builder failing to run configure
    • status: resolved for beta1. needs follow up
  • nthomas re-ran checksums (again) because we fixed source builders after above re-triggers and so source artifacts were missing. also missing, SUM and KEY bits which he manually pushed
    • [Bug 1243873] New: KEY file missing for releases

Firefox/Fennec_45.0b2 (jlund/rail/callek/nick/mtabara)

  • gtk3 known issue errors in update verify steps
  • flawless victory


Thunderbird_45.0b1 (jlund/rail/callek/nick/mtabara)

  • TODO: awaiting antivirus check and it's all good to release
  • we had two win32 repacks failing
    • failed at repack_6/10 on win32 - retriggered, intermittent timeout
    • failed at repack_2/10 on win32 - retriggered
      • retriggered upon 'da' locale failed while submitting to balrog, specifically around the script
      • retriggered upon loosing slave instance
      • retriggered upon timeout

Merge week

  • migration went smoothish
  • bug 1242892 - [Fennec] Wrong branding for Aurora 46.0a2 on API11 builds
    • resolved


Action items