Releases/Post-mortems/Firefox 17.0

From MozillaWiki
Jump to: navigation, search

l== Schedule / Location / Call Information ==

  • Tuesday, 12/4/2012 @ 9:00 am PT
  • Physical spaces: MTV-3G, SFO-7N, TOR-5G
  • Vidyo Room: Release Coordination

Other communication channels

  • irc.mozilla.org #planning for back channel
  • This etherpad for real-time notes (we'll copy the wiki page in at the beginning of the meeting, and out at the end)

Previous Actions

  • RelMgmt - what disabling an Android product means, and a proposal for future instances (re-building the prior version with a higher buildid) - QA & RelEng scheduling impacted
    • Good amount of work, we can keep product 'disabled' if we know the fix -- options still needed for if we don't know the fix?
    • Alex pinged Karen/Erin - need their input -- push to next post mortem to get mobile product input
  • All - before pulling the trigger on any releases, we need to make sure to have one final conversation (double check with security team, infra, make sure people are in the know about the impending release)
  • johnath - review of release-drivers for people who may not be appropriate to see
    • some removals, more emails sent
    • tabling to next post-mortem

Talking Points

Timeline

17.0

  • GTB for 17.0 build #2 Mobile 11/14
  • Shipped Mobile 17.0 on Monday 11/19
  • Shipped Desktop 17.0 10am PT 11/20

Release Issues

  • Hit bug 787373 – Aurora download page needs updating to point to Fx17 with Mobile -- cleanup of old builds too soon (discovered with bug 814469)
  • Runtime change for Linux - should have had a dev.platform thread, and possibly included it in the 10.5 desupport hotfix. These bugs should be nommed for tracking in the future to help prepare earlier. [Lukas will file a metrics bug to make pulling data about versions/supported libs on platforms easier next time - many thanks to Nthomas]
  • What can we do to just push at 8am the day of the release and just have it out there? What do we need to cut, prepare in advance?
    • Getting to mirrors -- we try to hold back on pushing to mirrors until late for final releases but can we get some testing that it's difficult to pull off CDN, can we find out what it would take?
    • Documentation of how to remove from CDN (regardless if we move to automatic mirror pushing or not)
    • Future Timeline (starting with FF18):
      • We go to build with our RC morning PT the Friday before
      • We'll get builds & updates available on Saturday (on betatest)
      • QA starts functional testing on betatest Sunday overnight & Monday
      • QA conditionally Signs off Monday EOD
      • updates are not available on releasetest until push to mirrors occurs (releasetest and release are same snippets, betatest has slightly different snippets)
      • Push to mirrors should happen on Monday (day before release)
      • releasetest testing occurs after push to mirrors and overnight
      • Pushing snippets at 6AM PT
      • Start testing on the release channel at ~7AM
      • Final sign-off by 8AM
  • Spun 17.0 build2 because of binary changes (not caught by relman/eng) so have filed bug bug 813809 – Implement an IDL checking hook on mozilla-beta and mozilla-release repos to refuse binary changes without specific approval to try and catch this much earlier next time - plan to enable on mozilla-beta and mozilla-release and mozilla-esrN repos
  • ARMv6 complaints in Google Play are being followed up on by mobile QA for resolution in the FF18 timeframe

QA

  • (ashughes) Proposal to test updates providing better coverage, quicker turnaround, and less confusion about what to test:https://intranet.mozilla.org/User:Ahughes@mozilla.com/Release-Updates-Proposal
  • (ashughes) nit: I'd like to see the "esr updates available on betatest" email to read "esr updates available on esrtest" to reflect the actual channel name [file bug in releng, cc bhearsum]
  • (ashughes) QA is trying to find ways to do better web compatibility testing
    • dedicating a particular Beta every release to web compatibility testing
    • investigating what test suites we have at our disposal to be checking a vast array of sites
    • Juan points out that the Chrome team hires an "army" to check the top-1000 websites before release
    • Cheng: need to know about these changes earlier
    • Alex: triage known broken sites into bugs
  • Regardless of Rapid Beta - we need to have:
    • enough QA infra to handle multiple release testing automation in parallel
      • A-team is getting close to having an ESX cluster online in the colo
      • In theory we can test serial faster or parallel -- goal for Ateam Q4
    • tie-in to Releng automation so that once mirrors are absorbed, QA mirror testing can be kicked off automatically
  • Release Notes/Web Content [marcia]:
    • Release Notes have been missing bug links in two releases (15 and 17), and some other web content had incorrect information- Dynamis has sent emails - what can do to improve in this area?

What web content had incorrect info?

  • buglist page is not found (linked from the releasenotes):

https://www.mozilla.org/en-US/firefox/17.0.1/releasenotes/buglist.html

  • Incorrect bug number on "Bug 815042 - leaving private browsing with social enabled doesn't reset all social components (edit)"
  1. Please *read* and also *test* the Preview that is sent out before release/notes publishing (RelMan does send out fully functional staging notes)
  2. [Action] RelMan can look at adding some automated web QA script to check links for releasenotes (talk to Stephen Donner - they have a test suite already, get the trigger)
  3. [Action] Release checklist page (RelMan) for product-details, fixing esr download page

RelEng

  • Can we nail down whether "Push to mirrors" can be done automatically? All that would be necessary seems to be to verify that we're able to successfully pull an update off the wire (bug needed?) (already discussed, action below)

Security

Marketing & Press

  • [Laura is in another meeting, here's an update] We didn't know about launching What's New page until last minute, but we did find out in time. Phew - thanks for being flexible. Earlier would have been better but there was no harm done here since we found out we didn't need it in time.
  • [Mark] No additional updates to share

Support

  • UA change was a suprise. We could have mitigated some of the issues through studies earlier in the cycle.
  • Azure landed in b4 which is too late for a major change like this. We also need a headsup if something like this lands.
    • Only a pref-off, not a new landing
    • Should have been getting testing for both on/off in the beta channel
  • What could we have done to mitigate the add-ons manager breakage in the beta channel?
    • Was disabling TMP the right choice?

Other

17.0.1

Drivers:

  • 815743 – Backout the UA changes in bug 588909 and its follow-ups from all branches
    • Ehsan prepared backouts
  • 814101 – Font Issue in FF 17

Ride alongs:

  • 795104 – crash in _VEC_memcpy | js_NewStringCopyN
  • 806192 – crash in inDOMUtils::GetBindingURLs with Firebug and ShareMeNot
  • 815042 – leaving private browsing with social enabled doesn't reset all social components
  • 813264 – 809865 or Bug 810726 broke Skype click-to-call (ESR17 landing only, already in 17.0)
  • 814554 – Firefox 17 silently stops processing permissions.sqlite when rejecting rules valid under Firefox 16
  • 815359 – Bing Maps v7 Ajax Pushpin Incompatibility

Other:

  • 813763 – Menu items like Addons Manager not working in FF 17 due to Tab Mix Plus
    • Was fixed by addon developer pushing update

Timeline:

  • 17.0.1 & 17.0.1esr GTB Wednesday EOD, Pacific
  • QA signed off Thursday EOD, Pacific
  • Shipped Friday am PT

Notes

Actions

  • akeybl to follow up on 17.0->17.0.1 uptake based upon the chunk size and update ping changes
  • Bhearsum to follow up on smarter scripts/address the issue of cleanup scripts with regards to Aurora downloads when we merge
  • Lukas will file a metrics bug to make pulling data about versions/supported libs on platforms easier next time
  • joduinn - to follow up with IT about CDN pull experiment, so that we're no longer afraid to push to mirrors (and please get documents on the process of doing so regardless of pain)
  • ashughes - make sure that new locales for FF19 won't break testing (more of a reminder)
  • [Action] RelMan can look at adding some automated web QA script to check links for releasenotes
  • akeybl to follow up on pipeline of broken site (cww) -> QA (ashughes) reproducibiliy/regression finding -> TE (who?) or Engineering (include Lawrence). We also need to decide on what constitutes a bad regression
  • [Action] RelMan (lukas) can look at adding some automated web QA script to check links for releasenotes (talk to Stephen Donner - they have a test suite already, get the trigger)
  • [Action] Release checklist page (RelMan) for product-details, fixing esr download page
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Post-mortems/Firefox_17.0&oldid=496561"