Firefox 33 Post mortem

Attendees: lmandel, sylvestre, jorge, marc, bsmedberg, kairo, tracy, ashuges, robert, lizzard, catlee

• OMTC issue
  • Our aurora/beta populations is not representative for graphics cards/drivers
  • In general we need a better way to communicate with our aurora/beta users. We have enough users, we just need to get better quality data
  • Some people might have thought "It is so big that someone is going to report it"
  • Having old machines with no up to date drivers to test during the cycle
  • (lmandel) More broadly, we need to come up with a way to get better information about potential graphics related issues. This may involve setting up a graphics lab, contracting another company that already has one, measuring beta and trying to improve the graphics coverage on this channel,...
    • Need about:support data submitted with feedback and support posts.+1
      • about:telemetry as well
  • We have been blind most of the process.
  • We couldn't turn OMTC off because of regressions and lack of testing
    • We should add a best practice here about testing features in on/off states
  • Need to do more testing on XP
  • Email process needs to be cleaned up.
  • (mschifer) lurking crashes like h264 and other noise we haven't seen these crashes
    • Can we expand the scope of what we look at
  • (bsmedberg) concerned about case where the browser is broken and we don't crash
    • need to be able to collect data
  • (bsmedberg) very poor signal of users just stopping to use beta
    • talked to metrics about getting some data (UA has ideas, can we get in on this meeting)
  • Turning off OMTC for one beta release killed our ability to see problems
  • (mschifer) spread out big feature landings in the same component
  • (cww) when we turn on/off a feature for one release it's hard to tell what the data is afterwards
    • heard about black screen issues after beta 6 but assumed it was related to disabling omtc
  • (cww) don't get feedback for startup crashes or anything that makes the browser unusable
  • (cww) very difficult to get about:support and other information when Firefox can't start
  • (bsmedberg) - making beta more representative by specifically targeting beta "invitations" to under-represented populations +1
    • In particular: geolocale/XP/older machines/poor network perf/corp-IT (or similar)
  • (cww) detect when people are installing on top of an existing installation
    • start in safe mode?
    • ask why they're reinstalling
  • (cww) can we tell if people are downgrading? bug 1091881
  • (cww) we can do outreach in advance if we expect something to break (OMTC-sized) but we didn't realize it was going to have the possible impact it would.

• Blacklist update broken
  • Nobody seems to understands it
  • We don't have tests
  • We should rewrite it
  • (lmandel) At the very least, there are serious limitations even if it's working and we don't see to have good ways to test changes to the list. bjacob has a bug on file and we should pursue a project the build a replacement. https://bugzilla.mozilla.org/show_bug.cgi?id=838845
  • We should be able to know which users a blacklist change will affect (in advance), based on their FHR profile data

• the pkix tests should be launched during the beta cycle

• Last minute patch with certificate - bug 1058812

• No way to properly test with OMTC = off.
  • For every new features, TBPL should run with pref = on & pref = off during the aurora & beta cycle

• we mixed two important changes at the same time:
  • OMTC = on
  • D3D11
    • I didn't even realize this until very late (Cww)

• Try on mozilla-release has some failures "because it is mozilla-release and tests are not planned for that"

 --> Not much that can be done about this in the short term given how Try is configured.

• (tracy) pulling the plug when we know we're shipping broken product
  • 33.1 handcuffed us
  • would it have been better to delay release a couple of days and push a build to beta (hindsight)

• QA doesn't have a good collection of machines to test things on
  • mschifer working on that

• Regressed bug 933733 (how?)
  • bug 937306 was supposed to fix this, QA called for help in verification, apparently didn't communicate well enough to UA about this.