Firefox/Channels/Postmortem/33

From MozillaWiki
Jump to: navigation, search
« previous release | index | next release »


Firefox 33 Post mortem

Attendees: lmandel, sylvestre, jorge, marc, bsmedberg, kairo, tracy, ashuges, robert, lizzard, catlee

  • OMTC issue
    • Our aurora/beta populations is not representative for graphics cards/drivers
    • In general we need a better way to communicate with our aurora/beta users. We have enough users, we just need to get better quality data
    • Some people might have thought "It is so big that someone is going to report it"
    • Having old machines with no up to date drivers to test during the cycle
    • (lmandel) More broadly, we need to come up with a way to get better information about potential graphics related issues. This may involve setting up a graphics lab, contracting another company that already has one, measuring beta and trying to improve the graphics coverage on this channel,...
      • Need about:support data submitted with feedback and support posts.+1
        • about:telemetry as well
    • We have been blind most of the process.
    • We couldn't turn OMTC off because of regressions and lack of testing
      • We should add a best practice here about testing features in on/off states
    • Need to do more testing on XP
    • Email process needs to be cleaned up.
    • (mschifer) lurking crashes like h264 and other noise we haven't seen these crashes
      • Can we expand the scope of what we look at
    • (bsmedberg) concerned about case where the browser is broken and we don't crash
      • need to be able to collect data
    • (bsmedberg) very poor signal of users just stopping to use beta
      • talked to metrics about getting some data (UA has ideas, can we get in on this meeting)
    • Turning off OMTC for one beta release killed our ability to see problems
    • (mschifer) spread out big feature landings in the same component
    • (cww) when we turn on/off a feature for one release it's hard to tell what the data is afterwards
      • heard about black screen issues after beta 6 but assumed it was related to disabling omtc
    • (cww) don't get feedback for startup crashes or anything that makes the browser unusable
    • (cww) very difficult to get about:support and other information when Firefox can't start
    • (bsmedberg) - making beta more representative by specifically targeting beta "invitations" to under-represented populations +1
      • In particular: geolocale/XP/older machines/poor network perf/corp-IT (or similar)
    • (cww) detect when people are installing on top of an existing installation
      • start in safe mode?
      • ask why they're reinstalling
    • (cww) can we tell if people are downgrading? bug 1091881
    • (cww) we can do outreach in advance if we expect something to break (OMTC-sized) but we didn't realize it was going to have the possible impact it would.
  • Blacklist update broken
    • Nobody seems to understands it
    • We don't have tests
    • We should rewrite it
    • (lmandel) At the very least, there are serious limitations even if it's working and we don't see to have good ways to test changes to the list. bjacob has a bug on file and we should pursue a project the build a replacement. https://bugzilla.mozilla.org/show_bug.cgi?id=838845
    • We should be able to know which users a blacklist change will affect (in advance), based on their FHR profile data
  • the pkix tests should be launched during the beta cycle
  • Last minute patch with certificate - bug 1058812
  • No way to properly test with OMTC = off.
    • For every new features, TBPL should run with pref = on & pref = off during the aurora & beta cycle
  • we mixed two important changes at the same time:
    • OMTC = on
    • D3D11
      • I didn't even realize this until very late (Cww)
  • Try on mozilla-release has some failures "because it is mozilla-release and tests are not planned for that"
 --> Not much that can be done about this in the short term given how Try is configured.
  • (tracy) pulling the plug when we know we're shipping broken product
    • 33.1 handcuffed us
    • would it have been better to delay release a couple of days and push a build to beta (hindsight)
  • QA doesn't have a good collection of machines to test things on
    • mschifer working on that
  • Regressed bug 933733 (how?)
    • bug 937306 was supposed to fix this, QA called for help in verification, apparently didn't communicate well enough to UA about this.