Releases/Fennec1.0maemo/Post Mortem

Agenda

What went not so well?

• Lack of communication between QA and other groups (aakash)
  • Starting with Beta 4, release dates weren't clearly communicated
  • bugs weren't filed, just talked about (john+)
    • Not enough visibility into time remaining for bugs
  • overworked?
  • bugs weren't acted on when triaged (bugs only worked on when assigned)
  • bugs ignored when they should be acknowledged
  • no backup for stewart
  • not enough test cases for every patch
  • no more meta bugs (tracking)
• Not enough communication between release and other teams, particularly around timelines
• Finding regression windows- time at the end of the year was dedicated to finding a regression after it's happened
  • need to find regressions quicker
    • disk cache parameter got turned on and took forever to find
    • automated perf testing
  • hard to know where we are in solving regressions
• lots of last minute time working on flash, risk should not be pushed towards the end of a release
• lead times on release engineering are long, took one year to set up mobile linux builds, tri-server took 6 months
• hard to ship 1.0 releases because you're doing everything at the same time- everything is priority
• Launching multiple OS's at the same time sapped resources
• Delayed early on due to unstable platform and hardware availability
  • N900's weren't easy to get until very late 2009/early 2010
  • Need to figure out how to get hardware faster if we move towards bundling
• Spent too much time blocking ourselves on stuff
  • Need to make decisions to cut features
  • Need to be realistic regarding perf/launch features
  • Lots of debate over choices which could be deferred and could be prioritized
• 1.0 was way too big
  • Prioritized features + quality over time
• Scheduling between desktop + mobile for firefox and weave was tricky
• Pressure to make last minute design changes as we approach 1.0 release
• unit tests were ignored
• Not sure which devices to support (n810 vs n900)
• Firefox processes for mobile may not have been the best fit
• L10n communication was abrupt and cantankerous, crankiness--, should have worked more closely with team to solve problems
• Releng didn't have a good sense at 1.0 to understand expectations for launch (signing, crosschecking)
• "Ship when ready" is hard on non-engineering teams
  • not enough warning on releases
  • unclear to releng what was in each release
• more communication between mobile and platform teams
  • tracemonkey drops would surprise mobile teams
  • winmo was affected for 2 weeks as a result- need automation
• wrong people at thursday release meetings
• Bugs get forgotten, and people don't necessarily know the ramifications of their changes
• no automated testing, but anecdotal evidence wasn't taken seriously
• didn't do a good enough job listening to users and support forum, particularly early adopters
• good "getting started" documentation for n900 but not a lot of FAQ info (how to set default browser, manage plugins, etc)
• Standards around good add-ons for mobile were fluid and hard for developers to understand