Releases/Fennec1.0maemo/Post Mortem

Agenda

What went not so well?

• Lack of communication between QA and other groups (aakash)
  • Starting with Beta 4, release dates weren't clearly communicated
  • bugs weren't filed, just talked about (john+)
    • Not enough visibility into time remaining for bugs
  • overworked?
  • bugs weren't acted on when triaged (bugs only worked on when assigned)
  • bugs ignored when they should be acknowledged
  • no backup for stewart
  • not enough test cases for every patch
  • no more meta bugs (tracking)
• Not enough communication between release and other teams, particularly around timelines
• Finding regression windows- time at the end of the year was dedicated to finding a regression after it's happened
  • need to find regressions quicker
    • disk cache parameter got turned on and took forever to find
    • automated perf testing
  • hard to know where we are in solving regressions
• lots of last minute time working on flash, risk should not be pushed towards the end of a release
• lead times on release engineering are long, took one year to set up mobile linux builds, tri-server took 6 months
• hard to ship 1.0 releases because you're doing everything at the same time- everything is priority
• Launching multiple OS's at the same time sapped resources
• Delayed early on due to unstable platform and hardware availability
  • N900's weren't easy to get until very late 2009/early 2010
  • Need to figure out how to get hardware faster if we move towards bundling
• Spent too much time blocking ourselves on stuff
  • Need to make decisions to cut features
  • Need to be realistic regarding perf/launch features
  • Lots of debate over choices which could be deferred and could be prioritized
• 1.0 was way too big
  • Prioritized features + quality over time
• Scheduling between desktop + mobile for firefox and weave was tricky
• Pressure to make last minute design changes as we approach 1.0 release
• unit tests were ignored
• Not sure which devices to support (n810 vs n900)
• Firefox processes for mobile may not have been the best fit
• L10n communication was abrupt and cantankerous, crankiness--, should have worked more closely with team to solve problems
• Releng didn't have a good sense at 1.0 to understand expectations for launch (signing, crosschecking)
• "Ship when ready" is hard on non-engineering teams
  • not enough warning on releases
  • unclear to releng what was in each release
• more communication between mobile and platform teams
  • tracemonkey drops would surprise mobile teams
  • winmo was affected for 2 weeks as a result- need automation
• wrong people at thursday release meetings
• Bugs get forgotten, and people don't necessarily know the ramifications of their changes
• no automated testing, but anecdotal evidence wasn't taken seriously
• didn't do a good enough job listening to users and support forum, particularly early adopters
• good "getting started" documentation for n900 but not a lot of FAQ info (how to set default browser, manage plugins, etc)
• Standards around good add-ons for mobile were fluid and hard for developers to understand

What went well?

• Firefox has never shipped a multilocale release, lots of cross pollination
  • learned a lot about shipping firefox on different distros which applies towards desktop builds
• we shipped!
• better than expected!
• built for quality instead of to a timeline
• made tough decision to remove flash for 1.0
  • turnaround was fast for youtube addon
• dramatic improvements in process and communication over the last month
• marketing videos were great for happiness, high leverage with community
• good press communication
• great attitude towards problem solving and dealing with crises
• dev group was entrepreneurial in the absence of releng, mktg, l10n etc in early days
  • those groups were understanding when introduced to team
• testing was great, amazing that we had 1 QA person
• versatile engineering team- "can-do attitude"
• first successful mobile firefox implementation
• everyone internalized the key goals of the project and kept that vision in mind

How will we change in the future? (themes for improvement)

• need communication/product human
• get technical contact at nokia for questions - stewart
• better product scoping of features, more tactical decision making around key features before spending too much time on them
• small prototypes of things more often
• better bug filing and triaging, goal of keeping things from slipping through cracks
• automated testing
  • find regressions faster, automated perf testing
• try to get more hardware as fast as possible or alter processes to mitigate for releng
• try to tune process to be more mobile specific
• cascade launch date decisions more effectively
• get add-ons quality improved, set UI standards and set expectations regarding good add-ons
• smaller meetings with the right people
• dealing with qualitative feedback
• no more major holiday cancellations