Post-mortems:2.0.0.2

What worked

How did the process go – what did we plan to do, what did we actually do? What did you like about the process?

Things we did well/better:

• Improved communication with build team to minimize QA waiting time.
  • discussions about build availability,
  • understanding the build process
• Worked more closely with dev. to understand and test Vista changes (rstrong and sspitzer did an awesome job!)
• Verified bugs sooner than later... at least the ones we could with testcases or through regression testing.
• Involved more community members by creating Vista users list and holding our regular test days
• Spent time testing top extensions and plugins (something we have overlooked in the past)
• Involved web-dev and product teams early to ensure that release day went smoothly with release notes and website changes.
• Steps for verifying and reproducing bugs were well communicated
• Tons of reminders from Jay about what is left to be done
  • about bugs that are not resolved for this release
• RC's on beta channel (for 2.0.0.3)

Things we need to improve

• Changes such as Bug 366113 should not silently land on branches.
• For every integration late in the release (after freeze) and for all the major groups of changes... make sure that there is good communication between developers and QA to ensure thorough focused testing.
• More input and testing from developers: if a testcase is not provided and QA is not able to create one, developers should either create them or think of cases that might break and let QA do exploratory testing around the fix.
• Evaluating risks that might not be apparent from looking at bug patches and descriptions (have dev. spend a bit more time *thinking* about what their fix might break)
• Build automation/turnaround time... this is in the works and hopefully will make for a smoother release/QA handoff for each RC.
• Better delegation of QA tasks, buy-in from other QA members to own certain areas (addons/plugins, topsite, js, partner, etc)... so 1-3 people are not trying to do everything at once.
• triage mtg annoucements - sometimes unclear when team was meeting
• Need better communication to IT and other interested parties about planned release dates
  • Week in advance
  • Immediate feedback if plans change
• Shouldn't deliver big features (Vista) into a small release
• Big code changes landing late in the release
• Could not adapt quickly when security bug came out because of the inertia of big release
• Need more input from developers on definition of change
  • To help QA do focused testing
• Better group understanding about why there are times when we can/cannot release during the day
• Not enough focused energy upfront on feature items like the G/Y switch
• Need basic system requirements for Win98
• Need clearly documented QA environs for clean/unclean testing on all platforms
• Make sure we do release planning with all affected functional teams
• Better review of release notes
  • revised 5 times!
• Build configs are error prone
  • Build configs not reviewed by anyone outside Build and mistakes were not caught
  • Either better review or improve autogeneration
  • Version numbers and locale errors not caught
• Code compelte date moved frequently
  • Code complete date was fuzzy by about 2 weeks
  • Build was in a holding pattern for too long
  • This was due to feature driven release (Vista) rather then date driven release.
  • Code complete date moved 2-3 times - too much
• Laked wider public testing for web regressions
• Developers not paying attention to dates
  • Some patches languished with no communication to the release team.
• Make release/build bugs and features part of release planning
  • Hot fixes to release automation changes not well baked
• Consistent notification about releases

What do we do from here

• Public RC tests as standard procedure
  • Do we have wide enough participation?
    • ~70K people updated to this RC!
    • Previously 50K-100K was what we used to consider a good number for beta testing
    • Average number of nightly testers is 10K
    • Need to determine if all locales are appropriately represented
  • Coordinator: Jay
• More proactive notification about releases to all the parties
  • Need to define the sets of interested parties and which notification methods to reach them
  • Coordinators: Basil/Schrep
• Planning for any new features in dot releases needs to be explicitly planned
  • Past major features in a maintenance release: Vista, major updates, UB)
  • Coordinator: Schrep
• Release mgr having clear escallation path to ensure bugs are landed on time for code freeze
  • Coordinator: Schrep
• Triage process/Better flag mgmt
  • Need to distinguish clearly between wanted and blocks release
  • Coordinator: Jay/MConnor
• Automated testing on the branches
  • Get backbase, MS, etc running on branch builds
  • Coordinator: Schrep

Other Ideas for change:

• Throttle back changes to the branches, be more strict with blocker/wanted criteria

1. this has not really been an issue, since dveditz's research has shown that most of the "extra" stuff has not been the cause of regressions
2. still, this sometimes takes away from QA resources to test/verify more bugs.
3. we might be better off with less bug fixes and more time for focused testing on a few critical areas

• Experiment with QA team assignments and approach to bug verifications

1. Test execution team can handle the majority of bugs with solid testcases.
2. More experienced/technical QA folks can follow up on bugs that are difficult to reproduce or need testcases (will be able to work better with dev in understanding the root of the issue)
3. 1-2 people focused on compatibility issues (topsites, ajax, addons/plugins, etc)
4. Create a specific test plan for the release based on the bugs that are going in. I have done this for the 2.0.0.3 release. It involved following up with the developers and getting more specific info about what to test [marcia]

Interested Attendees

• reed
• Axel
• Timr
• Jay
• rhelmer
• robcee
• marcia
• tracy
• tchung
• Tomcat