Post Mortem Notes
The following notes are a collection of feedback that the QA team has faced throughout the Firefox 3.0 development timeline.
What worked
- 8 alphas, 5 betas, and 3 RC builds. With 16 builds leading up to the final release, i felt there was lots of adequate testing on solidifying the release. [tchung]
- Colloborative efforts by build and QA team on each milestone release. As we had more, we moved quicker and more careful on our testpasses. [tchung]
- Having a blocking-firefox3 or blocking1.9 flag was key to getting the right bugs triaged. Knowing they were getting triaged each day was comforting that they werent just sitting in the queue. [tchung]
- Weekly firefox meetings with a structure and progress report for each team. [tchung]
What didnt work
- bug 430217 - Reporter shouldn't add a toolbar button to Firefox; is a good example of how poorly we handled theme refresh planning. All along I was raising the issue that the Reporter icon does not look right since it wasn't styled on the Mac theme. It was only a day or so ago that they won't fixed the bug to style that icon, so now in order for the theme to look right I feel it needs to be removed. But as you can see from Axel's comment he is not happy because of string changes. [marcia]
- Not having an up to date icon refresh document that is up to date makes it almost impossible for QA to figure out what is supposed to be correct in the UI regarding icons, such as the example cited above. Also I found out yesterday in reading bugmail that we will not have a crash reporter icon for final ship (currently it is the same as software update, which is confusing. [marcia]
- Every time a theme drop changes on the Mac, I don't have a clear picture of what is supposed to change. The find bar looks different today, but since I have no schema to go by, I am unsure as to whether the changes are expected or not. This results in possible unnecessary bugs, such as the one I filed the morning about the window sizer in Places (bug 430493) where I am not sure whether that is an intended change or a regression. [marcia]
- Some community members such as adelfino are obsessed with these minor UI changes and are clogging up bugzilla with hundreds of bugs regarding UI issues, especially on Linux on Mac. Many end up being duplicates and consume both community and QA time to resolve. [marcia]
Things to improve on
- Earlier interfacing with developers and testers. More cross-knowledge sharing of architecture and feature knowledge, so questions are asked earlier, testplans worked on earlier [tchung]
- Producing more tryserver builds earlier on. Helps QA verify fixes in a more 1-1 setting. [tchung]
- Keep the PRD up to date! An outdated PRD for certain sections, kept QA from knowing what new features were landing and obscuring out testplans. [tchung]
- Also, having a better ETA on dates when these features were going to land would help QA know when to expect them in builds [tchung]