Firefox Waverley/QA Postmortem

The purpose of the meeting is to identify any areas where we can improve our QA process for future releases, with Firefox 5 as an example. We'll focus on communication, testing approach and mechanics, and any other feedback you would like to provide.

1. Who: QA/Waverley Leads
2. What: What worked, what needs improvement, suggestions for changes that might make us more efficient

Communication

• What went well? What could improve?
  • Waverley: Desktop :
    • Week's highlights on etherpad are no longer being set
  • Waverley: Fennec :
    • Went well : Task for each build/channel and the adaptability to the new rapid process
  • Waverley: Automation
    • Went well : having henrik in our timestamp helps a lot. Also webQA team is very handy in communication, all Mozilla people are available constantly. Henrik said he has some ideas about improving communication. Communication can be improved
• Are we communicating frequently enough?
  • Waverley:Desktop
    • We could set another short meeting to discus specific issues (either by telephone or on the IRC)
  • Waverley: Fennec :
    • We consider that we could talk more, almost daily when a major task is given
  • Waverley: Automation:
    • We must communicate frequently with other from the automation teams. Desktop team misses the meeting, i don't like this.
• Was turn-round time for resolving issues or questions adequate?
  • Waverley:Desktop
    • Yes - IRC helpful with this (ie: When Litmus went down it was easy and fast to get in touch with tomcat to solve problem)
  • Waverley: Fennec :
• Are the topics we discuss in meetings the right set of things?
  • Waverley:Desktop
    • No - more focus on specific issues (what is blocking the work, what are the plans for the next period, what was discussed in other meetings and is of interest for everybody)
  • Waverley: Fennec :
    • We encountered some issues regarding the questions posted - unanswered ones or the follow up did not arrived
• Are the media we are using appropriate for the kind of information we share? (wiki, etherpads, irc, ...)
  • Waverley:Desktop
    • Yes - wiki pages need to be updated more often (ie: most of the feature pages haven't been updated since they were created)
  • Waverley: Fennec :
    • we consider that they are appropriate
  • Waveley: Automation
    • we get many info from the sync pad, rather than wiki pages. sync pads are very thorough

Planning

• What went well? What could improve?
  • Waverley:Desktop
    • Try to avoid 2 releases at the same time
  • Waverley: Fennec :
    • Test plans and the assignments
• Are tasks well defined and in the right places when you need them?
  • Waverley:Desktop
    • Overall yes - wiki pages need more organizing and clean-up so as to have the information more centralized
  • Waverley: Fennec :
    • Yes. they can be find easily
• Are assignments clear?
  • Waverley:Desktop
    • Yes - more pressure on devs to set flags for Firefox 6,7... when they upload patches so as for us not to leak bugs in the bug verification process
    • More organized wiki pages could lead to clearer assignments
  • Waverley: Fennec :
    • Mostly yes, There are some task without dates that can be improved.

Testing Approach and Mechanics

• What went well? What could improve?
  • Waverley: Desktop
    • We should improve the way in which attention is drawn to issues that are validated from the unconfirmed bug triage and from our own exploratory testing
  • Waverley: Fennec :
    • went well : tasks divided over the channels
    • can be improve : quicker response to our questions (exploratory, litmus, ownerships)
• Are we focusing on the right tasks per channel and milestone?
  • Waverley: Desktop
    • More emphasis on new features especially now that their number is increasing
    • Bug verification should be performed on all channels on which the patch has landed
  • Waverley: Fennec :
    • More attention for the features being in development
• Which tasks have the most impact? Which have lesser impact?
  • Waverley: Desktop
    • Less impact: Updating testcases in Litmus and vetting results - because less and less people are using Litmus in their testing activities
    • Greater impact: Work on new features
  • Waverley: Fennec : same as desktop
• What should we focus on for the next releases?
  • Waverley: Desktop
    • Thorough sign-off on features
    • Offer more support for x64 builds from the QA point of view
  • Waverley: Fennec :
    • Deeper testing for new features

Other Feedback

Take Aways

During the meeting we went through each of the sections and elaborated on the comments posted by the Waverley team. Among the topics discussed, some of the highlights where we could use improvement were:

• Come up with a centralized wiki structure that allows people to navigate to the right milestones easily.
• Establish a sort of office hours, where people from Mountain View can be available at certain times of the week, via IRC or Skype, in real time.
• Avoid reading status reports. Talk about outstanding issues, task planning, and talk about information given out in meetings other people cannot attend. Use meeting time to raise outstanding issues that aren't getting traction.
• Avoid scheduling more than one release at the same time, which is doable but tough. If we do need to turn around two releases or more, have a clear list of priorities.
• Some issues do not get traction. So how do we draw attention to new issues so they don't get lost or stagnant in the New state?
• There's a lot of effort going into vetting results and revising test cases in Litmus, but that is as valuable as the number of people using the system. We should keep promoting its use, such that the amount of work going into maintaining it does bear fruit.
• Make it clear how to use flags and how to mark verified bugs.
• During milestones, include feature sign-off tables.

One recurrent theme during the postmortem was the organization of wikis and the need to updating in a timely manner. Information is not easy to find or navigate and a lot of it is outdated. In addition, MV office hours might help clear any issues for weekly tasks. Communication and dissemination of information is key.

In general, we really appreciate the initiative and growing autonomy of the Waverley team as well as their direct involvement in new features, by attending providing feedback in feature meetings for example.