QA/Platform/Graphics/Features/Planning

Summary

As part of Mozilla's Quantum initiative, the Graphics team is interested in developing a feature qualification process based on the model used by the Electrolysis team. The general idea is to have a well-documented, clear path to success for any Graphics feature that is part of this initiative. The goal is to ship major features without losing users.

Owner

The development of this model is being driven by Anthony Hughes (:ashughes), the QA Engineer for the GFX team.

Status

This project is currently in the Exploratory Phase, expecting to be completed by July 31, 2016.

Exploratory Phase

The purpose of this phase is to consult with those involved in the Electrolysis project to gain awareness of the pros, cons, and evolution of the release model.

Discussion Notes

• Clearly define the KPIs for shipping a feature (eg. performance, stability, correctness, etc) [1][2]
• Democratize decision making and information needed to make decisions, this will enforce quality and prevent high-risk changes from backfiring.
• Clearly document what does and does not block shipping a feature
• Clearly document any flaws in the data being used to make decisions, people need to be able to trust the data
• Before shipping a feature be able to clearly answer, Are we going to lose users if we ship? If the answer is a clear no then you can ship. The e10s team used dau:mau ratio as a key metric to answer this question.
• Conduct many experiments to (dis)prove theories [3]
• Resist the temptation to increase scope or uplift risky changes given a longer release cycle, treat it as if you still just have 6 weeks.
• Make sure automated testing has a clearly defined role in the release criteria.
• Make sure status is clearly and frequently communicated to all stakeholders [4]
• When conflict breaks out between stakeholders break that out into a mediated meeting, this creates a safe space to work things out and prevents arguments from making the project look bad.
• Raise red flags early, often, and clearly
• System addons can be used to both experiment on Beta and release to a specific sub-population
• Clearly define owners/approvers for each feature and quality area, e10s team used the RASCI Responsibility Matrix
• Clearly define and track metrics for users utilizing your feature and users still on the old path
• Clearly define when a feature is good enough to ship to millions of users and how each bug gets you closer to that goal, not doing this can lead to developer stress/burnout