User:Ioana.budnar/Community Test Run Proposal

From MozillaWiki
Jump to: navigation, search

Where we are

The Mozilla community has always been involved in improving Firefox. They report, investigate, fix, verify, triage bugs, test, offer feedback etc, and this work pays off.

The part of their work of interest here is the testing, an activity in which the community is currently only involved through testdays. Considering that testdays are events only happening one day per week though, and that they are usually focused on one or very few features, they are not enough.

Please take into account the fact that this proposal is referring to Firefox Desktop, although it can be applied to any other Mozilla products.

Where we want to be

The main point here is that we would like to involve the community more in testing Firefox. Litmus used to contain BFT and FFT test runs that were active for whole 6 weeks cycles. These test runs were most usually ran by the community, as an addition to the testing done by the Firefox QA team.

Considering that since the Litmus days Firefox has acquired multiple new features, and got many of the old ones changed, simply returning to the Litmus tests will not suffice.

Where we want to be is the place where we have at least one new community test run per channel (aurora, beta, nightly), a test run anyone in the community can run at any time, and that covers enough of Firefox functionality to offer us confidence in its quality.

Please note a test run per channel doesn’t necessary mean a different set of test suites per channel. The same test suites can be used for all three test runs, if suitable.

Roadmap

There are several steps we need to take to get to a good community test run:

1. Decide on the level of coverage the community test run(s) should hold.

2. Select subsets of functionalities for each kind of test run we want to create.

3. For each selected functionality

  • Decide what level of coverage is needed (smoke, edge cases etc)
  • Create a corresponding test suite in MozTrap.
  • If it has test cases in Litmus, review those test cases and decide whether they should be imported to MozTrap or not.
  • Create the (rest of the) test cases in the corresponding MozTrap suite.

4. Create a test run containing all the test suites from step 3

  • Should be active for all the 6 weeks the Firefox version is on a certain channel.
  • The list of operating systems offered for this run should be very diverse (try to cover especially the most used OSs not officially tested by the QA team).

5. At each merge, create new test runs by cloning the initial ones and editing the test suites list as necessary (e.g. remove the suite for a feature that was backed out, add test suites for new features, old features that just passed through critical changes etc).

Ideally, we want to involve the community in all the above steps, not just hand them a test run. The help of component-owner developers is also important, especially for step 2. Ownership for each step should be as follows:

  • 1, 2, 3 (first bullet) - All Firefox teams willing to get involved + community.
  • 3 (second, third and fourth bullets) - QA team + community.
  • 4, 5 - QA team.

Results

Feedback

Please add here your feedback, questions, concerns etc.

Retrieved from "https://wiki.mozilla.org/index.php?title=User:Ioana.budnar/Community_Test_Run_Proposal&oldid=658177"