QA/Desktop Firefox/Automation/Commit Policy

From MozillaWiki
Jump to navigation Jump to search

Overview

The following document describes the commit policy for patches into the mozmill-tests repository and proposals for improvement of the policy.

Current Issues

1) Failure bug filed after first occurrence

  • PROBLEM: Creates bug churn due to uncontrollable circumstances (ex. internet outage)
  • SOLUTION: Only file a bug after 2 consecutive days of failure

2) Patch lands across all branches

  • PROBLEM: Creates considerable work when a patch fails in the real-world
  • SOLUTION: Land patch on default and let it prove itself. Once proven:
    • a) Let it ride the train, merge every 6 weeks, or
    • b) Port to all other branches, or
    • c) Port to one branch at a time and let it prove itself each step of the way

3) Patch must pass through a gatekeeper before landing

  • PROBLEM: Cost to quantity does not net a reciprocal benefit in quality
  • SOLUTION: No more gatekeeper reviews. Patch lands as soon as it passes peer review.

4) Patch passes through a single gatekeeper

  • PROBLEM: This does not scale at all
  • SOLUTION: r+ patches go into a checkin-needed queue which anyone with commit access can check-in

5) Only a chosen few get commit access

  • PROBLEM: This does not scale either and creates a huge barrier to contribution/responsibility/growth
  • SOLUTION: Open committer access to those who have "proven" themselves -- need to determine that criteria

Proposal

1) File failures after second consecutive occurrence.

  • WIN: Reduces bug churn due to glitches and uncontrollable circumstances, like internet connectivity.

2) Disable tests which are failing immediately upon filing the bug. In other words, file a bug with patch in hand to disable the test.

  • WIN: Increased day-to-day stability in our tests

3) Remove the necessity of a gatekeeper-review before check-in

  • WIN: Increase quantity, little decrease in quality

4) Only land patches on default initially. This will alleviate the problem of having to backout patches across multiple branches. Here are a couple of alternatives we can follow:

  • a) Check-in on default, let it ride the train
  • b) Check-in on default, port it up one channel per day
  • c) Check-in on default, port to all branches

Draft Commit Policy v1

Based on peer feedback to the above

Members with Commit Access
  • Anthony Hughes (Lead)
  • Vlad Maniac (Softvision Lead)
  • Alex Lakatos (Softvision 2nd)
Reviews

Every patch must pass one peer review meeting the following criteria:

  • Meet style requirements of Styleguide for readability to newcomers
  • Meet stability requirements given a full testrun on Windows, Mac, and Linux
    • Endurance tests need pass 100 iteration / 10 entity testruns
  • Meet coverage requirements of Litmus Test (new tests)
  • Appropriate usage of APIs where available
Check-in

Patches land on DEFAULT first (applies to new tests only)

  • Once a patch passes review, land it on default
  • RESOLVED FIXED if it passes the next daily testrun, backed-out if not
  • Land patch on aurora, beta, and release once it passes
  • VERIFIED FIXED if it passes the next daily testrun on all branches, backed-out if not

Check-in Responsibility

  • If developer has commit access: peer reviews the patch, developer checks it in
  • If developer has no commit access: peer reviews the patch, peer checks it in
Landing Flags
  • status-firefox<N>:?, unaffected, affected, wontfix, fixed
    • ?: ask for triage for a specific branch
    • unaffected: patch does not need to land for a specific branch
    • affected: patch needs to land for a specific branch
    • wontfix: patch will not be landed for a specific branch (bug won't be fixed)
    • fixed: patch has landed for a specific branch and passes the daily testrun
  • <N> based on whatever version is in the branch, as of Dec 12, 2011...
    • status-firefox8 means mozilla-release branch
    • status-firefox9 means mozilla-beta branch
    • status-firefox10 means mozilla-aurora branch
    • status-firefox11 means default branch
Points

Use points to track development progress

  • Every test/fail starts with 1 point
  • Each blocker adds 1 point to the test/fail
  • Each REOPENED bug adds 1 point to the test/fail
  • Examples:
  • Could potentially be used as onramp to commit access (ie. 100 points)
Shared Module Dependency
  • file a dependency bug
  • write patch for shared module needs
  • reviews should go to Automation Services team
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Desktop_Firefox/Automation/Commit_Policy&oldid=378080"