Dev-sanity

This page summarizes the work that the A-Team has put into addressing developer workflow/efficiency concerns in 2015Q2, as presented at the Whistler all-hands, with some additional projects that we won't have time to review. We also describe some cool new work coming up in the latter half of 2015.

MozReview

[mcote to fill in]

Intermittent Oranges

[jgriffin to fill in]

Tree Closures

Tree Closures for a number of the integration branches are currently on the rise. We can see this on Futurama. Some of the current tree closures can be attributed to the length that tests take to run (Normalising test run length would solve this).

For this year, the amount of tree closures have been attributed to the following

Tree Closed for a Test or Bustage	Count
Failing Test - No Try run done	7
Try breadth of run not sufficient	3
Bustage - No Try Run	5
Bustage - Try breadth of run not sufficient	3
Perf Regression	1

Developer Experience

Running tests from tests.zip

A new mach bootstrap environment is created specifically for running in the context of a tests.zip. This will allow us to write mach commands that will get packaged as part of the package-tests make target. Developers downloading tests.zip (or packaging it from a local build) can now run |mach help| to perform test package related tasks (most likely running tests).

Currently only the mochitest mach command has been implemented, but other harnesses may follow suit shortly depending on resources and demand. Because there is no |mozbuild| module to work with, the mach command cannot automatically detect the firefox binary, it must be passed in via --appname. For the same reason, the mach command cannot resolve tests and automatically detect which flavors and subsuites exist in a particular directory like the in-tree version of the script does. Usage:

mach mochitest --appname path/to/firefox [path/to/test/dir]

Test selection on try

Select tests within a test job to run on try. This is achieved by extending try syntax to support paths and tags as arguments. This means shorter turn around times for getting feedback on particular tests and less redundant machine time running tests that probably aren't going to fail as a result of a particular change.

This is primarily exposed through a new mach command, |mach try|, that takes paths and/or tags as arguments, calculates an appropriate try syntax based on the tests present in the tree, and pushes the result to try. The implementation was discussed in 1149670. Details can be found by running |mach help try|.

Automatic retrigger of oranges on try

Retrigger failed jobs on try automatically. This is a pulse service that listens for failed test jobs and triggers each one two additional times. The idea is to reduce time spent determining whether a particular failure on try is an intermittent issue or a persistent failure introduced by a specific change. A common response to orange jobs on try is to retrigger them to determine if the failure persists, and although it results in more test jobs on try, the service is intended to save developer time by replacing this manual process with an automated one.

A related feature implemented by this service is the ability to automatically build every test job on a push a certain number of times. This is a long-requested feature intended to be used by those investigating or proving a fix for an intermittent failures as well as those establishing a baseline for comparing Talos results across revisions.

The rollout of the service is ongoing and tracked by 1163698 (details on the implementation can be found there as well). As of this writing it's enabled for try pushes initiated by the autoland user.

Allow scheduling missing platforms on a try push

When a developer pushes to try, they have to select a list of platforms in their commit message. If they don't specify the right set of platforms, they would need to push again to try wasting machine resources and time. They now can use Mozilla CI tools (aka mozci) in order to trigger missing platforms.

They now can either use trigger.py or triggerbyfilters.py to add missing platforms.

In the future, an improved experience will be achieve through "Case scenario 8: Developer needs to add missing platforms/jobs for a Try push"

Others?

run-by-dir? chunk normalization?

e10s support...