This page exists to explore the remote possibility that we might skip some tests on each push, rather than running every test on every push.

Of course, there will always be a need to run all tests on a frequent basis, so we should keep doing it at least daily; but we might not need to run all tests on every push.

The hard part is to figure what tests can be skipped for a certain push.

Two basic questions to ask before we can even start

Question 1: do we want to specify:

• A: for each test, a list of source files whose modification should trigger running it?
• B: or, for each source directory, which tests to run when it's modified?

Note that the two possible answers of question 1 are converse of each other. I don't know which of A or B is better.

Question 2: what level of granularity should we use?

• C: whole test jobs such as "mochitest-2", "reftest" etc?
• D: or test directories?

Note that C is more short-term realistic, but D allows for far higher gains. For example, with C, any change touching Gecko would trigger about all tests, but with D, a change touching Gecko would typically trigger only a minority of tests.

For example, if we go for AD, we could use a text file looking like this:

 TEST_DIR content/canvas/test
   RELEVANT_DIRS:
     content/canvas/src          # canvas context impl
     content/html/content/src    # canvas element impl
     gfx                         # drawing, compositing, and GLContext
     dom/interfaces/canvas       # IDL