Problem

The current automation infrastructure Mozilla uses is running pretty far in the red. The amount of time it takes to get a test run completed, the tooling for in-house developers is nearly pitiful, and we are always two steps away from out-scaling our capacity.

We have few analytics w.r.t. how well we test anything. We have intermittent test issues that are threatening to sink the entire system. We're adding platforms like there is no tomorrow, and hoping the entire infrastructure doesn't collapse around our heads.

Solutions

To date, we've backed into solutions. We've racked mac minis because we needed SOMETHING to run performance tests on. And we've lived with that choice ever since. We turned off VMs for testing because we blamed them for intermittent failures without being able to 100% prove they were at fault. We don't have bugzilla wired to try because everyone thought they couldn't change bugzilla. We have mozharness but it doesn't hook well with buildbot, we have mach because you just can't easily change the build system, we have test suites that should be on TBPL but they can't because they require outside network access, etc etc etc.

We keep backing into solutions reactively. We should start thinking proactively. A lot of these solutions are already on the table, a lot of the half-baked stuff we've got sitting around can help tremendously if we were to just wire it together. Many of the developer headaches could likewise be easily solved. And some things can't be easily solved. Some things require an entirely new approach to automation at this kind of scale. Those are on the table too.

This project is to develop a prioritized road map backed by data for how to take Mozilla's automation infrastructure to the next level of scale and reliability.

Process

This is going to be an involved planning process. We will use two meetings a week, one to generate and ask questions and one to review the answers and decide on next steps of investigation. As a rough process these are the topics we plan to cover (note that this will change as we gain more information).

We also had a meeting with developers on 2012-10-26 in which we solicited ideas and opinions. We incorporated some of this into on-going plans.

Results

• Our identified areas of concern. This is what we'll be tackling within the scope of the Futurama project.
• Our immediate priorities. This is what we'll be starting with.
• The draft plan is here

Meetings

 # The generative meeting will be at 9AM PDT on Tuesdays  
 # The analysis meeting will be at noon PDT on Thursdays   
 # Vidyo:  https://v.mozilla.com/flex.html?roomdirect.html&key=e9MzlP1aYdfs
 # Phone:  650-903-0800 or 650-215-1282 x92 Conf# 9304 (US/INTL)
 #         1-800-707-2533 (pin 369) Conf# 9304 (free for anyone to join)
 # IRC:    irc://irc.mozilla.org:6697/#ateam
 # Take notes on this etherpad and copy to the wiki page for the date below.

Proposed schedule (subject to change)