Capacity Issues

• wait times are huge, releng scrambling for machines
  • Getting machines we can add more of

Running unit tests on AWS

• viable for xpcshell - doesn't use UI
• we would have to upload our own images into amazon to use AWS
  • could we clone/build one into a vm?
• AWS env go or no go
  • OS with xserver/desktop ui
  • upload our own images up there and test that
  • ensure it reports to orange factor and report the rate of oranges if it is running off the same builds that we run on main trees
  • Run same changeset on exact same code - run it on a project branch as well as in the vm.
  • Set up a separate orangefactor instance to get this information
  • We have to somehow automate/manually star the failures so they get populated in orange factor
  • would only need a small set of data - run the tests 1000 times.
  • could use auotmation on both teset envs to push orange occurrences into the database, then you're using the same code and should achieve comparable results - would also need to report unknown oranges.
  • might not need total orange factor support for this.

mobile capacity

• Could look into mobile testing on emulators
• Will be doing this on b2g already
• Might need to do similar types of testing to verify the valid results.
• Would also be a good idea to look into the x86 emulators for android testing/B2g
• How many emulators can you put on an ec2 image? Should be able to since that's where we are currently running the B2G emulators

Changing the "Test Everything all the time" paradigm

• Could coalesce testing eventually
• Run random/known sample each time, eventually run eveyrthing
• But need deep automation for when things turn orange to bisect issues and find the root cause
  • what would backouts look like in that case? would we back out a bunch of already landed things?
• --> Ted to talk to releng to see what this would involve.
• How to handle performance tests in this case? We do this now with PGO builds. We would need good automatic ways to automatically bisect issues to figure out regressions
• We will still need to do this even if we win on capacity.
• Would still want to do all the tests daily/nightly

Reproducibility

• people cannot easily get a hold of an environment to run a trial test on it*
• Possible Solutions
  • (x) Should be able to checkout entire system/toolchain for a slave.
  • Create a second automation system that could run the test until it hits a failure and it stops and waits for debugger input.
  • Need to talk to people to see which route they would prefer ^
• Solving this problem would make it easier to fix intermittent oranges as well
• Either we distribute a base VM and some tools to set it up like the test slave boxes and we can prove that running in this environment is the same as running in the buildbot environment.
• How to replicate the buildbot enviroinment for launching processes?
• Actions:
  • Poll developers to see if they prefer cloud base/local based systems
  • Query releng to see if we can repro the core env to see if we can produce that environment when running jobs.