CI Automation: Difference between revisions

2,284 bytes added ,  15 October 2019
→‎Other work we do: add new configurations with stub page
(→‎Other work we do: add new configurations with stub page)
 
(13 intermediate revisions by 4 users not shown)
Line 1: Line 1:
Welcome to the CI Automation team's landing page!
Welcome to the CI Automation team's landing page!


We work on a wide range of projects around Mozilla's core [https://en.wikipedia.org/wiki/Continuous_integration continuous integration] (CI) pipeline. Some areas we focus on include:
== Who we are ==


* '''Performance''': developing perf harnesses and standing up new benchmarks/measurements
We are a team of 9 engineers, 2 managers, 16 sheriffs (24x7 shift work in teams), and always a few interns, volunteers, and contractors.
* '''Testing''': developing test harnesses and improving test stability
* '''Data''': collecting CI related data, creating metrics and dashboarding
* '''Configuration''': developing taskgraph and modifying which tasks run in CI
* '''Tools''': developing the tools that Firefox core developers use to interact with CI
* Plus many other projects


Our team is split into 3 categories:
* '''Integration''': integration of tests, platforms, configs into our CI system
* '''Reporting''': dashboards and tools to view and work with data and the CI system
* '''Sheriffing''': code sheriffing (not perf sheriffing) for mozilla-central based trees
== Goal ==
The CI Automation and Reporting teams are responsible for ensuring that we schedule appropriate tests, those tests produce useful results, we can view the results of the tests and collect data to make decisions, and finally that we are experts with the tools and provide sheriffing features for Firefox.
== Mission ==
To make the scheduling and reporting of our test results more accurate and easier to understand, leading to a yes/no answer from a try push.
== Vision ==
Our CI system schedules tests as efficiently as possible, highlights regressions as quick as possible, and our tools to monitor and manage what we schedule, what is run, and what was run is easy to use.
== How we work ==
We don't write automated tests for Firefox features, but we love to work with teams that have unique scenarios for automation or just questions.  We work closely with product development to ensure that our tests are run in the right environments and configurations, that developers know how to test their code and understand their results, and when code lands the sheriffs actively monitor the trees. 
Requests are done via Bugzilla or IRC chat (#cia, #treeherder, #sheriffs).  We prefer to have 2-3 weeks heads up on projects and more if possible.  Our work is 20% reactive where we help put out fires or treat a new issue as a P1, and 80% proactive where we plan work and get it done.
== Integration: ==
=== Routine work we do ===
* new platforms
** windows10/aarch64
* new configs
** fission
* intermittents
* test efficiencies
=== Other work we do ===
* capacity planning
* test scheduling
* [[new test environments]]
* [[new configurations]]
** test-verify
* harness support


== Contact Us ==
== Contact Us ==
Line 15: Line 49:
We hang out on [[IRC|irc.mozilla.org]] in the '''#cia''' channel. Feel free to drop by and say hello, we're a friendly bunch. Here are some related channels that might also be of interest:
We hang out on [[IRC|irc.mozilla.org]] in the '''#cia''' channel. Feel free to drop by and say hello, we're a friendly bunch. Here are some related channels that might also be of interest:


* '''#ateam''': General automation channel where all automation related teams across Mozilla hang out
* '''#ci''': General channel for CI-related discussion
* '''#ci''': General channel for CI related discussion
* '''#cia''': Our team channel
* '''#cia''': Our team channel
* '''#developers''': Channel where developers all across Mozilla hang out
* '''#developers''': Channel where developers all across Mozilla hang out
Line 27: Line 60:
== Contributing ==
== Contributing ==


There are so many different aspects to contributing that implementing the patch is often the easy part! This is a high level overview of the contribution process, with links to resources that go into greater detail for each step.
There are so many different aspects to contributing that implementing the patch is often the easy part! This is a high-level overview of the contribution process, with links to resources that go into greater detail for each step.


==== Finding a Bug ====
==== Finding a Bug ====
Line 53: Line 86:
==== Building Firefox ====
==== Building Firefox ====


After cloning the source one of the most common first tasks is building Firefox. While the work our team tends to do rarely requires a build, it's still a good step to help familiarize yourself with the repository and some of the tools we use.
After cloning the source code, one of the most common first tasks is building Firefox. While the work our team tends to do rarely requires a build, it's still a good step to help familiarize yourself with the repository and some of the tools we use.


Follow [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_build these instructions] to setup your dependencies and perform a build (note: if you've already cloned, you can run '''./mach bootstrap''' instead of downloading bootstrap.py). Once your build is finished, use '''./mach run''' to spin up your very own Firefox!
Follow [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_build these instructions] to set up your dependencies and perform a build (note: if you've already cloned, you can run '''./mach bootstrap''' instead of downloading bootstrap.py). Once your build is finished, use '''./mach run''' to spin up your very own Firefox!


When going through this process, you'll notice that you'll interact a lot with [https://firefox-source-docs.mozilla.org/python/mach/index.html mach]. Mach is a command line dispatcher that we use to provide a central entry point to a vast array of tools. You can look through '''./mach help''' to get a sense of all the different tasks that can be performed in the repository.
When going through this process, you'll notice that you'll interact a lot with [https://firefox-source-docs.mozilla.org/python/mach/index.html mach]. Mach is a command-line dispatcher that we use to provide a central entry point to a vast array of tools. You can look through '''./mach help''' to get a sense of all the different tasks that can be performed in the repository.




'''Bonus Challenge'''
'''Bonus Challenge'''


Find a string in Firefox (e.g in one of the menus or preference dialogs) and make sure that it's fairly unique. Then try and use [https://searchfox.org/mozilla-central/source searchfox] to find where that string is defined. Change the value to something else and re-run a build. Then use '''./mach run''' to see if your change is there!
Find a string in Firefox (e.g. in one of the menus or preference dialogs) and make sure that it's fairly unique. Then try and use [https://searchfox.org/mozilla-central/source searchfox] to find where that string is defined. Change the value to something else and re-run a build. Then use '''./mach run''' to see if your change is there!


==== Fixing the Bug ====
==== Fixing the Bug ====
Line 83: Line 116:
* The [https://www.mercurial-scm.org/doc/evolution/user-guide.html evolve] extension
* The [https://www.mercurial-scm.org/doc/evolution/user-guide.html evolve] extension


See [https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/ Mercurial For Mozillians] for more information on Mercurial.
See [https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/ Mercurial For Mozillians] for more information.


If using Git, some ways to rewrite history include:
If using Git, some ways to rewrite history include:
Line 95: Line 128:
Congratulations, your patch just landed! But what does that mean?
Congratulations, your patch just landed! But what does that mean?


Mozilla has a multi-tiered branching system (in reality each branch is a separate repository, but let's gloss over that fact for now). As we mentioned earlier, all mainline development happens on mozilla-central. Since we are all human, every so often we accidentally land something that causes things to break (e.g we might cause a test to fail). Now imagine that developers were landing their patches directly on mozilla-central and occasionally causing things to break. If you were developing your own patch and notice a test failure it would be really hard for you to determine whether your patch was to blame, or if I just happened to be based on someone else's change that caused it to break. At a scale of hundreds of developers it's not hard to imagine how this would get out of hand very quickly!
Mozilla has a multi-tiered branching system (in reality each branch is a separate repository, but let's gloss over that fact for now). As we mentioned earlier, all mainline development happens on mozilla-central. Since we are all human, every so often we accidentally land something that causes things to break (e.g we might cause a test to fail). Now imagine that developers were landing their patches directly on mozilla-central and occasionally causing things to break. If you were developing your own patch and notice a test failure it would be really hard for you to determine whether your patch was to blame, or if you just happened to be based on someone else's change that caused it to break. At a scale of hundreds of developers it's not hard to imagine how this would get out of hand very quickly!


Instead Mozilla uses something called an "integration branch". These are branches where it's *ok* if things get broken since no one bases their local work on top of them. If your change breaks something, you'll get "backed out" (i.e the exact reverse of your commit will be committed). In this scenario, someone will comment on the bug to notify you of the backout and it will be your (and your mentor's) responsibility to figure out what failed and how to fix. Again don't feel bad if this happens, it's a normal part of our workflow.
Instead Mozilla uses something called an "integration branch". These are branches where it's *ok* if things get broken since no one bases their local work on top of them. If your change breaks something, you'll get "backed out" (i.e the exact reverse of your commit will be committed). In this scenario, someone will comment on the bug to notify you of the backout and it will be your (and your mentor's) responsibility to figure out what failed and how to fix. Again, don't feel bad if this happens, it's a normal part of our workflow.


If all the builds and tests pass on the integration branch, your patch will eventually be merged into mozilla-central (these merges typically happen twice a day). At this point you're in the clear! The next time someone pulls in the latest changes from central, your changes will be included. Congratulations!
If all the builds and tests pass on the integration branch, your patch will eventually be merged into mozilla-central (these merges typically happen twice a day). At this point you're in the clear! The next time someone pulls in the latest changes from central, your changes will be included. Congratulations!
You can see all the build and test results using a service called [https://treeherder.mozilla.org/#/jobs?repo=mozilla-central treeherder]. On the left you'll see the push information (who pushed and which commits were in the push). On the right you'll see the results. Each symbol represents a "task" which runs builds or tests. If something fails, the symbol will turn orange (or red). It's normal for every push to have a few "intermittent failures" on it. If your patch landed via phabricator, you'll be able to see it on the [https://treeherder.mozilla.org/#/jobs?repo=autoland autoland] branch of treeherder.
== [[Projects]] ==
This is an incomplete list of historical, current and future projects.
=== Windows 10 ARM64 ===
* Project page: [https://wiki.mozilla.org/CI_Automation/windows10_aarch64 link]
* Contact: gbrown@mozilla.com, egao@mozilla.com


== [[Glossary]] ==
== [[Glossary]] ==


Mozillians use a dizzying amount of terminology that can make any new contributor's head spin. You'll often be able to find what you're looking for in the [[Glossary]]. If you come across something not mentioned there, feel free to ask us in IRC (and remind us to add it to the list).
Mozillians use a dizzying amount of terminology that can make any new contributor's head spin. You'll often be able to find what you're looking for in the [[Glossary]]. If you come across something not mentioned there, feel free to ask us in IRC (and remind us to add it to the list).
75

edits