Balrog: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎Filing Bugs: escape space in link)
(Replaced content with "Moved to https://mozilla-balrog.readthedocs.io/en/latest/infrastructure.html")
 
(125 intermediate revisions by 6 users not shown)
Line 1: Line 1:
= Overview =
Moved to https://mozilla-balrog.readthedocs.io/en/latest/infrastructure.html
Balrog is the software that runs the server side component of the update system used by Firefox and other Mozilla products. It is the successor to [https://wiki.mozilla.org/AUS AUS (Application Update Service)], which did not scale to our current needs nor allow us to adapt to more recent business requirements. Balrog helps us ship updates faster and with much more flexability than we've had in the past.
 
= Filing Bugs =
Bugs and feature requests should be filed in the [https://bugzilla.mozilla.org/enter_bug.cgi?component=Balrog%3A%20Backend&product=Release%20Engineering Backend] or [https://bugzilla.mozilla.org/enter_bug.cgi?component=Balrog%3A%20Frontend&product=Release%20Engineering Frontend] components of Bugzilla.
 
= Database Model =
Balrog's model centres around two concepts: Rules and Releases. When a request for update from an application is received it is matched up against the rules. Once the correct rule has been found, it contains a pointer to a Release, which contains all of the metadata needed to construct a proper update response. Rules and Releases are described in greater detail below:
 
== Rules ==
The most important part of Balrog to understand is its rules. When a request comes in it is matched against each of Balrog's rule to find the one that best suits it (more in this below). Once found, Balrog looks at that rule's "mapping", which points at a release that has the required information to serve an update back to the client. Without any rules, Balrog will never serve an update. With badly configured rules Balrog could do bad things like serve Firefox updates to B2G devices.
 
In addition to the information found in the request, each rule also has a priority, which allows us to override updates for specific things while letting the rest "fall down" to a more general rule. For example, we block unsupported versions of OS' with a rule of the highest priority but continue to serve updates for users of other OS' with a rule of slightly lower priority. The ability to override and fallback by setting the priority is one of the reasons Balrog is so flexible, and a key way that it distinguishes itself from its predecessor.
 
=== What's in a rule? ===
Each rule has quite a lot of columns, but they all fall into one of the buckets below:
* '''Matchable''' - these correspond to information provided in the update request, and are used to filter out rules that don't apply to the request.
* '''Decision''' - these are also used to filter rules, but do not correspond to information in the request.
* '''Response''' - these contain information that ends up in the response
* '''Info''' - informational columns, not used as part of serving updates
 
Individual columns are detailed in the table below:
{| class="wikitable sortable"
|-
! Attribute
! Category
! Description
! Matching logic
! Examples
|- style="background-color: palegreen"
| '''Product'''
| Matchable
| The name of the application requesting an update.
| Exact string match only
| "Firefox" or "B2G"
|- style="background-color: palegreen"
| '''Version'''
| Matchable
| The version of the application requesting an update.
| Exact string match or operator plus version to compare the incoming one against
| "36.0" or ">=38.0a1"
|- style="background-color: palegreen"
| '''Channel'''
| Matchable
| The update channel of the application request an update.
| Exact string match or a string with "*" character to glob
| "nightly" or "beta*"
|- style="background-color: palegreen"
| '''buildTarget'''
| Matchable
| The "build target" of the application requesting an update. This is usually related to the target platform the app was built for.
| Exact string match only
| "Darwin_x86_64-gcc3-u-i386-x86_64" or "flame-kk-userdebug"
|- style="background-color: palegreen"
| '''buildID'''
| Matchable
| The build ID of the application requesting an update.
| Exact string match or operator plus buildid to compare the incoming one against
| "201410010830" or "<201512010830"
|- style="background-color: palegreen"
| '''Locale'''
| Matchable
| The locale of the application requesting an update.
| Exact string match or comma separated list of locales to do an exact match on
| "de" or "en-US,en-GB,id"
|- style="background-color: palegreen"
| '''osVersion'''
| Matchable
| The OS Version of the application requesting an update. This field is primarily used to point desupported operating systems to their last supported build.
| Partial string match or comma separated list of partial strings to match on
| "Windows_NT 5.0" or "Darwin 6,Darwin 7,Darwin 8"
|- style="background-color: palegreen"
| '''distribution'''
| Matchable
| The partner distribution name of the application requesting an update or "default" if the application is not a partner build.
| Exact string match only
| "default" or "yahoo"
|- style="background-color: palegreen"
| '''distVersion'''
| Matchable
| The version of the partner distribution of the application requesting an update or "default" if the application is not a partner build.
| Exact string match only
| "default" or "1.19"
|- style="background-color: palegreen"
| '''headerArchitecture'''
| Matchable
| The architecture of the OS of the client as guessed based on build target. This field is mostly deprecated now that this information is included in the build target.
| Exact string match only
| "PPC" and "Intel" are the only possible values
|- style="background-color: lightskyblue"
| '''Priority'''
| Decision
| The priority of the rule, relative to other rules. If multiple rules match an incoming request based on the Matchable columns, the rule with the highest priority is chosen.
| N/A
| Any number, by convention positive integers.
|- style="background-color: lightskyblue"
| '''backgroundRate'''
| Decision
| The percentage of background update requests that should receive an update if they match this rule. Generally this is used as a throttle to increase or decrease the rate at which the majority of users receive an update.
| N/A
| Any number 0 to 100.
|- style="background-color: orange"
| '''Mapping'''
| Response
| The Release to construct an update out of. This is a foreign key to the "name" column of the Releases table.
| N/A
| Any valid release name, or NULL.
|- style="background-color: orange"
| '''update_type'''
| Response
| The update_type to use in the XML response. It's very rare for a rule to use anything other than "minor" these days.
| N/A
| "minor" or "major"
|- style="background-color: crimson"
| '''id'''
| Info
| The id of the rule. This id is necessary to make changes to the rule through the REST API.
| N/A
| Autoincrementing integer
|- style="background-color: crimson"
| '''Comment'''
| Info
| A string describing the purpose of the rule. Not always necessary for obvious rules.
| N/A
| Any string
|}
 
=== How are requests matched up to rules? ===
The incoming request parts match up directly to incoming URL parts. For example, most update requests will send an URL in the following format:
/update/3/<product>/<version>/<buildID>/<buildTarget>/<locale>/<channel>/<osVersion>/<distribution>/<distVersion>/update.xml?force=1 # force can also be left off
 
The following logic is used to figure out which rule an update matches and what to respond with:
# If a rule specifies one of these fields and a request's field doesn't match it, the rule is considered not to be a match and the rule is ignored for that request. See above for details on how specific columns perform matching.
# If "force" wasn't specified, the backgroundRate of the selected rule is looked at.
# If we still choose serve an update after accounting for backgroundRate we look at the rule's mapping. This is a foreign key that points at an entry in the releases table. That row has most of the information we need to construct the update.
# Using the update_type and release that the mapping points to, construct and return an XML response with the details of the update for the client.
 
== Releases ==
'''needs fleshing out'''<br/>
To Balrog, a "release" is data about a related set of builds. This does _not_ match up with the concept of a "release" being on the "beta", "release" or "esr" channel elsewhere. In Balrog, each set of nightlies on any branch is considered a release.
 
While there's no enforced format on release names, there are a few conventions that we use:
* Nightly-style builds submit to releases named by product and branch. Each nightly generally submits to two different releases, one "dated" (eg: Firefox-mozilla-central-nightly-20150513010203) and one "latest" (eg: Firefox-mozilla-central-nightly-latest).
* Release-style builds submit to releases named by product, version number, and build number, eg: Firefox-38.0-build1
* GMP blobs are created by hand and generally named with the version of each plugin they contain in the name, eg: GMP-20150423-CDM-v4-OpenH264-v1.4
 
== Permissions ==
The permissions table is a simple list of usernames and the ACLs that they have. A user could be an "admin", giving them write access to everything, or could have one or more specific permissions. For example, our "ffxbld" system account has access to make PUT requests to add data to "Firefox" or "Fennec". These specific ACLs let us do things such as give B2G folks access to Balrog without the risk of them or their tools accidentally messing up Firefox updates.
 
== History Tables ==
Change attribution and recording is embedded deeply into Balrog. The rules, releases, and permissions tables all have a corresponding history table that records the time a change was made and who made it. This allows us to look back in time when debugging issues, attribute changes to people (aka blame), and quickly roll back bad changes.
 
= Admin UI Use Cases =
== Locking/Unlocking Nightlies ==
One of the most common uses for the Balrog UI is to lock a nightly update channel to a specific release for a period of time, and then unlock it later (so that users on that channel start receiving the latest available build again). This is often done if a serious bug is introduced to minimize the number of users affected by it.
 
Taking the B2G nightly channel as an example, let's see how we would lock it to the nightlies from 20150505160203:
# Log in to https://aus4-admin.mozilla.org
# Click on the "Rules" link at the top of the page
# Use the filter in the top right to narrow down the rules to "product:B2G channel:nightly"
# Locate the rule (or rules) on the "nightly" channel
#* Changing the sort to "Product, Channel" will group things together better.
# For each rule on the channel:
#* Click the "Update" button to enter edit mode
#* Find the mapping field and replace the "-latest" part with "-20150505160203" (the UI will autocomplete this for you if you start typing).
#* Scroll down and click "Save Changes"
 
When you're ready to unlock the updates, follow the same steps as above but replace the "-20150505160203" part of the mapping with "-latest" again.
 
== Adding a rule for a new update channel ==
When nightly builds are set up on a new branch, rules need to be added to Balrog for updates to be served. Note that the nightly build automation is responsible for providing metadata about each new set of builds to Balrog.
 
As an example, here is how B2G updates could be set up on a hypothetical "mozilla-b2g40" branch:
# Log in to https://aus4-admin.mozilla.org
# Click on the "Rules" link at the top of the page
# Click on "Add a new Rule" near the top left of the page
# Fill out the form as follows:
#* Product: B2G
#* Channel: nightly-b2g40
#* Mapping: B2G-mozilla-b2g40-nightly-latest
#* Rate: 100
#* Priority: 90
# Click "Save Changes"
 
== Modify an existing release ==
Most modifications to releases are done by automation, but sometimes we need to tweak them by hand. For example, when adjusting What's New page configuration.
 
For example, if you wanted to modify the "Firefox-38.0-build3" release, follow these steps:
# Log in to https://aus4-admin.mozilla.org/
# Click the "Releases" link at the top of the page
# Find the "Firefox-38.0-build3" release and click the "Download" link
# Save the file locally and modify it to your liking
# Click the "Update" link for "Firefox-38.0-build3"
# Click "Browse" and choose your new local version
# Click "Save Changes"
 
= Code Overview =
Balrog's code is organized into roughly the following parts:
* [https://github.com/mozilla/balrog/tree/master/auslib/blobs The blobs] -  These contain most of the brains (business logic) behind Balrog. They know how to validate new data coming into the system and translate existing data into useful responses to update requests.
* [https://github.com/mozilla/balrog/blob/master/auslib/db.py The database abstraction layer] - This layer sits between the actual database and the applications. It defines the database schema, performs permissions checking, and ensures all changes are written to history tables. Application should never touch the database directly - they should always go through this layer.
* [https://github.com/mozilla/balrog/tree/master/auslib/web The user-facing application] - The entry point to requests from applications looking for updates.
* [https://github.com/mozilla/balrog/tree/master/auslib/admin The admin API] - A simple RESTful API that allows the Admin UI and automation to make changes to Balrog's database.
* [https://github.com/mozilla/balrog-ui The admin UI] A human friendly interface to manage updates.
 
= Hacking =
Balrog's code is split between [https://github.com/mozilla/balrog the backend server] and the [https://github.com/mozilla/balrog-ui admin ui]. Both Github repositories use Travis for continuous integration and accept pull requests. To get both repositories do the following:
git clone https://github.com/mozilla/balrog
cd balrog
git submodule init
git submodule update
 
Basic Balrog development and testing can be done on your local machine. It is recommended that you use Vagrant to test your changes, because it is configured very similar to production. How to use it, and the standalone applications, is described below.
 
== Vagrant ==
Vagrant provides a VM that is configured very similar to production and includes the user-facing application, admin API, admin UI, and some basic sample data. To use it, follow these steps:
* [http://downloads.vagrantup.com install vagrant]
* Open a command line/terminal and browse to your Balrog clone
* Type "vagrant up" to bring up the development environment
* Add this to your hosts file
127.0.0.1 balrog.mozilla.dev
127.0.0.1 balrog-admin.mozilla.dev
* After "vagrant up" returns successfully, you can access the Balrog applications at the following URLs:
** Admin application - http://balrog-admin.mozilla.dev:9000
** User-facing application - http://balrog.mozilla.dev:9000
 
== Backend ==
=== Development Environment ===
Balrog is bundled with most of its required libraries, but you should still use a virtualenv to install the compiled and dev-only packages. To get started, create a new virtualenv and run the following:
<pre>
pip install -r requirements/compiled.txt
pip install -r requirements/dev.txt
</pre>
 
Once you've installed the necessary packages you can bring up the admin API and user-facing applications with the following commands:
<pre>
python admin.py
python balrog-server.py
</pre>
 
=== Unit Tests ===
Balrog comes with an extensive suite of unit tests which are a crucial component of ensuring that we can push changes without any interruption to service. You should always run tests before asking for review, and you should be adding new tests for most types of changes. To run the unit tests for the backend simply run:
<pre>
make test
</pre>
 
or if you want to generate a code coverage report:
<pre>
make test COVERAGE=1
</pre>
 
== Frontend ==
=== Development Environment ===
The frontend is NOT bundled with all of its required dependencies. To install them, run:
<pre>
npm install
npm install -g lineman
</pre>
 
With that done, you can bring up an instance of the UI with:
<pre>
lineman run
</pre>
 
In order for it to have data and be functional you'll need the admin API running as well (see above).
 
=== Unit Tests ===
To run the unit tests you'll need the UI running first ("lineman run"), and then run:
<pre>
lineman spec
</pre>
 
= Deploying Changes =
The dev, stage, and production deployments of Balrog are managed by the Web Operations team. [https://mana.mozilla.org/wiki/display/websites/aus4.mozilla.org Details of the overall deployment] can be found on Mana. This page describes how to go from a reviewed patch to deploying it in production.
 
== Is now a good time? ==
Before you deploy, consider whether or not it's an appropriate time to. Some factors to consider:
* Are we in the middle of an important release such as a chemspill? If so, it's probably not a good time to land non-trivial changes.
* How risky are your changes? If they're high risk, deploying on a Friday is probably a bad idea.
* Do you need to migrate any data? If you do, make sure you have time to do so right after deploying.
 
== Landing ==
=== UI Changes ===
If you've made a change to the [https://github.com/mozilla/balrog-ui ui repository], make sure you run "lineman build" and commit the result before pushing that change. This will rebuild the UI application and put the result into "dist", which is where the deployed instances run out of. You also need to commit the subrepository change to the [https://github.com/mozilla/balrog/ Balrog repository] and push that back. Generally, the workflow for this is something like:
<pre>
cd /your/balrog/repo
cd ui
lineman build
git commit -a -m "Rebuild UI."
git push origin
cd ..
git commit -a -m "Update to latest UI."
git push origin
</pre>
 
=== Backend Changes ===
Just push your change to the master branch of the [https://github.com/mozilla/balrog/ Balrog repository].
 
== Testing ==
The dev environment will automatically pick up any changes you push. You should do some testing against the [http://aus4-admin-dev.allizom.org/ admin interface] and [http://aus4-dev.allizom.org/ public interface] before proceeding further.
 
== Pushing to stage and production ==
Once you're satisfied with your results while testing in dev, simply [https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=server-ops-webops%40mozilla-org.bugs&bug_file_loc=http%3A%2F%2F&bug_ignored=0&bug_severity=normal&bug_status=NEW&cf_fx_iteration=---&cf_fx_points=---&component=WebOps%3A%20Product%20Delivery&contenttypemethod=autodetect&contenttypeselection=text%2Fplain&defined_groups=1&flag_type-354=X&flag_type-4=X&flag_type-407=X&flag_type-481=X&flag_type-607=X&flag_type-674=X&flag_type-707=X&flag_type-791=X&flag_type-800=X&flag_type-803=X&flag_type-811=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=All&priority=--&product=Infrastructure%20%26%20Operations&qa_contact=nmaul%40mozilla.com&rep_platform=All&short_desc=please%20update%20aus4%2Baus4-admin%20stage%2Fprod%20to%20XXXXXXXXXXXXXXXXX&target_milestone=---&version=other file an IT bug to have the changes pushed to stage and production]. Make sure to block the bug(s) containing the code that's being pushed on it. If you'd like to be around when the change is pushed be sure to give a time frame, too.
 
After the change is pushed it's a good idea to keep do some quick verification in production. This could mean making changes by hand, checking update URLs, watching jobs that submit to Balrog, or other things -- it depends what time of change you've made.
 
= Scripts =
These tools all live in [https://hg.mozilla.org/build/tools/file/default/scripts/updates tools/scripts/updates]. We use them to programmatically adjust the update server.
 
== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-submitter.py balrog-submitter.py] ==
 
Used to submit nightly and release style builds into release blobs. Called on build slaves, once per each combination of platform-locale.
 
== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-release-pusher.py balrog-release-pusher.py] ==
Used by the release automation to add the metadata to a release blob, and to push the new release onto the test channels. Example builder name - release-mozilla-beta-firefox_updates.
 
== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-release-shipper.py balrog-release-shipper.py] ==
Used by the release automation to push a release to the production chanenl. Example builder name - release-mozilla-beta-update_shipping.
 
== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-tweaker.py balrog-tweaker.py] ==
We don't have UI support for modifying blobs, so this is a helper to submit a blob fragment to replace content on the server. Doesn't handle removing keys.
 
Example:
<pre>python scripts/updates/balrog-tweaker.py --json json -b 'Firefox-33.0-build1' --api-root 'https://aus4-admin.mozilla.org'  --credentials-file cred -u 'nthomas@mozilla.com' -v</pre>
where json is a file containing the fragment of json to submit, cred is a file containing the password for the -u argument, with format
<pre>balrog_credentials = {
    'username': 'password'
}</pre>
 
== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-nightly-locker.py balrog-nightly-locker.py] ==
Used to 'freeze' nightly updates by pointing to a dated release blob instead of the latest, eg for big code landings, merges. Also to unfreeze afterwards. More details on usage at [[ReleaseEngineering/How_To/Enable_or_Disable_Updates_on_Aurora | Enable/disable updates on Aurora]]
 
== Common code ==
[https://hg.mozilla.org/build/tools/file/default/lib/python/balrog/submitter/cli.py cli.py] and [https://hg.mozilla.org/build/tools/file/default/lib/python/balrog/submitter/api.py api.py] in tools/lib/python/balrog/submitter/ provide the shared code.

Latest revision as of 14:49, 9 January 2020

Moved to https://mozilla-balrog.readthedocs.io/en/latest/infrastructure.html

Retrieved from "https://wiki.mozilla.org/index.php?title=Balrog&oldid=1222143"