= Overview =Balrog is If you are looking for the software general documentation that runs the server side component of the update system used by Firefox and other Mozilla products. It is the successor to live here, it has been moved into [https://wikigithub.com/mozilla.org/AUS AUS (Application Update Service)balrog the Balrog repository], which did not scale to our current needs nor allow us to adapt to more recent business requirementsand a built version of it [http://mozilla-balrog.readthedocs.io/en/latest/index. Balrog helps us ship updates faster and with much more flexability than we've had in the pasthtml is available on Read The Docs].
Bugs and feature requests should be filed in the [https://bugzilla.mozilla.org/enter_bug.cgi?component=Balrog%3A%20Backend&product=Release%20Engineering Backend] or [https://bugzilla.mozilla.org/enter_bug.cgi?component=This page will continue to host information about Balrog%3A%20Frontend&product=Release%20Engineering Frontend] components of Bugzilla. You can find that doesn't make sense to put into the Balrog developers in [irc://irc.mozilla.org/#balrog #balrog]repository, such as meeting notes and the [https://github.com/mozilla/balrog code in Github]things related to our hosted versions of Balrog.
= Infrastructure === Domains ==
Balrog admin interface is accessible at [https://aus4-admin.mozilla.org/ aus4-admin.mozilla.org] (vpn required).
The public interface that Firefox and other applications talk to is at aus5.mozilla.org. Many older update domains are also served by Balrog, including aus4.mozilla.org, aus3.mozilla.org, and aus2.mozilla.org. More details on these can be found on the [[Balrog/Client_Domains | Client Domains page]].
= Database Model =
Balrog's model centres around two concepts: Rules and Releases. When a request for update from an application is received it is matched up against the rules. Once the correct rule has been found, it contains a pointer to a Release, which contains all of the metadata needed to construct a proper update response. Rules and Releases are described in greater detail below:
== Rules ==
The most important part of Balrog to understand is its rules. When a request comes in it is matched against each of Balrog's rule to find the one that best suits it (more in this below). Once found, Balrog looks at that rule's "mapping", which points at a release that has the required information to serve an update back to the client. Without any rules, Balrog will never serve an update. With badly configured rules Balrog could do bad things like serve Firefox updates to B2G devices.
In addition to the information found in the request, each rule also has a priority, which allows us to override updates for specific things while letting the rest "fall down" to a more general rule. For example, we block unsupported versions of OS' with a rule of the highest priority but continue to serve updates for users of other OS' with a rule of slightly lower priority. The ability to override and fallback by setting the priority is one of the reasons Balrog is so flexible, and a key way that it distinguishes itself from its predecessor.
=== What's in a rule? ===
Each rule has quite a lot of columns, but they all fall into one of the buckets below:
* '''Matchable''' - these correspond to information provided in the update request, and are used to filter out rules that don't apply to the request.
* '''Decision''' - these are also used to filter rules, but do not correspond to information in the request.
* '''Response''' - these contain information that ends up in the response
* '''Info''' - informational columns, not used as part of serving updates
Individual columns are detailed in the table below:
{| class="wikitable sortable"
|-
! Attribute
! Category
! Description
! Matching logic
! Examples
|- style="background-color: palegreen"
| '''Product'''
| Matchable
| The name of the application requesting an update.
| Exact string match only
| "Firefox" or "B2G"
|- style="background-color: palegreen"
| '''Version'''
| Matchable
| The version of the application requesting an update.
| Exact string match or operator plus version to compare the incoming one against
| "36.0" or ">=38.0a1"
|- style="background-color: palegreen"
| '''Channel'''
| Matchable
| The update channel of the application request an update.
| Exact string match or a string with "*" character to glob
| "nightly" or "beta*"
|- style="background-color: palegreen"
| '''buildTarget'''
| Matchable
| The "build target" of the application requesting an update. This is usually related to the target platform the app was built for.
| Exact string match only
| "Darwin_x86_64-gcc3-u-i386-x86_64" or "flame-kk-userdebug"
|- style="background-color: palegreen"
| '''buildID'''
| Matchable
| The build ID of the application requesting an update.
| Exact string match or operator plus buildid to compare the incoming one against
| "201410010830" or "<201512010830"
|- style="background-color: palegreen"
| '''Locale'''
| Matchable
| The locale of the application requesting an update.
| Exact string match or comma separated list of locales to do an exact match on
| "de" or "en-US,en-GB,id"
|- style="background-color: palegreen"
| '''osVersion'''
| Matchable
| The OS Version of the application requesting an update. This field is primarily used to point desupported operating systems to their last supported build.
| Partial string match or comma separated list of partial strings to match on
| "Windows_NT 5.0" or "Darwin 6,Darwin 7,Darwin 8"
|- style="background-color: palegreen"
| '''systemCapabilities'''
| Matchable
| The supported hardware features of the application requesting an update. This field is primarily used to point desupported users based on their hardware. Eg: users who do not support SSE2
| Full string match or comma separated list of full strings to match on
| "SSE" or "MMX,SSE"
|- style="background-color: palegreen"
| '''distribution'''
| Matchable
| The partner distribution name of the application requesting an update or "default" if the application is not a partner build.
| Exact string match only
| "default" or "yahoo"
|- style="background-color: palegreen"
| '''distVersion'''
| Matchable
| The version of the partner distribution of the application requesting an update or "default" if the application is not a partner build.
| Exact string match only
| "default" or "1.19"
|- style="background-color: palegreen"
| '''headerArchitecture'''
| Matchable
| The architecture of the OS of the client as guessed based on build target. This field is mostly deprecated now that this information is included in the build target.
| Exact string match only
| "PPC" and "Intel" are the only possible values
|- style="background-color: palegreen"
| '''Whitelist'''
| Matchable
| A pointer to a Whitelist blob (stored in the Releases table) that can determine whether an update request is authorized to have the Release this rule is mapped to. Commonly used in whitelisting IMEIs for FirefoxOS updates.
| If a whitelist is present, its shouldServeUpdate is called. If it returns True, this rule is considered to be matching. If it returns False, this rule is thrown out.
| Any valid release name, or NULL
|- style="background-color: lightskyblue"
| '''Priority'''
| Decision
| The priority of the rule, relative to other rules. If multiple rules match an incoming request based on the Matchable columns, the rule with the highest priority is chosen.
| N/A
| Any number, by convention positive integers.
|- style="background-color: lightskyblue"
| '''backgroundRate'''
| Decision
| The percentage of background update requests that should receive the latest update if they match this rule. Others receive the update from Fallback Mapping, if specified. Generally, this is used as a throttle to increase or decrease the rate at which the majority of users receive the latest update.
| N/A
| Any number 0 to 100
|- style="background-color: crimson"
|- style="background-color: orange"
| '''Mapping'''
| Response
| The Release to construct an update out of if the user is on the right side of a background rate dice roll, or if the background rate is 100. This is a foreign key to the "name" column of the Releases table.
| N/A
| Any valid release name, or NULL.
|- style="background-color: orange"
| '''Fallback Mapping'''
| Response
| The Release to construct an update out of when the user is on the wrong side of a background rate dice roll. This is a foreign key to the "name" column of the Releases table.
| N/A
| Any valid release name, or NULL.
|- style="background-color: orange"
| '''update_type'''
| Response
| The update_type to use in the XML response. It's very rare for a rule to use anything other than "minor" these days.
| N/A
| "minor" or "major"
|- style="background-color: crimson"
| '''id'''
| Info
| The id of the rule. This id is necessary to make changes to the rule through the REST API.
| N/A
| Autoincrementing integer
|- style="background-color: crimson"
| '''Alias'''
| Info
| A unique alias for the rule. Can be used in place of id in any REST operation that doesn't involve rule history.
| N/A
| "firefox-release-betatest", "firefox-nightly"
|- style="background-color: crimson"
| '''Comment'''
| Info
| A string describing the purpose of the rule. Not always necessary for obvious rules.
| N/A
| Any string
|}
=== How are requests matched up to rules? ===
The incoming request parts match up directly to incoming URL parts. For example, most update requests will send an URL in the following format:
/update/3/<product>/<version>/<buildID>/<buildTarget>/<locale>/<channel>/<osVersion>/<distribution>/<distVersion>/update.xml?force=1 # force can also be left off
The following logic is used to figure out which rule an update matches and what to respond with:
# If a rule specifies one of these fields and a request's field doesn't match it, the rule is considered not to be a match and the rule is ignored for that request. See above for details on how specific columns perform matching.
# If "force" wasn't specified, the backgroundRate of the selected rule is looked at.
# If we still choose serve an update after accounting for backgroundRate we look at the rule's mapping. This is a foreign key that points at an entry in the releases table. That row has most of the information we need to construct the update.
# Using the update_type and release that the mapping points to, construct and return an XML response with the details of the update for the client.
=== Scheduled Changes ===
Rules may have changes scheduled in advance. Currently these changes may only be scheduled based on a timestamp, but there is the possibility that they could be driven by release uptake, crashes, or other data in the future.
Permissions for Scheduled Changes are inherited from regular Rule permissions. If you want to scheduled a change to a Rule, you must have the right permissions to modify that Rule directly. No special permission on top of that is needed to scheduled a change.
Scheduled Changes are stored in a separate table that mirrors the main Rules table, and tracks the extra information required to schedule and enact them.
== Releases ==
'''needs fleshing out'''<br/>
To Balrog, a "release" is data about a related set of builds. This does _not_ match up with the concept of a "release" being on the "beta", "release" or "esr" channel elsewhere. In Balrog, each set of nightlies on any branch is considered a release.
While there's no enforced format on release names, there are a few conventions that we use:
* Nightly-style builds submit to releases named by product and branch. Each nightly generally submits to two different releases, one "dated" (eg: Firefox-mozilla-central-nightly-20150513010203) and one "latest" (eg: Firefox-mozilla-central-nightly-latest).
* Release-style builds submit to releases named by product, version number, and build number, eg: Firefox-38.0-build1
* GMP blobs are created by hand and generally named with the version of each plugin they contain in the name, eg: GMP-20150423-CDM-v4-OpenH264-v1.4
More details on blobs can be found on [[Balrog/Blobs | the Blobs page]].
== Permissions ==
The permissions table is a simple list of usernames and the ACLs that they have. A user could be an "admin", giving them write access to everything, or could have one or more specific permissions. For example, our "ffxbld" system account has access to make create or modify "Firefox" or "Fennec" releases. These specific ACLs let us do things such as give Go Faster folks access to Balrog without the risk of them or their tools accidentally messing up Firefox updates (or visa versa).
The table below describe all possible permissions:
{| class="wikitable sortable"
|-
! Object
! Action
! Options
! Comments
|-
| admin
| No supported actions
| products - If specified, the user can perform any actions on Rules or Releases that affect the specified products.
| An admin user with no options specified has completely unrestricted access to Balrog.
|-
| rowspan="3" | rule
| create
| rowspan="9" | products - If specified, the user only has permission for the object and action if the changes they are making only affect the product specified.
| rowspan="12" |
|-
| modify
|-
| delete
|-
| rowspan="3" | release
| create
|-
| modify
|-
| delete
|-
| rowspan="2" | release_read_only
| set
|-
| unset
|-
| release_locale
| modify
|-
| rowspan="3" | permission
| create
| rowspan="3" | No supported options.
|-
| modify
|-
| delete
|-
| scheduled_change
| enact
| No supported options
| Only the Balrog Agent should be granted this permission.
|}
== History Tables ==
Change attribution and recording is embedded deeply into Balrog. The rules, releases, and permissions tables all have a corresponding history table that records the time a change was made and who made it. This allows us to look back in time when debugging issues, attribute changes to people (aka blame), and quickly roll back bad changes.
= Admin App =
== API ==
See the [[Balrog/Admin API | Admin API page]].
== UI Use Cases ==
=== Locking/Unlocking Nightlies ===
One of the most common uses for the Balrog UI is to lock a nightly update channel to a specific release for a period of time, and then unlock it later (so that users on that channel start receiving the latest available build again). This is often done if a serious bug is introduced to minimize the number of users affected by it.
Taking the B2G nightly channel as an example, let's see how we would lock it to the nightlies from 20150505160203:
# Log in to https://aus4-admin.mozilla.org
# Click on the "Rules" link at the top of the page
# Use the filter in the top right to narrow down the rules to "product:B2G channel:nightly"
# Locate the rule (or rules) on the "nightly" channel
#* Changing the sort to "Product, Channel" will group things together better.
# For each rule on the channel:
#* Click the "Update" button to enter edit mode
#* Find the mapping field and replace the "-latest" part with "-20150505160203" (the UI will autocomplete this for you if you start typing).
#* Scroll down and click "Save Changes"
When you're ready to unlock the updates, follow the same steps as above but replace the "-20150505160203" part of the mapping with "-latest" again.
=== Adding a rule for a new update channel ===
When nightly builds are set up on a new branch, rules need to be added to Balrog for updates to be served. Note that the nightly build automation is responsible for providing metadata about each new set of builds to Balrog.
As an example, here is how B2G updates could be set up on a hypothetical "mozilla-b2g40" branch:
# Log in to https://aus4-admin.mozilla.org
# Click on the "Rules" link at the top of the page
# Click on "Add a new Rule" near the top left of the page
# Fill out the form as follows:
#* Product: B2G
#* Channel: nightly-b2g40
#* Mapping: B2G-mozilla-b2g40-nightly-latest
#* Rate: 100
#* Priority: 90
# Click "Save Changes"
=== Modify an existing release ===
Most modifications to releases are done by automation, but sometimes we need to tweak them by hand.
For example, if you wanted to modify the "Firefox-38.0-build3" release, follow these steps:
# Log in to https://aus4-admin.mozilla.org/
# Click the "Releases" link at the top of the page
# Find the "Firefox-38.0-build3" release and click the "Download" link
# Save the file locally and modify it to your liking
# Click the "Update" link for "Firefox-38.0-build3"
# Click "Browse" and choose your new local version
# Click "Save Changes"
= Balrog Agent =
The Balrog Agent is a long running process that is responsible for monitoring and enacting [[#Scheduled Changes | Scheduled Changes]]. The Agent is a separate application that runs in its own Docker container. In order to ensure it cannot bypass permissions or history (accidentally or on purpose), the Agent acts as a client to the [[#Admin App | Admin App]] rather than interacting with database directly. This means that it must be granted [[#Permissions | the "scheduled_change" permission]] to function.
In the future, the Agent may also act as a client to Telemetry, Crash Stats, or other systems that contain data that we may want to scheduled changes with.
= Code Overview =
Balrog's code is organized into roughly the following parts:
* [https://github.com/mozilla/balrog/tree/master/auslib/blobs The blobs] - These contain most of the brains (business logic) behind Balrog. They know how to validate new data coming into the system and translate existing data into useful responses to update requests.
* [https://github.com/mozilla/balrog/blob/master/auslib/db.py The database abstraction layer] - This layer sits between the actual database and the applications. It defines the database schema, performs permissions checking, and ensures all changes are written to history tables. Application should never touch the database directly - they should always go through this layer.
* [https://github.com/mozilla/balrog/tree/master/auslib/web The user-facing application] - The entry point to requests from applications looking for updates.
* [https://github.com/mozilla/balrog/tree/master/auslib/admin The admin API] - A simple RESTful API that allows the Admin UI and automation to make changes to Balrog's database.
* [https://github.com/mozilla/balrog/tree/master/ui The admin UI] A human friendly interface to manage updates.
* [https://github.com/mozilla/balrog/tree/master/agent The Balrog Agent] A long running process that is responsible for enacting Scheduled Changes.
= Hacking =
Balrog's code is available in [https://github.com/mozilla/balrog the Github repository]. To get it and get ready to hack, run:
git clone https://github.com/mozilla/balrog
cd balrog
== Docker ==
You can run a complete Balrog environment locally with Docker, configured very similarly to production. You'll need Docker and Docker Compose installed; on Mac or Windows we require Docker for Mac/Docker for Windows v1.12 or higher. Then run the following in the root of the Balrog repo:
docker-compose up
Once it completes, you should be able to access
* the admin interface at [http://127.0.0.1:8080 http://127.0.0.1:8080]
* the public interface on port 9090, for example [http://127.0.0.1:9090/update/3/Firefox/33.0/20141202185629/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/default/default/default/update.xml?force=1 http://127.0.0.1:9090/update/3/Firefox/33.0/20141202185629/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/default/default/default/update.xml?force=1]
== UI hacking ==
Once the Docker images are running, any changes you make to js or css files should cause lineman to automatically rebuild them in the balrogui container - you do not need to run lineman or npm on your host machine. However, You will still need to shift-refresh in your browser to pick them up.
== Unit Tests ==
=== Backend ===
The recommended way to run backend tests is through Docker. Doing so will ensure your tests will run the same way they do in CI. To run them, use this command in the root of your Balrog repository:
./run-tests.sh backend
If you can't or don't want to run tests within Docker for some reason, you can run them with directly with:
tox
Tests run fine on any posix-like environment, but are only run regularly within the Docker image, so it's possible to have failures that aren't related to Balrog code or your changes when running directly with tox.
=== Frontend ===
To run the unit tests use:
./run-tests.sh frontend
More options can be found in [https://github.com/mozilla/balrog-ui/blob/master/README.md ui/README.md]
= Getting Involved =
If you like to get involved in the development of Balrog there are lots of areas where we could use some help. Below is a list of specific tasks that would be good first choices. Come talk to us in [irc://irc.mozilla.org/#balrog #balrog] if you're interested!
<bugzilla>
{
"quicksearch": "status:new,assigned,reopened,unconfirmed comp:balrog mentor:'@' assignee:nobody@mozilla.org whiteboard:[good first bug]",
"product": "Release Engineering",
"include_fields": "id,component,summary,status"
}
</bugzilla>
= Meeting Notes =
See the [[Balrog/Meetings | Meetings page]].
= Running without Docker Compose =
== Creating a database ==
Balrog's database is controlled through sqlalchemy-migrate. To initialize a new Balrog database, run the following:
docker run --entrypoint python mozilla/balrog /app/scripts/manage-db.py -d DBURI create
Similarly, to upgrade the schema of an existing Balrog database, run the following:
docker run --entrypoint python mozilla/balrog /app/scripts/manage-db.py -d DBURI upgrade
See the [[#Environment Variables | "Environment Variables" section below]] for DBURI format. If your testing out local changes that affect database creation or upgrades, you should replace "mozilla/balrog" with your local image.
== Environment Variables ==
The following environment variables are required by the Balrog WSGI apps:
* DBURI - The database to use, in the format: driver://user:password@host/database.
* SECRET_KEY - A pseudorandom string to use when generating CSRF tokens. Only used for the admin app.
These are optional:
* LOG_LEVEL - Controls the python level the app logs at. Set to INFO by default.
* LOG_FORMAT - Controls the log format. If unset, mozlog format (json) will be used. Can be overridden with "plain" to log simple plain-text messages. The former is recommended for production, the latter for local development.
* NOTIFY_TO_ADDR - An address to send e-mail to if a Rule or Permission changes. Unset by default, and only used for the admin app. If set, the following additional variables are required:
** SMTP_HOST, SMTP_PORT, SMTP_USERNAME, SMTP_PASSWORD - Information about the SMTP relay to send mail through.
** NOTIFY_FROM_ADDR - The "from" address to use when sending mail.
= Infrastructure =
== Support & Escalation ==
''If the issue may be visible to users, please make sure [irc://irc.mozilla.org/#moc #moc] is also notified. They can also assist with the notifications below.''
# Reverify in production. When production has been updated, verify your changes again there. If you need to tweak rules or releases to do so, be careful not to touch any live channels (create new rules or releases if necessary). This final verification is as more about making sure the right thing got deployed than whether or not your code is correct.
= Scripts Meeting Notes =These tools all live in * [https:[Balrog/Meetings/hg.mozilla.orgCloudOps - June 22, 2016 | CloudOps Migration Meeting - June 22, 2016]]* [[Balrog/buildMeetings/toolsBalrog+absearch Information Exchange | Balrog/file/default/scripts/updates tools/scripts/updatesabsearch Information Exchange - June 28, 2016]]. We use them to programmatically adjust the update server. == * [[https://hg.mozilla.org/build/tools/file/default/scriptsBalrog/updatesMeetings/balrogCloudOps -submitter.py balrogJune 29, 2016 | CloudOps Migration Meeting -submitter.pyJune 29, 2016]] == Used to submit nightly and release style builds into release blobs. Called on build slaves* [[Balrog/Meetings/CloudOps - July 6, once per each combination of platform2016 | CloudOps Migration Meeting -locale.July 6, 2016]] == * [[https://hg.mozilla.org/build/tools/file/default/scriptsBalrog/updatesMeetings/balrogCloudOps -releaseJuly 12, 2016 | CloudOps Final Cut Over Planning -pusher.py balrog-release-pusher.pyJuly 12, 2016]] ==Used by the release automation to add the metadata to a release blob* [[Balrog/Meetings/Balrog Worker Brainingstorming - July 19, and to push the new release onto the test channels. Example builder name 2016 | Balrog Worker Brainstorming - release-mozilla-beta-firefox_updates.July 19, 2016]] == * [[https://hg.mozilla.org/build/tools/file/default/scriptsBalrog/updatesMeetings/balrogCloudOps -releaseJuly 27, 2016 | CloudOps Meeting -shipper.py balrog-release-shipper.pyJuly 27, 2016]] ==Used by the release automation to push a release to the production chanenl. Example builder name * [[Balrog/Meetings/CloudOps - releaseAugust 10, 2016 | CloudOps Meeting -mozilla-beta-update_shipping.August 10, 2016]] == * [[https:Balrog/Meetings/hg.mozilla.org/build/tools/file/default/scripts/updates/balrogCloudOps -tweaker.py balrogSeptember 7, 2016 | CloudOps Meeting -tweaker.pySeptember 7, 2016]] ==We don't have UI support for modifying blobs, so this is a helper to submit a blob fragment to replace content on the server. Doesn't handle removing keys. Example:<pre>python scripts* [[Balrog/updatesMeetings/balrogCloudOps -tweaker.py September 14, 2016 | CloudOps Meeting --json json -b 'Firefox-33.0-build1' --api-root 'https:September 14, 2016]]* [[Balrog/Meetings/aus4-admin.mozilla.org' --credentials-file cred -u 'nthomas@mozilla.com' CloudOps -v</pre>where json is a file containing the fragment of json to submitSeptember 21, cred is a file containing the password for the 2016 | CloudOps Meeting -u argumentSeptember 21, with format2016]]<pre>balrog_credentials = { 'username': 'password'}</pre> == * [[https://hg.mozilla.org/build/tools/fileBalrog/defaultMeetings/scripts/updates/balrogCloudOps -nightlySeptember 28, 2016 | CloudOps Meeting -locker.py balrog-nightly-locker.pySeptember 28, 2016]] ==Used to 'freeze' nightly updates by pointing to a dated release blob instead of the latest, eg for big code landings, merges. Also to unfreeze afterwards. More details on usage at * [[ReleaseEngineeringBalrog/How_ToMeetings/Enable_or_Disable_Updates_on_Aurora CloudOps - October 19, 2016 | Enable/disable updates on AuroraCloudOps Meeting - October 19, 2016]] == Common code ==* [[https:Balrog/Meetings/hg.mozilla.org/build/tools/file/default/lib/python/balrog/submitter/cli.py cli.pyCloudOps - November 2, 2016 | CloudOps Meeting - November 2, 2016]] and * [[https:Balrog/Meetings/hg.mozilla.org/build/tools/file/default/lib/python/balrog/submitter/api.py api.pyCloudOps - November 9, 2016 | CloudOps Meeting - November 9, 2016]] in tools/lib/python/balrog/submitter/ provide the shared code.
