Balrog: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎What's in a rule?: fix systemCapabilities field name)
(Replaced content with "Moved to https://mozilla-balrog.readthedocs.io/en/latest/infrastructure.html")
 
(60 intermediate revisions by 4 users not shown)
Line 1: Line 1:
= Overview =
Moved to https://mozilla-balrog.readthedocs.io/en/latest/infrastructure.html
Balrog is the software that runs the server side component of the update system used by Firefox and other Mozilla products. It is the successor to [https://wiki.mozilla.org/AUS AUS (Application Update Service)], which did not scale to our current needs nor allow us to adapt to more recent business requirements. Balrog helps us ship updates faster and with much more flexability than we've had in the past.
 
Bugs and feature requests should be filed in the [https://bugzilla.mozilla.org/enter_bug.cgi?component=Balrog%3A%20Backend&product=Release%20Engineering Backend] or [https://bugzilla.mozilla.org/enter_bug.cgi?component=Balrog%3A%20Frontend&product=Release%20Engineering Frontend] components of Bugzilla. You can find the Balrog developers in [irc://irc.mozilla.org/#balrog #balrog], and the [https://github.com/mozilla/balrog code in Github].
 
= Domains =
Balrog admin interface is accessible at [https://aus4-admin.mozilla.org/ aus4-admin.mozilla.org] (vpn required).
 
The public interface that Firefox and other applications talk to is at aus5.mozilla.org. Many older update domains are also served by Balrog, including aus4.mozilla.org, aus3.mozilla.org, and aus2.mozilla.org. More details on these can be found on the [[Balrog/Client_Domains | Client Domains page]].
 
= Database Model =
Balrog's model centres around two concepts: Rules and Releases. When a request for update from an application is received it is matched up against the rules. Once the correct rule has been found, it contains a pointer to a Release, which contains all of the metadata needed to construct a proper update response. Rules and Releases are described in greater detail below:
 
== Rules ==
The most important part of Balrog to understand is its rules. When a request comes in it is matched against each of Balrog's rule to find the one that best suits it (more in this below). Once found, Balrog looks at that rule's "mapping", which points at a release that has the required information to serve an update back to the client. Without any rules, Balrog will never serve an update. With badly configured rules Balrog could do bad things like serve Firefox updates to B2G devices.
 
In addition to the information found in the request, each rule also has a priority, which allows us to override updates for specific things while letting the rest "fall down" to a more general rule. For example, we block unsupported versions of OS' with a rule of the highest priority but continue to serve updates for users of other OS' with a rule of slightly lower priority. The ability to override and fallback by setting the priority is one of the reasons Balrog is so flexible, and a key way that it distinguishes itself from its predecessor.
 
=== What's in a rule? ===
Each rule has quite a lot of columns, but they all fall into one of the buckets below:
* '''Matchable''' - these correspond to information provided in the update request, and are used to filter out rules that don't apply to the request.
* '''Decision''' - these are also used to filter rules, but do not correspond to information in the request.
* '''Response''' - these contain information that ends up in the response
* '''Info''' - informational columns, not used as part of serving updates
 
Individual columns are detailed in the table below:
{| class="wikitable sortable"
|-
! Attribute
! Category
! Description
! Matching logic
! Examples
|- style="background-color: palegreen"
| '''Product'''
| Matchable
| The name of the application requesting an update.
| Exact string match only
| "Firefox" or "B2G"
|- style="background-color: palegreen"
| '''Version'''
| Matchable
| The version of the application requesting an update.
| Exact string match or operator plus version to compare the incoming one against
| "36.0" or ">=38.0a1"
|- style="background-color: palegreen"
| '''Channel'''
| Matchable
| The update channel of the application request an update.
| Exact string match or a string with "*" character to glob
| "nightly" or "beta*"
|- style="background-color: palegreen"
| '''buildTarget'''
| Matchable
| The "build target" of the application requesting an update. This is usually related to the target platform the app was built for.
| Exact string match only
| "Darwin_x86_64-gcc3-u-i386-x86_64" or "flame-kk-userdebug"
|- style="background-color: palegreen"
| '''buildID'''
| Matchable
| The build ID of the application requesting an update.
| Exact string match or operator plus buildid to compare the incoming one against
| "201410010830" or "<201512010830"
|- style="background-color: palegreen"
| '''Locale'''
| Matchable
| The locale of the application requesting an update.
| Exact string match or comma separated list of locales to do an exact match on
| "de" or "en-US,en-GB,id"
|- style="background-color: palegreen"
| '''osVersion'''
| Matchable
| The OS Version of the application requesting an update. This field is primarily used to point desupported operating systems to their last supported build.
| Partial string match or comma separated list of partial strings to match on
| "Windows_NT 5.0" or "Darwin 6,Darwin 7,Darwin 8"
|- style="background-color: palegreen"
| '''systemCapabilities'''
| Matchable
| The supported hardware features of the application requesting an update. This field is primarily used to point desupported users based on their hardware. Eg: users who do not support SSE2
| Full string match or comma separated list of full strings to match on
| "SSE" or "MMX,SSE"
|- style="background-color: palegreen"
| '''distribution'''
| Matchable
| The partner distribution name of the application requesting an update or "default" if the application is not a partner build.
| Exact string match only
| "default" or "yahoo"
|- style="background-color: palegreen"
| '''distVersion'''
| Matchable
| The version of the partner distribution of the application requesting an update or "default" if the application is not a partner build.
| Exact string match only
| "default" or "1.19"
|- style="background-color: palegreen"
| '''headerArchitecture'''
| Matchable
| The architecture of the OS of the client as guessed based on build target. This field is mostly deprecated now that this information is included in the build target.
| Exact string match only
| "PPC" and "Intel" are the only possible values
|- style="background-color: palegreen"
| '''Whitelist'''
| Matchable
| A pointer to a Whitelist blob (stored in the Releases table) that can determine whether an update request is authorized to have the Release this rule is mapped to. Commonly used in whitelisting IMEIs for FirefoxOS updates.
| If a whitelist is present, its shouldServeUpdate is called. If it returns True, this rule is considered to be matching. If it returns False, this rule is thrown out.
| Any valid release name, or NULL
|- style="background-color: lightskyblue"
| '''Priority'''
| Decision
| The priority of the rule, relative to other rules. If multiple rules match an incoming request based on the Matchable columns, the rule with the highest priority is chosen.
| N/A
| Any number, by convention positive integers.
|- style="background-color: lightskyblue"
| '''backgroundRate'''
| Decision
| The percentage of background update requests that should receive an update if they match this rule. Generally this is used as a throttle to increase or decrease the rate at which the majority of users receive an update.
| N/A
| Any number 0 to 10
|- style="background-color: crimson"
|- style="background-color: orange"
| '''Mapping'''
| Response
| The Release to construct an update out of. This is a foreign key to the "name" column of the Releases table.
| N/A
| Any valid release name, or NULL.
|- style="background-color: orange"
| '''update_type'''
| Response
| The update_type to use in the XML response. It's very rare for a rule to use anything other than "minor" these days.
| N/A
| "minor" or "major"
|- style="background-color: crimson"
| '''id'''
| Info
| The id of the rule. This id is necessary to make changes to the rule through the REST API.
| N/A
| Autoincrementing integer
|- style="background-color: crimson"
| '''Alias'''
| Info
| A unique alias for the rule. Can be used in place of id in any REST operation that doesn't involve rule history.
| N/A
| "firefox-release-betatest", "firefox-nightly"
|- style="background-color: crimson"
| '''Comment'''
| Info
| A string describing the purpose of the rule. Not always necessary for obvious rules.
| N/A
| Any string
|}
 
=== How are requests matched up to rules? ===
The incoming request parts match up directly to incoming URL parts. For example, most update requests will send an URL in the following format:
/update/3/<product>/<version>/<buildID>/<buildTarget>/<locale>/<channel>/<osVersion>/<distribution>/<distVersion>/update.xml?force=1 # force can also be left off
 
The following logic is used to figure out which rule an update matches and what to respond with:
# If a rule specifies one of these fields and a request's field doesn't match it, the rule is considered not to be a match and the rule is ignored for that request. See above for details on how specific columns perform matching.
# If "force" wasn't specified, the backgroundRate of the selected rule is looked at.
# If we still choose serve an update after accounting for backgroundRate we look at the rule's mapping. This is a foreign key that points at an entry in the releases table. That row has most of the information we need to construct the update.
# Using the update_type and release that the mapping points to, construct and return an XML response with the details of the update for the client.
 
=== Scheduled Changes ===
Rules may have changes scheduled in advance. Currently these changes may only be scheduled based on a timestamp, but there is the possibility that they could be driven by release uptake, crashes, or other data in the future.
 
Permissions for Scheduled Changes are inherited from regular Rule permissions. If you want to scheduled a change to a Rule, you must have the right permissions to modify that Rule directly. No special permission on top of that is needed to scheduled a change.
 
Scheduled Changes are stored in a separate table that mirrors the main Rules table, and tracks the extra information required to schedule and enact them.
 
== Releases ==
'''needs fleshing out'''<br/>
To Balrog, a "release" is data about a related set of builds. This does _not_ match up with the concept of a "release" being on the "beta", "release" or "esr" channel elsewhere. In Balrog, each set of nightlies on any branch is considered a release.
 
While there's no enforced format on release names, there are a few conventions that we use:
* Nightly-style builds submit to releases named by product and branch. Each nightly generally submits to two different releases, one "dated" (eg: Firefox-mozilla-central-nightly-20150513010203) and one "latest" (eg: Firefox-mozilla-central-nightly-latest).
* Release-style builds submit to releases named by product, version number, and build number, eg: Firefox-38.0-build1
* GMP blobs are created by hand and generally named with the version of each plugin they contain in the name, eg: GMP-20150423-CDM-v4-OpenH264-v1.4
 
More details on blobs can be found on [[Balrog/Blobs | the Blobs page]].
 
== Permissions ==
The permissions table is a simple list of usernames and the ACLs that they have. A user could be an "admin", giving them write access to everything, or could have one or more specific permissions. For example, our "ffxbld" system account has access to make create or modify "Firefox" or "Fennec" releases. These specific ACLs let us do things such as give Go Faster folks access to Balrog without the risk of them or their tools accidentally messing up Firefox updates (or visa versa).
 
The table below describe all possible permissions:
{| class="wikitable sortable"
|-
! Object
! Action
! Options
! Comments
|-
| admin
| No supported actions
| products - If specified, the user can perform any actions on Rules or Releases that affect the specified products.
| An admin user with no options specified has completely unrestricted access to Balrog.
|-
| rowspan="3" | rule
| create
| rowspan="9" | products - If specified, the user only has permission for the object and action if the changes they are making only affect the product specified.
| rowspan="12" |
|-
| modify
|-
| delete
|-
| rowspan="3" | release
| create
|-
| modify
|-
| delete
|-
| rowspan="2" | release_read_only
| set
|-
| unset
|-
| release_locale
| modify
|-
| rowspan="3" | permission
| create
| rowspan="3" | No supported options.
|-
| modify
|-
| delete
|-
| scheduled_change
| enact
| No supported options
| Only the Balrog Agent should be granted this permission.
|}
 
== History Tables ==
Change attribution and recording is embedded deeply into Balrog. The rules, releases, and permissions tables all have a corresponding history table that records the time a change was made and who made it. This allows us to look back in time when debugging issues, attribute changes to people (aka blame), and quickly roll back bad changes.
 
= Admin App =
== API ==
See the [[Balrog/Admin API | Admin API page]].
== UI Use Cases ==
=== Locking/Unlocking Nightlies ===
One of the most common uses for the Balrog UI is to lock a nightly update channel to a specific release for a period of time, and then unlock it later (so that users on that channel start receiving the latest available build again). This is often done if a serious bug is introduced to minimize the number of users affected by it.
 
Taking the B2G nightly channel as an example, let's see how we would lock it to the nightlies from 20150505160203:
# Log in to https://aus4-admin.mozilla.org
# Click on the "Rules" link at the top of the page
# Use the filter in the top right to narrow down the rules to "product:B2G channel:nightly"
# Locate the rule (or rules) on the "nightly" channel
#* Changing the sort to "Product, Channel" will group things together better.
# For each rule on the channel:
#* Click the "Update" button to enter edit mode
#* Find the mapping field and replace the "-latest" part with "-20150505160203" (the UI will autocomplete this for you if you start typing).
#* Scroll down and click "Save Changes"
 
When you're ready to unlock the updates, follow the same steps as above but replace the "-20150505160203" part of the mapping with "-latest" again.
 
=== Adding a rule for a new update channel ===
When nightly builds are set up on a new branch, rules need to be added to Balrog for updates to be served. Note that the nightly build automation is responsible for providing metadata about each new set of builds to Balrog.
 
As an example, here is how B2G updates could be set up on a hypothetical "mozilla-b2g40" branch:
# Log in to https://aus4-admin.mozilla.org
# Click on the "Rules" link at the top of the page
# Click on "Add a new Rule" near the top left of the page
# Fill out the form as follows:
#* Product: B2G
#* Channel: nightly-b2g40
#* Mapping: B2G-mozilla-b2g40-nightly-latest
#* Rate: 100
#* Priority: 90
# Click "Save Changes"
 
=== Modify an existing release ===
Most modifications to releases are done by automation, but sometimes we need to tweak them by hand. For example, when adjusting What's New page configuration.
 
For example, if you wanted to modify the "Firefox-38.0-build3" release, follow these steps:
# Log in to https://aus4-admin.mozilla.org/
# Click the "Releases" link at the top of the page
# Find the "Firefox-38.0-build3" release and click the "Download" link
# Save the file locally and modify it to your liking
# Click the "Update" link for "Firefox-38.0-build3"
# Click "Browse" and choose your new local version
# Click "Save Changes"
 
= Balrog Agent =
The Balrog Agent is a long running process that is responsible for monitoring and enacting [[#Scheduled Changes | Scheduled Changes]]. The Agent is a separate application that runs in its own Docker container. In order to ensure it cannot bypass permissions or history (accidentally or on purpose), the Agent acts as a client to the [[#Admin App | Admin App]] rather than interacting with database directly. This means that it must be granted [[#Permissions | the "scheduled_change" permission]] to function.
 
In the future, the Agent may also act as a client to Telemetry, Crash Stats, or other systems that contain data that we may want to scheduled changes with.
 
= Code Overview =
Balrog's code is organized into roughly the following parts:
* [https://github.com/mozilla/balrog/tree/master/auslib/blobs The blobs] -  These contain most of the brains (business logic) behind Balrog. They know how to validate new data coming into the system and translate existing data into useful responses to update requests.
* [https://github.com/mozilla/balrog/blob/master/auslib/db.py The database abstraction layer] - This layer sits between the actual database and the applications. It defines the database schema, performs permissions checking, and ensures all changes are written to history tables. Application should never touch the database directly - they should always go through this layer.
* [https://github.com/mozilla/balrog/tree/master/auslib/web The user-facing application] - The entry point to requests from applications looking for updates.
* [https://github.com/mozilla/balrog/tree/master/auslib/admin The admin API] - A simple RESTful API that allows the Admin UI and automation to make changes to Balrog's database.
* [https://github.com/mozilla/balrog/tree/master/ui The admin UI] A human friendly interface to manage updates.
* [https://github.com/mozilla/balrog/tree/master/agent The Balrog Agent] A long running process that is responsible for enacting Scheduled Changes.
 
= Hacking =
Balrog's code is available in [https://github.com/mozilla/balrog the Github repository]. To get it and get ready to hack, run:
git clone https://github.com/mozilla/balrog
cd balrog
 
== Docker ==
You can run a complete Balrog environment locally with Docker, configured very similarly to production. You'll need Docker and Docker Compose installed; on Mac or Windows we require Docker for Mac/Docker for Windows v1.12 or higher. Then run the following in the root of the Balrog repo:
docker-compose up
 
Once it completes, you should be able to access
* the admin interface at [http://127.0.0.1:8080 http://127.0.0.1:8080]
* the public interface on port 9090, for example [http://127.0.0.1:9090/update/3/Firefox/33.0/20141202185629/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/default/default/default/update.xml?force=1 http://127.0.0.1:9090/update/3/Firefox/33.0/20141202185629/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/default/default/default/update.xml?force=1]
 
== UI hacking ==
Once the Docker images are running, any changes you make to js or css files should cause lineman to automatically rebuild them in the balrogui container - you do not need to run lineman or npm on your host machine. However, You will still need to shift-refresh in your browser to pick them up.
 
== Unit Tests ==
=== Backend ===
The recommended way to run backend tests is through Docker. Doing so will ensure your tests will run the same way they do in CI. To run them, use this command in the root of your Balrog repository:
./run-tests.sh backend
 
If you can't or don't want to run tests within Docker for some reason, you can run them with directly with:
tox
 
Tests run fine on any posix-like environment, but are only run regularly within the Docker image, so it's possible to have failures that aren't related to Balrog code or your changes when running directly with tox.
 
=== Frontend ===
To run the unit tests use:
./run-tests.sh frontend
 
More options can be found in [https://github.com/mozilla/balrog-ui/blob/master/README.md ui/README.md]
 
= Getting Involved =
If you like to get involved in the development of Balrog there's lots of areas where we could use some help. Below is a list of specific tasks that would be good first choices. Come talk to us in [irc://irc.mozilla.org/#balrog #balrog] if you're interested!
 
<bugzilla>
{
"quicksearch": "status:new,assigned,reopened,unconfirmed comp:balrog mentor:'@' assignee:nobody@mozilla.org whiteboard:[good first bug]",
"product": "Release Engineering",
"include_fields": "id,component,summary,status"
}
</bugzilla>
 
= Meeting Notes =
See the [[Balrog/Meetings | Meetings page]].
 
= Running without Docker Compose =
== Creating a database ==
Balrog's database is controlled through sqlalchemy-migrate. To initialize a new Balrog database, run the following:
docker run --entrypoint python mozilla/balrog /app/scripts/manage-db.py -d DBURI create
 
Similarly, to upgrade the schema of an existing Balrog database, run the following:
docker run --entrypoint python mozilla/balrog /app/scripts/manage-db.py -d DBURI upgrade
 
See the [[#Environment Variables | "Environment Variables" section below]] for DBURI format. If your testing out local changes that affect database creation or upgrades, you should replace "mozilla/balrog" with your local image.
 
== Environment Variables ==
The following environment variables are required by the Balrog WSGI apps:
* DBURI - The database to use, in the format: driver://user:password@host/database.
* SECRET_KEY - A pseudorandom string to use when generating CSRF tokens. Only used for the admin app.
 
These are optional:
* LOG_LEVEL - Controls the python level the app logs at. Set to INFO by default.
* LOG_FORMAT - Controls the log format. If unset, mozlog format (json) will be used. Can be overridden with "plain" to log simple plain-text messages. The former is recommended for production, the latter for local development.
* NOTIFY_TO_ADDR - An address to send e-mail to if a Rule or Permission changes. Unset by default, and only used for the admin app. If set, the following additional variables are required:
** SMTP_HOST, SMTP_PORT, SMTP_USERNAME, SMTP_PASSWORD - Information about the SMTP relay to send mail through.
** NOTIFY_FROM_ADDR - The "from" address to use when sending mail.
 
= Infrastructure =
== Support & Escalation ==
''If the issue may be visible to users, please make sure [irc://irc.mozilla.org/#moc #moc] is also notified. They can also assist with the notifications below.''
 
RelEng is the first point of contact for issues. To contact them, follow [[ReleaseEngineering#Contacting_Release_Engineering|the standard RelEng escalation path]].
 
If RelEng is unable to correct the issue, they may [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations escalate to CloudOps].
 
== Monitoring & Metrics ==
TODO: add more, flesh out.
 
* [https://app.datadoghq.com/dash/156924/balrog-web-aus5mozillaorg?live=true&page=0&is_auto=false&tile_size=m&fullscreen=false Datadog Dashboard]
 
== Nginx Logs ==
The nginx logs for the public-facing application are replicated to the '''balrog-us-west-2-elb-logs''' S3 bucket, located in us-west-2. Logs are rotated very quickly, and we end up with tens of thousands of separate files each day. Because of this, and the fact that S3 has a lot of overhead per-file, it can be tricky to do analysis on them. You're unlikely to be able to download the logs locally in any reasonable amount of time (ie, less than a day), but mounting them on an EC2 instance in us-west-2 should provide you with reasonably quick access. Here's an example:
* Launch EC2 instance (you probably a compute-optimized one, and at least 100GB of storage).
* Generate an access token for your CloudOps AWS account. If you don't have a CloudOps AWS account, talk to Ben Hearsum or Bensong Wong. Put the token in a plaintext file somewher on the instance.
** If you've chosen local storage, you'll probably need to format and mount volume.
* Install s3fs by following the instructions on https://github.com/s3fs-fuse/s3fs-fuse.
* Mount the bucket on your instance, eg:
s3fs balrog-us-west-2-elb-logs /media/bucket -o passwd_file=pw.txt
* Do some broad grepping directly on the S3 logs, and store it in a local file. This should speed up subsequent queries. Eg:
grep '/Firefox/.*WINNT.*/release/' /media/bucket/AWSLogs/361527076523/elasticloadbalancing/us-west-2/2016/09/17/* | gzip > /media/ephemeral0/sept-17-winnt-release.txt.gz
* Do additional queries on the new logfile.
 
== Backups ==
Balrog uses the built-in RDS backups. The database in snapshotted nightly, and incremental backups are done throughout the day. If necessary, we have the ability to recover to within a 5 minute window. Database restoration is done by CloudOps, and they should be contacted immediately if needed.
 
== Deploying Changes ==
Balrog's [https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Update+Service stage and production infrastructure] is managed by [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations the Cloud Operations team]. This section describes how to go from a reviewed patch to deploying it in production.
 
=== Is now a good time? ===
Before you deploy, consider whether or not it's an appropriate time to. Some factors to consider:
* Are we in the middle of an important release such as a chemspill? If so, it's probably not a good time to deploy.
* How risky are your changes? If they're high risk, deploying on a Friday is probably a bad idea.
* Do you need to migrate any data? If you do, make sure you have time to do so right after deploying.
* Do you have enough time to safely do a push? Most pushes take at most 60 minutes to complete after the stage push has been done. This time is mostly affected by how long it takes you to verify your changes in stage and production.
 
=== Landing ===
==== Schema Upgrades ====
If you need to do a schema change you '''must''' ensure that either the current production code can run with your schema change applied, or that your new code can run with the old schema. Code and schema changes '''cannot''' be done at the same instant, so you must be able to support one of these scenarios. Generally, additive changes (column or table additions) should do the schema change first, while destructive changes (column or table deletions) should do the schema change second. You can simulate the upgrade with your local Docker containers to verify which is right for you.
 
When you file the deployment bug (see below), include a note about the schema change in it. Something like:
This push requires a schema change that needs to be done _prior_ to the new code going out. Thas can be performed by running the Docker image with the "upgrade-db" command, with DBURI set.
 
{{bug|1295678}} is an example of a push with a schema change.
 
=== Testing ===
Before asking for a push, you should do some functional testing on your local machine with the [[#Docker | Docker]] images. You should do this against [https://github.com/mozilla/balrog the master branch of the upstream repository] to ensure you're testing the exact code that is to be deployed. At the very least, you should do explicit testing of all the new code that would be included in the push. Eg: if you're changing the format of a blob, make sure that you can add a new blob of that type, and that the XML response looks correct.
 
=== Pushing to stage and production ===
Pushing live is a two step process. First, you must push to the stage environment and ensure things are working there. Then, you can push live.
# Bump the [https://github.com/mozilla/balrog/blob/master/version.txt in-repo version].
# [https://github.com/mozilla/balrog/releases Tag the repository] with a "vX.Y" tag. Eg: "git tag -s vX.Y"
# Wait for CI jobs to complete. Unit tests must pass and a new Docker Image [https://hub.docker.com/r/mozilla/balrog/tags/ for the webapps] and [https://hub.docker.com/r/mozilla/balrogagent/tags/ the Agent] must be pushed to Dockerhub before you proceed.
# [https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=bwong%40mozilla.com&bug_file_loc=http%3A%2F%2F&bug_ignored=0&bug_severity=normal&bug_status=NEW&cc=bwong%40mozilla.com&cc=ckolos%40mozilla.com&cc=bhearsum%40mozilla.com&cf_blocking_b2g=---&cf_blocking_fennec=---&cf_fx_iteration=---&cf_fx_points=---&cf_status_b2g_2_6=---&cf_status_firefox48=---&cf_status_firefox49=---&cf_status_firefox50=---&cf_status_firefox51=---&cf_status_firefox_esr45=---&cf_tracking_b2g=---&cf_tracking_e10s=---&cf_tracking_firefox48=---&cf_tracking_firefox49=---&cf_tracking_firefox50=---&cf_tracking_firefox51=---&cf_tracking_firefox_esr45=---&cf_tracking_firefox_relnote=---&comment=Balrog%20version%20X.Y%20is%20ready%20to%20be%20pushed%20to%20stage.%20Please%20deploy%20the%20new%20Docker%20image%20%28TAG_NAME%29%20the%20admin%20and%20public%20webapps%2C%20and%20the%20new%20Docker%20image%20for%20the%20Agent%20%28TAG_NAME%29.%0D%0A%0D%0AWe%27d%20like%20the%20production%20push%20for%20this%20to%20happen%20sometime%20between%2011am%20and%201pm%20pacific%20on%20%28DATE%20GOES%20HERE%29&component=Operations%3A%20Deployment%20Requests&contenttypemethod=autodetect&contenttypeselection=text%2Fplain&defined_groups=1&flag_type-37=X&flag_type-4=X&flag_type-5=X&flag_type-607=X&flag_type-708=X&flag_type-720=X&flag_type-721=X&flag_type-737=X&flag_type-781=X&flag_type-787=X&flag_type-800=X&flag_type-803=X&flag_type-846=X&flag_type-864=X&flag_type-875=X&flag_type-892=X&flag_type-901=X&flag_type-905=X&flag_type-908=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=--&product=Cloud%20Services&rep_platform=Unspecified&short_desc=please%20deploy%20balrog%20X.Y%20to%20stage%2Fprod&target_milestone=---&version=unspecified File a bug] to have the new version pushed to stage. Be sure to include the new version number, and Docker image tags for the webapps and the Agent you want deployed.
# Once stage has been updated, verify your changes again. Even though you've tested locally, it's important to retest in stage to make sure there's no deployment-specific issues.
# When stage looks good, you're ready to comment in the bug to ask for production to be updated.
# Reverify in production. When production has been updated, verify your changes again there. If you need to tweak rules or releases to do so, be careful not to touch any live channels (create new rules or releases if necessary). This final verification is as more about making sure the right thing got deployed than whether or not your code is correct.
 
= Scripts =
These tools all live in [https://hg.mozilla.org/build/tools/file/default/scripts/updates tools/scripts/updates]. We use them to programmatically adjust the update server.
 
== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-submitter.py balrog-submitter.py] ==
 
Used to submit nightly and release style builds into release blobs. Called on build slaves, once per each combination of platform-locale.
 
== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-release-pusher.py balrog-release-pusher.py] ==
Used by the release automation to add the metadata to a release blob, and to push the new release onto the test channels. Example builder name - release-mozilla-beta-firefox_updates.
 
== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-release-shipper.py balrog-release-shipper.py] ==
Used by the release automation to push a release to the production chanenl. Example builder name - release-mozilla-beta-update_shipping.
 
== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-tweaker.py balrog-tweaker.py] ==
We don't have UI support for modifying blobs, so this is a helper to submit a blob fragment to replace content on the server. Doesn't handle removing keys.
 
Example:
<pre>python scripts/updates/balrog-tweaker.py --json json -b 'Firefox-33.0-build1' --api-root 'https://aus4-admin.mozilla.org'  --credentials-file cred -u 'nthomas@mozilla.com' -v</pre>
where json is a file containing the fragment of json to submit, cred is a file containing the password for the -u argument, with format
<pre>balrog_credentials = {
    'username': 'password'
}</pre>
 
== [https://hg.mozilla.org/build/tools/file/default/scripts/updates/balrog-nightly-locker.py balrog-nightly-locker.py] ==
Used to 'freeze' nightly updates by pointing to a dated release blob instead of the latest, eg for big code landings, merges. Also to unfreeze afterwards. More details on usage at [[ReleaseEngineering/How_To/Enable_or_Disable_Updates_on_Aurora | Enable/disable updates on Aurora]]
 
== Common code ==
[https://hg.mozilla.org/build/tools/file/default/lib/python/balrog/submitter/cli.py cli.py] and [https://hg.mozilla.org/build/tools/file/default/lib/python/balrog/submitter/api.py api.py] in tools/lib/python/balrog/submitter/ provide the shared code.

Latest revision as of 14:49, 9 January 2020

Moved to https://mozilla-balrog.readthedocs.io/en/latest/infrastructure.html

Retrieved from "https://wiki.mozilla.org/index.php?title=Balrog&oldid=1222143"