Balrog: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Replaced content with "Moved to https://mozilla-balrog.readthedocs.io/en/latest/infrastructure.html")
 
(37 intermediate revisions by 3 users not shown)
Line 1: Line 1:
If you are looking for the general documentation that used to live here, it has been moved into [https://github.com/mozilla/balrog the Balrog repository], and a built version of it [http://mozilla-balrog.readthedocs.io/en/latest/index.html is available on Read The Docs].
Moved to https://mozilla-balrog.readthedocs.io/en/latest/infrastructure.html
 
This page will continue to host information about Balrog that doesn't make sense to put into the repository, such as meeting notes and things related to our hosted versions of Balrog.
 
= Infrastructure =
== Domains ==
Balrog admin interface is accessible at [https://aus4-admin.mozilla.org/ aus4-admin.mozilla.org] (vpn required).
 
The public interface that Firefox and other applications talk to is at aus5.mozilla.org. Many older update domains are also served by Balrog, including aus4.mozilla.org, aus3.mozilla.org, and aus2.mozilla.org. More details on these can be found on the [[Balrog/Client_Domains | Client Domains page]].
 
== Support & Escalation ==
''If the issue may be visible to users, please make sure [irc://irc.mozilla.org/#moc #moc] is also notified. They can also assist with the notifications below.''
 
RelEng is the first point of contact for issues. To contact them, follow [[ReleaseEngineering#Contacting_Release_Engineering|the standard RelEng escalation path]].
 
If RelEng is unable to correct the issue, they may [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations escalate to CloudOps].
 
== Monitoring & Metrics ==
Metrics from RDS, EC2, and Nginx are available in the [https://app.datadoghq.com/dash/156924/balrog-web-aus5mozillaorg?live=true&page=0&is_auto=false&tile_size=m&fullscreen=false Datadog Dashboard].
 
We aggregate exceptions from both the [https://sentry.prod.mozaws.net/operations/prod-public/ public apps] and [https://sentry.prod.mozaws.net/operations/prod-admin/ admin app] to [https://sentry.prod.mozaws.net/operations/ CloudOps' Sentry instance].
 
== Change Notification ==
Changes made to Rules, Scheduled Rule Changes, Permissions, or the read-only flag of a Release will send e-mail notification to the [https://groups.google.com/a/mozilla.com/forum/#!forum/balrog-db-changes balrog-db-changes mailing list]. This serves us an alert system - if we see changes made that we weren't expecting, we can go investigate them.
 
== ELB Logs ==
'''NOTE: ''' These instructions were written before [https://aws.amazon.com/blogs/aws/amazon-athena-interactive-sql-queries-for-data-in-amazon-s3/ Amazon Athena existed]. The next time we need to do such analysis, it's probably worth giving it a try. Other techniques may be better too - the instructions below are just something we've done in the past.
 
The ELB logs for the public-facing application are replicated to the '''balrog-us-west-2-elb-logs''' S3 bucket, located in us-west-2. Logs are rotated very quickly, and we end up with tens of thousands of separate files each day. Because of this, and the fact that S3 has a lot of overhead per-file, it can be tricky to do analysis on them. You're unlikely to be able to download the logs locally in any reasonable amount of time (ie, less than a day), but mounting them on an EC2 instance in us-west-2 should provide you with reasonably quick access. Here's an example:
* Launch EC2 instance (you probably a compute-optimized one, and at least 100GB of storage).
* Generate an access token for your CloudOps AWS account. If you don't have a CloudOps AWS account, talk to Ben Hearsum or Bensong Wong. Put the token in a plaintext file somewhere on the instance.
** If you've chosen local storage, you'll probably need to format and mount volume.
* Install s3fs by following the instructions on https://github.com/s3fs-fuse/s3fs-fuse.
* Mount the bucket on your instance, eg:
s3fs balrog-us-west-2-elb-logs /media/bucket -o passwd_file=pw.txt
* Do some broad grepping directly on the S3 logs, and store it in a local file. This should speed up subsequent queries. Eg:
grep '/Firefox/.*WINNT.*/release/' /media/bucket/AWSLogs/361527076523/elasticloadbalancing/us-west-2/2016/09/17/* | gzip > /media/ephemeral0/sept-17-winnt-release.txt.gz
* Do additional queries on the new logfile.
 
== Backups ==
Balrog uses the built-in RDS backups. The database in snapshotted nightly, and incremental backups are done throughout the day. If necessary, we have the ability to recover to within a 5 minute window. Database restoration is done by CloudOps, and they should be contacted immediately if needed.
 
== Deploying Changes ==
Balrog's [https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Update+Service stage and production infrastructure] is managed by [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations the Cloud Operations team]. This section describes how to go from a reviewed patch to deploying it in production. You should generally begin this process '''at least 1 hour''' before you want the new code live in production. This gives CloudOps plenty of time to respond to the request.
 
=== Is now a good time? ===
Before you deploy, consider whether or not it's an appropriate time to. Some factors to consider:
* Are we in the middle of an important release such as a chemspill? If so, it's probably not a good time to deploy.
* How risky are your changes? If they're high risk, deploying on a Friday is probably a bad idea.
* Do you need to migrate any data? If you do, make sure you have time to do so right after deploying.
* Do you have enough time to safely do a push? Most pushes take at most 60 minutes to complete after the stage push has been done. This time is mostly affected by how long it takes you to verify your changes in stage and production.
 
=== Schema Upgrades ===
If you need to do a schema change you '''must''' ensure that either the current production code can run with your schema change applied, or that your new code can run with the old schema. Code and schema changes '''cannot''' be done at the same instant, so you must be able to support one of these scenarios. Generally, additive changes (column or table additions) should do the schema change first, while destructive changes (column or table deletions) should do the schema change second. You can simulate the upgrade with your local Docker containers to verify which is right for you.
 
When you file the deployment bug (see below), include a note about the schema change in it. Something like:
This push requires a schema change that needs to be done _prior_ to the new code going out. That can be performed by running the Docker image with the "upgrade-db" command, with DBURI set.
 
{{bug|1295678}} is an example of a push with a schema change.
 
=== Testing ===
The stage environment automatically deploys new code from the master branch of [https://github.com/mozilla/balrog the Balrog repository] (including any necessary schema changes). Before asking for a push, you should do some functional testing there. At the very least, you should do explicit testing of all the new code that would be included in the push. Eg: if you're changing the format of a blob, make sure that you can add a new blob of that type, and that the XML response looks correct. When you've completed testing take note of the "commit" on https://balrog-admin.stage.mozaws.net/__version__ - you'll need it when filing the production push bug.
 
=== Pushing to production ===
Pushing live requires CloudOps. For non-urgent pushes, please make sure you file the bug for them X hours in advance. For urgent pushes, file the bug immediately and [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations escalate if no action is taken quickly]. Either way, you must follow this procedure to push:
 
# [https://github.com/mozilla/balrog/releases Tag the repository] with a "vX.Y" tag. Eg: "git tag -s vX.Y"
# Diff against the previous release tag. Eg: "git diff v2.24 v2.25"
#* Look for anything unexpected, or any '''schema changes'''. If schema changes are present, see the above section for instructions on handling them.
# Wait for CI jobs to complete. Unit tests must pass and a new Docker Image [https://hub.docker.com/r/mozilla/balrog/tags/ for the webapps] and [https://hub.docker.com/r/mozilla/balrogagent/tags/ the Agent] must be pushed to Dockerhub before you proceed.
# [https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=dthorn%40mozilla.com&bug_file_loc=http%3A%2F%2F&bug_ignored=0&bug_severity=normal&bug_status=NEW&cc=dthorn%40mozilla.com&cc=miles%40milescrabill.com&cf_blocking_fennec=---&cf_fx_iteration=---&cf_fx_points=---&cf_status_firefox52=---&cf_status_firefox53=---&cf_status_firefox54=---&cf_status_firefox55=---&cf_status_firefox_esr45=---&cf_status_firefox_esr52=---&cf_tracking_firefox52=---&cf_tracking_firefox53=---&cf_tracking_firefox54=---&cf_tracking_firefox55=---&cf_tracking_firefox_esr45=---&cf_tracking_firefox_esr52=---&cf_tracking_firefox_relnote=---&comment=Balrog%20version%20X.Y%20is%20ready%20to%20be%20pushed%20to%20prod.%20Please%20deploy%20the%20new%20Docker%20images%20%28master-%24COMMIT%29%20for%20admin%2C%20public%2C%20and%20the%20agent.%0D%0A%0D%0AWe%27d%20like%20the%20production%20push%20for%20this%20to%20happen%20sometime%20between%2011am%20and%201pm%20pacific%20on%20%28DATE%20GOES%20HERE%29&component=Operations%3A%20Deployment%20Requests&contenttypemethod=autodetect&contenttypeselection=text%2Fplain&defined_groups=1&flag_type-37=X&flag_type-4=X&flag_type-5=X&flag_type-607=X&flag_type-708=X&flag_type-720=X&flag_type-721=X&flag_type-737=X&flag_type-781=X&flag_type-787=X&flag_type-800=X&flag_type-803=X&flag_type-846=X&flag_type-864=X&flag_type-905=X&flag_type-914=X&flag_type-916=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=--&product=Cloud%20Services&rep_platform=Unspecified&short_desc=please%20deploy%20balrog%20X.Y%20to%prod&target_milestone=---&version=unspecified File a bug] to have the new version pushed to production. Be sure to substitute the new version number, and commit SHA for the Docker image tags.
#* This bug should generally be filed at least a few hours in advance of the desired production push time to ensure that CloudOps has time to react to it.
#* Wednesdays are usually the best day to push to production, because they are generally free of release events. Unless you have a specific need to deploy on a different day, you should request the prod push for a Wednesday.
#* You should link any bugs being deployed is the "Blocks" field.
# Once the push has happened, verify the changes in production. If you need to tweak rules or releases to do so, be careful not to touch any live channels (create new rules or releases if necessary). This final verification is as more about making sure the right thing got deployed than whether or not your code is correct.
# Bump the [https://github.com/mozilla/balrog/blob/master/version.txt in-repo version] to the next available one to ensure the next push gets a new version.
 
= Meeting Notes =
* [[Balrog/Meetings/CloudOps - June 22, 2016 | CloudOps Migration Meeting - June 22, 2016]]
* [[Balrog/Meetings/Balrog+absearch Information Exchange | Balrog/absearch Information Exchange - June 28, 2016]]
* [[Balrog/Meetings/CloudOps - June 29, 2016 | CloudOps Migration Meeting - June 29, 2016]]
* [[Balrog/Meetings/CloudOps - July 6, 2016 | CloudOps Migration Meeting - July 6, 2016]]
* [[Balrog/Meetings/CloudOps - July 12, 2016 | CloudOps Final Cut Over Planning - July 12, 2016]]
* [[Balrog/Meetings/Balrog Worker Brainingstorming - July 19, 2016 | Balrog Worker Brainstorming - July 19, 2016]]
* [[Balrog/Meetings/CloudOps - July 27, 2016 | CloudOps Meeting - July 27, 2016]]
* [[Balrog/Meetings/CloudOps - August 10, 2016 | CloudOps Meeting - August 10, 2016]]
* [[Balrog/Meetings/CloudOps - September 7, 2016 | CloudOps Meeting - September 7, 2016]]
* [[Balrog/Meetings/CloudOps - September 14, 2016 | CloudOps Meeting - September 14, 2016]]
* [[Balrog/Meetings/CloudOps - September 21, 2016 | CloudOps Meeting - September 21, 2016]]
* [[Balrog/Meetings/CloudOps - September 28, 2016 | CloudOps Meeting - September 28, 2016]]
* [[Balrog/Meetings/CloudOps - October 19, 2016 | CloudOps Meeting - October 19, 2016]]
* [[Balrog/Meetings/CloudOps - November 2, 2016 | CloudOps Meeting - November 2, 2016]]
* [[Balrog/Meetings/CloudOps - November 9, 2016 | CloudOps Meeting - November 9, 2016]]
* [[Balrog/Meetings/CloudOps - January 18, 2017 | CloudOps Meeting - January 17, 2017]]
* [[Balrog/Meetings/CloudOps - February 22, 2017 | CloudOps Meeting - February 22, 2017]]

Latest revision as of 14:49, 9 January 2020

Moved to https://mozilla-balrog.readthedocs.io/en/latest/infrastructure.html

Retrieved from "https://wiki.mozilla.org/index.php?title=Balrog&oldid=1222143"