Changes

Jump to: navigation, search

Balrog

352 bytes added, 15:17, 13 October 2017
Deploying Changes
== Deploying Changes ==
Balrog's [https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Update+Service stage and production infrastructure] is managed by [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations the Cloud Operations team]. This section describes how to go from a reviewed patch to deploying it in production. You should generally begin this process '''at least 24 hours''' before you want the new code live in production. This gives the new code a chance to bake in stage.
 
At a high level, the deployment process looks like this:
* Verify the new code in dev
* Bake the new code in stage
* Deploy to prod
 
Each part of this process is described in more detail below.
=== Is now a good time? ===
{{bug|1295678}} is an example of a push with a schema change.
=== Testing Verification in dev ===
The dev environment automatically deploys new code from the master branch of [https://github.com/mozilla/balrog the Balrog repository] (including any necessary schema changes). Before beginning the deployment procedure, you should do some functional testing there. At the very least, you should do explicit testing of all the new code that would be included in the push. Eg: if you're changing the format of a blob, make sure that you can add a new blob of that type, and that the XML response looks correct.
'''If you have schema changes''' you must also ensure that the existing deployed code will work with the new schema. To do this, CloudOps will downgrade the dev apps. You should do some routine testing (make some changes to some objects, try some update requests) to ensure that everything works. If you have any issues you CANNOT proceed to production.
=== Pushing to production Baking in stage ===Pushing live requires CloudOps. For non-urgent pushes, you should begin this procedure '''at least 24 hours To get the new code in advance'''. For urgent pushes, file the bug immediately and [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations escalate if no action is taken quickly]. Either way, stage you must follow this procedure to pushcreate a new Release in Github as follows
# Tag the repository with a "vX.Y" tag. Eg: "git tag -s vX.Y && git push --tags"
# Diff against the previous release tag. Eg: "git diff v2.24 v2.25"
#* Look for anything unexpected, or any '''schema changes'''. If schema changes are present, see the above section for instructions on handling them.
# [https://github.com/mozilla/balrog/releases Create a new Release on Github]. This create new Docker images tagged with your version, and deploy them to stage. It may take upwards of 30 minutes for the deployment to happen.
# Let things Once the changes are deployed to stage, let them bake on stage for at least 24 hours. You can do additional targeted testing here if you wish, or simply wait for nightlies/releases to prod things along.It's a good idea to watch Sentry for new exceptions that may show up, and Datadog for any notable changes in the shape of the traffic. === Pushing to production ===Pushing live requires CloudOps. For non-urgent pushes, you should begin this procedure a few hours in advance to give CloudOps time to notice and respond. For urgent pushes, file the bug immediately and [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations escalate if no action is taken quickly]. Either way, you must follow this procedure to push:
# [https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=dthorn%40mozilla.com&bug_file_loc=http%3A%2F%2F&bug_ignored=0&bug_severity=normal&bug_status=NEW&cc=dthorn%40mozilla.com&cc=miles%40milescrabill.com&cf_blocking_fennec=---&cf_fx_iteration=---&cf_fx_points=---&cf_status_firefox56=---&cf_status_firefox57=---&cf_status_firefox58=---&cf_status_firefox_esr52=---&cf_tracking_firefox56=---&cf_tracking_firefox57=---&cf_tracking_firefox58=---&cf_tracking_firefox_esr52=---&cf_tracking_firefox_relnote=---&comment=Balrog%20version%20X.Y%20is%20ready%20to%20be%20pushed%20to%20prod.%20Please%20deploy%20the%20new%20Docker%20images%20%28vX.Y%29%20for%20admin%2C%20public%2C%20and%20the%20agent.%0D%0A%0D%0AWe%27d%20like%20the%20production%20push%20for%20this%20to%20happen%20sometime%20between%2011am%20and%201pm%20pacific%20on%20%28DATE%20GOES%20HERE%29.%0D%0A%0D%0AONE%20OF%3A%0D%0A%28NO%20SCHEMA%20CHANGE%20OR%20ADDITIVE%20SCHEMA%20CHANGE%29%20If%20anything%20goes%20wrong%20with%20the%20new%20version%20in%20production%2C%20we%20can%20safely%20rollback%20to%20the%20previous%20versions%20of%20admin%2C%20agent%2C%20public%2C%20and%20public%20in%20that%20order.%0D%0A%0D%0A%28DESTRUCTIVE%20SCHEMA%20CHANGE%29%20Because%20this%20push%20contains%20a%20destructive%20schema%20change%2C%20rollback%20to%20the%20previous%20version%20must%20be%20done%20as%20follows%3A%0D%0A%2A%20Revert%20the%20schema%20change%20%28run%20%22python%20%2Fapp%2Fscripts%2Fmanage-db.py%20-d%20%24DBURI%20--version%20X%22%20in%20the%20admin%20container%29%0D%0A%2A%20Rollback%20admin%2C%20agent%2C%20and%20public%20in%20that%20order.%0D%0A%0D%0A%28UNABLE%20TO%20COME%20UP%20WITH%20SAFE%20ROLLBACK%20PROCEDURE%20-%20THIS%20SHOULD%20NEVER%20HAPPEN%29%20This%20push%20contains%20changes%20that%20cannot%20be%20safely%20rolled%20back.%20If%20anything%20goes%20wrong%20with%20the%20new%20version%20in%20production%2C%20please%20escalate%20to%20me.&component=Operations%3A%20Deployment%20Requests&contenttypemethod=autodetect&contenttypeselection=text%2Fplain&defined_groups=1&flag_type-37=X&flag_type-4=X&flag_type-5=X&flag_type-607=X&flag_type-708=X&flag_type-721=X&flag_type-737=X&flag_type-787=X&flag_type-800=X&flag_type-803=X&flag_type-846=X&flag_type-864=X&flag_type-914=X&flag_type-916=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=--&product=Cloud%20Services&rep_platform=Unspecified&short_desc=please%20deploy%20balrog%20X.Y%20to%20prod&target_milestone=---&version=unspecified File a bug] to have the new version pushed to production.
#* This bug should generally be filed at least a few hours in advance of the desired production push time to ensure that CloudOps has time to react to it.
#* Wednesdays are usually the best day to push to production, because they are generally free of release events. Unless you have a specific need to deploy on a different day, you should request the prod push for a Wednesday.
#* You should link any bugs being deployed is the "Blocks" field.
Canmove, confirm
6,438
edits

Navigation menu