canmove, Confirmed users
6,442
edits
Balrog: Difference between revisions
Jump to navigation
Jump to search
→Deploying Changes
| Line 42: | Line 42: | ||
== Deploying Changes == | == Deploying Changes == | ||
Balrog's [https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Update+Service stage and production infrastructure] is managed by [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations the Cloud Operations team]. This section describes how to go from a reviewed patch to deploying it in production. | Balrog's [https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Update+Service stage and production infrastructure] is managed by [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations the Cloud Operations team]. This section describes how to go from a reviewed patch to deploying it in production. You should generally begin this process '''at least 1 hour''' before you want the new code live in production. This gives CloudOps plenty of time to respond to the request. | ||
=== Is now a good time? === | === Is now a good time? === | ||
| Line 53: | Line 51: | ||
* Do you have enough time to safely do a push? Most pushes take at most 60 minutes to complete after the stage push has been done. This time is mostly affected by how long it takes you to verify your changes in stage and production. | * Do you have enough time to safely do a push? Most pushes take at most 60 minutes to complete after the stage push has been done. This time is mostly affected by how long it takes you to verify your changes in stage and production. | ||
=== Schema Upgrades === | |||
If you need to do a schema change you '''must''' ensure that either the current production code can run with your schema change applied, or that your new code can run with the old schema. Code and schema changes '''cannot''' be done at the same instant, so you must be able to support one of these scenarios. Generally, additive changes (column or table additions) should do the schema change first, while destructive changes (column or table deletions) should do the schema change second. You can simulate the upgrade with your local Docker containers to verify which is right for you. | If you need to do a schema change you '''must''' ensure that either the current production code can run with your schema change applied, or that your new code can run with the old schema. Code and schema changes '''cannot''' be done at the same instant, so you must be able to support one of these scenarios. Generally, additive changes (column or table additions) should do the schema change first, while destructive changes (column or table deletions) should do the schema change second. You can simulate the upgrade with your local Docker containers to verify which is right for you. | ||
| Line 63: | Line 60: | ||
=== Testing === | === Testing === | ||
The stage environment automatically deploys new code from the master branch of [https://github.com/mozilla/balrog the Balrog repository] (including any necessary schema changes). Before asking for a push, you should do some functional testing there. At the very least, you should do explicit testing of all the new code that would be included in the push. Eg: if you're changing the format of a blob, make sure that you can add a new blob of that type, and that the XML response looks correct. When you've completed testing take note of the "commit" on https://balrog-admin.stage.mozaws.net/__version__ - you'll need it when filing the production push bug. | |||
=== Pushing to production === | |||
Pushing live requires CloudOps. For non-urgent pushes, please make sure you file the bug for them X hours in advance. For urgent pushes, file the bug immediately and [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations escalate if no action is taken quickly]. Either way, you must follow this procedure to push: | |||
# [https://github.com/mozilla/balrog/releases Tag the repository] with a "vX.Y" tag. Eg: "git tag -s vX.Y" | # [https://github.com/mozilla/balrog/releases Tag the repository] with a "vX.Y" tag. Eg: "git tag -s vX.Y" | ||
# Wait for CI jobs to complete. Unit tests must pass and a new Docker Image [https://hub.docker.com/r/mozilla/balrog/tags/ for the webapps] and [https://hub.docker.com/r/mozilla/balrogagent/tags/ the Agent] must be pushed to Dockerhub before you proceed. | # Wait for CI jobs to complete. Unit tests must pass and a new Docker Image [https://hub.docker.com/r/mozilla/balrog/tags/ for the webapps] and [https://hub.docker.com/r/mozilla/balrogagent/tags/ the Agent] must be pushed to Dockerhub before you proceed. | ||
# [https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to= | # [https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=dthorn%40mozilla.com&bug_file_loc=http%3A%2F%2F&bug_ignored=0&bug_severity=normal&bug_status=NEW&cc=dthorn%40mozilla.com&cc=miles%40milescrabill.com&cf_blocking_fennec=---&cf_fx_iteration=---&cf_fx_points=---&cf_status_firefox52=---&cf_status_firefox53=---&cf_status_firefox54=---&cf_status_firefox55=---&cf_status_firefox_esr45=---&cf_status_firefox_esr52=---&cf_tracking_firefox52=---&cf_tracking_firefox53=---&cf_tracking_firefox54=---&cf_tracking_firefox55=---&cf_tracking_firefox_esr45=---&cf_tracking_firefox_esr52=---&cf_tracking_firefox_relnote=---&comment=Balrog%20version%20X.Y%20is%20ready%20to%20be%20pushed%20to%20stage.%20Please%20deploy%20the%20new%20Docker%20images%20%28master-%24COMMIT%29%20for%20admin%2C%20public%2C%20and%20the%20agent.%0D%0A%0D%0AWe%27d%20like%20the%20production%20push%20for%20this%20to%20happen%20sometime%20between%2011am%20and%201pm%20pacific%20on%20%28DATE%20GOES%20HERE%29&component=Operations%3A%20Deployment%20Requests&contenttypemethod=autodetect&contenttypeselection=text%2Fplain&defined_groups=1&flag_type-37=X&flag_type-4=X&flag_type-5=X&flag_type-607=X&flag_type-708=X&flag_type-720=X&flag_type-721=X&flag_type-737=X&flag_type-781=X&flag_type-787=X&flag_type-800=X&flag_type-803=X&flag_type-846=X&flag_type-864=X&flag_type-905=X&flag_type-914=X&flag_type-916=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=--&product=Cloud%20Services&rep_platform=Unspecified&short_desc=please%20deploy%20balrog%20X.Y%20to%20stage%2Fprod&target_milestone=---&version=unspecified File a bug] to have the new version pushed to production. Be sure to substitute the new version number, and commit SHA for the Docker image tags. | ||
#* This bug should generally be filed a about 24h in advance of the desired production push time to give adequate time for the stage deployment and testing. | #* This bug should generally be filed a about 24h in advance of the desired production push time to give adequate time for the stage deployment and testing. | ||
#* Wednesdays are usually the best day to push to production, because they are generally free of release events. Unless you have a specific need to deploy on a different day, you should request the prod push for a Wednesday. | #* Wednesdays are usually the best day to push to production, because they are generally free of release events. Unless you have a specific need to deploy on a different day, you should request the prod push for a Wednesday. | ||
#* You should link any bugs being deployed is the "Blocks" field. | #* You should link any bugs being deployed is the "Blocks" field. | ||
# Once | # Once the push has happened, verify the changes in production. If you need to tweak rules or releases to do so, be careful not to touch any live channels (create new rules or releases if necessary). This final verification is as more about making sure the right thing got deployed than whether or not your code is correct. | ||
# Bump the [https://github.com/mozilla/balrog/blob/master/version.txt in-repo version] to the next available one to ensure the next push gets a new version. | |||
= Meeting Notes = | = Meeting Notes = | ||