canmove, Confirmed users
6,441
edits
Balrog: Difference between revisions
Jump to navigation
Jump to search
no edit summary
No edit summary |
|||
| Line 48: | Line 48: | ||
== Deploying Changes == | == Deploying Changes == | ||
Balrog's [https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Update+Service stage and production infrastructure] is managed by [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations the Cloud Operations team]. This section describes how to go from a reviewed patch to deploying it in production. You should generally begin this process '''at least | Balrog's [https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Update+Service stage and production infrastructure] is managed by [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations the Cloud Operations team]. This section describes how to go from a reviewed patch to deploying it in production. You should generally begin this process '''at least 24 hours''' before you want the new code live in production. This gives the new code a chance to bake in stage. | ||
=== Is now a good time? === | === Is now a good time? === | ||
| Line 66: | Line 66: | ||
=== Testing === | === Testing === | ||
The | The dev environment automatically deploys new code from the master branch of [https://github.com/mozilla/balrog the Balrog repository] (including any necessary schema changes). Before beginning the deployment procedure, you should do some functional testing there. At the very least, you should do explicit testing of all the new code that would be included in the push. Eg: if you're changing the format of a blob, make sure that you can add a new blob of that type, and that the XML response looks correct. | ||
'''If you have schema changes''' you must also ensure that the existing deployed code will work with the new schema. To do this, CloudOps will downgrade the | '''If you have schema changes''' you must also ensure that the existing deployed code will work with the new schema. To do this, CloudOps will downgrade the dev apps. You should do some routine testing (make some changes to some objects, try some update requests) to ensure that everything works. If you have any issues you CANNOT proceed to production. | ||
=== Pushing to production === | === Pushing to production === | ||
Pushing live requires CloudOps. For non-urgent pushes, | Pushing live requires CloudOps. For non-urgent pushes, you should begin this procedure '''at least 24 hours in advance'''. For urgent pushes, file the bug immediately and [https://mana.mozilla.org/wiki/display/SVCOPS/Contacting+Cloud+Operations escalate if no action is taken quickly]. Either way, you must follow this procedure to push: | ||
# | # Tag the repository with a "vX.Y" tag. Eg: "git tag -s vX.Y && git push --tags" | ||
# Diff against the previous release tag. Eg: "git diff v2.24 v2.25" | # Diff against the previous release tag. Eg: "git diff v2.24 v2.25" | ||
#* Look for anything unexpected, or any '''schema changes'''. If schema changes are present, see the above section for instructions on handling them. | #* Look for anything unexpected, or any '''schema changes'''. If schema changes are present, see the above section for instructions on handling them. | ||
# | # [https://github.com/mozilla/balrog/releases Create a new Release on Github]. This create new Docker images tagged with your version, and deploy them to stage. It may take upwards of 30 minutes for the deployment to happen. | ||
# [https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=dthorn%40mozilla.com&bug_file_loc=http%3A%2F%2F&bug_ignored=0&bug_severity=normal&bug_status=NEW&cc=dthorn%40mozilla.com&cc=miles%40milescrabill.com&cf_blocking_fennec=---&cf_fx_iteration=---&cf_fx_points=---&cf_status_firefox56=---&cf_status_firefox57=---&cf_status_firefox58=---&cf_status_firefox_esr52=---&cf_tracking_firefox56=---&cf_tracking_firefox57=---&cf_tracking_firefox58=---&cf_tracking_firefox_esr52=---&cf_tracking_firefox_relnote=---&comment=Balrog%20version%20X.Y%20is%20ready%20to%20be%20pushed%20to%20prod.%20Please%20deploy%20the%20new%20Docker%20images%20% | # Let things bake on stage for at least 24 hours. You can do additional targeted testing here if you wish, or simply wait for nightlies/releases to prod things along. | ||
# [https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=dthorn%40mozilla.com&bug_file_loc=http%3A%2F%2F&bug_ignored=0&bug_severity=normal&bug_status=NEW&cc=dthorn%40mozilla.com&cc=miles%40milescrabill.com&cf_blocking_fennec=---&cf_fx_iteration=---&cf_fx_points=---&cf_status_firefox56=---&cf_status_firefox57=---&cf_status_firefox58=---&cf_status_firefox_esr52=---&cf_tracking_firefox56=---&cf_tracking_firefox57=---&cf_tracking_firefox58=---&cf_tracking_firefox_esr52=---&cf_tracking_firefox_relnote=---&comment=Balrog%20version%20X.Y%20is%20ready%20to%20be%20pushed%20to%20prod.%20Please%20deploy%20the%20new%20Docker%20images%20%28vX.Y%20for%20admin%2C%20public%2C%20and%20the%20agent.%0D%0A%0D%0AWe%27d%20like%20the%20production%20push%20for%20this%20to%20happen%20sometime%20between%2011am%20and%201pm%20pacific%20on%20%28DATE%20GOES%20HERE%29.%0D%0A%0D%0AONE%20OF%3A%0D%0A%28NO%20SCHEMA%20CHANGE%20OR%20ADDITIVE%20SCHEMA%20CHANGE%29%20If%20anything%20goes%20wrong%20with%20the%20new%20version%20in%20production%2C%20we%20can%20safely%20rollback%20to%20the%20previous%20versions%20of%20admin%2C%20agent%2C%20public%2C%20and%20public%20in%20that%20order.%0D%0A%0D%0A%28DESTRUCTIVE%20SCHEMA%20CHANGE%29%20Because%20this%20push%20contains%20a%20destructive%20schema%20change%2C%20rollback%20to%20the%20previous%20version%20must%20be%20done%20as%20follows%3A%0D%0A%2A%20Revert%20the%20schema%20change%20%28run%20%22python%20%2Fapp%2Fscripts%2Fmanage-db.py%20-d%20%24DBURI%20--version%20X%22%20in%20the%20admin%20container%29%0D%0A%2A%20Rollback%20admin%2C%20agent%2C%20and%20public%20in%20that%20order.%0D%0A%0D%0A%28UNABLE%20TO%20COME%20UP%20WITH%20SAFE%20ROLLBACK%20PROCEDURE%20-%20THIS%20SHOULD%20NEVER%20HAPPEN%29%20This%20push%20contains%20changes%20that%20cannot%20be%20safely%20rolled%20back.%20If%20anything%20goes%20wrong%20with%20the%20new%20version%20in%20production%2C%20please%20escalate%20to%20me.&component=Operations%3A%20Deployment%20Requests&contenttypemethod=autodetect&contenttypeselection=text%2Fplain&defined_groups=1&flag_type-37=X&flag_type-4=X&flag_type-5=X&flag_type-607=X&flag_type-708=X&flag_type-721=X&flag_type-737=X&flag_type-787=X&flag_type-800=X&flag_type-803=X&flag_type-846=X&flag_type-864=X&flag_type-914=X&flag_type-916=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=--&product=Cloud%20Services&rep_platform=Unspecified&short_desc=please%20deploy%20balrog%20X.Y%20to%20prod&target_milestone=---&version=unspecified File a bug] to have the new version pushed to production. | |||
#* This bug should generally be filed at least a few hours in advance of the desired production push time to ensure that CloudOps has time to react to it. | #* This bug should generally be filed at least a few hours in advance of the desired production push time to ensure that CloudOps has time to react to it. | ||
#* Wednesdays are usually the best day to push to production, because they are generally free of release events. Unless you have a specific need to deploy on a different day, you should request the prod push for a Wednesday. | #* Wednesdays are usually the best day to push to production, because they are generally free of release events. Unless you have a specific need to deploy on a different day, you should request the prod push for a Wednesday. | ||
#* You should link any bugs being deployed is the "Blocks" field. | #* You should link any bugs being deployed is the "Blocks" field. | ||
#* Make sure you choose the correct rollback option from the bug template. | #* Make sure you substitute the version number and choose the correct rollback option from the bug template. | ||
# Once the push has happened, verify that the code was pushed to production by checking the __version__ endpoints on [https://aus4-admin.mozilla.org/__version__ the Admin] and [https://aus5.mozilla.org/__version__ Public] apps. | # Once the push has happened, verify that the code was pushed to production by checking the __version__ endpoints on [https://aus4-admin.mozilla.org/__version__ the Admin] and [https://aus5.mozilla.org/__version__ Public] apps. | ||
# Bump the [https://github.com/mozilla/balrog/blob/master/version.txt in-repo version] to the next available one to ensure the next push gets a new version. | # Bump the [https://github.com/mozilla/balrog/blob/master/version.txt in-repo version] to the next available one to ensure the next push gets a new version. | ||