ReleaseEngineering/Merge Duty/Steps

From MozillaWiki
Jump to: navigation, search

Up to date documentation currently at [1]. This page: candidate-for-deletion

Bump the buildbot-configs gecko version for mozilla-release

  • Bump the mozilla-release version gecko_versions.json
    • Get review, land, and merge to production before the RC build is kicked off in Ship It.

Bump the buildbot-configs gecko version for other branches

  • bump mozilla-central, mozilla-aurora, mozilla-beta, comm-beta, comm-aurora in gecko_versions.json
    • get review, land when you're ready for the next reconfig to push this live.
    • these need a reconfig to go live.

Throttle Aurora nightly updates

Unthrottle Aurora nightly updates

Perform mozilla-beta -> mozilla-release migration

  • Get the list of l10n locales to remove (if required) from release management.
  • be available in #r-d as there is often some chatter about merge day in there.

Set up HG Extensions

Edit ~/.hgrc to include 

[extensions]
transplant=
mq=
rebase=
graphlog=

Close mozilla-beta

  • Use https://mozilla-releng.net/treestatus to close mozilla-beta.
    • Please use the "Remember previous state" option, so you can easily restore the previous status messages when you are finished (there will be a restore button at the bottom of the page).
    • In order to change tree statuses, your browserID email needs to be added with sheriff permissions (ping edmorley/philor/catlee or ask in #developers).
    • mozilla-beta will remain closed until after the Aurora->Beta uplift the following Monday (or a beta re-spin is necessary).

Sync mozilla-beta to mozilla-release

Overview of the process:

  • Mercurial repositories used for final releases will be synced from mozilla-beta.
  • Tip of default on mozilla-release will be tagged and closed.
  • mozilla-beta will be tagged and pulled. It creates new tip of default on mozilla-release.
  • L10N repositories use noop (dummy) merge if needed to simplify localizers life.

Beta to Release Merge day helper Script

To handle this process use the following scripts: Gecko migration script: gecko_migration.py

Running gecko_migration.py for mozilla-release

Example invocation: 

# checkout mozharness from the gecko tree using archiver
wget https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py
python archiver_client.py mozharness --destination mozharness-beta --repo releases/mozilla-beta --rev default --debug
# run the script
python mozharness-beta/scripts/merge_day/gecko_migration.py -c merge_day/beta_to_release.py
  • the script will tag mozilla-beta
  • it will continue by pulling mozilla-beta to mozilla-release, merging the heads in a fast-forward manner (using hg debugsetparents), adjusting shipped-locales and changing some configs.
  • Once the script finishes, run an hg diff to see the uncommitted changes that the script generated:
hg -R build/mozilla-release diff
  • It's best to get a visual review by someone else as well.
  • Commit+push the final bits by
# Make sure you already pushed your release version bump and reconfiged before this part!
python mozharness-beta/scripts/merge_day/gecko_migration.py \
  -c selfserve/production.py -c merge_day/beta_to_release.py \
  --create-virtualenv --commit-changes --push

In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o.

Bump ESR version

This will bump the second digit in ESR's version. Repeat for all ESR that require the bump. 

ESR_VERSION=45
# checkout mozharness from the gecko tree using archiver
wget https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py
python archiver_client.py mozharness --destination mozharness-esr$ESR_VERSION --repo releases/mozilla-esr$ESR_VERSION --rev default --debug
# run the script
python mozharness-esr$ESR_VERSION/scripts/merge_day/gecko_migration.py -c mozharness-esr$ESR_VERSION/configs/merge_day/bump_esr.py
hg -R build/mozilla-esr$ESR_VERSION diff
python mozharness-esr$ESR_VERSION/scripts/merge_day/gecko_migration.py -c mozharness-esr$ESR_VERSION/configs/merge_day/bump_esr.py \
  --commit-changes --push

Uplift locales

Helper script: beta2release_l10n.sh

  • The script uses ffxbld_rsa key in the current directory (make sure to copy it and chmod 600) to push l10n repos on behalf on ffxbld user. It can be run from cruncher.
  • The script is based on Axel's merge script

Example invocation: 

mkdir l10n && cd l10n
wget https://hg.mozilla.org/build/braindump/raw-file/default/releases-related/beta2release_l10n.sh
chmod 755 beta2release_l10n.sh
# copy ffxbld_rsa
chmod 600 ffxbld_rsa
./beta2release_l10n.sh
  • It takes 30-45 minutes
  • In case of failure it can be rerun and it will skip already cloned locales. If a locale failed while pushing, you can delete it to make the script process it again.

Notify r-d that the migration has completed!

By responding to the migration request.

Perform mozilla-aurora -> mozilla-beta and mozilla-central -> mozilla-aurora migrations

Gecko migration script : https://hg.mozilla.org/build/mozharness/file/production/scripts/merge_day/gecko_migration.py

Wait for Go

Wait for an email to r-d with the subject: "Please bump & merge m-c, m-a, m-b", with the following contents:

  • all locales confirmed (or a list of changes needed with bug numbers)
  • no config changes needed (or a list of features to enable/disable with bug numbers)
  • confirm that all IDL checks have been done, and that RelMan will ensure that any remaining IDL issues are addressed prior to go-to-build for m-a.

Ensure that you understand all of the requested changes.

Set up HG Extensions

Edit ~/.hgrc to include 

[extensions]
transplant=
mq=
rebase=
graphlog=

Close mozilla-aurora

  • Use https://mozilla-releng.net/treestatus to close the tree.
  • Please use the "Remember previous state" option, so you can easily restore the previous status messages when you are finished.
  • mozilla-beta should already be closed from the previous Monday.

Running gecko_migration.py for mozilla-beta

You can run this in the exact same directory as you ran the beta_to_release migration.

Example invocation: 

# checkout mozharness from the gecko tree using archiver
wget https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py
python archiver_client.py mozharness --destination mozharness-aurora --repo releases/mozilla-aurora --rev default --debug
# run the script
python mozharness-aurora/scripts/merge_day/gecko_migration.py -c merge_day/aurora_to_beta.py
  • the script will tag mozilla-aurora
  • it will continue by pulling mozilla-aurora to mozilla-beta, merging the heads in a fast-forward manner (using hg debugsetparents), adjusting shipped-locales and changing some configs.
  • run an hg -R build/mozilla-beta diff to view the uncommitted changes.
  • Push the final bits by
python mozharness-aurora/scripts/merge_day/gecko_migration.py \
  -c selfserve/production.py -c merge_day/aurora_to_beta.py \
  --create-virtualenv --commit-changes --push --trigger-builders

In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o. If --trigger-builders fails, you can rerun the same script with only that action specified (no --commit-changes --push).

Leave mozilla-beta closed, hand it off to #sheriffs

  • Best case, until we can run gecko-decision on merges we need another no-op push after the merge to see whether TaskCluster jobs are working; worst case it will take a day of fixing up to get the tree green.
  • If there isn't an obvious sheriff on duty hanging around in #releng, shouting in #sheriffs should work.

Running gecko_migration.py for mozilla-aurora

You can run this in the exact same directory as you ran the aurora_to_beta migration.

Example invocation: 

# checkout mozharness from the gecko tree using archiver
wget https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py
python archiver_client.py mozharness --destination mozharness-central --repo mozilla-central --rev default --debug
# run the script
python mozharness-central/scripts/merge_day/gecko_migration.py -c merge_day/central_to_aurora.py
  • the script will tag mozilla-central
  • it will continue by pulling mozilla-central to mozilla-aurora, merging the heads in a fast-forward manner (using hg debugsetparents), adjusting shipped-locales and changing some configs.
  • Once the diff is shown and reviewed on screen, the script will commit the changes.
  • Push the final bits by
scp cltbld@buildbot-master81.bb.releng.scl3.mozilla.com:/builds/buildbot/build_scheduler/master/BuildSlaves.py oauth.txt
python mozharness-central/scripts/merge_day/gecko_migration.py \
  -c balrog/production.py -c selfserve/production.py \
  -c merge_day/central_to_aurora.py \
  --create-virtualenv --lock-update-paths --commit-changes \
  --push --trigger-builders

In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o. If --trigger-builders fails, you can rerun the same script with only that action specified (no --commit-changes --push).

  • Freeze aurora updates using page until --lock-update-paths is fixed (bug 1163998)

Change Template Version Numbers

Now that everything has been bumped edit the following so the Channel Notes and other wiki pages have the correct info and queries now (increment by 1). Also they are very important for daily cronjobs that read the values and nag many many engineers based on triage queries that use them.

The script is at https://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/update_merge_day_wiki.sh

If the ESR version is NOT changing, 

export WIKI_USERNAME=asasaki
export WIKI_PASSWORD=*******
update_merge_day_wiki.sh

If the ESR version IS changing, 

export WIKI_USERNAME=asasaki
export WIKI_PASSWORD=*******
NEW_ESR_VERSION=52
./update_merge_day_wiki.sh -e $NEW_ESR_VERSION

Double check that the wiki changes happened properly!

Send Merge Completion Email

Send an email to r-d with a subject of "Bump & Merge m-c,m-a,m-b complete". Make sure to CC thunderbird-drivers@mozilla.org.

The contents should confirm that:

  • wikimo version templates have been updated
  • Notify qa-drivers that nighlty builds are in progress and aurora updates are frozen.

Leave mozilla-aurora closed, hand it off to #sheriffs

  • Best case, until we can run gecko-decision on merges we need another no-op push after the merge to see whether TaskCluster jobs are working; worst case it will take a day of fixing up to get the tree green.
  • If there isn't an obvious sheriff on duty hanging around in #releng, shouting in #sheriffs should work.

Notify interested people (Optional)

  • Ping Callek and Pike in #r-d to warn about push

Bump bouncer entry versions

NB: it is expected that the two stub products have a win and win64 location which both point to the same location. We don't have a win64 stub installer, instead the mislabeled 32bit stub selects the correct full installer at runtime.

Trim bouncer's Check Now list

  • Once per release cycle we should stop checking old releases to see if they're present
  • Visit the list of check now enabled products
  • you should leave all current and upcoming releases enabled, as well as the updates for watershed releases. Typically this is about 100 products
  • select all the old releases not covered above, and use the 'Remove Check Now on selected products' option in the Action dropdown
Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/Merge_Duty/Steps&oldid=1212294"