ReleaseEngineering/Merge Duty/Steps old

From MozillaWiki
Jump to: navigation, search

Contents

Bump the buildbot-configs gecko version for mozilla-release

  • Bump the mozilla-release version gecko_versions.json
    • Get review, land, and merge to production before the RC build is kicked off in Ship It.

Bump the buildbot-configs gecko version for other branches

  • bump mozilla-central, mozilla-aurora, mozilla-beta, comm-beta, comm-aurora in gecko_versions.json
    • get review, land when you're ready for the next reconfig to push this live.
    • these need a reconfig to go live.

Throttle Aurora nightly updates

Unthrottle Aurora nightly updates

(Deprecated) File a bug for new gaia hg repository

(Deprecated) File a bug for new Gaia l10n repositories

Perform mozilla-beta -> mozilla-release migration

  • Get the list of l10n locales to remove from release management.
  • be available in #r-d as there is often some chatter about merge day in there.

Close mozilla-beta

  • Use https://treestatus.mozilla.org/ to close mozilla-beta.
    • Please use the "Remember previous state" option, so you can easily restore the previous status messages when you are finished (there will be a restore button at the bottom of the page).
    • In order to change tree statuses, your browserID email needs to be added with sheriff permissions (ping edmorley/philor/catlee or ask in #developers).
    • mozilla-beta will remain closed until after the Aurora->Beta uplift the following Monday (or a beta re-spin is necessary).

Sync mozilla-beta to mozilla-release

Mercurial repositories used for final releases should be synced from mozilla-beta.

Tip of default on mozilla-release should be tagged and closed. mozilla-beta should be tagged and pulled. It creates new tip of default on mozilla-release. See Aurora specific sync mechanics for the details.

L10N repositories use noop (dummy) merge if needed to simplify localizers life.

To handle this process use the following scripts:

Beta to Release Merge day helper Script

Gecko migration script: gecko_migration.py

Running gecko_migration.py for mozilla-release

Example invocation: 

# clone mozharness
hg clone http://hg.mozilla.org/build/mozharness
# run the script
python mozharness/scripts/merge_day/gecko_migration.py -c mozharness/configs/merge_day/beta_to_release.py
  • the script will tag mozilla-beta
  • it will continue by pulling mozilla-beta to mozilla-release, merging the heads in a fast-forward manner (using hg debugsetparents), adjusting shipped-locales and changing some configs.
  • Once the script finishes, run an hg diff to see the uncommitted changes that the script generated:
(cd build/mozilla-release; hg diff) | more
# an |hg out| will show the migrated changesets; an |hg out --patch| will show the patches for the migrated changesets.
  • It's best to get a visual review by someone else as well.
  • Commit+push the final bits by
# Make sure you already pushed your release version bump and reconfiged before this part!
python mozharness/scripts/merge_day/gecko_migration.py \
  -c mozharness/configs/selfserve/production.py \
  -c mozharness/configs/merge_day/beta_to_release.py \
  --create-virtualenv --commit-changes --push --trigger-builders

In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o.

Uplift locales

Helper script: beta2release_l10n.sh

  • Note you need permission to be able to run this script successfully. File a server ops bug similar to bug 869654
  • ATM, the script uses ffxbld_rsa key in the current directory (make sure to copy it and chmod 600) to push l10n repos on behalf on ffxbld user. It can be run from cruncher.
  • The script is based on Axel's merge script

Notify r-d that the migration has completed!

By responding to the migration request.

Perform mozilla-aurora -> mozilla-beta and mozilla-central -> mozilla-aurora migrations

Gecko migration script : http://hg.mozilla.org/build/mozharness/file/production/scripts/merge_day/gecko_migration.py

Wait for Go

Wait for an email to r-d with the subject: "Please bump & merge m-c, m-a, m-b", with the following contents:

  • all locales confirmed (or a list of changes needed with bug numbers)
  • no config changes needed (or a list of features to enable/disable with bug numbers)
  • confirm that all IDL checks have been done, and that RelMan will ensure that any remaining IDL issues are addressed prior to go-to-build for m-a.

Ensure that you understand all of the requested changes.

Set up HG Extensions

Edit ~/.hgrc to include 

[extensions]
transplant=
mq=
rebase=
graphlog=

Close mozilla-aurora

  • Use https://treestatus.mozilla.org/ to close the tree.
  • Please use the "Remember previous state" option, so you can easily restore the previous status messages when you are finished.
  • mozilla-beta should already be closed from the previous Monday.

(Deprecated) Pause vcs-sync systems

The critical portions should already have happened in previous merge day steps.

  • TODO: Add notes how to verify that vcs sync is paused.

Running gecko_migration.py for mozilla-beta

You can run this in the exact same directory as you ran the beta_to_release migration, assuming you've already committed and pushed those changes. (If not, the clean-repos action will make sure you're ok for this migration, but you'll lose your beta_to_release changes.)

Example invocation: 

# clone mozharness
hg clone http://hg.mozilla.org/build/mozharness
# run the script
python mozharness/scripts/merge_day/gecko_migration.py -c mozharness/configs/merge_day/aurora_to_beta.py
  • the script will tag mozilla-aurora
  • it will continue by pulling mozilla-aurora to mozilla-beta, merging the heads in a fast-forward manner (using hg debugsetparents), adjusting shipped-locales and changing some configs.
  • run an hg -R build/mozilla-beta diff to view the uncommitted changes.
  • Push the final bits by
python mozharness/scripts/merge_day/gecko_migration.py \
  -c mozharness/configs/selfserve/production.py \
  -c mozharness/configs/merge_day/aurora_to_beta.py \
  --create-virtualenv --commit-changes --push --trigger-builders

In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o.

Notify interested people

  • Ping Standard8, Callek, and Pike in #r-d to warn about push

Re-open mozilla-beta

(Deprecated) Push to mozilla-b2gXX_vX_X (even numbered central gecko version)

You created this branch, and it may not be in sync with aurora. It's best to push aurora there before the central->aurora migration.

  • Since I had already pushed the in-tree changes, this required a merge:
cd mozilla-b2g32_v2_0/
hg pull -u
hg pull http://hg.mozilla.org/releases/mozilla-aurora
hg merge -r default
hg commit -m "merge aurora -> b2g32. a=release CLOSED TREE"
hg push
  • Also re-sync gaia-l10n:
# https://hg.mozilla.org/build/braindump/file/7908431e686d/releases-related/gaia_l10n.sh
# edit gaia_l10n.sh to update BRANCH, TO_REPO_PATH, and probably empty out IGNORE_LOCALES
rm -rf l10n
hg clone https://hg.mozilla.org/build/braindump
sh braindump/releases-related/gaia_l10n.sh 2>&1 | tee gl10n.log
# Watch for errors.
# If you see something like "WARNING releases/gaia-l10n/v2_0/es is newer!",
# check that repo to make sure that any changes newer than gaia-l10n/es are intentional;
# otherwise they need to be backported.

Running gecko_migration.py for mozilla-aurora

You can run this in the exact same directory as you ran the aurora_to_beta migration, assuming you've already committed and pushed those changes. (If not, the clean-repos action will make sure you're ok for this migration, but you'll lose your aurora_to_beta changes.)

Example invocation: 

# clone mozharness
hg clone http://hg.mozilla.org/build/mozharness
# run the script
python mozharness/scripts/merge_day/gecko_migration.py -c mozharness/configs/merge_day/central_to_aurora.py
  • the script will tag mozilla-central
  • it will continue by pulling mozilla-central to mozilla-aurora, merging the heads in a fast-forward manner (using hg debugsetparents), adjusting shipped-locales and changing some configs.
  • Once the diff is shown and reviewed on screen, the script will commit the changes.
  • Push the final bits by
scp cltbld@buildbot-master81.bb.releng.scl3.mozilla.com:/builds/buildbot/build_scheduler/master/BuildSlaves.py oauth.txt
python mozharness/scripts/merge_day/gecko_migration.py \
  -c mozharness/configs/balrog/production.py \
  -c mozharness/configs/selfserve/production.py \
  -c mozharness/configs/merge_day/central_to_aurora.py \
  --create-virtualenv --lock-update-paths --commit-changes \
  --push --trigger-builders

In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o.

Bump bouncer entry versions

  • When we have nightlies from mozilla-central with the new version, update the bouncer locations for firefox-nightly-latest win32 and win64 only (installer) to use the new version.
  • Notify qa-drivers that nighlty builds are in progress and aurora updates are frozen.
  • Continue without waiting for nightlies to complete

Change Template Version Numbers

Now that everything has been bumped edit the following so the Channel Notes and other wiki pages have the correct info and queries now (increment by 1). Also they are very important for daily cronjobs that read the values and nag many many engineers based on triage queries that use them.

The script is at http://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/update_merge_day_wiki.sh

If the ESR version is NOT changing, 

export WIKI_USERNAME=asasaki
export WIKI_PASSWORD=*******
B2G_TRUNK_VERSION=2.2
update_merge_day_wiki.sh -b $B2G_TRUNK_VERSION

If the ESR version IS changing, 

export WIKI_USERNAME=asasaki
export WIKI_PASSWORD=*******
B2G_TRUNK_VERSION=2.2
NEW_ESR_VERSION=31
./update_merge_day_wiki.sh -b $B2G_TRUNK_VERSION -e $NEW_ESR_VERSION

For posterity, This was/should have been B2G_TRUNK_VERSION 2.1 and NEW_ESR_VERSION 31 on July 21 2014, since the b2g version only changes every other merge day.

Double check that the wiki changes happened properly!

Send Merge Completion Email

Send an email to r-d with a subject of "Bump & Merge m-c,m-a,m-b complete". The contents should confirm that:

  • wikimo version templates have been updated

Re-open mozilla-aurora

(Deprecated) Add new B2G gecko branch

This can and should be done ahead of merge day, so you can test the plumbing.

Add to Graphs

mysql> select * from branches where name like '%b2g28%'
    -> ;
+-----+----------------------------+
| id  | name                       |
+-----+----------------------------+
| 221 | Mozilla-B2g28-v1.3         |
| 223 | Mozilla-B2g28-v1.3-Non-PGO |
+-----+----------------------------+
2 rows in set (0.00 sec) 

mysql> select * from machines where name like '%b2g28%';
+------+-------+---------------+-----------+-------------------------------------------+-----------+------------+
| id   | os_id | is_throttling | cpu_speed | name                                      | is_active | date_added |
+------+-------+---------------+-----------+-------------------------------------------+-----------+------------+
| 9415 |     6 |             0 | NULL      | Linux_mozilla-b2g28_v1_3                  |         1 | 1390520203 |
| 9417 |     6 |             0 | NULL      | Linux_mozilla-b2g28_v1_3_leak_test        |         1 | 1390520203 |
| 9419 |     7 |             0 | NULL      | OS_X_10.5.2_mozilla-b2g28_v1_3            |         1 | 1390520204 |
| 9421 |     7 |             0 | NULL      | OS_X_10.5.2_mozilla-b2g28_v1_3_leak_test  |         1 | 1390520204 |
| 9423 |     8 |             0 | NULL      | WINNT_5.2_mozilla-b2g28_v1_3              |         1 | 1390520204 |
| 9425 |     8 |             0 | NULL      | WINNT_5.2_mozilla-b2g28_v1_3_leak_test    |         1 | 1390520205 |
| 9427 |    19 |             0 | NULL      | WINNT_6.1_mozilla-b2g28_v1_3              |         1 | 1390520205 |
| 9429 |    19 |             0 | NULL      | WINNT_6.1_mozilla-b2g28_v1_3_leak_test    |         1 | 1390520205 |
| 9431 |    17 |             0 | NULL      | OS_X_10.6.2_mozilla-b2g28_v1_3            |         1 | 1390520206 |
| 9433 |    17 |             0 | NULL      | OS_X_10.6.2_mozilla-b2g28_v1_3_leak_test  |         1 | 1390520206 |
| 9435 |    18 |             0 | NULL      | Linux_x86-64_mozilla-b2g28_v1_3           |         1 | 1390520207 |
| 9437 |    18 |             0 | NULL      | Linux_x86-64_mozilla-b2g28_v1_3_leak_test |         1 | 1390520208 |
+------+-------+---------------+-----------+-------------------------------------------+-----------+------------+
12 rows in set (0.00 sec)

Connection info is in the releng secrets repo, with https://wiki.mozilla.org/ReleaseEngineering/How_To/Create_new_ESR_branch#Update_graphs.m.o_and_graphs.allizom.org_with and https://wiki.mozilla.org/ReleaseEngineering:Buildduty:Slave_Management#Slavealloc giving more info on which server to connect from.

Add to Treeherder

  • File a bug in [2].

Add to TreeStatus

  • File a bug in [3].

Test new b2g branch

  • Push aurora to the new b2g branch, and watch the builds+tests on tbpl.
    • sheriffs will sometimes do this for you.

(Deprecated) Merge the B2G version from Central to Aurora

  • puppet
    • add mozilla-aurora to the b2g bumper puppet configs here
  • in-tree aurora (gecko, gaia l10n)
  • in-tree central (bump version)
    • Bump the gecko and b2g versions in central in confvars.sh.
  • If needed, manually bump gaia.json to point to the tip of the current gaia mirror like in this automatic commit

Roll out and reconfig and verify that Aurora has the b2g builds+tests.

Add aurora back to legacy gecko.git vcs-sync

(Deprecated) Kill Aurora B2G

  • Kill aurora b2g in buildbot-configs
    • review, land, reconfig
  • Mozharness: move the aurora version to the new b2gXX_vX_X branch, and update the b2g bumper configs.
    • review, land, transplant or merge to production
  • sheriffs will start merging beta -> b2gXX_vX_X

(Deprecated) Create new B2G git branches

These docs need testing!

b2g git branch prep work

  • verify configs/merge_day/b2g_branch_repos.py
    • The " manifest_repo_revision specifies which branch to branch b2g-manifest from. In most cases, this is 'master' (master -> v2.1, f/e). However, when we create a partner branch, we may want to branch from v1.4 -> v1.4d. This can be overridden here in the config file, or by using --manifest-repo-revision .
    • The branch_order will matter here. Right now we have 'master' first, and then we're preferring jellybean and ice cream sandwich branches over kitkat. This is because more of our supported devices use jb or ics than kk, and some repos we're branching have both sets of branches live on b2g-manifest master; we can only branch off of one of them. In the future, we may want to prefer kitkat. We will hit CRITICAL error messages if/when new branch names conflict and aren't represented in the branch_order. These will need to be changed in this config file. Discuss with others if you need to change the branch_order list.
  • pull the sources. This will take a while (could be hours?) the first time, so prepping before merge day is good.
BRANCH=v2.1  # This is the new branch you want to create!
mozharness/scripts/merge_day/b2g_branch_repos.py -c mozharness/configs/merge_day/b2g_branch_repos.py --branch-name $BRANCH --pull --pull-branch-repos
  • make sure your logs/b2g_branch_repos_critical.log is size 0. If not, you probably have an issue with conflicting branches, and you'll need to decide which branch(es) take priority in branch_order. We dump a complete list of branches per repo and the manifests that use them in the info log -- look for the line "Outputting branch_repo_dict:".
  • If you have time, run through the whole thing:
BRANCH=v2.1  # This is the new branch you want to create!
mozharness/scripts/merge_day/b2g_branch_repos.py -c mozharness/configs/merge_day/b2g_branch_repos.py --branch-name $BRANCH --clean-repos
mozharness/scripts/merge_day/b2g_branch_repos.py -c mozharness/configs/merge_day/b2g_branch_repos.py --branch-name $BRANCH

b2g git branching, merge day

  • Run through the whole thing. Hopefully this won't take too long now that you have the repos on disk. (If you don't, you probably want to look at the above section.):
BRANCH=v2.1  # This is the new branch you want to create!
mozharness/scripts/merge_day/b2g_branch_repos.py -c mozharness/configs/merge_day/b2g_branch_repos.py --branch-name $BRANCH --clean-repos
mozharness/scripts/merge_day/b2g_branch_repos.py -c mozharness/configs/merge_day/b2g_branch_repos.py --branch-name $BRANCH
  • Make sure your critical log is size 0 and you exited successfully.
  • |git diff| in build/b2g-manifest to make sure the manifest munging looks good.
  • When you're ready, commit and push.
BRANCH=v2.1
pushd build/b2g-manifest
# verify diff
git diff

# If it's all good, commit and push
git commit -va -m "bug XXXXXXX - create $BRANCH branch"
popd
mozharness/scripts/merge_day/b2g_branch_repos.py -c mozharness/configs/merge_day/b2g_branch_repos.py --branch-name $BRANCH --push

(Deprecated) EOL a B2G branch

  • Check the b2g train schedule to see if a B2G version is EOLing. For example, on 3/17/2014, 1.1 is EOLing. If so, do the below:
    • kill the branch in buildbot-configs. For example, on 3/17/2014, remove all references to mozilla-b2g18.
      • to roll out: review, land, reconfig
    • tools -- I'm not sure if we need/want to remove these from production-branches.json.
    • mark the branch as closed in treestatus and possibly remove it from tbpl altogether.

(Deprecated) Bump gecko.git vcs-sync (odd numbered central gecko version)

  • Move the b2g train from mozilla-central to mozilla-aurora, and bump the mozilla-central version.
    • Legacy vcs-sync: You'll want a similar patch, but with the right version number (2.1 for 9/2/2014).
    • New vcs-sync: We need a patch like this, but for hg.m.o/build/mozharness, and with the right version number (1.4 for 3/17/2014).
      • review, land
      • this will need to be transplanted/merged to production to take effect.

(Deprecated) Pause gecko.git vcs-sync (even numbered central gecko version)

python mozharness/configs/vcs-sync/gecko-git.py
# this should exit without errors
    • Review + land on mozharness default
    • This needs to transplant/merge to mozharness production to go live.

(Deprecated) Restart gecko.git vcs-sync (even numbered central gecko version)

  • Legacy vcs-sync: Add the new b2gXX_vX_X repo to gecko.git, like this and this (but not 1.3t, 1.3).
  • New vcs-sync: Add the new b2gXX_vX_X repo to gecko.git, like the first part of this.
    • review, land
    • This needs to transplant/merge to production to go live.

(Deprecated) Roll out a legacy gecko.git change

  • To roll out, ssh to github-sync3.dmz.scl3.mozilla.com.
    • You'll need vcs2vcs_screen.
    • To show all screen windows, use ^A-" and use the up/down arrows and enter when you're at the right window.
    • To create a new screen window, use ^A-c
bin/vcs2vcs_screen -x
# get a bash screen window
cd /opt/vcs2vcs/bin
hg in  # verify the change(s) coming in
hg pull -u
cd /opt/vcs2vcs/etc
hg in  # verify the change(s) coming in
hg pull -u
# Run ~/idle_screen 
~/idle_screen
# Switch to the 'combo' screen window
^C
keep_updated
^A-d

(Deprecated) Pause Gecko l10n vcs-sync

  • Empty GECKO_BRANCHES = {} (commenting out the contents is probably easier for restarting this later).
  • get review; land on default; transplant to production.
  • (ignoring legacy vcs-sync due to bug 967282)

(Deprecated) Add new Gaia and l10n vcs-sync

  • New vcs-sync: You'll need a patch like this for the new Gaia l10n version. This will begin populating the gecko l10n, and create a new gaia versioned l10n branch that follows gaia l10n master.
    • review, land on mozharness default. It'll need to merge/transplant to production to go live.
  • Legacy Gaia vcs-sync: follow the instructions here.
    • Review, land
    • To roll out legacy vcs-sync, ssh to github-sync3.dmz.scl3.mozilla.com;
sudo su - vcs2vcs
cd /opt/vcs2vcs/etc
hg in  # verify the change(s) coming in
hg pull -u

(Deprecated) Update Gaia l10n vcs-sync

  • New vcs-sync: The previous merge day we landed a patch like this for the new Gaia l10n version. This time, we need to change the hg_url to https://hg.mozilla.org/releases/gaia-l10n/v2_0/%(locale)s (except replace the v2_0 with the appropriate b2g version; this is the version that's getting the new b2gXX_vX_X branch).
    • review, land on mozharness default. It'll need to merge/transplant to production to go live.
  • Legacy Gaia l10n vcs-sync: You'll need a patch like this for the new version, per locale tracked.
    • make sure that releases-gecko/hgrc is included in the patch, see this patch
    • Review, land
    • To roll out both sets of patches, ssh to github-sync3.dmz.scl3.mozilla.com;
sudo su - vcs2vcs
cd /opt/vcs2vcs/etc
hg in  # verify the change(s) coming in
hg pull -u

(Deprecated) Restart Gecko l10n vcs-sync (odd numbered central gecko version)

'v1.5': 'mozilla-central',
    • The previous version (that just moved to a b2gXX_vX_X branch) will be on mozilla-beta. For example, on 4/28/2014,
'v1.4': 'mozilla-beta',
    • There will be no mozilla-aurora entry.
  • get review; land on default; transplant to production.

(Deprecated) Restart Gecko l10n vcs-sync (even numbered central gecko version)

  • Set GECKO_BRANCHES:
    • The version on mozilla-central will be on mozilla-central. For example, on 7/21/2014,
'v2.1': 'mozilla-central',
    • The version on mozilla-aurora will be on mozilla-aurora on odd-versioned m-c merge days; on even, there will be no mozilla-aurora. For example, on 6/09/2014,
'v2.0': 'mozilla-aurora',
      • However, on 7/21/2014, there was no mozilla-aurora line.
    • The version on the latest b2gXX_vX_X branch will be on mozilla-beta. For example, on 7/21/2014,
'v2.0': 'mozilla-beta',
  • get review; land on default; transplant to production.

Tag B2G Gecko+Gaia for MERGEDAY

This step tags supported B2G branches to simplify landing security fixes. 

# Pull all the repos.  This will take a long time; do this ahead of time if you can!
mozharness/scripts/merge_day/b2g_tag.py -c mozharness/configs/merge_day/b2g_tag.py --pull

# Do everything.  This will also pull, so if you missed the previous --pull step, skip it and just run this.
mozharness/scripts/merge_day/b2g_tag.py -c mozharness/configs/merge_day/b2g_tag.py

# To clean local repos, you could
# mozharness/scripts/merge_day/b2g_tag.py -c mozharness/configs/merge_day/b2g_tag.py --clean-repos
# To [re]run one, f/e only pull/tag/push mozilla-b2g18:
# mozharness/scripts/merge_day/b2g_tag.py -c mozharness/configs/merge_day/b2g_tag.py --gecko-repo mozilla-b2g18

(Deprecated) Update mozharness for b2gXX

The b2g version has moved from Aurora to b2gXX.

  • Write a patch for b2g_build like this
  • Write a patch for b2g_bumper like this

Review, land, merge. (No puppet patch this time because it was already added to puppet last merge day.)

Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/Merge_Duty/Steps_old&oldid=1057802"