ReleaseEngineering/Merge Duty/Steps: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 168: Line 168:
== Change Template Version Numbers ==
== Change Template Version Numbers ==
Now that everything has been bumped edit the following so the Channel Notes and other wiki pages have the correct info and queries now (increment by 1). Also they are very important for daily cronjobs that read the values and nag many many engineers based on triage queries that use them.
Now that everything has been bumped edit the following so the Channel Notes and other wiki pages have the correct info and queries now (increment by 1). Also they are very important for daily cronjobs that read the values and nag many many engineers based on triage queries that use them.
The script is at http://hg.mozilla.org/build/tools/file/a52cd8b0f1e7/buildfarm/maintenance/update_merge_day_wiki.sh
If the ESR version is NOT changing,
WIKI_USERNAME=asasaki
WIKI_PASSWORD=*******
B2G_TRUNK_VERSION=2.2
update_merge_day_wiki.sh -b $B2G_TRUNK_VERSION
If the ESR version IS changing,
WIKI_USERNAME=asasaki
WIKI_PASSWORD=*******
B2G_TRUNK_VERSION=2.2
NEW_ESR_VERSION=31
update_merge_day_wiki.sh -b $B2G_TRUNK_VERSION -e $NEW_ESR_VERSION
=== Double check that the wiki changes happened properly! ===


* [https://wiki.mozilla.org/index.php?title=Template:NEXT_VERSION&action=edit NEXT_VERSION]
* [https://wiki.mozilla.org/index.php?title=Template:NEXT_VERSION&action=edit NEXT_VERSION]
Line 181: Line 200:
** [https://wiki.mozilla.org/index.php?title=Template:FIREFOX_MERGE_DATE&action=edit FIREFOX_MERGE_DATE] (the next ship date)
** [https://wiki.mozilla.org/index.php?title=Template:FIREFOX_MERGE_DATE&action=edit FIREFOX_MERGE_DATE] (the next ship date)
** [https://wiki.mozilla.org/index.php?title=Template:CURRENT_CYCLE&action=edit CURRENT_CYCLE] based upon [https://wiki.mozilla.org/RapidRelease/Calendar the calendar]
** [https://wiki.mozilla.org/index.php?title=Template:CURRENT_CYCLE&action=edit CURRENT_CYCLE] based upon [https://wiki.mozilla.org/RapidRelease/Calendar the calendar]
== Send Merge Completion Email ==
== Send Merge Completion Email ==
Send an email to r-d with a subject of "Bump & Merge m-c,m-a,m-b complete". The contents should confirm that:
Send an email to r-d with a subject of "Bump & Merge m-c,m-a,m-b complete". The contents should confirm that:

Revision as of 15:33, 14 July 2014

Bump the buildbot-configs gecko version for mozilla-release

Bump the buildbot-configs gecko version for other branches

Throttle Aurora nightly updates

Unthrottle Aurora nightly updates

Create new hg bundles

File a bug for new gaia hg repository

File a bug for new Gaia l10n repositories

Perform mozilla-beta -> mozilla-release migration

  • Get the list of l10n locales to remove from release management.

Close mozilla-beta

  • Use https://treestatus.mozilla.org/ to close mozilla-beta.
    • Please use the "Remember previous state" option, so you can easily restore the previous status messages when you are finished (there will be a restore button at the bottom of the page).
    • In order to change tree statuses, your browserID email needs to be added with sheriff permissions (ping edmorley/philor/catlee or ask in #developers).
    • mozilla-beta will remain closed until after the Aurora->Beta uplift the following Monday (or a beta re-spin is necessary).

Sync mozilla-beta to mozilla-release

Mercurial repositories used for final releases should be synced from mozilla-beta.

Tip of default on mozilla-release should be tagged and closed. mozilla-beta should be tagged and pulled. It creates new tip of default on mozilla-release. See Aurora specific sync mechanics for the details.

L10N repositories uses noop (dummy) merge if needed to simplify localizers life.

To handle this process use the following scripts.

Beta to Release Merge day helper Script

Gecko migration script: gecko_migration.py

Clobber mozilla-release builders

We may not need to do this now that gecko_migration.py touches the CLOBBER file! Was: Go to https://secure.pub.build.mozilla.org/clobberer/?branch=mozilla-release and clobber the mozilla-release builders

Running gecko_migration.py for mozilla-release

Example invocation: 

# clone mozharness
hg clone http://hg.mozilla.org/build/mozharness
# run the script
python mozharness/scripts/merge_day/gecko_migration.py -c mozharness/configs/merge_day/beta_to_release.py --remove-locale sw
  • the script will tag mozilla-beta
  • it will continue by pulling mozilla-beta to mozilla-release, merging the heads in a fast-forward manner (using hg debugsetparents), adjusting shipped-locales and changing some configs.
  • Once the diff is shown and reviewed on screen, the script will commit the changes.
  • Push the final bits by
python mozharness/scripts/merge_day/gecko_migration.py -c mozharness/configs/merge_day/beta_to_release.py --commit-changes --push

In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o.

Uplift locales

Helper script: beta2release_l10n.sh

  • Note you need permission to be able to run this script successfully. File a server ops bug similar to bug 869654
  • ATM, the script uses ~/.ssh/ffxbld_dsa key to push l10n repos on befalf on ffxbld user. It can be run from cruncher.
  • The script is based on Axel's merge script

Perform mozilla-aurora -> mozilla-beta and mozilla-central -> mozilla-aurora migrations

Gecko migration script : http://hg.mozilla.org/build/mozharness/file/28a22339cb48/scripts/merge_day/gecko_migration.py

Wait for Go

Wait for an email to r-d with the subject: "Please bump & merge m-c, m-a, m-b", with the following contents:

  • all locales confirmed (or a list of changes needed with bug numbers)
  • no config changes needed (or a list of features to enable/disable with bug numbers)
  • confirm that all IDL checks have been done, and that RelMan will ensure that any remaining IDL issues are addressed prior to go-to-build for m-a.

Ensure that you understand all of the requested changes.

Set up HG Extensions

Edit ~/.hgrc to include 

[extensions]
transplant=
mq=
rebase=
graphlog=

Close the Trees

DO NOT CLOSE mozilla-central FOR MERGING

  • Use https://treestatus.mozilla.org/ to close the trees.
  • Please use the "Remember previous state" option, so you can easily restore the previous status messages when you are finished (there will be a restore button at the bottom of the page).
  • Close mozilla-aurora; mozilla-beta should already be closed from the previous Monday.

Pause vcs-sync systems

The critical portions should already have happened in previous merge day steps.

Running gecko_migration.py for mozilla-beta

You can run this in the exact same directory as you ran the beta_to_release migration, assuming you've already committed and pushed those changes. (If not, the clean-repos action will make sure you're ok for this migration, but you'll lose your beta_to_release changes.)

Example invocation: 

# clone mozharness
hg clone http://hg.mozilla.org/build/mozharness
# run the script
python mozharness/scripts/merge_day/gecko_migration.py -c mozharness/configs/merge_day/aurora_to_beta.py
  • the script will tag mozilla-aurora
  • it will continue by pulling mozilla-aurora to mozilla-beta, merging the heads in a fast-forward manner (using hg debugsetparents), adjusting shipped-locales and changing some configs.
  • Once the diff is shown and reviewed on screen, the script will commit the changes.
  • Push the final bits by
python mozharness/scripts/merge_day/gecko_migration.py -c mozharness/configs/merge_day/aurora_to_beta.py --commit-changes --push

In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o.

Notify interested people

  • Ping Standard8, Callek, and Pike in #r-d to warn about push

Running gecko_migration.py for mozilla-aurora

You can run this in the exact same directory as you ran the aurora_to_beta migration, assuming you've already committed and pushed those changes. (If not, the clean-repos action will make sure you're ok for this migration, but you'll lose your aurora_to_beta changes.)

Example invocation: 

# clone mozharness
hg clone http://hg.mozilla.org/build/mozharness
# run the script
python mozharness/scripts/merge_day/gecko_migration.py -c mozharness/configs/merge_day/central_to_aurora.py
  • the script will tag mozilla-central
  • it will continue by pulling mozilla-central to mozilla-aurora, merging the heads in a fast-forward manner (using hg debugsetparents), adjusting shipped-locales and changing some configs.
  • Once the diff is shown and reviewed on screen, the script will commit the changes.
  • Push the final bits by
python mozharness/scripts/merge_day/gecko_migration.py -c mozharness/configs/merge_day/central_to_aurora.py --commit-changes --push

In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will be sufficient). It should be faster the second time, since you won't be pulling in as many changesets from hg.m.o.


Kick off m-c nightlies

  • When we have nightlies from mozilla-central with the new version, update the bouncer location for firefox-nightly-latest (installer) to use the new version
  • Continue without waiting for mozilla-central nightlies to complete

Change Template Version Numbers

Now that everything has been bumped edit the following so the Channel Notes and other wiki pages have the correct info and queries now (increment by 1). Also they are very important for daily cronjobs that read the values and nag many many engineers based on triage queries that use them.

The script is at http://hg.mozilla.org/build/tools/file/a52cd8b0f1e7/buildfarm/maintenance/update_merge_day_wiki.sh

If the ESR version is NOT changing, 

WIKI_USERNAME=asasaki
WIKI_PASSWORD=*******
B2G_TRUNK_VERSION=2.2
update_merge_day_wiki.sh -b $B2G_TRUNK_VERSION

If the ESR version IS changing, 

WIKI_USERNAME=asasaki
WIKI_PASSWORD=*******
B2G_TRUNK_VERSION=2.2
NEW_ESR_VERSION=31
update_merge_day_wiki.sh -b $B2G_TRUNK_VERSION -e $NEW_ESR_VERSION

Double check that the wiki changes happened properly!

Send Merge Completion Email

Send an email to r-d with a subject of "Bump & Merge m-c,m-a,m-b complete". The contents should confirm that:

  • wikimo version templates have been updated

Add new B2G gecko branch

This can and should be done ahead of merge day, so you can test the plumbing.

Add to Graphs

mysql> select * from branches where name like '%b2g28%'
    -> ;
+-----+----------------------------+
| id  | name                       |
+-----+----------------------------+
| 221 | Mozilla-B2g28-v1.3         |
| 223 | Mozilla-B2g28-v1.3-Non-PGO |
+-----+----------------------------+
2 rows in set (0.00 sec) 

mysql> select * from machines where name like '%b2g28%';
+------+-------+---------------+-----------+-------------------------------------------+-----------+------------+
| id   | os_id | is_throttling | cpu_speed | name                                      | is_active | date_added |
+------+-------+---------------+-----------+-------------------------------------------+-----------+------------+
| 9415 |     6 |             0 | NULL      | Linux_mozilla-b2g28_v1_3                  |         1 | 1390520203 |
| 9417 |     6 |             0 | NULL      | Linux_mozilla-b2g28_v1_3_leak_test        |         1 | 1390520203 |
| 9419 |     7 |             0 | NULL      | OS_X_10.5.2_mozilla-b2g28_v1_3            |         1 | 1390520204 |
| 9421 |     7 |             0 | NULL      | OS_X_10.5.2_mozilla-b2g28_v1_3_leak_test  |         1 | 1390520204 |
| 9423 |     8 |             0 | NULL      | WINNT_5.2_mozilla-b2g28_v1_3              |         1 | 1390520204 |
| 9425 |     8 |             0 | NULL      | WINNT_5.2_mozilla-b2g28_v1_3_leak_test    |         1 | 1390520205 |
| 9427 |    19 |             0 | NULL      | WINNT_6.1_mozilla-b2g28_v1_3              |         1 | 1390520205 |
| 9429 |    19 |             0 | NULL      | WINNT_6.1_mozilla-b2g28_v1_3_leak_test    |         1 | 1390520205 |
| 9431 |    17 |             0 | NULL      | OS_X_10.6.2_mozilla-b2g28_v1_3            |         1 | 1390520206 |
| 9433 |    17 |             0 | NULL      | OS_X_10.6.2_mozilla-b2g28_v1_3_leak_test  |         1 | 1390520206 |
| 9435 |    18 |             0 | NULL      | Linux_x86-64_mozilla-b2g28_v1_3           |         1 | 1390520207 |
| 9437 |    18 |             0 | NULL      | Linux_x86-64_mozilla-b2g28_v1_3_leak_test |         1 | 1390520208 |
+------+-------+---------------+-----------+-------------------------------------------+-----------+------------+
12 rows in set (0.00 sec)

Connection info is in the releng secrets repo, with https://wiki.mozilla.org/ReleaseEngineering/How_To/Create_new_ESR_branch#Update_graphs.m.o_and_graphs.allizom.org_with and https://wiki.mozilla.org/ReleaseEngineering:Buildduty:Slave_Management#Slavealloc giving more info on which server to connect from.

Add to TBPL

  • Write an hg.m.o/webtools/tbpl patch like this.
    • Get review from a sheriff (edmorley, ryanvm), land. Ask them to update Chief at their convenience (unless you're adding this late, in which case it'll be a rush job).
    • Ping edmorley, ryanvm, or hwine to add to treestatus.

Test new b2g branch

  • Push aurora to the new b2g branch, and watch the builds+tests on tbpl.
    • sheriffs will sometimes do this for you.

Merge the B2G version from Central to Aurora

  • puppet
    • add mozilla-aurora to the b2g bumper puppet configs here
  • in-tree aurora (gecko, gaia l10n)
  • in-tree central (bump version)
    • Bump the gecko and b2g versions in central in confvars.sh.
    • Until bug 1022907 is resolved, you'll need to land a patch like this to set the b2g update channels to aurora.

Roll out and reconfig and verify that Aurora has the b2g builds+tests.

Add aurora back to legacy gecko.git vcs-sync

Add the new Central version to update.boot2gecko.org

This should be obsolete with b2g balrog !!!

The docs are here.

Kill Aurora B2G

  • Kill aurora b2g in buildbot-configs
    • review, land, reconfig
  • Mozharness: move the aurora version to the new b2gXX_vX_X branch, and update the b2g bumper configs.
    • review, land, transplant or merge to production
  • sheriffs will start merging beta -> b2gXX_vX_X

Create new B2G git branches

These docs need testing!

b2g git branch prep work

  • verify configs/merge_day/b2g_branch_repos.py
    • The <a href="http://hg.mozilla.org/build/mozharness/file/b189cc9b9161/configs/merge_day/b2g_branch_repos.py#l8">>manifest_repo_revision</a> specifies which branch to branch b2g-manifest from. In most cases, this is 'master' (master -> v2.1, f/e). However, when we create a partner branch, we may want to branch from v1.4 -> v1.4d. This can be overridden here in the config file, or by using --manifest-repo-revision .
    • The <a href="http://hg.mozilla.org/build/mozharness/file/b189cc9b9161/configs/merge_day/b2g_branch_repos.py#l27">branch_order</a> will matter here. Right now we have 'master' first, and then we're preferring jellybean and ice cream sandwich branches over kitkat. This is because more of our supported devices use jb or ics than kk, and some repos we're branching have both sets of branches live on b2g-manifest master; we can only branch off of one of them. In the future, we may want to prefer kitkat. We will hit CRITICAL error messages if/when new branch names conflict and aren't represented in the branch_order. These will need to be changed in this config file. Discuss with others if you need to change the branch_order list.
  • pull the sources. This will take a while (could be hours?) the first time, so prepping before merge day is good.
BRANCH=v2.1  # This is the new branch you want to create!
mozharness/scripts/merge_day/b2g_branch_repos.py -c mozharness/configs/merge_day/b2g_branch_repos.py --branch-name $BRANCH --pull --pull-branch-repos
  • make sure your logs/b2g_branch_repos_critical.log is size 0. If not, you probably have an issue with conflicting branches, and you'll need to decide which branch(es) take priority in branch_order. We dump a complete list of branches per repo and the manifests that use them in the info log -- look for the line "Outputting branch_repo_dict:".
  • If you have time, run through the whole thing:
BRANCH=v2.1  # This is the new branch you want to create!
mozharness/scripts/merge_day/b2g_branch_repos.py -c mozharness/configs/merge_day/b2g_branch_repos.py --branch-name $BRANCH --clean-repos
mozharness/scripts/merge_day/b2g_branch_repos.py -c mozharness/configs/merge_day/b2g_branch_repos.py --branch-name $BRANCH

b2g git branching, merge day

  • Run through the whole thing. Hopefully this won't take too long now that you have the repos on disk. (If you don't, you probably want to look at the above section.):
BRANCH=v2.1  # This is the new branch you want to create!
mozharness/scripts/merge_day/b2g_branch_repos.py -c mozharness/configs/merge_day/b2g_branch_repos.py --branch-name $BRANCH --clean-repos
mozharness/scripts/merge_day/b2g_branch_repos.py -c mozharness/configs/merge_day/b2g_branch_repos.py --branch-name $BRANCH
  • Make sure your critical log is size 0 and you exited successfully.
  • |git diff| in build/b2g-manifest to make sure the manifest munging looks good.
  • When you're ready, commit and push.
BRANCH=v2.1
pushd build/b2g-manifest
# verify diff
git diff

# If it's all good, commit and push
git commit -m "bug XXXXXXX - create $BRANCH branch"
popd
mozharness/scripts/merge_day/b2g_branch_repos.py -c mozharness/configs/merge_day/b2g_branch_repos.py --branch-name $BRANCH --push

old style b2g git branching, DEPRECATED

cd b2g-manifest
git checkout -B v2.0 origin/master  # new branch based on new b2g version
cd ..
python braindump/git-related/b2g-branching/b2g_branch_repos.py -m b2g-manifest -b v1.4 -v --branch-order master 2>&1 | tee log

# for 2.0, this was
# python braindump/git-related/b2g-branching/b2g_branch_repos.py -m b2g-manifest -b v2.0 -v --branch-order master,b2g-4.3_r2.1,b2g-jellybean 2>&1 | tee log
  • verify that only mozilla maintained repos have had their revision modified, to avoid bug 969556 type bustage. (bug 971741 filed to fix the tooling, but until that is done...)
  • Verify changeset-specific revisions weren't changed (e.g. bug 1023353
  • Verify non-master platform_build manifest branches haven't been updated to the new branch bug 1023966
  • Look at the log for warnings... maybe check with Rail.
  • Every manifest in b2g-manifest/*.xml requires a new gaia+gecko version; you need to manually edit these.
    • These will also become the same B2G version. In this example, v1.4
  • Get review on the diffs
  • On merge day, commit+push every git repo.

EOL a B2G branch

  • Check the b2g train schedule to see if a B2G version is EOLing. For example, on 3/17/2014, 1.1 is EOLing. If so, do the below:
    • kill the branch in buildbot-configs. For example, on 3/17/2014, remove all references to mozilla-b2g18.
      • to roll out: review, land, reconfig
    • tools -- I'm not sure if we need/want to remove these from production-branches.json.
    • mark the branch as closed in treestatus and possibly remove it from tbpl altogether.

Bump gecko.git vcs-sync (odd numbered central gecko version)

  • Move the b2g train from mozilla-central to mozilla-aurora, and bump the mozilla-central version.
    • Legacy vcs-sync: You'll want the reverse of this patch, but with the right version number (1.4 for 3/17/2014).
    • New vcs-sync: We need a patch like this, but for hg.m.o/build/mozharness, and with the right version number (1.4 for 3/17/2014).
      • review, land
      • this will need to be transplanted/merged to production to take effect.

Pause gecko.git vcs-sync (even numbered central gecko version)

python mozharness/configs/vcs-sync/gecko-git.py
# this should exit without errors
    • Review + land on mozharness default
    • This needs to transplant/merge to mozharness production to go live.

Restart gecko.git vcs-sync (even numbered central gecko version)

  • Legacy vcs-sync: Add the new b2gXX_vX_X repo to gecko.git, like this and this (but not 1.3t, 1.3).
  • New vcs-sync: Add the new b2gXX_vX_X repo to gecko.git, like the first part of this.
    • review, land
    • This needs to transplant/merge to production to go live.

Roll out a legacy gecko.git change

  • To roll out, ssh to github-sync3.dmz.scl3.mozilla.com.
    • You'll need vcs2vcs_screen.
    • To show all screen windows, use ^A-" and use the up/down arrows and enter when you're at the right window.
    • To create a new screen window, use ^A-c
bin/vcs2vcs_screen -x
# get a bash screen window
cd /opt/vcs2vcs/bin
hg in  # verify the change(s) coming in
hg pull -u
cd /opt/vcs2vcs/etc
hg in  # verify the change(s) coming in
hg pull -u
# Switch to the 'combo' screen window
^C
keep_updated
^A-d

Pause Gecko l10n vcs-sync

  • Empty GECKO_BRANCHES = {} (commenting out the contents is probably easier for restarting this later).
  • get review; land on default; transplant to production.
  • (ignoring legacy vcs-sync due to bug 967282)

Add new Gaia and l10n vcs-sync

  • New vcs-sync: You'll need a patch like this for the new Gaia l10n version. This will begin populating the gecko l10n, and create a new gaia versioned l10n branch that follows gaia l10n master.
    • review, land on mozharness default. It'll need to merge/transplant to production to go live.
  • Legacy Gaia vcs-sync: follow the instructions here.
    • Review, land
    • To roll out legacy vcs-sync, ssh to github-sync3.dmz.scl3.mozilla.com;
sudo su - vcs2vcs
cd /opt/vcs2vcs/etc
hg in  # verify the change(s) coming in
hg pull -u

Update l10n vcs-sync

  • New vcs-sync: The previous merge day we landed a patch like this for the new Gaia l10n version. This time, we need to change the hg_url to https://hg.mozilla.org/releases/gaia-l10n/v2_0/%(locale)s (except replace the v2_0 with the appropriate b2g version; this is the version that's getting the new b2gXX_vX_X branch).
    • review, land on mozharness default. It'll need to merge/transplant to production to go live.
  • Legacy Gaia l10n vcs-sync: You'll need a patch like this for the new version, per locale tracked.
    • Review, land
    • To roll out both sets of patches, ssh to github-sync3.dmz.scl3.mozilla.com;
sudo su - vcs2vcs
cd /opt/vcs2vcs/etc
hg in  # verify the change(s) coming in
hg pull -u

Restart Gecko l10n vcs-sync (odd numbered central gecko version)

'v1.5': 'mozilla-central',
    • The previous version (that just moved to a b2gXX_vX_X branch) will be on mozilla-beta. For example, on 4/28/2014,
'v1.4': 'mozilla-beta',
    • There will be no mozilla-aurora entry.
  • get review; land on default; transplant to production.

Restart Gecko l10n vcs-sync (even numbered central gecko version)

'v1.5': 'mozilla-central',
    • The version on mozilla-aurora will be on mozilla-aurora. For example, on 3/17/2014,
'v1.4': 'mozilla-aurora',
    • The previous version will be on mozilla-beta. For example, on 3/17/2014,
'v1.3': 'mozilla-beta',
  • get review; land on default; transplant to production.

Tag B2G Gecko+Gaia for MERGEDAY

See the b2g train schedule for versioning. 

# Pull all the repos.  This will take a long time; do this ahead of time if you can!
mozharness/scripts/merge_day/b2g_tag.py -c mozharness/configs/merge_day/b2g_tag.py --pull

# Do everything.  This will also pull, so if you missed the previous --pull step, skip it and just run this.
mozharness/scripts/merge_day/b2g_tag.py -c mozharness/configs/merge_day/b2g_tag.py

# To clean local repos, you could
# mozharness/scripts/merge_day/b2g_tag.py -c mozharness/configs/merge_day/b2g_tag.py --clean-repos
# To [re]run one, f/e only pull/tag/push mozilla-b2g18:
# mozharness/scripts/merge_day/b2g_tag.py -c mozharness/configs/merge_day/b2g_tag.py --gecko-repo mozilla-b2g18

Enable RIL repacks

Update /home/buildduty/bin/ril_repack.sh on cruncher and update gecko versions if needed.

  • If it's an odd m-c gecko version, add the new aurora branch.
  • If it's an even m-c gecko-version, change all aurora references to the appropriate b2gXX_vX_X branch name.
Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/Merge_Duty/Steps&oldid=996680"