ESR Landing Process

From MozillaWiki
Jump to: navigation, search


Goal:

We'd like to have a lightweight process that prevents "surprise" ESR landings at the 11th hour and makes sure ESR patches are created very soon after their related mainline patches.

REPOS

The ESR78 repo: https://hg.mozilla.org/releases/mozilla-esr78/ The ESR78 repo: https://hg.mozilla.org/releases/mozilla-esr78/

More info about the ESR

What should land on mozilla-esr78/mozilla-esr78

Security and some major stability fixes when they're landed/merged onto mozilla-beta, or fixes for regressions specific to the ESR. In the first few cycles of ESR we may be more flexible on these criteria. As the versions progress we should limit this to the most critical security fixes.

Exception: If patches only make changes to tests, test harnesses or anything else that does not affect the shipped builds, they may land with self approval (use a=testonly, a=npotb etc).

The associated flags

approval-mozilla-esr78: ?, +, -

approval-mozilla-esr78: ?, +, -

tracking-esr78: ?, -, 125+, 126+, ...

tracking-esr78: ?, -, 125+, 126+, ...

status-esr78: ?, unaffected, wontfix, affected, checkin-pending, fixed, verified

status-esr78: ?, unaffected, wontfix, affected, checkin-pending, fixed, verified

The process

1) When an engineer believes that a stability bug needs to be addressed for the ESR, or a security fix that's landed on a branch affects the ESR:

  • Set tracking-firefox-esr78 (and if we're in the middle of supporting 2 versions, then also tracking-firefox-esr78) to ?
  • Set status-firefox-esr78 (and if we're in the middle of supporting 2 versions, then also status-firefox-esr78) to affected if an ESR patch is still being worked on
  • Set approval-mozilla-esr78 (and if we're in the middle of support 2 versions, then also approval-mozilla-esr78) to ? and status-firefox-esr78/78 to checkin-pending if a patch is ready to land on mozilla-esr78/78

2) Members of both Security and Release Management teams will triage tracking-firefox-esr78/78=?, and if agreement is that it's needed for the ESR, the tracking flag will be set to the first version of mainline Firefox that the patch is present in and the approval-mozilla-esr78/78 flag will be set to +.

  • In the case of an ESR-specific regression, we'll track for the next ESR release.
  • Why are we using mainline versions to track the ESR? According to the ESR proposal, a chemspill uses a minor version number, thus throwing off our tracking flags. Additionally, engineers shouldn't have to know what version of the ESR we're on.

3) Once the mozilla-beta version matches the tracking-firefox-esr78/78 version(s), Security and Release Management teams will triage those bugs and make sure affected/checkin-pending fixes are landed at least 1 week prior to release so that we can go-to-build.

  • Approval is required, so add "a={whoever set the approval-mozilla-esr78 flag to +}" to the commit message.
  • After landing the patch, set status-firefox-esr78/78 to "fixed".
  • Our beta period for the ESR will be to push this build to FTP a week early for qualification by enterprises.

ESR Triage Queries

CURRENT TEMPLATE VALUES

ESR_CURRENT (click to edit): 78

ESR_BRANCH_DATE (click to edit): 2019-05-20 // Firefox 78 beta branch date

ESR_NEXT (click to edit): 78

ESR_NEXT_BRANCH_DATE (click to edit): 2020-06-01 // Firefox 78 beta branch date

ESR78


ESR78

Note: not useful until Firefox 78 is on Beta


ESR Timeline and Activities

Weekly

Review ESR queries

  • for potential ESR uplifts
  • to review uplift requests
  • to follow up on approved uplifts that have not landed
  • Note: if you don't have security bug access, please ask another release manager to check the sec-critical bugs to nominate them and cc you.

Week 4

Tue: Send smoke test e-mail to enterprise list

Week 5

Mon:

  1. Kick off RC build in ship-it
  2. Send go to build e-mail for ESR candidate to release-drivers. (This happens automatically if you fill in the form in ship-it)
  3. Create release notes in Nucleus

Release Week

Monday: ESR sign off on esr-localtest channel should have been received, so now email release-drivers list to request the push to esr-cdntest (more on these channels in Releases/Update_Channels.

Release Day

All happens after mainline release goes live (usually 8am Pacific time)

  1. You must have received sign off for esr-cdntest updates from QA
  2. Talk with sec team to coordinate timing for sec advisories going live
  3. Send push release to esr channel e-mail request for releng to r-d
  4. Mark release notes as public in Nucleus
  5. Update product details in SVN (instructions; add the version to history/firefoxHistory.class.php and update the constant in FIREFOX_ESR.php)
  6. Add release to historical releases page
  7. Post release announcement to enterprise mailing list
  8. Update as needed the wiki template values - Release_Management/ESR_Landing_Process#CURRENT_TEMPLATE_VALUES - at the very least you are bumping the dot version of the current ESR
  9. File bugs in the BMO: Adminstration component to create and remove ESR related tracking flags and attachment flags at the boundaries between major versions. Examples can be found in this bugzilla query: https://bugzilla.mozilla.org/buglist.cgi?list_id=12657880&short_desc=esr%20flag&resolution=FIXED&query_format=advanced&short_desc_type=allwordssubstr

Sample E-mails

Smoke Test Request

Subject: Request: Smoke Test Firefox {{ESR_NEXT}}esr

As we approach the ESR {{ESR_NEXT}} release date of {{FIREFOX_SHIP_DATE}}, we'd like your help in smoke testing the latest pre-release builds. Our request is not for full qualification of the builds but rather exploratory testing of internal websites and applications for which our QA team does not have access.

Please file bugs for any critical regressions that you find in Bugzilla and make sure to set the need-info? flag on release-mgmt at mozilla.com <mailto:release-mgmt at mozilla.com> in order to alert the release management team to the issues. Alternatively, you can directly email release-mgmt at mozilla.com <mailto:release-mgmt at mozilla.com> or this list.

Thank you for your continued support in ensuring high quality ESR releases.

<name>
Firefox Release Manager


*NOTE*: Do not deploy any of the pre-release builds listed below. These pre-release builds will not update correctly and will not be supported by Mozilla.


ESR {{ESR_NEXT}} Windows:
https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-esr38-win32/latest/firefox-38.1.0esrpre.en-US.win32.installer.exe

ESR {{ESR_NEXT}} Mac OS X:
https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-esr38-macosx64/latest/firefox-38.1.0esrpre.en-US.mac.dmg

ESR {{ESR_NEXT}} Linux 32 bits:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-esr38-linux/latest/firefox-38.1.0esrpre.en-US.linux-i686.tar.bz2

ESR {{ESR_NEXT}} Linux 64 bits:
https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-esr38-linux64/latest/firefox-38.1.0esrpre.en-US.linux-x86_64.tar.bz2

Request to push release to esr channel

Subject: Please push Firefox 24.5.0esr build #1 to the ESR channel

Hello,

QA has signed off on the desktop release. Please push ESR to the esr release channel. 

Thanks,
<name>

Release Announcement

Subject: Firefox 24.6.0esr released

I'm pleased to announce the release of Firefox ESR 24.6.0. This release includes security fixes in line with Firefox 30.

You can find installers for Windows / Mac OS X / GNU/Linux here:
https://www.mozilla.org/en-US/firefox/organizations/all/

And the release notes are available here:
https://www.mozilla.org/en-US/firefox/24.6.0/releasenotes/

<name>
Firefox Release Manager

Branching Logistics

When we have the overlapping ESR (and the first build of a major version number for the ESR.next - 10, 17, 24, 31, etc) we do build a separate build with its own branding from the newly minted branch. See https://wiki.mozilla.org/Releases/Firefox_24.0.esr/BuildNotes for how to kick off the builds and http://hg.mozilla.org/releases/mozilla-esr31/rev/e91c79c4c04b for an example of the custom configs needed for that first build's partials that has to land before kicking off the build in ship-it.