Release Management/ESR Landing Process

Goal: We'd like to have a lightweight process that prevents "surprise" ESR landings at the 11th hour and makes sure ESR patches are created very soon after their related mainline patches

The ESR repo: https://hg.mozilla.org/releases/mozilla-esr60/

More info about the ESR:

• https://wiki.mozilla.org/Enterprise/Firefox/ExtendedSupport:Proposal
• https://www.mozilla.org/en-US/firefox/organizations/
• Download

What should land on mozilla-esr60: Security and some major stability fixes when they're landed/merged onto mozilla-beta, or fixes for regressions specific to the ESR

The associated flags:

approval-mozilla-esr60: ?, +, -

tracking-esr60: ?, -, 11+, 12+, 13+, ...

status-esr60: ?, unaffected, wontfix, affected, checkin-pending, fixed, verified

The process:

1) When an engineer believes that a stability bug needs to be addressed for the ESR, or a security fix that's landed on a branch affects the ESR:

• Set tracking-esr60 to ?
• Set status-esr60 to affected if an ESR patch is still being worked on
• Set approval-mozilla-esr60 to ? and status-esr60 checkin-pending if a patch is ready to land on mozilla-esr60

2) Members of both Security and Release Management teams will triage tracking-esr60=?, and if agreement is that it's needed for the ESR, the tracking flag will be set to the first version of mainline Firefox that the patch is present in and the approval-mozilla-esr60 flag will be set to +.

• In the case of an ESR-specific regression, we'll track for the next ESR release.
• Why are we using mainline versions to track the ESR? According to the ESR proposal, a chemspill uses a minor version number, thus throwing off our tracking flags. Additionally, engineers shouldn't have to know what version of the ESR we're on.

3) Once the mozilla-beta version matches the tracking-esr60 version, Security and Release Management teams will triage those bugs and make sure affected/checkin-pending fixes are landed at least 1 week prior to release so that we can go-to-build.

• Approval is required, so add "a={whoever set the approval-mozilla-esr10 flag to +}" to the commit message.
• After landing the patch, set status-esr60 to "fixed".
• Our beta period for the ESR will be to push this build to FTP a week early for qualification by enterprises.

ESR Triage Queries

ESR60

• ESR60 Nominations - track for an upcoming ESR release, or don't track at all
• ESR-60 approval requests - approve/deny, make sure 150+ is set

• Security bugs fixed on mainline, status not marked for ESR60 - mark the ESR as affected/unaffected
  • Minus b2g,tbird: Security bugs fixed on mainline, status not marked for ESR60 (this link needs work to update the changed to-from dates)

• Bugs marked as affected for ESR, but not tracked - track for a release or wontfix
• May need to land now - 150+ if necessary, or wontfix

• Bugs that need to be fixed on ESR24 this cycle - follow up with the engineers
• Tracked, but not for a specific version - follow up and decide on what version of the ESR these fixes should go into

B2G18

NOTE: Post-merge, we'll likely need to create new static links for 20+ b2g bugs (still landing to v1.1, along with 21+ bugs)

• B2G-18 security approval requests - approve/deny, make sure 150+ is set

• Security bugs fixed on mainline, status not marked for B2G18 - mark B2G as affected/unaffected
  • security bugs fixed on mainline, status not marked for B2G18 (alternate query, includes high priority unhidden bugs, skips Thunderbird et al)
• Security bugs marked as affected for B2G, but not tracked - track for a release or wontfix
• Security bugs that may need to land now - 150+ if necessary, or wontfix
• Security bugs that also may need to land now

• Bugs that need to be fixed on B2G18 this cycle - follow up with the engineers

B2G26(v1.2)

• Security bugs fixed on mainline, status not marked for 1.2 - mark status-b2g-v1.2: as affected/unaffected/wontfix

ESR Timeline and Activities

Weekly

Review ESR queries

• for potential ESR uplifts
• to review uplift requests
• to follow up on approved uplifts that have not landed.

Week 4

Send smoke test e-mail to enterprise list

Week 5

1. Kick off build in ship-it
2. Send go to build for ESR candidate
3. Create release notes in Nucleus

Release date

All happens after mainline release goes live

1. Post request for QA sign-off to r-d
2. Wait for QA sign-off e-mail to r-d from Matt Wobensmith
3. Post to r-d requesting releng push release to esr channel
4. Mark release notes as public in Nucleus
5. Update product details in SVN
6. Add release to historical releases page
7. Post release announcement to enterprise mailing list

Sample E-mails

Smoke Test Request

Subject: Pre-release builds of FF24.4.0esr available for smoke testing

As we approach our next targeted ESR release date of March 18th, when we'll be shipping 24.4.0esr, we'd like your help in smoke testing the latest pre-release builds. We're not asking for a full qualification of the builds prior to release, but rather minor exploratory testing of internal websites and applications that our QA team wouldn't otherwise have access to.

Please file bugs for any critical regressions you find and make sure to set the need-info? flag on release-mgmt at mozilla.com <mailto:release-mgmt at mozilla.com> so that we have visibility into the issue. Alternatively, you can directly email release-mgmt at mozilla.com <mailto:release-mgmt at mozilla.com> or this list.

Thank you for your continued support in ensuring high quality releases of the ESRs.

<name>
Firefox Release Manager

*NOTE*: Please do not deploy any of the builds linked to below - they are pre-release software. Updates will not function correctly, and these pre-release builds will not be supported by Mozilla.

ESR 24 Windows:
https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-esr24/firefox-24.3.0esrpre.en-US.win32.installer.exe
<https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-esr24/firefox-24.0esrpre.en-US.win32.installer.exe>
ESR 24 Mac OS X:
https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-esr24/firefox-24.3.0esrpre.en-US.mac.dmg
ESR 24 Linux 32 bits:
https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-esr24/firefox-24.3.0esrpre.en-US.linux-i686.tar.bz2
<https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-esr24/firefox-24.0esrpre.en-US.linux-i686.tar.bz2>ESR
24 Linux 64 bits:
https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-esr24/firefox-24.3.0esrpre.en-US.linux-x86_64.tar.bz2

Request to push release to esr channel

Subject: Please push Firefox 24.5.0esr build #1 to the ESR channel

Hello,

QA has signed off on the desktop release. Please push ESR to the esr release channel?

Thanks,
<name>

Release Announcement

Subject: Firefox 24.5.0esr released

We just released Firefox ESR 24.5.0. This release includes security fixes in line with Firefox 29.

You can find installers for Windows / Mac OS X / GNU/Linux here:
https://www.mozilla.org/en-US/firefox/organizations/all/

And the release notes are available here:
https://www.mozilla.org/en-US/firefox/24.5.0/releasenotes/

<name>
Firefox Release Manager