Jetpack/Release Process

This document describes the process by which versions of the Add-on SDK are released. For information about the process by which versions of the SDK are developed, see Jetpack/Development Process.

Prepare

Note: many of this step's substeps can be done in parallel.

Clone Checklist

Click on 'Create' ( below ) to copy this checklist to a new wiki page at Jetpack/SDK/Release_Checklist/VERSION (f.e. Jetpack/SDK/Release_Checklist/1.10), and choose people to fill the Release Manager (RM), Release Engineer (RE), Technical Lead (TL), and Documentation Lead (DL) roles. Use the checklist to track the status of the release.

<createbox> align=left type=create preload=Jetpack/SDK/TEMPLATE/Release_Checklist default=1.xx prefix=Jetpack/SDK/Release_Checklist/ </createbox>

The template for this form is here.

Past SDK Release checklists

• Jetpack/SDK/Release Checklist/1.10
• Jetpack/SDK/Release Checklist/1.11
• Jetpack/SDK/Release Checklist/1.12
• Jetpack/SDK/Release Checklist/1.13
• Jetpack/SDK/Release Checklist/1.14
• Jetpack/SDK/Release Checklist/1.15
• Jetpack/SDK/Release Checklist/1.16
• Jetpack/SDK/Release Checklist/1.17

Merge Master to Stabilization

Clone the canonical repository, and enter the clone's working directory:

git clone git@github.com:mozilla/addon-sdk.git
cd addon-sdk

Merge the master (development) branch to the stabilization branch:

git checkout stabilization
git merge --no-commit master
# resolve conflicts, if any (don't resolve install.rdf yet!)
git commit -m "merge master into stabilization to start the N.N release cycle"

If there are merge conflicts, you will need to resolve them first. In general, favor the version coming from the 'master' side, but keep an eye out for changes which somehow appeared on stabilization and *not* on master (perhaps these need to be cherry-picked to master). When done, compare the two trees to check for merge artifacts. The only differences should be in the minVersion/maxVersion fields, that will be changed next:

git diff master stabilization
# only differences should be in install.rdf, minVersion/maxVersion

Change the Firefox minVersion/maxVersion compatibility identifiers on the stabilization branch per the Jetpack/Development Process.

Commit the changes:

git commit -a -m"update Firefox minVersion/maxVersion to anticipated current/next versions of Firefox"

Change back to the master branch:

git checkout master

Set JPCV (Jetpack Current Version) to the version for which you are starting the release process: what the stabilization branch will turn into. If 'cfx --version' on master at this point reports "1.2-dev-SOMETHING", then JPCV will be "1.2":

export JPCV=(UPCOMING VERSION)

Set JPNV (Jetpack Next Version) to the next-higher version identifier: what master will turn into the next time it is merged to stabilization, e.g. 1.3:

export JPNV=(NEXT VERSION)

Record an empty commit, to mark where master development switches from one release to the next:

git commit --allow-empty -m "branched off $JPCV, master is now destined to become $JPNV"

Add a development tag to master, so subsequent checkouts self-identify as headed towards the next release:

git tag $JPNV-dev

Do NOT change the Firefox minVersion/maxVersion compatibility identifiers on the master branch at this point. Master branch maintains compatibility with all current Firefox development channels, from release to mozilla-central (eg, from Firefox 9.0 through 12.0a1). When the next version of Firefox ships three weeks from now, THEN you can bump compatibility numbers (eg, to 10.0/13.0a1).

Push the changes to the canonical repository, and push the JPNV-dev tag too:

git push origin master stabilization
git push --tags origin

File Tracking Bug

File a bug to track the release using this release tracking bug template.

Notify Relevant Parties

Community

Notify project participants in the discussion forum. Mention the tentative release date, and reference the tracking bug and release checklist.

Product Planning

Notify product planners via their weekly meeting.

PR

Notify the Mozilla PR team (communications@mozilla.com).

PR likes to have the email, despite sitting in the planning meetings.

Spin Test Builds

To ensure the release meets our quality standards, spin test builds and distribute them to testers. Do this at least weekly during the stabilization period, spinning beta builds initially and one or more release candidate builds ultimately.

Clone the canonical repository, and enter the clone's working directory, and check out the stabilization branch:

git clone git@github.com:mozilla/addon-sdk.git
cd addon-sdk
git checkout stabilization

Determine the version identifier for the beta or release candidate build. This should be the final release identifier (e.g. 1.2) with a suffix like "b1" or "rc2", e.g. "1.2b1":

Set JPNV (Jetpack Next Version) to the next version being built (f.e. 1.2b2):

export JPNV=1.2b1

Tag the repository with the new version identifier:

git tag ${JPNV}

Create a tarball and a ZIP archive:

git archive --format=zip --output addon-sdk-${JPNV}.zip --prefix addon-sdk-${JPNV}/ ${JPNV}
git archive --format=tar --output addon-sdk-${JPNV}.tar --prefix addon-sdk-${JPNV}/ ${JPNV}
gzip addon-sdk-${JPNV}.tar # makes addon-sdk-${JPNV}.tar.gz

Note: at present, the final release contains slightly different bits than the last release candidate: python-lib/cuddlefish/_version.py contains strings which are expanded by 'git archive' to include a tag name, which will be different in the rc and the final release. In addition, the unpacked directory name will be different.

Copy the tarball/ZIP archive to the distribution server:

scp addon-sdk-${JPNV}.tar.gz addon-sdk-${JPNV}.zip stage.mozilla.org:/pub/mozilla.org/labs/jetpack/

Note: make sure the two files have the correct permissions before proceeding.

Verify that the tarball/ZIP archive is ready for distribution by running integration checks on it:

mkdir TEST_ZIP && cd TEST_ZIP && python ../bin/integration-scripts/integration-check --url https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-${JPNV}.zip
cd ..
mkdir TEST_TGZ && cd TEST_TGZ && python ../bin/integration-scripts/integration-check --url https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-${JPNV}.tar.gz
cd ..

Note: you may need to set the --binary flag to the location of the binary of Firefox against which you want to test the build.

Push the stabilization branch and the tag to the canonical repository:

git push origin stabilization ${JPNV}

Notify testers about the build via a post to the discussion forum. Have them run automated and manual tests (including submitting generated XPIs to the AMO test server) and report any bugs they discover. (Mention that testers should not try to submit add-ons built from the test builds to AMO, as it won't pass the validator.)

File a bug in the Builder component to get the test build uploaded to the -dev server.

Stabilize Codebase

Land stabilizing changes on the master branch.

Note: because of conflicts due to incompatible SDK and Firefox minVersion/maxVersion changes between the stabilization and master branches, do not land stabilizing changes on the stabilization branch and merge that branch to master. In the future, we may resolve the conflicts and do this.

Periodically generate a list of changes between the master and stabilization branches:

git rev-list --right-only --cherry-pick --no-merges --pretty --reverse stabilization...master

Note: --right-only shows only commits on master (on the right in the symmetric diff notation stabilization...master) that are not on stabilization; --cherry-pick omits equivalent commits; --no-merges excludes merge commits (it's usually better to cherry-pick only non-merge commits); --pretty shows useful info about each commit; and --reverse shows commits chronologically from older to newer (the same order in which you are most likely to cherry-pick them successfully).

Identify stabilizing changes landed on the master branch, and cherry-pick them to the stabilization branch:

git cherry-pick -x (REVISION ID)

Draft Release Announcement

Draft a release announcement blog post for the Add-ons Blog.

To draft the post, access the blog's Wordpress admin page and go to Posts > Add New. Use previous release posts as a template for your new one. Make sure to include a <!-- more --> element to reduce the initial size of the post, and put the post into the developers, jetpack, general, and releases categories.

After drafting the post, share it with other folks who can help edit it, such as the project's marketing manager.

Note: don't publish the post in this step. That should happen in the Release step.

Write Release Notes

Release notes live on the wiki at /Labs/Jetpack/SDK/Release_Notes/(VERSION) (e.g. /Labs/Jetpack/SDK/Release_Notes/1.2). Create a page for the release notes, using the previous release's notes as a template, and include important information about the release in the notes. Mention the issues raised/addressed in each resolved bug that was fixed in the release and has been tagged with the relnote keyword. Also mention issues in each unresolved bug tagged with the relnote keyword.

Reference the release notes from the List of Release Notes.

Choose Candidate

Decide to ship a candidate as the final release.

Push Docs

After you have decided to ship a candidate as the final release, generate the docs for the release.

Clone the canonical repository, and enter the clone's working directory, check out the candidate tag, and activate the SDK:

git clone git@github.com:mozilla/addon-sdk.git
cd addon-sdk
git checkout (CANDIDATE VERSION)
source bin/activate

Generate and expand the docs:

cfx sdocs --baseurl="https://addons.mozilla.org/en-US/developers/docs/sdk/(FINAL VERSION)/"
tar xzf addon-sdk-docs.tgz

Clone the docs repository and delete everything it contains (wbamberg: this is evidently a bit mad, but it makes no sense to me to pretend that this is a new revision, since it's generated):

git clone git@github.com:mozilla/addon-sdk-sdocs.git
cd addon-sdk-sdocs
rm -rf *
git rm *

Copy the docs to the docs repository, commit the changes, tag the revision as (CANDIDATE VERSION)-amo (f.e. 1.2rc1-amo), and push the changes to the docs repository:

cp -r ../doc/* .
git add *
git commit -m "add (CANDIDATE VERSION) docset"
git tag -a (CANDIDATE VERSION)-amo -m "tagged (CANDIDATE VERSION) docset"
git push --tags origin master

Ask IT to push the docs to production and update the "latest" redirect to point to the new docs via this docs push bug template.

Note: IT pushes the docs at their earliest convenience, so we can test them, but they update the latest docs redirect on the day of the release.

(Myk: should there be separate bugs for the Push Docs and Update Latest Docs Redirect steps?)

Release

Create Final Build

Create a final tarball/ZIP archive, from the same sources as the most recent release candidate (JPRCV). This uses the same steps as "Spin Test Builds" above, but with a finalized version name:

git clone git@github.com:mozilla/addon-sdk.git
cd addon-sdk
git checkout stabilization

Recall the version of the last release candidate:

export JPRCV=1.2rc2

Determine the version identifier for the final release (e.g. 1.2), and store it in JPFV:

export JPFV=1.2

Tag the repository with the final version identifier, but do not push the tag until all subsequent validation steps have been completed:

git tag ${JPFV}

Confirm that the revision id for the final release is the same as the last release candidate:

git diff ${JPRCV} ${JPFV}
# should produce no output

Create a tarball and ZIP archive:

git archive --format=zip --output addon-sdk-${JPFV}.zip --prefix addon-sdk-${JPFV}/ ${JPFV}
git archive --format=tar --output addon-sdk-${JPFV}.tar --prefix addon-sdk-${JPFV}/ ${JPFV}
gzip addon-sdk-${JPFV}.tar # makes addon-sdk-${JPFV}.tar.gz

Retrieve, unpack, and compare the last release candidate tarball against the newly generated one. The only difference should be the embedded version string. Since the same git revision referenced by both tags (JPRCV and JPFV), the embedded git_refnames= string in the final tarball will contain both tags (JPRCV,JPFV). The code that analyzes git_refnames= will prefer the shorter number-bearing string:

scp stage.mozilla.org:/pub/mozilla.org/labs/jetpack/addon-sdk-${JPRCV}.tar.gz ./
tar xf addon-sdk-${JPRCV}.tar.gz
tar xf addon-sdk-${JPFV}.tar.gz
diff -urN addon-sdk-${JPRCV} addon-sdk-${JPFV}
# should show one line different in python-lib/cuddlefish/_version.py
# git_refnames= should change from (JPRCV,..) to (JPRCV,JPFV,..). e.g.:
#   git_refnames = " (HEAD, 1.4rc4, origin/stabilization, stabilization)"
#     changes to
#   git_refnames = " (HEAD, 1.4rc4, 1.4, origin/stabilization, stabilization)"

In the newly generated and unpacked tarball, confirm that the code reports the correct version (and *not* the rcN version):

addon-sdk-${JPFV}/bin/cfx --version
# Add-on SDK $JPFV (hex revision id)

Everything looks good.

Commit (to the) New Release

Now copy the archives to the distribution server:

scp addon-sdk-${JPFV}.tar.gz addon-sdk-${JPFV}.zip stage.mozilla.org:/pub/mozilla.org/labs/jetpack/

Note: make sure file permissions are correct (644) on both files!

Merge the stabilization branch to the release branch: this should be a fast-forward merge, because after each release, the release branch is merged back into the stabilization branch:

git checkout release
git merge --ff-only stabilization

And push the final release tag and the release branch to the public repo:

git push origin ${JPFV} stabilization release

Update AMO Validator

Requirements: working checkout of AMO-validator & prerequisites. Please see the AMO Validator Readme for the gory details. We should fork the amo-validator repo and then for each release do the following:

Step 1: update the hash file in our forked repo:

git pull mozamo master # update from the upstream 'mozamo' repo
cd jetpack
./generate_jp_whitelist.sh
cd ../

Make extra-super sure the pickle file can be git add'ed. If it doesn't show up:

rm validator/testcases/jetpack_data.txt.pickle
nosetests

Then go ahead:

git add validator/testcases/jetpack_data.txt
git add validator/testcases/jetpack_data.txt.pickle
git commit -m "Updated amo-validator with hashes for SDK version x.x.x"
git push origin master

Step 2: We issue a pull request to the mozilla/amo-validator and pester engage the AMO devs to accept our updated hash file, *and* add the tracking bug to the milestone for the next push. WARNING: unless the tracking bug is added to the milestone for the next push, the change will not be deployed!!!

Step 3: We do some testing on AMO's existing test infrastructure on allizom.com. The test should be essentially:

• run the validator on xpi files that use the new version
• ensure the validator does not raise errors on SDK core files.

Step 4: The changes are pushed with the regular Thursday AMO push.

Step 5: Once the AMO push is done, generate an XPI using the release candidate and validate it using AMO in order to verify that the validator has been updated and is working.

File Builder bug to push build

Once the Build has been blessed, we need to ensure that it is made available to Builder users as well. The process seems to be:

• File a bug AMO with the 'Add-on Builder' component, to add the new version to production.
• In a branch of your forked repo for Flightdeck, run these commands to add the new version as a submodule to the FlightDeck repo, then issue a pull request:

git submodule add git://github.com/mozilla/addon-sdk.git lib/addon-sdk-(VERSION)
cd lib/addon-sdk-(VERSION)
git checkout (VERSION)
cd ../..
git add lib/addon-sdk-(VERSION)
#Don't forget to update LOWEST_APPROVED_SDK and TEST_SDK in settings.py for the new SDK version!
git add settings.py

• ensure that seanmonstr / zalun / arron accepts the pull request and tags it. IT can then take this tag and complete the deployment.
• get IT[1] to run the 'add sdk' command on -dev, make sure things work. Also, stephend should be pinged to ensure tests pass on -dev with the new revision:

./manage.py add_core_lib addon-sdk-1.x --useversion=1.x

• Schedule a push with IT[1] and push the site
• Run the 'add sdk' command[1] in production

[1] It seems to help to assign a bug to IT (oremj specifically?) to get things done on IT's side, including a description of what needs to be done.

Update Latest Builds Redirect

Use a text editor like vi or emacs to update the /pub/mozilla.org/labs/jetpack/.htaccess file on ftp.mozilla.org to redirect the "latest" symlinks to the new release, making the content of the file look like this:

Redirect 307 /pub/mozilla.org/labs/jetpack/addon-sdk-latest.zip https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-(FINAL VERSION).zip
Redirect 307 /pub/mozilla.org/labs/jetpack/addon-sdk-latest.tar.gz https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-(FINAL VERSION).tar.gz
Redirect 307 /pub/mozilla.org/labs/jetpack/jetpack-sdk-latest.zip https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-(FINAL VERSION).zip
Redirect 307 /pub/mozilla.org/labs/jetpack/jetpack-sdk-latest.tar.gz https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-(FINAL VERSION).tar.gz

Test the symlinks to ensure they correctly provide the new release.

Update Latest Docs Redirect

Update the latest docs redirect to point to the new version's docs.

Publish Release Announcement

Access the Add-ons blog's Wordpress admin page and go to Posts. Publish the post you previously drafted.

Note: changes to the blog can take several hours to show up on the site because of server-side caching. To check that the post has been published, add "?(SOME UNIQUE STRING)" to the end of the URL, which bypasses the cache.

Notify Community

Notify the discussion forum about the release. Include a link to the release tag:

https://github.com/mozilla/addon-sdk/tree/(FINAL VERSION)

Update the IRC topic on #jetpack to announce the new release.

File a bug in the Builder component to get the new release pushed to the production Builder server.

Bask

Bask in the glow of the latest and greatest release!

Physically or virtually high-five or fist-jab contributors.

Review

Review the release process via a post-mortem, solicitation of feedback in the weekly meeting and discussion forum, and other methods as appropriate. Identify things that went well and we should continue to do, things that went badly that we should do differently next time, and parts of the process that have changed and for which this document needs to be updated. Make changes as appropriate.

Panic

Hot-Fixes are releases that are made directly on the "release" branch, rather than the usual "stabilization" branch, and contain just one or two fixes relative to the previous release. The Release Manager may decide to handle critical bugs in e.g. 1.5 by creating a hot-fix release (1.5.1) instead of waiting for the next scheduled release cycle (1.6).

To create a hot-fix release, start by creating a branch based off the most recent release tag, cherry pick the important fix to it from master, make the release, then merge back to stabilization.

The basic idea is that the "release" branch should always be a descendant of the "stabilization" branch, so that normal releases (made on stabilization) cause the "release" branch to be fast-forwarded to the new revision. The normal release cycle takes care of this automatically: the code is developed and tagged on stabilization, then "release" is moved forward to point to the same revision. But since hotfixes are developed on "release", an extra post-release merge step is necessary to bring "stabilization" up-to-date.

The general process is:

• git clone git@github.com:mozilla/addon-sdk
• cd addon-sdk
• git checkout release
• cherry-pick fixes from master to fix the bug, repeat until it works
• git tag NEWRELEASE
• use 'git archive' to create tarballs
• upload tarballs (644 permissions!)
• git checkout stabilization
• git merge release (this may require conflict-resolution)
• git push origin NEWRELEASE release stabilization

It may be easier to use a personal branch while developing the fix, to share release candidates with others (especially when asking the original bug submitter to validate the fix). In that case, the process will look something like:

• git clone MYGITHUBREPO ; cd MYREPO
• git remote add official git@github.com:mozilla/addon-sdk
• git fetch official
• git checkout -b hotfix official/release
• cherry-pick fixes, share RC builds
• git push origin hotfix (to share code)
• git checkout release
• git merge --ff-only hotfix
• git tag NEWRELEASE
• make+upload tarballs (644 permissions!)
• git checkout stabilization
• git merge official/release (and resolve conflicts)
• git push official NEWRELEASE release stabilization
• git branch -D hotfix

After you make the hotfix, you'll need to get the hotfix pushed to Builder and update the AMO Validator, as detailed further up this page.