Jetpack/Release Process
Overview
Jetpack SDK development happens in roughly four week cycles, as described in Development Process, each of which culminates in stabilization and release of the codebase. This document describes the release process.
The overall process is:
- freeze development
- prepare to release
- notify relevant parties
- build/test release candidate(s)
- create marketing/PR/evangelism materials
- thaw development
- release
- feel good
- review the process
In general, we freeze on a Tuesday night at 11:59pm, make the first release candidate available the next day, spend about a week stabilizing the code and preparing the materials, and release the following Wednesday. To ensure that the code is well-baked, we release no sooner than a week after the first candidate and two days after the last one.
We use numerically consecutive versions to denote major releases (0.1, 0.2, etc.). Development cycles use the version of the next release followed by "pre" (0.1pre before 0.1, 0.2pre before 0.2, etc.). Release candidates use the final release version internally (i.e. in all code, documentation, and other files inside the package) and that version followed by "rc" and the number of the candidate (0.1rc1, 0.1rc2, etc.) in references external to the package (e.g. in the names of the packages).
The rest of this document describes each step in more detail. When following these steps to release a new version, track the non-trivial work being done by filing a meta-bug for the release that depends on individual bugs for each discrete task.
Freeze Development
Freezing is easy. Just notify Jetpack code contributors that development has been frozen via a post to the forum. The post should specify that the only changes that can be committed to the repository until it thaws are release blockers and low risk fixes that have been approved by the release drivers.
Kill two birds with one stone! Use the post to notify the forum about the release, as specified in Prepare to Release > Notify Relevant Parties > Notify Forum below.
Update Credits
Update the Credits page at static-files/md/dev-guide/credits.md to include the names of folks who have contributed to the project since the last release.
Prepare to Release
Preparing to release involves doing three different sets of things, which can be done in parallel:
- file tracking bugs
- notify relevant parties
- build/test release candidate(s)
- create marketing/PR/evangelism materials
File Tracking Bugs
File a bug to track code blockers of the release and another to track the overall release.
Notify Relevant Parties
Notify Forum
Notify Jetpack participants in the discussion forum about the release. Mention the tentative release date, tracking bug, and release notes. Notify the forum again if the release date changes and when each release candidate becomes available.
Notify Ambassadors
Notify Jetpack ambassadors about the release. Include the same information in the notice sent to the discussion forum.
Notify PR and Engagement
Contact the Mozilla PR team (press@mozilla.com) as well as the folks in charge of Jetpack engagement on the Engagement team (currently Mayumi Matsuno and Dan Horner) and let them know about the release by emailing them. Tell them the tentative release date. Note whether this is just another in the regular monthly sequence of releases or a major milestone (which determines whether or not the release is likely to be a PR event).
Notify them again if anything changes (f.e. the release date).
Notify Product Delivery Team
Notify the product delivery team about the release by adding a summary of its goals, status of the release candidates, and tentative release date to the weekly meeting notes. Attend the weekly meeting and talk about the release.
Notify IT
IT is responsible for pushing changes to the website (including new versions of the documentation) to production, and they should be notified in advance that such changes are forthcoming, even though they don't actually push the changes until the day of the release.
To notify them, file an IT request (IT request bug template), cc:ing ":zandr", to have the website changes pushed to production, including:
- pushing the new version's docs to the website;
- updating the /sdk/latest/docs/ redirect to point to the new version's docs;
- pushing the files in the changeset tagged "website-production" in the "website" subdirectory in the repository https://hg.mozilla.org/labs/jetpack to the production server.
Make sure to note that the work shouldn't be done until the day of the release, specify the tentative release date, and update it if it changes.
Build/Test Release Candidate(s)
In order to make sure a release meets our quality standards, we build and test one or more release candidates before releasing the final version.
To build and test a release candidate, start by cloning the canonical repository and entering its working directory:
git clone git@github.com:mozilla/addon-sdk.git cd addon-sdk
If this is the first RC, create a branch named branch-[version] (e.g. branch-1.0b1), update references to the version in static-files/index.html and other files to the final version you are releasing (e.g. 1.0b1), then commit the change:
git checkout -b branch-1.0b1 (update the version) git commit -a -m"update version for next release"
Otherwise, if this is the second or a subsequent RC, change to the previously-created branch:
git checkout branch-1.0b1
Create tarball and ZIP archives whose names include a release candidate identifier but whose directory prefixes (i.e. the directory that will be created when the archives are expanded) do not include a release candidate identifier:
git archive --format=zip --output ~/addon-sdk-1.0b1rc1.zip --worktree-attributes --prefix addon-sdk-1.0b1/ HEAD git archive --format=tar --output ~/addon-sdk-1.0b1rc1.tar --worktree-attributes --prefix addon-sdk-1.0b1/ HEAD gzip ~/addon-sdk-1.0b1rc1.tar # makes addon-sdk-1.0b1rc1.tar.gz
Verify that the builds are ready for distribution by running integration and other checks on them.
Tag the repository with the release candidate version:
git tag 1.0b1rc1 branch-1.0b1
Push the branch and tag back to the canonical repository:
git push origin --tags branch-1.0b1
Push the archives to the staging server:
scp addon-sdk-1.0b1rc1.tar.gz addon-sdk-1.0b1rc1.zip stage.mozilla.org:/pub/mozilla.org/labs/jetpack/
The packages should immediately become available on the distribution server:
https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-1.0b1rc1.tar.gz https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-1.0b1rc1.zip
Notify QA, the forum, and other testers about the candidate. Have them run automated & manual tests (including submitting generated XPIs to the AMO test server) and report any bugs they discover.
Triage reported bugs at least daily, determine which are blockers, and get Jetpack developers to fix those blockers. If any blockers are found during an RC test cycle, spin another RC after those blockers are fixed and get testers to verify the fixes and make sure they didn't cause regressions.
An additional criteria for build quality is approval of the documentation. The documentation lead, or a Jetpack driver, must sign off on the documentation for a given release candidate to be approved for final release.
Create Marketing/PR/Evangelism Materials
Update Website
The Jetpack website lives at jetpack.mozillalabs.com. Much of its content is release-neutral. For example, links to SDK documentation point to a "latest" redirect that takes you to the latest release version of the docs. And buttons/links to download the SDK packages point to "latest" symlinks that link to the latest release packages.
Some content, however, might need updating. For example, the description of the SDK on the home page might change over time as the SDK acquires additional APIs and other functionality.
To update the website, clone its code from the old Jetpack Prototype repository (the code lives in the website subdirectory), change the code as needed, test your changes by loading your local working copy of the code in your browser, commit your changes, tag the changeset with the "website-production" tag (you'll need to specify -f to force Mercurial to update the existing tag), and push your changes back to the central repository.
Draft Blog Post
Draft a blog post for the Add-ons Blog to announce the release.
To draft the post, access the blog's Wordpress admin page and go to Posts > Add New. Use previous release posts as a template for your new one. Make sure to include a <!-- more --> element to reduce the initial size of the post, and put the post into the developers, jetpack, general, and releases categories.
After drafting the post, share it with other folks who can help edit it, such as the project's marketing manager.
Note: don't publish the post in this step. That should happen in the Release step.
Write Release Notes
Release notes live on the wiki at /Labs/Jetpack/SDK/Release_Notes/[VERSION] (e.g. /Labs/Jetpack/SDK/Release_Notes/0.2). Create a page for the release notes, using the previous release's notes as a template, and include important information about the release in the notes. Mention the issues raised/addressed in each resolved bug that was fixed in the release and has been tagged with the relnote keyword. Also mention issues in each unresolved bug tagged with the relnote keyword.
Thaw Development
Clone the canonical repository and enter its working directory:
git clone git@github.com:mozilla/addon-sdk.git cd addon-sdk
Update references to the version in static-files/index.html and other files to the development cycle version, which is the conservative anticipated next release followed by the "pre" suffix (e.g. 1.0b2pre), then commit and push the change:
(update references to the version) git commit -a -m"update version for next development cycle" git push origin master
Notify Jetpack code contributors that development has been thawed via a post to the forum. The post should specify that the repository is open for general checkins and that checkins no longer need driver approval.
Note: in theory, you can thaw development any time after the first release candidate, since it's possible to fix release blockers on the branch. In practice, it's better to keep the tree frozen longer to focus the team's attention on release blockers.
Release
Bless Candidate
Copy the archives for the latest release candidate to files without the release candidate identifier in the name:
ssh stage.mozilla.org cd /pub/mozilla.org/labs/jetpack/ cp addon-sdk-1.0b1rc1.tar.gz addon-sdk-1.0b1.tar.gz cp addon-sdk-1.0b1rc1.zip addon-sdk-1.0b1.zip
Update the /pub/mozilla.org/labs/jetpack/.htaccess file to redirect the "latest" symlinks to the new release:
Redirect 307 /pub/mozilla.org/labs/jetpack/addon-sdk-latest.zip https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-1.0b1.zip Redirect 307 /pub/mozilla.org/labs/jetpack/addon-sdk-latest.tar.gz https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-1.0b1.tar.gz Redirect 307 /pub/mozilla.org/labs/jetpack/jetpack-sdk-latest.zip https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-1.0b1.zip Redirect 307 /pub/mozilla.org/labs/jetpack/jetpack-sdk-latest.tar.gz https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/addon-sdk-1.0b1.tar.gz
Test the latest symlinks to ensure they correctly provide the new release.
Update Website
In the bug you filed earlier about pushing the website changes to production, comment that now is the time to do the work and wait for its assignee to do it.
Publish Blog Post
Access the Add-ons blog's Wordpress admin page and go to Posts. Publish the post you previously drafted.
Note: changes to the blog can take several hours to show up on the site because of server-side caching. To check that the post has been published, add "?[some unique string]" to the end of the URL, which bypasses the cache.
Notify Forum
Notify the discussion forum about the release.
Feel Good
For minor releases (every month):
- physically high-five or fist jab local Jetpack contributors;
- virtually high-five or fist jab remote Jetpack contributors.
For major milestones (every 6-9 months):
- do everything you do for minor releases;
- celebrate over dinner/drinks with local Jetpack contributors;
- celebrate over the internet with remote Jetpack contributors.
Review the Process
After every release, it's useful to review the process described here and update it as appropriate. This can take the form of a formal post mortem, solicitation of feedback in the regular weekly meeting, or individual review by folks who were involved in the release.
Checklist Template
The checklist table below can be used for tracking the status of each step in the process, per release.
Jetpack Release Checklist
| Step | Status | Owner |
| Freeze Development | ||
| Update Credits | ||
| Prepare to Release | ||
| Notify Relevant Parties | ||
| Notify Forum | ||
| Notify Ambassadors | ||
| Notify PR | ||
| Notify Product Delivery Team | ||
| Notify IT | ||
| Build/Test Release Candidate(s) | ||
| Create Marketing/PR/Evangelism Materials | ||
| Update Website | ||
| Write Blog Post | ||
| Write Release Notes | ||
| Thaw Development | ||
| Release | ||
| Bless Candidate | ||
| Update Website | ||
| Publish Blog Post | ||
| Notify Forum | ||
| Feel Good | ||
| Review the Process |