Jetpack/Release Process
Overview
Jetpack SDK development happens in roughly four week cycles, as described in Development Process, each of which culminates in stabilization and release of the codebase. This document describes the release process.
The overall process is:
- freeze development
- prepare to release
- notify relevant parties
- build/test release candidate(s)
- create marketing/PR/evangelism materials
- thaw development
- release
- feel good
- review the process
In general, we freeze late Sunday night, make the first release candidate available the next day, spend a week stabilizing the code and preparing the materials, and release the following Monday. To ensure that the code is well-baked, we release no sooner than a week after the first candidate and two days after the last one.
We use numerically consecutive versions to denote major releases (0.1, 0.2, etc.). Development cycles use the version of the next release followed by "pre" (0.1pre before 0.1, 0.2pre before 0.2, etc.). Release candidates use the final release version internally (i.e. in all code, documentation, and other files inside the package) and that version followed by "rc" and the number of the candidate (0.1rc1, 0.1rc2, etc.) in references external to the package (e.g. in the names of the packages).
The rest of this document describes each step in more detail. When following these steps to release a new version, track the non-trivial work being done by filing a meta-bug for the release that depends on individual bugs for each discrete task.
Freeze Development
Freezing is easy. Just notify Jetpack code contributors that development has been frozen via a post to the forum. The post should specify that the only changes that can be committed to the repository until it thaws are release blockers and low risk fixes that have been approved by the release drivers.
Prepare to Release
Preparing to release involves doing three different sets of things, which can be done in parallel:
- notify relevant parties
- build/test release candidate(s)
- create marketing/PR/evangelism materials
Notify Relevant Parties
Notify Forum
Notify Jetpack participants in the discussion forum about the release. Mention the tentative release date, tracking bug, and release notes. Notify the forum again if the release date changes and when each release candidate becomes available.
Notify Ambassadors
Notify Jetpack ambassadors about the release. Include the same information in the notice sent to the discussion forum.
Notify PR
Contact the Mozilla PR team and let them know about the release by emailing press@mozilla.com. Tell them the tentative release date. Note whether this is just another in the regular monthly sequence of releases or a major milestone (which determines whether or not the release is likely to be a PR event).
Notify them again if anything changes (f.e. the release date).
Notify Product Delivery Team
Notify the product delivery team about the release by adding a summary of its goals, status of the release candidates, and tentative release date to the weekly meeting notes. Attend the weekly meeting and talk about the release if asked to do so.
Build/Test Release Candidate(s)
In order to make sure a release meets our quality standards, we build and test one or more release candidates before releasing the final version.
To build and test a release candidate, start by cloning the repository and entering its working directory:
hg clone ssh://hg.mozilla.org/labs/jetpack-sdk/ jetpack-sdk cd jetpack-sdk
Update references to the version in static-files/index.html and other files to the final version (e.g. 0.3), then commit the change:
hg commit -m"update version for next release"
Tag the repository with both the release candidate version and the final version (e.g. 0.3rc1 and 0.3):
hg tag 0.3rc1 hg tag 0.3
Note: you tag the repository with the release candidate version to record which version the release candidate represents, while you tag it with the final version so the name of the directory in the packages is correct.
Push the changes to the central repository:
hg push
The central repository will automatically build tarball and ZIP archives based on the version tags you created. Download the packages for the final version tag:
https://hg.mozilla.org/labs/jetpack-sdk/archive/0.3.tar.gz https://hg.mozilla.org/labs/jetpack-sdk/archive/0.3.zip
The packages you download will have "jetpack-sdk-" prepended to them, e.g. jetpack-sdk-0.3.tar.gz and jetpack-sdk-0.3.zip. Rename the files to reflect the release candidate version:
mv jetpack-sdk-0.3.tar.gz jetpack-sdk-0.3rc1.tar.gz mv jetpack-sdk-0.3.zip jetpack-sdk-0.3rc1.zip
Push the packages to the staging server:
scp jetpack-sdk-0.3rc1.tar.gz jetpack-sdk-0.3rc1.zip stage.mozilla.org:/pub/mozilla.org/labs/jetpack/
The packages should immediately become available on the distribution server:
https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/jetpack-sdk-0.3rc1.tar.gz https://ftp.mozilla.org/pub/mozilla.org/labs/jetpack/jetpack-sdk-0.3rc1.zip
Notify QA, the forum, and other testers about the candidate. Have them run manual tests and report any bugs they discover. Triage reported bugs at least daily, determine which are blockers, and get Jetpack developers to fix those blockers. If any blockers are found during an RC test cycle, spin another RC after those blockers are fixed and get testers to verify the fixes and make sure they didn't cause regressions.
Create Marketing/PR/Evangelism Materials
Update Website
The Jetpack website lives at jetpack.mozillalabs.com. Some of its content is release-neutral (e.g. links to SDK documentation point to the "latest" version of the docs rather than the docs for a particular release). Other content, however, is release-specific, like the buttons and links that download a particular release of the SDK. Those links must be updated to point to the new release. (Should we set up "latest" symlinks to the latest release on the distribution server so we don't have to update the website with each release?)
To update the website, clone its code from the old Jetpack Prototype repository (the code lives in the website subdirectory), change the code as needed, test your changes by loading your local working copy of the code in your browser, commit your changes, tag the changeset with the "website-production" tag, and push your changes back to the central repository.
Push Documentation
Documentation lives on the website in /sdk/[VERSION]/docs/ directories (with a redirect from /sdk/latest/docs/ to the latest version's docs), but the process for pushing a new version's docs to the website is different than for other pages on the site.
To push a new version's docs to the website, ask Zandr to do it. (We should make this process more generic.)
Note: don't update the "latest" redirect in this step. That should happen in the Release step.
Write Blog Post
The Jetpack blog lives at mozillalabs.com/jetpack. Each release should be accompanied by a blog post that announces and describes the release.
To write the post, access the blog's Wordpress admin page and go to Posts > Add New.
Note: don't publish the post in this step. That should happen in the Release step.
Write Release Notes
Release notes live on the wiki at /Labs/Jetpack/SDK/Release_Notes/[VERSION] (e.g. /Labs/Jetpack/SDK/Release_Notes/0.2). Create a page for the release notes, using the previous release's notes as a template, and include important information about the release in the notes. Mention the issues raised/addressed in each resolved bug that was fixed in the release and has been tagged with the relnote keyword. Also mention issues in each unresolved bug tagged with the relnote keyword.
Thaw Development
Update references to the version in static-files/index.html and other files to the development cycle version (e.g. 0.4pre), then commit and push the change:
hg commit -m"update version for next development cycle" hg push
Notify Jetpack code contributors that development has been thawed via a post to the forum. The post should specify that the repository is open for general checkins and that checkins no longer need driver approval.
Note: in theory, you can thaw development any time after the first release candidate, since it's possible to branch the tree at that point and fix release blockers on the branch. In practice, it's better to keep the tree frozen longer, both to focus developers' attention on release blockers and to obviate the encumberance of branching. However, as the number of active Jetpack code contributors increases, and if stabilization cycles become longer, it may start to become better to thaw earlier.
Release
Bless Candidate
Blessing the last release candidate simply entails copying its archives to files with names reflecting the final release:
ssh stage.mozilla.org cd /pub/mozilla.org/labs/jetpack/ cp jetpack-sdk-0.3rc1.tar.gz jetpack-sdk-0.3.tar.gz cp jetpack-sdk-0.3rc1.zip jetpack-sdk-0.3.zip
Update Website
File an IT request (cc:ing :zandr) to have the changes pushed to production. Specify that the changeset tagged "website-production" is the version of the site to push to production. Ideally this should happen a couple days before the target release date, so IT has time to push the changes to production.
Update Latest Redirect
Ask Zandr to do it. (We should make this process more generic.)
Update Blog
The Jetpack blog lives at mozillalabs.com/jetpack. Most of its content (by which I mean the content that surrounds the blog posts, i.e. logos, navigation links, etc.) is release-neutral, but the download links are release-specific, just as they are on the website. Since there is no way to stage changes to the blog, this happens here in the Release step rather than in the Prepare to Release step.
To update the links, access the blog's Wordpress admin page, go to Appearance > Widgets, and edit the text of the Get It Now widget.
Note: changes to the blog can take several hours to show up on the site because of server-side caching. To check that your changes have been applied, add "?[some unique string]" to the end of the URL, which bypasses the cache.
Publish Blog Post
Access the blog's Wordpress admin page and go to Posts. Publish the post you previously wrote.
Note: changes to the blog can take several hours to show up on the site because of server-side caching. To check that the post has been published, add "?[some unique string]" to the end of the URL, which bypasses the cache.
Feel Good
For minor releases (every month):
- physically high-five local Jetpack contributors;
- virtually high-five remote Jetpack contributors.
For major milestones (every 6-9 months):
- do everything you do for minor releases;
- celebrate over dinner/drinks with local Jetpack contributors.
Review the Process
After every release, it's useful to review the process described here and update it as appropriate. This can take the form of a formal post mortem, solicitation of feedback in the regular weekly meeting, or individual review by folks who were involved in the release.