NSS/Release Management

This article describes the release process for NSS on the example of version 3.26. There are additional commands in the helper script, for doing releases like 3.27.1 or 3.27.1.2 We do two-digit releases whenever we have new features. We do three-digit releases if we must fix a stable branch. We reserve four-digit release numbers to scenarios, where 3.27.1 and 3.27.2 have been released, an important branch (e.g. ESR) still uses 3.27.1 and we can't upgraded to 3.27.2, but it requires an important backported fix. In that case we'd use numbers 3.27.1.1 3.27.1.2 etc.

Prepare the NSS repo

1. clone it
   1. $ hg clone https://hg.mozilla.org/projects/nss
   2. or reuse your local tree and run

      $ hg pull -u

2. Ensure you're on the correct branch, and that you don't have anything locally applied. We usually release from the "default" branch.

   $ hg summ

3. Check that there haven't been any unexpected commits by someone else.
   1. Either have a brief look at https://hg.mozilla.org/projects/nss
   2. or check the log

      $ hg log or $ hg glog

Communication

1. (We usually declare the tree as frozen prior to release, and usually we have created a beta tag a couple of days ago, and there shouldn't have been any commits after the last beta tag.)
2. Also, we usually discuss on the nss-dev list when we freeze nspr/nss.Someone should have sent a note to the list, announcing that a new NSS release is being worked on, and that the NSS tree is frozen.
3. Please send email to the "nss-dev" list saying "The NSS trunk is open for 3.27 development" after finishing the process.
4. After the release is done:
   1. Please send email to nss-dev, saying that the NSS release has been uploaded to FTP/S3.
   2. Ask for the list of changes in the release, and ask who will send the announcement.
   3. Send the announcement to the dev-tech-crypto mailing list https://lists.mozilla.org/listinfo/dev-tech-crypto

Verify correctness

Look at the diff between the previous released version, and the latest snapshot. Verify that the changes to *.def files are clean, meaning, nobody has accidentally added new symbols to old version definitions, and any additions have been added with the correct new NSS version number.

$ hg diff -r NSS_3_25_RTM -r default | lsdiff |grep \.def$

If the above reports that any *.def files changed, inspect those changes. Mistakes must be fixed before a release.

Verify branch readiness

Make sure the branch is passing all appropriate tests in CI. Sometimes older branches may have expected failures, but if you aren't sure, confirm before making a release.

You may have to search the prior jobs in treeherder to find the relevant branch.

Prepare a beta

1. Make sure the beta tags are there in lib/nss/nss.h, lib/softoken/softkver.h, lib/util/nssutil.h
2. If the branch hasn't been made yet, make it

   $ hg branch NSS_3_39_BRANCH

3. Otherwise, ensure you're on the correct branch.
4. Then produce the beta tag on the branch. We number betas from BETA1 upward

   $ hg tag NSS_3_39_BETA1

5. Push the changes

   $ hg outgoing

   $ hg push --new-branch

Prepare the release

1. Make sure that the release notes are in NSS before the release
2. Ensure the required NSPR version has been updated
   1. Edit file automation/release/nspr-version.txt and set it to the minimum NSPR version that is required for this NSS release.
3. Remove beta tags

   $ cd nss

   $ python automation/release/nss-release-helper.py remove_beta

   $ hg diff

   There should be differences in lib/nss/nss.h, lib/softoken/softkver.h, lib/util/nssutil.h

4. Verify that
   1. _VERSION definition was changed to have the Beta string removed
   2. _VMINOR was changed if this is a minor release
   3. the _BETA definition has been changed to PR_FALSE
5. Commit the version numbers.

   $ hg commit -m "Set version numbers to 3.26 final"

6. Tag the version
   1. We start release candidate tags with RC0.
   2. When ready for RTM, let's do an RTM tag.

   $ hg tag NSS_3_26_RTM

7. Push the changes

   $ hg outgoing

   $ hg push

Prepare repo for next version

1. On the default branch, increment the version number (to 3.27 in this example) and set the beta flag again. (If you are on the default branch, the beta flag should already be set and only a version increment should happen.)
   1. $ python automation/release/nss-release-helper.py set_beta
   2. $ python automation/release/nss-release-helper.py set_version_to_minor_release 3 27
2. Push changes
   1. $ hg commit -m "Set version numbers to 3.27 Beta"
   2. $ hg outgoing
   3. $ hg push

Create the NSS release archive

1. Make sure you have a stage directory next to the NSS dir.

   $ mkdir stage

2. Go to the NSS dir

   $ cd /your/path/to/nss

3. Make sure that the respective NSPR release archive file (for the version listed in automation/release/nspr-version.txt) is contained in your stage directory. (Download it, if you don't have it, from https://ftp.mozilla.org/pub/nspr/releases/) A copy of this NSPR release will be automatically included in the combined NSS release archived. It needs to be stored in stage/v<NSPR version number>/src/.
4. Run the following command to create two release archive files (nss-3.26.tar.gz and nss-3.26-with-nspr-4.12.tar.gz) and two files SHA1SUMS and SHA256SUMS in /your/path/to/stage/NSS_3_26_RTM/src/.

   $ python automation/release/nss-release-helper.py create_nss_release_archive 3.26 NSS_3_26_RTM ../stage

Some additional check one may want to perform

This section is only for the paranoid, only in case you want to double-check that the created file is correct.

1. Extract the archive
2. Check that all files inside the directory use the following two prefix dirs:

   nss-3.26/nss/

   nss-3.26/nspr/

3. Check that file nss-3.26/nss/.hg_archival.txt mentions tag: NSS_3_26_RTM
4. You could double-check that the three version number files have all _BETA flags set to PR_FALSE and no version string line _VERSION contains Beta.
5. Remove the extracted directory, to ensure you won't upload the individual files.

Upload release to public ftp/google cloud.

When running the following command ensure that you use the directory name NSS_3_26_RTM twice, both in source and in destination.

$ cd /your/path/to/stage
$ find NSS_3_26_RTM

expected output:

NSS_3_26_RTM/
NSS_3_26_RTM/src
NSS_3_26_RTM/src/nss-3.26.tar.gz
NSS_3_26_RTM/src/nss-3.26-with-nspr-4.12.tar.gz
NSS_3_26_RTM/src/SHA256SUMS
NSS_3_26_RTM/src/SHA1SUMS

1. Check that there's no NSS_3_26_RTM directory at https://ftp.mozilla.org/pub/security/nss/releases/.
2. Ensure you have the gcloud command installed.
3. (see also: https://cloud.google.com/sdk/docs/install )
4. Upload the release

   $ NSS_RELEASE=NSS_3_26_RTM \
   GCP_PROJECT=moz-fx-productdelivery-pr-38b5 \
   GCP_ENV=prod \
   gcloud --project $GCP_PROJECT --impersonate-service-account=nss-team-$GCP_ENV@$GCP_PROJECT.iam.gserviceaccount.com \
   storage cp --recursive --no-clobber $NSS_RELEASE gs://$GCP_PROJECT-productdelivery/pub/security/nss/releases/

5. Check https://ftp.mozilla.org/pub/security/nss/releases/ (the indexing of the new directory can be delayed but it can be accessed immediately through the url.)
6. Check upload:

   $ gcloud storage ls gs://moz-fx-productdelivery-pr-38b5-productdelivery/pub/security/nss/releases/

   $ gcloud storage ls gs://moz-fx-productdelivery-pr-38b5-productdelivery/pub/security/nss/releases/NSS_3_26_RTM/

   $ gcloud storage ls gs://moz-fx-productdelivery-pr-38b5-productdelivery/pub/security/nss/releases/NSS_3_26_RTM/src/

Upgrading mozilla-central (automated)

1. RelEng to Uplift the new RTM tag using the nss-uplift scripts.

   Amend the uplift commit to set old-configure.in file in mozilla-central to point AM_PATH_NSS to the new minimum version of NSS

Example:

PATH=$PATH:/path/to/xpcshell LD_LIBRARY_PATH=/path/to/xpcshell NOBUILD=: /path/to/nss-tools/nss-uplift-unified.sh NSS_X_YY_RTM

Upgrading mozilla-central (manual instructions)

1. If you already upgraded mozilla-central with the nss-uplift-unified.sh script, you can ignore these manual instructions.
2. Someone should approve the upgrade.
3. For the upgrade task, we have docs https://developer.mozilla.org/en/docs/Updating_NSPR_or_NSS_in_mozilla-central Go to a mozilla-inbound checkout and run the following.

   $ python client.py update_nss NSS_3_26_RTM

   $ hg commit -m "Bug 1216283, land NSS_3_26_RTM, UPGRADE_NSS_RELEASE, r=the-reviewer-name"

   $ hg push

4. If we must land into branches, like aurora or beta, we must get approvals, etc. Note that we don't do patches for branch landing. We use a placeholder attachment

with the command to be executed. The reason is, that the state of those dependency files might be different between branches (where we upgrade or remove newlines as necessary, to trigger full rebuilds). By using the upgrade script, this is handled automatically.

Logistics

1. Update the NSS:Release Versions page.
2. Update the MDN NSS Release page.