Release Management/Release Process Metrics
This page describes the release process/quality metrics gathered during the 68 cycle, and how.
Each week, grab a snapshot of the following data points in the tracking spreadsheet.
Usage numbers
Nightly and Beta ADI
The numbers are a 30-day moving average. There's a couple of days of latency before they show up but you can grab the latest available and be close enough.
Go to https://dataviz.mozilla.org/#/views/PlatformVersionFirefoxADI/DesktopADIbyPlatform?:iid=1 and filter on channel
Stability
Mission Control crash rates
On https://missioncontrol.telemetry.mozilla.org/#/?channel=nightly or https://missioncontrol.telemetry.mozilla.org/#/?channel=beta, get the "Firefox summary score", the overall scores for 3 desktop platforms, and the "Fennec summary score".
Number of backouts from central
Count changesets that landed while 68 was in nightly, whose commit message matches what `hg qbackout` generates, and that landed as their own push to mozilla-central (i.e. not combined with other changes like a merge from integration). The query will undercount if there are cases where the commit message is different and/or where the backout didn't land directly on m-c. This might be acceptable as those corner cases, where developers land backouts on integration, are typically not the ones where nightly is broken enough that we need an immediate backout and respin.
This requires:
- an up-to-date mozilla-central clone
- the pushlog and mozext hg extensions from version-control-tools
- run the following command, adjusting for version numbers:
hg log -r 'FIREFOX_NIGHTLY_67_END::(present(FIREFOX_NIGHTLY_68_END) or central) and pushhead()' --template "{if(backsoutnodes, ifeq(node, pushbasenode, '{node|short} {desc|firstline}\n', ''))}" | wc -l
Features
Number of planned features
Counts the number of cards in the trello list for the release of interest.
Requires a trello API key and token from https://trello.com/app-key, the Firefox trello board id and the id of the list corresponding to your release.
#!python3 import requests key = '....' token = '...' board = '...' listid = '...' url = 'https://api.trello.com/1/lists/{idList}/cards?key={yourKey}&token={yourToken}'.format(idList=listid, idBoard=board, yourKey=key, yourToken=token) r = requests.get(url) print(len(r.json())
Number of disabled/deferred features
Keep track (manually :( ) of features that get added and removed from a given release.
Regressions
Number of new regressions
For 68 these queries were run anonymously so didn't include security or confidential bugs.
New regressions in nightly
This counts bugs:
- filed during the Firefox X nightly cycle
- tagged as regressions (at the time of the query)
- either unresolved or resolved as FIXED (to filter out duplicate / invalid / ...) (at the time of the query)
- where the status for Firefox X-1 is either "unaffected", "disabled" or unset (at the time of the query)
- where the status for Firefox X is *not* "unaffected", "disabled" or unset (at the time of the query)
Excluding "disabled" on the previous version means the bug is a regression from a user PoV even though the bug may have been present in the code beforehand.
https://bugzilla.mozilla.org/rest/bug?count_only=1&keywords=regression&o1=anyexact&o2=notequals&chfield=%5BBug%20creation%5D&chfieldfrom=2019-03-18&o4=notequals&v1=disabled%2Cunaffected%2C---&v2=disabled&v4=---&f1=cf_status_firefox67&o3=notequals&v3=unaffected&resolution=---&resolution=FIXED&f4=cf_status_firefox68&chfieldto=2019-05-20&f3=cf_status_firefox68&f2=cf_status_firefox68
New regressions in beta
This counts bugs:
- filed during the Firefox X beta cycle
- tagged as regressions
- either unresolved or resolved as FIXED (to filter out duplicate / invalid / ...)
- where the status for Firefox X-1 is either "unaffected", "disabled" or unset
- where the status for Firefox X is *not* "unaffected", "disabled" or unset
https://bugzilla.mozilla.org/rest/bug?count_only=1&keywords=regression&o1=anyexact&o2=notequals&chfield=%5BBug%20creation%5D&chfieldfrom=2019-05-20&o4=notequals&v1=disabled%2Cunaffected%2C---&v2=disabled&v4=---&f1=cf_status_firefox67&o3=notequals&v3=unaffected&resolution=---&resolution=FIXED&f4=cf_status_firefox68&chfieldto=2019-07-08&f3=cf_status_firefox68&f2=cf_status_firefox68
Unfixed new regressions
This counts bugs:
- filed during the Firefox X nightly or beta cycle
- tagged as regressions
- either unresolved or resolved as FIXED
- where the status for Firefox X-1 is either "unaffected", "disabled" or unset
- where the status for Firefox X is either "affected", "fix-optional" or "wontfix"
https://bugzilla.mozilla.org/rest/bug?count_only=1&keywords=regression&chfield=%5BBug%20creation%5D&chfieldfrom=2019-03-18&chfieldto=2019-07-08&o1=anyexact&f1=cf_status_firefox67&v1=disabled%2Cunaffected%2C---&o2=anyexact&f2=cf_status_firefox68&v2=affected%2Cfix-optional%2Cwontfix&resolution=---&resolution=FIXED
Adjust version numbers and dates as needed.
Beta uplifts
Count the number of bugs uplifted to beta. The below query looks at changesets on beta after the NIGHTLY_END tag, filters out test-only, npotb and releng changes, and changesets that don't refer to a bug.
Requires an up-to-date mozilla-unified clone. Replace "beta" with "release" during RC week.
#!python3 import subprocess mu = ##PATH_TO_MOZILLA_UNIFIED## v = ##VERSION## uplifts = subprocess.check_output(['hg', 'log', '-r', '(present(FIREFOX_NIGHTLY_%d_END) or central)::(present(FIREFOX_BETA_%d_END) or beta) and desc("re:(?i)^bug")' % (v, v), '--template', '{desc|firstline}\\n'], cwd=mu) bugs = set(line.split()[1] for line in uplifts.splitlines() if b'a=release' not in line.lower() and b'a=test' not in line.lower() and b'a=npotb' not in line.lower()) print(len(bugs))