Release Management/Release Process Metrics

From MozillaWiki
Jump to: navigation, search

This page describes the release process/quality metrics gathered during the 68 cycle, and how.

Each week, grab a snapshot of the following data points in the tracking spreadsheet.

Usage numbers

Nightly and Beta ADI

The numbers are a 30-day moving average. There's a couple of days of latency before they show up but you can grab the latest available and be close enough.

Go to https://dataviz.mozilla.org/#/views/PlatformVersionFirefoxADI/DesktopADIbyPlatform?:iid=1 and filter on channel

Stability

Mission Control crash rates

On https://missioncontrol.telemetry.mozilla.org/#/?channel=nightly or https://missioncontrol.telemetry.mozilla.org/#/?channel=beta, get the "Firefox summary score", the overall scores for 3 desktop platforms, and the "Fennec summary score".

Number of backouts from central

Count changesets that landed while 68 was in nightly, whose commit message matches what `hg qbackout` generates, and that landed as their own push to mozilla-central (i.e. not combined with other changes like a merge from integration). The query will undercount if there are cases where the commit message is different and/or where the backout didn't land directly on m-c. This might be acceptable as those corner cases, where developers land backouts on integration, are typically not the ones where nightly is broken enough that we need an immediate backout and respin.

This requires:

  • an up-to-date mozilla-central clone
  • the pushlog and mozext hg extensions from version-control-tools
  • run the following command, adjusting for version numbers:
hg log -r 'FIREFOX_NIGHTLY_67_END::(present(FIREFOX_NIGHTLY_68_END) or central) and pushhead()' --template "{if(backsoutnodes, ifeq(node, pushbasenode, '{node|short} {desc|firstline}\n', ''))}" | wc -l

Features

Number of planned features

Counts the number of cards in the trello list for the release of interest.

Requires a trello API key and token from https://trello.com/app-key, the Firefox trello board id and the id of the list corresponding to your release. 

#!python3
import requests
key = '....'
token = '...'
board = '...'
listid = '...'
url = 'https://api.trello.com/1/lists/{idList}/cards?key={yourKey}&token={yourToken}'.format(idList=listid, idBoard=board, yourKey=key, yourToken=token)
r = requests.get(url)
print(len(r.json())

Number of disabled/deferred features

Keep track (manually :( ) of features that get added and removed from a given release.

Regressions

Number of new regressions

For 68 these queries were run anonymously so didn't include security or confidential bugs.

New regressions in nightly

This counts bugs:

  • filed during the Firefox X nightly cycle
  • tagged as regressions (at the time of the query)
  • either unresolved or resolved as FIXED (to filter out duplicate / invalid / ...) (at the time of the query)
  • where the status for Firefox X-1 is either "unaffected", "disabled" or unset (at the time of the query)
  • where the status for Firefox X is *not* "unaffected", "disabled" or unset (at the time of the query)

Excluding "disabled" on the previous version means the bug is a regression from a user PoV even though the bug may have been present in the code beforehand. 

https://bugzilla.mozilla.org/rest/bug?count_only=1&keywords=regression&o1=anyexact&o2=notequals&chfield=%5BBug%20creation%5D&chfieldfrom=2019-03-18&o4=notequals&v1=disabled%2Cunaffected%2C---&v2=disabled&v4=---&f1=cf_status_firefox67&o3=notequals&v3=unaffected&resolution=---&resolution=FIXED&f4=cf_status_firefox68&chfieldto=2019-05-20&f3=cf_status_firefox68&f2=cf_status_firefox68

New regressions in beta

This counts bugs:

  • filed during the Firefox X beta cycle
  • tagged as regressions
  • either unresolved or resolved as FIXED (to filter out duplicate / invalid / ...)
  • where the status for Firefox X-1 is either "unaffected", "disabled" or unset
  • where the status for Firefox X is *not* "unaffected", "disabled" or unset
https://bugzilla.mozilla.org/rest/bug?count_only=1&keywords=regression&o1=anyexact&o2=notequals&chfield=%5BBug%20creation%5D&chfieldfrom=2019-05-20&o4=notequals&v1=disabled%2Cunaffected%2C---&v2=disabled&v4=---&f1=cf_status_firefox67&o3=notequals&v3=unaffected&resolution=---&resolution=FIXED&f4=cf_status_firefox68&chfieldto=2019-07-08&f3=cf_status_firefox68&f2=cf_status_firefox68

Unfixed new regressions

This counts bugs:

  • filed during the Firefox X nightly or beta cycle
  • tagged as regressions
  • either unresolved or resolved as FIXED
  • where the status for Firefox X-1 is either "unaffected", "disabled" or unset
  • where the status for Firefox X is either "affected", "fix-optional" or "wontfix"
https://bugzilla.mozilla.org/rest/bug?count_only=1&keywords=regression&chfield=%5BBug%20creation%5D&chfieldfrom=2019-03-18&chfieldto=2019-07-08&o1=anyexact&f1=cf_status_firefox67&v1=disabled%2Cunaffected%2C---&o2=anyexact&f2=cf_status_firefox68&v2=affected%2Cfix-optional%2Cwontfix&resolution=---&resolution=FIXED

Adjust version numbers and dates as needed.

Beta uplifts

Count the number of bugs uplifted to beta. The below query looks at changesets on beta after the NIGHTLY_END tag, filters out test-only, npotb and releng changes, and changesets that don't refer to a bug.

Requires an up-to-date mozilla-unified clone. Replace "beta" with "release" during RC week. 

#!python3
import subprocess
mu = ##PATH_TO_MOZILLA_UNIFIED##
v = ##VERSION##
uplifts = subprocess.check_output(['hg', 'log',
    '-r', '(present(FIREFOX_NIGHTLY_%d_END) or central)::(present(FIREFOX_BETA_%d_END) or beta) and desc("re:(?i)^bug")' % (v, v),
    '--template', '{desc|firstline}\\n'], cwd=mu)
bugs = set(line.split()[1] for line in uplifts.splitlines()
           if b'a=release' not in line.lower() and
              b'a=test' not in line.lower() and
              b'a=npotb' not in line.lower())
print(len(bugs))
Retrieved from "https://wiki.mozilla.org/index.php?title=Release_Management/Release_Process_Metrics&oldid=1214890"