QA/Desktop Firefox/Releases: Difference between revisions

Revision as of 22:42, 20 December 2013

Regression	Versions	Manual Coverage	Automation Coverage
bug 848663 Can’t update from FF19.0 to FF19.0.1	Firefox 19+	testcase	Nominated: bug 853147
bug 834667 Playing youtube video in html5 starts to leak memory	Firefox 19+	testcase	Nominated: bug 851502
bug 839548 Excessive memory while loading PDF thumbnail view	Pending bug fix	Pending bug fix	Nominated: bug 851507
bug 789958 Verify update applies with root-owned files	Firefox 15+	testcase	n/a - not automatable
bug 789422 What's New page on Firefox update	Firefox 16+	testcase	nominated
bug 650355: Stop accepting md5 hash algorithm	Firefox 16+	testcase	nominated
Mac OSX 10.8 code-signing	Firefox 17+	testcase	?
Breakpad end-to-end functionality	Firefox 17+	testcase	?
Mac OSX 32-bit mode compatibility	Firefox 17+	testcase	?
Mac OSX 10.5 De-support	Firefox 17+	testcase	?
bug 833823 Random, unpredictable behavior with multiple YouTube embeds	Firefox 20+	?	nominated
bug 787743 Private Browsing data does not cache	Firefox 15+	testcase	testcase

Goal:

Try to identify valuable metrics in Release so we can evaluate Nightly and develop smarter strategies for Aurora
Value for Engineering is identifying problem areas earlier

New metrics to gather:

Stop gather metrics on:

Firefox 23.0.1:
- bug 901944 caused by bug 847267 which landed on mozilla-central on 2013-05-05 and rode the trains
- bug 902532 caused by bug 857830 which landed on mozilla-central on 2013-05-07 and rode the trains
- bug 901527 caused by bug 886886 which landed on mozilla-central on 2013-07-22 and was uplifted to mozilla-beta and mozilla-aurora after only 5 hours on Nightly
Firefox 20.0.1:
- bug 857291 caused by bug 807678 which landed on mozilla-central on 2012-12-24 and rode the trains
- bug 846848 caused by OS.File not being on QA's radar as it rode the trains

The following shows when a bug was fixed (based on the state of the status flag during the 6-week period).

Uplifts account for ~66% of all fixes going into a given release
All versions requiring at least one point release occurred when uplifts accounted for 55% or more of the fixes
All versions requiring at least two point releases occurred when uplifts accounted for ~60% or more of the fixes
Hypothesis: volume of landings is increasing instability (ie. likelihood of needing a follow-up release)

@@ Line 265: / Line 265: @@
 === Crashes by Version ===
+{| class="querytable"
+ | class="header" | Version
+ | class="header" | Reported
+ | class="header" | Fixed
+ | class="header" | Dropped
+ | class="header" | Ever Reopened
+ | class="header" | Chemspills
+ |-
+ | Firefox 26
+ | 75
+ | 26
+ | 2
+ | 2
+ | 0
+ |-
+ | Firefox 25
+ | 101
+ | 35
+ | 0
+ | 2
+ | 1
+ |-
+ | Firefox 24
+ | 143
+ | 52
+ | 3
+ | 4
+ | 0
+ |-
+ | Firefox 23
+ | 122
+ | 45
+ | 11
+ | 6
+ | 1
+ |-
+ | Firefox 22
+ | 155
+ | 40
+ | 9
+ | 7
+ | 0
+ |-
+ | Firefox 21
+ | 93
+ | 20
+ | 11
+ | 5
+ | 1
+ |-
+ | Firefox 20
+ | 123
+ | 42
+ | 16
+ | 5
+ | 1
+ |-
+ | Firefox 19
+ | 145
+ | 45
+ | 15
+ | 11
+ | 2
+ |-
+ | Firefox 18
+ | 197
+ | 63
+ | 10
+ | 11
+ | 2
+ |-
+ | Firefox 17
+ | 161
+ | 53
+ | 10
+ | 11
+ | 1
+ |-
+ | Firefox 16
+ | 141
+ | 37
+ | 16
+ | 1
+ | 2
+ |-
+ | Firefox 15
+ | 218
+ | 65
+ | 10
+ | 14
+ | 1
+ |-
+ | Firefox 14
+ | 117
+ | 36
+ | 26
+ | 7
+ | 1
+ |-
+ | Firefox 13
+ | 130
+ | 42
+ | 13
+ | 9
+ | 1
+ |-
+ | Firefox 12
+ | 93
+ | 15
+ | 7
+ | 3
+ | 0
+ |-
+ | Firefox 11
+ | 104
+ | 22
+ | 8
+ | 10
+ | 0
+ |-
+ | Firefox 10
+ | 194
+ | 23
+ | 23
+ | 6
+ | 1
+ |-
+ | Firefox 9
+ | 114
+ | 17
+ | 16
+ | 5
+ | 1
+ |-
+ | Firefox 8
+ | 82
+ | 12
+ | 8
+ | 4
+ | 1
+ |-
+ | Firefox 7
+ | 69
+ | 10
+ | 8
+ | 1
+ | 1
+ |-
+ | Firefox 6
+ | 73
+ | 5
+ | 15
+ | 1
+ | 2
+ |-
+ | Firefox 5
+ | 109
+ | 14
+ | 22
+ | 1
+ | 1
+ |}
 == Effect of Uplifts ==