Releases/Firefox 58/Test Plan

Summary

This page is to track testing of Firefox 57 from mozilla-central (Nightly) through to mozilla-release. This is intended to be a minimal set of information to help Firefox QA understand and communicate the quality of upcoming releases. Our current walkthrough of the release and QA process might be changing due to recent reorgs.

The QA efforts are mostly split up between two teams:

• Engineering QA
  • team reports to Lawrence Mandel
  • the Project Manager (and main contact) for the team is Rares Bologa
  • team is based in Romania (most part) and Las Vegas (5 people)
  • the team mainly tests the Desktop app, but covers the Mobile (Android, iOS) side as well
  • main responsibilities:
    • Feature Testing - new Firefox features, prior to landing and during Nightly
    • Bug Triage - older as well as incoming Unconfirmed bugs
    • Bug Investigation - various investigation as requested by other people (e.g. finding steps, setting component, finding regression windows)
    • Misc - other miscellaneous requests
• Release QA Desktop and Release QA Mobile
  • team reports to Lawrence Mandel
  • the Project Manager (and main contact) for the team is Florin Mezei
  • team is based in Romania
  • main responsibilities:
    • Build Validation - mostly Beta, Release, and ESR builds (including automated ondemand update, and throttling tests)
    • Feature Testing - new features not covered by the Engineering QA team, and those covered by the Engineering QA team after they enter Beta
    • Bug Verification - verifying the most important / high risk fixes, with focus on those closer to release
    • Bug Investigation - various investigation as requested by other people (e.g. finding steps, setting component, finding regression windows)
    • Misc - other miscellaneous requests

Schedule

Milestone	Date	Checks
Nightly	2017-11-14	Conduct daily bug triage, regular build validation, and feature testing
Beta	2017-01-08	Feature testing and sign off every beta release, regular testing of DevEdition builds
Release	2018-01-16	Review tests and sign off release

Please consult the Rapid Release Calendar for more information.

Meetings

Meeting	Purpose	When	Vidyo Room	IRC Backchannel	Notes
Channel Status	Raise quality concerns with the Release Management team	10:00 AM Pacific on Tuesday & 08:00 AM Pacific on Thursday	ReleaseCoordination	#planning	Notes archive

Nightly

While Firefox 58 is on the Nightly channel, QA will mainly focus on:

• Feature Testing - EngQA team (for Firefox features assigned to them) + RelQA team (for other new features - Platform or Firefox)
• Regular Nightly validation - 3 rounds per cycle - EngQA team + RelQA team, alternately
• Triaging newly reported bugs - EngQA team
• Bug Verification - both teams - with main focus on: fixes related to new features, fixes tracked as Sprint work, fixes for major bugs

Reference: https://wiki.mozilla.org/QA/Desktop_Firefox/Walkthroughs/Release_Coordination#Nightly

End of cycle status

• Total bugs filed affecting 57: [TBD]
• Bugs fixed / verified on 57: [TBD] / [TBD]
• Conditional signoff for merge: <date>, <person> (email sent to release-drivers at )
• QE signoff: N/A

Beta

During the Beta cycle, there are several beta releases. Communication with the release coordination, release engineering, and automation teams is key.

While Firefox 58 is on the Beta channel, QA will mainly focus on:

• Signing off each Beta build prior to being pushed to the beta channel - RelQA
• Feature Testing - RelQA team
  • Focus is on fixes that land in Beta, and Regression and Smoke testing
• Regular validation of DevEdition builds - about once every two weeks
• Triaging newly reported bugs - EngQA team
• Bug Verification - mostly RelQA - with main focus on: fixes related to new features, fixes for major bugs, fixes assessed as high risk

QA also analyzes the results of OnDemand Update tests running on Jenkins. We create config files for the update tests, check their validity, and run them on at least two channels for each beta release cycle.

Pivotal bugs may need verification during each beta cycle.

• Link to release notes. review & update.
• Crash evaluation/escalation.
• Feature evaluation.

QA Activities

QA wanted

• QA Needed Bugs

The following tracks bugs which have requests for urgent QA assistance.

• Choose a bug to test from the list below
• Read it and make sure you understand what information is needed (this may be steps or URLs to reproduce, a regression range, or something else altogether)
• Use the latest build of this Firefox version and test until you've discovered the information necessary. Ask for help if you need it.
• Report this information to the bug and remove the qawanted keyword

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Stability

This section tracks bugs representing the most urgent stability issues. Consult this wiki for more information.

• Ensure new and explosive crashes have actionable bug reports which are in progress
• Top Crashes - ensure the highest ranked crashes have actionable bug reports which are in progress
• Untriaged Bugs - ensure bugs are actionable and in the correct component
• topcrash bugs

Unconfirmed Bug Triage

The following tracks triage and testing of incoming, unconfirmed bug reports.

• Choose a bug to test from the list below
• Use the latest Nightly build to confirm the bug reproduces
• Flag with reporter with needinfo if you cannot reproduce and need more information
• Set the status to NEW and ensure the bug is in the appropriate component if you can reproduce the bug (provide as much detail about your testing as possible)
• If it's a Firefox bug you may want to add the firefox-backlog=? flag to it.

Queries:

• All Unconfirmed Bugs
  • https://bugzilla.mozilla.org/buglist.cgi?bug_status=UNCONFIRMED&version=58%20Branch
• Unchanged Since One Week
  • These bugs may need extra attention.
  • https://bugzilla.mozilla.org/buglist.cgi?bug_status=UNCONFIRMED&version=58%20Branch&chfieldto=-1w
• Unchanged Since One Month
  • These are likely to be confusing or non-reproducible issues. We should make decisions to either close these or tag these non-actionable, and filter them out
  • https://bugzilla.mozilla.org/buglist.cgi?bug_status=UNCONFIRMED&version=58%20Branch&chfieldto=-1m

Verification Needed Bugs

• Review the bugs below in the Pending Triage section and select a bug you want to test
• Make sure you understand the bug before you begin testing it
• Test on a previously known broken build to make sure you can reproduce the bug
• Test on the latest build from the branches which are fixed to confirm the bug no longer reproduces
• Mark the bug VERIFIED FIXED if the bug appears to be fixed and set the appropriate status-firefox flag to verified
• Reopen the bug if it still reproduces

Iteration Development

Firefox development during Nightly works on two-week iterations as of 2014. This section tracks the bugs selected for development in these iterations. Please consult the Iteration Dev Walkthrough and the Firefox/IterativeDevelopment page for more information on this process.

• Ensure all bugs are marked qe-verify:+ or -
• Ensure all qe-verify:+ bugs have a QA Contact (The Feature Owner page may help here)
• Ensure all selected bugs are VERIFIED before the end of the iteration
• Anything incomplete at the end of the iteration will be automatically carried over to the next iteration

Queries:

• Dashboard showing current active iteration: https://crash-analysis.mozilla.com/rkaiser/crash-report-tools/qa/
• Iteration 58.1, all bugs: https://bugzilla.mozilla.org/buglist.cgi?f1=OP&j1=OR&f2=cf_fx_iteration&o2=substring&v2=58.1&f3=status_whiteboard&o3=substring&v3=s%3D58.1&f4=CP
• Iteration 58.2, all bugs: https://bugzilla.mozilla.org/buglist.cgi?f1=OP&j1=OR&f2=cf_fx_iteration&o2=substring&v2=58.2&f3=status_whiteboard&o3=substring&v3=s%3D58.2&f4=CP
• Iteration 58.3, all bugs: https://bugzilla.mozilla.org/buglist.cgi?f1=OP&j1=OR&f2=cf_fx_iteration&o2=substring&v2=58.3&f3=status_whiteboard&o3=substring&v3=s%3D58.3&f4=CP

Milestones

Regular Nightly Validation

SIGN-OFF DATE	SIGN-OFF STATUS	REASON(S)	MAJOR ISSUES	MINOR ISSUES	TEST REPORT
2017-10-06	GREEN	6 new bugs uncovered, none of them blockers: Bugzilla link. Known (old) bugs encountered are low-impact.	---	Bugzilla link	TestRail link
2017-10-19	GREEN	3 new bugs uncovered, none of them blockers: bug 1409673, bug 1409768, bug 1410009.	---	Bugzilla link	TestRail link
2018-11-02	GREEN	7 new bugs uncovered, none of them high-impact: Bugzilla link. Known (old) bugs encountered are low-impact.	---	Bugzilla link	TestRail link

Nightly Migration

Pre-sign-off Checks:

• [NOT STARTED] Manual Smoke testing - ensure there are no major issues blocking release to the beta channel
  • Areas covered: TBD
  • Signed off by <name> on <date>
    • Detailed test results: in TestRail (requires access rights) or as a pdf.
• [NOT STARTED] Features - all scoped features are signed off as ready for Beta by owners. (If you haven't heard anything to the contrary, that is probably true, but make sure we don't know of blocking issues.)

Post-sign-off Checks (as soon as updates are enabled again):

• None yet for migration of Nightly to Beta

Developer Edition

Note: Developer Edition builds are usually signed off once every two weeks.

BUILD	SIGN OFF			ENCOUNTERED BUGS
	Status	Reason(s)	Test Report	New	Known (old)
58.0b1-build1	GREEN	No new issues uncovered. Known issues are low-impact.	TestRail link	Regressions: none. Non-regressions: none.	High-impact: none. Low-impact: none.
58.0b2-build1	GREEN	No new issues uncovered. Known issues are low-impact.	TestRail link	Regressions: none. Non-regressions: none.	High-impact: none. Low-impact: Bugzilla link.
58.0b7-build1	GREEN	No new issues uncovered. Known issues are low-impact.	TestRail link	Regressions: none. Non-regressions: none.	High-impact: none. Low-impact: Bugzilla link.
58.0b11-build1	GREEN	No new issues uncovered. Known issues are low-impact.	TestRail link	Regressions: none. Non-regressions: none.	High-impact: none. Low-impact: Bugzilla link.
58.0b13-build1	GREEN	One new issue uncovered during testing: 1427735. Known issues are low-impact.	TestRail link	Regressions: 1427735. Non-regressions: none.	High-impact: none. Low-impact: Bugzilla link.

Beta

Note: Beta builds are usually signed off twice a week.

Beta 3

Build Information

• Build 1: 58.0b3-build2 | f7eb3bbcd5f8 | (ID: 20171114032831)

Testing

• Manual - Firefox 58 Beta 3
  • 58.0b3-build2
    • [DONE] Smoke and Exploratory Testing with focus on: Crash reporter, Session restore and Media playback & support.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel (Linux report, Windows & Mac report
  • [NOT STARTED] Updates on beta channel (Linux report, Windows & Mac report

Beta 4

Build Information

• Build 1: 58.0b4-build1 | 03402f586daa | (ID: TBD)

Testing

• Manual - Firefox 58 Beta 3
  • 58.0b3-build2
    • [NOT STARTED] Smoke and Exploratory Testing with focus on: TBD.
      • Detailed test results: in TestRail.
    • [NOT STARTED] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [NOT STARTED] Updates on beta-cdntest channel (Linux report, Windows & Mac report
  • [NOT STARTED] Updates on beta channel (Linux report, Windows & Mac report

Beta 5

Build Information

• Build 1: 58.0b5-build1 | f155e109bb41 | (ID: 20171120142222)

Testing

• Manual - Firefox 58 Beta 5
  • 58.0b5-build1
    • [DONE] Smoke and Exploratory Testing with focus on: Customization and Forms Autofill.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel Linux report, Windows & Mac report
  • [DONE] Updates on beta channel Linux report, Windows & Mac report

Beta 6

Build Information

• Build 1: 58.0b6-build1 | a38e0adbe442 | (ID: 20171123161455)

Testing

• Manual - Firefox 58 Beta 6
  • 58.0b6-build1
    • [DONE] Smoke and Exploratory Testing with focus on: Search, Migration, Geolocation and Mac OS X signing.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel Linux report, Windows & Mac report
  • [DONE] Updates on beta channel Linux report, Windows & Mac report

Beta 7

Build Information

• Build 1: 58.0b7-build1 | e8f054b3ac68 | (ID: 20171127135700)

Testing

• Manual - Firefox 58 Beta 7
  • 58.0b7-build1
    • [DONE] Smoke and Exploratory Testing with focus on: Drag and Drop, Printing.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel Linux report, Windows & Mac report
  • [DONE] Updates on beta channel Linux report, Windows & Mac report

Beta 8

Build Information

• Build 1: 58.0b8-build1 | 27dc691eb4a0 | (ID: 20171130160223)

Testing

• Manual - Firefox 58 Beta 8
  • 58.0b8-build1
    • [DONE] Smoke and Exploratory Testing with focus on: Search, PDF Viewer.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel Linux report, Windows & Mac report
  • [DONE] Updates on beta channel Linux report, Windows & Mac report

Beta 9

Build Information

• Build 1: 58.0b9-build1 | [1] | (ID: 20171204150510)

Testing

• Manual - Firefox 58 Beta 9
  • 58.0b9-build1
    • [DONE] Smoke and Exploratory Testing with focus on: Graphics, Downloads and Download Manager.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel report
  • [DONE] Updates on beta channel report

Beta 10

Build Information

• Build 1: 58.0b10-build1 | 4fba9ec9d4ba | (ID: 20171207170405)

Testing

• Manual - Firefox 58 Beta 10
  • 58.0b10-build1
    • [DONE] Smoke and Exploratory Testing with focus on: Add-on compatibility, Safe browsing and bookmarks.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel Linux report, Windows & Mac report
  • [DONE] Updates on beta channel Linux report, Windows & Mac report

Beta 11

Build Information

• Build 1: 58.0b11-build1 | d7ddb4b0dc03 | (ID: 20171211020921)

Testing

• Manual - Firefox 58 Beta 11
  • 58.0b11-build1
    • [DONE] Smoke and Exploratory Testing with focus on: Plug-in compatibility and Srcolling and zooming.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel Linux report, Windows & Mac report
  • [DONE] Updates on beta channel Linux report, Windows & Mac report

Beta 12

Build Information

• Build 1: 58.0b12-build1 | a59d9203232d | (ID: 20171218174357)

Testing

• Manual - Firefox 58 Beta 12
  • 58.0b12-build1
    • [DONE] Smoke and Exploratory Testing with focus on: Windows Child Mode, Password Manager and Image support.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel Linux report, Windows & Mac report
  • [NOT STARTED] Updates on beta channel Linux report, Windows & Mac report

Beta 13

Build Information

• Build 1: 58.0b13-build1 | cb9a0bfe28ea | (ID: 20171218174357)

Testing

• Manual - Firefox 58 Beta 13
  • 58.0b13-build1
    • [DONE] Smoke and Exploratory Testing with focus on: Media playback and support.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel Linux report, Windows & Mac report
  • [DONE] Updates on beta channel Linux report, Windows & Mac report

Beta 14

Build Information

• Build 2: 58.0b14-build2 | 26b83e9df201 | (ID: 20180103230655)

Testing

• Manual - Firefox 58 Beta 14
  • 58.0b14-build2
    • [DONE] Smoke and Exploratory Testing with focus on: Customization and Web compatibility.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel Linux report, Windows & Mac report
  • [DONE] Updates on beta channel Linux report, Windows & Mac report

Beta 15

Build Information

• Build 1: 58.0b15-build1 | 60e4ebb5f150 | (ID: 20180108140638)

Testing

• Manual - Firefox 58 Beta 15
  • 58.0b15-build1
    • [DONE] Smoke and Exploratory Testing with focus on: Activity Stream and Graphics.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel Linux report, Windows & Mac report
  • [DONE] Updates on beta channel Linux report, Windows & Mac report

Beta 16

Build Information

• Build 1: 58.0b16-build1 | 8bb8f895a740 | (ID: 20180111124119)

Testing

• Manual - Firefox 58 Beta 16
  • 58.0b15-build1
    • [DONE] Smoke and Exploratory Testing with focus on: Printing, Smart Card feature and PDF Viewer.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel Linux report, Windows & Mac report
  • [NOT STARTED] Updates on beta channel Linux report, Windows & Mac report

58.0 Release Candidate

Build Information

• Build 2: 58.0-build2 | 4c230b89e05e | (ID: 20180115093319)

Testing

• Manual - Firefox 58 Release candidate
  • 58.0-build2
    • [DONE] Smoke and Exploratory Testing with focus on: Web Compat.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel report
  • [DONE] Updates on beta channel report
  • [NOT STARTED] Updates on release-cdntest channel report
  • [NOT STARTED] Updates on release channel report

58.0 Release Candidate 2

Build Information

• Build 6: 58.0-build6 | 40755aa80f41 | (ID: 20180118215408)

Testing

• Manual - Firefox 58 Release candidate 2
  • 58.0-build6
    • [DONE] Smoke and Exploratory Testing with focus on: Web Compat.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Automated (configs for ondemand)
  • [DONE] Updates on beta-cdntest channel report
  • [NOT STARTED] Updates on beta channel report
  • [NOT STARTED] Updates on release-cdntest channel [TBD report]
  • [NOT STARTED] Updates on release channel [TBD report]

58.0.1

Build Information

• Build 1: 58.0.1-build1 | c2db4a50dc5c | (ID: 20180128191252)

Testing

• Manual - Firefox 58.0.1
  • 58.0.1-build1
    • [DONE] Smoke and Exploratory Testing with focus on: Web Compat.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Updates
  • [DONE] Updates on release-cdntest channel report
  • [DONE] Updates on release channel report

58.0.2

Build Information

• Build 1: 58.0.2-build1 | 849c090094db | (ID: 20180206200532)

Testing

• Manual - Firefox 58.0.2
  • 58.0.2-build1
    • [DONE] Smoke and Exploratory Testing with focus on: Web Compat.
      • Detailed test results: in TestRail.
    • [DONE] Bug verification with focus on the latest fixes to land in this build.
• Updates
  • [DONE] Updates on release-cdntest channel report
  • [DONE] Updates on release channel report