QA/E10s Test Plan

From MozillaWiki
< QA
Jump to: navigation, search

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
03/02/2015 1.0 Brindusa Tot Created first draft

Overview

Scope

This is the general QA Test Plan for Electrolysis ("e10s" for short). With e10s enabled, Firefox content runs in a different process than the browser itself. This will help to improve general performance and security. Our goals of testing are to identify areas of concerns and potential risk and to be able to give a recommendation when e10s will be able to be pushed to a wider audience (release). The goal of the current wiki page is to gather the testing efforts done in support of the Electrolysis project. It defines the overall testing requirements and provides an integrated view of the test activities.

Enabling and Disabling Electrolysis
On current Nightly and Aurora versions, e10s is enabled by default. To disable e10s, open Preferences->General and un-check the "Enable multi-process" check-box and then restart the browser.

Ownership

Manager, Firefox Quality Engineering - Ryan VanderMeulen
PM, QA Engineeting Team - Rares Bologa
QA Engineering Team - Brindusa Tot
PM, Mozilla Release QA Team - Florin Mezei
Mozilla Release QA Team - Andrei Vaida

Testing summary

Scope of Testing

In Scope

Full testing of Firefox browser will be performed. Testing will be performed against Nightly, Aurora and Beta builds. Areas that need to be included in testing are:

- User Engagement
- Stability
- Scrolling and Zooming
- UI Smoothness
- Page Load
- Plugins
- Graphics
- Startup/Shutdown Time
- Memory Usage
- Performance

Not In Scope

Add-on Compatibility and Accessibility are not in the scope of current release. New test plans will be created for these areas of functionality once ready for testing. Link available for e10s Add-ons compatibility is http://arewee10syet.com.

Test types

Manual testing is performed by both Mozilla QA Engineering and Mozilla Release QA teams.
We will focus our testing on the at-risk features identified by the engineering teams, validation of experiment code functionality prior to deployment, assistance with bug triage and verification, general functional and exploratory testing

Automated Testing
A large number of automated tests are written and run against different OSes. There is a dedicated team led by Blake Kaplan that is responsible for ensuring that automated tests are run where possible with e10s enabled in continuous integration.

Note: Because of constraints on Windows hardware test pools, automated tests on e10s are not enabled on Windows XP and Windows 8 on production branches. Instead, Windows 7 is the focus of all e10s test coverage on production branches for Gecko 47+. The Ash project branch which tracks mozilla-central was also created to provide test coverage across all desktop platforms where resource constraints are an issue.

Details about test coverage can be found at Test Coverage link

Requirements for testing

Environments

It is desirable for testing to be performed on a wide range of environments, but focus will be on:

- Windows 7
- Windows 10
- OSX 10.10 (OSX 10.6-10.8 are EOL)
- Ubuntu 14/15

For testing, we will alternate 32 and 64 FF builds and we will do the same with the above mentioned OSes.

Notes:

Testing will also be performed on lower-end hardware, like single core machines with Windows 7 or Windows Vista. The testing focus on these machines will focus on graphics and performance/responsiveness.

Builds

This section should contain links for builds with the feature -

  • Nightly builds are available at link
  • Aurora builds are available at link
  • Beta builds are available at link

Bug Management

Any issue identified during testing will be logged in Bugzilla. When filing a new e10s bug, the template from the following link: http://is.gd/aTza8A, will be used.
The tracking-e10s:? flag will be set so that it shows up in the e10s team's weekly triage bug list. Each e10s bug will be triage by engineering teams and prioritized based on severity.

Features covered

e10s specific testing

Manual testing with e10s enabled has been performed for the below features and all issues found have also been verified with e10s disabled prior to logging them.

Full functional testing

General testing was performed on Aurora 46 and Beta 45 on the following browser areas: Audio Compatibility, Video Compatibility, Web Compatibility, Geolocation, Safe Browsing, Plug-in compatibility, WEBGL Compatibility, PDF Compatibility, Image support, Crash Reporter, Location bar, Search Suggestions, Sync, Downloads, Bookmarks and History, READER VIEW & READABILITY.JS, Browser customization, Scroll and Zoom

QA Manager contact Ryan VanderMeulen
QA Contact Engineering Team Michelle Funches
QA Contact Desktop Team Andrei Vaida
Team Responsible FF Version tested Test type Test Cases Automation e10s bugs Signoff Signoff Date
Moz QA Eng Aurora 46 Manual Aurora_test -

1252262
1253610

n/a 03.10.2016
Desktop QA Team Beta 45 Manual Beta_tests -

1248599
1247611

n/a
Desktop QA Team Beta 46 Manual Beta_tests -

n/a

n/a


Password Manager

Tested Password Manager against Nightly 47.

QA Manager contact Ryan VanderMeulen
QA Contact Vlad Bacia
Team Responsible FF Version tested Test Plan Test type Test Cases Automation e10s bugs Signoff Signoff Date
Moz QA Eng Nightly 47 - Manual tc TBD - -
  • Meta bug on Password Manager - 1118955


APZ bugs verification

Engineering Manager, Pan/Zoom Kartikaya Gupta
QA Manager contact Ryan VanderMeulen
QA Contact Brindusa Tot

On this area, Kats' team was assisted with bug triage on APZ bugs and verification of major impact bugs.

Hello(loop)

Engineering Manager, Hello Ian Bicking
Firefox Hello Desktop Technical Lead Mark Banner
QA Manager contact Ryan VanderMeulen
QA Contact Brindusa Tot
Team Responsible FF Version tested Test Plan Test type Test Cases Automation e10s bugs Signoff Signoff Date
Moz QA Eng Nightly 48 Hello TP Manual Hello TC -

1257243
1257575
1257790

-


E10s Experiment Testing

Tested Experiments on Beta 45 and Nightly 47.

QA Manager contact Ryan VanderMeulen
QA Contact Michelle Funches
Team Responsible FF Version tested Test Plan Test type Test Cases Automation e10s bugs Signoff Signoff Date
Moz QA Eng Multi-process Firefox A/B test Beta 45 TP Manual TC TBD - Yes 01/28/2016
Moz QA Eng Multi-process Firefox A/B Experiment Beta 2nd Experiment TP Manual TC TBD - Yes 02/11/2016
Moz QA Eng DisplayPort Size Tuning aka Checkerboard Experiment TP Manual TC TBD - Yes 03/02/2016
  • Meta bug on Multi-process Firefox A/B test Beta 45 - 1238802
  • Meta bug on Multi-process Firefox A/B Experiment Beta 2nd Experiment - 1244187
  • Meta bug on DisplayPort Size Tuning aka Checkerboard Experiment - 1251052


Other Features tested generally

The following features have been tested on Nightly channel: functional validation has been performed for each feature with e10s enabled and disabled.

Push Notifications

Development contact Edwin Wong
QA Manager contact Ryan VanderMeulen
QA Contact Michelle Funches
Team Responsible FF Version tested Test Plan Test type Test Cases Automation e10s bugs Signoff Signoff Date
Moz QA Eng Nightly 45 Test Plan Manual tc TBD [DONE] 01/06/2016


Synced Tabs Sidebar

Development contact Edwin Wong
QA Manager contact Ryan VanderMeulen
QA Contact Brindusa Tot


Team Responsible FF Version tested Test Plan Test type Test Cases Automation e10s bugs Signoff Signoff Date
Moz QA Eng Nightly 47 Push Notification TP Manual tc TBD Conditional Signoff 03.08.2016


Activity Stream

Development contact Edwin Wong
QA Manager contact Ryan VanderMeulen
QA Contact Paul Oiegas
Vlad Bacia


Team Responsible FF Version tested Test Plan Test type Test Cases Automation e10s bugs Signoff Signoff Date
Moz QA Eng Nightly 48 Activity Stream TP Manual tc TBD - -
  • Meta bug: bug [-]


Risk analysis

Bug Handling/Triage

  • Bug triage has been mostly focusing on bugs specifically flagged and nominated as e10s issues. We may be missing incoming issues that are filed as generic bugs and don't get the right e10s nominations. This can be possibly mitigated by more focused review of all incoming bugs.
  • Bug prioritization is based on subjective opinions about the relative severity of a reported issue. This is mitigated to some extent by these decisions being made in a group setting with representatives from different stakeholders present and able to make their case.

Test Coverage

  • The Engineering QA team focus has been on validating e10s functionality primarily on the Nightly channel, and not on channels which are closer to potential release. Only when specific requests are received from engineering teams to validate e10 functionality on other channels (like Aurora) has focused testing been performed.
  • The Softvision QA teams are small relative to the overall number of users, even on pre-release channels, and we have a relatively homogenous set of hardware we're testing on. We acknowledge the risk of missing bugs due to this lack of diversity. This risk is at least somewhat mitigated by shipping e10s enabled by default to our Nightly and DevEdition populations and by the recent experiments on Beta.
  • Manual testing will never be an exhaustive way of testing changes that affect the entire codebase. By choosing to focus on specific features for targeted testing, we are intentionally *not* exhaustively testing other functionality in a deliberate manner. This is hopefully mitigated by the ongoing efforts to expand our automated test coverage across a larger percentage of the test suite and across a wider variety of platforms in CI.
  • Our automated test infrastructure is severely resource constrained, limiting the number of platforms we can test on and the frequency with which those tests can be run. We've partially mitigated some of these issues by using the Ash branch to run the test suite against a larger number of platforms, but this only tracks mozilla-central, leaving us potentially exposed to uncaught problems or missed backports on release branches.
  • Intermittent test failures occur at high frequency in our automated test infrastructure, making it difficult to find and track down new issues that get lost in the noise. Intermittent test failures are generally prioritized below other work, adding further difficulty to this. We don't have a great mitigation for this at this point in time, other than disabling flaky tests and accepting the loss in test coverage that results from it.

At-Risk Features/Functionality

  • The initial rollout plan is for only users with add-ons disabled. Code has been added to detect that at runtime and may possibly have edge cases where users end up with e10s enabled that shouldn't. This has been mostly-mitigated by ongoing exploratory testing and crash/Telemetry data analysis and verification.
  • Significant changes have been made to the graphics pipeline to support accelerated graphics over IPC. Prior experience has shown that any major changes to graphics can cause regressions that are difficult to find until the changes are shipping to release populations. We need to ensure that we have effective blocklisting mechanisms in place and as thorough testing as possible to minimize unanticipated regressions and point releases once e10s ships.

Test Areas

Test Areas Covered Details
Private Window Yes
Theme (high contrast) Yes
UI
Mouse-only operation Yes
Keyboard-only operation Yes
Display (HiDPI) Yes
Interraction (scroll, zoom) Yes
Usable with a screen reader No e.g. with NVDA
Help/Support
Help/support interface required No Make sure links to support/help pages exists and are easy reachable.
Support documents planned(written) No Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected Yes
Does sync work across upgrades Yes
Requires install testing No separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding  ??? Florin/Lawrence are investigating if there is a dedicated QA for this. If this must be tested for current feature, then we should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing  ??? yes-no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work differently on ESR builds
Enterprise administration N/A
Network proxies/autoconfig N/A
ESR behavior changes N/A
Locked preferences N/A
Data Monitoring
Temporary or permanent telemetry monitoring Yes List of error conditions to monitor
Telemetry correctness testing
Server integration testing  ?
Offline and server failure testing  ?
Load testing  ?
Add-ons No Separate test plan, will be on a different release, as a separate ship criteria
Add-on API required -
Comprehensive API testing -
Permissions -
Testing with existing/popular add-ons -
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review
Privilege escalation testing
Fuzzing
Web Compatibility depends on the feature
Testing against target sites Yes
Survey of many sites for compatibility Yes
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. -
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS -
Interaction of this feature with other browser features Yes

Sign off

Criteria

Check list

  • All blocking bugs must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA).
  • There is a separate wiki page in regards to Electrolysis release criteria, at the following wiki page.

Results

For each tested feature, the links to the executed manual test-cases can be found at section "Features covered", see Features covered

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE] Make sure e10s is enabled
Test Plan Creation [IN PROGRESS]
Test Cases Creation [IN PROGRESS]
Full Functional Tests Execution [IN PROGRESS]
Automation Coverage Blake Kaplan (mrbkap)
Performance Testing Need to see who is in charge
All Defects Logged [IN PROGRESS]
Critical/Blockers Fixed and Verified [IN PROGRESS] Should verify that for each tested feature criticals and blockers are fixed and verified
QA Aurora - Full Testing [DONE] Email sent on 03/11/2016
QA Push Notifications [DONE] Email sent on 01/06/2016
QA Synced Tabs Sidebar [DONE] Email sent on 03/08/2016
QA Next feature to be tested Email to be sent

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation Google docs
Test case execution Google docs
Bugs management Bugzilla


For Bugzilla, when filing a new e10s bug , use the template from the following link: http://is.gd/aTza8A
Please add the tracking-e10s:? flag or place 'e10s' in the bug title so that it shows up in the team's weekly triage bug list.


References

List and links related to E10s effort

Electrolysis wiki pages

Summary: Goal, Enabling and Disabling Electrolysis, What to Expect, Schedule, Contributing, Accessibility Support, Add-ons Compatibility, Past Milestones, Communication, People, Reference, Bug Lists, Meeting Notes, Weekly Status

Electrolysis Test Coverage wiki page

Summary: Table with Test coverage on diff OSes, Related Bugs, Individual Test Issues, Mochitest a11y, Mochitest chrome, Windows Coverage

Electrolysis Release Criteria

Summary: APZ Regressions, Release Criteria, User Engagement, Stability, Jank, Scrolling, Slow Scripts, UI Smoothness, Page Load, Plugin Jank, Graphics Performance, Startup/Shutdown Time, Memory Usage, Release Blocking Bugs, M8/M9 bugs, Release Criteria meta bugs' blockers, APZ Bugs, Tests, Accessibility Add-ons

Release Criteria Document: contains information related to criteria for releasing E10s specific to different browser functionalities, like Stability, Performance, Automation testing


E10s performance tasks meeting minutes (Feb-19)


E10s meetings agenda and notes