QA/e10s Multi
< QA
Overview
Purpose
This wiki details the testing that will be performed by QA and other teams for e10s-multi. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
- Areas of risk
- What will be tested
- How testing will be performed
- criteria
- deliverables
- ownership
- schedule
- What data will be monitored
To help ensure the best possible release and long term maintenance of this feature.
Risk Analysis
| Risk area | Requirement | Status |
|---|---|---|
| Performance | Improved browser responsiveness | TBD |
| Memory Usage | No regression in comparable system memory usage | TBD |
| Security | No security holes found in fuzz testing | TBD |
| Stability | No regressions in crash rate | TBD |
| Major site compatibility | Improved perf at major websites | TBD |
Testing Strategy
Scope
| Test Areas | Covered |
|---|---|
| Private Window | Yes |
| Multi-Process Enabled | Yes |
| Single-Content Process Enabled | Yes (baseline) |
| e10s dis-abled | Yes (baseline) |
| Stability | Yes |
| UI | |
| Interraction (scroll, zoom) | Yes |
| Multi-Tab/Window | Yes |
| Security | |
| Fuzzing | Yes |
| Web Compatibility | |
| Testing against target sites | Yes |
| Data Monitoring | |
| Temporary or permanent telemetry monitoring | Yes |
If it's not listed above, it is currently out of scope.
Test Strategy
Objectives
| Criteria Description | Metric | Single-Content Process | Multi-Content Process | Criteria Met? | QA Owner |
|---|---|---|---|---|---|
| Manual testing | Tests passed | (test results) | (test results) | TBD (Date status updated) | SV? |
| Unit testing | Automated tests pass | (test results) | (test results) | TBD (Date status updated) | Blake |
| Talos testing | Startup time | ts_paint - perherder <link tbd> | ts_paint - perherder <link tbd> | TBD (Date status updated) | Talos team? |
| UI | Tab/Window responsiveness | Perherder <link(s) tbd> | Perherder <link(s) tbd> | TBD (Date status updated) | Talos team? |
| Memory monitoring | System memory use | <link tbd> | <link tbd> | TBD (Date status updated) | TBD |
| Fuzz testing | Automated tests pass | (test results) | (test results) | TBD (Date status updated) | Fuzzing team (tbd) |
| Stability | Crash rate stable | telemetry and crash stats | telemetry and crash stats | TBD (Date status updated) | Telemetry team |
Testcases
TBD:
- General browser testing as in e10s [1]
- e10s-multi specific tests
- process assignment
Environments
Full Testing will be performed using Nightly for Desktop builds on:
- Windows
- Mac OS X
- Linux
Channel dependent settings (configs) and environment setups
- In Nightly dom.ipc.processCount should be 4 by default
Project Information
Ownership
Developer contacts: Blake Kaplan, Gabor Krizsanits
QA: Tracy Walker
QA Peer: TBD
Builds
This section should contain links for builds with the feature
Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | Q3/2016 | Q3/2017 |
| QA - Test plan creation | Q1/2017 | Q1/2017 |
| QA - Test cases/Env preparation | Q1/2017 | Q2/2017 |
| QA - Nightly Testing | Q2/2017 | Q2/2017 |
| QA - Aurora Testing | Q2/2017 | Q3/2017 |
| QA - Beta Testing | Q3/2017 | Q3/2017 |
| Release Date | FF55 |
References
- List and links for specs
- [1] - e10s multi wiki page
- [2] - e10s multi release strategy
- [3] - e10s release criteria
- [4] - e10s UI smoothness
- Meta bug: - Turn e10s-multi on in Nightly