Continuous Integration
From MozillaWiki
This program ended before work was completed.The content within this wiki remains for archival purposes.
Contents
Program Description
The purpose of the Firefox OS Quality initiative was to drive the engineering team to Continuous Integration through testing platform improvements, test improvements, test automation and code coverage, where every commit on Gecko and Gaia triggers a full test run, with failures and performance issues reported on Treeherder for immediate backout. Work on these projects ended with the decision to sunset Firefox OS.
Requirements:
The following Firefox OS 2.6 work must be completed to achieve Continuous Integration:
* Automated Test Coverage * Convert integration tests from GIP to GIJ * Fix intermittently failing tests * Fix tests that work on mulet but not on device * Create a Test Matrix to define which platform the tests are going to run on (device, mulet, emulator) * Make sure every single commit in gecko/gaia triggers automated tests * Automate MozTrap scenarios using GIJ when feasible * Actively maintain working Mulet tests with reporting on Treeherder * Ensure code coverage * Marionette Improvements * Remove python from Marionette harness * Run Marionette.js tests on devices in the Bitbar testing farm with reporting on Treeherder * Improve Marionette stability and stability of sandbox environment * Kitkat Emulator Improvements * Fix kitkat emulator testing issues * Put emulators in place for testing with reporting on Treeherder * Port emulator to next android x86 version * Raptor Activation on Treeholder * Run performance tests on devices with reporting on Raptor and Treeherder (Bitbar testing farm) * Integrate the device service into Raptor * Achieve ability for sheriffs to back out commits if post-commit data is bad * Ability to block commits with pre-commit testing
Program Status
Automated Test Coverage
| Milestone | Due Date | Status | Notes |
|---|---|---|---|
| Convert integration tests from GIP to GIJ | 03/25/16 | CHALLENGED | Due date revised to end of Sprint 10. All but 10 have been converted. This deliverable is challenged because intermittents are a competing priority. |
| Intermittently failing tests fixed | 02/26/16 | ON TARGET | Due date revised to end of Sprint 8. Fixed parts of Marionette that were causing issues. Disabled all the tests were failing intermittently. Module owners need to ensured they're fixed before re-enabling them. Need support to prioritize this work with the engineers. |
| Tests that work on mulet work on device | 02/26/16 | CHALLENGED | It's being reported that some of the flagged tests actually work on device. Need to reassess all. This deliverable isn't being given much attention. |
| Test Matrix created that defines platform(s) the tests are going to run on | 02/26/16 | ON TARGET | Due date pushed out to one that is more realistic. Draft of matrix completed but there are still some tests that need to be evaluated for appropriate testing environment. Meeting with eng. mgmt. scheduled for Jan. 27th to review test matrix. |
| Automation of tests according to Test Matrix | 05/31/16 | CHALLENGED | Draft of matrix completed but there are still some tests that need to be evaluated for appropriate testing environment. Need to finalize matrix before automation can begin. |
| Every single commit in gecko/gaia triggers automated tests | 05/31/16 | ON TARGET | There's still work that needs to be identified and scheduled. |
| MozTrap scenarios automated using GIJ when feasible | 05/31/16 | ON TARGET | |
| Working Mulet tests actively maintained with reporting on Treeherder | 05/31/16 | ON TARGET | |
| Code Coverage Tool developed | 05/31/15 | ON TARGET | Originally targeted for end of Q4, but assistance at Mozlando didn't happen. Re-prioritized and given new due date. |
Marionette Improvements
| Milestone | Due Date | Status | Notes |
|---|---|---|---|
| Removal of python from the Marionette harness | 02/26/16 | ON TARGET | New marionette-client promises driver developed. Now working on test for new api. Due date revised (pushed out). |
| Device testing on TaskCluster/Treeherder with Bitbar | 05/31/16 | ON HOLD | Uncertaintly about the future of phone testing at Bitbar has resulted in questions regarding how to proceed. |
| Marionette stability | 03/31/16 | ON TARGET | Bugs have been entered and work is slotted for Q1. Fixes have been made to tackle problem associated with "intermittents". |
| Sandbox environment stability | 03/31/16 | ON TARGET | Work is slotted for Q1. |
Kitkat Emulator Improvements
| Milestone | Due Date | Status | Notes |
|---|---|---|---|
| Milestone 1 | ON TARGET | ||
| Milestone 2 | CHALLENGED | ||
| Milestone 3 | AT RISK | ||
| Milestone 4 | ON TARGET | ||
| Milestone 5 | CHALLENGED | ||
| Milestone 6 | AT RISK | ||
| Milestone 7 | ON TARGET | ||
| Milestone 8 | CHALLENGED | ||
| Milestone 9 | AT RISK |
Raptor Activation on Treeherder
| Milestone | Due Date | Status | Notes |
|---|---|---|---|
| Integrate the fxos device service into Raptor | DONE | ||
| Enable automated testing of devices at Bitbar | NOT STARTED | ||
Status Key
| Color | Status | Key |
|---|---|---|
| On Target | The project or deliverable is expected to meet its due date. | |
| Challenged | The project or deliverable is facing an issue that might cause it to miss its due date, but a “get well” plan has been developed to get it back on track. | |
| At Risk or Late | The project or deliverable is blocked or facing an issue that might cause it to miss its due date, and there’s no “get well” plan to get it back on track, or it is already late. | |
| Done | The project or deliverable has been completed. | |
| On Hold | The project or deliverable has been placed on hold. |
MVP Scope
Querying by 2.6+ features
(please add correct bug tracking number)
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Dependency Tracking (TBD)
Detailed Program Plan
| Action Item | Engineering Owner | QA Owner | UX Owner | Bugzilla ID | Planned Done | Actual Done |
|---|---|---|---|---|---|---|
Program Stakeholders
| Role | Name | IRC |
|---|---|---|
| EPM | ||
| EM | ||
| PM | ||
| TL | ||
| UX | ||
| QA |
- EPM = Engineering Program Manager
- EM = Engineering Manager
- PM = Product Manager
- TL = Tech Lead
- UX = User Experience
- QA = Quality Assurance
