Engineering Quality Program: Difference between revisions
Jump to navigation
Jump to search
(→Programs and Projects: Proposed changes to project list - removal of two no longer needed and addition of two projects for which we need wikis) |
(→Programs and Projects: removed deliverables from project list) |
||
| (13 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
=='''Program Description'''== | =='''Program Description'''== | ||
The purpose of the Firefox OS Quality initiative is to drive the engineering team to Continuous Integration through testing platform improvements, test improvements, test automation and code coverage, where every commit on Gecko and Gaia triggers a full test run, with failures and performance issues reported on Treeherder for immediate backout. | |||
The following Firefox OS 2.6 work must be completed to achieve Continuous Integration: | |||
* [[Automated Test Coverage]] | |||
* Convert integration tests from GIP to GIJ | |||
* Fix intermittently failing tests | |||
* Fix tests that work on mulet but not on device | |||
* Create a Test Matrix to define which platform the tests are going to run on (device, mulet, emulator) | |||
* Make sure every single commit in gecko/gaia triggers automated tests | |||
* Automate MozTrap scenarios using GIJ when feasible | |||
* Actively maintain working Mulet tests with reporting on Treeherder | |||
* Ensure code coverage | |||
* [[Marionette Improvements]] | |||
* Remove python from Marionette harness | |||
* Run Marionette.js tests on devices in the Bitbar testing farm with reporting on Treeherder | |||
* Improve Marionette stability and stability of sandbox environment | |||
* [[Kitkat Emulator Improvements]] | |||
* Fix kitkat emulator testing issues | |||
* Put emulators in place for testing with reporting on Treeherder | |||
* Port emulator to next android x86 version | |||
* [[Raptor Activation on Treeholder]] | |||
* Run performance tests on devices with reporting on Raptor and Treeherder (Bitbar testing farm) | |||
* Integrate the device service into Raptor | |||
* Achieve ability for sheriffs to back out commits if post-commit data is bad | |||
* Ability to block commits with pre-commit testing | |||
The following Firefox OS 2.6 work must be completed to get to the Train Model (OTA branch): | |||
* Low amount of bugs in master | |||
* 0 Smoke tests blockers / foxfood blockers in the OTA branch | |||
* Automated / Semi manual OTA delivery to foxfooders from the OTA branch | |||
* Automated update of OTA branch from master once OTA has been delivered | |||
The following Firefox OS 2.6 work must be completed to for software architecture improvement: | |||
* Split BE / FE and apps views (NGA - bridge.js) | |||
* Mock app BE for FE testing with Telemetry libraries (NGA) | |||
* BE tests with Telemetry reports (NGA) | |||
=='''Programs and Projects'''== | =='''Programs and Projects'''== | ||
| Line 6: | Line 39: | ||
! style="text-align: center; font-size:22px" colspan="3" | [[Continuous Integration]] | ! style="text-align: center; font-size:22px" colspan="3" | [[Continuous Integration]] | ||
|- | |- | ||
| | ! style="text-align: left; font-size:16px"| Project Name | ||
! style="text-align: center; font-size:16px"| Leads | |||
! style="text-align: center; font-size:16px"| Status | |||
! style=" | |||
|- | |- | ||
| [[ | | [[Automated Test Coverage]] | ||
| PM: <br /> | | PM: N/A <br /> | ||
EPM:<br /> | EPM: Julie McCracken <br /> | ||
EM: | EM: Dylan Oliver | ||
! style="background:#00EC00;" | ON TARGET | ! style="background:#00EC00;" | ON TARGET | ||
|- | |- | ||
| [[ | | [[Marionette Improvements]] | ||
| PM: <br /> | | PM: N/A <br /> | ||
EPM:<br /> | EPM: Julie McCracken <br /> | ||
EM: | EM: Dylan Oliver | ||
! style="background:#00EC00; | ! style="background:#00EC00; width: 25%;" | ON TARGET | ||
|- | |- | ||
| [[Kitkat Emulator Improvements]] | | [[Kitkat Emulator Improvements]] | ||
| PM: | | PM: N/A <br /> | ||
EPM: Josh Cheng <br /> | |||
EM: Hsinyi Tsai | |||
EPM:<br /> | |||
EM: | |||
! style="background:#00EC00;" | ON TARGET | ! style="background:#00EC00;" | ON TARGET | ||
|- | |- | ||
| [[Raptor Activation on Treeholder]] | | [[Raptor Activation on Treeholder]] | ||
| PM: <br /> | | PM: N/A <br /> | ||
EPM:<br /> | EPM: Bobby Chien <br /> | ||
EM: | EM: Gregor Wagner | ||
! style="background:#00EC00;" | ON TARGET | ! style="background:#00EC00;" | ON TARGET | ||
|- | |- | ||
| Line 51: | Line 70: | ||
|- | |- | ||
| [[Train Model OTA Branch]] | | [[Train Model OTA Branch]] | ||
| PM: <br /> | | PM: N/A <br /> | ||
EPM:<br /> | EPM: Julie McCracken <br /> | ||
EM: | EM: Mahe Potharaju | ||
! style="background:#00EC00;" | ON TARGET | ! style="background:#00EC00;" | ON TARGET | ||
|- | |- | ||
| Line 59: | Line 78: | ||
|- | |- | ||
| [[Split BE / FE and apps views]] | | [[Split BE / FE and apps views]] | ||
| PM: <br /> | | PM: <br /> | ||
EPM:<br /> | EPM: Maria Oteo<br /> | ||
EM: | EM: David Scravaglieri | ||
! style="background:#00EC00;" | ON TARGET | ! style="background:#00EC00;" | ON TARGET | ||
|- | |- | ||
| [[Mock app BE for FE testing with Telemetry Libraries]] | | [[Mock app BE for FE testing with Telemetry Libraries]] | ||
| PM: <br /> | | PM: <br /> | ||
EPM:<br /> | EPM: Maria Oteo<br /> | ||
EM: | EM: David Scravaglieri | ||
! style="background:#00EC00;" | ON TARGET | ! style="background:#00EC00;" | ON TARGET | ||
|- | |- | ||
| [[BE tests with Telemetry reports]] | | [[BE tests with Telemetry reports]] | ||
| PM: <br /> | | PM: <br /> | ||
EPM:<br /> | EPM: Maria Oteo<br /> | ||
EM: | EM: David Scravaglieri | ||
! style="background:#00EC00;" | ON TARGET | ! style="background:#00EC00;" | ON TARGET | ||
|- | |- | ||
Latest revision as of 19:06, 30 December 2015
Program Description
The purpose of the Firefox OS Quality initiative is to drive the engineering team to Continuous Integration through testing platform improvements, test improvements, test automation and code coverage, where every commit on Gecko and Gaia triggers a full test run, with failures and performance issues reported on Treeherder for immediate backout.
The following Firefox OS 2.6 work must be completed to achieve Continuous Integration:
* Automated Test Coverage * Convert integration tests from GIP to GIJ * Fix intermittently failing tests * Fix tests that work on mulet but not on device * Create a Test Matrix to define which platform the tests are going to run on (device, mulet, emulator) * Make sure every single commit in gecko/gaia triggers automated tests * Automate MozTrap scenarios using GIJ when feasible * Actively maintain working Mulet tests with reporting on Treeherder * Ensure code coverage * Marionette Improvements * Remove python from Marionette harness * Run Marionette.js tests on devices in the Bitbar testing farm with reporting on Treeherder * Improve Marionette stability and stability of sandbox environment * Kitkat Emulator Improvements * Fix kitkat emulator testing issues * Put emulators in place for testing with reporting on Treeherder * Port emulator to next android x86 version * Raptor Activation on Treeholder * Run performance tests on devices with reporting on Raptor and Treeherder (Bitbar testing farm) * Integrate the device service into Raptor * Achieve ability for sheriffs to back out commits if post-commit data is bad * Ability to block commits with pre-commit testing
The following Firefox OS 2.6 work must be completed to get to the Train Model (OTA branch):
* Low amount of bugs in master * 0 Smoke tests blockers / foxfood blockers in the OTA branch * Automated / Semi manual OTA delivery to foxfooders from the OTA branch * Automated update of OTA branch from master once OTA has been delivered
The following Firefox OS 2.6 work must be completed to for software architecture improvement:
* Split BE / FE and apps views (NGA - bridge.js) * Mock app BE for FE testing with Telemetry libraries (NGA) * BE tests with Telemetry reports (NGA)
Programs and Projects
| Continuous Integration | ||
|---|---|---|
| Project Name | Leads | Status |
| Automated Test Coverage | PM: N/A EPM: Julie McCracken |
ON TARGET |
| Marionette Improvements | PM: N/A EPM: Julie McCracken |
ON TARGET |
| Kitkat Emulator Improvements | PM: N/A EPM: Josh Cheng |
ON TARGET |
| Raptor Activation on Treeholder | PM: N/A EPM: Bobby Chien |
ON TARGET |
| Release Train Model | ||
| Train Model OTA Branch | PM: N/A EPM: Julie McCracken |
ON TARGET |
| Software Architecture Improvement | ||
| Split BE / FE and apps views | PM: EPM: Maria Oteo |
ON TARGET |
| Mock app BE for FE testing with Telemetry Libraries | PM: EPM: Maria Oteo |
ON TARGET |
| BE tests with Telemetry reports | PM: EPM: Maria Oteo |
ON TARGET |