Engineering Quality Program: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(template)
 
(→‎Programs and Projects: removed deliverables from project list)
 
(17 intermediate revisions by 2 users not shown)
Line 1: Line 1:
<div style="float: right; margin: 0px 0px 10px 25px;">[[File:Fxossmall.png|link=]]</div>
=='''Program Description'''==
=='''Program Description'''==
Enter text here
The purpose of the Firefox OS Quality initiative is to drive the engineering team to Continuous Integration through testing platform improvements, test improvements, test automation and code coverage, where every commit on Gecko and Gaia triggers a full test run, with failures and performance issues reported on Treeherder for immediate backout. 


===Product Requirements:===
The following Firefox OS 2.6 work must be completed to achieve Continuous Integration:
Enter text here
  * [[Automated Test Coverage]]
    * Convert integration tests from GIP to GIJ
    * Fix intermittently failing tests
    * Fix tests that work on mulet but not on device
    * Create a Test Matrix to define which platform the tests are going to run on (device, mulet, emulator)
    * Make sure every single commit in gecko/gaia triggers automated tests
    * Automate MozTrap scenarios using GIJ when feasible
    * Actively maintain working Mulet tests with reporting on Treeherder
    * Ensure code coverage
  * [[Marionette Improvements]]
    * Remove python from Marionette harness
    * Run Marionette.js tests on devices in the Bitbar testing farm with reporting on Treeherder
    * Improve Marionette stability and stability of sandbox environment
  * [[Kitkat Emulator Improvements]]
    * Fix kitkat emulator testing issues
    * Put emulators in place for testing with reporting on Treeherder
    * Port emulator to next android x86 version
  * [[Raptor Activation on Treeholder]]
    * Run performance tests on devices with reporting on Raptor and Treeherder (Bitbar testing farm)
    * Integrate the device service into Raptor
    * Achieve ability for sheriffs to back out commits if post-commit data is bad
    * Ability to block commits with pre-commit testing
The following Firefox OS 2.6 work must be completed to get to the Train Model (OTA branch):
    * Low amount of bugs in master
    * 0 Smoke tests blockers / foxfood blockers in the OTA branch
    * Automated / Semi manual OTA delivery to foxfooders from the OTA branch
    * Automated update of OTA branch from master once OTA has been delivered
The following Firefox OS 2.6 work must be completed to for software architecture improvement:
    * Split BE / FE and apps views (NGA - bridge.js)
    * Mock app BE for FE testing with Telemetry libraries (NGA)
    * BE tests with Telemetry reports (NGA)


===User Stories and Acceptance Criteria===
=='''Programs and Projects'''==
{| class="wikitable"
{| class="wikitable" style="width: 85%; margin: auto;"
! style="text-align: center;" | Title
! style="text-align: center; font-size:22px" colspan="3" | [[Continuous Integration]]
! style="text-align: center;" | BUG ID
! style="text-align: center;" | User story
! style="text-align: center;" | Acceptance Criteria
|-
|-
| rowspan="2" | Title Goes Here
! style="text-align: left; font-size:16px"| Project Name
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1180659 Bug ID]
! style="text-align: center; font-size:16px"| Leads
| User Story 1
! style="text-align: center; font-size:16px"| Status
| Acceptance Criteria 1
|-
|-
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1180659 Bug ID]
| [[Automated Test Coverage]]
| User Story 2
| PM: N/A <br />
| Acceptance Criteria 2
EPM: Julie McCracken <br />
EM: Dylan Oliver
! style="background:#00EC00;" | ON TARGET
|-
|-
| rowspan="1" | Help/Onboarding
| [[Marionette Improvements]]
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1180659 Bug ID]
| PM: N/A <br />
| User Story 3
EPM: Julie McCracken <br />
| Acceptance Criteria 3
EM: Dylan Oliver
|}
! style="background:#00EC00; width: 25%;" | ON TARGET
 
=='''Program Status'''==
{| class="wikitable"
! style="text-align: center;" | Milestone
! style="text-align: center;" | Date
! style="text-align: center;" | Status
|-
|-
| Milestone 1
| [[Kitkat Emulator Improvements]]
|  
| PM: N/A <br />
EPM: Josh Cheng <br />
EM: Hsinyi Tsai
! style="background:#00EC00;" | ON TARGET
! style="background:#00EC00;" | ON TARGET
|-
|-
| Milestone 2
| [[Raptor Activation on Treeholder]]
|  
| PM: N/A <br />
! style="background:#FFFF00;" | CHALLENGED
EPM: Bobby Chien <br />
EM: Gregor Wagner
! style="background:#00EC00;" | ON TARGET
|-
|-
| Milestone 3
! style="text-align: center; font-size:22px" colspan="3" | [[Release Train Model]]
|
! style="background:#FF2800;" | AT RISK
|}
 
'''Status Key'''
{| class="wikitable"
! style="text-align: center;" | Color
! style="text-align: center;" | Status
! style="text-align: center;" | Key
|-
|-
! style="background:#00EC00;" |  
| [[Train Model OTA Branch]]
| On Target
| PM: N/A <br />
| The project or deliverable is expected to meet its due date.
EPM: Julie McCracken <br />
EM: Mahe Potharaju
! style="background:#00EC00;" | ON TARGET
|-
|-
! style="background:#FFFF00;" |
! style="text-align: center; font-size:22px" colspan="3" | [[Software Architecture Improvement]]
| Challenged
| The project or deliverable is facing an issue that might cause it to miss its due date, but a “get well” plan has been developed to get it back on track.
|-
|-
! style="background:#FF2800;" |  
| [[Split BE / FE and apps views]]
| At Risk or Late
| PM:  <br />
| The project or deliverable is blocked or facing an issue that might cause it to miss its due date, and there’s no “get well” plan to get it back on track, or it is already late.
EPM: Maria Oteo<br />
EM: David Scravaglieri
! style="background:#00EC00;" | ON TARGET
|-
|-
! style="background:#00B2FF;" |  
| [[Mock app BE for FE testing with Telemetry Libraries]]
| Done
| PM: <br />
| The project or deliverable has been completed.
EPM: Maria Oteo<br />
EM: David Scravaglieri
! style="background:#00EC00;" | ON TARGET
|-
|-
! style="background:#C1C6CA;" |  
| [[BE tests with Telemetry reports]]
| On Hold
| PM:  <br />
| The project or deliverable has been placed on hold.
EPM: Maria Oteo<br />
|}
EM: David Scravaglieri
 
! style="background:#00EC00;" | ON TARGET
----
 
=='''Program Timeline'''==
[[File:Release Timeline.png]]
 
----
 
=='''MVP Scope'''==
 
Querying by 2.6+ features
 
(please add correct bug tracking number)
<bugzilla>
    {
        "blocks":"1188667",
        "cf_feature_b2g":"2.6+",
        "status":["NEW","REOPENED","UNCONFIRMED","ASSIGNED","RESOLVED","VERIFIED","CLOSED"],
        "include_fields": "id, summary, status, target_milestone, resolution, assigned_to, depends_on, blocks, whiteboard, cf_feature_b2g"
    }
</bugzilla>
 
----
 
=='''Dependency Tracking'''==
[[File:Dependency Gantt Chart.png]]
 
----
 
=='''Detailed Program Plan'''==
 
{| class="wikitable"
! style="text-align: center;" | Action Item
! style="text-align: center;" | Engineering Owner
! style="text-align: center;" | QA Owner
! style="text-align: center;" | UX Owner
! style="text-align: center;" | Bugzilla ID
! style="text-align: center;" | Planned Done
! style="text-align: center;" | Actual Done
|-
|-
|
|
|
|
|
|
|
|-
|
|
|
|
|
|
|
|-
|
|
|
|
|
|
|
|-
|
|
|
|
|
|
|
|}
|}


----
----
=='''Program Stakeholders'''==
{| class="wikitable"
! style="text-align: center;" | Role
! style="text-align: center;" | Name
! style="text-align: center;" | IRC
|-
| EPM
|
|
|-
| EM
|
|
|-
| TL
|
|
|-
| UX
|
|
|-
| QA
|
|
|}
*EPM = Engineering Program Manager
*EM = Engineering Manager
*TL = Tech Lead
*UX = User Experience
----
==Reference Links==

Latest revision as of 19:06, 30 December 2015

Program Description

The purpose of the Firefox OS Quality initiative is to drive the engineering team to Continuous Integration through testing platform improvements, test improvements, test automation and code coverage, where every commit on Gecko and Gaia triggers a full test run, with failures and performance issues reported on Treeherder for immediate backout.

The following Firefox OS 2.6 work must be completed to achieve Continuous Integration: 

 * Automated Test Coverage
   * Convert integration tests from GIP to GIJ
   * Fix intermittently failing tests
   * Fix tests that work on mulet but not on device
   * Create a Test Matrix to define which platform the tests are going to run on (device, mulet, emulator)
   * Make sure every single commit in gecko/gaia triggers automated tests
   * Automate MozTrap scenarios using GIJ when feasible
   * Actively maintain working Mulet tests with reporting on Treeherder 
   * Ensure code coverage 
 * Marionette Improvements
   * Remove python from Marionette harness
   * Run Marionette.js tests on devices in the Bitbar testing farm with reporting on Treeherder
   * Improve Marionette stability and stability of sandbox environment
 * Kitkat Emulator Improvements
   * Fix kitkat emulator testing issues
   * Put emulators in place for testing with reporting on Treeherder
   * Port emulator to next android x86 version
 * Raptor Activation on Treeholder
   * Run performance tests on devices with reporting on Raptor and Treeherder (Bitbar testing farm) 
   * Integrate the device service into Raptor
   * Achieve ability for sheriffs to back out commits if post-commit data is bad
   * Ability to block commits with pre-commit testing

The following Firefox OS 2.6 work must be completed to get to the Train Model (OTA branch): 

   * Low amount of bugs in master
   * 0 Smoke tests blockers / foxfood blockers in the OTA branch
   * Automated / Semi manual OTA delivery to foxfooders from the OTA branch
   * Automated update of OTA branch from master once OTA has been delivered

The following Firefox OS 2.6 work must be completed to for software architecture improvement: 

   * Split BE / FE and apps views (NGA - bridge.js)
   * Mock app BE for FE testing with Telemetry libraries (NGA)
   * BE tests with Telemetry reports (NGA)

Programs and Projects

Continuous Integration
Project Name Leads Status
Automated Test Coverage PM: N/A

EPM: Julie McCracken
EM: Dylan Oliver

ON TARGET
Marionette Improvements PM: N/A

EPM: Julie McCracken
EM: Dylan Oliver

ON TARGET
Kitkat Emulator Improvements PM: N/A

EPM: Josh Cheng
EM: Hsinyi Tsai

ON TARGET
Raptor Activation on Treeholder PM: N/A

EPM: Bobby Chien
EM: Gregor Wagner

ON TARGET
Release Train Model
Train Model OTA Branch PM: N/A

EPM: Julie McCracken
EM: Mahe Potharaju

ON TARGET
Software Architecture Improvement
Split BE / FE and apps views PM:

EPM: Maria Oteo
EM: David Scravaglieri

ON TARGET
Mock app BE for FE testing with Telemetry Libraries PM:

EPM: Maria Oteo
EM: David Scravaglieri

ON TARGET
BE tests with Telemetry reports PM:

EPM: Maria Oteo
EM: David Scravaglieri

ON TARGET
Retrieved from "https://wiki.mozilla.org/index.php?title=Engineering_Quality_Program&oldid=1110968"