B2G/QA/Meetings/Automation/2014-12-18

Overview

This is a weekly meetup to discuss all things regarding B2G automation and Quality. Please use wiki style.

• Meeting Name: QA Automation Roundtable
• Meeting Time: Thursday @ 9:30am - 10am PST
• Vidyo Room: QA Vidyo Room
• Video Link: https://v.mozilla.com/flex.html?roomdirect.html&key=nL4PESn1IaVh
• Dial in: 1-800-707-25331
• Meeting Notes: https://wiki.mozilla.org/B2G/QA/Meetings
• Previous Meeting:https://wiki.mozilla.org/B2G/QA/Meetings/Automation/2014-12-11
  • Q4 Goals B2G (Draft)
    • Complete 2.1 QA criteria: https://wiki.mozilla.org/Release_Management/FirefoxOS/Release_Milestones#Code_Complete_Criteria
    • Identify 3-5 active contributors that has completed One and Done tasks
    • Identify 2.2 features and have testplans drafted by end of sprint 2
    • Automate 20 gaia UI tests from backlog. have them running on device [geo: add "if appropriate"?], and checked into the main repo. and reporting against Tree Herder
      • [geo] Running on device (for MarionetteJS) and reporting to Treeherder are both harness features and belong to FxOS Auto, delivery of those is out of scope for QA. We should stick to "automate 20 gaia UI tests," implication being against whatever harness is currently supported and reasonably stable. I'd suggest leaving out "from backlog"--that's an implementation detail.
    • Clean up 2.1 invalidated testcases and prepare for 2.2 testsuite
      • performance goal for acceptance testing

Attendees

Previous Action Items

• [Geo] will send out best practices doc by EOD friday (1st draft)
• [Vio] take the feedback and refine it into a final public document on how-tos for managing the tests
  • https://docs.google.com/document/d/1_N278eyUo7fjuGGJ93kDle6wcvENckC09jBjQY_HtKc/edit

Status Highlights

Agenda

• [jlorenzo] Creating another job for less important tests (like https://bugzilla.mozilla.org/show_bug.cgi?id=1083731 )
  • We'll create a weekly job denominated "weekly" and put this test there
  • Also, design review for future examples
• [jlorenzo] What is the current status of MTBF?
  • Which mailing the report is sent to?
  • What's the plan to increase the number of tests?
  • https://github.com/Mozilla-TWQA/MTBF-Driver/tree/master/mtbf_driver/tests/mtbf
• [jlorenzo] Is there a way to delete voicemails on AT&T and T-Mobile SIM cards without calling the Voicemail?
• [Otilia] SoftVision PTO planning around holidays - you can find the dates here https://docs.google.com/spreadsheets/d/1cWHkm2SAoAG0TlLJK-h-QEeQyfyTKDID2ww5YtxcOPo/edit#gid=0
• US/EU PTO:
  • Geo: 12/22-01/02
  • Johan: 12/24-12/28 & 1/1
  • No-Jun: 12/25, 26, 1/1, 1/5 - 1/9, 1/19 - 1/23
• Q4 Goals Recap

Q1 2015 Goals

2.2 - (OS QA (in Taiwan)

• • Support the stability of 2.2 branch with focused test runs, qawanted, bug triage, and daily smoke tests (manual and automation)
  • Support new feature testing
  • Manage the release out the door until CC date, and hand off to Devices QA team into Q2.

3.0 - (OS QA (in US/EU)

• • Automation goals:
    • Re-evaluate the smoke tests on v3 that will be needed for automation in javascript
    • Focus on getting existing Gij tests running against the lab of devices
    • Creating a set of best practices on creating JS gaia-UI automation tests
    • Focus on getting smoke tests running against the lab of devices (could be a combination of Gij, python, or JS tests).
  • Support the stability of master with focused test runs, qawanted, bug triage, and daily smoke tests (manual and automation)
  • Support lightweight dogfooding program off of nightly
  • Develop feature testplans when features are concrete

US/EU Holidays: Christmas Eve - December 24 (Not in Canada and France) Christmas Day - December 25 Boxing Day - December 26 (Canada only) New Year's Day - January 1

• [Tony] Goals check and Q12015 Draft
• [Bebe] Can we move the master build time 2h earlier? will file a bug
  • Filed - https://bugzilla.mozilla.org/show_bug.cgi?id=1113255

This Week - Decisions & Action Items

• [Johan] work with Bebe on creating the weekly testrun suite in jenkins, and add this new test
  • [Geo] Also, we should seek early design review for any test that might be atypical in terms of externals, execution characteristics, etc.
• [Tony] follow up with MTBF questions
  • whats the latest MTBF report? on 2.1, trunk?
  • when will they start running this against 2.2?
  • currently 10-12 tests, whats the plan to add more?
  • more frequent updates sent to the fxos-watchlist. currently only b2g-release-drivers
• Voicemail issue, options we have;
  • Look into the voicemail AT&T API (https://vm.att.com/ ). This might be something for the Automation team
  • Create a cleanup job that dial the sequence to delete a voicemail like a regular user would do. (The dial sequence may differ between AT&T and T-Mobile)
  • Ask the carrier to clean up all the voicemails each week/month.
  • Create our own carrier network (http://openbts.org/ ) and put our voicemail there
• [Tony/GEo] work with functional teams on a plan to process preventative broken feature landings
• [Geo] work with functional teams on owning the Gij based per-checkin tests on master