QA/Test Automation/2011-06-29

From MozillaWiki
Jump to: navigation, search

previous meeting | Meetings | next meeting »

Dial in

 # 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL)
 # 1-800-707-2533 (pin 369) Conf# 315 (US)
 # #mozmill for backchannel


  • Aaron, Henrik, Anthony, Matt, Cameron, Geo, David, Dave, Matt B.

Last action items

  • David: Work with Geo and Anthony to get some first ideas for changes of the review process; we will schedule a meeting once Henrik is back (should happen in June)
    • We should have a deeper discussion during our upcoming work week (Henrik)

Goals Overview

Risky Goals

  • Complete the refactoring of the API
  • Create the top 5 tests for the discovery pane
    • 4 tests now in SR, remainder need revisions

Project Status

Redesign Shared Modules (Geo/Henrik)

  • Window object model changing from a mixin-based model to an inheritance based model. Is in debug now, code will be delivered today.
  • Known state implemented for test start/end:
    • fresh default bookmarks
    • no tab history
    • all tabs closed except about:blank.
  • Default tabs can be left for test start if desired (probably only for the test that verifies the default tab set).
  • Still should add "close all open windows but the original and focus the browser" to known state; will be settled once model change is in.
  • Windowing is the last fundamental being worked on. In-content windows (addons manager, notably) still unaddressed, bumped to next quarter.
  • Implicit waitFor added on element finding (and is there for window finding as well).
  • Test areas and top tests have been triaged and prioritized and added to Pivotal.

On-demand release tests (Geo/Henrik)

  • Henrik is working on the final version of the staging script, which will also cover candidate builds and daily builds
  • Geo is working on adding BFT kickoff to the daemons, as well as code cleanup per Henrik's code review.

Endurance Tests (Dave/Anthony)

  • Flash videos landed on and litmus-data, will target test page and test case writing next quarter
  • Waverley to assist next quarter by making a portion of their test development Endurance tests related to the focus area
  • Looking to align with MemShrink and Telemetry


Mozmill Crowd

  • We are about to release Mozmill Crowd 0.1.3 by latest tomorrow (bug 667902)
  • Waiting for preliminary review on
  • Testday for Mozmill Crowd (and test development) this Friday, July 1st


Current dashboard installation (Henrik)
  • no updates
New dashboard driven by ES (Owen)

Sync Server Load Testing (Owen)

  • Project on hold until we have agreed upon goals with the services team.

On Demand Provisioning (Cloud Automation) (Al/Henrik)

  • Vsphere/ESX box is fully operational
  • Al is creating reference virtual machines on it from scratch (porting existing VMs is impossible...mostly)
    • First post-reference VM will be a squid proxy
  • What specific needs do people have here for on demand machines?

Tests / Shared Modules

General (Henrik)
Broken Tests
  • A couple of broken tests across brances in the last days. It has been caused by Henrik's patch for the Mozmill change in handling 'documentLoaded'. As is turned out today it's a Mozmill bug which blocks the thread execution if the window passed to the controller is null. See 668202 bug 668202.
Softvision (Anthony)
  • 4 Discovery Pane tests in SR, remainder need revisions
  • Will focus on Add-ons Manager restart tests next quarter, tacking on Web Apps or a Firefox 8 feature pending feature readiness

TCM (Cameron/Rebecca B.)

  • .4 progress hampered a bit by platform instability. We will not make all features by 7/15.
  • Negotiating with uTest on process and staffing improvements on their side.
  • Hashing out task assignments for Rebecca and Matt B.
  • Work on Environment management continuing.

Shadow web server (Al/Henrik)

  • SSL will require more IPs for VPS
    • Cannot reliably use multiple certs on one IP
      • Could use SNI to do this but it requires recompile of both OpenSSL and Apache, which carries its own maintenance and other headaches.
    • Getting three more IPs from IT (and Rackspace)
    • We are taking over last year's summit book cert and address ( as its SSL cert expires soon and IT was going to retire it.
    • We have the OV and DV certs we asked for (as of this last Friday). These two plus the summitbook and self-signed certs will fill four total IP addresses.
  • Flash test files have been deployed.
  • Mozmill Crowd files are being maintained by Henrik.
  • What other needs do people have for

Automation Scripts / Infrastructure (Henrik)

  • Download script has been enhanced to also download daily builds of Nightly or Aurora
  • We can now even download Nightly builds from project branches - anyone interested in?

Personal Status

For the personal status please check the weekly status updates:


  • Changes to our bi-weekly meeting regarding the org changes (Henrik)
    • We should setup preliminary pages which describe our new team next week
    • We should move the meeting pages below the new team page and call it "QA Automation Services Meeting"
  • Review process (Henrik)
    • A more detailed discussion should happen during our work week in Cambridge
    • We have to find an interim solution to get not stuck in review requests
      • If you are not able to react to a review request from Softvision or contributors within one day, you should pass the review to another person
      • If there are issues you are running into, spread this information across the team. It takes lesser time to solve it together
  • Discovery Pane tests
    • Do we want to allow tests to auto-skip themselves? If not we should mark those tests as skipped on default and aurora as long as the dependencies with WebDev are not solved.

Meeting Notes

  • General
    • Geo: We have to rethink how we set the goals especially for those with dependencies
  • Changes to the meeting
    • Matt: We want to cycle through all team members for driving a meeting
    • Henrik: Lets create the new wiki page directly after the current meeting when meeting notes and action items have been defined
    • Henrik: Everyone has to update the own content as early as possible so others have the chance to read through
    • Geo: Lets make the project updates just a read and ask questions part
  • Q2 goals
    • API (shared modules) refactor
      • Geo: We will not be able to finish all the tasks we have set for this goal
      • Henrik: Compared to the former quarters we did a huge amount of work this quarter in any way
      • Matt: Deliver summary of what has been done
    • Test porting
      • Matt: Also deliver a summary for this project with tasks finished
    • Discovery Pane
      • Anthony: Good feeling that we can get it definitely done by end of this week
      • Henrik: I will work with Softvision on the final reviews during EU time zones
      • Matt: Deliver summary what has been achieved
    • Endurance
      • Dave: We can call it done
      • Matt: We need plans for getting the new investigated metrics implemented
    • On-demand
      • Geo: on-demand driven updates already rolled out and run perfect
      • Geo: currently working on BFT integration with Henrik
      • Geo: We will make this goal
    • TCM
      • Cam: We successfully completed the design of the new filtering mechanism
      • Cam: We gathered user stories. This is ongoing, but we accomplished what we needed for this quarter
      • Cam: Buy in was a debulous goal, so this was dropped
      • Cam: We released .3 features internally on schedule
      • Cam: .4 feature implementation changed scope so that it was to complete 7/15 (after Q2 end)
        • However, we also will miss that date, because we were delayed due to Platform instability
  • Q3 goals
    • Most comments are in the etherpad, see goals section above
    • Overall QA goals
      • Henrik (mail): Make webapps goal more a goal to a generic test framework instead of Mozmill
      • Henrik (mail): We can't rely on Mozmill for e10s. Mozmill 2 will not be e10s ready and no schedule yet for the release
      • Matt: We should change the goal to add a good amount of endurance test enhancements
    • Team Goals
    • Geo: we should measure goals by 50-60% of our time
  • Roundtable
    • Meeting pages have to be put into a new tree onto the wiki
    • David will drive the next meeting

Action items

  • Geo: Deliver summary for API refactoring and test porting project by latest end of this week
  • Anthony: Deliver summary for Discover Pane project by latest end of this week
  • Henrik: Work with Softvision to ensure that DiscoPane tests will be checked in latest by Friday
  • All: Reply to the discussion email about auto-skipping DiscoPane tests
  • Matt: update QA overall goals
  • Henrik/David: Finalize Q3 goals and get team feedback
  • Henrik/David: Work out preliminary wiki pages for QA Automation Services team (incl. meeting page)
  • Henrik/David: Come up with the first draft of the team structure and tasks