previous meeting | Meetings | next meeting »

Dial in

 # 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL)
 # 1-800-707-2533 (pin 369) Conf# 315 (US)
 # irc.mozilla.org #mozmill for backchannel

Attendees

• Aaron, Henrik, Anthony, Matt, Cameron, Geo, David, Dave, Matt B.

Last action items

• David: Work with Geo and Anthony to get some first ideas for changes of the review process; we will schedule a meeting once Henrik is back (should happen in June)
  • We should have a deeper discussion during our upcoming work week (Henrik)

Goals Overview

• Revisit our Automation Team goals for 11/Q2
• Work on Automation Services goals for 11/Q3 (proposals)

Risky Goals

• Complete the refactoring of the API
• Create the top 5 tests for the discovery pane
  • 4 tests now in SR, remainder need revisions

Project Status

Redesign Shared Modules (Geo/Henrik)

• Window object model changing from a mixin-based model to an inheritance based model. Is in debug now, code will be delivered today.
• Known state implemented for test start/end:
  • fresh default bookmarks
  • no tab history
  • all tabs closed except about:blank.
• Default tabs can be left for test start if desired (probably only for the test that verifies the default tab set).
• Still should add "close all open windows but the original and focus the browser" to known state; will be settled once model change is in.
• Windowing is the last fundamental being worked on. In-content windows (addons manager, notably) still unaddressed, bumped to next quarter.
• Implicit waitFor added on element finding (and is there for window finding as well).
• Test areas and top tests have been triaged and prioritized and added to Pivotal.

On-demand release tests (Geo/Henrik)

• Henrik is working on the final version of the staging script, which will also cover candidate builds and daily builds
• Geo is working on adding BFT kickoff to the daemons, as well as code cleanup per Henrik's code review.

Endurance Tests (Dave/Anthony)

• Flash videos landed on mozqa.com and litmus-data, will target test page and test case writing next quarter
• Waverley to assist next quarter by making a portion of their test development Endurance tests related to the focus area
• Looking to align with MemShrink and Telemetry

Others

Mozmill Crowd

• We are about to release Mozmill Crowd 0.1.3 by latest tomorrow (bug 667902)
• Waiting for preliminary review on addons.mozilla.org
• Testday for Mozmill Crowd (and test development) this Friday, July 1st

Dashboard

Current dashboard installation (Henrik)

• no updates

New dashboard driven by ES (Owen)

• Link to current Pilot Application

Sync Server Load Testing (Owen)

• Project on hold until we have agreed upon goals with the services team.

On Demand Provisioning (Cloud Automation) (Al/Henrik)

• Vsphere/ESX box is fully operational
• Al is creating reference virtual machines on it from scratch (porting existing VMs is impossible...mostly)
  • First post-reference VM will be a squid proxy
• What specific needs do people have here for on demand machines?

Tests / Shared Modules

General (Henrik)

• Check-ins since the last meeting

Broken Tests

• A couple of broken tests across brances in the last days. It has been caused by Henrik's patch for the Mozmill change in handling 'documentLoaded'. As is turned out today it's a Mozmill bug which blocks the thread execution if the window passed to the controller is null. See 668202 bug 668202.

Softvision (Anthony)

• 4 Discovery Pane tests in SR, remainder need revisions
• Will focus on Add-ons Manager restart tests next quarter, tacking on Web Apps or a Firefox 8 feature pending feature readiness

TCM (Cameron/Rebecca B.)

• .4 progress hampered a bit by platform instability. We will not make all features by 7/15.
• Negotiating with uTest on process and staffing improvements on their side.
• Hashing out task assignments for Rebecca and Matt B.
• Work on Environment management continuing.

Shadow web server (Al/Henrik)

• SSL will require more IPs for VPS
  • Cannot reliably use multiple certs on one IP
    • Could use SNI to do this but it requires recompile of both OpenSSL and Apache, which carries its own maintenance and other headaches.
  • Getting three more IPs from IT (and Rackspace)
  • We are taking over last year's summit book cert and address (https://summitbook.mozilla.org) as its SSL cert expires soon and IT was going to retire it.
  • We have the OV and DV certs we asked for (as of this last Friday). These two plus the summitbook and self-signed certs will fill four total IP addresses.
• Flash test files have been deployed.
• Mozmill Crowd files are being maintained by Henrik.
• What other needs do people have for mozqa.com?

Automation Scripts / Infrastructure (Henrik)

• Download script has been enhanced to also download daily builds of Nightly or Aurora
• We can now even download Nightly builds from project branches - anyone interested in?

Personal Status

For the personal status please check the weekly status updates:

• 110620 - 110624
• 110627 - 110701

Roundtable

General

• Changes to our bi-weekly meeting regarding the org changes (Henrik)
  • We should setup preliminary pages which describe our new team next week
  • We should move the meeting pages below the new team page and call it "QA Automation Services Meeting"
• Review process (Henrik)
  • A more detailed discussion should happen during our work week in Cambridge
  • We have to find an interim solution to get not stuck in review requests
    • If you are not able to react to a review request from Softvision or contributors within one day, you should pass the review to another person
    • If there are issues you are running into, spread this information across the team. It takes lesser time to solve it together
• Discovery Pane tests
  • Do we want to allow tests to auto-skip themselves? If not we should mark those tests as skipped on default and aurora as long as the dependencies with WebDev are not solved.

Meeting Notes

• General
  • Geo: We have to rethink how we set the goals especially for those with dependencies

• Changes to the meeting
  • Matt: We want to cycle through all team members for driving a meeting
  • Henrik: Lets create the new wiki page directly after the current meeting when meeting notes and action items have been defined
  • Henrik: Everyone has to update the own content as early as possible so others have the chance to read through
  • Geo: Lets make the project updates just a read and ask questions part

• Q2 goals
  • API (shared modules) refactor
    • Geo: We will not be able to finish all the tasks we have set for this goal
    • Henrik: Compared to the former quarters we did a huge amount of work this quarter in any way
    • Matt: Deliver summary of what has been done
  • Test porting
    • Matt: Also deliver a summary for this project with tasks finished
  • Discovery Pane
    • Anthony: Good feeling that we can get it definitely done by end of this week
    • Henrik: I will work with Softvision on the final reviews during EU time zones
    • Matt: Deliver summary what has been achieved
  • Endurance
    • Dave: We can call it done
    • Matt: We need plans for getting the new investigated metrics implemented
  • On-demand
    • Geo: on-demand driven updates already rolled out and run perfect
    • Geo: currently working on BFT integration with Henrik
    • Geo: We will make this goal
  • TCM
    • Cam: We successfully completed the design of the new filtering mechanism
    • Cam: We gathered user stories. This is ongoing, but we accomplished what we needed for this quarter
    • Cam: Buy in was a debulous goal, so this was dropped
    • Cam: We released .3 features internally on schedule
    • Cam: .4 feature implementation changed scope so that it was to complete 7/15 (after Q2 end)
      • However, we also will miss that date, because we were delayed due to Platform instability

• Q3 goals
  • Most comments are in the etherpad, see goals section above
  • Overall QA goals
    • Henrik (mail): Make webapps goal more a goal to a generic test framework instead of Mozmill
    • Henrik (mail): We can't rely on Mozmill for e10s. Mozmill 2 will not be e10s ready and no schedule yet for the release
    • Matt: We should change the goal to add a good amount of endurance test enhancements
  • Team Goals
  • Geo: we should measure goals by 50-60% of our time

• Roundtable
  • Meeting pages have to be put into a new tree onto the wiki
  • David will drive the next meeting

Action items

• Geo: Deliver summary for API refactoring and test porting project by latest end of this week
• Anthony: Deliver summary for Discover Pane project by latest end of this week
• Henrik: Work with Softvision to ensure that DiscoPane tests will be checked in latest by Friday
• All: Reply to the discussion email about auto-skipping DiscoPane tests
• Matt: update QA overall goals
• Henrik/David: Finalize Q3 goals and get team feedback
• Henrik/David: Work out preliminary wiki pages for QA Automation Services team (incl. meeting page)
• Henrik/David: Come up with the first draft of the team structure and tasks