QA/Test Automation/2010-12-15

previous meeting | Meetings | next meeting »

Dial in

 # 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL)
 # 1-800-707-2533 (pin 369) Conf# 315 (US)
 # irc.mozilla.org #mozmill for backchannel

Attendees

• Aaron, Antony, Al, Henrik, Geo, Dave, Cameron

Last weeks action Items

• [DONE] Matt/Henrik: final leads for Panorama
• [DONE] Matt/Henrik: final priority list/leads for open project
• [DONE] Dave/Aaron: consolidate past discussion re: Endurance tests into one Etherpad
• [DONE] Dave/Aaron: project page for Endurance Tests
• [ON TRACK] Anthony: (Panorama) project page/tracking bugs/clarification of which tests to implement
• [DONE] Adrian: update L10N section of notes

Goals Overview

• QA Q4 goals

Project Status

Endurance Tests (Dave/Aaron)

• Project wiki page updated with consolidated notes taken during yesterdays meeting
  • Roadmap written with Q4 goal plan

Panorama Tests (Anthony/Al)

• Currently investigating existing tests for potential overlaps in testing

Triggered software updates (Geo/Henrik)

• Triggered Update Tester wiki page
• We are going with Pulse
• Have all pieces in place except the download code (am assured it's very simple)
• Are gluing pieces together today, targeting demo tomorrow

Mozmill Crowd Extension (Henrik/Aaron)

• Moved spawning external processes to nsIProcess
• Implemented downloader for test environment
• More stripped down version of the Windows test-environment; Linux and OS X under construction
• Talked with Jorge about a place to host test environments - HTTPS via QMO?

Redesign Shared Modules (Geo/Henrik)

• Shared Modules Refactor wiki page
• Have made lots of progress getting basic Element/Widget binding going
• Frequent code updates happening right now while map structure is defined
• See Geo's github repo for more details

Mozmill Result Dashboard (Henrik/Anthony)

• Mockups for update test results
• Investigating Elastic Search for CouchDB replacement

Build bot - local data (Anthony/Aaron)

• nothing new to report

Others

DOM Walker + L10n API (Adrian)

• Short updates

Tests / Shared Modules

• General (Henrik)
  • TabView shared module has been updated, there shouldn't be any dependencies anymore
  • Readme files in repository have been populated with content

• Test Refactoring (Anthony)
  • Phase 1 work defined
  • Need final say on use of "==" vs "===" in conditionals

• Broken Tests
  • Firefox 4.0 (Geo)
    • No progress
  • Firefox 3.5/3.6 (Anthony)
    • nothing new to report

Automation Scripts / Infrastructure

• • Short updates

Personal Status

For the personal status please check the weekly status updates:

• 101206 - 101210
• 101213 - 101217

Roundtable

General

Issues

• How we handle broken tests in order to make test runs and litmus syncs cleaner
  • We should fix broken tests as soon as possible, but history tells us they stick around for awhile so we need a process to handle them while in progress
  • Even under perfect circumstances, some tests are hard to fix. This process will be needed for those if no others.
    • Proposed flow
      1. Check to see if it has failed (twice?) during the last seven days, if so:
         1. Open a bug on the test failure
         2. Add whiteboard tag to the bug for tracking current failures
         3. Remove test from mozmill Litmus test suite (make sure it is still in BFT and/or Smoketest manual suites)
         4. Mark it for Mozmill as a skip test with the bug number from #2.
      2. Once fixed,
         1. Unmark skip
         2. Replace in Litmus
         3. Remove whiteboard tag
         4. Resolve bug

Meeting Notes

• Endurance Tests
  • It's not realistic to have a wrapper script by end of this year (personal constraints)
  • For now don't use a Python wrapper script, but a faked JSON object in the test itself to populate the data
  • Need to find more ideas what the exact next steps will be
  • We need more personal resources for coding Python

• Panorama
  • Work together with panorama devs
    • Aza is leaving, so Ian Gilman is our contact person now
    • Identifying extisting Mochitests one by one to find most useful Mozmill tests
  • Geo still has some performance stuff in petto but no time to complete it
    • Could be part of the endurance tests
    • Performance tests with checkpoints in the background (dependencies on Mozmill)
  • would be great to have 5 tests for the top nasty bugs by end of the year
    • Get list of high priority tests to get written
    • Most likely some restart tests

• Triggered Updates
  • With the work from today a demo should be possible tomorrow
    • Create a screencast for the demo

• Mozmill Crowd
  • Demo tomorrow during the QA Roadmap meeting

• Refactor Modules
  • High hierarchy map could be auto-generated by Mozmill
  • Silktest style for the map management, selenium style for page objects

• Dashboard
  • Sanity check for number of tests run - 0 passed tests should be treated as fail bug 620386
  • Matrix for release candidate testing next year
  • Elastic search
    • Get use cases so we can make sure it fits all our expectation

• Regression Test
  • What's the process how to create mozmill tests from bugzilla
  • Heathers regression tool (bisect)

• Others
  • Coding Style: Always use === in our tests

• How to treat test failures
  • Add litmus test id into bug
  • Always disable tests, even when they only fail on one single platform
  • Link from mozmill tests to litmus tests
    • We really shouldn't add id's to tests because of Litmus branches (Litmus 2?)
    • Should be easier via a Litmus search query