QA/Test Automation/2010-09-22

From MozillaWiki
Jump to: navigation, search

previous meeting | Meetings | next meeting »

Dial in

 # 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL)
 # 1-800-707-2533 (pin 369) Conf# 315 (US)
 # #qa for backchannel


  • Adrian, Henrik, Anthony, Geo, Matt, Al, Aaron

Last weeks action Items

  • Geo: [MISSED] Send out list of broken tests w/ Fx4 and MM1.5
  • Matt: [DONE] Check what limits test harnesses on buildbot to use other local webservers (connect with John O'Duin)
  • Matt: [MISSED] Action items from release tasks meeting? Need to notify release leads that they're responsible for coordinating resources.
  • Geo: [DONE] For Auto-tools meeting next wk, how to incorporate train model going forward
  • Henrik & Matt: [MISSED] How to streamline patch checkin process
  • Geo: [MISSED] Generate wiki page to sum up tab candy objective

Goals Overview

  • Current status for Q3 goals
    • All: Set of tests for a main feature of Firefox 4
      • Can we make it an accomplished goal?
      • What takes lesser resources: Addons Manager vs. Panorama
    • Henrik: Dropping Mozmill Crowd extension?
    • Henrik: Web dashboard will be usable for the general test-run
    • Geo: Model for re-designing the shared modules
    • Adrian: DOM walker and set of l10n tests
  • Planning for Q4 (proposals)
    • QA Q4 goals
    • Henrik: First public version of the Mozmill Crowd extension
    • Henrik: Dashboard for software updates and addons tests
    • Henrik: Continue preparation of tests to get run on buildbot (incl. local data and stablization)
    • Henrik: Continue creation of tests for Panorama (performance?)
    • Henrik: Refactoring the shared module system
    • Henrik: Enhancements to the on demand testing infrastructure


Mozmill (Henrik)
  • Buildbot integration with 10 tests will happen today or tomorrow
  • Common JS module for a more streamlined module inclusion will be available with Mozmill 1.5.1 (smooth transition for Mozmill 2.0)
Test creation / shared modules
  • General (Henrik)
    • Creation of a PoC for Panorama tests during the QA workweek + Demo
  • DOM Walker + L10n API (Adrian)
    • see my personal notes
  • Redesign Shared Modules (Geo)
    • Got first "refactored" test working
Local Test Data (Anthony)
  • 4 r?, 11 checkin-needed, 10 need patches
Broken Tests
  • Firefox 4.0 (Geo)
  • Firefox 3.5/3.6 (Anthony)
    • No update
Update Tests (Henrik)
  • Fixed problem with missing fallback string in wiki results
  • Updated the SoftwareUpdateAPI for recent changes on trunk
  • bug 597664 gives us the chance to track slow mirrors
MozMill Crowd extension (Henrik)
  • No update
Add-ons Testing (Henrik)
  • Working with M.D.-Arche on Fidesfit client tests (bug 570493
Automation (Henrik)
  • No update
Dashboard (Henrik)
  • Staging server
  • Problems with filtering can be solved by using Ajax for view requests

Risky Goals

  • Mozmill Crowd extension


  • Last two weeks
    • [DONE] Attended CouchCamp
    • [DONE] QA workweek - Creation of a PoC for Panorama + work on presentation
    • [MISSED] Finish work on the dashboard (general testrun)
    • Fixes for TabbedBrowsingAPI and SoftwareUpdateAPI
  • Next two weeks
    • Finish work on the dashboard (general testrun)
    • Depending on the goals we want to achieve this quarter:
      • Continue work on the crowd extension
      • Update AddonsAPI and prepare list of Mozmill tests to implement
  • Last two weeks
    • [DONE] Patch review queue to 0
    • [DONE] Patch checkins to 0
  • Next two weeks
    • Review new patches to 0 (3 remain)
    • Commit new check-ins to 0 (11 remain)
    • Ensure 10 remaining local-data tests complete
    • Contribute 1 Panorama test
    • [AT RISK] QA results dashboard (if/when ready)
    • [AT RISK] QA crowd extension (if/when ready)
  • Last two weeks
    • Work on tab candy as possible during work week.
    • Finish carryover goals.
  • Next two weeks
    • Going to Bali.
  • Last two weeks:
    • refactored testAccessLocationBar.js, functional
      • Was functional as of last night, minorly broken again right now on a module chaining issue. Will have code pushed to personal hg as soon as that's fixed.
    • [CARRY OVER] collecting team feedback on prototype
    • [CARRY OVER] Fixes for bug 573582, bug 573584, bug 575250
    • Presentation w/ Aaron
      • Rough draft (info dump, really) created at Google Docs
    • QA work week
      • Finished first rev of performance checkpointing module
      • Demoed for Tab Candy team
      • Much hilarity had with all
  • Next two weeks:
    • Two more tests for refactor project, one from AwesomeBar, one from elsewhere.
    • Bug fixes for broken 4.0 tests
    • Any other necessary work to fulfill team goals
  • Last two weeks:
    • [CARRY OVER] of conversion of 31 tests to use local-data (patches and fixes upon reviews)
    • Attend QA work week
      • Presentation work w/Henrik & Geo
      • Panorama Challenge w/Henrik & Geo
      • local-data project work
  • Next two weeks:
    • Either divert focus to creating BFT focused Panorama tests with Anthony or carry on with local-data
  • Last two weeks:
    • [DONE] (mostly) finish Documentation + UML Diagrams for CommonA11yL10nAPI (renamed to: DOMUtilitiesAPI) and L10nAPI
    • [DONE] (mostly) implementation of the CommonA11yL10nAPI (renamed to: DOMUtilitiesAPI) and L10nAPI: WIP patch attached to bug 562084 - including two tests for the preferences window
  • Next two weeks:
    • implement the L10n tests for the update window
    • bring the 2 shared-modules and 4 tests through the reviews

Contributor Status

  • Name


  • Brasstacks
    • Issues with vhost/reversed proxy will remain for a while until VM's can be moved to Colo in Phoenix; AT team has to wait for Releng and prepare the move. Will be solved next quarter.
  • QA-Horus / QA-Set
    • Too less memory when running all VM's in parallel
    • Requested 32GB for both machines (bug 598305) - Matt has to approve

Meeting Notes

  • Last action items
    • Limits of the test harness on buildbot to use other local webservers
      • Pretty strict handling for accessing other servers
      • It would be possible to have tests running on a Talos server; Therefore a test environment has to be setup
  • Goals Q3
    • We drop the Mozmill crowd extension from our list
    • Dashboard will be ready for general test results by end of this week
    • Anthony and Aaron will finish the first part of the local test data project
    • Henrik updates the AddonsAPI and the existing tests until end of next week (Aaron will help out once the local data project is done)
    • Geo will have a solid plan for the refactoring work by end of this month
    • Adrian will work on the DOMWalker to have it ready by end of this month
  • Goals Q4 (Finalized by next week)
    • [CARRY OVER] Develop and deploy a Mozmill crowd-sourcing extension that will allow community users to execute and report results of Firefox 4 mozmill automated tests across the available platforms and configurations used by the Mozilla testing community.
    • [NEW] Integrate local data-based Firefox 4 mozmill tests under Buildbot execution control and reporting
    • [NEW] Collaborate with development team to expand Mozmill Panorama support to include an automated test suite for functional and performance testing of the Firefox 4 Panorama feature.
    • [NEW] Implement Mozmill Dashboard views for software update and addon tests
    • [NEW] Refactoring the shared module system based on the specification from Q3
  • Others
    • Geo: More often conversations about goals and blockers except our 1-1's and the bi-monthly meeting
      • We will setup our own IRC #mozmill channel and have an internal mailing list (qa-auto)
    • On demand test execution improvements
      • Geo: Shouldn't be a team goal regarding time constraints
      • Henrik: We can have at least the download script integration

Action Items

  • Geo: [CARRY OVER] Send out list of broken tests w/ Fx4 and MM1.5
  • Geo: schedule in-IRC meeting to do patch-handling proposal
  • Geo: [CARRY OVER] Generate wiki page to sum up tab candy objective
  • Henrik: Talk with John O'Duin about specifics of the test environment
  • Henrik: [DONE] Setup IRC channel #mozmill + Mailing list (qa-auto) [bug 598883]
  • Henrik: [DONE] Send out goals proposals for final rework
  • Matt: [CARRY OVER] Action items from release tasks meeting? Need to notify release leads that they're responsible for coordinating resources.