QA/Test Automation/2011-01-26

From MozillaWiki
Jump to: navigation, search

previous meeting | Meetings | next meeting »

Dial in

 # 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL)
 # 1-800-707-2533 (pin 369) Conf# 315 (US)
 # #mozmill for backchannel


  • Anthony, Dave, Aaron, Al, Cameron, Geo, Matt, Henrik

Last weeks action Items

  • [MISSED] Geo: Update documentation for handling of failing Mozmill tests
    • At this point, have lost the thread on this. Need to touch base with Henrik to find out what final plan was and where docs should be updated.
  • [DONE] Anthony: Update coding styles and keep discussion on-going to get remaining important rules defined

Goals Overview

Risky Goals

Project Status

On-demand update (bft) tests (Geo/Henrik)

  • Project page started here, still filling in info.
  • Henrik has made great progress on staging downloads
  • Other part is simultaneous test launch
    • Currently planning on using ssh/dsh, or similar
    • Will be staging a test version on qa-mozmill

Endurance Tests (Dave/Anthony)

  • Tracking bug created. bug 629065
  • Patch uploaded for shared-module and initial test snippet. bug 629069
  • Patch uploaded for endurance testrunner. bug 629077
  • Created branch for endurance tests reports in Mozmill dashboard. bug 629083 repository
  • Reports have been generated using the current patches and published to a couchone account:

Panorama Tests (Anthony/Al)

  • Community contributor (uyclay) has 4 tests which are nearly ready to checkin
  • Drag&Drop issue bug 515776 "resolved"
  • All tests should now be non-blocked, in theory
  • If you want to help out, see the list and assign something to yourself

Redesign Shared Modules (Geo/Henrik)

  • Tentative project plan has been added here. Still undergoing review b/w Henrik and Geo, but will be finalized this week.
  • Will be sending out periodic information for review by team as we hit milestones
  • Plan on team-wide pilot at the end of Q1, where everyone can try implementing tests against the library and give feedback.
  • Waverley has also expressed interest in ramping up into automation development; they can help us out a lot by being a "test community."

Shadow test server (Al/?)

  • Initial Apache server being created in an IT-run ESX server. See bug 621794.
  • Once server is up, will need to create some sort of rsync script to mirror content from QA repo initially.
  • Will also need to determine what certificates we want for https testing.


Mozmill Crowd

  • Mozmill Crowd 0.1 and an improved version 0.1.1 has been released
  • We already have some daily users (81 downloads, 8 active users)
  • Working on project page updates and milestones for this quarter
  • Next planned version of Mozmill Crowd by next week
  • Mozmill Crowd talk during FOSDEM 2011 on Sunday, Feb 6th


  • Working on project updates and milestones for this quarter
  • Implementation of necessary views for l10n tests earlier this week

Cloud Automation (Al/?)

  • Creating initial project page
    • Determining initial milestones, requirements, and dependencies.
    • This is investigation in the immediate, near-term

Tests / Shared Modules

General (Henrik)
  • No progress - first Panorama tests still under review
Broken Tests
  • Firefox 4.0
    • Landed a lot of fixes; down to one single test failure on Linux
  • Firefox 3.5/3.6 (Anthony/Aaron)
    • Bug with window.close() not being properly handled (bug 626674) - fix landed, will check testrun tomorrow to check for residual failures resolved
    • Bug with testPasteLocationBar (bug 624682) - reverted from new controller MenuAPI to using the old controller click access of popup menu items, revealed another issue on 3.5 with keypress copying

Automation Scripts / Infrastructure (Henrik)

  • Download script for builds has been finished and can be used
  • Work happens now on the wrapper script to auto-start download and the test-run.

Personal Status

For the personal status please check the weekly status updates:


  • Testing of 32bit builds on Ubuntu 64?
  • Value of the execution of Mozmill tests for all locales (Henrik)
    • l10: A huge amount of issues has been covered by the testrun
    • Update: 1 crasher bug has been identified for 3 locales (bug 628829)
      • Results
      • Tests take about 2h for partial updates
    • Pros vs. Cons
      • Check stability and quality of all localized builds
      • No need for manual update spot checks
      • Limited number of machines (what about the new order?)
      • We should test from Tier1 -> Tier3 (limit to Tier1 for smoketests and test channels?)
      • What about BFT tests? A single test-run for a locale will take about 12 minutes. We could run in parallel while Waverly tests manually

Meeting Notes

  • Endurance
    • A valuable resource of information would be graphs over graphs
    • Compare test-runs to see improvement/degradation of memory usage or performance
  • Shadow Test Server
    • We need a 2nd lead on the project.
    • Send an email to the list if you have interest
    • Otherwise Henrik will take the position
  • Mozmill Dashboard
    • Overlay results with appropriate bug ids
    • Overlay results with additionally meta data added by the logged in user
  • General
    • Everyone should check the own handling of projects and collect ideas for improvements in next quarter
    • For test failures we have to focus on 3.6 test failures before 3.5 failures

Action items

  • Update refactoring discussion items and communicate to team
  • Make sure spreadsheet is updated with Panorama status
  • Talk with Anthony to find the requirements for SSL certificats on the shadow server
  • Finish on-demand updates project page
  • Figure out what jetpack is doing for docgen (ask myk/analyze sdk)
  • Document failed test process from henrik's email at appropriate place
  • Send Geo desired place for failed test docs
  • Add 32-bit version of Firefox to our daily testruns on Ubuntu 64-bit
  • Matt/Al: work to find the goal for cloud automation goal