QA/Test Automation/2011-02-09

From MozillaWiki
Jump to: navigation, search

previous meeting | Meetings | next meeting »

Dial in

 # 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL)
 # 1-800-707-2533 (pin 369) Conf# 315 (US)
 # #mozmill for backchannel


  • Anthony, Dave, Henrik, Aaron, Al, Matt

Last weeks action Items

  • [CARRY OVER] Update refactoring discussion items and communicate to team
  • [DONE] Make sure spreadsheet is updated with Panorama status
  • [DONE] Talk with Anthony to find the requirements for SSL certificates on the shadow server
  • Finish on-demand updates project page
  • Figure out what jetpack is doing for docgen (ask myk/analyze sdk)
  • Document failed test process from henrik's email at appropriate place
  • [DONE] Send Geo desired place for failed test docs
  • [DONE] Add 32-bit version of Firefox to our daily testruns on Ubuntu 64-bit
  • Matt/Al: work to find the goal for cloud automation goal

Goals Overview

Risky Goals

Project Status

On-demand update (bft) tests (Geo/Henrik)

  • Script to stage builds for testing is in the work; WIP can already read the configuration file and download all referenced builds bug 628659
  • We preferred to rely on a Pulse implementation instead of triggers via ssh

Endurance Tests (Dave/Anthony)

  • Awaiting review from Henrik
  • Looking into getting the current test name from Mozmill
  • Looking into an endurance test for bug 631494

Panorama Tests (Anthony/Al)

  • 9 / 21 tests now under development
  • no new tests landed yet, but we are getting close
  • New community contributor, adamFishHook
    • Undergrad at National University Singapore
    • found his way to Mozilla through researching open-source contribution
    • came on IRC, Aaron steered him toward Panorama
    • wants to spend a year working on Mozmill and apply learning to other Mozilla projects

Redesign Shared Modules (Geo/Henrik)

  • Geo started to work on sprints, Henrik will chime in this week
  • Documentation for several basic widgets has been started
  • jsdoc toolkit looks good so far except some minor issues with inheritance

Shadow test server (Al/Henrik)

  • This is bug 621794.
    • Al has pinged IT again for an update and an ETA for when this can go live. There has been no response on this since 2/2. We're gated on IT here.


Mozmill Crowd

  • Henrik had a talk at FOSDEM (slides)
  • We got some great feedback and were able to confirm some unknown problems
  • Change to non-blocking process execution is still in work - demo went fine


  • No updates

Cloud Automation (Al/Henrik)

  • This is blocked by needing IT to give us the server. This is bug 630426. Mrz and Phong are going back and forth on this.
  • Once we have a server, initial virtual machines will be put onto it.
  • Al is looking at the VMware command line tools for how we can manage VMs.

Tests / Shared Modules

General (Henrik)
  • First Mozmill test by a developer (bug 631052)
  • Improvements for l10n tests
  • Improvements for general tests when run in localized builds
Broken Tests
  • Firefox 4.0
    • Down to 1-2 constant failures (some random orange)
  • Firefox 3.5/3.6 (Anthony/Aaron)
    • 3.6: 6 failures (down from 11)
    • 3.5: 13 failures (down from 14)

Automation Scripts / Infrastructure (Henrik)

  • See on-demand update tests for the staging script

Personal Status

For the personal status please check the weekly status updates:


  • WebQA Test Automation status inclusion?

Meeting Notes

  • Shadow Server
    • Different vendors for SSL certificates
    • Those certificates will contain whitelists for subdomains
  • Cloud automation
    • Blocked in some areas from IT (server setup)
    • Research for initial plan is on Al's personal goals list
  • Endurance
    • Tests with add-ons or even other software (AV software) installed
    • Adding other kinds of metrics
  • Panorama
    • Partly overlap with Mochitests is wanted
  • Refactor Shared Modules
    • Check if we can acquire community members to help us on the documentation; mention it on the project page
  • Mozmill Crowd
    • Still searching for the best location for test environments
  • Web QA
    • How do we want to include WebQA automation?
      • Renaming Test Automation -> Desktop Test Automation (2 separate teams)
      • Or adding WebQA projects to the Test Automation team page
      • There are interests to also talk about WebQA related stuff during our meeting

Action items

  • Create project page for Cloud Testing project
  • Find out if we can get metrics for the upcoming qatest vm
  • Find out if it is possible for us to upload files to QMO (test environment)
  • [DONE] Add links to project pages for project headings
  • Create PHP script for downloading test environments