QA/Test Automation/2011-01-26

previous meeting | Meetings | next meeting »

Dial in

 # 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL)
 # 1-800-707-2533 (pin 369) Conf# 315 (US)
 # irc.mozilla.org #mozmill for backchannel

Last weeks action Items

• [MISSED] Geo: Update documentation for handling of failing Mozmill tests
  • At this point, have lost the thread on this. Need to touch base with Henrik to find out what final plan was and where docs should be updated.
• [DONE] Anthony: Update coding styles and keep discussion on-going to get remaining important rules defined

Goals Overview

• Q1 goals

Project Status

On-demand update (bft) tests (Geo/Henrik)

• Project page started [here], still filling in info.
• Henrik has made great progress on staging downloads
  • bug 528064
  • bug 628659
• Other part is simultaneous test launch
  • Currently planning on using ssh/dsh, or similar
  • Will be staging a test version on qa-mozmill

Endurance Tests (Dave/Anthony)

• Tracking bug created. bug 629065
• Patch uploaded for shared-module and initial test snippet. bug 629069
• Patch uploaded for endurance testrunner. bug 629077
• Created branch for endurance tests reports in Mozmill dashboard. bug 629083 repository
• Reports have been generated using the current patches and published to a couchone account:
  • Open/Close Add-ons Manager
    • Iterations: 5, 10, 20, 50, 100, 200

Panorama Tests (Anthony/Al)

• Community contributor (uyclay) has 4 tests which are nearly ready to checkin
• Drag&Drop issue bug 515776 "resolved"
• All tests should now be non-blocked, in theory
• Will be putting a call out soon to team for assignments (stay tuned)
• If you want to get a jump on it, see the list and assign something to yourself

Redesign Shared Modules (Geo/Henrik)

• Tentative project plan has been added [here]. Still undergoing review b/w Henrik and Geo, but will be finalized this week.
• Will be sending out periodic information for review by team as we hit milestones
• Plan on team-wide pilot at the end of Q1, where everyone can try implementing tests against the library and give feedback.
• Waverley has also expressed interest in ramping up into automation development; they can help us out a lot by being a "test community."

Shadow test server (Al/?)

• Initial Apache server being created in an IT-run ESX server. See bug 621794.
• Once server is up, will need to create some sort of rsync script to mirror content from QA repo initially.
• Will also need to determine what certificates we want for https testing.

Others

Mozmill Crowd

• Mozmill Crowd 0.1 and an improved version 0.1.1 has been released
• We already have some daily users (81 downloads, 8 active users)
• Working on project page updates and milestones for this quarter
• Next planned version of Mozmill Crowd by next week
• Mozmill Crowd talk during FOSDEM 2011 on Sunday, Feb 6th

Dashboard

• Working on project updates and milestones for this quarter
• Implementation of necessary views for l10n tests earlier this week

Cloud Automation (Al/?)

• Creating initial project page
  • Determining initial milestones, requirements, and dependencies.
  • This is investigation in the immediate, near-term

Tests / Shared Modules

General (Henrik)

• No progress - first Panorama tests still under review

Broken Tests

• Firefox 4.0
  • Landed a lot of fixes; down to one single test failure on Linux
• Firefox 3.5/3.6 (Anthony/Aaron)
  • Bug with window.close() not being properly handled (bug 626674) - fix should be landed today after branch release work, eliminating failures related to windows hanging around
  • Bug with testPasteLocationBar (bug 624682) - reverted from new controller MenuAPI to using the old controller click access of popup menu items, revealed another issue on 3.5 with keypress copying

Automation Scripts / Infrastructure (Henrik)

• Download script for builds has been finished and can be used
• Work happens now on the wrapper script to auto-start download and the test-run.

Personal Status

For the personal status please check the weekly status updates:

• 110117 - 110121
• 110124 - 110128

Roundtable

General

• Testing of 32bit builds on Ubuntu 64?
• Value of the execution of Mozmill tests for all locales (Henrik)
  • l10: A huge amount of issues has been covered by the testrun
    • Results
    • Screenshots
    • Tests take about 80 minutes
  • Update: 1 crasher bug has been identified for 3 locales (bug 628829)
    • Results
    • Tests take about 2h for partial updates
  • Pros vs. Cons
    • Check stability and quality of all localized builds
    • No need for manual update spot checks
    • Limited number of machines (what about the new order?)
    • We should test from Tier1 -> Tier3 (limit to Tier1 for smoketests and test channels?)
    • What about BFT tests? A single test-run for a locale will take about 12 minutes. We could run in parallel while Waverly tests manually

Issues