QA/Test Automation/2011-01-12

From MozillaWiki
Jump to: navigation, search

previous meeting | Meetings | next meeting »

Dial in

 # 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL)
 # 1-800-707-2533 (pin 369) Conf# 315 (US)
 # irc.mozilla.org #mozmill for backchannel

Attendees

  • Geo, Matt, Al, Henrik, Aaron, Anthony, Adrian

Last weeks action Items

  • Geo: Update documentation for handling of failing Mozmill tests
    • Did not update this, but the handling is somewhat up in the air right now. After the roundtable from this meeting should know better how to proceed.
  • [DONE] Anthony: Update coding styles and keep discussion on-going to get remaining important rules defined

Goals Overview

  • Proposals
    • Triggered Update Tests
      • Integrate triggered Mozmill update tests into nightly automation process
        • One goal, but lower priority because it's an incremental change
      • what about release candidate builds? For Q2?
        • Release candidate tests are the higher priority for Q1. They will be on-demand, rather than triggered
        • Separate goal
        • Measurement will be "Matt can run them."
    • Endurance Tests
      • Create suite of Mozmill tests to stress-test Firefox in different areas
      • Performance tests
    • Module Refactoring
      • Finalize the project by updating modules and tests
        • Only for mostly used modules and tests for the beginning (focus areas)
        • Updates per component
    • Collaborate with development teams to get XXX BFT tests automated
      • We never found a good way to specify a realistic goal, whether via number of tests nor per component. Do we have other options?
      • Given the Firefox 4 priority and other goals, how much time we would have this quarter?
        • Note: let's carry over panorama test goal for this.
      • Should we better move our efforts to keep the testsuite green?
        • Not a Collaboration goal, can be a process goal
    • Testing Server
      • Setting up a testing server which allows kill our dependencies on external websites.
        • iterative effort
        • need to create a feature milestone implementation plan to define what will be delivered in chronological order

Risky Goals

Project Status

Panorama Tests (Anthony/Al)

  • 4 / 15 doable tests currently in development/review
  • 6 tests currently blocked by drag & drop (bug 515776)
  • Will be looking for team assistance once 4.0b9 work dies down (early next week)
  • Proposed Team Goal: All non-blocking tests automated

Software updates (Geo/Henrik)

  • Finished PoC for triggered nightly updates.
    • Takes Pulse message
    • Downloads appropriate build
    • Runs update tests, including checking that end-build == expected build ID.
  • Emphasis on updates for next few weeks will be streamlining on-demand release updates, since that's expected to be the immediate process pain point

Mozmill Crowd Extension (Henrik/Aaron)

  • Version 0.1 is live on AMO
  • We have to inform localizers ASAP so they can run our tests against their locales and fix remaining issues before RC1
  • Version 0.2 planned for end of this month (non-blocking process)
  • Submitted proposal for my talk at FOSDEM 2011

Endurance Tests (Dave/Aaron)

Redesign Shared Modules (Geo/Henrik)

  • PoC stage reached in December, running "toy" tests.
  • Currently porting real tests over to shake out early-stage issues.
  • Determining a milestone-based plan including pilot periods where the team as a whole can help contribute to porting (and also test out how easy the framework is to develop against)

Mozmill Result Dashboard (Henrik/Anthony)

  • Update tests now have their own views
  • A couple of minor issues identified which have to be fixed

Build bot - local data (Anthony/Aaron)

  • No status
  • Most likely will not see progress this quarter due to Firefox 4 work and Automation priorities

Others

DOM Walker + L10n API (Adrian)
  • Slower progress but working on the last major items to get fixed by Adrian
    • For the most important bug left (bug 614579), we need input from people knowing about theming stuff, as it seems to be a Firefox bug...
  • Can be executed by the Crowd extension
Tests / Shared Modules
  • General (Henrik)
    • Nothing major to report
  • Test Refactoring (Anthony)
    • Following is moot until Shared Modules are refactored
    • project in place to begin once Firefox 4b9 work dies down
    • will need team assistance to complete everything this quarter (please let me know if you are willing to help)
    • we still have a few outstanding items for discussion
      • if possible, I'd like to discuss these during the team meetings (resolve 1 per meeting)
      • agreement would be integrated into refactoring, not styleguide
  • Broken Tests
    • General
      • We have started combined efforts to get some of the failing tests fixed
      • We have now owners for the different sections which have to take care of broken tests
    • Firefox 4.0 (Geo)
      • No particular branch-based news, see General above and Round Table below.
    • Firefox 3.5/3.6 (Anthony)
      • Focusing on 4.0 failures for b9 release
      • Will focus on branch failures once b9 is out
Automation Scripts / Infrastructure
    • Download script for builds from FTP is in progress. It already works for release and candidate builds

Personal Status

For the personal status please check the weekly status updates:

Roundtable

General
  • Styleguide & Refactoring - on hold until Shared Module refactoring complete
    • we should fork into two separate projects
    • freeze the styleguide
    • new additions go into refactoring work
    • changes/additions to styleguide should reflect "style" not logical/design changes
  • Failures
    • adopt a module-ownership model to deal with all failures (4.0, 3.6, and 3.5)
    • once we are green, go back to a branch-based model or stay with module-ownership?
Issues

Meeting Notes

Action items

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Test_Automation/2011-01-12&oldid=277458"