Auto-tools/Meetings/2012-11-12

From MozillaWiki
Jump to: navigation, search

Goals Progress

Below are our top level goals. Beneath each goal, note any progress made towards that goal this week.

  • Ensure B2G has all test automation available to prepare for launch
    • [ON TRACK] Deploy B2G Test Automation into TBPL running on Emulator and Panda platforms.
      • Mochitests, reftests, and WebAPI tests are all running on emulators on TBPL, on all the trees that they're needed. All suites need additional coverage.
    • [ON TRACK] Maintain B2G dog food update server as well as builds for development and QA while working to hand them off to releng
    • [ON TRACK] Ensure all webAPIs testable through the emulator have comprehensive automated tests
    • [ON TRACK] Continue to aid QA in creating automated tests for B2G
  • Expand and stabilize Test Automation on new and existing device platforms for NEON and ARMv6 support.
    • [ON TRACK] Have APC.io boards chassis-ready -- rooted, smoketested, and working for ARMv6 testing
    • [ON TRACK] Monitor deployment of large scale android and B2G panda board testing, aid with bug investigations
    • [ON TRACK] Implement a new pool-based approach for device management in automation to increase stability on all device testing
    • [ON TRACK] Deploy Autophone with 40 phones and 70% up time
  • Create and deploy performance automation for desktop, fennec, and B2G product benchmarks
    • [ON TRACK] Create Eideticker tests for B2G on panda boards and run them once a day
    • [ON TRACK] Port Eideticker to win32 and run tests there once a day to measure win32 snappiness for desktop
    • [ON TRACK] Create pageload benchmark with networking shaper tests so that we can adequately track user-perceived performance across all devices and network types
    • [ON TRACK] Monitor Datazilla webservices and ensure our accuracy in detecting performance regressions on a per-push basis
    • [ON TRACK] Deploy Datazilla per-push UI to make it easier for developers to dig into the causes of performance regressions
  • Update our bugzilla instance to 4.2 providing many benefits from UI to helping people keep track of bugs/projects from within Bugzilla.
    • [ON TRACK] Complete the 'Sandstone' CSS update
    • [ON TRACK] Complete the REST API integration
    • [ON TRACK] Rework browserID <--> BMO integration
    • [ON TRACK] Deliver user and product dashboards
    • [ON TRACK] Deploy tracking flags extension database optimization
  • Make the process of sheriffing easier by updating TBPL and helping make it easier to handle the intermittent test problem
    • [ON TRACK] Aid with B2G changes so that B2G oranges tracked in Orange factor, B2g tests show up in TBPL
    • [ON TRACK] Make it easier to sheriff and diagnose issues through better notification and automatic suggestions of issues for intermittent failures
      • TBPL patch to allow distinguishing between Android Tegra and Android Panda now in production.
      • Various patches and bugs filed to make mozharness+TBPL more compatible with each other.
      • Investigation started into test harness timeout support issues, that limit TBPL starring potential.
    • [DONE] Split mochitest browser-chrome from mochitest-other to improve parallelism and overall turnaround time.
    • [ON TRACK] Make orange factor update more than once a day to help make it easier to see changes over time
    • [ON TRACK] Assist with getting Datazilla's UI integrated with TBPL, phase 1
  • Expand and Modernize the QA Automation framework as well as make it easier and more clear how to write automated tests for QA.
    • [ON TRACK] Finish and deploy mozmill 2.0 changes to frameworks and existing QA tests
    • [ON TRACK] Update selenium grid to use ESX on-demand to improve web QA capacity
    • [ON TRACK] Re-organize and improve the automated testing documentation at MDN
    • [ON TRACK] Develop tests for WebRTC, B2G and other high profile projects

Highlights

  • [henrik] We will have our first automation testday this Wednesday (Nov 14th) which is about B2G and getting people ready to run tests on devices or the emulator.

In-Depth Project Discussion

Pick one project/presenter per week who will give a detail presentation on their project.

The Overview

Progress, notes, problems, and other statuses about current projects. Add any new projects to this template!

AutoPhone/Noah's Ark

  • [bc] Have local versions of robocop, mochitest, reftest, crashtest and jsreftest running. Issues with displaying autolog results and logging that need to be worked out.
  • [mcote] Work on MozPool continues. It's been merged with bmm so it will take a bit more work to get it back into Autophone; focus is on deploying for B2G first.

Bughunter

Bugzilla

Eideticker

Firebug Testing

Jetperf

Marionette

  • [DBurns] Landed {bug|801733} which is a new visibility bug and fixed in the Selenium tree

Mobile Automation

  • [jmaher] - reftest resolution change landed, we can run on small resolutions without orange
    • still have to fix the tests

Mobile Evangelism Site-Scraping Tool

Mochitest

MozBase

  • [wlach] The usual set of fixes/improvements to mozdevice in flight

Mozmill

MozTrap

  • [camd] Split the work feature is done, and under test
  • [camd] need to talk to ctalbert and bmoss about continuing work on MozTrap.
  • [camd] upgrade to django 1.4 unblocked and will be submitted
  • [camd] working on performance improvements to test runs that exceet 1000 cases.

New Graphserver (Datazilla)

Open Web Apps

Peptest

Powerball

Pulse

Signal From Noise

Speedtests

Systems

Talos

Sheriffing & Trees

  • [edmorley] Was on PTO Friday.
  • [edmorley] Day-to-day sheriffing of the trees.
  • [edmorley] Filed bugs for the majority of cases where mozharness regresses TBPL starring capability (see dependants of bug 778688).
  • [edmorley] More preparation work for the switch from "[orange]" in whiteboard to keyword.
  • [edmorley] Investigation into bug 808410.
  • [edmorley] Patch for bug 808545.
  • [edmorley] Created a bookmarklet-based script to simplify the TBPL -> 'viewing job on the buildbot master' workflow (bug 809908).
  • [edmorley] Investigation into maxTime & timeout inconsistencies in our various test harnesses, causing wasted infra capacity due to excessive timeouts, as well as meaning we hit buildbot timeouts rather than handling in-harness and so lose debugging info & ability for TBPL to star.

Treestatus

Tinderboxpushlog

War on Orange

  • [edmorley] OrangeFactor patch for bug 807930.
  • [edmorley] Major drive this week to lower our OrangeFactor - is now ~25% lower than what it was the week before.
  • [edmorley] Ongoing backports of orange fixes to aurora/beta/esr.

Web QA

Automation Development

  • [rwood] WebSMS test development complete (pending post-review changes); now starting Proximity Sensor

Upcoming Events

Standing weekly meeting related to our projects should go here

Round Table

Put your questions and things to raise with the entire team here

Take notes here and copy and paste them to this section afterwards.

Notes

Holidays and Trips

Next week MC

Misc

  • [mcote] gave a presentation (slides here) entitled "Network Programming for Automation Developers" at the first ever PyCon Canada.
Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Meetings/2012-11-12&oldid=486791"