Auto-tools/Meetings/2012-07-16

From MozillaWiki
Jump to: navigation, search

Goals Progress

Below are our top level goals. Beneath each goal, note any progress made towards that goal this week.

  • [ON TRACK] Support the B2G and platform team by moving existing automation to that platform as well as expanding automation on B2G and B2G devices for basecamp deliverables.
  • [ON TRACK] Expand Mobile testing platforms for B2G and Fennec Native to include user devices, NEON Chipsets (panda boards), and ARMv6 (Beagle boards).
    • [mcote] Project plan in place for AutoPhone.
    • [mcote] Refactoring and self-tests.
  • [ON TRACK] Establish new Datazilla system as a useful tool for capturing and diagnosing talos performance regressions on a per-push basis, as well as capturing data for network and B2G performance tests.
    • NOTE: This is largely a change to the data harvesting and analysis side of the Talos system (even though some changes are required to the test harness and tests themselves). We will not be replacing existing Talos data harvesting system in Q3. By the end of Q3, we want to have the new performance data harvesting and analysis system Datazilla to a point at which it can shadow the old system and we can use Q4 to verify that the new approach captures just as many regressions as the old system does. Potential timeframe for conversion of the old system to the new Datazilla system is in Q1 2013 (earliest "thumb in the wind" estimate).
  • [ON TRACK] Upgrade BMO to 4.2, deploy tracking flags optimizations, UX improvements (comment tagging, mid-air resolution)
  • [ON TRACK] Make TBPL More Orange aware to make it easier to file intermittent orange bugs, and so that we can more adequately track (and therefore address) issues around android-related and memory leak intermittent failures
    • [edmorley] Fixed bug 772388 - Make ShutdownLeakLogger output the list of leaking tests in a format that can be individually starred.
  • [ON TRACK] Enable QA and contributors to increase their skills and participation by crafting automated tests for B2G and desktop smoketests by providing examples, mentoring, and documentation

Check out Pivotal Tracker

Highlights

This is a place for people to put announcements or to describe activities that they'd like to highlight. It's not the responsibility of the MC to find things to put here.

In-Depth Project Discussion

The Overview

Progress, notes, problems, and other statuses about current projects. Add any new projects to this template!

AutoPhone/Noah's Ark

  • [mcote] - Added a few unit tests.
  • [mcote] - Some refactoring and lots of documentation.
  • [mcote] - Developed project plan coordinated with releng's plans for foopy-less tegras.

Bughunter

Bugzilla

Eideticker

  • [wlach] - Finished merging Orangutan input simulation framework into master branch (see: http://wrla.ch/blog/2012/07/the-evolution-of-simulating-events-in-eideticker-from-monkeys-to-orangutns/)
  • [wlach] - Set up Nagios to monitor status of Eideticker machines / phones
  • [wlach] - Various work to try and make Eideticker dashboard update scripts more resilient (retrying failed jobs, etc.)
  • [wlach] - In Toronto this week to set up Eideticker machines and help anyone interested get up to speed on running/developing Eideticker tests.

Firebug Testing

  • [ahal] Option to specify custom database in order to move away from couchone

Jetperf

Marionette

  • [DBurns] Currently aligning client API with Selenium which allowed me to do what I did in the Open Web Apps section
  • [mdas] added a workaround for bug 771224 so chrome tests can run again.
  • [mdas] readjusting performance test runner job, adding one more perf test bug 774278

Mobile Automation

  • [jmaher] - investigating problem when boards reboot in the middle of a test with no indication of what is happening.
  • [jmaher] - investigating resource consumption during tests to see if that causes reboots/crashes/oranges.
  • [wlach] - various patches to SUTAgent and client library, mostly to fix eideticker-related stuff but may be useful elsewhere as well: commands now run as su by default (bug 763497), fall back to wifiman to get MAC address on older versions of Android (bug 768597), give better error when we fail to push a file (bug 772105), and don't consume tons of CPU when waiting for a response in the client (bug 773463).
  • [ahal] - b2g reftests running again after emulator fix and fix to bug 772307
    • continuing triaging process
  • [mihneadb] - got xpcshell tests running in b2g emulator
  • [mihneadb] - various fixes/refactorings to marionette and devicemanager

Mobile Evangelism Site-Scraping Tool

  • [samliu, carljm] fixed a number of bugs in the scraper, worked on writing tests

Mochitest

MozBase

Mozmill

MozTrap

[rbillings] - Doing a test run pass on the 1.1.X codebase

New Graphserver (Datazilla)

  • [carljm] Added support for Oauth to datazilla_client (also added tests, fixed a couple bugs, and added support for sending prebuilt JSON)
  • [camd] Added storage and importing of pushlog data, merged to master
  • [camd] Adding batch processing to certain management commands for cache updates and data processing. Added batch field to be able to assign projects to specific cron batches. (not yet merged)
  • [jeads] Pulled datazilla data for the talos test suite tp5n from 06/20/2012 to 07/05/2012 and ran t-test metrics on all of the associated pushes. This time range contains the changeset a9ae09d4f1d8 which caused a performance regression. Metrics is analyzing the data to further validate the t-test method.
  • [jeads] Cleaning up some loose ends to get ready for a release. This includes adding handling to set a default branch, making some schema changes, and cleaning up how reference data is cached.

Open Web Apps

  • [DBurns] I have marionette working logging in and installing web app. Login done via WebQA BrowserID Page Objects

Peptest

  • [ahal] Wrote test to catch unresponsiveness in tab switching
    • updating mozharness script to provide proxy sites

Powerball

Pulse

Signal From Noise

Speedtests

Systems

Talos

  • [jmaher] - worked with jhammel to get reporting to graph server for new sunspider, v8, kraken tests
  • [jmaher] - wrote blog post about compare.py for comparing talos numbers from graph server.
  • [jmaher] - backed out specialpowers for talos as it caused massive performance regressions.

Sheriffing & Trees

  • [edmorley] Day-to-day sheriffing of the trees.
  • [edmorley] Infra/tree closure issues:
    • Tracking down winxp pgo-only jsreftest failure (problematic due to not doing pgo-per-push & high levels of load at the time). Was due to bug 772303 and meant an 8-bug backout due to conflicts and dependencies :-/
    • Resolving several layers of webgl bustage, masked by coalesing.
    • Helping #it with debugging of bug 766810, which intermittently prevents people from pushing to inbound.
  • [edmorley] Updated all Wiki links for the clobberer to the new location & got IT to correct the revision in production, since the last rollout had been to the old site only, meaning different people were using different versions of the clobberer without realising it, *facepalm*.
  • [edmorley] Email discussions with joduinn about his idea of adding extra integration repos to reduce mozilla-inbound contention at peak times.

Treestatus

Tinderboxpushlog

  • [edmorley] Discussions about moving primary TBPL development out of a user repo and under hg.mozilla.org/{automation,webtools,build}.
  • [edmorley] Fixed bug 772084.
  • [edmorley] Items to ease starring/identification of failures more easily:
    • Fixed bug 772388 - Make ShutdownLeakLogger output the list of leaking tests in a format that can be individually starred.
    • Filed bug 772610 - results.py shouldn't say "Graph server unreachable" when the test name isn't recognised.
  • [edmorley] Posted about TBPL/OrangeFactor changes in dev.platform.

War on Orange

Web QA

Automation Development

  • [jmaher] - bug 762908 - add specialpowers to jsreftests - in review queue, required for landing ionmonkey

Upcoming Events

Standing weekly meeting related to our projects should go here

Round Table

  • Review the goals - are they specific enough while maintaining some amount of flexibility (since this is going to be a quarter that requires a ton of flexibility from us), do you have an idea how we'll measure them?
  • Discuss how to attack the Reftest Resolution problem for mobile. Come up with plan of attack and priority in terms of other actions.

Take notes here and copy and paste them to this section afterwards.

  • We discussed the goals, refining the language of them
  • We watched a fantastic presentation overview on peptest by Ahal
  • Held a long discussion around how to address the reftest screen size issue:
    • Reftest requires a resolution that is too large for mobile phones - need to figure out a way to reduce it so that we can perform true reftests on mobile and resolution constrained devices
    • Decided to change reftest to save all the canvases at the high resolution, then compare a subset of that canvas at our target small resolution, to determine tests that might fail (i.e. would be drawing anything out of the frame in the smaller resolution). We will generate a list of tests for further investigation from the set of tests that draw content outside the small resolution. Jmaher and ahal are going to take a crack at instrumenting reftest to support this and create the list.

Notes

Holidays and Trips

  • [DBurns] Tuesday 17 July off to sort house for window fitters

Next week MC

  • Ted will be the MC
  • Wlach will present Eideticker!!

Misc

Find something you think is cool, interesting, funny, or exciting and put it here

Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Meetings/2012-07-16&oldid=451200"