Auto-tools/Meetings/2013-01-14: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(18 intermediate revisions by 10 users not shown)
Line 1: Line 1:
== Goals Progress ==
== Goals Progress ==


Below are our [https://wiki.mozilla.org/Auto-tools/Goals/2012Q4 top level goals].  Beneath each goal, note any progress made towards that goal this week.
Below are our [https://wiki.mozilla.org/Auto-tools/Goals/2013Q1 top level goals].  Beneath each goal, note any progress made towards that goal this week.
* '''Ensure B2G has all test automation available to prepare for launch'''
 
** {{ok|Deploy B2G Test Automation into TBPL running on Emulator and Panda platforms. Specifically we are turning on:}}
* <insert goals here>
*** {{done|Mochitest-plain (emulator first, panda second)}}
*** {{skip|Mochitest-chome (emulator first, panda second)}}
*** {{done|Reftest (emulator first, panda second}}
*** {{ok|xpcshell (panda)}}
*** {{done|webAPI (emulator only}}
*** {{ok|webQA - gaia front end tests emulator and panda}}
*** {{ok|gaia dev integration/unit tests emulator and panda}}
** {{ok|Maintain B2G dog food update server as well as builds for development and QA while working to hand them off to releng}}
** {{ok|Ensure all webAPIs testable through the emulator have comprehensive automated tests}}
** {{ok|Continue to aid QA in creating automated tests for B2G}}
* '''Expand and stabilize Test Automation on new and existing device platforms for NEON and ARMv6 support.'''
** {{ok|Have APC.io (ARMv6)boards smoketest ready - rooted, working, plugged into Autophone}}
** {{ok|Monitor deployment of large scale android and B2G panda board testing, aid with bug investigations}}
** {{ok|Implement a new pool-based approach for device management in automation to increase stability on all mobile device testing}}
** {{ok|Deploy Autophone (Noah's Ark) mochitest and robocop test abilities.}}
* '''Create and deploy performance automation for desktop, fennec, and B2G product benchmarks'''
** {{ok|Create Eideticker tests for B2G on panda boards and run them once a day}}
** {{ok|Monitor Datazilla webservices and ensure our accuracy in detecting performance regressions on a per-push basis}}
** {{ok|Deploy Datazilla per-push UI to make it easier for developers to dig into the causes of performance regressions}}
* '''Update our bugzilla instance to 4.2 providing many benefits from UI to helping people keep track of bugs/projects from within Bugzilla.'''
** {{ok|Deploy 4.2 version of Bugzilla}}
** {{ok|Ensure that bug migration tools are in place for B2G so that we can share bugs with outside services and partners.}}
** {{ok|Complete the 'Sandstone' CSS update}}
** {{ok|Rework browserID <--> BMO integration}}
** {{ok|Deliver user and product dashboards}}
** {{ok|Deploy tracking flags extension database optimization}}
* '''Make the process of sheriffing easier by updating TBPL and helping make it easier to handle the intermittent test problem'''
** {{done|Aid with B2G changes so that B2G oranges tracked in Orange factor, B2G tests show up in TBPL}}
*** {{done|Add TBPL support for B2G emulator/Panda/Unagi/Otoro}}
*** {{done|Make TBPL's log parser compatible with mozharness-style output}}
*** {{done|File/track B2G intermittent failures, report issues with mozharness' error detection/reporting & unhide builds/testsuites on TBPL once they pass consistently}}
*** {{done|Add the new B2G18 tree to OrangeFactor}}
** {{done|Make it easier to sheriff and diagnose issues through better notification and automatic suggestions of issues for intermittent failures}}
*** {{done|Add support for more types of failures to TBPL's log parser}}
*** {{done|Improve test harness & buildbot output for failures & make the output more compatible with TBPL's parser}}
*** {{done|Improve TBPL's bug suggestions for failures}}
*** {{done|Make TBPL's tree open/closed status update more frequently}}
*** {{done|Resolve timeouts during TBPL log parsing/summary generation}}
*** {{done|Simplify the 'TBPL job -> recent history for that machine' workflow}}
*** {{done|Obtain BuildVPN access & investigate BuildVPN-only tools and their use to sheriffs}}
*** {{done|Simplify the TBPL -> 'viewing job on the buildbot master' workflow}}
** {{done|Split mochitest browser-chrome from mochitest-other to improve parallelism and overall turnaround time.}}
** {{ok|Make orange factor update more than once a day to help make it easier to see changes over time}}
*** Patch in progress, see {{bug|797811}}.
** {{risk|Assist with getting Datazilla's UI integrated with TBPL, phase 1}}
*** Awaiting finalised datazilla/talos output format, see: {{bug|816634}} & {{bug|794901}}.
** {{done|Reduce regression windows for tree bustage by switching off unused builds/tests to reduce coalescing}}
** {{done|Reduce TBPL & OrangeFactor load on BzAPI/b.m.o, by updating use of BzAPI & transitioning to keywords rather than whiteboard annotations}}
** {{done|Fix our test-harness detection of crashes on Android}}
** {{done|Make it easier to differentiate open/resolved/test-disabled bugs on OrangeFactor, to improve prioritisation as well as finding potential backport candidates}}
** {{done|Track down intermittent-failures missing from OrangeFactor}}
** {{done|Make the tree-closure hook fail closed if treestatus is down}}
** {{done|Increase the SnR of dev.tree-management regression emails by suppressing small percentage changes & cleaning up the subject lines}}
* '''Expand and Modernize the QA Automation framework as well as make it easier and more clear how to write automated tests for QA.'''
** {{ok|Re-organize and improve the automated testing documentation at MDN}}
** {{ok|Develop first round of unit tests for WebRTC}}
** {{ok|Ensure all B2G device WebAPIs testable through emulators have comprehensive low level test coverage}}


== Highlights ==
== Highlights ==


''This is a place for people to put announcements or to describe activities that they'd like to highlight.  It's not the responsibility of the MC to find things to put here.''
''This is a place for people to put announcements or to describe activities that they'd like to highlight.  It's not the responsibility of the MC to find things to put here.''
* Fennec x86
** android-x86
*** Installed android-x86 vms into ESXi, VMWare Fusion, VirtualBox. Android 2.2, 2.3 eeepc are supported. Later versions do not have ethernet support. Unfortunately can not run fennec x86 due to ABORT: OpenGL-accelerated layers are a hard requirement on this platform. Cannot continue without support for them. See: {{bug|771774}}, {{bug|778175}}. Are the performance and battery issues enough to continue to hard block non OpenGL devices? Being able to deploy android-x86 virtual machines for non-performance related testing may be a big enough win to allow non OpenGL devices.
** emulator-x86
*** Appear to be able to run fennec on emulator x86 on OSX with HAX installed. Still investigating.
* New intern has joined!
** please welcome Yiming to the team. She'll be with us for 4 months, working on Marionette
* [Henrik] We finally turned the knob and switched over to the new CI system for Windows and Linux (this includes a speed-up from 2h to 20min for ondemand update tests!!!)


== In-Depth Project Discussion ==
== In-Depth Project Discussion ==
Line 88: Line 41:


=== [[Project_Eideticker|Eideticker]] ===
=== [[Project_Eideticker|Eideticker]] ===
* [wlach] - Landed faster/better dashboard, new test manifest support (in preparation for B2G stuff)
* [wlach] - B2G dashboard results *almost* there
* [wlach] - Much of the data generated in December / early Jan was wrong. Trying to regenerate it.


=== [https://github.com/mozilla/Firebug-Test-Runner Firebug Testing] ===
=== [https://github.com/mozilla/Firebug-Test-Runner Firebug Testing] ===
Line 94: Line 50:


=== [[Auto-tools/Projects/Marionette|Marionette]] ===
=== [[Auto-tools/Projects/Marionette|Marionette]] ===
* [mdas] had to change how we send touch events in the abstraction layer for b2g, due to https://bugzilla.mozilla.org/show_bug.cgi?id=823619. Dealing with fallout now.


=== Mobile Automation ===
=== Mobile Automation ===
* [ahal] Larger set of reftests now running on B2G emulator arm ({{bug|811779}})


=== [[Auto-tools/Projects/Mobile_Evangelism_Site_Scraper|Mobile Evangelism Site-Scraping Tool]] ===
=== [[Auto-tools/Projects/Mobile_Evangelism_Site_Scraper|Mobile Evangelism Site-Scraping Tool]] ===
Line 102: Line 60:


=== [[Auto-tools/Projects/MozBase|MozBase]] ===
=== [[Auto-tools/Projects/MozBase|MozBase]] ===
* [mcote] Various mozdevice fixes: made sut_tests pass again, improved reboot-callback handling
* [jhammel] Various minor cleanup fixes; looking into getting as much of m-c on mozbase as possible soon


=== [[Auto-tools/Projects/Mozmill|Mozmill]] ===
=== [[Auto-tools/Projects/Mozmill|Mozmill]] ===


=== [[Auto-tools/Projects/MozTrap|MozTrap]] ===
=== [[Auto-tools/Projects/MozTrap|MozTrap]] ===
* [camd] Wrapping up final changes for this dev push.
* [camd] I will be moving away from dedicated dev on MozTrap to work on other projects.  New requests will be triaged on a case by case basis after 1/17.
* [camd] wrapping up scalability fix for multiselect for tags, suites and runs.
* [camd] Will release 1.4.0 this week


=== [[Auto-tools/Projects/Datazilla|New Graphserver (Datazilla)]] ===
=== [[Auto-tools/Projects/Datazilla|New Graphserver (Datazilla)]] ===
* [jeads] Updated https://datazilla.readthedocs.org
* [jeads] Built the in application help
* [jeads] Fixed cross browser UI layout bug
* [jeads] Adding feature to display the comparison revision for a given push to help validate the data.


=== [[Auto-tools/Projects/Open_Web_Apps_Testing|Open Web Apps]] ===
=== [[Auto-tools/Projects/Open_Web_Apps_Testing|Open Web Apps]] ===
Line 126: Line 94:


=== Sheriffing & Trees ===
=== Sheriffing & Trees ===
* [edmorley] Day-to-day sheriffing of the trees.
* [edmorley] Notifying top users of Try about TryServer best practices ({{bug|829932}})
* [edmorley] TryChooser improvements in production: https://bugzil.la/823711,823907,823909,827886
* [edmorley] Reviewed {{bug|813132}}.
* [edmorley] Patches:
** Trychooser: https://bugzil.la/827886
** Mozcrash/mozlog: https://bugzil.la/828324,829332,829773,829941
** Talos: https://bugzil.la/829334,829367
** Misc: https://bugzil.la/828946,782633


=== [https://treestatus.mozilla.org/ Treestatus] ===
=== [https://treestatus.mozilla.org/ Treestatus] ===


=== [https://tbpl.mozilla.org/ Tinderboxpushlog] ===
=== [https://tbpl.mozilla.org/ Tinderboxpushlog] ===
* [edmorley] New things in production: https://bugzil.la/828272,829092,826671,826028
* [edmorley] Patches:
** Parser: https://bugzil.la/829092
** UI/UX: https://bugzil.la/828272,764883,829935,829950


=== [[Auto-tools/Projects/WarOnOrange|War on Orange]] ===
=== [[Auto-tools/Projects/WarOnOrange|War on Orange]] ===
* [edmorley] Patch to suppress the weekly War on Orange email to dev.tree-management if the OF is 0.0 ({{bug|827856}})
* [edmorley] Ongoing backouts of newly introduced flaky tests & disabling of chronic top-oranges.


=== Web QA ===
=== Web QA ===
Line 150: Line 133:


=== Holidays and Trips ===
=== Holidays and Trips ===
* mcote out from Jan 21 - Feb 4 inclusive.
* jmaher out the week of Jan 28 - Feb 4 (with some internets during the time, email only)


=== Next week MC ===
=== Next week MC ===

Latest revision as of 18:04, 21 January 2013

Goals Progress

Below are our top level goals. Beneath each goal, note any progress made towards that goal this week.

  • <insert goals here>

Highlights

This is a place for people to put announcements or to describe activities that they'd like to highlight. It's not the responsibility of the MC to find things to put here.

  • Fennec x86
    • android-x86
      • Installed android-x86 vms into ESXi, VMWare Fusion, VirtualBox. Android 2.2, 2.3 eeepc are supported. Later versions do not have ethernet support. Unfortunately can not run fennec x86 due to ABORT: OpenGL-accelerated layers are a hard requirement on this platform. Cannot continue without support for them. See: bug 771774, bug 778175. Are the performance and battery issues enough to continue to hard block non OpenGL devices? Being able to deploy android-x86 virtual machines for non-performance related testing may be a big enough win to allow non OpenGL devices.
    • emulator-x86
      • Appear to be able to run fennec on emulator x86 on OSX with HAX installed. Still investigating.
  • New intern has joined!
    • please welcome Yiming to the team. She'll be with us for 4 months, working on Marionette
  • [Henrik] We finally turned the knob and switched over to the new CI system for Windows and Linux (this includes a speed-up from 2h to 20min for ondemand update tests!!!)

In-Depth Project Discussion

Pick one project/presenter per week who will give a detail presentation on their project.

The Overview

Progress, notes, problems, and other statuses about current projects. Add any new projects to this template!

AutoPhone/Noah's Ark

  • Temporarily began running unit tests on phones in haxxor. Need to develop reduced test manifests.
  • Currently rerunning s1s2 tests.

Bughunter

Bugzilla

Eideticker

  • [wlach] - Landed faster/better dashboard, new test manifest support (in preparation for B2G stuff)
  • [wlach] - B2G dashboard results *almost* there
  • [wlach] - Much of the data generated in December / early Jan was wrong. Trying to regenerate it.

Firebug Testing

Jetperf

Marionette

Mobile Automation

  • [ahal] Larger set of reftests now running on B2G emulator arm (bug 811779)

Mobile Evangelism Site-Scraping Tool

Mochitest

MozBase

  • [mcote] Various mozdevice fixes: made sut_tests pass again, improved reboot-callback handling
  • [jhammel] Various minor cleanup fixes; looking into getting as much of m-c on mozbase as possible soon

Mozmill

MozTrap

  • [camd] Wrapping up final changes for this dev push.
  • [camd] I will be moving away from dedicated dev on MozTrap to work on other projects. New requests will be triaged on a case by case basis after 1/17.
  • [camd] wrapping up scalability fix for multiselect for tags, suites and runs.
  • [camd] Will release 1.4.0 this week

New Graphserver (Datazilla)

  • [jeads] Updated https://datazilla.readthedocs.org
  • [jeads] Built the in application help
  • [jeads] Fixed cross browser UI layout bug
  • [jeads] Adding feature to display the comparison revision for a given push to help validate the data.

Open Web Apps

Peptest

Powerball

Pulse

Signal From Noise

Speedtests

Systems

Talos

Sheriffing & Trees

Treestatus

Tinderboxpushlog

War on Orange

  • [edmorley] Patch to suppress the weekly War on Orange email to dev.tree-management if the OF is 0.0 (bug 827856)
  • [edmorley] Ongoing backouts of newly introduced flaky tests & disabling of chronic top-oranges.

Web QA

Automation Development

Upcoming Events

Standing weekly meeting related to our projects should go here

Round Table

Put your questions and things to raise with the entire team here

Take notes here and copy and paste them to this section afterwards.

Notes

Holidays and Trips

  • mcote out from Jan 21 - Feb 4 inclusive.
  • jmaher out the week of Jan 28 - Feb 4 (with some internets during the time, email only)

Next week MC

Misc

Find something you think is cool, interesting, funny, or exciting and put it here

Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Meetings/2013-01-14&oldid=503353"