|
|
| (11 intermediate revisions by 8 users not shown) |
| Line 1: |
Line 1: |
| == Goals Progress == | | == Goals Progress == |
|
| |
|
| Below are our [https://wiki.mozilla.org/Auto-tools/Goals/2012Q4 top level goals]. Beneath each goal, note any progress made towards that goal this week. | | Below are our [https://wiki.mozilla.org/Auto-tools/Goals/2013Q1 top level goals]. Beneath each goal, note any progress made towards that goal this week. |
| * '''Ensure B2G has all test automation available to prepare for launch'''
| | |
| ** {{ok|Deploy B2G Test Automation into TBPL running on Emulator and Panda platforms. Specifically we are turning on:}}
| | * <insert goals here> |
| *** {{done|Mochitest-plain (emulator first, panda second)}}
| |
| *** {{skip|Mochitest-chome (emulator first, panda second)}}
| |
| *** {{done|Reftest (emulator first, panda second}}
| |
| *** {{ok|xpcshell (panda)}}
| |
| *** {{done|webAPI (emulator only}}
| |
| *** {{ok|webQA - gaia front end tests emulator and panda}}
| |
| *** {{ok|gaia dev integration/unit tests emulator and panda}}
| |
| ** {{ok|Maintain B2G dog food update server as well as builds for development and QA while working to hand them off to releng}}
| |
| ** {{ok|Ensure all webAPIs testable through the emulator have comprehensive automated tests}}
| |
| ** {{ok|Continue to aid QA in creating automated tests for B2G}}
| |
| * '''Expand and stabilize Test Automation on new and existing device platforms for NEON and ARMv6 support.'''
| |
| ** {{ok|Have APC.io (ARMv6)boards smoketest ready - rooted, working, plugged into Autophone}}
| |
| ** {{ok|Monitor deployment of large scale android and B2G panda board testing, aid with bug investigations}}
| |
| ** {{ok|Implement a new pool-based approach for device management in automation to increase stability on all mobile device testing}}
| |
| ** {{ok|Deploy Autophone (Noah's Ark) mochitest and robocop test abilities.}}
| |
| * '''Create and deploy performance automation for desktop, fennec, and B2G product benchmarks''' | |
| ** {{ok|Create Eideticker tests for B2G on panda boards and run them once a day}}
| |
| ** {{ok|Monitor Datazilla webservices and ensure our accuracy in detecting performance regressions on a per-push basis}}
| |
| ** {{ok|Deploy Datazilla per-push UI to make it easier for developers to dig into the causes of performance regressions}}
| |
| * '''Update our bugzilla instance to 4.2 providing many benefits from UI to helping people keep track of bugs/projects from within Bugzilla.'''
| |
| ** {{ok|Deploy 4.2 version of Bugzilla}}
| |
| ** {{ok|Ensure that bug migration tools are in place for B2G so that we can share bugs with outside services and partners.}}
| |
| ** {{ok|Complete the 'Sandstone' CSS update}}
| |
| ** {{ok|Rework browserID <--> BMO integration}}
| |
| ** {{ok|Deliver user and product dashboards}}
| |
| ** {{ok|Deploy tracking flags extension database optimization}}
| |
| * '''Make the process of sheriffing easier by updating TBPL and helping make it easier to handle the intermittent test problem'''
| |
| ** {{done|Aid with B2G changes so that B2G oranges tracked in Orange factor, B2G tests show up in TBPL}}
| |
| *** {{done|Add TBPL support for B2G emulator/Panda/Unagi/Otoro}}
| |
| *** {{done|Make TBPL's log parser compatible with mozharness-style output}}
| |
| *** {{done|File/track B2G intermittent failures, report issues with mozharness' error detection/reporting & unhide builds/testsuites on TBPL once they pass consistently}}
| |
| *** {{done|Add the new B2G18 tree to OrangeFactor}}
| |
| ** {{done|Make it easier to sheriff and diagnose issues through better notification and automatic suggestions of issues for intermittent failures}}
| |
| *** {{done|Add support for more types of failures to TBPL's log parser}}
| |
| *** {{done|Improve test harness & buildbot output for failures & make the output more compatible with TBPL's parser}}
| |
| *** {{done|Improve TBPL's bug suggestions for failures}}
| |
| *** {{done|Make TBPL's tree open/closed status update more frequently}}
| |
| *** {{done|Resolve timeouts during TBPL log parsing/summary generation}}
| |
| *** {{done|Simplify the 'TBPL job -> recent history for that machine' workflow}}
| |
| *** {{done|Obtain BuildVPN access & investigate BuildVPN-only tools and their use to sheriffs}}
| |
| *** {{done|Simplify the TBPL -> 'viewing job on the buildbot master' workflow}}
| |
| ** {{done|Split mochitest browser-chrome from mochitest-other to improve parallelism and overall turnaround time.}}
| |
| ** {{ok|Make orange factor update more than once a day to help make it easier to see changes over time}}
| |
| *** Patch in progress, see {{bug|797811}}.
| |
| ** {{risk|Assist with getting Datazilla's UI integrated with TBPL, phase 1}}
| |
| *** Awaiting finalised datazilla/talos output format, see: {{bug|816634}} & {{bug|794901}}.
| |
| ** {{done|Reduce regression windows for tree bustage by switching off unused builds/tests to reduce coalescing}}
| |
| ** {{done|Reduce TBPL & OrangeFactor load on BzAPI/b.m.o, by updating use of BzAPI & transitioning to keywords rather than whiteboard annotations}}
| |
| ** {{done|Fix our test-harness detection of crashes on Android}}
| |
| ** {{done|Make it easier to differentiate open/resolved/test-disabled bugs on OrangeFactor, to improve prioritisation as well as finding potential backport candidates}}
| |
| ** {{done|Track down intermittent-failures missing from OrangeFactor}}
| |
| ** {{done|Make the tree-closure hook fail closed if treestatus is down}}
| |
| ** {{done|Increase the SnR of dev.tree-management regression emails by suppressing small percentage changes & cleaning up the subject lines}}
| |
| * '''Expand and Modernize the QA Automation framework as well as make it easier and more clear how to write automated tests for QA.'''
| |
| ** {{ok|Re-organize and improve the automated testing documentation at MDN}}
| |
| ** {{ok|Develop first round of unit tests for WebRTC}}
| |
| ** {{ok|Ensure all B2G device WebAPIs testable through emulators have comprehensive low level test coverage}}
| |
|
| |
|
| == Highlights == | | == Highlights == |
| Line 71: |
Line 14: |
| ** emulator-x86 | | ** emulator-x86 |
| *** Appear to be able to run fennec on emulator x86 on OSX with HAX installed. Still investigating. | | *** Appear to be able to run fennec on emulator x86 on OSX with HAX installed. Still investigating. |
| | * New intern has joined! |
| | ** please welcome Yiming to the team. She'll be with us for 4 months, working on Marionette |
| | |
| | * [Henrik] We finally turned the knob and switched over to the new CI system for Windows and Linux (this includes a speed-up from 2h to 20min for ondemand update tests!!!) |
|
| |
|
| == In-Depth Project Discussion == | | == In-Depth Project Discussion == |
| Line 103: |
Line 50: |
|
| |
|
| === [[Auto-tools/Projects/Marionette|Marionette]] === | | === [[Auto-tools/Projects/Marionette|Marionette]] === |
| | * [mdas] had to change how we send touch events in the abstraction layer for b2g, due to https://bugzilla.mozilla.org/show_bug.cgi?id=823619. Dealing with fallout now. |
|
| |
|
| === Mobile Automation === | | === Mobile Automation === |
| | * [ahal] Larger set of reftests now running on B2G emulator arm ({{bug|811779}}) |
|
| |
|
| === [[Auto-tools/Projects/Mobile_Evangelism_Site_Scraper|Mobile Evangelism Site-Scraping Tool]] === | | === [[Auto-tools/Projects/Mobile_Evangelism_Site_Scraper|Mobile Evangelism Site-Scraping Tool]] === |
| Line 112: |
Line 61: |
| === [[Auto-tools/Projects/MozBase|MozBase]] === | | === [[Auto-tools/Projects/MozBase|MozBase]] === |
| * [mcote] Various mozdevice fixes: made sut_tests pass again, improved reboot-callback handling | | * [mcote] Various mozdevice fixes: made sut_tests pass again, improved reboot-callback handling |
| | * [jhammel] Various minor cleanup fixes; looking into getting as much of m-c on mozbase as possible soon |
|
| |
|
| === [[Auto-tools/Projects/Mozmill|Mozmill]] === | | === [[Auto-tools/Projects/Mozmill|Mozmill]] === |
|
| |
|
| === [[Auto-tools/Projects/MozTrap|MozTrap]] === | | === [[Auto-tools/Projects/MozTrap|MozTrap]] === |
| | * [camd] Wrapping up final changes for this dev push. |
| | * [camd] I will be moving away from dedicated dev on MozTrap to work on other projects. New requests will be triaged on a case by case basis after 1/17. |
| | * [camd] wrapping up scalability fix for multiselect for tags, suites and runs. |
| | * [camd] Will release 1.4.0 this week |
|
| |
|
| === [[Auto-tools/Projects/Datazilla|New Graphserver (Datazilla)]] === | | === [[Auto-tools/Projects/Datazilla|New Graphserver (Datazilla)]] === |
| * [jeads] Updated https://datazilla.readthedocs.org | | * [jeads] Updated https://datazilla.readthedocs.org |
| * [jeads] Built the in application help | | * [jeads] Built the in application help |
| | * [jeads] Fixed cross browser UI layout bug |
| * [jeads] Adding feature to display the comparison revision for a given push to help validate the data. | | * [jeads] Adding feature to display the comparison revision for a given push to help validate the data. |
|
| |
|
| Line 158: |
Line 113: |
|
| |
|
| === [[Auto-tools/Projects/WarOnOrange|War on Orange]] === | | === [[Auto-tools/Projects/WarOnOrange|War on Orange]] === |
| | * [edmorley] Patch to suppress the weekly War on Orange email to dev.tree-management if the OF is 0.0 ({{bug|827856}}) |
| | * [edmorley] Ongoing backouts of newly introduced flaky tests & disabling of chronic top-oranges. |
|
| |
|
| === Web QA === | | === Web QA === |
| Line 177: |
Line 134: |
| === Holidays and Trips === | | === Holidays and Trips === |
| * mcote out from Jan 21 - Feb 4 inclusive. | | * mcote out from Jan 21 - Feb 4 inclusive. |
| | * jmaher out the week of Jan 28 - Feb 4 (with some internets during the time, email only) |
|
| |
|
| === Next week MC === | | === Next week MC === |