Auto-tools/Goals/2012Q4

From MozillaWiki
Jump to: navigation, search
  • Ensure B2G has all test automation available to prepare for launch
    • [ON TRACK] Deploy B2G Test Automation into TBPL running on Emulator and Panda platforms. Specifically we are turning on:
      • [DONE] Mochitest-plain (emulator first)
      • [SKIPPED] Mochitest-chome (emulator first)
      • [DONE] Reftest (emulator first
      • [DONE] xpcshell (emulator)
      • [DONE] webAPI (emulator only
      • [AT RISK] webQA - gaia front end tests emulator and panda
      • [AT RISK] gaia dev integration/unit tests emulator and panda
    • [DONE] Maintain B2G dog food update server as well as builds for development and QA while working to hand them off to releng
    • [DONE] Ensure all webAPIs testable through the emulator have comprehensive automated tests
    • [DONE] Continue to aid QA in creating automated tests for B2G
  • Expand and stabilize Test Automation on new and existing device platforms for NEON and ARMv6 support.
    • [DONE] Have APC.io (ARMv6)boards smoketest ready - rooted, working, plugged into Autophone
    • [DONE] Monitor deployment of large scale android and B2G panda board testing, aid with bug investigations
    • [DONE] Implement a new pool-based approach for device management in automation to increase stability on all mobile device testing
      • Version 1.1 released and deployed.
    • [DONE] Deploy Autophone (Noah's Ark) mochitest and robocop test abilities.
  • Create and deploy performance automation for desktop, fennec, and B2G product benchmarks
    • [ON TRACK] Create Eideticker tests for B2G on panda boards and run them once a day
    • [DONE] Monitor Datazilla webservices and ensure our accuracy in detecting performance regressions on a per-push basis
    • [DONE] Deploy Datazilla per-push UI to make it easier for developers to dig into the causes of performance regressions
  • Update our bugzilla instance to 4.2 providing many benefits from UI to helping people keep track of bugs/projects from within Bugzilla.
    • [DEFER] Deploy 4.2 version of Bugzilla
      • Deferred because of risk of deployment during critical b2g week. Rescheduled for early 2013.
    • [DONE] Ensure that bug migration tools are in place for B2G so that we can share bugs with outside services and partners.
    • [DONE] Complete the 'Sandstone' CSS update
      • Core work is done. Will probably need some tweaks as we go along. Regardless, won't be deployed until next year, with 4.2.
    • [DEFER] Rework browserID <--> BMO integration
      • Deferred in favour of some optimizations and github-migration scripts.
    • [DONE] Deliver user and product dashboards
      • As with the Sandstone skin, development is done but deployment delayed with 4.2 until next year.
    • [DEFER] Deploy tracking flags extension database optimization
      • Deferred as 4.2 is not going to be ready in time (new clusters not ready), given that the holidays will start shortly after its deployed. Targetting early 2013Q1.
  • Make the process of sheriffing easier by updating TBPL and helping make it easier to handle the intermittent test problem
    • [DONE] Aid with B2G changes so that B2G oranges tracked in Orange factor, B2G tests show up in TBPL
      • [DONE] Add TBPL support for B2G emulator/Panda/Unagi/Otoro
      • [DONE] Make TBPL's log parser compatible with mozharness-style output
      • [DONE] File/track B2G intermittent failures, report issues with mozharness' error detection/reporting & unhide builds/testsuites on TBPL once they pass consistently
      • [DONE] Add the new B2G18 tree to OrangeFactor
    • [DONE] Make it easier to sheriff and diagnose issues through better notification and automatic suggestions of issues for intermittent failures
      • [DONE] Add support for more types of failures to TBPL's log parser
      • [DONE] Improve test harness & buildbot output for failures & make the output more compatible with TBPL's parser
      • [DONE] Improve TBPL's bug suggestions for failures
      • [DONE] Make TBPL's tree open/closed status update more frequently
      • [DONE] Resolve timeouts during TBPL log parsing/summary generation
      • [DONE] Simplify the 'TBPL job -> recent history for that machine' workflow
      • [DONE] Obtain BuildVPN access & investigate BuildVPN-only tools and their use to sheriffs
      • [DONE] Simplify the TBPL -> 'viewing job on the buildbot master' workflow
    • [DONE] Split mochitest browser-chrome from mochitest-other to improve parallelism and overall turnaround time.
    • [ON TRACK] Make orange factor update more than once a day to help make it easier to see changes over time
    • [AT RISK] Assist with getting Datazilla's UI integrated with TBPL, phase 1
    • [DONE] Reduce regression windows for tree bustage by switching off unused builds/tests to reduce coalescing
    • [DONE] Reduce TBPL & OrangeFactor load on BzAPI/b.m.o, by updating use of BzAPI & transitioning to keywords rather than whiteboard annotations
    • [DONE] Fix our test-harness detection of crashes on Android
    • [DONE] Make it easier to differentiate open/resolved/test-disabled bugs on OrangeFactor, to improve prioritisation as well as finding potential backport candidates
    • [DONE] Track down intermittent-failures missing from OrangeFactor
    • [DONE] Make the tree-closure hook fail closed if treestatus is down
    • [DONE] Increase the SnR of dev.tree-management regression emails by suppressing small percentage changes & cleaning up the subject lines
  • Expand and Modernize the QA Automation framework as well as make it easier and more clear how to write automated tests for QA.
    • [DEFER] Re-organize and improve the automated testing documentation at MDN
    • [DONE] Develop first round of unit tests for WebRTC
    • [DONE] Ensure all B2G device WebAPIs testable through emulators have comprehensive low level test coverage
  • Projects Defered due to pivoting hard toward B2G
    • [DEFER] Port Eideticker to win32 and run tests there once a day to measure win32 snappiness for desktop
    • [DEFER] Create pageload benchmark with networking shaper tests so that we can adequately track user-perceived performance across all devices and network types
    • [DEFER] Complete REST API integration with Bugzilla
    • [DEFER] Update selenium grid to use ESX on-demand to improve web QA capacity - start date pushed back due to B2G
    • [DEFER] Finish and deploy mozmill 2.0 changes to frameworks and existing QA tests - start date pushed back due to B2G
    • [DEFER] Autophone goal scaled back and started later
    • [DEFER] Talos work scaled back as well.
Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Goals/2012Q4&oldid=496877"