Auto-tools/Goals/2012Q1

From MozillaWiki
Jump to: navigation, search

Official Q1 Goals

  • Native Fennec Automation Support and rollout
    • Ensure Reftest, Robotium are working with less than 5% intermittent orange in automation
    • Ensure Cross browser onload startup testing is running with an uptime of 90%.
    • Wire checkerboarding analysis (native vs xul first, then cross browser) into Eideticker
  • Complete Phase I of Talos Signal from Noise
    • Complete Milestone 1 so that we can implement Stephen's first recommendation of throwing out the first value on a test rather than the highest value.
    • Complete Milestone 2 to make recommended changes on TdHTML and verify the results with metrics.
    • Complete Milestone 3 so that our ability to track and analyze our talos data (especially the new data being generated) is improved alongside the tests themselves.
  • Complete Build & Test Automation system for B2G
    • Land Marionette into m-c
    • Ensure Marionette has full JSON protocol test running support
    • Complete end-to-end build and test infrastructure ensure support can be rolled out as needed using ESXi/EC
    • Implement and deploy ability for developers to write automated tests for Gaia apps
    • Running on-checkin build and test by end of quarter for DOM backend and gaia apps
  • Secure and modernize dashboard and automation infrastructure
    • Build and deploy an IT managed webserver for orange factor, autolog, cross browser startup
    • Remove couchdb from all systems in favor of elastic search
    • Work with IT to make Pulse easier to use and more reliable

Crucial Projects

These projects must be completed to achieve the above goals.

GOAL: Native Fennec Automation Rollout

  • Q1 Outcomes
    • [DONE] Deploy reftest to native fennec with less than 5% intermittent orange rate
    • [DONE] Deploy robotium for UI automation testing with less than 5% intermittent orange rate
    • [DONE] Deploy robotium based tpan test
    • [DONE] Deploy robotium based tzoom test
    • [DONE] Ensure crossbrowser onload startup testing achieves uptime of 90%
    • [DONE] Wire checkerboarding analysis (native vs xul first, then cross browser) into Eideticker
    • [DONE] Provide Native Fennec build on ARMv6
  • Stakeholders
    • Mobile QA
    • Mobile Developers
  • Depends On
    • Releng for deployment
    • Mobile developers for feedback/fixing issues uncovered in product
    • Eideticker Project

GOAL: Complete Phase I of Talos Signal From Noise

  • Q1 Outcomes
    • [DONE] Complete Milestone 1 so that we can implement Stephen's first recommendation of throwing out the first value on a test rather than the highest value.
    • [SKIPPED] Complete Milestone 2 to make recommended changes on TdHTML and verify the results with metrics.
    • [DONE] Complete Milestone 3 so that our ability to track and analyze our talos data (especially the new data being generated) is improved alongside the tests themselves.
    • [DONE] Decided to focus on tp5 since it is the most quoted and used metric in talos and also one of the most noisy. Goal is to reduce the noise by changing how the data is collected
  • Stakeholders
    • Desktop and mobile developers
  • Community involvement
    • Extensive community involvement surface area/low-hanging bugs
  • Depends On
    • Metrics for further research/feedback on directions
    • Releng for deployment
    • Webdev for graph server changes/deployment

GOAL: Complete Build and Test Automation for B2G

  • Q1 Outcomes
    • [DONE] Ensure all bugs fixed in test automation system
    • [DONE] Procure machines/vms for test slaves
    • [DONE] Land Marionette in m-c
    • [MISSED] Implement full test driver protocol in Marionette
      • NOTE: We decided to use full selenium atom support for this and won't have it done by EOQ
    • [DONE] Get Security Review for Marionette
    • [DONE] Implement and deploy ability for developers to write automated tests for Gaia apps
    • [DONE] Deploy on-checkin automated testing for backend (DOM based) and gaia apps
    • [DONE] Implement dashboard for B2G on Auto-log
  • Stakeholders
    • B2G Team, WebAPI team
  • Community Involvement
    • Ensure effort is more visible and bugs are adequately marked for community involvement
  • Depends On
    • Remote debugging protocol landing in m-c
    • Sec review for landing in m-c
    • A11y support in B2G

GOAL: Secure and modernize dashboard and automation infrastructure

  • Q1 Outcomes
    • [DONE] Build and deploy an IT managed webserver for our dashboards: orange factor, autolog, cross browser startup
      • NOTE: We managed to get orange factor and autolog stood back up. Due to IT-lag, we had to get cross browser startup hosted on an external site.
    • [MISSED] Remove couchdb from all systems in favor of elastic search
      • NOTE: IT support to make this happen came too late.
    • [MISSED] Work with IT to make Pulse easier to use and more reliable
      • NOTE: IT has not powered on new hardware for Pulse yet, we also only managed to identify the cause of the instability in pulse, we have not yet fixed it.
  • Stakeholders
    • Infastructure Security
    • QA Automation Engineering
    • Platform Engineering
    • Ourselves
  • Depends On
    • Pulse Stabilization Project (see below)
    • Metrics for extending Elastic Search use

Supporting Projects

Many of our projects are inter-related, and there are many supporting projects that we want to call out as they are important and cannot be entirely neglected in favor of these goals. This section calls these out.

Bughunter

  • Rollout the Bughunter system to the community of users as the project has reached completion. It will enable our QA team to more easily diagnose reproducible crashes and provide that information to development.
  • Q1 Outcomes
    • [DONE] Conduct Rollout Training for Security/QA team
    • [DONE] Implement safe mechanisms/policies for working with system
    • [DONE] Find/fix with UI given feedback from users
  • Stakeholders
    • Security team
    • Crash kill/Project Mgmt
    • QA

Bugzilla

  • Pave the way toward better Bugzilla integration with our tools and Bugzilla 5.0
  • Q1 Outcomes
    • [MISSED] Deploy 4.2 Upgrade
    • [MISSED] Change the way the Release Tracking Flags are implemented in Bugzilla so they are more sustainable as we move forward
    • [MISSED] Implement REST API native to bugzilla
    • [MISSED] Implement PULSE integration
      • NOTE: A bunch of things happened here. IT was unable to finish hardware upgrade/move for 4.2. Also, great work was done on both REST API and Pulse but it was not completed due to the Bugzilla guys completing a bunch of extensions for a bunch of different groups - first patch extension, secure mail extension, tell us more extension, etc. We need to work on better focus here, and that will happen in Q2.
  • 'Depends On
    • Pulse stabilization
    • IT support, networking between SJC/PHX

DXR Support

  • Ensure that DXR unit tests are running as we move toward making DXR production so that MXR can be retired
  • Q1 Outcomes
    • [SKIPPED] Ensure DXR automated tests conform to logging/return code standards for Buildbot automation
    • [SKIPPED] Understand and document how to run DXR automated tests
    • [MISSED] Work with Releng to deploy DXR tests into automation
      • NOTE: There are no DXR tests, evidently. This was a miscommunication. We are instead working to ensure the DXR build is properly integrated into releng automation.
      • NOTE: Unable to deploy builds into releng because we could not get vm from IT to put our git->hg mirror on in order to get the code into hg so releng could work with it.
  • Stakeholders
    • DXR Team
    • Web Tools team
  • Depends On
    • Releng for deployment

Eideticker

  • Create a Mountain View Eideticker system and demo its capabilities to developers.
    • Support the Native Fennec goal of automating the visual startup performance tests
  • Q1 Outcomes
    • [DONE] Set up mountain view Eideticker machine
    • [DONE] Perform Checkerboard measurement demo in MV
    • [DONE] Set up Eideticker measurement of native vs xul checkerboarding
    • [DONE] Set up Eideticker measurement of cross browser checkerboarding
    • [SKIPPED] Analyze Eideticker checkerboarding measurements using video capture for phones that do not have HDMI out, aim to enable us to measure checkerboarding within 10% of internal telemetry measurements
    • [DONE] Get Eideticker running with pandaboard (stretch)
      • NOTE: There wasn't enough interest at this time to spend effort on working with phones that do not have HDMI out. So we skipped video camera integration at this time
  • Stakeholders
    • Mobile Developers
    • Mobile QA
    • B2G Team
  • Depends On
    • None

Flying Tanks Staging Server

  • Enable the A*team to work with a staging environment for all the systems running on our brasstacks web server. This will ensure better up-time and provide an easier contribution path for contributors interested in aiding with A*team web development.
    • Will also enable faster turn around on A*team web projects
  • Q1 Outcomes
    • [DONE] Ensure Staging vms created properly on PHX cluster
    • [MISSED] Move all systems from brasstacks to staging system
    • [MISSED] Recreate brasstacks system as a clone of staging environment
    • [DONE] Create methodology for deployment from staging to production with IT
      • NOTE: We did finally get IT support for the VMs but it came late in the quarter, most of the quarter was held up by IT blocking on getting us proper VLANs created.
  • Stakeholders
    • (many)
  • Depends On
    • IT support

Internal Pypi Server

  • Implement an internal pypi-like server so that our tools running inside restricted environments (like build VPN) can use normal python dependency management systems
  • Q1 Outcomes
    • [DONE] Get Requirements from Release Engineering on what to build
    • {{miss|Deploy or reuse a machine in the MPT colo that has access from buildVPN and internal networks (so that it can be used by both automation and staging systems)
    • [DONE] Build the system with tests for functionality.
    • [MISSED] Deploy to a machine/vm within the MPT colo network
      • NOTE: Still do not have VM from IT to use for this.
  • Stakeholders
    • Release Engineering
  • Depends On
    • IT for machine/vm procurement and networking setup

Jetpack Talos Test

  • The jetpack team would like a talos test running on their tree
  • Q1 Outcomes
    • [DONE] Define what the talos test should be, what it should measure, and how it should be deployed
    • [DONE] Define other work -- TBD
      • NOTE: This was defined and was worked on. It hit a wall with mozharness integration, when that was resolved, we realized that it was still blocked on the pypi server, which in turn was blocked on IT.
  • Stakeholders
    • Jetpack Team (aka addons-sdk team)
  • Depends On
    • Jetpack team for complete definition of requirements
    • Releng for deployment

Mozbase Internal Automation

  • Provide sanity check for our mozbase software stack as we start using the software across several projects to reduce possibilities for regressions.
  • Q1 Outcomes
    • [DONE] Implement buildbot based system to run mozbase tests
    • [MISSED] Deploy to widows and linux vm and mac mini as test slaves
    • [DONE] Run tests on checkin to mozbase repo
    • [MISSED] Report results to autolog
      • NOTE: Dropped in favor of higher priority projects, still have no VMs.
  • Stakeholders
    • A-team
  • Depends On
    • IT for VM creation and machine procurement

Peptest

  • Deploy our user responsiveness regression testing system that complements the Talos User Responsiveness measurement we landed last quarter.
  • Q1 Outcomes
    • [DONE] Complete rollout to Try
    • [DONE] Create visibility, integrate with profiler once it is completed
    • [DONE] Ensure developers writing and landing tests for system
    • [DONE] If developers write tests, deploy tests into on-checkin automation
  • Stakeholders
    • Project Snappy
  • Community Involvment
    • Simpler project, advertise for community support
    • Ensure bugs well-marked and filed
    • Potential for many "goodfirstbugs"
  • Depends On
    • Benwa's profiler code working on all three systems
    • Releng support to deploy on try (all buildbot code already finished)

PULSE Stabilization

  • Many systems are depending on Pulse; we have no team dedicated to finding and fixing stability problems in that tool. We will dive into pulse to measure its stability/capacity and fix any issues we identify.
  • Q1 Outcomes
    • [DONE] Work with IT to determine measurement for pulse stability (stress/uptime/etc)
    • [SKIPPED] Profile Pulse to find hotpoints/weak areas and fix the worst of them, increase stability measurement
    • [DONE] Ensure buildbot messages coming from pulse are predictable and simplified
      • NOTE: We found that there aren't necessary performance issues with pulse. The instability comes from run-away queues that are not being checked. So instead of doing detailed profiling, we are working on fixing this bug.
  • Stakeholders
    • TBPL team, Releng, Ateam, QA,
    • Other consumers of pulse messages
  • Depends On
    • IT support for monitoring/stress testing/uptime analysis

Talos Mozbase/Mozharness

  • Move our oldest, cruftiest test harness atop the Mozbase software stack to make it easier for developers to run Talos as well as simplifying the releng code that runs Talos in automation. This will also make Talos easier to test and simpler to deploy which should require less Tree closures going forward.
  • Q1 Outcomes
    • [DONE] Create an internal pypi server for our automation systems
    • [MISSED] Merge Talos mozharness scripts for desktop/remote
    • [MISSED] Resolve all setup dependencies
    • [MISSED] Add in mozprocess and mozprofile support
    • [MISSED] Test and deploy
      • NOTE: We took mozharness stuff as far as we could in Q1 without releng support (which we knew we wouldn't have) We are working toward this goal with releng support in Q2.
  • Stakeholders
    • Releng
    • Developers (this work will make it easy to run talos exactly the way it is run in automation).
  • Community Involvement
    • Ample set of "goodfirstbugs" for community involvement
    • Very visible project, ensure widely publicized
  • Depends On
    • Releng to transition the buildbot code that drives Talos to Mozharness calls (expected to happen in Q2)

Talos RSS Desktop Measurement

  • Implement RSS measurement for all spawned processes during a Talos run. (Currently we do not measure RSS from the plug-in container process on desktop)
  • Q1 Outcomes
    • [DONE] Implement all process measurement of RSS on desktop during Talos runs
    • TBD
  • Stakeholders
    • Platform developers
  • Depends On
    • Releng for Deployment - will likely require side-by-side deployment

Community Involvement Goals

  • [DONE] Identify all stakeholders and audience for each project in 2012 Q1. (ctalbert + project leads)
    • [MISSED] Ensure we have docs and good feature pages for each project to detail our phases, our plans and how to get involved (check out code, setup etc) (ctalbert to work with dria on making us easy-to-write feature page style wikis for this)
  • [DONE] Integrate yourself with the teams that you're designing/implementing tools for (live in their IRC channel, go to their meetings, post to their newsgroup (as well as the tools newsgroup) (all of us will need to do this)
  • [DONE] Ensure there are clear requirements for each phase of the project, including the polish/test/deployment phase - (ctalbert + project leads will work on this)
  • [DONE] Increase visibility - Goal - by end of quarter, we have 75% of paid staff understanding who the Ateam is and what we do.
    • Use Twitter with #mozauto and newsgroups to help build visibility
    • Use yammer if you want, we decided it doesn't seem that useful
    • put "auto-tools" or "a*team" in the title of your blog post if appropriate
    • don't rely on blog posts for interaction, use them more for resources to have reference material for our projects, cross post to newsgroups where/when appropriate (usually when you want/expect discussion).
    • we are hard to find (as a group) on social networks. Optionally, you're encouraged to put your twitter/other-social-group-info-you're-comfortable-with-sharing into your A*team members entry: https://wiki.mozilla.org/Auto-tools/The_Ateam
    • Feel free to think about proposing talks for conferences and community events if that's something you're interested in. It's an excellent way to bring in more contributors. OpenSourceBridge is coming up in May/June in Portland if people want to start thinking about presenting there.
Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Goals/2012Q1&oldid=412639"