Auto-tools/Projects

From MozillaWiki
Jump to: navigation, search

Welcome to the Automation and Tools Projects Page.

We are the people who write the code that enables all our automated test systems (correctness, performance etc) to run. We are continually expanding, upgrading, and inventing new and better ways to do automated quality analysis at Mozilla. Most of our systems run on a per-checkin basis, and at the time of this writing we do roughly 300 checkins a day, and we run hundreds of thousands of tests per checkin, so you can imagine how many times our code gets run. If you like the idea of that kind of a challenge, we could use your help on some of the projects below. If you don't know how to get started, feel free to hop on irc (irc.mozilla.org), into the channel #ateam and ask some questions.

The Goals

Every calendar quarter, we figure out how best to support Mozilla's efforts and translate that into a set of goals. Most of our goals these days have tracking bugs. So if you want to directly help us the most, look through those tracking bugs and find a bug you can start hacking on. Ask in #ateam if you need help.

2014

Goal List Post-Mortem Notes
2014 Q4
2014 Q3
2014 Q2
2014 Q1

2013

Goal List Post-Mortem Notes
2013 Q4 Awaiting scheduling Pulled together the highest priority items into the "top list" on goals page to help focus
2013 Q3 2013 Q3 Postmortem Quarter felt very unfocused, company wide goals done differently, had some impact on us.
2013 Q2 No postmortem performed
2013 Q1 No postmortem performed

Other Projects

Every quarter there are more things we'd like to do than we have time for. Below are some of these projects, broken down by areas and technologies so that you can find something that gets you excited.

Overview

Work in progress effort to create an index for all existing/WIP automation tools - Here.

Guidelines

The guidelines are evaluated based on the link we have to documentation (wiki, readme.md, readthedocs, etc.) This link should contain the information or in the first page of text have a link to additional information so that we can easily learn about the project.

Guidelines for what to include in project documentation:

  • The goals of the project
  • Why the project is important to Mozilla and/or the A-Team
  • Dependencies on other projects/teams, and its place in the greater ecosystem
  • List out who to contact, mentors, active developers
  • Documentation for how to use the project (or how it will be used)
  • How to setup / develop / test the project
  • Milestones and features to be developed (bugzilla table, link to well annotated github issues, etc.)
  • Good first bugs and how to get started (bugzilla table, link to well annotated github issues, etc.)
  • How to submit a patch
  • Coding style guidelines and expectations

When setting the priority of a project, consider this scale as a starting point:

  • 1 - This is a stated goal for the current quarter | There are active people working on this | Other projects/teams depend on this completing
  • 2 - There are active people working on this | We are committed to delivering at least one milestone in the current quarter
  • 3 - This is an project that we and others believe would be beneficial to Mozilla, but we haven't promised to deliver it on a specific timeline. There are clear plans and maybe a prototype of one or more milestones to start with
  • 4 - This is a good idea or a project that we worked on in the past which could use maintenance. We don't have resources for this now, but still feel it is a valuable project!

Firefox for Android

The Firefox for Android project is a continual expansion of our test harnesses to better support the Android platform. Currently we only release Firefox on that platform (aka Fennec), but we might also begin testing web apps there in the near future. Familiarity with Android and a great knowledge of Python is very useful here.

  • Help expand our reach to x86 Android systems by aiding us in debugging and fixing test failures on that platform: see the collection of bugs beneath bug 891959

Firefox Desktop and General Automation Support

The desktop web browser Firefox continues to be our flagship product, and there is always ongoing work needed to ensure that we continue to support the new features that regularly land in Firefox. Many of the bugs here are more tractable simply because the test harnesses and tests involved are older code. This is a good spot for first-time contributors. These will involve Python knowledge and some JavaScript, depending on the bug.

Performance

We also maintain all the systems and the code that perform per-checkin testing on performance. There are many performance automation systems at Mozilla, and one of our current efforts is to pull them under one high-level dashboard. Performance will demand a working knowledge of the web, Python, JavaScript, statistics, and great debugging skills.

Tools & Dashboards

We create web-based tools and dashboards to help illustrate how our automation is doing. Datazilla is a performance dashboarding system, and Treeherder surfaces the status of our automation runs. We also maintain and continually improve Mozilla's Bugzilla installation, often contributing patches to the upstream general Bugzilla project. Python, JavaScript, and, in Bugzilla's case, Perl, are all used here, along with both relational and NoSQL databases.

ATeam Handbook

The ATeam Handbook is meant to be the definitive source of information on how we do things in the ateam, useful for new and old contributors alike. Currently in the planning phase.

Mentored Bugs

These bugs are things that we have identified as great starter bugs. Each bug contains a focused technology that is required and a mentor who has volunteered to help out people starting to work on the issue. If you've done a few mentored bugs, talk to your mentors about becoming a mentor yourself!

ID Summary Priority Status Bug mentor
592653 Use Services.prefs instead of preferences-service / gPrefService -- ASSIGNED
649563 Convert all existing uses of nsIPrefBranch to use SpecialPowers -- NEW
682901 catch the case where .finish() gets called before waitForExplicitFinish() -- NEW
695937 pageloader extension/talos should die more gracefully when pageset doesn't exist -- NEW
718487 SimpleTest/EventUtils.js: Improve sendKey() to correctly handle arguments beginning with VK_ -- NEW
744387 SimpleTest._logResult() should report when test.name is 'undefined' -- NEW
768239 Add progress indicator for [current/total] test count to Mozmill output -- ASSIGNED
841086 mozprofile - test(s) should be written for bug 759594 -- NEW
847275 make assertions cause test failures in mochitest-browser-chrome -- ASSIGNED
862253 [mozrunner] Error Message for a wrong binary is not helpful -- ASSIGNED
884828 mozinstall.get_binary() should check for executable bit and be less confusing -- ASSIGNED
925408 [mozprocess] mozprocess should have the ability to kill a specific process -- ASSIGNED
933103 mochitest's EventUtils.synthesizeMouse functions not e10s friendly -- NEW
939755 With httpd.js we sometimes don't get the most recent version of the file -- NEW
940882 Consolidate waitForCondition implementations in SimpleTest -- NEW
951272 Convert SpecialPowers.addPermission usage to SpecialPowers.pushPermissions -- NEW
958147 Choose one of run-if or skip-if and get rid of the other -- REOPENED
969334 mozinstall should not magically try to find the mount point of the DMG -- NEW
975663 Give more detail message to notice people to install virtualenv in gaia-marionette -- NEW
980788 [manifestdestiny] Add greater-than/less-than (equal) support -- NEW
982852 Make run_test() in xpcshell tests optional -- NEW
985207 getTitle should return the top level browsing context title P1 ASSIGNED
989266 Output docker download image status as tty -- NEW
996652 Scope the private symbols of head.js -- NEW
1001318 Don't need to generate profile at marionette/install. -- NEW
1002518 [mozversion] Add gaia_version to mozversion -- NEW
1003314 Dump more Android emulator info before and after tests -- NEW
1011167 Worker CI -- NEW
1016007 mozregression aborts in the case of a network error -- NEW
1033651 [ui] AlertManager view/hide alerts over and over again -- NEW
1033655 [ui] AlertManager should show proper branch names for android and osx on the treeherder -- NEW
1033657 [ui] AlertManager should detect missing jobs and trigger them -- NEW
1034243 [UI] Alert Manager should generate a report for uplift statistics -- NEW
1036328 Support bisection in case of timeout/crash -- NEW
1051561 diagnostics: Periodic production tests (meta) -- NEW
1051717 Docker Worker "status" page -- NEW
1051728 Run docker-worker tests on "production" ami instances -- NEW
1056282 docs: Design for docs.taskcluster.net and general refactoring -- NEW
1056384 docker-worker: Enforce max runtime at the task level account for pulls / kills / etc.. -- NEW
1057882 docker-worker: Support "Services" -- ASSIGNED
1057932 [docker worker] Failure to extend task graph should be considered task failure -- NEW
1058732 [docker-worker] be more aggressive with shutdowns -- NEW
1060576 [titanic] Use requests-cache to improve performance -- NEW
1061813 Make mochitest-browser tests provide a useful message for tests with neither add_task nor test() methods -- NEW
1063348 docker worker: Extended task graphs should be dependent on the task which created them. -- NEW
1063706 User build phones that have the screen lock setting enabled cannot be reached with adb after a reboot after 15-20 seconds - find a way to determine this and notify the user -- NEW
1068361 docker-worker: Fail tasks which fail to start rather then bailing with infrastructure "task error" which causes task to be retried. -- NEW
1071995 diagnostics: Test that we can create a task and that it gets resolved -- NEW
1072854 package.json after installing from an npm-mirror repository doesn't work with CommonJS require() -- UNCONFIRMED
1072912 Remove makefile targets for running tests from testsuite-targets.mk -- NEW
1078894 Create example test of do_await_remote_message -- NEW
1079132 Drop explicit support for Firefox OS v1.3 -- NEW
1080806 Port runreftestb2g.py to use mozbase -- NEW
1084469 [UI] Alert Manager - table view needs to support mouse over for additional data and user transactions -- NEW
1086570 Don't hardcode urls in the report html as they can vary from suite to suite -- NEW
1088238 taskcluster-notify: Email notification component for tasks and task-graphs -- NEW
1091284 Remove systemMemory, environment from automationutils -- NEW
1091541 [backend] Alert Manager - needs to fix treeherder links for android -- NEW
1092274 [backend] Alert Manager - fix the merge detection code -- NEW
1092340 Remove initializeProfile from automation.py.in -- NEW
1100155 Mention expected response key when raising MarionetteException on unexpected response -- NEW
1103064 Add Futuremark Peacekeeper benchmark to mozbench -- ASSIGNED
1103134 Add support for Android stock browser and Dolphin to mozbench -- NEW
1110285 titanic could use a intermittent mode to help narrow down intermittents -- NEW
1113227 Mozbench should set user profile on Android -- NEW
1114717 Mozbench should allow user to specify which benchmark to run on command line -- NEW
1114972 Mozbench should automatically run on Android if the firefox url ends with .apk -- NEW
1118313 newSession looks for non-conforming session_id property on Command -- NEW
1121393 Make JS callers of ios.newChannel call ios.newChannel2 in mozmill/ -- ASSIGNED
1124689 Test informant should use structured logs instead of parsing manifests -- ASSIGNED
1124695 [mozrunner] and/or [mozprocess] should send SIGTERM before sending SIGKILL by default -- NEW
1124746 create a script when tells us if we have confidence that the regression we see includes all related regressions/improvements -- NEW
1124779 talos compare.py should pull data from treeherder -- NEW
1125098 Remove XML report functionality from Marionette test runner -- NEW
1125863 Reports should differentiate enabled from added and disabled from removed -- NEW
1129899 Don't silently use the unsupported x64 builds on 64bit Windows -- NEW
1130979 Calling find_element() should use constants from marionette.By instead of hard-coded values -- ASSIGNED
1132057 [mozlog] HTML log should show the passed/failed tests on the top of the page as default -- NEW
1132867 Allow using mozregression with a particular pref -- NEW
1135091 convert remaining SpecialPowers.setBoolPref to pushPrefEnv -- NEW

80 Total; 80 Open (100%); 0 Resolved (0%); 0 Verified (0%);


For reference, the old Projects Page (which is largely out of date, but interesting for historical reasons) is accessible here