Personal tools

Auto-tools/Projects

From MozillaWiki

Jump to: navigation, search

Welcome to the Automation and Tools Projects Page.

We are the people who write the code that enables all our automated test systems (correctness, performance etc) to run. We are continually expanding, upgrading, and inventing new and better ways to do automated quality analysis at Mozilla. Most of our systems run on a per-checkin basis, and at the time of this writing we do roughly 300 checkins a day, and we run hundreds of thousands of tests per checkin, so you can imagine how many times our code gets run. If you like the idea of that kind of a challenge, we could use your help on some of the projects below. If you don't know how to get started, feel free to hop on irc (irc.mozilla.org), into the channel #ateam and ask some questions.

Contents

The Goals

Every calendar quarter, we figure out how best to support Mozilla's efforts and translate that into a set of goals. Most of our goals these days have tracking bugs. So if you want to directly help us the most, look through those tracking bugs and find a bug you can start hacking on. Ask in #ateam if you need help.

2014

Goal List Post-Mortem Notes
Current Quarter: 2014 Q1

2013

Goal List Post-Mortem Notes
2013 Q4 Awaiting scheduling Pulled together the highest priority items into the "top list" on goals page to help focus
2013 Q3 2013 Q3 Postmortem Quarter felt very unfocused, company wide goals done differently, had some impact on us.
2013 Q2 No postmortem performed
2013 Q1 No postmortem performed

Other Projects

Every quarter there are more things we'd like to do than we have time for. Below are some of these projects, broken down by areas and technologies so that you can find something that gets you excited.

Firefox OS

The Firefox OS (aka B2G) project is to create an OS for a mobile phone based on the web platform. Our job is to both automate the OS tests and make the OS itself easily testable so we can ensure a very high quality bar. Building and working with an OS is not for the feint of heart; it is a moving platform where many things change all the time. It's helpful to have very good debugging skills, deep knowledge of JavaScript, working knowledge of Python and some C++/build system internals.

Firefox for Android

The Firefox for Android project is a continual expansion of our test harnesses to better support the Android platform. Currently we only release Firefox on that platform (aka Fennec), but we might also begin testing web apps there in the near future. Familiarity with Android and a great knowledge of Python is very useful here.

  • Attempt to run our automated tests using the awesome work done with Android emulators in bug 910092
  • Help expand our reach to x86 Android systems by aiding us in debugging and fixing test failures on that platform: see the collection of bugs beneath bug 891959

Firefox Desktop and General Automation Support

The desktop web browser Firefox continues to be our flagship product, and there is always ongoing work needed to ensure that we continue to support the new features that regularly land in Firefox. Many of the bugs here are more tractable simply because the test harnesses and tests involved are older code. This is a good spot for first-time contributors. These will involve Python knowledge and some JavaScript, depending on the bug.

Performance

We also maintain all the systems and the code that perform per-checkin testing on performance. There are many performance automation systems at Mozilla, and one of our current efforts is to pull them under one high-level dashboard. Performance will demand a working knowledge of the web, Python, JavaScript, statistics, and great debugging skills.

Tools & Dashboards

We create web-based tools and dashboards to help illustrate how our automation is doing. Datazilla is a performance dashboarding system, and TBPL surfaces the status of our automation runs. We also maintain and continually improve Mozilla's Bugzilla installation, often contributing patches to the upstream general Bugzilla project. Python, JavaScript, and, in Bugzilla's case, Perl, are all used here, along with both relational and NoSQL databases.

Mentored Bugs

These bugs are things that we have identified as great starter bugs. Each bug contains a focused technology that is required and a mentor who has volunteered to help out people starting to work on the issue. If you've done a few mentored bugs, talk to your mentors about becoming a mentor yourself!

ID Summary Priority Status
780807 browser and other reporting statistics should not be literal strings in run_tests.py (and possibly elsewhere) -- NEW
841086 mozprofile - test(s) should be written for bug 759594 -- NEW
958147 Choose one of run-if or skip-if and get rid of the other -- NEW
518436 Move do_check_throws into testing/xpcshell/head.js -- NEW
574871 [exit] mozmill should have a per-test timeout to stop the test and continue if a test takes too long -- NEW
613454 Mozmill reports do not distinct between 32bit and 64bit platforms -- NEW
643473 endTest passes redundant data -- passes and passed -- NEW
656633 JS Bridge connects twice -- NEW
662972 intuit controller based on application -- NEW
680837 Instrument Firebug test-bot to run with SeaMonkey -- NEW
682901 catch the case where .finish() gets called before waitForExplicitFinish() -- NEW
695937 pageloader extension/talos should die more gracefully when pageset doesn't exist -- NEW
698769 mutt should be brought up to the reality that the MozMill class is now used via the API -- NEW
705864 [mozprocess] mozprocess tests should use mozprocess.pid -- NEW
718487 SimpleTest/EventUtils.js: Improve sendKey() to correctly handle arguments beginning with VK_ -- NEW
757171 Marionette should gracefully handle navigation attempts that throw exceptions -- NEW
795187 Clearly indicate phone's ID/POOL -- NEW
795375 Report actual fennec version, not the one we are trying to test -- NEW
797711 Test duration wrong for restart tests (only last module is taking into account) -- NEW
798507 Empty exception if addHttpResource() is called with an invalid path -- NEW
804158 Mozmill restart tests should skip following tests in case of a failure -- NEW
828573 Latest Mozmill doesn't work with Python 3 -- NEW
834502 Need to cover more cases in test_click.py -- NEW
841808 mozfile.rmtree should handle windows directory in use better -- NEW
846749 fix up marionette-listener.js&marionette-actors.js style and comments -- NEW
872174 Improve mutt test function blocks to better reflect what the test is doing -- NEW
897543 Create mutt tests for Touch Events (for Metro) -- NEW
898374 Move mutt tests from js/frame/restart/states into the python tests folder -- NEW
909831 Ouija pages should display the date range they are using the calculate the data -- NEW
932651 [e10s] getClipboardData in specialpowersAPI not e10s friendly -- NEW
933103 mochitest's EventUtils.synthesizeMouse functions not e10s friendly -- NEW
935691 Colour property values must be standardized to rgba format in getElementValueOfCssProperty -- NEW
938712 Mach mochitest commands should be smart enough to pick the suite for you -- NEW
939755 With httpd.js we sometimes don't get the most recent version of the file -- NEW
944471 Introduce a Mutt testcase class -- NEW
957081 [Tracking bug] Make Mozmill ready for multi-process architecture -- NEW
973560 Mutt test 'testController/testExpectedEvents.js' fails under xvfb with 'TypeError: callback is not a function' -- NEW
974859 controller.waitForPageLoad() doesn't timeout for 0 -- NEW
980794 Extend the new mouse events to accept modifier keys -- NEW
982852 Make run_test() in xpcshell tests optional -- NEW
984208 [Marionette Client] Modifying shuffle to put its seed outside -- NEW
985207 getTitle should return the top level browsing context title -- NEW
989266 Output docker download image status as tty -- NEW
992401 Extend b2ghaystack so that it can be used to download proper builds and guide someone through the regression hunting process -- NEW
994302 disable speculative connections for talos -- NEW
994860 update compare.py to use current platforms/tests -- NEW
996236 screenshot() should allow decoding from base64 automatically -- NEW
996652 Scope the private symbols of head.js -- NEW
996871 add preferences to talos to remove external network access for rss -- NEW
997389 Should choose 32-bit builds by default on Windows -- NEW
975663 Give more detail message to notice people to install virtualenv in gaia-marionette -- NEW
949543 Use seconds for defining time in Marionette Python client P4 NEW
964278 Integrate mozversion conditions into manifest parsing -- ASSIGNED
592653 Use Services.prefs instead of preferences-service / gPrefService -- ASSIGNED
637500 EXCLUDED_PLATFORMS utilized but not used -- ASSIGNED
632069 mozmill events arent documented -- ASSIGNED
873364 Mozmill.finish() calls a 'print' command when screenshots have been made, as opposed to calling the logger. -- ASSIGNED
992139 [mozversion] Set "application_display_name" to "application_name" if CodeName doesn't exist -- ASSIGNED
847275 make assertions cause test failures in mochitest-browser-chrome -- ASSIGNED
884828 mozinstall.get_binary() should check for executable bit and be less confusing -- ASSIGNED
905400 Inherit from mozprofile's general Profile class and add all testing preferences -- ASSIGNED
768239 Add progress indicator for [current/total] test count to Mozmill output -- ASSIGNED
795282 Further improve time until first test gets started by not blocking on get_appinfo() -- ASSIGNED
913100 Add shuffle option for tests -- ASSIGNED
867076 Mutt tests should take disabled tests into account for the final passed/failed/skipped test count -- ASSIGNED
840109 Pandas running b2g stop sending network traffic -- ASSIGNED
925408 [mozprocess] mozprocess should have the ability to kill a specific process -- ASSIGNED
940882 Consolidate waitForCondition implementations in SimpleTest -- ASSIGNED
960421 Mozmill doesn't kill the running application if KeyboardInterrupt error is raised -- ASSIGNED
982043 Improve report sending code for handling all kind of HTTP failures -- ASSIGNED
862253 [mozrunner] Error Message for a wrong binary is not helpful -- ASSIGNED
875660 Remove the "driver.js" module from mozmill code -- ASSIGNED
873035 Marionette should have a CLI for stdout log -- REOPENED

Open; Resolved; Total (0% complete)


For reference, the old Projects Page (which is largely out of date, but interesting for historical reasons) is accessible here