Why I Didn't Write a Test: Stuff that's hard to test, and what we could do to fix it

Host: Ted Mielczarek

We have several automated test suites running these days, but they don't cover everything. There are still plenty of areas that aren't being tested due to testing requirements that currently aren't covered by our test harnesses.

This session will be a roundtable discussion about deficiencies in our testing, and what we could do to fix them. Bring your own examples of tests you didn't write, and what we could do to make that code testable.

Transcript

• Crash Reporter tests
• Extension Manager
  • needs to restart browser
  • use sisyphus? merge some logic into automation.py?
• Font metrics (CSS length units)
  • blocked on @font-face
• Caret positioning
• Invalidation
• OS integration
  • don't want to leave build boxes in an unknown state
    • cleanup phase? VM reset?
  • no tests for updater
  • hard in debug builds, timeouts etc
• Drag and drop
  • use gristmill?
• Profile tests
  • need restart (similar to EM tests)
• Tests with extensions
  • possibly gristmill as well?
• Plugin tests
  • Plugin code has lots of plugin-specific branches
  • JRE tests - licensing?
  • Simple test plugin (jst mentioned)
  • Flash tests
• Testing of timezones/locales
  • is the required locale even installed?
• Chrome/theme
  • contrast of foreground/background colors
• Printing / Print Preview
  • specific driver bugs
  • cairo printing issues (very large spool file)
  • transparent PNGs not printing properly
  • testing printable area on paper
  • print selection (can't do with reftest currently)
• Multiple Monitors
  • getting screen size
  • positioning popups on screen properly
• Window managers
  • focus issues with specific window managers
• Focus
  • lots of issues
• JS API
  • no unit tests, just running JS tests
• Migration
  • profiles from other apps
  • have canned profiles checked in?
  • don't want to break build boxes
• Session restore
  • Needs restart + crash
• Certificates
  • ssltunnel, mayhemer has a patch to integrate into mochitest
• Net Error tests (l10n specifically)
• C++ test harness
  • Waldo has a patch (but he's gone right now)
  • linkage issues with libxul - how to test non-exported symbols?
  • should we just use cppunit or Google's unit test library?
• Ability to run tests without compiling
• XPCshell tests for extension components
  • can't register new interfaces
• XPCshell hangs - don't get any output
  • need to make xpcshell not buffer output?
• hard to run xpcshell in a debugger
  • make-interactive works, but could be easier (like firefox -g)
• xpcshell startup time makes it slow to run multiple tests
• mochitest focus issues make it annoying to run mochitest
  • linux unit test boxes use Xvfb, could we build this into the test harness?
  • Windows, use a separate workstation?
• using threads in xpcshell asserts a lot
• running mochitests all-at-once vs. individual has problems
  • hard to see what's failing when running all-at-once
• xpcshell TODO support
• xpcshell run-to-completion
  • currently quits at first test failure, would like to see all failures
• reftest features for importing other test suites easily
  • "this test should have no red pixels"
  • "there should be green pixels in this test"
• reftest with multiple reference pages-match any to pass
• security tests
  • tests are sensitive, may reveal working exploits or novel exploit concepts
• can't run mochitests from file:// URLs
• mochitest security tests (signed scripts)