QA/Execution/Web Testing/Socorro/Test Plan

Socorro test plan

Socorro roadmap:

• http://spreadsheets.google.com/ccc?key=0AhiX365xacl1dDZ6eVdMc1Vpbld3elMyX2lnUzJnclE&hl=en
  

Staging server:

• http://crash-stats.stage.mozilla.com

Supported browser(s):

• Firefox 3.6 (Tier 1)
• IE 8 (major functionality/layout only)
• Chrome (major functionality/layout only)
• Safari (major functionality/layout only)

Manual Tests:

• Ensure that the various top-level application views work (Firefox, Thunderbird, etc.)
  • http://crash-stats.stage.mozilla.com/products/Firefox
  • http://crash-stats.stage.mozilla.com/products/Camino
  • http://crash-stats.stage.mozilla.com/products/SeaMonkey
  • http://crash-stats.stage.mozilla.com/products/Thunderbird
  • http://crash-stats.stage.mozilla.com/products/Fennec
• As well as their listed versions, beneath
  • Check that the versions are accurate/up-to-date/complete (ask!)
  • Check they're sorted correctly
  • Load each "Report" type:
    • Overview
    • Crashes per User
    • Nightly Builds
    • Top Changers
    • Top Crashers
    • Top Crashers by URL
    • Top Crashers by Domain
    • Top Crashers by Topsite
• Browse around and ensure there are no JS errors trapped in the Error Console
• Ensure that searching for crashes by ID -and- signature works
• Ensure that the "More Reports" dropdown loads data for each option
• Advanced Search (contains: "flash"):
  • http://crash-stats.mozilla.com/query/query?product=Firefox&version=ALL%3AALL&range_value=1&range_unit=weeks&date=09%2F21%2F2010+22%3A10%3A40&query_search=signature&query_type=contains&query=flash&build_id=&process_type=any&hang_type=any&do_query=1
  • Since we can't test every combination, ensure that both the "All" filters work, in tandem with random testing of other platforms/versions
  • What should be tested in the Advanced Filters section?
    • Test JavaScript in form
      • Product, Version should be populated based on clicks
    • Switch to "Plugins Only," click search - using results (click through to details on a couple)
  • Test "Search by Branch" returns only crashes for Gecko on that branch
    • Do a general search; using results, test each in isolation:
      • Test Stack signature startswith, contains, isexactly
      • Build Id returns only crashes for that build
    • Play with before and within the last fields to change search results
    • The "Crashing Thread" section is intact, and the link to the source works
    • The "Show/hide other threads" link works
  • Topchangers
    • Verify switching # of days works
      • Download CSV file and ensure it has content
      • Ensure RSS feed has data (validate the feed, too)
• Load a crash report, and:
  • "Details" tab: Ensure that the "Related Bugs" block shows up, and the links work
  • "Modules" tab: raw list of DLLs, their versions, debug identifiers, and filenames
  • "Raw dump" tab: just what it says, ensure there is data there
  • "Extensions" tab:
    • If there is an extension listed, click on it; it should go to its AMO details page
    • It lists the extension's GUID (email address or hex), version, and "current?" (if available) version
  • "Comments" tab:
    • Comments show up (crashes don't always have comments; just ensure some do)
    • Their timestamps are links to the original report, which should have the same top frame
  • "Correlations" tab: click on the "Load" buttons, and ensure that data loads
  • Try the "Next" and "Previous" buttons
• Make sure to test out http://crash-stats.stage.mozilla.com/status
  • Refresh, ensure that the server time updates
  • What else can be tested here, on staging? (Socorro devs, please fill in)

Automated Tests:

• What can be automated from above?
  • Simple search
    • Possible to automate by ID?
    • How do we search for stacktraces?
• Next steps:
  • Figure out tests
  • Write them
  • Get review
  • Land in http://viewvc.svn.mozilla.org/vc/projects/socorro_qa/
  • Get cron for auto-dumping prod -> staging set up
  • Configure Hudson to pull and run