QA/Execution/Web Testing/Socorro/Test Plan

Socorro test plan

Socorro roadmap:

• http://spreadsheets.google.com/ccc?key=0AhiX365xacl1dDZ6eVdMc1Vpbld3elMyX2lnUzJnclE&hl=en
  

Staging server:

• http://crash-stats.stage.mozilla.com

Release checklist:

• https://intranet.mozilla.org/Socorro/Release_Checklist

Code Respository / Commits

• Repo: http://code.google.com/p/socorro/source/browse/
• Latest commits: http://code.google.com/p/socorro/source/list

Supported browser(s):

• Firefox 3.6+ (Tier 1)
• IE 8 (major functionality/layout only)
• Chrome (major functionality/layout only)
• Safari (major functionality/layout only)

Litmus Tests:

• Socorro trunk test run -- run this after bugs have been verified, before certifying the release
• Tests to run
• Tests to write

Manual Tests:

• Ensure that the various top-level application views work (Firefox, Thunderbird, etc.)
  • http://crash-stats.stage.mozilla.com/products/Firefox
  • http://crash-stats.stage.mozilla.com/products/Camino
  • http://crash-stats.stage.mozilla.com/products/SeaMonkey
  • http://crash-stats.stage.mozilla.com/products/Thunderbird
  • http://crash-stats.stage.mozilla.com/products/Fennec
• As well as their listed versions, beneath
  • Check that the versions are accurate/up-to-date/complete (ask!)
  • Check they're sorted correctly
  • Load each "Report" type:
    • Overview
    • Crashes per User
    • Nightly Builds
    • Top Changers
    • Top Crashers
    • Top Crashers by URL
    • Top Crashers by Domain
    • Top Crashers by Topsite
• Browse around and ensure there are no JS errors trapped in the Error Console
• Ensure that searching for crashes by ID -and- signature works
• Ensure that the "More Reports" dropdown loads data for each option
• Advanced Search (contains: "flash"):
  • http://crash-stats.mozilla.com/query/query?product=Firefox&version=ALL%3AALL&range_value=1&range_unit=weeks&date=09%2F21%2F2010+22%3A10%3A40&query_search=signature&query_type=contains&query=flash&build_id=&process_type=any&hang_type=any&do_query=1
  • Since we can't test every combination, ensure that both the "All" filters work, in tandem with random testing of other platforms/versions
  • What should be tested in the Advanced Filters section?
    • Test JavaScript in form
      • Product, Version should be populated based on clicks
    • Switch to "Plugins Only," click search - using results (click through to details on a couple)
  • Test "Search by Branch" returns only crashes for Gecko on that branch
    • Do a general search; using results, test each in isolation:
      • Test Stack signature startswith, contains, isexactly
      • Build Id returns only crashes for that build
    • Play with before and within the last fields to change search results
    • The "Crashing Thread" section is intact, and the link to the source works
    • The "Show/hide other threads" link works
• Load a crash report, and:
  • "Details" tab: Ensure that the "Related Bugs" block shows up, and the links work
  • "Modules" tab: raw list of DLLs, their versions, debug identifiers, and filenames
  • "Raw dump" tab: just what it says, ensure there is data there
  • "Extensions" tab:
    • If there is an extension listed, click on it; it should go to its AMO details page
    • It lists the extension's GUID (email address or hex), version, and "current?" (if available) version
  • "Comments" tab:
    • Comments show up (crashes don't always have comments; just ensure some do)
    • Their timestamps are links to the original report, which should have the same top frame
  • "Correlations" tab: click on the "Load" buttons, and ensure that data loads
  • Try the "Next" and "Previous" buttons
• Make sure to test out http://crash-stats.stage.mozilla.com/status
  • Refresh, ensure that the server time updates
  • What else can be tested here, on staging? (Socorro devs, please fill in)

lars: the dailyUrl cron job creates a csv file at a configurable location. That csv file in staging is at /home/processor and is named in the form YYYYMMDD-crashdata.csv.gz [09:41am] stephend|mtg: ah [09:41am] lars: cpu_info is column 12 (0 based) [09:42am] lars: I've just noticed that the column name is degenerate, though that is not likely a problem

How to verify that the right data is showing up on staging:

• https://bugzilla.mozilla.org/show_bug.cgi?id=601971

Automated Tests (via Selenium):

• What can be automated from the above list?
  • Can we pull product versions from product-details, store it in an array, and compare what's on staging against that
  • Values of important <select> / <option> widgets
    • For each resulting page, verify the most important elements, and that there is *some* data
      • Also verify that column headings, that they have data, and, if it has a sort indicator, it's sorting correctly
  • Under Top Crashers / Top Changers, ensure three versions are present, and have 20 entries (15 right now on staging; which is right? Does it vary?)
  • View All link
  • Search
    • Ensure form elements are present/correct
    • Search by crash ID: find an incident via the DOM, get its ID, then execute a search for that ID
    • How do we search for stacktraces (data-dependent)?
      • Are there common/long-standing ones that staging always has available?
• Next steps:
  • Figure out tests
  • Write them
  • Get review
  • Land in http://viewvc.svn.mozilla.org/vc/projects/socorro_qa/
  • Get cron for auto-dumping prod -> staging set up
  • Configure Hudson to pull and run

Gotchas / Notes

• If certain versions / views have no data, it could be that we have to re-enable them manually on staging from the admin interface, when data in postgres gets flushed