Troubleshooting failed AMO test runs

Test suite driver output log in Hudson The test suite driver appends the output from all tests’ stderr output followed by all tests’ stdout. Output from tests are separated by a bar of ‘====’. If the suite runs on multiple environments, all output for each environment is kept together.

   Windows 7; Firefox 
   ====== stderr  ======
   stderr from 1st test with exceptions 
   ========
   stderr from 2nd test with exceptions 
  ... 
   ====== stdout  ======
   Stdout from 1st test run 
   ========
   Stdout from 2nd test run 
  ... 
   
   mac OS X; Firefox 
   ====== stderr  ======
   ... 

Flavors of exceptions

Driver does not run test suite

Before running any test cases the suite driver checks that the website is available. If it receives an http error in return the suite driver immediately throws an exception and ends the run. Occasionally the test suite setup has, such as failing to get test cases from SVN. When a suite fails to start there are no stdout or stderr sections.

Exception with traceback stacks

This is the vanilla traceback stack from Python or Selenium throwing an exception (not caught in a try-except block), or the test case raising an exception. The module names and corresponding line#s provide a starting point to investigate the error that’s listed at the bottom of the traceback.

Saved exceptions

Test cases might capture certain errors and report them as a group at the end of the test. This is done where the same verifications are applied to multiple add-ons or pages, and it’s desirable to not end the test because of a minor discrepancy. (More on the SavedExceptions object in the Conventions in AMO Tests wiki page.)

At then end of the test case all saved exceptions are printed to to stderr and assertion error is thrown with the message “exceptions reported by test case”. In the output log the saved exceptions and the assertion error appear in separate blocks. Since they both contain a test case name they can be tied together as being from the same test.

Each saved exception contains three parts that are appended together on one line.

1. error summary.  ex: “Updated date not found”, “Exception in reviews count link”
2. error details including values that failed the verification. An error captured in a try-except block might use the string returned by the exception class.  ex: “text ‘reviews’ does not match regex ‘[0-9]+ reviews’ ”, “rating on browse page is ‘5 stars’, on detail page is ‘4 stars’ “.
3. test case breadcrumbs to help reconstruct the context the error happened on.  ex: “User type: anonymous, App: firefox, Sort: popular, Add-on: #2 ‘Firefox sync’

Counts for each error summary are also printed to avoid loosing visibility of an exception because it is buried amongst others.

The saved exceptions do not contain a line#, unlike raised exceptions with tracebacks. Searching the test case source code for the error summary will often return the line# at or near where the saved exeption was created. If the error summary text is not in the test case it will be in a function library or a page class method which was called by the test case.

Errors that indicate failure due to environmental factors

• http status 500 errors
• page load timeout
• “service is unavailable” message [1]
• missing elements on page when other tests in the suite experience a 500 errors or load timeouts [2]
  

Follow up

• if website is still down then file a bug to IT, if one hasn’t been filed.
• if website appears healthy consider rerunning (Hudson’s Build Now option) to restore green-ness to dashboard.

Notes

• Although the test suite driver checks that the website is available before launching the suite, test cases can still experience a 500 error if the there’s an intermittent issue or the server becomes unavailable after the driver starts the suite.
• [1] A few functions/methods, such as the one that submits an add-on search, checks the resulting page for the presence of expected elements, and if they aren’t found the function gets whatever text is on screen and include it in an exception.
• [2] When pre-POM tests open a new page they wait for a new page to load and then continue. If the page that loaded was simply an error message such as “Service is temporarily unavailable” the test case will throw exceptions when attempts a selenium call to any element that’s not there. POM tests cases avoid this by checking the presence of certain elements after it opens a new page.

Element not found

Exceptions with tracebacks include the locator of the missing element.  Saved exceptions include an element name and perhaps location, such as 'add-on rating in info table on detail page'.

• Potentially side effect of environmental issues if other tests in the same suite run have environmental problems.
• Element no longer appears on the web page. Might be an intentional or unintentional change by Web Dev.
• Element DOM has changed and element still appears on the screen.
• Unexpected alternate page layout that hadn’t been encountered when test case was written.
• Bad data as a result of an up-stream bug.
  

Followup

(if not an environmental issue)

• When the locator appears in the exception the intended element and page might not be immediately obvious.  Try searching the source code workspace for the locator to find the variable it is assigned to [1]. The variable name containing the locator should offer some context regarding the intended element.
• Since first segment of saved exceptions is normally a literal in the test cases, it can be search for in the source code to find the line that saved the exception.
  
• An element that’s universally missing from the preview website can be compared to the corresponding page in production.
• Se IDE’s Find button will show what element a locator points to, if it’s on the page. also see [2] below. (??? Issue with IDE 1.0.8 and CSS locators?)
  
• An element that is apparently present might be missing in a particular context: user logon, listing with a particular sort, or a particular add-on. Test case breadcrumbs might provide clues to the context.

Notes

• [1] Locator strings are generally defined outside of the test case. AMOlocators for pre-POM, page classes for POM. They are purposely defined as a string literal rather than concatenating segments together.
  • Locators may contain an index which should be truncated when searching the source code. Locators for items in lists contain a formater which is used to insert an index value at run time. A locator at run time containing div[2] or li:nth-child(3) could be generated by from div[%s] or li:nth-child(%d) in the source code.
    • There are rare cases of locators with two indexes for listed items that contain a list (ex: Install button).
    • A locator might use a hardcoded index for different adjacent elements when there is no other attributes to distinguish them. (ex: Sidebar's add-on category list that's followed by add-on type list)

• • [2] If copying a locator from source code to paste into Se IDE, remember to check for occurrences of %s or %d which require substitution (see [1]).  %s or %d would not appear in a locator in a run time exception unless the test case had a bug.

Value checking errors

After text or an element attribute is retrieved it is often checked for some expected consistency. Failure can be due to:

• Unexpected alternate data formatting that hadn’t been encountered when test case was written.
• Bad data as a result of an up-stream bug.

Some examples of value checking:

• text does not match a regular expression. 
  
• link URL does not match a regular expression
  
• two values from the same page are not equal
  
• a element does not have the same value on two pages (ex: add-on rating on browse page and detail page)
  
• value is null
  

Information on the general theory of relativity can be found here and maybe here