QA/Execution/Web Testing/Docs/Automation/Testcases/ConventionsAmoTests

From MozillaWiki
Jump to: navigation, search

Conventions in AMO test cases

Element Locators

  • Element locators are defined outside of the test case to centralize their definition and simplify test case code.
  • Locators may contain %d or %s formatting specifiers to support elements in a list, or a common locator for sibling elements that differ by a short segment.
  • Locators are defined in a combination of Xpath and CSS formats. CSS is the preferred format for POM.

Locators in Pre-POM test cases

  • Defined in AMOlocators module. The AMOlocators class is divided into sub-classes that contain locators by page which is further sub-classed into sections of the page.
  • Index or option subsitution is done with the % operator. For example, for the locator to select one of the sort options on the AMO home is:

In AMOlocators a locator is defined for links to sort options on a page (note the %s near the end of the string)

 sortLink_Keyed =
   “css=div[class~='featured-inner'] div[class='listing-header'] ul li a[href*='?browse=%s']”

To click on the ‘popular’ sort option:

 sort_locator = sortLink_Keyed % ‘popular’
 selenium.click(sort_locator)

Locators in POM test cases

  • Defined in the page classes. Defined as strings or a Locator object.
  • The POM Locator class is a child class of string which contains methods with_index( ) and with_option( ) for inserting values. For example:

In a page class, the locator for items in the themes category list:

 loc = locator.Locator
 theme_category_link =
   loc("css=div[class~='other-categories'] h3:contains('Themes') + ul + ul li:nth-child(%d) > a")

In a test case, to create a locator for the 2nd category in the list :

 cat_locator = theme_category_link.with_index(2)

If a list requires an offset to index an element (i.e. the 1st element is accessed by “:nth-child(2)” ) the page class may define an alternate locator class which defines with_index( ) to include the offset.

In the case of the Themes landing page the ThemesLoc class maps a single index into the (row, column) pair the page is implemented in.

Parameter driven tests

Test cases use values from TCparams to drive the test.

  • Option lists such as user types, Mozilla applications and sort options are used in nested for-loops.
  • Values such as maxItemsPerPage or maxCategories are used to limit how many items on a page or number of categories are included in the test run.

(The TCparams class contains three child classes which contains different values for the same option/variable name. In the child classes an option list may contain all valid values or a subset. A max* variable will contain a numeric value to partially drive the length of the test.  The suite driver passes one of the child classes to the test case, overriding options indicated in the tests.py file)

The iterating through option lists has been implemented in two ways:

A) Simple nesting example

 for user_type in tc_parms.userTypeList:
 
     # login or not, as per user_type
     ...
     for app_name in tc_params.mozillaApplicationOptions:
 
         # select the application from the navigation menu
         ...
                         # eventually the destination page is reached
 
                         # determine the maximum add-ons to check on this page
                         max_count = min(selenium.get_xpath_count(addon_item_locator), tc_params.maxItemsPerPage)

                         for addon_index in range(1, max_count + 1):
                              # process an add-on


Because of the multiple nested loops the innermost loop can be deeply indented resulting in less horizontal space for statements when following the 80 character line limit.

B) Option tuples

To avoid deep indention from multiple options lists, another implementation creates tuples of all combinations of options.

 param_list = [tc_parms.userTypeList,
                    tc_params.mozillaApplicationOptions, ...]
 
 # create tuples from parameter lists
 param_combos = shared_lib.combos_from_lists(param_list)
 
 for param_tuple in param_combos:
 
     user_type = param_tuple[0]
     app_name = parm_tuple[1]
     ...
     # login or not, as per user_type
     ...
     # select the application from the navigation menu
     ...
     # eventually the destination page is reached
     ...
     # determine the number of add-ons to check on this page
     max_count = min(selenium.get_xpath_count(addon_item_locator), tc_params.maxItemsPerPage)

     for addon_index in range(1, max_count + 1):
         # process an add-on


shared_lib is the shared library where combos_from_lists is defined: general_functions in pre-POM test cases, data_functions in POM.

Verifying a Variety of Add-ons

In an effort to check a variety of add-ons in page layout tests, two mechanisms are used to weakly mix the add-ons:

  • multiple sort options
  • multiple categories

While it could be argued that traversing multiple pages of one category and one sort option also provides a variety of add-ons, the current implementation does not require a decision on the “best” sort option or category for a test case.

Sort options can affect layout of a page:

  • "Recently Added" sort may display Added date, "Recently Updated" sort may display Updated date.
  • On page 1 the "Popular" sort displays add-ons with ratings, the "Recently Added" displays add-ons that are not yet rated.

Breadcrumbs, Verbose mode, Tracing back to a page

In test cases that iterate through multiple add-ons, the test cases breadcrumbs provide context to when something happened during a test run. The TC breadcrumbs contain the current sequence of parameters and items being processed.  They can appear in exceptions and verbosity messages.

Tracing back to a page: The text “User: anonymous, App: firefox, Sort: popular, Add-on #2 ‘Firefox sync’” appearing in an exception would indicate the error occurred on the 2nd add-on on the page that’s sorted with the Popular option, etc.

When testcases generate exceptions with extended information, the third part of the extended information is the breadcrumbs string.

Verbose mode:

  • pre-POM test cases typically use the variable CONSOLE_TRACE to indicate when breadcrumbs are printed to the console (ex: 0 = no verbose messages, 1 = print options being processed, 2 = print options and add-ons being processed). Currently they are turned off in all test cases.
  • The POM test cases don’t have a verbose mode convention. A print statement using the breadcrumbs variable could be added to a test case as needed.


Building Breadrumbs

Pre-POM test cases use string variables named tc_breadcrumbs* and build breadcrumbs them by concatenating a new value to “parent” breadcrumbs. (The tc_* variable name prefix distinguishes it from the breadcrumb element that often appears in the page header.)

The following example sets the breadcrumbs which contains the current options (user, application, etc) and the next add-on in the list that will be processed. Based on the verbosity level the breadcrumbs string is printed.

 for addon_index in range(1, max_addons + 1):
  
    f = '%s, add-on #%s'
    tc_breadcrumbs_addon = f % (tc_breadcrumbs_options, addon_index)

    if CONSOLE_TRACE >= 2: print tc_breadcrumbs_addon


The POM test cases use TestCastBreadcrumbs object that's defined in the testcase_breadcrumbs module which stores a list of (type, value) pairs. The breadcrumbs are updated with an add(type, value) method. A __str__ method creates a formated string with the labels and values.

The SavedExceptions data type contains a breadcrumbs attribute.  POM test cases typically use the breadcrumbs attribute that's inside a SavedExceptions object, typically named ex.

 for addon_index in range(1, max_addons + 1):
  
    ex.breadcrumbs.add('add-on', '#%d' % addon_index)
 
    if CONSOLE_TRACE >= 2: print ex.breadcrumbs

Although the example has a conditional print, most POM test cases haven't implemented a verbosity.

Saved Exceptions

Test cases might capture certain errors and report them as a group at the end of the test. This is done where the same verifications are applied to multiple add-ons or pages, and it’s desirable to not end the test because of a minor discrepancy. (also see Troubleshooting AMO Test Runs wiki page.)

Each saved exception contains three parts that are appended together on one message:

  1. error summary. ex: “Updated date not found”, “Exception in reviews count link”
  2. error details including values that failed the verification. An error captured in a try-except block might use the string returned by the exception class. ex: “text ‘reviews’ does not match regex ‘[0-9]+ reviews’ ”, “rating on browse page is ‘5 stars’, on detail page is ‘4 stars’ “.
  3. test case breadcrumbs to help reconstruct the context the error happened on. ex: “User type: anonymous, App: firefox, Sort: popular, Add-on: #2 ‘Firefox sync’

Implementation

Pre-POM test cases

  • Exception messages saved in a list
  • Error summary counts saved in a dictionary
  • saveException( ) function is defined locally in each test case, taking the error summary, error detail and TC breadcrumbs as parameters.
  • saved exceptions detail and summary counts are printed by teardown( ) using a for-loop

POM test cases

  • The SavedExceptions class is defined in the saved_exceptions module. The object contains the exception list, error summary counts, and breadcrumbs. It also has a has_exceptions attribute which is True when any exceptions have been saved.
  • The class method save_exception(error_summary, error_detail) updates the exception detail list and error summary count. The internal breadcrumb attribute of the object is added to the detailed exception message.
  • The class method print_exceptions_to_stderr( ) prints the error summary counts and exception details, and raise an exception. Test cases call it from teardown( ).

Skipping Exceptions ( aka exception to the exception )

Sometimes a test fails because of a narrow set of conditions and a fix to the web page is not imminent. The known add-ons involved might be only one or a few. An actual example is a verification that expects add-on descriptions on on the browse and detail pages to be the same (accounting for the possibility the browse page description is truncated). The description for a particular add-on has an embedded URL which one the add-on detail page is followed by a space and not followed by space on the browse page.

Because the problem is minor it’s undesirable to disable the entire test in the suite, disable the verification in the test, or dilute the verification to avoid the condition (i.e. compare only the first 25 chars of the descriptions). To address situations similar to this, additional logic is added to to the exception handling that checks for the limited condition before generating an exception. If the limited condition is detected the normal exception message is printed to the console, otherwise the exception is generated as usual. The result is the test had the potential of finishing green despite the minor issue, without losing visibility to the exception-to-the-exception.

Implementation in test cases often takes the form similar to

 # if conditionX then there's an exception
 # if conditionY then we prefer not to generate a real exception

 if conditionX:  
     m1 = “this did not match that”
     m2 = “value found ‘%s’, expected ‘%s’ “ % (valueA, valueB)

     #  Bugzilla bug 123456
     #  skip exception if condition Y exists
     if conditionY:
         print “. skipped exception”, m1, “bug 123456”, m2
    else:
         ex.save_exception(m1, m2)      # or  raise Exception, (m1+m2)

Points to note

  • The word Bugzilla is included in the comment to provide a keyword for source code searches across all test cases.
  • The bug# appears in the comment, and more importantly in the print statement.
  • The information that would appear in the exception appears in the print statement
  • conditionY may include add-on IDs or sub-strings to search for, to keep the condition very narrow. The POM amo_page class has variables that define a list of add-ons related to particular bugs.
  • conditions X and Y appear in separate statements so that logic for the exception and exception-to-the-exception is clear.

Normally, a test case would not entirely skip add-ons because of a known condition in one element. On the other hand, an add-on might be skipped completely if a condition affects multiple elements or verifications, and would require an exception to multiple exceptions.