|
|
| (14 intermediate revisions by 2 users not shown) |
| Line 3: |
Line 3: |
| <br> | | <br> |
| <br> | | <br> |
| It is also recommended to review Geo's draft of [https://github.com/geoelectric/ui_auto_best_practice/blob/master/uiabp.md automation best practices], which gives a theoretical overview of writing automation tests.
| |
|
| |
|
| = [[B2G/QA/Automation/Style Guide/General Concepts|General Concepts]] = | | == General Disclaimer == |
| = External Tools / Resources =
| | As we develop more knowledge some tests might fall behind the standards in this style guide. It can be tempting to want to fix all of the outdated style but in order to keep patches/pulls small (see above!) we are happy to have new and old standards of code sit side by side. As we regularly review and update tests the project will be brought completely up to our current standards. |
| = Tests - Style =
| |
| == Naming ==
| |
| * Module names should be called test_ and then behavioral areas.
| |
| test_search.py
| |
|
| |
|
| * Test method names should always show the intent of the test case.
| | Or if you prefer, file a [https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20OS&component=Gaia%3A%3AUI%20Tests Bugzilla issue] to have a section of code addressed separately to the job you are doing. |
|
| |
|
| <source lang="python">
| | == Outside Test Consumers == |
| # Good
| | * Be aware if you change or remove the Gaia UI test API, that outside consumers might get broken. |
| def test_that_advanced_search_does_not_find_item(self):
| |
|
| |
|
| # Bad
| | Current outside consumers of Gaiatest are: |
| def test_advanced_search(self):
| | * [https://github.com/Mozilla-TWQA/MTBF-Driver MTBF] |
| </source>
| | * [https://github.com/mozilla-b2g/gaia/tree/master/tests/python/gaia-ui-tests/gaiatest/tests/accessibility a11y] |
| | * [https://github.com/JonHylands/power-tests Power-tests] |
| | * [https://github.com/mozilla/marketplace-tests-gaia/tree/master/marketplacetests Marketplace tests] |
| | * [https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/Run_ImageMagick_Tool_with_Python_Marionette Imagecompare] tests |
| | * [https://arewefastyet.com/#machine=26 awfy] |
|
| |
|
| == File Headers == | | == [[B2G/QA/Automation/Style Guide/General Concepts|General Concepts]] == |
| * Each file should have a completed copy of the [http://www.mozilla.org/MPL/2.0/ MPL2] license block, immediately followed by an empty line.
| | == [[B2G/QA/Automation/Style Guide/Howtos| How-Tos ]] == |
| * Each file should pass [http://www.python.org/dev/peps/pep-0008/ PEP8] except for line length, see below.
| |
|
| |
|
| <source lang="python">
| | == [[B2G/QA/Automation/Style_Guide/Python_Script_Style|Python Script Style Guide]] == |
| # Good
| |
| def method(self, parameter)
| |
|
| |
|
| # Bad
| | == [[B2G/QA/Automation/Style Guide/Best Practices|Best Practices]] == |
| def method(self,parameter)
| |
| </source>
| |
|
| |
|
| * Lines should try not to have more than 100 characters.
| | == [[B2G/QA/Automation/Style_Guide/Avoiding_Duplication|Avoiding Duplication]] == |
| * Docstrings should conform to [http://www.python.org/dev/peps/pep-0257/ PEP0257] and should be on a single line wherever possible.
| |
|
| |
|
| <source lang="python">
| | == [[B2G/QA/Automation/Style_Guide/Submission_Review|Submission and Review]] == |
| # Good
| |
| def click_login():
| |
| """Clicks the login link."""
| |
|
| |
|
| # Bad
| | == [https://github.com/geoelectric/ui_auto_best_practice/blob/master/uiabp.md Further Reading] == |
| def click_login():
| |
| """
| |
| Clicks the login link.
| |
| """
| |
| </source>
| |
| | |
| Where not possible, the first line should be a summary.
| |
| | |
| <source lang="python">
| |
| # Good
| |
| def login():
| |
| """Logs in.
| |
| | |
| Clicks the login link and then waits for the home page to load.
| |
| | |
| """
| |
| | |
| # Bad
| |
| def login():
| |
| """Logs in.
| |
| Clicks the login link and then waits for the home page to load."""
| |
| </source>
| |
| | |
| * Indenting should be a soft tab (4 spaces) as common with in Python. Do not mix tabs and spaces!
| |
| * There should be no whitespace at the end of the file (as per PEP8).
| |
| * Comments should be on the line above. Remember to update comments when changing code so that code matches the comments.
| |
| * Comments should be used carefully. Try not to put comments that don't provide any additional value
| |
| * Class names should be in Pascal style as this is Python idiomatic.
| |
| | |
| <source lang="python">
| |
| # Good
| |
| class TestThisSite:
| |
|
| |
| # Bad
| |
| class test_this_site:
| |
| </source>
| |
| | |
| == Actions ==
| |
| * Methods that perform actions on the page should indicate the action in the method name.
| |
| | |
| <source lang="python">
| |
| # Good
| |
| def click_report_with_length(length)
| |
| | |
| # Bad
| |
| def report_length(length)
| |
| </source>
| |
| | |
| * Actions should wait for the appropriate action to complete. This could be an implicit or explicit wait. For example, clicking a login button might explicitly wait for a username field to be visible.
| |
| | |
| == Use testvars.json ==
| |
| TBD - njpark
| |
| == Using high-level methods ==
| |
| TBD - jlorenzo
| |
| * In a test, make sure to only have steps that are high level (for instance: messages.send_an_sms_to_yourself() instead of detailing every single tap and click)
| |
| == Avoid sleep() calls ==
| |
| TBD - njpark
| |
| == Variable naming ==
| |
| TBD - jlorenzo
| |
| * Name your variables with units. For example: timeout_in_seconds, width_in_pixels
| |
| * Don't shorten variable names. A bad example: self.t_out_err_tim = 1000 instead of self.time_out_error_time
| |
| == app.py and regions/helper.py ==
| |
| TBD - njpark
| |
| == How to create filename strings ==
| |
| TBD - njpark
| |
| http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/apps/settings/app.py#296
| |
| == Meaningful custom Assert() messages ==
| |
| TBD - njpark
| |
| https://bugzilla.mozilla.org/show_bug.cgi?id=1198449#c2
| |
| == Clean up afterwards ==
| |
| TBD - njpark
| |
| == Avoid code duplication ==
| |
| TBD - jlorenzo
| |
| * Use parameterized()
| |
| * Centralize workarounds
| |
| == Simulate end-user check ==
| |
| TBD - jlorenzo
| |
| * Check what an end-user would check.
| |
| * Make sure manifest.ini is updated with right flags
| |
| | |
| == Making test multi-locale ==
| |
| TBD - njpark
| |
| * we check the raw text in order to verify the input from the user (for instance a phone number put in the dialer, we verify it appears in the call log)
| |
| * we test the l10n-id for any string that comes from Gaia only (like an error message)
| |
| == Be aware of outside consumers of ui-test ==
| |
| TBD - mwargers
| |
| * Be aware if you change the Gaia UI test API, that outside consumers (mtbf, etc) might get broken
| |
| | |
| | |
| = Logic =
| |
| == Handling shadow DOM ==
| |
| TBD - njpark
| |
| == Switching frames and system frame ==
| |
| TBD - mwargers
| |
| == Handling browser instances ==
| |
| TBD - mwargers
| |
| == Returning the page object after completing an action ==
| |
| TBD - njpark
| |
| When writing methods that is doing some action on the device, for instance opening an app or opening a subpage, make sure that the resulting action is finished and return the object of the resulting action, e.g. the app or the subpage (PageRegion)
| |
| | |
| | |
| = Use of Libraries =
| |
| == Limited use of conditionals ==
| |
| * Methods should not contain logic that depends on properties of the page. The logic and expectations should be within the test, and adding this to the page object could guard your tests against genuine failures.
| |
| | |
| <source lang="python">
| |
| # Good
| |
| def click_login(self)
| |
| self.selenium.find_element(*self._login_locator).click()
| |
| | |
| # Bad
| |
| def click_login(self)
| |
| if not self.is_user_logged_in:
| |
| self.selenium.find_element(*self._login_locator).click()
| |
| else:
| |
| pass
| |
| </source>
| |
| | |
| == Locators ==
| |
| TBD - jlorenzo
| |
| * Locator variables should be prefixed with <code>_</code> to show that it is [http://docs.python.org/tutorial/classes.html#private-variables private].
| |
| * Variables should be descriptive of the area and not clash with any properties.
| |
| * Should have a suffix of <code>_locator</code>.
| |
| * Accessing locators should be done through a property or method as this keeps the locator as read-only.
| |
| | |
| <source lang="python">
| |
| @property
| |
| def search_term(self):
| |
| return self.selenium.find_element(*self._search_box_locator).value
| |
| </source>
| |
| | |
| * We should use locators in the following order of preference (there will be exceptions):
| |
| ** ID
| |
| ** Name
| |
| ** Class name
| |
| ** CSS selector
| |
| ** XPath
| |
| * CSS locators should use whitespace for readability when using direct descendants.
| |
| | |
| <source lang="python">
| |
| # Good
| |
| _my_locator = "css=#content > p > a"
| |
|
| |
| # Bad
| |
| _my_locator = "css=#content>p>a"
| |
| </source>
| |
| | |
| * Use Python tuples to define locators:
| |
| | |
| <source lang="python">
| |
| # Good
| |
| _my_locator = (By.ID, "content")
| |
| </source>
| |
| | |
| == PageRegions ==
| |
| TBD - jlorenzo
| |
| * How to deal with stale root_elements
| |
| In some circumstances, for example where a header/navigation is common across the website, we will use a page region. The page region is a child class of the base Page object, which is inherited by all page objects. This means that the navigation can be reached from any page object and herein lies the DRY!
| |
| | |
| A brief example:
| |
| | |
| <source lang="python">
| |
| class BasePage(Page):
| |
| | |
| @property
| |
| def header(self):
| |
| return BasePage.HeaderRegion(self.testsetup)
| |
|
| |
| class HeaderRegion(Page):
| |
| | |
| _login_link = (By.ID, "home")
| |
| | |
| @def click_login(self):
| |
| self.selenium.find_element(*self._login_link).click()
| |
| </source>
| |
| | |
| Referring to this page region with a property makes it very readable and concise from within the test. Clicking login during a test would be performed like this:
| |
| <source lang="python">
| |
| my_page.header.click_login()
| |
| </source>
| |
| | |
| Another example where this might be used is on a search results page, the page region being the search results element.
| |
| | |
| == Assertions ==
| |
| TBD - jlorenzo
| |
| * Tests should handle the asserts -- not the page objects.
| |
| * Tests should use Python's native [https://docs.python.org/2/reference/simple_stmts.html#the-assert-statement assert] statement.
| |
| * When doing equivalency assertions, put the expected value first, followed by the actual value, for example:
| |
| <source lang="python">
| |
| # Good
| |
| a = some_function()
| |
| assert 'expected result' == a
| |
| | |
| # Bad
| |
| a = some_function()
| |
| assert a == 'expected result'
| |
| </source>
| |
| == How to use GaiaHeader and GaiaBinaryControl ==
| |
| TBD - jlorenzo
| |
| | |
| = Submitting and Reviewing Patches =
| |
| == Submission ==
| |
| To make sure that we can review your patch as quickly and efficiently as possibly we would like patches to have a single test in them and the necessary changes to the page objects. This also limits the chances of merge conflicts later.
| |
| <br>
| |
| Please refer to [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/Submitting_a_Gaia_patch this page] for gaia patch submission steps.
| |
| == Review ==
| |
| First of all, make sure above mentioned guidelines are not violated, and check for the following:
| |
| <br>
| |
| === Look for transitions that are not waited on ===
| |
| TBD - jlorenzo
| |
| === Check for change/removal of methods ===
| |
| In order to make sure that the change is cascaded, `git grep` is your friend.
| |
| TBD - jlorenzo
| |
| === Are the locators correct? Could they be improved? ===
| |
| TBD - jlorenzo
| |
| === Are the locators/methods used that were added? ===
| |
| TBD - mwargers
| |
| A linter would also help to enforce it, see: https://bugzilla.mozilla.org/show_bug.cgi?id=1186388
| |
| === Are workarounds explained in the code file? ===
| |
| TBD - jlorenzo
| |