|
|
| (21 intermediate revisions by 2 users not shown) |
| Line 3: |
Line 3: |
| <br> | | <br> |
| <br> | | <br> |
| It is also recommended to review Geo's draft of [https://github.com/geoelectric/ui_auto_best_practice/blob/master/uiabp.md automation best practices], which gives a theoretical overview of writing automation tests.
| |
|
| |
|
| = External Tools / Resources = | | == General Disclaimer == |
| == Using webIDE to define locators == | | As we develop more knowledge some tests might fall behind the standards in this style guide. It can be tempting to want to fix all of the outdated style but in order to keep patches/pulls small (see above!) we are happy to have new and old standards of code sit side by side. As we regularly review and update tests the project will be brought completely up to our current standards. |
| njpark - TBD
| |
|
| |
|
| == CSS Selectors ==
| | Or if you prefer, file a [https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20OS&component=Gaia%3A%3AUI%20Tests Bugzilla issue] to have a section of code addressed separately to the job you are doing. |
| Use following references as guides.
| |
| * [http://www.w3.org/TR/css3-selectors/ W3]
| |
| * [http://www.cheat-sheets.org/saved-copy/Locators_table_1_0_2.pdf Locator Cheat Sheet]
| |
|
| |
|
| = Tests - Style = | | == Outside Test Consumers == |
| == Naming ==
| | * Be aware if you change or remove the Gaia UI test API, that outside consumers might get broken. |
| * Module names should be called test_ and then behavioral areas. | |
| test_search.py
| |
|
| |
|
| * Test method names should always show the intent of the test case. | | Current outside consumers of Gaiatest are: |
| | * [https://github.com/Mozilla-TWQA/MTBF-Driver MTBF] |
| | * [https://github.com/mozilla-b2g/gaia/tree/master/tests/python/gaia-ui-tests/gaiatest/tests/accessibility a11y] |
| | * [https://github.com/JonHylands/power-tests Power-tests] |
| | * [https://github.com/mozilla/marketplace-tests-gaia/tree/master/marketplacetests Marketplace tests] |
| | * [https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/Run_ImageMagick_Tool_with_Python_Marionette Imagecompare] tests |
| | * [https://arewefastyet.com/#machine=26 awfy] |
|
| |
|
| <source lang="python">
| | == [[B2G/QA/Automation/Style Guide/General Concepts|General Concepts]] == |
| # Good
| | == [[B2G/QA/Automation/Style Guide/Howtos| How-Tos ]] == |
| def test_that_advanced_search_does_not_find_item(self):
| |
|
| |
|
| # Bad
| | == [[B2G/QA/Automation/Style_Guide/Python_Script_Style|Python Script Style Guide]] == |
| def test_advanced_search(self):
| |
| </source>
| |
|
| |
|
| == File Headers == | | == [[B2G/QA/Automation/Style Guide/Best Practices|Best Practices]] == |
| * Each file should have a completed copy of the [http://www.mozilla.org/MPL/2.0/ MPL2] license block, immediately followed by an empty line.
| |
| * Each file should pass [http://www.python.org/dev/peps/pep-0008/ PEP8] except for line length, see below.
| |
|
| |
|
| <source lang="python">
| | == [[B2G/QA/Automation/Style_Guide/Avoiding_Duplication|Avoiding Duplication]] == |
| # Good
| |
| def method(self, parameter)
| |
|
| |
|
| # Bad
| | == [[B2G/QA/Automation/Style_Guide/Submission_Review|Submission and Review]] == |
| def method(self,parameter)
| |
| </source>
| |
|
| |
|
| * Lines should try not to have more than 100 characters.
| | == [https://github.com/geoelectric/ui_auto_best_practice/blob/master/uiabp.md Further Reading] == |
| * Docstrings should conform to [http://www.python.org/dev/peps/pep-0257/ PEP0257] and should be on a single line wherever possible.
| |
| | |
| <source lang="python">
| |
| # Good
| |
| def click_login():
| |
| """Clicks the login link."""
| |
| | |
| # Bad
| |
| def click_login():
| |
| """
| |
| Clicks the login link.
| |
| """
| |
| </source>
| |
| | |
| Where not possible, the first line should be a summary.
| |
| | |
| <source lang="python">
| |
| # Good
| |
| def login():
| |
| """Logs in.
| |
| | |
| Clicks the login link and then waits for the home page to load.
| |
| | |
| """
| |
| | |
| # Bad
| |
| def login():
| |
| """Logs in.
| |
| Clicks the login link and then waits for the home page to load."""
| |
| </source>
| |
| | |
| * Indenting should be a soft tab (4 spaces) as common with in Python. Do not mix tabs and spaces!
| |
| * There should be no whitespace at the end of the file (as per PEP8).
| |
| * Comments should be on the line above. Remember to update comments when changing code so that code matches the comments.
| |
| * Comments should be used carefully. Try not to put comments that don't provide any additional value
| |
| * Class names should be in Pascal style as this is Python idiomatic.
| |
| | |
| <source lang="python">
| |
| # Good
| |
| class TestThisSite:
| |
|
| |
| # Bad
| |
| class test_this_site:
| |
| </source>
| |
| | |
| == Actions ==
| |
| * Methods that perform actions on the page should indicate the action in the method name.
| |
| | |
| <source lang="python">
| |
| # Good
| |
| def click_report_with_length(length)
| |
| | |
| # Bad
| |
| def report_length(length)
| |
| </source>
| |
| | |
| * Actions should wait for the appropriate action to complete. This could be an implicit or explicit wait. For example, clicking a login button might explicitly wait for a username field to be visible.
| |
| | |
| == Use testvars.json ==
| |
| TBD - njpark
| |
| == Using high-level methods ==
| |
| TBD - jlorenzo
| |
| * In a test, make sure to only have steps that are high level (for instance: messages.send_an_sms_to_yourself() instead of detailing every single tap and click)
| |
| == Avoid sleep() calls ==
| |
| TBD - njpark
| |
| == Variable naming ==
| |
| TBD - jlorenzo
| |
| * Name your variables with units. For example: timeout_in_seconds, width_in_pixels
| |
| * Don't shorten variable names. A bad example: self.t_out_err_tim = 1000 instead of self.time_out_error_time
| |
| == app.py and regions/helper.py ==
| |
| TBD - njpark
| |
| == How to create filename strings ==
| |
| TBD - njpark
| |
| http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/apps/settings/app.py#296
| |
| == Meaningful custom Assert() messages ==
| |
| TBD - njpark
| |
| https://bugzilla.mozilla.org/show_bug.cgi?id=1198449#c2
| |
| == Clean up afterwards ==
| |
| TBD - njpark
| |
| == Avoid code duplication ==
| |
| TBD - jlorenzo
| |
| * Use parameterized()
| |
| * Centralize workarounds
| |
| == Simulate end-user check ==
| |
| TBD - jlorenzo
| |
| * Check what an end-user would check.
| |
| == Making test multi-locale ==
| |
| TBD - njpark
| |
| * we check the raw text in order to verify the input from the user (for instance a phone number put in the dialer, we verify it appears in the call log)
| |
| * we test the l10n-id for any string that comes from Gaia only (like an error message)
| |
| == Be aware of outside consumers of ui-test ==
| |
| TBD - mwargers
| |
| * Be aware if you change the Gaia UI test API, that outside consumers (mtbf, etc) might get broken
| |
| | |
| | |
| = Logic =
| |
| == Handling shadow DOM ==
| |
| TBD - njpark
| |
| == Switching frames and system frame ==
| |
| TBD - mwargers
| |
| == Handling browser instances == | |
| TBD - mwargers
| |
| == Returning the page object after completing an action ==
| |
| TBD - njpark
| |
| When writing methods that is doing some action on the device, for instance opening an app or opening a subpage, make sure that the resulting action is finished and return the object of the resulting action, e.g. the app or the subpage (PageRegion)
| |
| | |
| | |
| = Use of Libraries =
| |
| == Page Objects ==
| |
| TBD - jlorenzo
| |
| * where to call marionette's APIs
| |
| * why a page class shouldn't import data from the test
| |
| == General ==
| |
| * All page objects should inherit from <code>Page</code> in page.py.
| |
| * Page objects should not do asserts. This should be done within the test.
| |
| * Each page should be grouped within one module.
| |
| * If using multiple words to describe a module separate them with underscores '_'
| |
| * Single quotes (') should be used instead of double (") throughout.
| |
| * Methods should have a single purpose.
| |
| | |
| == Limited use of conditionals ==
| |
| * Methods should not contain logic that depends on properties of the page. The logic and expectations should be within the test, and adding this to the page object could guard your tests against genuine failures.
| |
| | |
| <source lang="python">
| |
| # Good
| |
| def click_login(self)
| |
| self.selenium.find_element(*self._login_locator).click()
| |
| | |
| # Bad
| |
| def click_login(self)
| |
| if not self.is_user_logged_in:
| |
| self.selenium.find_element(*self._login_locator).click()
| |
| else:
| |
| pass
| |
| </source>
| |
| | |
| == Locators ==
| |
| TBD - jlorenzo
| |
| * Locator variables should be prefixed with <code>_</code> to show that it is [http://docs.python.org/tutorial/classes.html#private-variables private].
| |
| * Variables should be descriptive of the area and not clash with any properties.
| |
| * Should have a suffix of <code>_locator</code>.
| |
| * Accessing locators should be done through a property or method as this keeps the locator as read-only.
| |
| | |
| <source lang="python">
| |
| @property
| |
| def search_term(self):
| |
| return self.selenium.find_element(*self._search_box_locator).value
| |
| </source>
| |
| | |
| * We should use locators in the following order of preference (there will be exceptions):
| |
| ** ID
| |
| ** Name
| |
| ** Class name
| |
| ** CSS selector
| |
| ** XPath
| |
| * CSS locators should use whitespace for readability when using direct descendants.
| |
| | |
| <source lang="python">
| |
| # Good
| |
| _my_locator = "css=#content > p > a"
| |
|
| |
| # Bad
| |
| _my_locator = "css=#content>p>a"
| |
| </source>
| |
| | |
| * Use Python tuples to define locators:
| |
| | |
| <source lang="python">
| |
| # Good
| |
| _my_locator = (By.ID, "content")
| |
| </source>
| |
| | |
| == PageRegions ==
| |
| TBD - jlorenzo
| |
| * How to deal with stale root_elements
| |
| In some circumstances, for example where a header/navigation is common across the website, we will use a page region. The page region is a child class of the base Page object, which is inherited by all page objects. This means that the navigation can be reached from any page object and herein lies the DRY!
| |
| | |
| A brief example:
| |
| | |
| <source lang="python">
| |
| class BasePage(Page):
| |
| | |
| @property
| |
| def header(self):
| |
| return BasePage.HeaderRegion(self.testsetup)
| |
|
| |
| class HeaderRegion(Page):
| |
| | |
| _login_link = (By.ID, "home")
| |
| | |
| @def click_login(self):
| |
| self.selenium.find_element(*self._login_link).click()
| |
| </source>
| |
| | |
| Referring to this page region with a property makes it very readable and concise from within the test. Clicking login during a test would be performed like this:
| |
| <source lang="python">
| |
| my_page.header.click_login()
| |
| </source>
| |
| | |
| Another example where this might be used is on a search results page, the page region being the search results element.
| |
| | |
| == Assertions ==
| |
| TBD - jlorenzo
| |
| * Tests should handle the asserts -- not the page objects.
| |
| * Tests should use Python's native [https://docs.python.org/2/reference/simple_stmts.html#the-assert-statement assert] statement.
| |
| * When doing equivalency assertions, put the expected value first, followed by the actual value, for example:
| |
| <source lang="python">
| |
| # Good
| |
| a = some_function()
| |
| assert 'expected result' == a
| |
| | |
| # Bad
| |
| a = some_function()
| |
| assert a == 'expected result'
| |
| </source>
| |
| == How to use GaiaHeader and GaiaBinaryControl ==
| |
| TBD - jlorenzo
| |
| | |
| | |
| | |
| = Submitting and Reviewing Patches =
| |
| == Submission ==
| |
| To make sure that we can review your patch as quickly and efficiently as possibly we would like patches to have a single test in them and the necessary changes to the page objects. This also limits the chances of merge conflicts later.
| |
| <br>
| |
| Please refer to [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/Submitting_a_Gaia_patch this page] for gaia patch submission steps.
| |
| == Review ==
| |
| First of all, make sure above mentioned guidelines are not violated, and check for the following:
| |
| <br>
| |
| === Look for transitions that are not waited on ===
| |
| TBD - jlorenzo
| |
| === Check for change/removal of methods ===
| |
| In order to make sure that the change is cascaded, `git grep` is your friend.
| |
| TBD - jlorenzo
| |
| === Are the locators correct? Could they be improved? ===
| |
| TBD - jlorenzo
| |
| === Are the locators/methods used that were added? ===
| |
| TBD - mwargers
| |
| A linter would also help to enforce it, see: https://bugzilla.mozilla.org/show_bug.cgi?id=1186388
| |
| === Are workarounds explained in the code file? ===
| |
| TBD - jlorenzo
| |