TestEngineering/Web/Automation/Gaia UI Testing

Summary

This is the home for all documentation related to the Gaia UI tests written and maintained by the Web QA Team.

Task List

Please see the Gaia UI Testing Task List for a list of daily tasks, in order of priority.

Running the Tests

Please see Gaia UI Tests Guidelines on MDN for information on running and writing Gaia UI tests.

Running on Desktop B2G

Please see Testing on Desktop B2G on MDN.

Running on Device

Please see Testing on Devices on MDN.

Monitoring Jenkins Jobs For Failures and XPasses

Note: This is very much a work in progress. Do not take anything below as gospel until it has been finalized.

The first priority, when supporting Gaia UI Testing, is to ensure that our existing tests are accurate and reliable. To that end, we need to monitor the jobs that run in our CI and address any that are failing or are xpassing (passing but marked as expected to fail).

To review the jobs for failures and xpasses, complete the following steps:

1. Choose the job to review. This involves choosing the branch and device.
   1. Which branch to review?
      1. We currently run jobs against the master and v1.2 branches of Gaia. When reviewing the job results for failures and xpasses, always review master first, and then v1.2.
   2. Which device to review?
      1. We currently run jobs on two different devices, Hamachi and Inari. When reviewing the job results for failures and xpasses, always review Hamachi first, and then Inari.
   3. Taking the above into consideration, the order for review should be:
      1. Hamachi master
      2. Hamachi v1.2
      3. Inari master
      4. Inari v1.2
2. Review the results of the latest run of the job in question.
   1. Review failures.
      1. You can use the links above to view the latest results of the job. This will give you a quick summary of which tests are currently failing. Looking at the Age of the failure will give you an idea of whether this is a new failure, or one that has been happening for awhile. Also, failures with an Age of 1 often indicate intermittent failures.
      2. For each failure that you find:
         1. If the failure details say: Error Message: test failure, Stacktrace: N/A, then the failure is likely actually an xpass. Keep this in mind when going through the next steps.
         2. Search for a bug in Bugzilla that has been opened for the failure. Every failure that needs to be addressed must have a bug opened for it. If you find an open bug for a failure, examine it:
            1. If it has a pull request attached to it which is waiting to be reviewed, review it. A pull request that is waiting to be reviewed is one that has no negative reviews and less than two positive reviews.
            2. If it is unassigned, and you would like to take the time to address it, assign it to yourself and update its status to Assigned. Do not do this unless you have the time to address it.
         3. If you do not find an open bug for the failure, open one and include as much detail about the failure as possible.
            1. If you have time to address the failure, assign it to yourself and update its status to Assigned.
   2. Address xpasses.
      1. For each xpass that you found during the Review failures step:
         1. Run the test locally to see if it also passes for you.
            1. If the test passes locally for you and has been passing on CI for more than one consecutive run, then the test should be un-xfailed:
               1. Look at the manifest.ini file that contains the test in question. It should contain a reference to a bug number for which the test was xfailed.
               2. Look at the bug in question and examine it:
                  1. If the bug has been closed, that likely explains why the test is now passing. :)
                  2. If the bug has not been closed, add a comment to the bug seeking clarification, explaining that the test is now passing.
               3. Check for an existing bug to un-xfail the test. If one exists it should be listed as being blocked by the current bug.
               4. If no bug exists to un-xfail the test, open one.
               5. Create a pull request to un-xfail the test and attach it to the un-xfail bug mentioned above.

Completing Requested Reviews

Your second priority, when supporting Gaia UI Testing, is to complete any reviews that have been requested of you. Reviews generally mean reviewing a pull request and Bugzilla is your friend in finding these. You can see all of your requested reviews on your Buzilla Dashboard in the section labelled Flags Requested of You.

Each bug that has you listed as a reviewer with a ? flag will appear on this list. Here is a recommended workflow for each pull request you review. Note that this uses the master branch as an example.

1. Make sure you are on the correct branch and that it is up to date:
   1. git checkout master && git pull upstream master
2. Create a new branch for the review:
   1. git checkout -b branch_name
3. Pull the changes for the pull request from the requester's branch:
   1. git pull url_of_requesters_repo branch_name
      1. Note that you may find it useful to add remotes for any repos for which you are frequently asked to perform reviews. You can do this using git remote add remote_name repo_url, and you can then use git pull remote_name branch_name instead of the above.
      2. You should not experience any merge conflicts when doing this pull. If you do, check the pull request. If it indicates that it cannot be merged then add a comment to the pull request that conflicts need to be resolved and flag the attachment on the bug as - to indicate it did not pass your review.
         1. If the pull request is not marked as being unable to be merged then something went wrong with your pull. Perhaps your local branch was not up to date. Start the process again at #1 above.
4. Run the affected test(s) against the branch in question to ensure that they all pass.
   1. Note that there may be several tests affected which may not be included in the pull request. For example, if a locator or method in an app object changes, then any tests that make use of that app object property or method would need to be tested.
   2. If the tests do not pass:
      1. Check to see if any are xfailed and note these. They are not expected to pass, so they should not affect your review.
      2. For any that are not xfailed, add a comment to the pull request to indicate which tests failed. Be sure to include the full stack trace of the failure(s) in your comments. Then flag the attachment on the bug as - to indicate it did not pass your review.
   3. Assuming that the tests are passing, continue to the next step.
5. Review the code in the pull request:
   1. Check against the expected style of the gaia-ui-tests.
   2. Check for pep8 errors.
   3. If any problems are found, add comments to the pull request and flag the attachment on the bug as - to indicate it did not pass your review.
   4. If you got this far and there are no problems with the pull request, flag the attachment on the bug as + to indicate it passed your review.
6. Merge the pull request:
   1. If you are the second reviewer to r+ the pull request then you may merge it. See merging pull requests for details.

Merging Pull Requests

Before merging a pull request, please ensure that:

1. The attachment to the bug has at least two positive (r+) reviews.
2. The Travis build of the pull request has passed.
   1. If the Travis build failed check it to see if the gaia_ui_tests step failed, and if it did whether any of the affected tests failed. Although it is preferred that no pull request be merged with a failing Travis build, if you are confident that the Travis failure had nothing to do with the changes in the pull request you may merge it, at your own discretion.
   2. You may also request a rerun of the Travis build, which you do via the Travis UI, if you would prefer to see it pass before merging.

When it is time to merge, do the following:

1. Visit the pull request and click the big green Merge pull request button.
2. Go to the bug and ensure that all reviews are cleared.
3. Add a comment to the bug with a link to the merge commit, indicating on which branch the patch landed.
4. If this was a merge to master, consider whether the patch should be uplifted. Follow the procedure in Changes Have Been Committed To Master.
5. If the bug is finished (e.g., no uplifts are needed), mark it as Resolved - Fixed.

Dealing with Travis and TBPL Failures

Currently, we do not monitor Travis or TBPL for failures, but we are sometimes informed of them and asked to investigate. When investigating failures on Travis or TBPL, be sure to run your tests against B2G Desktop, not a device, as that is how the tests are run in those environments.

You can find the executables for B2G desktop at:

• Master
• v1.2 - look for the ones prefixed with mozilla-b2g26_v1_2, e.g. mozilla-b2g26_v1_2-macosx64_gecko

Running like Travis locally.

There are two steps to running tests like Travis locally; install (downloads Gecko and builds Gaia) and run tests.

The tests must be run from the gaia/ root directory.

When running locally we need to be aware of 3 things:

1. The travis install script will not download a new Gecko if one already exists. The solution is to "rm -rf b2g" and "rm -rf profile" to remove the artifacts that the install script creates
2. The install script will install the Python packages at global level. The solution is to remove "sudo" from line 5 of the install script.
3. I forgot wot this one woz but for some reason in my mind I thought there were 3

The install script is run by: > tests/travis_ci/gaia-ui-tests/install

To run the tests, modify the script file to include the command line options you need. All normal command line options are valid.

> tests/travis_ci/gaia-ui-tests/script

Keeping Branches in Sync with Master

Currently, all development of Gaia UI tests is done on the master branch, with the exception of fixes that are needed specifically for a different branch. It is our goal to keep other branches of interest as closely synced with master as possible. When changes are committed to master, it is important to determine whether those changes should also be uplifted to other branches, and if so then the uplift must be completed.

Current Branches of Interest

• Master - this is the current development branch of Gaia and is priority #1 for failing tests. All new development (e.g., new tests) should be completed in the master branch and then, if necessary, uplifted to other branches of interest.
• v1.2 - this is the next shipping version of Gaia, and is priority #2 for failing tests. Changes should only be committed to v1.2 if they either:
  • land first in master and are deemed necessary for v1.2, or
  • contain fixes that are specific to v1.2

This means that the only current branches of interest are master and v1.2. For this reason the rest of this page will use v1.2 to mean any branch of interest.

After Changes Have Been Committed To Master

The following steps should be taken:

1. Determine whether the change should be uplifted to v1.2. There are a number of considerations for this including:
   1. If it is a new test:
      1. Are the features being tested available in v1.2? If yes, then it should be uplifted, otherwise do not uplift.
   2. If it is a fix to a failing test on master:
      1. Is it a locator or other change that causes the test to fail on master but not on v1.2? If yes, then do not uplift.
      2. Is it a test enhancement, such as an improved wait? If yes, then it should be uplifted.
   3. If, after considering the above, you are unsure as to whether the commit should be uplifted to v1.2, then flag it for uplifting and add a comment that you are not sure if it is needed or not. By flagging it for uplifting you are ensuring that someone will look at it and make that determination.
2. Flag the bug to indicate if it is to be uplifted or not, or complete the uplift yourself:
   1. If the commit should be uplifted (or you are unsure, as above), either:
      1. Flag it for uplifting by setting the status-b2g-v1.2 flag to affected, or
      2. Complete the uplift yourself. This is preferable to flagging it for someone else to complete. See the section below about Uplifting Commits.
      3. In either case, the bug should not be closed.
   2. If you determine that the commit does not need to be uplifted, flag it as such by setting the status-b2g-v1.2 flag to unaffected. The bug can be closed.

Uplifting Commits

Uplifting means to take the changes in a commit made to master, and bring those same changes into the branch of interest. Here is a recommended workflow. Note that this uses the v1.2 branch as an example.

1. Make sure you are on the correct branch and that it is up to date:
   1. git checkout v1.2 && git pull upstream v1.2
2. Create a new branch for the uplift:
   1. git checkout -b branch_name
3. Cherry pick the commit that was made to master. You can find the commit hash by looking at the Github pull request:
   1. git cherry-pick commit-hash
   2. Please note: Use the commit hash of the original commit of code (the one submitted by the pull request author) as opposed to the commit hash of the merge commit (the one created by the person who merged the pull request). For example, for this pull request, you would choose commit hash a9798fc, which is the hash of Bob's original commit, not commit hash 42de07a, which is the commit hash of Zac's merge commit.
   3. The cherry pick may or may not complete without conflicts. If there are conflicts, resolve them manually.
      1. If there were conflicts, commit your changes to your branch:
         1. git commit -a
         2. The commit message should be pre-populated with the commit message from when the changes were committed to master. Leave this commit message intact. You may add a second line to the message to indicate that this is an uplift to the branch of interest.
4. Run the affected test(s) against v1.2 to ensure that they still pass.
   1. Note that there may be several tests affected which may not be included in the commit. For example, if a locator or method in an app object changes, then any tests that make use of that app object property or method would need to be tested.
   2. If the tests do not pass, try to determine why and whether or not they can be made to pass by making other changes. If you are unable to get the tests to pass on v1.2, and it isn't because they are attempting to test features that do not exist on v1.2, then make a comment in the bug about this, with the full details of the failures, and leave the bug flagged as status-b2g-v1.2: affected.
   3. Assuming that the tests are passing, continue to the next step.
5. Push your changes to your Github repo:
   1. git push origin branch_name
6. Go to Github and issue a pull request against your branch.
   1. Make sure the pull request is being issued against the correct upstream branch (e.g., v1.2). This will likely not be the default so you will have to change it manually.
7. Attach the pull request to the bug.
   1. This can be done from the pull request if you have the Github Bugzilla Tweaks add-on installed.
   2. This can also be done by pasting the url to the Github pull request into an attachment to the bug using 'paste text as attachment'.
   3. Flag the attachment for review and add a comment that this patch is to uplift to the branch of interest.
8. When the pull request for the uplift has been merged, set the status-b2g-v1.2 flag to fixed and close the bug.

Reviewing and Merging Uplifts

Often a pull request which is an uplift from master to a branch of interest will contain code that is identical to that which was committed to master. That code has already been reviewed for correctness and style, so it may not be necessary to review the code itself as carefully. You will need to determine whether that is the case yourself. You could do that by comparing the files changed, after applying the commit in the pull request, with the same files in the master branch.

Regardless of the decision made above, follow the procedure in Merging Pull Requests

Finding Bugs That Require Uplifts

All bugs that require uplifts should be flagged as status-b2g-v1.2: affected (or whichever flag corresponds to the branch of interest). You can execute a Bugzilla query such as this one, to view all such bugs and then assign yourself to any that you plan to complete.

Keeping On Top of Changes

There are some steps that can be taken to try to keep the branches in sync. These include:

1. As mentioned above, if you can complete an uplift yourself, by creating a pull request, rather than simply flagging a bug as affected, that is preferable.
2. Monitor bugs that are flagged as status-b2g-v1.2: affected as described in Finding Bugs That Require Uplifts.
3. Complete a manual audit of commits that have landed in master but not v1.2. This is something that should be done once or twice per month.
   1. Generate a list of all the commits that have landed in master but not v1.2 using this command: git show master ^v1.2 --no-merges -- tests/python/gaia-ui-tests.
   2. Go through the list and verify whether any of those commits need to be uplifted to v1.2.
      1. Some will not require uplifting because they are not applicable to v1.2.
      2. Some of the changes may already be in v1.2, but were added using different commits (e.g, via a cherry pick which resulted in conflicts).
      3. For any that you find that do need to be uplifted to v1.2, reopen the bug to which the original pull request was added and either:
         1. Complete the uplift yourself by opening a pull request for v1.2 and attaching it to the bug, or
         2. Flag it as status-b2g-v1.2: affected.