81
edits
TestEngineering/Web/Automation/Gaia UI Testing: Difference between revisions
TestEngineering/Web/Automation/Gaia UI Testing (view source)
Revision as of 20:24, 26 August 2016
, 26 August 2016StuartPhilp moved page QA/Execution/Web Testing/Automation/Gaia UI Testing to TestEngineering/Web/Automation/Gaia UI Testing: reorganizing wiki under Test Engineering
StuartPhilp (talk | contribs) m (StuartPhilp moved page QA/Execution/Web Testing/Automation/Gaia UI Testing to TestEngineering/Web/Automation/Gaia UI Testing: reorganizing wiki under Test Engineering) |
|||
| (8 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
= Summary = | = Summary = | ||
This is the home for all documentation related to the Gaia UI tests written and maintained by the Web QA Team. | This is the home for all documentation related to the Gaia UI tests written and maintained by the Web QA Team. | ||
= Task List = | |||
Please see the [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing/Task_List|Gaia UI Testing Task List]] for a list of daily tasks, in order of priority. | |||
= Running the Tests = | = Running the Tests = | ||
| Line 95: | Line 99: | ||
# Go to the bug and ensure that all reviews are cleared. | # Go to the bug and ensure that all reviews are cleared. | ||
# Add a comment to the bug with a link to the merge commit, indicating on which branch the patch landed. | # Add a comment to the bug with a link to the merge commit, indicating on which branch the patch landed. | ||
# If this was a merge to master, consider whether the patch should be uplifted. Follow the procedure in [ | # If this was a merge to master, consider whether the patch should be uplifted. Follow the procedure in [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing#After_Changes_Have_Been_Committed_To_Master After|Changes Have Been Committed To Master]]. | ||
# If the bug is finished (e.g., no uplifts are needed), mark it as Resolved - Fixed. | # If the bug is finished (e.g., no uplifts are needed), mark it as Resolved - Fixed. | ||
| Line 104: | Line 108: | ||
You can find the executables for B2G desktop at: | You can find the executables for B2G desktop at: | ||
* [https://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/latest-mozilla-central/ Master] | * [https://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/latest-mozilla-central/ Master] | ||
* [https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ v1.2] - look for the ones prefixed with mozilla-b2g26_v1_2, e.g. [https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-b2g26_v1_2-macosx64_gecko/ mozilla-b2g26_v1_2-macosx64_gecko] | * [https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ v1.2] - look for the ones prefixed with mozilla-b2g26_v1_2, e.g. [https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-b2g26_v1_2-macosx64_gecko/ mozilla-b2g26_v1_2-macosx64_gecko] | ||
Running like Travis locally. | |||
There are two steps to running tests like Travis locally; install (downloads Gecko and builds Gaia) and run tests. | |||
The tests must be run from the gaia/ root directory. | |||
When running locally we need to be aware of 3 things: | |||
# The travis install script will not download a new Gecko if one already exists. The solution is to "rm -rf b2g" and "rm -rf profile" to remove the artifacts that the install script creates | |||
# The install script will install the Python packages at global level. The solution is to remove "sudo" from line 5 of the install script. | |||
# I forgot wot this one woz but for some reason in my mind I thought there were 3 | |||
The install script is run by: | |||
> tests/travis_ci/gaia-ui-tests/install | |||
To run the tests, modify the script file to include the command line options you need. All normal command line options are valid. | |||
> tests/travis_ci/gaia-ui-tests/script | |||
= Keeping Branches in Sync with Master = | = Keeping Branches in Sync with Master = | ||
| Line 133: | Line 156: | ||
## If the commit should be uplifted (or you are unsure, as above), either: | ## If the commit should be uplifted (or you are unsure, as above), either: | ||
### Flag it for uplifting by setting the ''status-b2g-v1.2'' flag to ''affected'', or | ### Flag it for uplifting by setting the ''status-b2g-v1.2'' flag to ''affected'', or | ||
### Complete the uplift yourself. This is preferable to flagging it for someone else to complete. See the section below about [ | ### Complete the uplift yourself. This is preferable to flagging it for someone else to complete. See the section below about [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing#Uplifting_Commits|Uplifting Commits]]. | ||
### In either case, the bug should not be closed. | ### In either case, the bug should not be closed. | ||
## If you determine that the commit does not need to be uplifted, flag it as such by setting the ''status-b2g-v1.2'' flag to ''unaffected''. The bug can be closed. | ## If you determine that the commit does not need to be uplifted, flag it as such by setting the ''status-b2g-v1.2'' flag to ''unaffected''. The bug can be closed. | ||
| Line 158: | Line 181: | ||
# Push your changes to your Github repo: | # Push your changes to your Github repo: | ||
## <code>git push origin branch_name</code> | ## <code>git push origin branch_name</code> | ||
# Go to Github and issue a pull request against your branch. | # Go to Github and issue a pull request against your branch. | ||
## Make sure the pull request is being issued against the correct upstream branch (e.g., v1.2). This will likely not be the default so you will have to change it manually. | ## Make sure the pull request is being issued against the correct upstream branch (e.g., v1.2). This will likely not be the default so you will have to change it manually. | ||
| Line 165: | Line 187: | ||
## This can also be done by pasting the url to the Github pull request into an attachment to the bug using 'paste text as attachment'. | ## This can also be done by pasting the url to the Github pull request into an attachment to the bug using 'paste text as attachment'. | ||
## Flag the attachment for review and add a comment that this patch is to uplift to the branch of interest. | ## Flag the attachment for review and add a comment that this patch is to uplift to the branch of interest. | ||
# When the pull request for the uplift has been merged, set the ''status-b2g-v1.2'' flag to ''fixed'' and close the bug. | |||
== Reviewing and Merging Uplifts == | == Reviewing and Merging Uplifts == | ||
| Line 170: | Line 193: | ||
Often a pull request which is an uplift from master to a branch of interest will contain code that is identical to that which was committed to master. That code has already been reviewed for correctness and style, so it may not be necessary to review the code itself as carefully. You will need to determine whether that is the case yourself. You could do that by comparing the files changed, after applying the commit in the pull request, with the same files in the master branch. | Often a pull request which is an uplift from master to a branch of interest will contain code that is identical to that which was committed to master. That code has already been reviewed for correctness and style, so it may not be necessary to review the code itself as carefully. You will need to determine whether that is the case yourself. You could do that by comparing the files changed, after applying the commit in the pull request, with the same files in the master branch. | ||
Regardless of the decision made above, follow the procedure in [ | Regardless of the decision made above, follow the procedure in [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing#Merging_Pull_Requests|Merging Pull Requests]] | ||
== Finding Bugs That Require Uplifts == | == Finding Bugs That Require Uplifts == | ||
| Line 181: | Line 204: | ||
# As mentioned above, if you can complete an uplift yourself, by creating a pull request, rather than simply flagging a bug as affected, that is preferable. | # As mentioned above, if you can complete an uplift yourself, by creating a pull request, rather than simply flagging a bug as affected, that is preferable. | ||
# Monitor bugs that are flagged as ''status-b2g-v1.2: affected'' as described in [ | # Monitor bugs that are flagged as ''status-b2g-v1.2: affected'' as described in [[QA/Execution/Web_Testing/Automation/Gaia_UI_Testing#Finding_Bugs_That_Require_Uplifts|Finding Bugs That Require Uplifts]]. | ||
# Complete a manual audit of commits that have landed in master but not v1.2. This is something that should be done once or twice per month. | # Complete a manual audit of commits that have landed in master but not v1.2. This is something that should be done once or twice per month. | ||
## Generate a list of all the commits that have landed in master but not v1.2 using this command: <code>git show master ^v1.2 --no-merges -- tests/python/gaia-ui-tests</code>. | ## Generate a list of all the commits that have landed in master but not v1.2 using this command: <code>git show master ^v1.2 --no-merges -- tests/python/gaia-ui-tests</code>. | ||