QA/Ondemand Update Tests

From MozillaWiki
< QA
Jump to: navigation, search

Summary

This page describes in detail the steps you need to take to run Ondemand Update tests for various Firefox builds. The setup below assumes you have access to Jenkins, either from within the internal network or via VPN.

Complementary to update tests on the "release" channel, you may need to also run Throttling Tests. Visit the link for instructions on how to run these.

Setting up the config file

  1. Configuration data should be readily available prior to builds coming out
  2. Steps to find/update configuration data, and set up the config file:
    1. Search Mozilla wiki for the Test Plan page for the version you are testing - e.g. if you need to test update for 52.0.2, then go to the Firefox 52 Test Plan page - https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan
    2. Look for the section dedicated to the exact version that you need to test
      1. E.g. if you need to test update for 52.0.2, it should be under Milestones -> Release Candidates -> 52.0.2 (see https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#52.0.2)
      2. E.g. if you need to test update for Beta 6, it should be under Milestone -> Betas -> Beta 6 (see https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#Beta_6)
    3. If there is a section dedicated to the exact version that you need to test, do the following:
      1. Under "Testing"->"Automated" there should be a link that says “configs” - click on it
      2. The Config page that opens (e.g. https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx52b6), will display information on update restrictions and watersheds, and below you should see the contents needed in the config files to run the tests - usually there will be two sections: one for the Win 64 builds config file (the short one on the bottom), and a larger one for all the other platforms
      3. Copy each section of pre-formatted text (starting with [testrun]) in a “config.ini” file on your computer - you can name the files any way you want but it's recommended you give it a name that will help you keep track of it (e.g. "52b6_build1_beta-cdntest_win64.ini" for a config file used to test 52 Beta 6 build 1, with a 64 bit Windows build, on the beta-cdntest channel)
      4. Make sure that the “channel” value is the one that you want to test - notice how the order to test the channels is specified in each corresponding test plan:
        1. Beta - https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#Beta_6 -> Testing -> Automated = order is: “beta-cdntest”, then “beta”
        2. Release Candidate - https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#52.0_RC2 -> Testing -> Automated = order is: “beta-cdntest”, then "beta", then “release-cdntest”, then “release”
        3. Dot Release - https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#52.0.2 -> Testing -> Automated = order is: “release-localtest”, then “release-cdntest”, then “release”
        4. ESR - https://wiki.mozilla.org/Releases/Firefox_52_ESR/Test_Plan#Firefox_52.1.0_ESR -> Checklist = order is: “esr-localtest”, then “esr-cdntest”, then “esr”
      5. Save the files locally - they are now ready to be used to run the tests
    4. If there is NO section dedicated to the exact version that you need to test, do the following:
      1. Look for the section dedicated to the previous build: e.g. if you are testing 52 Beta 6, look for the 52 Beta 5 section in the Test Plan (https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#Beta_5)
      2. Under the "Testing"->"Automated" section there should be a link that says “configs” - click on it
      3. The Config page that opens (e.g. https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx52b5), will display information on update restrictions and watersheds, and below you should see the contents needed in the config files to run the tests - usually there will be two sections: one for the Win 64 builds config file (the short one on the bottom), and a larger one for all the other platforms
      4. Copy each section of pre-formatted text (starting with [testrun]) in a “config.ini” file on your computer - you can name the files any way you want but it's recommended you give it a name that will help you keep track of it (e.g. "52b6_build1_beta-cdntest_win64.ini" for a config file used to test 52 Beta 6 build 1, with a 64 bit Windows build, on the beta-cdntest channel)
      5. Open each config file for editing
      6. Modify the “target-version” value to the version corresponding to the build you are testing, making sure the correct format is used (see examples below):
        1. Beta - 52.0b6#1 - means 52 Beta 6, build 1 (found here: https://archive.mozilla.org/pub/firefox/candidates/52.0b6-candidates/)
        2. Release/Dot release - 52.0.1#2 - means 52.0.1, build 2 (found here: https://archive.mozilla.org/pub/firefox/candidates/52.0.1-candidates/)
        3. ESR - 52.1.0esr#3 - means ESR 52.1.0, build 3 (found here: https://archive.mozilla.org/pub/firefox/candidates/52.1.0esr-candidates/)
      7. Make sure to always use the latest available build, as that's the one Firefox should be updating to
      8. Modify the “channel” value to the channel that you want to test (notice how the order to test the channels is specified in each corresponding test plan:
        1. Beta - https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#Beta_6 -> Testing -> Automated = order is: “beta-cdntest”, then “beta”
        2. Release Candidate - https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#52.0_RC2 -> Testing -> Automated = order is: “beta-cdntest”, then "beta", then “release-cdntest”, then “release”
        3. Dot Release - https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan#52.0.2 -> Testing -> Automated = order is: “release-localtest”, then “release-cdntest”, then “release”
        4. ESR - https://wiki.mozilla.org/Releases/Firefox_52_ESR/Test_Plan#Firefox_52.1.0_ESR -> Checklist = order is: “esr-localtest”, then “esr-cdntest”, then “esr”
      9. Modify the config files:
        1. At a minimum make sure that the previous version is updated (e.g. you'll notice that on each platform we test the previous version with en-US and/or another locale - in https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx52b5 you'll see that the previous version - 52.0b4 - is the first one to be covered for each platform - if you are updating the file to be used for 52 Beta 6, then, at a minimum, you need to change 52.0b4 everywhere to 52.0b5)
        2. For a complete update of the config data, we aim at covering for each version:
          1. all builds of version V, with one locale each - e.g. for https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx52b5, under [windows xp 32bit], there is one locale for each of 52.0b4, 52.0b3, 52.0b2, 52.0b1 (with en-US for 52.0.b4) - if we cannot cover all builds of version V under a platform, then we aim at splitting them as evenly as possible among various similar platforms (Windows, Mac, Linux)
          2. a minimum of one locale for V-1, V-2, V-3 versions - e.g. for https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx52b5, we cover a random version of each of 51b, 50b, and 49b for Windows, but because we afford it, we cover multiple versions for 51b on Linux
        3. NOTE: The configuration data is designed to cover ALL existing locales exactly once on any platform (except for en-US), distribute the tests as evenly as possible between Windows, Mac and Linux, cover versions only up to V-3 (except for ESR), and focus more on the more recent previous versions, and less on the older previous versions
      10. Once you've modified the data in your config files, save the files locally (they're now ready to use to run the tests), then also create a new Wiki page for the new config data, using the wiki page from the previous config (e.g. for 52 Beta 6, go to https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx52b6 to create it, and add the info from https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx52b5, with the modified config file content)

Notes

  1. Example of test plans for Firefox and ESR (just replace the version number in the URL to get to the wiki you need):
    1. https://wiki.mozilla.org/Releases/Firefox_52/Test_Plan
    2. https://wiki.mozilla.org/Releases/Firefox_52_ESR/Test_Plan

Running the tests with the config file

  1. Connect to MozillaVPN
  2. Go to http://mm-ci-production.qa.scl3.mozilla.com:8080/
  3. Login with valid credentials
  4. Click on the trigger-ondemand link
  5. Click on Build with Parameters on the left side of the page
  6. Click on the “Choose File” button and select your config file for the run
  7. Click the “Build” button -> The job will start to load in the panel on the left
  8. As you will usually use two config files, repeat the steps above to start the tests with the second file as well (you can do it immediately, the system will know to start the jobs from the second file, once the ones in the first file are all started)
  9. Monitor progress in: http://mm-ci-production.qa.scl3.mozilla.com:8080/job/ondemand_update/
  10. Monitor test results in Treeherder:
    1. If the Test Plan section already exists for your version - Click on the "report" link corresponding to the channel that you are testing
    2. If the Test Plan section doesn't/didn't already exist for your version - Follow the next steps:
      1. Open the report for the previous version (see examples below):
        1. Beta/Release/Dot Release - https://wiki.mozilla.org/Releases/Firefox_46/Test_Plan#Beta_4 -> Testing -> Automated -> “report” link => https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=2f6f69a19150&filter-tier=3&filter-searchStr=Fxup-beta-cdntest(
        2. ESR - https://wiki.mozilla.org/Releases/Firefox_45_ESR/Test_Plan#Firefox_45.0.1_ESR -> Checklist -> Update automation on esr-localtest -> “report” link => https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr45&revision=32eb8c423d67&filter-tier=3&filter-searchStr=Fxup-esr-localtest(
      2. In the URL of the report for the previous version, just replace the revision string with the one for the current version (that you are testing now) - to find this value do the following:
        1. Go to the location of your build, and select any platform and any language - e.g. for 44.0.2 you could go to: http://archive.mozilla.org/pub/firefox/candidates/44.0.2-candidates/build3/win32/en-US/
        2. Click on the txt file at this location to view it - e.g. in this case: http://archive.mozilla.org/pub/firefox/candidates/44.0.2-candidates/build3/win32/en-US/firefox-44.0.2.txt
        3. Navigate to the URL in the txt file - e.g. in this case: https://hg.mozilla.org/releases/mozilla-release/rev/60e96806ff1c9fb51f3df31ccd1db33f5f320e75
        4. Note the text on the top page - e.g. in this case: “Mercurial > releases > mozilla-release / changeset / 60e96806ff1c”
        5. The last string is the value to use in the URl of the report for “revision” - e.g. in this case the value will be “60e96806ff1c”
      3. Also make sure that the “filter-searchStr” value in the URL corresponds to the channel you are testing
      4. You should now be able to see the results in real time, similar to the pic below:

Tree1.png

Analyzing the results

Green = PASS

  1. No action

Orange = TEST FAILURE

  1. If there’s a star displayed next to the text then it’s already starred
  2. If there is no star then you need to classify the failure:
    1. Click on the string -> The “Pinboard” will open
    2. See more details in section 4. Classifying failures below
    3. Rebuild:
      1. In the pinboard (picture below) click on the “Job details” tab
      2. Click on the link next to “Inspect Jenkins Build (VPN required):”
      3. This will bring you to the Jenkins page for that test - e.g. http://mm-ci-production.qa.scl3.mozilla.com:8080/job/ondemand_update/23245/
      4. On this Jenkins page click the “Rebuild” link on the left of the page
      5. Now click the “Rebuild” button on the bottom of the page
      6. Several seconds after the test was rebuilt you should see a new entry on the Treeherder report page with the same name as the one that failed (e.g. “lij-1”) but in gray (meaning it’s running) - if the rebuild passes you will then see it in green and are all done

Red = BUSTAGE

  1. The exact same actions apply as for Orange

Pink = CANCELLED

  1. Possible causes:
    1. Manual cancellation - Example
      1. HOW - you can manually cancel jobs in Jenkins by clicking the "x" icon next to a job in http://mm-ci-production.qa.scl3.mozilla.com:8080/job/ondemand_update/, or clicking on the link for a job and then clicking the "x" icon on top right of the page
      2. ACTIONS - just classify all pink jobs as "expected fail" with a comment as to why you've cancelled them, and then either rebuild all pink jobs (if just a few) or re-trigger the whole ondemand job using the config.ini file
    2. Network issues - Example
      1. HOW - network issues may cause the test machines to not be able to setup the environment, so tests won't even be start
      2. ACTIONS - just classify all pink jobs as "infra" with a comment stating that they were caused by an internal network issue, and then either rebuild all pink jobs (if just a few) or re-trigger the whole ondemand job using the config.ini file

Classifying failures

  1. Issue is in the "Failure Summary"/"Failure Classification" list
    1. Look for all errors for that test - in the picture below there is only one error: “TEST-UNEXPECTED-FAIL | test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: The staged update to buildid 20170504103226 has not been applied” - which shows on the first line under "Failure Classification" or "Failure Summary" if you select it
    2. Look underneath the error for the bug number that best corresponds to your error - in the picture below that would be: “1355818 - [tier-3] Intermittent test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: The staged update to buildid 20170412030252 has not been applied” - which is the very first line under the error (most of the times, the first line will also be the bug that you're looking for)
    3. Click the pin icon in front of the bug number (Note: you need to be logged in)
    4. Click save - you should get a notification on the top left of the page that the error has been classified, and the job will now display a star on the top right Ondemand fail 1.png
    5. Note: this also works if you have multiple jobs selected with Ctrl+click (if all have failed with the same error)
  2. Issue is NOT in the Failure Summary
    1. Search this wiki or bugzilla for the error (search Product=Testing, Component=Firefox UI Tests - example of query)
      1. If found:
        1. Ctrl+Click on the failed job OR click on “Open pinboard ^” (additionally you can Ctrl+Click on all jobs that failed with the same error)
        2. Under “classification” select “intermittent”
        3. Add the bug number in the "+ click to add a related bug" field on the left of the "classification" field, and press ENTER - if this doesn't work, add it in the comment field
        4. Click “save” (if “save” button is disabled, CTRL+Click the same test again - you should see it added on the left, and “save” is enabled) - classify the issue as "intermittent" with the bug number
      2. If not found file a new bug:
        1. Via Treeherder (the quick way to do it)
          1. In the URL bar add "&bugfiler" parameter to the URL and press "Enter"
          2. Click on the job that failed
          3. Select the Failure Summary tab (the second one, in between Job Details and Failure Classification)
          4. Click the small bug icon in front of the failure that you want to file (the line that says: "TEST-UNEXPECTED-ERROR | ...") -> This should open the Intermittent Bug Filer pop-up window
          5. In the Product entry field add Testing :: Firefox UI Tests and click "Find Product" -> Make sure the component is found, and the radio box in front of it appears as selected
          6. Double-check the summary to make sure it is the desired one
          7. Leave all three check-boxes selected:
            1. Include Parsed Log Link
            2. Include Full Log Link
            3. This is an intermittent failure (unless you are hitting a permanent failure, in which case you need to uncheck it)
          8. Add any additional comments, as you consider necessary (e.g. where it was found, how many times it was seen, etc.)
          9. Click "Submit Bug"
          10. The failure should now appear as classified, and within several seconds you should also see the link to the newly filed bug
        2. Via Bugzilla
          1. Choose to create a new bug for Testing::Firefox UI
          2. Set "Version"=<version_that_you've_run_the_tests_for
          3. Set appropriate "Platform" values (or leave Unspecified)
          4. In the "Summary" field enter the error encountered (the error that shows in Treeherder under "Failure summary" - e.g. "TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update, test_page_info_window.py TestPageInfoWindow.test_close_window | MarionetteException: win.document.documentElement is null")
          5. In the "Description" field add the following information:
            1. Where was the error seen (with link to the Treeherder job) - for example: "Seen once while running ondemand update tests for 47.0b1 (https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=5bbf2e7c2fc6&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta(&selectedJob=1025489)."
            2. Traceback log - to get this do the following:
              1. For the Treeherder job that failed, click on the "Job details" tab
              2. Click the "report.html" link, to open the report in a new tab
              3. Copy the contents displayed under the "Results" section (e.g. "Traceback (most recent call last): ..."
              4. This will be your Traceback log to paste in the bug description - see the example here
  3. No logs available in the "Failure Summary"
    1. This usually indicates a problem with the Mozilla infrastructure
    2. Actions to take:
      1. If Jenkins indicates only a few failure
        1. Rebuild and see if it passes (if it doesn't then it may be a real problem)
        2. Once rebuilt and passed classify the issue as "infra" (comment is not mandatory)
      2. If Jenkins indicates a lot of failures
        1. Reverify the config file
        2. If config file is OK, contact :whimboo on #automation - he should know/investigate such issues
        3. Once notified that the problem was remedied:
          1. Classify all failures in treeherder as "infra" (comment is not mandatory)
          2. Start the entire run once again (could rebuild individual jobs that failed, but that takes a much longer time)
          3. Examples: 47 Beta 4

Common issues

  1. Logs not fully parsed, please wait
    1. This is an infrastructure issue - no need to wait as the logs won't be parsed
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Open pinboard (or Ctrl+Click on the job), in the classification drop-down select infra, then click “save”
  2. TEST-UNEXPECTED-FAIL | test_direct_update.py TestDirectUpdate.test_update | AssertionError: Available update has been found
    1. This usually indicates that Firefox has NOT found any updates available
    2. Actions:
      1. Check the results for patterns showing that only some versions get this, and double check for existing watersheds (there may be an existing watershed or rule in place that does not allow update of specific versions - #releng can check for this)
        1. If a watershed is in place then all tests for the same version should fail in the exact same way - if you only have a few failures then this is likely bug 1316567 - Actions to take in case of bug:
          1. Re-build - If tests pass on rebuild, then this confirms you are dealing with bug 1316567
          2. If/after the test passes on re-build, Classify the error:
            1. Click the pin icon in front of bug 1316567 + Click “save” ***OR***
            2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
        2. If ALL tests fail, and most failures are with this error message and the one below, then it means that the server is not properly configured - Actions to take:
          1. Notify people on #release-drivers or #releng about this, and ask them to check the server settings
          2. Classify the error as "expected fail" with a comment explaining that this is due to the server not being set properly:
            1. Ctrl+Click all failures, in the classification drop-down select "expected fail", in the field below add a comment explaining that this is due to the server not being set properly, press ENTER, and then Click “save”
          3. After the rules on the update the server are fixed, re-run all jobs (trigger aggain with the same config files)
  3. TEST-UNEXPECTED-FAIL | test_direct_update.py TestDirectUpdate.test_update | AssertionError: u'20160427030215' != u'20160425205003'
    1. This usually indicates that Firefox was updated to a different version than the expected one
      1. The first string is the BuildID of the ACTUAL version that Firefox got updated to
      2. The second string is the BuildID of the EXPECTED version that Firefox was supposed to get updated to
    2. Actions:
      1. Check the results for patterns showing that only some versions get this, and double check for existing watersheds (there may be an existing watershed or rule in place that does not allow update directly to your expected version, but instead force update to an intermediate version - #releng can check for this)
        1. If a watershed is in place then all tests for the same version should fail in the exact same way - if you only have 1-2 failures then this is likely bug 1285340 - Actions to take in case of bug:
          1. Re-build
          2. If/after the test passes on re-build, Classify the error:
            1. Click the pin icon in front of bug 1285340 + Click “save” ***OR***
            2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
        2. If ALL tests fail, and most failures are with this error message and the one above, then it means that the server is not properly configured - Actions to take:
          1. Notify people on #release-drivers or #releng about this, and ask them to check the server settings
          2. Classify the error as "expected fail" with a comment explaining that this is due to the server not being set properly:
            1. Ctrl+Click all failures, in the classification drop-down select "expected fail", in the field below add a comment explaining that this is due to the server not being set properly, press ENTER, and then Click “save”
          3. After the rules on the update the server are fixed, re-run all jobs (trigger aggain with the same config files)
  4. TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: 'beta-cdntest' != u'nightly'
    1. This means that the machine running the test is stuck on Nightly or Aurora - ISSUE USED TO BE WIDESPREAD BUT SHOULD NOW BE FIXED
    2. To confirm click the “Job details” tab then the “report.html” link - if you see screenshots showing an open Nightly/Aurora window, then it’s confirmed that the machine is stuck on Nightly or Aurora - If you don't see a Nightly or Aurora build, but instead get the expected build, then just ignore the error and classify the other errors (if multiple errors), or classify as "infra" if it's the only error
    3. Actions:
      1. Make sure your VPN is connected
      2. Open a VNC Viewer app on Windows (e.g. VNC Viewer, Tight VNC Viewer)
      3. In the “VNC Server” or “Remote Host” field enter the name of the machine where the failure occurred (you can find it on the left of the expanded pinboard - e.g. "Machine: mm-win-7-64-2.qa.scl3.mozilla.com")
      4. Enter the password to connect to the machine (if you do not know the password you can ask in #automation)
      5. You should now be connected to that machine and see an open Nightly/Aurora window
      6. Note down: name of the machine, version of Firefox that was open (e.g. from "about:support" -> "49.0a1, BuildID=20160512205003"), and page that was open in the active window (e.g. "about:home")
        1. Do this for all other machines in this case - you should now have a list of affected machines, firefox versions, and open pages
      7. Close the Firefox window and exit from the VNC Viewer app - NOTE: Make sure you close ONLY the FIREFOX windows
        1. In case of multiple machines in this situation do the following
          1. Close ONLY the Firefox window on all of them
          2. Take at least one machine offline (see "Taking node offline" at the end of this wiki page
          3. Send email to mozmill-ci@mozilla.org, stating that there are multiple machines stuck on Nightly/Aurora, and list the details picked in the previous steps (machine name, firefox version that was stuck, with BuildID, page that was open)
          4. Proceed then to classify issues and rebuild
      8. Re-build
      9. Classify the issue:
        1. Click on the failed test
        2. Click on “Open pinboard ^”
        3. If “save” button is disabled, CTRL+Click the same test again (you should see it added on the left, and “save” is enabled)
        4. Under “classification” select “infra”
        5. Add a comment like: "Machines stuck on Nightly/Aurora"
        6. Click “save”
        7. Note: you can classify multiple issues of the same kind by using CTRL+Click for multiple selection
    4. NOTE: These errors can also show in case of expected failures, caused by:
      1. Bad config.ini - For example here a config.ini file was used where the target build was 46.0b11, but for some locales the source for the update was also 46.0b11 - obviously a build will not update to itself
      2. Watershed rules - are rules set so that some older versions will NOT update beyond some version, for example:
        1. 44.0b1 is an update "watershed" for Windows, no updates from before go to builds beyond it. This means that if you set up a test to update 43.0b7 for example to the latest Beta version (e.g. 47.0b4), then the update test will fail, as 43.0b7 would update to 44.0b1
        2. 44.0rc3 is an update "watershed" for Linux, 44.0b4 and earlier only get updates to that version.
      3. Blocked updates - temporary/partially disabled updates - For example here a lot of tests failed because when releasing 46.0.1 to the release channel, Release Engineering decided that updates from 45.0.2 and lower should be temporarily disabled (bug 1269848), which naturally caused all update tests to fail, except for the ones that tested update from 46.0
        1. Basically in this case, the failures confirmed the fix for bug 1269848
        2. These failures were NOT rebuilt since they were never going to pass (until the change from bug 1269848 would have been reverted)
        3. All these failures were marked as "expected fail"
  5. TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | AttributeError: 'dict' object has no attribute 'type'
    1. This is bug 1247596
    2. This is NOT necessarily a problem with Firefox
    3. This usually indicates there are some watersheds in play - to find out more information:
      1. Click on a job and then on "Job details"
      2. Click on the "log_info.log" link
      3. Go through the log trying to figure out:
        1. What might have caused a failure
        2. What's the actual build that Firefox got updated to (look for "build_post" in the logs)
    4. Re-building jobs is unlikely to help in this case, but you can try to do it at least once
  6. TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | NoSuchWindowException: NoSuchWindowException: No window found for '<function <lambda> at 0x03226830>'
    1. This is bug 1274943 - should show up in the "Failure summary"
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1274943 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  7. TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | error: [Errno 10054] An existing connection was forcibly closed by the remote host
    1. This is bug 1314627 - should show up in the "Failure summary"
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1314627 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  8. TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | IOError: process died with returncode 0
    1. This is bug 1289343 - should show up in the "Failure summary"
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1289343 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  9. TEST-UNEXPECTED-FAIL | test_direct_update.py TestDirectUpdate.test_update | AssertionError: <firefox_puppeteer.ui.about_window.deck.DownloadFailedPanel object at 0x02830CD0> == <firefox_puppeteer.ui.about_window.deck.DownloadFailedPanel object at 0x02830650>
    1. This is bug 1249466, which is in fact a duplicate of bug 1249467 - should show up in the "Failure summary"
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1249467 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  10. TEST-UNEXPECTED-FAIL | test_direct_update.py TestDirectUpdate.test_update | AssertionError: <firefox_puppeteer.ui.about_window.deck.DownloadingPanel object at 0x032BD9F0> != <firefox_puppeteer.ui.about_window.deck.CheckForUpdatesPanel object at 0x032BD770>
    1. This is bug 1233679
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1233679 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  11. TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | IOError: Timed out waiting for port! *** OR *** TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | IOError: Timed out waiting for port!
    1. This is bug 1202375 - should show up in the "Failure summary"
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1202375 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  12. TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | MarionetteException: MarionetteException: win.document.documentElement is null *** OR *** TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | MarionetteException: MarionetteException: win.document.documentElement is null
    1. This is bug 1268087
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1268087 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  13. TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | TimeoutException: TimeoutException: Timed out after 5.1 seconds *** OR *** TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | TimeoutException: TimeoutException: Timed out after 5.1 seconds with message: Cannot get window type for chrome window handle "1"
    1. This is bug 1256425 - should show up in the "Failure summary"
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1256425 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  14. TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | JavascriptException: JavascriptException: TypeError: ums.activeUpdate is null
    1. This is bug 1260383
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1260383 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  15. TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | UnknownException: UnknownException: Error loading page *** OR *** TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | UnknownException: UnknownException: Error loading page
    1. This is bug 1262122
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1262122 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  16. TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | TimeoutException: TimeoutException: Timed out after 360.2 seconds with message: Download has been completed.
    1. This is bug 1238002 - should show up in the "Failure summary"
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1238002 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  17. TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | NoSuchWindowException: NoSuchWindowException: Unable to locate window: [1|3]
    1. This is bug 1207042
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1207042 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  18. TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | TimeoutException: TimeoutException: Timed out after 30.2 seconds with message: Check for updates has been finished.
    1. This is bug 1288731
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1288731 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  19. TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | IOError: Process killed because the connection to Marionette server is lost. Check gecko.log for errors (Reason: Connection timed out after 60s)
    1. This is bug 1311987
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1311987 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  20. TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | IOError: Process killed because the connection to Marionette server is lost. Check gecko.log for errors (Reason: Timed out waiting for port 2828!)
    1. This is bug 1303834
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1303834 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  21. TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | UnknownWindowError: UnknownWindowError: Window with handle "1" does not exist
    1. This is bug 1323174
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1323174 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  22. TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | IOError: Process has been closed (Exit code: 0) (Reason: [Errno 61] Connection refused)
    1. This is bug 1324059
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1324059 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  23. TEST-UNEXPECTED-FAIL | test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Additional update found due to watershed release 50.1.0
    1. This is bug 1316564
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1316564 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  24. TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | MarionetteException: Please start a session
    1. This is bug 1322109
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1322109 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  25. TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | NoSuchWindowException: No window found for '1'
    1. This is bug 1377092
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1377092 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  26. IOError: Process killed because the connection to Marionette server is lost. Check gecko.log for errors (Reason: Timed out waiting for connection on localhost:2828!)
    1. This is bug 1362293
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1362293 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  27. TEST-UNEXPECTED-FAIL | test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: The staged update to buildid 20170504103226 has not been applied
    1. This is bug 1355818
    2. Actions:
      1. On localtest or cdntest channels ONLY - Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1355818 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
    3. NOTE: This error causes repeated failures on WINDOWS, that pass after multiple re-runs, and seriously affect the official channels (beta, release, esr), to the point where you might even end up re-building the same job up to 100 times until it passes - this error was not reported in the real world - current agreement is that WE DO NOT RE-BUILD JOBS THAT FAILED WITH THIS ERROR ON OFFICIAL CHANNELS (beta, release, esr) - we only re-build on localtest or cdntest channels
  28. TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | IOError: Process killed because the connection to Marionette server is lost. Check gecko.log for errors (Reason: Timed out waiting for connection on localhost:2828!)
    1. This is bug 1303834
    2. Actions:
      1. On localtest or cdntest channels ONLY - Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1303834 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
    3. NOTE: This error causes repeated failures on LINUX, that pass after multiple re-runs, and seriously affect the official channels (beta, release, esr), to the point where you might even end up re-building the same job up to 100 times until it passes - this error was not reported in the real world - current agreement is that WE DO NOT RE-BUILD JOBS THAT FAILED WITH THIS ERROR ON OFFICIAL CHANNELS (beta, release, esr) - we only re-build on localtest or cdntest channels
  29. TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | TimeoutException: Timed out after 5.1 seconds with message: No new chrome window has been opened.
    1. This is bug 1372982
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1372982 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”
  30. TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update | JavascriptException: ReferenceError: log is not defined
    1. This is bug 1383032
    2. Actions:
      1. Re-build
      2. If/after the test passes on re-build, Classify the error:
        1. Click the pin icon in front of bug 1383032 + Click “save” ***OR***
        2. Open pinboard (or Ctrl+Click), manually add the bug number (in the "+ click to add a related bug" field on the left of the "classification" field), press ENTER + Click “save”

Reporting

See below a series of emails sent when signing off various builds, on various channels. Note that the procedure can vary on the same channel at different phases.

BETA - aurora-cdntest/beta-cdntest

BETA 1-2 (DevEdition)

To: 'release-signoff' <release-signoff@mozilla.org>
Subject: [desktop] Please push DevEdition 62.0b1 to the aurora channel

Hi everyone,

Manual QA has finished testing DevEdition 62.0b1-build1 updates on the aurora-cdntest channel. Results are here:
* https://public.etherpad-mozilla.org/p/devedition-62.0b1-build1

Regression and manual testing efforts found no blocking issues.

For more information on testing of DevEdition 62.0b1 see our test plan:
* https://wiki.mozilla.org/Releases/Firefox_62/Test_Plan#Developer_Edition

Regards,

BETA 3-last

To: 'release-signoff' <release-signoff@mozilla.org>
Subject: [desktop] Please push Firefox 62.0b3 to beta and aurora, respectively

Drivers,
 
QE has finished testing Firefox 62.0b3 (build1), and updates on the beta-cdntest and aurora-cdntest channels. Regression and manual testing efforts found one new issue (bug 1468327) that is not severe enough to stop this release, so we consider the build ready for release to Beta.
 
Please make the build live on the beta channel - and the corresponding DevEdition build live on the aurora channel.

For more information on testing of 62.0b3, see our test plan at:
https://wiki.mozilla.org/Releases/Firefox_62/Test_Plan#Beta_3

Regards,

BETA - aurora/beta

BETA 1-2 (DevEdition)

To: 'release-signoff' <release-signoff@mozilla.org>
Subject: [desktop] DevEdition 62.0b1 - Updates on aurora channel signed off by QA

Content: 
Hi everyone,

Update tests on aurora channel look good. Results are here:
* https://public.etherpad-mozilla.org/p/devedition-62.0b1-build1

For more information on testing of DevEdition 62.0b1 see our test plan:
* https://wiki.mozilla.org/Releases/Firefox_62/Test_Plan#Developer_Edition

Regards,

BETA 3-last


To: 'release-signoff' <release-signoff@mozilla.org>
Subject: [desktop] Firefox 62.0b1 - Updates on Beta Channel signed off by QA

Drivers,

Update tests on beta channel look good. Results are here:
* https://public.etherpad-mozilla.org/p/devedition-62.0b1-build1

For more information on testing of DevEdition 62.0b3 see our test plan:
* https://wiki.mozilla.org/Releases/Firefox_62/Test_Plan#Beta_3

Regards,

RC - sign off on beta-cdntest

To: 'release-signoff' <release-signoff@mozilla.org>
Subject: [desktop] Preliminary sign-off for Firefox 61.0RC on the beta-cdntest channel

Drivers,

QE has finished testing Firefox 61.0RC (build1), and updates on the beta-cdntest channel. Regression and manual testing efforts found no blocking issues for pushing to Beta.

From the QE side, we are ready to make the builds live on the beta channel, so handing over to RelMan to give the "go" on that.

For more information on testing of 61.0RC build1, see our test plan:
* https://wiki.mozilla.org/Releases/Firefox_61/Test_Plan#61.0_RC

Regards,

RC - sign off on beta

To: 'release-signoff' <release-signoff@mozilla.org>
Subject: [desktop] Firefox 61.0 RC updates on beta channel signed off by QE
Hi,

Update tests on the beta channel look good. Results can be found here:
* https://public.etherpad-mozilla.org/p/devedition-62.0b1-build1

Signing off for 47.0 RC (build 1) on this channel.

Regards,

RC - sign off on release-cdntest

To: 'release-signoff' <release-signoff@mozilla.org>
Subject: [desktop] Preliminary sign-off for Firefox 61.0RC on the beta-cdntest channel

Drivers,

QE has finished testing Firefox 61.0RC (build1), and updates on the beta-cdntest channel. Regression and manual testing efforts found no blocking issues for pushing to Beta.

From the QE side, we are ready to make the builds live on the beta channel, so handing over to RelMan to give the "go" on that.

For more information on testing of 61.0RC build1, see our test plan:
* https://wiki.mozilla.org/Releases/Firefox_61/Test_Plan#61.0_RC

Regards,

RC - sign off on release

To: 'release-signoff' <release-signoff@mozilla.org>
Subject: [desktop] QE signing off on 61.0 release
Hi,

Update tests look good on the release channel.
Results of the testing can be found here:
* https://public.etherpad-mozilla.org/p/rc-61.0-build3

More details in the Test Plan: 
* https://wiki.mozilla.org/Releases/Firefox_61/Test_Plan#61.0_Release_Candidate_3

Regards,

DOT RELEASE - sign off on release-localtest

To: 'release-signoff' <release-signoff@mozilla.org>
Subject: [desktop] Firefox 60.0.1 signed off on the release-localtest channel by QE

Hi,

Update tests on the release-localtest channel look good, manual testing showed no new issues, and bug/s <bug/s that caused the dot release> has been verified on 60.0.1.

With that, I'm signing off for QE on release-localtest and handing over to release management.

For more information on testing of 60.0.1, see our test plan at:
* https://wiki.mozilla.org/Releases/Firefox_60/Test_Plan/Beta/60.0.1
Regards,

RELEASE & DOT RELEASE - sign off on release-cdntest

To: 'release-signoff' <release-signoff@mozilla.org>
Subject: [desktop] Firefox 60.0.2 signed off on the release-cdntest channel by QE

Hi,

Update tests on the release-cdntest channel look good. 
Results can be seen at: 
* https://public.etherpad-mozilla.org/p/Firefox_60.0.2-build1. 

Test Plan can be found at:
* https://wiki.mozilla.org/Releases/Firefox_60/Test_Plan#60.0.2_Build_1

Pending Release Management decision, we can go forward with releasing this one.

Regards,

RELEASE & DOT RELEASE - sign off on release

To: 'release-signoff' <release-signoff@mozilla.org>
Subject: [desktop] QE signing off on 60.0.2 release

Hi, 

Update tests look good on the release channel.
Results of the testing can be found here:
* https://public.etherpad-mozilla.org/p/rc-60.0.2-build3

More details in the Test Plan: 
* https://wiki.mozilla.org/Releases/Firefox_61/Test_Plan#Release_60.0.2
Regards,

ESR - sign off on esr-localtest

To: 'release-signoff' <release-signoff@mozilla.org>
Subject: [desktop] QE Signs off on Firefox 52.9.0 ESR on the "esr-localtest" channel

Hi, 

We have completed the following manual update testing using "esr-localtest" channel for Firefox 52.9.0 ESR (build1) and found no new issues.
Results of the testing can be found here:
* https://public.etherpad-mozilla.org/p/esr-52.9.0-build1

More details can be found in the Test Plan: 
* https://wiki.mozilla.org/Releases/Firefox_52_ESR/Test_Plan#Firefox_52.9.0_ESR

Regards,

ESR - sign off on esr-cdntest

To: 'release-signoff' <release-signoff@mozilla.org>
Subject: [desktop] QE Signs off on Firefox 52.9.0 ESR build1 on the "esr-cdntest" channel

Hi,

We have completed the following manual update testing using "esr-cdntest" channel for Firefox 52.9.0 ESR (build1) and found no new issues.
Results of the testing can be found here:
* https://public.etherpad-mozilla.org/p/esr-52.9.0-build1

More details can be found in the Test Plan:
* https://wiki.mozilla.org/Releases/Firefox_52_ESR/Test_Plan#Firefox_52.9.0_ESR

Regards,

ESR - sign off on esr

To: 'release-signoff' <release-signoff@mozilla.org>
Subject: [desktop] QE Signs off on Firefox 52.9.0 ESR on the "esr" channel

Hi, 

We have completed the following manual update testing using "esr" channel for Firefox 52.9.0 ESR (build1) and found no new issues.
Results of the testing can be found here:
* https://public.etherpad-mozilla.org/p/esr-52.9.0-build1

More details can be found in the Test Plan: 
* https://wiki.mozilla.org/Releases/Firefox_52_ESR/Test_Plan#Firefox_52.9.0_ESR

Regards,

Troubleshooting

  1. All tests fail
    1. Usually it’s a sign of:
      1. Incorrect settings in the config file:
        1. Action: check the config file again and make sure the correct values are set for “target-version”, and “channel”
      2. Some massive problem with Jenkins/Marionette (e.g. Network issues)
        1. Actions:
          1. Check https://mozilla-releng.net/treestatus/ to see if mozilla-beta, mozilla-release, or mozilla-esrXX are OPEN or CLOSED
          2. Ask on #automation or #releng if still unclear if there is a problem or not
  2. Many test failures
    1. Whenever ~25% or more of the tests fail it’s a sign of one of the following:
      1. Some problem in the config file
        1. Action: check again the config file
      2. There is some watershed rule in place that causes older versions to not update directly to the latest version, but instead go through an intermediary version (e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1234277#c4)
        1. Action: classify failures as "expected fail" with comment stating that it's due to watershed rules from bug X
      3. Some problem on Jenkins/Marionette (haven’t encountered such a situation)
        1. Action: contact :whimboo, :AutomatedTester, :maja_zf (#automation, #releng, #release-drivers), or write directly on #automation, or #releng and ask for help
  3. Few test failures
    1. A low number of test failures can be caused by all sorts of intermittent bugs
      1. Action: pin the failure and rebuild
      2. If the test still fails after rebuilding it’s possible that the localized build does not exist, or there is a real issue so notify people on #release-drivers (especially whimboo if available as he can check what the problem is)

Rebuilding

If a run failed or you want to run it again for a platform, all you have to do is to rebuild it:

  1. In the pinboard click on the “Job details” tab
  2. Click on the link next to “Inspect Jenkins Build (VPN required):”
  3. This will bring you to the Jenkins page for that test - e.g. http://mm-ci-production.qa.scl3.mozilla.com:8080/job/ondemand_update/23245/
  4. On this Jenkins page click the “Rebuild” link on the left of the page
  5. Now click the “Rebuild” button on the bottom of the page
  6. Several seconds after the test was rebuilt you should see a new entry on the Treeherder report page with the same name as the one that failed (e.g. “lij-1”) but in gray (meaning it’s running), right next to the previously failed job - if the rebuild passes you will then see it in green and are all done

Taking node offline

If a run is aborted without clear reason (or tests fail because of some issues), you may need to take some node(s) offline, so people on the infra team can fix the problem and then take them back online. To take a node offline:

  1. In the pinboard (picture below) click on the “Job details” tab
  2. Click on the link next to “Inspect Jenkins Build (VPN required):”
  3. This will bring you to the Jenkins page for that test - e.g. http://mm-ci-production.qa.scl3.mozilla.com:8080/job/ondemand_update/23245/
  4. Click on the machine link on the right of the page (e.g. mm-win-81-64-3)
  5. Click “Make this node temporarily offline”

Cancel runs

In some cases (e.g. run triggered with bad config.ini) you may want to cancel the run, or individual jobs one by one:

  1. Cancel individual jobs - TESTED (should work properly)
    1. Go to Jenkins ondemand update page
    2. In the Job Queue (left side) you should see which jobs are done (full green/red bar), which in progress (in progress green/red bar), and which pending (text)
    3. To cancel a job (In Progress or Pending):
      1. Click on the close (x) button on the right of the job link
      2. Click on the link for a job and then click the "x" icon on top right of the page
  2. Cancel the run - UNTESTED (may NOT work properly)
    1. Go to Jenkins trigger-ondemand page
    2. Look for the run you want to cancel in the Build Queue (left side)
    3. Click on the close (x) button of the build you want to cancel
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Ondemand_Update_Tests&oldid=1202131"