View source for Sheriffing/How To/Mobile Trees

'''The content of this page is obsolete. See https://wiki.mozilla.org/Sheriffing/How_To/Intermittent_bugs instead.'''


The following trees set up only for mobile projects are being monitored:
* [https://treeherder.mozilla.org/#/jobs?repo=android-components&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable android-components]
** Description: A collection of Android libraries to build browsers or browser-like applications.
** Repository: [https://github.com/mozilla-mobile/android-components android-components] on GitHub
** Escalation channel: [https://chat.mozilla.org/#/room/#android-components:mozilla.org android-components] (Matrix)
* [https://treeherder.mozilla.org/#/jobs?repo=fenix&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable fenix]
** Description: Android browser "Firefox Preview" - codename "Fenix"
** Repository: [https://github.com/mozilla-mobile/fenix fenix] on GitHub
** Escalation channel: #fenix-team on Slack
* [https://treeherder.mozilla.org/#/jobs?repo=reference-browser&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable reference-browser]
** Description: browser based on android-components intended to test those. It's not a product for end users.
** Repository: [https://github.com/mozilla-mobile/reference-browser reference-browser] on GitHub
** Escalation channel: [https://chat.mozilla.org/#/room/!nzmqfSxszBNCPTXRYz:mozilla.org reference-browser] (Matrix)

Other communication channels:
* [https://chat.mozilla.org/#/room/#mobile-test-eng:mozilla.org mobile-test-eng] (Matrix): questions about issues with test tasks and tests
* #releaseduty-mobile (Slack): For problems with release tasks like cron, decision or signing tasks
* [https://chat.mozilla.org/#/room/#perftest:mozilla.org perftest]: For issues with performance tasks like Raptor

===Differences to autoland and mozilla-central===
* If a task shall run once more, this must be done from "Custom Actions" > "rerun".
* Code sheriffs don't perform backouts or close trees but escalate to developers/teams to get issues fixed.
* Failures are tracked in multiple tools
** Raptor failures in Bugzilla's [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Raptor Testing :: Raptor] component
** Issues with the release pipeline like signing issues into Bugzilla's [https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering Release Engineering]
** Non-Raptor test failures shall be reported into the Github repository mentioned above.
** Test results are often stored externally and not in the log.
*** In these cases, Treeherder cannot provide failure suggestions.

===Firebase Test Lab===
[[Sheriffing/How To/Mobile Trees/Firebase Test Lab|Sheriffing Outages (Guide)]]

===Accessing the test results===
Many test tasks stored the test results in an external database and the log will only provide the url to the results in that database.
<ol>
<li>Select the failed task.</li>
<li>Open the unformatted log.</li>
<li>Go the end of the log and look for a failure line like this: 
<code>[task 2020-06-17T10:41:25.807Z] Error: Matrix failed: matrix-147257ou2tpig FINISHED failure  https://console.firebase.google.com/project/moz-reference-browser-230023/testlab/histories/bh.b4e77beaed81bc1c/matrices/6129539140917351455</code></li>
<li>Copy the url.</li>
<li>Open the url in a new tab.</li>
<li>The main pane lists the devices on which the tests are run. Repeat the following steps for each device with a failure.</li>
<li>Click on the device with the failure. A list of 'shards' will reopen which are similar to 'chunks' on autoland and mozilla-central: The tests are distributed across the shards.</li>
<li>Open each failed shard in a new tab (click link with mouse wheel).</li>
<li>Depending on the task, the view opened will provide details on the failure. The tab '''Test cases''' shows which test failed.</li>
<li>If don't know if there is already a Github issue tracking the issue, go to the related Github project mentioned above.</li>
<li>Click onto the tab 'Issues'.</li>
<li>Insert the test name from the clipboard and search for it.</li>
<li>If there is no open Github issue, create one.
  <ul>
    <li>Set the issue title as 'Intermittent testName errorSummary'.</li>
    <li>Provide a link to the log.</li>
    <li>Copy and paste the stacktrace (if present).</li>
    <li>For further questions about test failures, the [https://chat.mozilla.org/#/room/#mobile-test-eng:mozilla.org mobile test engineering channel] is the primary contact.</li>
  </ul>
</li>
<li>Copy the url of the issue.</li>
<li>Back in Treeherder, select the task (e.g. Ctrl + click and paste the url into the comment field.</li>
<li>Save the classification.</li>
</ol>