Sheriffing/How To/Mobile Trees

The content of this page is obsolete. See https://wiki.mozilla.org/Sheriffing/How_To/Intermittent_bugs instead.

The following trees set up only for mobile projects are being monitored:

• android-components
  • Description: A collection of Android libraries to build browsers or browser-like applications.
  • Repository: android-components on GitHub
  • Escalation channel: android-components (Matrix)
• fenix
  • Description: Android browser "Firefox Preview" - codename "Fenix"
  • Repository: fenix on GitHub
  • Escalation channel: #fenix-team on Slack
• reference-browser
  • Description: browser based on android-components intended to test those. It's not a product for end users.
  • Repository: reference-browser on GitHub
  • Escalation channel: reference-browser (Matrix)

Other communication channels:

• mobile-test-eng (Matrix): questions about issues with test tasks and tests
• #releaseduty-mobile (Slack): For problems with release tasks like cron, decision or signing tasks
• perftest: For issues with performance tasks like Raptor

Differences to autoland and mozilla-central

• If a task shall run once more, this must be done from "Custom Actions" > "rerun".
• Code sheriffs don't perform backouts or close trees but escalate to developers/teams to get issues fixed.
• Failures are tracked in multiple tools
  • Raptor failures in Bugzilla's Testing :: Raptor component
  • Issues with the release pipeline like signing issues into Bugzilla's Release Engineering
  • Non-Raptor test failures shall be reported into the Github repository mentioned above.
  • Test results are often stored externally and not in the log.
    • In these cases, Treeherder cannot provide failure suggestions.

Firebase Test Lab

Sheriffing Outages (Guide)

Accessing the test results

Many test tasks stored the test results in an external database and the log will only provide the url to the results in that database.

1. Select the failed task.
2. Open the unformatted log.
3. Go the end of the log and look for a failure line like this: [task 2020-06-17T10:41:25.807Z] Error: Matrix failed: matrix-147257ou2tpig FINISHED failure https://console.firebase.google.com/project/moz-reference-browser-230023/testlab/histories/bh.b4e77beaed81bc1c/matrices/6129539140917351455
4. Copy the url.
5. Open the url in a new tab.
6. The main pane lists the devices on which the tests are run. Repeat the following steps for each device with a failure.
7. Click on the device with the failure. A list of 'shards' will reopen which are similar to 'chunks' on autoland and mozilla-central: The tests are distributed across the shards.
8. Open each failed shard in a new tab (click link with mouse wheel).
9. Depending on the task, the view opened will provide details on the failure. The tab Test cases shows which test failed.
10. If don't know if there is already a Github issue tracking the issue, go to the related Github project mentioned above.
11. Click onto the tab 'Issues'.
12. Insert the test name from the clipboard and search for it.
13. If there is no open Github issue, create one.
    • Set the issue title as 'Intermittent testName errorSummary'.
    • Provide a link to the log.
    • Copy and paste the stacktrace (if present).
    • For further questions about test failures, the mobile test engineering channel is the primary contact.
14. Copy the url of the issue.
15. Back in Treeherder, select the task (e.g. Ctrl + click and paste the url into the comment field.
16. Save the classification.