New test environments

Overview

From time to time, there rises a need to upgrade the underlying operating system of a platform. This need arises in sync with new major releases of various operating systems that form part of the CI infrastructure.

For instance, as of 2019-08-07, all Firefox builds for Linux is executed on Ubuntu 16.04.5 docker containers. In other words, the version of Linux distribution used for testing is at least 2 major releases behind the likely dominant version on the market, which is Ubuntu 18.04.

Upgrade of the underlying operating system version has been, in the past, considered a large undertaking often taking upwards of 6 months. This causes a chicken-and-egg problem where regular upgrades do not occur due to the perceived amount of work, which in turn causes the amount of issues to multiply when the upgrade is finally tackled.

The aim of this document, and process is to establish a standardized process that can be used by anyone in Mozilla engineering to perform operating system upgrades.

Task Overview

Broadly speaking, the following are the discrete phases involved.

• ensure availability of machines (if hardware)

   responsibility: release engineering
   

• enable platform on try

   placeholder

• run all suites on try

   placeholder

• begin greening process

   placeholder

• create, review and land patches

   placeholder

Enable platform on taskgraph

The first step of any new test environment is to enable the test platform on Tryserver.

At each step, ensure the taskgraph is sound. This can be verified using:

./mach taskgraph full -v

If the taskgraph can be generated, a push to Tryserver would at least pass the Gecko Decision Task.

Enable build

This step may have already been performed by other teams (eg. Releng). If so, skip to the next step.

First, the platform must have builds enabled before tests can be run.

From taskcluster/ci/build, select the appropriate YML file for the platform. For example, if adding a new Windows build type, select taskcluster/ci/build/windows.yml.

Define all of the required attributes, using existing configurations as a template.

Example with windows10-aarch64 builds:

win64-aarch64/opt:
   description: "AArch64 Win64 Opt"
   index:
       product: firefox
       job-name: win64-aarch64-opt
   attributes:
       enable-full-crashsymbols: true
   treeherder:
       platform: windows2012-aarch64/opt
       symbol: B
       tier: 1
   worker-type: b-win2012
   worker:
       max-run-time: 7200
       env:
           TOOLTOOL_MANIFEST: "browser/config/tooltool-manifests/win64/aarch64.manifest"
           PERFHERDER_EXTRA_OPTIONS: aarch64
   run:
       actions: [get-secrets, build]
       options: [append-env-variables-from-configs]
       script: mozharness/scripts/fx_desktop_build.py
       secrets: true
       config:
           - builds/releng_base_firefox.py
           - builds/taskcluster_base_windows.py
           - builds/taskcluster_base_win64.py
       extra-config:
           stage_platform: win64-aarch64
           mozconfig_platform: win64-aarch64
   fetches:
       toolchain:
           - win64-clang-cl
           - win64-rust
           - win64-rust-size
           - win64-cbindgen
           - win64-sccache
           - win64-nasm
           - win64-node

Push to Tryserver using the interactive Try Fuzzy selector:

./mach try fuzzy

Worker configuration

This step may have already been performed by other teams (eg. Releng), or not required at all (eg. OS upgrade). If so, skip to the next step.

Once the build task has been successfully enabled, test workers must be defined.

There are several files that need to have the new platform added in order to satisfy the taskgraph algorithm:

• taskcluster/taskgraph/transforms/tests.py
• taskcluster/taskgraph/util/workertypes.py

Check and add the new platform details in the following categories:

• worker types
• tiers
• treeherder name translations