B2G/UpdateTesting

From MozillaWiki
< B2G
Jump to: navigation, search

This document is a WIP for the upcoming B2G update test automation in bug 821412

Overview

When a system update is applied in B2G, it causes either a process restart (OTA) or a device reboot (FOTA), requiring special high level support from a testing framework to automate testing of the various stages of the update.

To solve this problem, specialized Marionette based frontends have been written to solve two different use cases:

  1. A script based frontend for use by developers, and eventually TBPL
  2. A "continuous build" smoke testing frontend designed to automatically test updates between builds in a Continuous Integration environment.

Smoke Tests

Setup

To get started, just make a directory that will store updates and flash scripts for each build, and inside it create a smoketest-config.json.

$ mkdir $B2G_UPDATE_TESTDATA

Example smoketest-config.json:

{
  "devices": {
    "unagi": {
      "system_fs_type": "ext4",
      "system_location": "/dev/block/mmcblk0p19",
      "data_fs_type": "ext4",
      "data_location": "/dev/block/mmcblk0p22",
      "sdcard": "/mnt/sdcard",
      "sdcard_recovery": "/sdcard",
      "serials": ["full_unagi"]
    }
  },

  "public_key":   "/Users/mculpepper/Code/B2G-dev/build/target/product/security/testkey.x509.pem",
  "private_key":  "/Users/mculpepper/Code/B2G-dev/build/target/product/security/testkey.pk8"
}

CI integration

  • Each CI build should have its update staged for each test device:
$ testing/marionette/update-smoketests/stage-update.py unagi $B2G_UPDATE_TESTDATA
  • Once at least 2 builds are staged, the smoketests can be run by passing a list of build IDs to test between:
$ testing/marionette/update-smoketests/run-smoketests.py --build-dir $B2G_UPDATE_TESTDATA --run-dir $B2G_UPDATE_RUNDATA 20121219101023 20121220172647