B2G/QA/BuildProcess

Nightly Build Status

• please see https://wiki.mozilla.org/B2G/QA/Flame_OTA

Dogfood Build Status

• please see https://wiki.mozilla.org/B2G/QA/Spark

Processes for Smoke and Dogfood

Smoke Test Process

Smoke Test

• Builds are daily tested by QAnalysts; nightly-latest and dogfood-latest builds are tested:
  • Build location for Flame nightly: https://tools.taskcluster.net/index/artifacts/#gecko.v1.mozilla-central.latest.linux.flame-kk-ota/gecko.v1.mozilla-central.latest.linux.flame-kk-ota.debug
  • Build location for Aries dogfood: https://tools.taskcluster.net/index/artifacts/#gecko.v1.mozilla-central.latest.linux.aries-dogfood/gecko.v1.mozilla-central.latest.linux.aries-dogfood.debug
  • Test Suite in MozTrap: https://moztrap.mozilla.org/manage/cases/?filter-suite=241
  • Initial Logging : https://docs.google.com/spreadsheets/d/13BjtbYDVaiC6TvOAd1x-SuoYNNM8aH5-xdJS7TU3o-U/edit#gid=328354895
• If there is any failure, these will fail the build from being tested further for release for nightly/dogfood.
• Depending on the failure, the build may or not be used QA for feature testing.
• Daily automation results from jenkins running GIP tests are looked at, reviewed, and seeing if they manually reproduce.
• Detailed information revolving the Smoke tests are listed here : https://wiki.mozilla.org/B2G/QA/Smoke_Tests

Smoke Test Results

• Results are posted in 4 different places:
  • Announced in QA Standup meeting daily : #QA-B2G in vidyo 10 am PDT Daily
  • Posted in QAStandup etherpad daily : https://public.etherpad-mozilla.org/p/b2g-qa-daily-standup
  • Posted in https://mail.mozilla.org/listinfo/fxosqa-report-watchlist daily
  • Posted in QA Weekly round table weekly: https://public.etherpad-mozilla.org/p/b2g-qa-roundtable ; #QA-B2G in vidyo 5 pm PDT Wednesdays
• Daily automation results are reviewed; Failures are manually tested and are posted to "gaia-ui-automation@mozilla.org"

Dogfood Test Process

Dogfood Test

• If smoke test passes :
  • For nightly builds, nightly-latest is pushed to nightly on [1]. ( currently on hold until new base build/FOTA occurs)
    • https://wiki.mozilla.org/B2G/QA/Flame_OTA is updated when the change is made
  • Dogfood-latest build is moved to dogfood-test
  • Dogfood Tests are executed for further testing:
    • Dogfood Test Suites : https://moztrap.mozilla.org/manage/cases/?filter-suite=912&filter-suite=719

Dogfood Test Results

• Dogfood test results are posted in https://mail.mozilla.org/listinfo/fxosqa-report-watchlist
• If passed then dogfood channel is updated on balrog with the build
  • a quick sanity test to make sure people download the right build is done
  • https://wiki.mozilla.org/B2G/QA/Spark is updated when the change is made

Process in event of failure

• Bugs that are smoke test failures should be investigated for regression ranges
  • If a regression window is found :
    • Dev is notified and discussion occurs
    • If a dev cannot be notified in time, process will occur
• Dogfood failures may end up having the similar process. Smoke test failures is higher priority.
• If a patch is made for an issue the following day and a new failure occurs, the tested build is patched with the new patch and a new build is kicked off and tested with the dogfood test suite.

Challenges

• backouts/patching needs to occur faster
  • Gecko often may have a regression that cannot necessarily be backed out easily
    • This is the main reason why there are delays in a dogfood build beyond any new bug failures.
• more often than not, there can be multiple dogfood and smoke test failures that occur.
• Close monitoring of smoke test failures and pushing of backouts has not occurred since Bhavana (release manager) left; QA voice is not enough to convince a developer to back out faster. We need a manager to fill this role.

T2M Base Builds

• The occurrence of needing a new T2M Base build is when there has been a new gonk level change, that a new build is considered.
• T2M base builds follows a similar path as dogfood.
• When making a build, the img files are written over top of the original 18D base build
  • https://github.com/nhirata/Nhirata_Scripts/blob/master/T2M_Flame.sh outlines what is done in the build.
  • Note: It's probably better to fix/create a taskcluster script to kick off this build [to be worked on]
• The build is sent to the TAM for T2M and they are asked to hosts the files
• Balrog will need to point to the FOTA build accordingly.