Firefox OS/DeviceQA/Gaia-UI-Tests-On-Jenkins

From MozillaWiki
Jump to: navigation, search

How to setup Jenkins

How to setup gaia-ui-tests

Where to Get Report

  • We have CI set up in Taipei and it pushes reports on Github.IO everyday.

How to Read Build Report

  • Test Result Trend of a Project (e.g.: B2G.v2.2.pvt.flame.gaiatest.smoketest)
    • In project view (Jenkins>>Project), we can see a figure like below and quickly find out if the failed test number increases.
    • Blue: number of passed cases, Red: number of fail cases, Yellow: number of skipped cases
Test Result Trend
  • Test Result Overview of a Build
    • In Jenikns>>Project>>Build>>Test Result, we can quickly know which cases failed and if it is failed last last time.
      • If Fail=1 && diff="", it is usually a known issue.
      • If Fail=1 && diff=+1, we need to figure out the reason: if it resulted from a bug.
      • If Pass=1 && diff=+1, the issue in last build might be resolved.
      • Please make sure the status of each diff value when a test script is disabled/enabled.
Test Result Overview
  • Test Result Details
    • We can find index.html at Jenkins>>Project>>Build>>index.html, just like the red rectangle in below figure.
Where to click index.html
  • It includes test result details, especially build information, logs and screenshot of failed cases.
    • From log, we usually can determine if it is a common non-bug error. Otherwise, we need to rerun test script locally or rerun test manually.
Failed Case Details

  • How to Verify a Failed Case
    • Common non-bug errors
      • Timeout
Unable to locate element
  • Unable to locate element
Unable to locate element
  • ...?

How to write daily report

  • Subject
 E.G. 02/09/2015 Flame KK v188-1 Gaia-ui Automation v2.1 Mozilla RIL Full Flash Build 38/38 smoketests, 90/90 non-smoketests
  • Failed cases & failed reason
    • List all failures and expected failures cases for both Smoketest and Non-smoketest. Please also write down the result of failure survey.
   Test failures - 0 replicable manually:
     [Not reproduced] - Cannot reproduce locally.
     [Not reproduced] - Cannot reproduce locally.
   Expected failures / disabled tests:
     [Disabled] - Bug 1028192 - Problems with ActiveSync accounts
     [Disabled] - Bug 1028192 - Problems with ActiveSync accounts
     [Disabled] - Bug 1061698 - Need ability to work in the shadow root/shadow DOM
     [Disabled] - Bug 1096385 - Investigate failure in, call log is displayed erroneously
   Test failures - 0 replicable manually:
     [Not reproduced] - Cannot reproduce locally.
     [Not reproduced manually] - Bug 1117804 - [v2.2] Investigate test_call_log_groups failure
     [Not reproduced manually] -Bug 1117813 - Make PlivoUtil use a valid 'answer_url' with a valid   Plivo XML
   Expected failures / disabled tests:
     [Disabled] - Test runs only on desktop
     [Disabled] - Test requires device with 2 sims
     [Disabled] - Bug 1000863 - Investigate failure
  • Build info
 Build under test:
   Device firmware (base)	L1TC000118D0
   Device firmware (date)	28 Jan 2015 16:59:20
   Device firmware (incremental)	eng.cltbld.20150128.035910
   Device firmware (release)	4.4.2
   Device identifier	flame
   Gaia date	28 Jan 2015 08:05:23
   Gaia revision	 cd42b034fd28 
   Gecko build	20150128002506
   Gecko revision	d824c65a6a2b 
   Gecko version	37.0a2
  • Summary
     45 tests ran in 3697 seconds.
     43 passed, 132 skipped, 0 failed, 2 errors.
     0 expected failures, 0 unexpected passes.
     135 tests ran in 9441 seconds.
     132 passed, 55 skipped, 1 failed, 2 errors.
     0 expected failures, 0 unexpected passes.
  • Attachments
    • Please rename 2 index.html files to smoke.html and non-smoke.html, then attach them in mail.
      • Where to get index.html?
        • You can download both index.html from our github: Gaiatest-Reports (please find the correct date to get them)
        • Or, click the job number at CI then get the index.html.
        • 01_JenkinsReporting.png
        • 02_JenkinsReporting.png


  • If 2 error records for one test case (It seems race condition with the logging, Bug 1050170 tracks this issue), please do not double counting the same fail case.
Double counting error


  • To hide the passed cases to see the failed/error cases more clearly, do the following:
    1. Press F12 to open the Firefox Developer Tool > Console
    2. Run the following code:
      []'passed'), function(elem){"none"})