QA/Automation/Projects/Mozmill Automation/On Demand Test Framework/CI Documentation

From MozillaWiki
Jump to: navigation, search

NOTE

This is a work in progress and should be more detailed soon

Overview

Master Responsibilities

  • Fetch build environment artifacts
    • Centralized on master to limit dependencies
    • Master itself is configured as a build node specifically to fetch these environments
  • Create core jobs on itself via Jenkins API
    • Grabs from davehunt's Github repo
    • Uses python-jenkins to update the jobs on itself
    • Will leave old jobs behind if a platform is removed from requirements
  • Run the job cascade from the configuration file via the on-demand job
  • Push jobs to slaves

Slave Responsibilities

  • Pull build environment artifacts from master
    • Copy Artifact plugin
  • Run the test jobs (Mozmill scripts)
  • Mozmill scripts push to brasstacks (slaves need network visibility to results repo)
  • Currently one build executor per node because no simultaneous Mozmill. New focus monitor changes gives us a shot at parallelism

Flow

Fetch Mozmill environments

  1. We kick off an environment job on master
  2. Master contacts itself as a build node
  3. Master fetches environment from http://people.mozilla.com/~hskupin/downloads/mozmill-env/
  4. Now available as an artifact

Test job

  1. We kick off on-demand job on master w/ configuration file
  2. Configuration job used to create a new specific job w/ UUID in name
  3. New specific job has build triggers in it for each configuration to trigger parametrized core jobs. Immediately finishes after kicking all these off.
  4. Core jobs added to build queue and processed as nodes come available
  5. Core jobs have a dependencies on environments. Use artifacts as built above.
  6. Core jobs each translate to one run of the Mozmill scripts.

Environment Targets

  • Retrieve them from a known place
  • Poll for newer environments and fetch on demand

Build Targets

Coming soon

Test Responsibilities

Coming soon

Installation

Install Prerequisites

Instructions

  1. Install and run Jenkins (java -jar jenkins.war)
  2. Open http://localhost:8080
  3. Install the following plugins (Manage Jenkins > Manage Plugins)
    • Mercurial Plugin
    • Green Balls
    • Xvnc (for Linux slaves)
    • Parametrized Trigger
    • GitHub
    • Git
    • Show Build Parameters
    • Rebuild

Configuration

Master

SSH login

  1. Generate public/private keys using RSA
  2. Keep the public key

Jenkins

  1. Set # of executors to 1

Windows Slave

Jenkins

  1. Set # of executors to 1

Linux, OS X Slave

SSH login

  1. Create activated_keys with master's public key

Jenkins

  1. Set # of executors to 1

Creating jobs

Mozmill Environments

  • Create a new (multi-configuration) job named download-mozmill-environments with the following configuration:

Core Testrun Jobs

Trigger On-Demand Job

  • Create a new (free-style) project named trigger-ondemand with the following configuration:

Running test suites

On-demand test suite

  • Open the Jenkins front end: http://localhost:8080/
  • Locate the 'trigger-ondemand' job. Use the 'admin' view to make this easier to find.
  • Click 'Schedule a build' to the right of the job (or click the job name followed by 'Build Now' on the left)
  • You will be prompted for a configuration file. This is almost identical to the file used by the existing testrun_release.py script. You can find two suitable examples below.
  • Click 'Build'
  • That's all you need to do! What happens next is:
    • The job uses the Jenkins API to create a new job named 'ondemand-<testrun_type>-<uuid>'
    • This job does nothing itself, but triggers additional jobs with parameters as specified by the configuration file.
    • The triggered jobs are added to the build queue and executed as the appropriate slaves become available.
    • These trigger jobs perform the following:
      1. Clear the workspace.
      2. Copy the latest artifact from the appropriate mozmill-environment job.
      3. Unzip and configure the Mozmill environment for use.
      4. Clone mozmill-automation repository.
      5. Run the download.py script to get the appropriate Firefox binary.
      6. Run the testrun_<testrun_type>.py script.
        • The configuration of all of these testrun jobs is controlled by a rather crude script named create_testrun_jobs.py available here: https://github.com/davehunt/mozmill-jenkins-config. It creates a job for each node type (determined by combining a number of labels). At the moment this is limited to OS name/version but could be more granular in the future.

Daily test suite

For now these need to be set up manually, but it would only require a few small tweaks to the configuration file in order to have these created for us. I have manually configured an example daily test suite for addons, endurance, functional, l10n, and remote on our MPT instance

Proof of Concepts

Selenium

  • CI server: Atlassian Bamboo (xserve.openqa.org:8085)
  • Trigger: The build for Selenium IDE is triggered whenever a change is committed to the SVN repository, or an upstream job is built. Whenever the add-on is built the tests are executed.
  • Reports:
  • Prerequisites: The job is run on a Bamboo agent that has Python, Mercurial, and Mozmill preinstalled. Any updates to these would have to be taken care of manually. It would be possible to install Mozmill as part of the job, however Mercurial is an absolute requirement as it's needed to check out mozmill-tests and mozmill-automation from hg.mozilla.org
  • Customizations: As we are running against a snapshot version of Selenium IDE, it is necessary to overwrite the addons.ini file with the location of the snapshot XPI file. This requires first checking out mozmill-tests from hg.mozilla.org, creating the replacement file, committing the change to the local repository, and then specifying the location of the local repository when running the mozmill-automation script.
  • Notifications:
    • An e-mail report is sent to Dave Hunt.
    • selbot2 in #selenium on irc.freenode.net reports build results via RSS feed.

Mozmill

  • CI server: Jenkins (running in a local VM at this time)

download-latest-nightly

  • Purpose: Download the latest Nightly build for testing.
  • Execute: ./download.py --directory=latest-nightly --platform=linux64 --type=daily --branch=mozilla-central
  • Trigger: This job is triggered whenever the contents at http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/ is updated. This may fail if the contents has updated but the build is not yet fully available, so we are likely to want a more robust build trigger in the future.
  • Reports: Standard Jenkins build report.
  • Prerequisites: Requires Python and Mercurial.
  • Notifications: An e-mail report is sent to Dave Hunt if this job fails.
  • Notes:
    • The downloaded build is accessible to other jobs.
    • On completion, the Mozmill jobs listed below are executed.

mozmill-addons-latest-nightly

  • Purpose: Run the addons tests against the latest Nightly build.
  • Execute: ./testrun_addons.py --junitxml=report.xml --logfile=addons.log --report=http://mozmill-archive.brasstacks.mozilla.com/db/ --with-untrusted /var/lib/jenkins/jobs/download-latest-nightly/workspace/latest-nightly
  • Reports:
  • Notifications: An e-mail report is sent to Dave Hunt if this job fails.

mozmill-endurance-latest-nightly

  • As mozmill-addons-latest-nightly but for endurance tests.

mozmill-functional-latest-nightly

  • As mozmill-addons-latest-nightly but for functional tests.

mozmill-l10n-latest-nightly

  • As mozmill-addons-latest-nightly but for l10n tests.

mozmill-remote-latest-nightly

  • As mozmill-addons-latest-nightly but for remote tests.

Notes

  • URL build trigger for downloading latest build is not the best solution. We need to come up with another.
  • Mozmill automation scripts always exit with a successful exit code. This means that the jobs are reported as successful even if there are failures. Fortunately, at least Jenkins updates the job result based on the JUnit report, so failures would not go unnoticed. See bug 626712 for introducing valid exit codes.
  • Jenkins jobs support parameters when manually triggered. We could use this to target specific locales, or to make the download script more flexible. These parameters cannot be controlled when using build triggers however, and defaults would be used.
  • We can use multiple-configuration builds to run the same job with different parameters. For example I have set up download-latest-nightly-multiple-locales to download de, en-GB, and en-US. This triggers mozmill-l10n-latest-nightly-multiple-locales with the same configuration.

Questions/Proposals

  1. A plugin like the following could help us to prepare the workspace for testruns across slaves of the same platform. Downloads will not have to happen in parallel by the slaves but will all be done by the "master" and handed over to the slaves
  2. Do we want to place the master's Jenkins folder under the home directory of the user? Therefore we should work with the .war file instead of installing a native Jenkins build onto the system?
  3. Do we have a fallback strategy if something is broken? Could we still trigger/execute tests manually for a given platform.
  4. For environment targets: Unzipping vs downloading without packing. Which is more complicated?
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Automation/Projects/Mozmill_Automation/On_Demand_Test_Framework/CI_Documentation&oldid=887755"