Apps/QA/Test Infrastructure

From MozillaWiki
< Apps‎ | QA
Jump to: navigation, search

Test Infrastructure Requirements

The Problem

The problem we are currently facing in our testing environments is that each group has their own infrastructure. The infrastructure is built around varying concepts of dev , beta, staging environments.

The problem is those environments all have different schedules that code is pushed, and released, and it is not possible to guarantee what version of the marketplace, browser id, Apps , Sync Tests have gone through a level of integration testing.

Each group has their own infrastructure, and it's not entirely clear to each group what the benefits of moving to a shared environment are. Current Release Schedules Are the following:

Sync Service -

 Weekly Push
 Wed Push to staging 
 Monday - Release to production

MarketPlace -

 Infrastructure Diagram Release Infra
 addons.allizom is staging server, but only updated as needed
 Release happens 2pm Thursday
 DBA Sheeri / MPressman

BrowserID -

 Thursdays: deployment to Production, Stage (QA), and Dev
 Thursdays/Fridays: open testing and experimentation by Dev, QA, and community
 Following week: Monday - Wednesday: QA testing and sign off of current deployment 
 DBA: petef

Apps - Web Services (HTML / JS / Dashboard)

 As needed

The Purpose

 Define an apps test environment that allows for testing javascript api's, the firefox browser, and the android Soup app. 

QA should have the ability to qualify a build, and know with certainty that those are the bits being shipped. Production releases should be done using the jenkins server to push the files to production.

Minimum Requriements

Initial Hardware Requirements 


Webapps test infrastructure.png


Jenkins Cluster

The brains of our jenkins infrastructure resides on:

Jenkins machine is to be snapshotted monthly, as well as having its configuration backed up to github

Slaves will be deployed as needed to support additional builds / jobs. Current machine load is minimal, the intention is to add this as needed by the demands of the project.

There is an open request for a windows virtual machine, which will also function as a jenkins slave.

The requirements are detailed in the bug at the bottom of the page.

High Availability

High Availability Concerns are addressed using the following two solutions.

I have also checkpointed within Automation Services team, and these seem to be standard mechanisms to achieve the required result

Multiple Build Targets

Within each build we have hardcoded values for myapps.mozillalabs, and apps.mozillalabs. Ideally we need to parameterize these inputs to the build process.

 lib/api.js:      "":"", "":"",
 lib/api.js:      "":"", "":"",
 lib/main.js:const APP_SYNC_URL = "";
 lib/main.js:      "https?://",
 lib/main.js:      "*",
 lib/main.js:      "https?://",
 lib/main.js:   * should be got from
 lib/main.js:  let button = win.document.getElementById("" + TOOLBAR_ID);
 lib/main.js:    let dboard = "";
 lib/main.js: * library, hosted at
 lib/main.js:      contentURL: "",
 lib/main.js:        if (origin == "") {
 lib/main.js:      if (!found)"");
 lib/nativeshell.js:                                "");
 lib/nativeshell.js:                                      "");
 lib/nativeshell.js:        subKey.writeStringValue("Contact", "");
 lib/nativeshell.js:        //subKey.writeStringValue("Readme", "");
 package.json:    "id": "",

The same is true for android. It is important to be able to modify these parameters for QA / Release / Automation. So that when we click on the Apps icon in the lower right of the browser it launches the staging myapps instead of the production

We will probably need to be able to generate our own nightlies based upon a github repo build of firefox. This would be necessary for QA to be able to test based upon what is checked in to github vs having to wait until patches are reviewed.

Staging Servers

  • This is a where the dashboard will be hosted. Gives us the ability to test dashboard changes before they go live.
  • Serve test apps that are staged for qa. Currently a lot of the tests we run are based upon external apps. When local apps provide a more reliable alternative.
  • apps-staging will also host a staging version of the html / js version of mozApps apis.

Server Rearrangement

Some thoughts from Ian:

I think we can move everything to a single server (we'll call it myapps-staging): the HTML shim, the dashboard, the test appdir, and we should be able to generate an XPI there as well, and in addition future sync deployments.

If we can automate this well, then we can potentially base it all on a manifest of repositories. A manifest might look like:


And do the same thing for each component (since we have multiple repos). If we can build that on a brand new VM (maybe just an externally hosted cloud VM) we can really let people create their own manifests to test other combinations.

Open Questions and Answers

  • (jsmith) We should consider building out a community landing page on the build machine (similar to how firefox has a aurora, beta, nightly build)
    • +1 by David and Jason
  • (jsmith) Jenkins - How could we address fault tolerance in our continuous integration?
    • (dclarke) Snapshots. The internet connection was lost due to a NAT / Firewall issue that affected all the virtual machines.
    • (dclarke) But I can see a benefit to having a backup process, fault tolerance would indicate that we are not able to tolerate faults, and i think for a CI server, especially where we are at in the release process. Time could be more appropriately assigned.
  • (jsmith) With the apps hosting server (, how it will address the issue with the one app per origin problem?
    • (dclarke) Single app per origin problem: you can host different apps on different ports, and it's not a problem.
  • (jsmith) Right now, the system only supports allowing one app to get hosted on a single origin
  • (jsmith) How do we get around that issue?
  • (jsmith) For developers - How would they say, test their own implementations for their own specific work? Is the plan to do it all locally on their machine? Or is there a way to deploy it remotely to specific areas for their individual testing?
  • (jsmith) How does the test infrastructure support separation of concerns of different build needs (building different builds, running unit tests, running system tests)?
    • (dclarke) Good point, ian brought up a scheme which would allow us to kick off jobs based upon branches, so developers can kick off and view the reports to their own jobs.
  • (jsmith) Right now - I see one server all handling this, could that be too much load on one particular system?
  • (jsmith) Might want to consider using Jenkins support for master-slave relationships to cut back the centralized load on the infrastructure
    • (dclarke) Master - Slave is fine, and probably the direction we will go. For builds that are quite labor intensive Ex; building firefox, running test automation, but for right now the build process can be kept local, because it is so light weight.

Project Tracking

 Infrastructure Phase I Tracker Bug:
 Linux Build Hosting Machine:
 Windows Jenkins Virtual Machine:
 Linux Myapps Staging Machine: