From MozillaWiki
Jump to: navigation, search

Document Overview

Feature/Product: Case Conductor
Projected Feature Freeze Date: 3/31/2012
Product Champions: Cameron Dawson, Rebecca Billings, Carl Meyer
Privacy Champions: Sid Stamm
Security Contact: Adam Muntner
Document State: [AT RISK] follow-up / log policy


Architectural Overview: (date TBD)
Recommendation Meeting: (date TBD)
Review Complete ETA: tbd


In this section, the product's architecture is described. Any individual components or actors are identified, their "knowledge" or what data they store is identified, and data flow between components and external entities is described.

The main objective of this feature/product is: This is a test case management system. It is Free and open-source. This is our upcoming replacement for Litmus. The system is designed to provide test case management, and status on test execution. Test managers can write tests for testers to execute, and they can see the results of the tests in real-time.

We will host an instance of this product as our own test case management system. However, the source is checked into github here and other organizations will deploy their own instances.

Use Cases

Roles and functions

There are 4 main roles. These are the roles and main use-cases.

  • Testers - Testers can only execute tests. In this case, usually community members will be given a URL to an existing Test Run.
  • Test Creators - These users can do everything that Testers do, but they also can create new tests. These are "semi-deputized" members of the community. They will create new tests, and a Test Manager or Admin will activate them, if they pass muster.
  • Test Managers - These are usually in-house QA or Product owners, or fully deputized Community members. These users can specify new Products, and Product Versions that should be tested. They specify the environments that must be tested for those products. They can create new Test Cases and Suites of Test Cases and activate them so they can be run. They can also create Test Runs (collections of Test Suites that can be executed). Once this use activates the Test Run, then all the users above can execute the test cases and provide results.
  • Admins - These users can do everything from above, and can also manipulate user accounts.

Running Tests

All users can run tests. They nav to a test run, specify the environment they're using (such as Windows 7) and begin executing test cases. They mark them Pass / Fail / or Invalid (if the case is unclear to the point they can't tell if it passes or fails).

Viewing Results

All users can login and see results of executed test cases. They can see what passed, what didn't and what cases were not clear. Users can also see % coverage for test runs. Users can also "drill down" and see if all the environments for a specific test case of a run have been executed.


This is where the user changes the items in the system. Management is specific per role (as outlined above).


Originally this product was designed to be in two parts. A "Platform" written in Java providing REST endpoint APIs and a "User Interface" written in Django. Despite some people's investment in this relationship with uTest and the separation of the two parts, eventually it was determined that this architecture was untenable. At this point (Nov 2011) we decided to re-architect as pure Django.

One result of this change, is that the product is no longer a joint project with uTest, however they own the name "Case Conductor." So we will be changing the name. The branding search is underway.


Project Plan:Pivotal Tracker



Describe any major components in the system and how they interact. Also include any third-party APIs (those Mozilla does not control) and what type of data is sent or received via those APIs.

Deployment Considerations: readthedocs

The main components of the product are:

  • Web App
  • BrowserID (external component)
  • File Store / Attachments (external component)

Web App

This component manages:

  • Products & Versions
  • Environments
  • Test Runs, Suites & Cases
  • Test Results
  • Users / Authentication

The user can choose to login with either browserID, or they can just have a username, email and password stored with us. Email and username are stored as strings in the db. Passwords are salted and hashed with bcrypt, using the django-sha2 app built by mozillian Frederic Wenzel.

The tables below simply summarize the data encountered by this component.

Stored Data:

What Where
Products - String definitions of products such as Firefox or Thunderbird. Products under test. This is the top level item to which other items are bound. For instance, Test Cases will be specific to a Product. database
Product Versions - Products can have several versions. These are string or numerical versions. Environments that need testing are specified against the Product Version, because they may be different for different versions. There is one version of each test case for each Product Version for the same reason. database
Environments - These are categorized things like the category "language" may have "english, spanish, italian" as well as the category of "OS" may have "OS X, Windows 7, Linux, etc." Environments can be any set of strings that need test coverage for the Product Version. database
Test Runs - A collection of test cases, specific to a Product Version, that need testing. Once "activated" a tester can begin providing results for each environment of each test case. database
Test Suites - Collections of test cases that can be added to Test Runs for execution. database
Test Cases - Specify a set of actions that must be executed. A case is often executed for several different environments. database
Users - Users of the system that can have roles such as Admin (can change anything), Test Mangager (can edit and create tests), Test Creator (can create new tests, but not edit existing ones), and Tester (can only execute and provide results for tests) database
Attachment URLs - Files attached to test cases to help with the test. Such as a video file used to test WEBM video. database

Communication with File-Store / Attachments Attachments are files stored in a file store. In our deployment, it's just the local file system. Full documentation on the Django file store API can be found here: Django File Store API

Direction Message Data Notes
In: url(name) hyperlink pointing to the file in question attachments are referenced by hyperlinks. when the user clicks a link for this file, it opens the hyperlink in the browser.
Out: get_available_name(name) text an available name in the file store to save this file
Out: save(name, content) text, binary the name of the file, and the binary content of that file

Communication with BrowserID Documentation on using BrowserID with a web app can be found here: BrowserID Dev Guide

Direction Message Data Notes
In: assertion string this is a signed claim that proves the user is who they say they are. This contains the email that the user signed into BrowserID with. We use that email to decide which user to authenticate.
Out: assertion an encoded assertion
Out: audience the host and port of this web app

User Data Risk Minimization

In this section, the privacy champion will identify areas of user data risk and recommendations for minimizing the risk.

Breach and Log Disclosure

The Risk is that information logged by the Web App server will be unintentionally disclosed to untrusted third parties through a breach event.

Recommendation: Work with infrasec and IT/Ops to minimize logging, minimize retention window, and deploy a secure logging infrastructure.


Alignment with Privacy Operating Principles

In this section, the privacy champion will identify how the feature lines up with Mozilla's privacy operating principles.

See Also: Privacy/Roadmap_2011#Operating_Principles:

Principle: Transparency / No Surprises

Users of the system must create an account and use the system specifically for testing. The system's operation must be clear to the users, both by opening the source (done) and by providing information to users at appropriate times about data collection, etc. For users who upload test cases (things they created themselves), it's obvious and unsurprising what happens.

The Risk: is that test-performing users may be submitting data to the Web App without knowing they're submitting it. For example, I may run a test and submit my results but it may not be clear whether the results contain my browsing history, search queries, etc. It should be clear when I submit data what I'm submitting.

Recommendations: Ensure that users submitting test data to the system are aware of the types of data being submitted (and have access to the data itself). This can be done through a prompt ("Hey, thanks for finishing the test, we're going to collect x, y, z from you now") or a dashboard where users can see the type of data they submit.

[RESOLVED] Users type or paste anything they submit into a text box. It's clear they understand what is being submitted.

Principle: Real Choice

Users must be able to participate in the system without being asked for unreasonably wide data sets, and they must understand how the system interacts with their data. It's pretty clear that all users of this system intend to help with creating, performing or analyzing tests.

Recommendations: None.

Principle: Sensible Defaults

Users of our other products should not be required to use this system (they're not).

Recommendations: None.

Principle: Limited Data

Data collected by the system should be limited to what is required for the intended functionality.

The Risk is that we may end up with data that never gets used for the clear value proposed to users of the system.

Recommendations: Work with infrasec and IT/Ops to minimize logging, minimize retention window, and deploy a secure logging infrastructure.


Follow-up Tasks and tracking

What Who Bug Details
[DONE] Initial Overview Discussion Sid, Curtis, Cam Done via Email, March-2012
[DONE] Initial Risk Analysis Privacy Eng. Team 28-March-2012
[DONE] Public Discussion dev.planning 10-Apr-2012
[AT RISK] log retention policy Michael Coates (mcoates) 741810