Firefox/Shield/Shield Studies

From MozillaWiki
Jump to: navigation, search

Overview

Shield Studies are controlled A/B tests deployed as part of the Shield project. These studies allow us to compare proposed changes to the default experience in Firefox for representative populations before shipping those changes to everyone.

Shield Studies are available on all channels. Individual studies can be opt-out or opt-in depending on the nature of the study and the data being collected. Opt-out Shield Studies can only collect Type 1 and 2 data. This is the same type of basic interaction data we collect by default as defined in our privacy policy. There may be instances where we want to collect data that is not covered by our default data collection policies. In those instances you will be prompted to opt-in to the study. A complete description of the study and any additional data that would be collected will be disclosed before enrolling. A great example of this is Pioneer, which is actually an opt-in Shield Study currently in the wild.

At the end of the study period, the study will expire and return your system to the previous state. When the study expires, you may be asked to fill out a survey based on their experience.

Shield Studies can be targeted at various populations by specifying conditions in a Shield recipe. Additional configurations and conditions can be set through the add-on for the individual study.

Shield Studies FAQ

How Do I Request/Make a Shield Study

Documentation on the entire process of building and launching a Shield Study (including a sample timeline)

Who Approves a Shield Study before it ships?

Shield Studies must be approved by

  • A Firefox Product Manager
  • QA
  • Release Management
  • A member of the core Shield Team

Additional approvals may be required by:

  • Data Steward
  • Legal
  • Others

You can find detailed specifics in the link above. Most of these reviews are pro forma and straightforward.

What data do Shield Studies normally collect?

By default, Shield Studies will collect this data:

  • Firefox was used on a day
  • Usage hours
  • Study info: Study Name, Study Name
  • Answers provided in followup survey
  • User uninstalled / opted-out

Mechanism:

- at STARTUP, SHUTDOWN, INSTALL, UNINSTALL, - send a `shield-study` packet containing the Unified Telemetry Environment.


Additional measures may be added to individual studies pending approval from a data steward. All data collected will be disclosed prominently to the user at the time of consent.

Are Shield Studies just Add-ons?

Shield Studies are a special type of restartless extension. They can do anything a regular restartless extension can do.

Do Shield Studies need to be signed?

Yes! Shield Studies need to be signed to run in FF 42+, just like all other add-ons.

How long do Shield Studies last?

Shield Studies live for a fixed period and are uninstalled by the system at the end date or after the specified elapsed time. Usually that time is 14 days from the install date, but studies can last much longer if needed.

What are participation rates for opt-in Shield Studies?

Participation for opt-in Shield Studies is currently around 1-2% of randomly selected participants. We are working on improving the overall flow.