Difference between revisions of "Firefox/Shield"

From MozillaWiki
Jump to: navigation, search
(Complete re-write and revision to reflect State of Shield in Aug 2017)
(Supported Study and Messaging Designs)
 
Line 19: Line 19:
  
 
; Preference-flip to control feature
 
; Preference-flip to control feature
; Preferenc-flip to control (disabled by default) system addon
+
; Preference-flip to control (disabled by default) system addon
 
; Add-on Based study (opt-in) [[Firefox/Shield/Shield_Studies|Shield Studies]]
 
; Add-on Based study (opt-in) [[Firefox/Shield/Shield_Studies|Shield Studies]]
 
; Add-on Based Study (opt-out) [[Firefox/Shield/Shield_Studies|Shield Studies]]
 
; Add-on Based Study (opt-out) [[Firefox/Shield/Shield_Studies|Shield Studies]]
Line 29: Line 29:
 
; Pioneer: On-going opt-in cohort for deeper data collection  
 
; Pioneer: On-going opt-in cohort for deeper data collection  
 
; Rollout preference study
 
; Rollout preference study
 
  
 
=== Choosing A Design for Evaluating Features ===
 
=== Choosing A Design for Evaluating Features ===

Latest revision as of 09:40, 10 November 2017

SHIELD PROGRAM

Shield Turns Superstition into Science

Shield is a Firefox user testing platform for proposed, new and existing features and ideas. Broad Applications The Shield system addon targets specific subsets of users, enables features and messaging, and measures the results of those interventions to evaluate the effect of the features/messages on measures (retention, uptake, performance, crashiness) to create insights for the Firefox product and Mozilla as an organization.


Supported Study and Messaging Designs

Shield supports these Study and Outreach Designs

Preference-flip to control feature
Preference-flip to control (disabled by default) system addon
Add-on Based study (opt-in) Shield Studies
Add-on Based Study (opt-out) Shield Studies
Recruits users to participate in testing new features and ideas that help us validate product decisions. Our users tell us what to ship.
Heartbeat Star Rating with action link
Messaging Study with action link to external site (survey, Brain Games, interface testing, external user task tool)
Contextual Feature Recommendation
Customized action after user event
Pioneer
On-going opt-in cohort for deeper data collection
Rollout preference study

Choosing A Design for Evaluating Features

Issues in design choice:

  • Code deployment: in-tree (either as native code, or system addon) vs. Extension (Legacy Extension or Addon)
  • Sensitivity of data / probes collected. In particular, full urls usually require an opt-in design
  • Speed vs. fidelity: if possible, use Images, fake prototypes, etc FIRST to do validation. Gather data using **surveys** of Firefox users, Other Browser users or whoever.

Deciding:

  • Just an Idea: Do surveys! Minimal wireframe prototypes! Pile-sort! Don't build! Identify needs, user models, and terminology before solving and building!
  • if you are already in tree (mozilla-central), including your probes, use "pref-flip"
  • Not in tree, use an opt-out (if data is benign).
  • Not in tree, need more sensitive data, needs opt-in (at the risk of recruitment bias), choose opt-in (one-off study) or Pioneer.

Alternatives To Shield

Other methods support these designs better:

  • early stage ideation: simple hallway / cafe testing and paper prototypes
  • You need Chrome / IE / Edge users - we can recruit those using other methods
  • You want in-depth narrative data.


Other methods for launching and testing features or messages include

  • AMO addon
  • Test Pilot
  • Snippets


Launch a Shield Study

Overview: File Bug, Send email to Release-Drivers, tell Shield Team to start recruiting.


Launch Schedule, Backlog, In Progress

Engineering and Building

Shield System Addon - downloads, interprets and runs recipes

The Shield system addon pulls a list potential "recipes" from the Normandy server on a regular (daily) basis. It allows us to quickly control the delivery of recipes outside of the trains. The addon then filters recipes based on client configuration (channel, locale, geo, addons, prefs). The client-side component lets us constrain what actions are available to recipes for security purposes, and also provides a good point for testing the functionality of our fixes without removing our ability to ship fixes fast.

Read details at: Shield client addon concepts


Normandy - User Profile Matching and Recipe Deployment

Shield Studies Addon Utils

  • Single File `.jsm`.
  • Embed these inside your Legacy Addon or Embedded WebExtension addon.
  • Shield-Studies-Addon-Utils on Github
  • if you ARE NOT ON VERSION 4+, please stop :)
  • No more Jetpack / Addon-sdk!

Experienced Shield Study Authors Include:

  • Gregg Lind
  • Kamyar Ardekani
  • Benton Case
  • Luke Crouch
  • (others!)


Contact Us

Not Satisfied? For all additional questions or live support

Visit our SLACK Channel: #shield on mozilla.slack.com and say "help".


Shield Core Team

  • Project Lead: Gregg Lind [:gregglind]
  • Product Owner: Matt Grimes [:Matt_G]
  • Data Steward: Rebecca Weiss [:rweiss]
  • Development: Michael Cooper [:mythmon], Michael Kelly [:Osmose], Web Engineering * Systems
  • Operations: Daniel Thorn [:relud], Benson Wong [:mostlygeek]
  • QA: Chris Hartjes [:chartjes], Adrian Florinescu [:aflorinescu], Ovidiu Boca [:Ovidiu]
  • Status: Launched / Active Development


History

  • June 2014 * mconner files a tracker bug for enhancing FHR to proactively detect and solve problems for users.
  • December 2014 * bsmedberg files bug 1111016 for implementing the smallest-possible set of changes necessary to support Heartbeat (a prompt shown to users asking them to rate Firefox) and Self-Repair (a minimal version of the problem-solving functionality, described in detail in dependent bugs).
  • Early 2015
    • UITour is enhanced to support Heartbeat prompts, sending user responses to input.mozilla.org.
    • Self-Repair is implemented as a set of static files hosted on AWS that Firefox retrieves.
  • Late 2015 * Self Repair is renamed to SHIELD since the project covers more than just repairing Firefox. The name covers the server that deploys configuration files (recipes) and the in-product component that executes the recipe.
    • Example 1: The Shield server deploys a recipe to a client indicating that the user should be included in the daily Heartbeat sample. The in-product component responds by offering the Heartbeat prompt.
    • Example 2: The SHIELD server matches a “preference repair recipe” to the user configuration, and returns it. Firefox runs the recipe steps to fix the broken preference.
  • February 2016 * Development begins on the SHIELD backend service, Normandy. Work on the system add-on begins as well.
  • April 2016 * Normandy replaces the self-repair server at the domain self-repair.mozilla.org. Development on the system add-on continues.
  • December 2016 * SHIELD System Add-on lands in mozilla-central
  • 2017 * Shield adds pref-flip studies, v4 of shield utils and many other things.