Confirm
67
edits
Changes
Create page from google doc
Feature Flagging with Normandy Pref Rollout
Normandy Pref Rollout is a feature that allows Mozilla to change the default value of a preference for a targeted set of users, without deploying an update to Firefox. This document focuses on the use of Pref Rollout as a mechanism to enable feature flagging in Firefox.
== Background: Feature Flags ==
It isn't always desirable to turn on a feature for all users at once, or exactly when a new version of Firefox is released. In this case, it is common to make the feature controllable via a flag, or a switch that can be used to turn the feature on and off. In Firefox this is generally a preference. There are then several mechanism we can use to turn this flag on to enable the feature off-train, Normandy pref rollout is a simple, safe, and featureful way.
== Capabilities of Normandy Pref Rollout ==
Normandy can change one or more preferences for a targeted set of users. This can be used to enable a feature that is controlled by a flag. Targeting can be based on many criteria, including:
* Firefox version
* channel (release, beta, nightly)
* a percentage of users
* country
* Firefox locale
* installed add-ons
* profile age
* any preference value
* many keys in Telemetry
Rollout should be considered permanent. Once a user matches a rollout targeting criteria, that user is enrolled in the rollout until it is explicitly undone on the server (rollback), or the built in defaults change to match the rollout (graduation). A version change, removing an add-on, changing a preference, or any other change in targeting criteria will not automatically remove a user from a rollout.
== When not to use Pref Rollout for feature flags==
You should not use Normandy's pref rollout feature for a feature if any of the following apply:
* If you are trying to answer a question, the feature is experimental in nature, or you need multiple branches.
* Consider a Shield preference study
* The feature is not already in the target version of Firefox, in a disabled state
* System add-ons are able to ship new code off-train
* You are targeting a version of Firefox older than 61.
* The feature requires a restart after the preference is changed
== When to use Pref Rollout for feature flags ==
Normandy pref rollout may be suitable if
* The feature is already confirmed to have a positive effect.
* It is undesirable to wait until the next train switch to turn on the feature.
* Gradual rollout is desired.
* The new value is expected to become the default value in future versions.
* The feature is already on in Firefox, and should be turned off or adjusted.
* The feature is well tested by QA and automated tests.
Additional considerations:
* Feature flags are not a way to skip the trains.
* Feature flagging is not a replacement for thorough testing.
* Normandy pref rollout is not suitable for preferences that require a browser restart.
== Who's involved ==
* Feature Flag Champion
* Generally an engineer or PM from the team making the feature.
* Responsible for:
* defining rollout plan, timeline, and targeting criteria
* sending an intent-to-ship email
* writing the Normandy recipe
* Requires a thorough understanding of the feature at hand
* Release Management
* Relman must approve all feature flag changes.
* The champion should send an intent-to-ship to release-drivers@mozilla.com to inform release management of your feature's rollout
* They will provide guidance about possible conflicts with other feature being deployed.
* Product Integrity
* All features should be approved for release to the target audience by product integrity.
* Normandy team
* At this time, someone from the Normandy team should be involved as early as possible in feature development.
* Please file a bug in your component to deploy the feature via Normandy, and needinfo Michael Cooper (:mythmon) if you're considering using preference rollout.
== Intent to Ship ==
When sending an intent-to-ship email, it is important to be clear about the following items:
* This is an off-train feature flag change
* The testing plan, and the current state of testing
* The timeline and rollout plan
* The targeted audience, including channel and version
* What would cause the feature to be rolled back
Checklist
* Make a feature, controlled by a preference
* Get the preference tested by PI
* Make a rollout plan
* File a rollout bug, and needinfo the Normandy team
* Send an intent-to-ship email
* Create a Normandy recipe
* Get Normandy recipe approved
* Ship it!
== Rollback ==
Sometimes mistakes happen. A bug is found, the feature doesn't interact well with an important website, or there is some conflict found late in the process. In this case, a rollback is needed. This is a separate step, simply disabling the Normandy recipe will not cause a rollback.
Rollbacks can be targeted separately from rollouts, but should generally re-use the same targeting as the rollout. Rollbacks do not specify the preferences to change or their new values, but instead specify a specific rollout by name to undo.
Rollouts are semi-permanent: they won't change unless some other action is taken. Rollbacks are actually permanent. Once a rollout has been rolled-back, it can't be re-rolled-out without starting over with a new rollout.
== Relationship with Shield ==
Shield is a program for studies and experimentation on Firefox. Shield also use Normandy to make changes to Firefox, including changing preferences. The difference between Shield and preference rollout is one of intent. Preference rollout is meant for permanent changes that we are sure of. Shield is meant for testing variations and figuring out what, if anything, is the best thing to do.
In short, Shield is for asking questions, and preference rollout is for applying the answers that come out.
Normandy Pref Rollout is a feature that allows Mozilla to change the default value of a preference for a targeted set of users, without deploying an update to Firefox. This document focuses on the use of Pref Rollout as a mechanism to enable feature flagging in Firefox.
== Background: Feature Flags ==
It isn't always desirable to turn on a feature for all users at once, or exactly when a new version of Firefox is released. In this case, it is common to make the feature controllable via a flag, or a switch that can be used to turn the feature on and off. In Firefox this is generally a preference. There are then several mechanism we can use to turn this flag on to enable the feature off-train, Normandy pref rollout is a simple, safe, and featureful way.
== Capabilities of Normandy Pref Rollout ==
Normandy can change one or more preferences for a targeted set of users. This can be used to enable a feature that is controlled by a flag. Targeting can be based on many criteria, including:
* Firefox version
* channel (release, beta, nightly)
* a percentage of users
* country
* Firefox locale
* installed add-ons
* profile age
* any preference value
* many keys in Telemetry
Rollout should be considered permanent. Once a user matches a rollout targeting criteria, that user is enrolled in the rollout until it is explicitly undone on the server (rollback), or the built in defaults change to match the rollout (graduation). A version change, removing an add-on, changing a preference, or any other change in targeting criteria will not automatically remove a user from a rollout.
== When not to use Pref Rollout for feature flags==
You should not use Normandy's pref rollout feature for a feature if any of the following apply:
* If you are trying to answer a question, the feature is experimental in nature, or you need multiple branches.
* Consider a Shield preference study
* The feature is not already in the target version of Firefox, in a disabled state
* System add-ons are able to ship new code off-train
* You are targeting a version of Firefox older than 61.
* The feature requires a restart after the preference is changed
== When to use Pref Rollout for feature flags ==
Normandy pref rollout may be suitable if
* The feature is already confirmed to have a positive effect.
* It is undesirable to wait until the next train switch to turn on the feature.
* Gradual rollout is desired.
* The new value is expected to become the default value in future versions.
* The feature is already on in Firefox, and should be turned off or adjusted.
* The feature is well tested by QA and automated tests.
Additional considerations:
* Feature flags are not a way to skip the trains.
* Feature flagging is not a replacement for thorough testing.
* Normandy pref rollout is not suitable for preferences that require a browser restart.
== Who's involved ==
* Feature Flag Champion
* Generally an engineer or PM from the team making the feature.
* Responsible for:
* defining rollout plan, timeline, and targeting criteria
* sending an intent-to-ship email
* writing the Normandy recipe
* Requires a thorough understanding of the feature at hand
* Release Management
* Relman must approve all feature flag changes.
* The champion should send an intent-to-ship to release-drivers@mozilla.com to inform release management of your feature's rollout
* They will provide guidance about possible conflicts with other feature being deployed.
* Product Integrity
* All features should be approved for release to the target audience by product integrity.
* Normandy team
* At this time, someone from the Normandy team should be involved as early as possible in feature development.
* Please file a bug in your component to deploy the feature via Normandy, and needinfo Michael Cooper (:mythmon) if you're considering using preference rollout.
== Intent to Ship ==
When sending an intent-to-ship email, it is important to be clear about the following items:
* This is an off-train feature flag change
* The testing plan, and the current state of testing
* The timeline and rollout plan
* The targeted audience, including channel and version
* What would cause the feature to be rolled back
Checklist
* Make a feature, controlled by a preference
* Get the preference tested by PI
* Make a rollout plan
* File a rollout bug, and needinfo the Normandy team
* Send an intent-to-ship email
* Create a Normandy recipe
* Get Normandy recipe approved
* Ship it!
== Rollback ==
Sometimes mistakes happen. A bug is found, the feature doesn't interact well with an important website, or there is some conflict found late in the process. In this case, a rollback is needed. This is a separate step, simply disabling the Normandy recipe will not cause a rollback.
Rollbacks can be targeted separately from rollouts, but should generally re-use the same targeting as the rollout. Rollbacks do not specify the preferences to change or their new values, but instead specify a specific rollout by name to undo.
Rollouts are semi-permanent: they won't change unless some other action is taken. Rollbacks are actually permanent. Once a rollout has been rolled-back, it can't be re-rolled-out without starting over with a new rollout.
== Relationship with Shield ==
Shield is a program for studies and experimentation on Firefox. Shield also use Normandy to make changes to Firefox, including changing preferences. The difference between Shield and preference rollout is one of intent. Preference rollout is meant for permanent changes that we are sure of. Shield is meant for testing variations and figuring out what, if anything, is the best thing to do.
In short, Shield is for asking questions, and preference rollout is for applying the answers that come out.
