Mobile/Fennec/Android/Switchboard

Switchboard is a service we use to segment users for A/B testing or staged rollout of features.

An experiment can be enabled for a specified percentage of users and limited to release channels, app versions, languages, countries (locale), device names and device manufacturers. An experiment can contain additional metadata to "configure" the experiment. This can be used to change parameters of an experiment without needing to ship a new app version. Which experiments a user is enrolled in is reported to our telemetry systems via the core ping.

History

We started with KeepSafe's Switchboard library and server component but since then the code and architecture has changed significantly. The original implementation used a special switchboard server that did decides which experiments a client is part of and then just returns a simplified list for the client to consume. This required the client to send data (including a unique id) to the server. To avoid this and make the administration of experiments simpler we moved to using Kinto as storage and server of the experiment configuration. Now clients load the whole configuration and decide independently what experiments they are enrolled it.

Summary

• The experiment configuration is stored in Kinto and served from a CDN: https://firefox.settings.services.mozilla.com/v1/buckets/fennec/collections/experiments/records
• The client regularly loads the configuration (or diffs, see Kinto API) and stores it locally.
• Based on a UUID for bucketing the client and using the experiment filters the client decide which experiments it is part of.
• The experiment configuration can be modified using the Kinto web admin (VPN + account whitelisting required)

Adding a new experiment

Client

Server side

• Go to https://settings-writer.prod.mozaws.net/v1/admin (You need to be on the Mozilla network, either in an office or via VPN)
• Login with your LDAP credentials (Your account needs to be whitelisted)
• Follow the Web UI to create or modify an experiment. See below for an explanation of the fields.
• Currently saved changes go live immediately! The Kinto team is setting up a process for a mandatory review step.

Client side

• Add your experiment name to Experiments.java
• Use SwitchBoard.isInExperiment() to check whether the client is part of your experiment.
• Use SwitchBoard.getExperimentValuesFromJson() if you have configured additional metadata for your experiment.

Testing

• There's an add-on that you can use for overriding the local switchboard configuration: Addon Download Source code. This works great for switching experiments on/off - but doesn't let you configure additional metadata if required by the experiment.

Experiment configuration

• Name: A descriptive name for the experiment. This name is used in JSON that is being sent by the server and stored by the client. Examples are: bookmark-history-menu, malware-download-protection, promote-add-to-homescreen
• Description: A more detailed description what this experiment is about. In the best case this contains a bug number for context. Currently we send this data to the clients too (Don't make it too big).
• Regions: It check the country we get in the search engine manager (e.g. US, TW). If there's no value stored we just think we are not in the experiment. In the admin interface, it should be a list (e.g {'US','TW'} ).
• Buckets: Every user is in one of 100 buckets (0-99). For every experiment you can set up a min and max value (0 <= min <= max <= 100). The bounds are [min, max). An experiment set to buckets min=0, max=50 means that users in buckets 0-49 will have it enabled. There is no check for out of bounds values, so "disabled" experiments can have min/max buckets keys that are either out of bounds or have min == max. A disabled experiment is usually set to min=0, max=0.
• Filters - In addition to buckets additional filters can be used to limit the experiments to a subset of users.
  • appId (regex): The Android app ID (e.g. org.mozilla.fennec, org.mozilla.firefox_beta, org.mozilla.firefox)
  • version (regex): The Firefox app version number (e.g. 47.0a1', 46.0)
  • lang (regex): Language, pulled from the default locale (e.g. eng)
  • country (regex): Country, pulled from the default locale (e.g. USA)
  • device (regex): Android device name
  • manufacturer (regex): Android device manufacturer
• values: Every experiment can include an optional JSON object with values that are used by the experiment. For example an experiment showing a bookmark dialog after a page has been visited multiple times, could use a value that defines how many visits are needed to display the dialog. With that the experiment can change without needing to update the app.