CloudServices/SoftRelease

The SoftRelease service offers a way to activate/degrade a new feature to a subset of Firefox OS / Firefox users.

Use cases examples: - ramping up Firefox Hello for our user base by making it accessible to 10% of the user base and growing it to 100% once we are confident that the server infrastructure works well. - activating a feature for specific regions, like a new FxA Avatar system only for France. - making small UI variations like what they're doing at teh Foundation for end of year campaign, see https://fundraising.mozilla.org/testing-testing-and-more-testing/

General Principle

When the client starts (Firefox OS, Firefox or a WebApp), it sends a request to the SoftRelease service to ask if a feature has to be activated or not.

The proposed API is a single HTTP endpoint that contains the name of the feature:

GET https://features.services.mozilla.com/<feature_name>

The server analyzes the client IP and specific headers like the User Agent, and returns a JSON mapping containing the answer:

{'enabled': true}

When the enabled key is sent back, the client gets a YES/NO answer and acts upon it. For example, for Firefox HELLO, the decision to display the Hello button or not could be done by this call:

GET https://features.services.mozilla.com/hello

In some other cases, the feature is activated but we want to make different versions. it's preferrable to let the client decide what to do, given a list of options values sent back by the server.

For example, in a campaign page, an UI has a button with two options that may vary amongst users: its color and its text. The client can call the server to get back the values to use:

GET https://features.services.mozilla.com/campaign-2015

{'color': '#ff0000',

'value': 'Click Here'}

When the client wants to get several features at once, it can batch its requests by calling the root endpoint:

GET https://features.services.mozilla.com?features=campaign-2015,hello {'hello': {'enabled': true},

'campaign-2015': {'color': '#ff0000', 'value': 'Click Here'}}

Server implementation details

For the simple yes/no answer, we can use a DNS-based system. XXX

Ramp up

A feature in Firefox has a set of prefs under a namespace. For example, Hello uses "loop". A feature that whishes to use the soft release mechanism has two pref set to "true": <name>.enabled and <name>.throttled.

Upon first startup, each client will select a random number in the range of 1 to 2²⁴-2 and write it into the "<name>.soft_start_ticket_number" pref. Then, upon this and every subsequent startup, each client will check the value of the "<name>.throttled" pref. If set to true, then the client checks the value of a DNS A record (tentatively "<name>.sofstart.services.mozilla.com" -- which is required to be in the range 127.0.0.0 - 127.255.255.255.

If the record is outside this range, or if there is an error retrieving the A record, then the client does not activate the feature.

If the record is successfully retrieved, then the low 24 bits of the address are treated as a "now serving" number, and compared to the value stored in "<name>.soft_start_ticket_number". If the value is strictly greater than the selected ticket number, then the feature is activated, and the "<name>.throttled" pref is set to false (which will bypass this procedure for all subsequent startups).

This allows us to increase load on the system very gradually after launch. The recommended handling of this number is as follows:

1. Ensure that the TTL for the DNS record is set to a relatively short value, so as to allow changes to propagate through the system rapidly. recommended value is in the range of 600 to 3600 seconds (10 minutes to an hour).
2. When initially launched, set the load level to 10%. Leave it at that level for at least 24 hours and observe server load.
3. If server utilization is sufficiently low, increase the load level incrementally, waiting at least 24 hours between each change to ensure that server load can settle.
4. Once server load is ramped all the way to 100%, file a bug to remove the throttling logic from the Loop feature.

For easy reference, the following table calculates the IP address values for loads from 0% to 100%, in 5% increments:

Load (%)	Load (24-bit integer)	IP Address
0%	0	127.0.0.0
5%	838860	127.12.204.204
10%	1677721	127.25.153.153
15%	2516582	127.38.102.102
20%	3355443	127.51.51.51
25%	4194303	127.63.255.255
30%	5033164	127.76.204.204
35%	5872025	127.89.153.153
40%	6710886	127.102.102.102
45%	7549746	127.115.51.50
50%	8388607	127.127.255.255
55%	9227468	127.140.204.204
60%	10066329	127.153.153.153
65%	10905189	127.166.102.101
70%	11744050	127.179.51.50
75%	12582911	127.191.255.255
80%	13421772	127.204.204.204
85%	14260632	127.217.153.152
90%	15099493	127.230.102.101
95%	15938354	127.243.51.50
100%	16777215	127.255.255.255

GeoLoc

The Geoloc mode allows a feature to be activated in specific locations. Example of locations: Mozilla VPN, Europe, France, California, etc.

A feature that whishes to use the geoloc release mechanism has two pref set to "true": <name>.enabled and <name>.geoloc.

Upon every startup, each client will check the value of the "<name>.geoloc" pref. If set to true, then the client checks the value of a DNS A record (tentatively "<name>.geoloc.services.mozilla.com" -- which is required to be 127.0.0.0 or 127.255.255.255. If the value is 127.0.0.0, the feature is activated. If it's 127.255.255.255, it's not activated.

If the record is different from these two values, or if there is an error retrieving the A record, then the client does not activate the feature.

Server Side Architecture

The SoftRelease service is composed of two components:

• Amazon's Route 53 DNS system
• A custom Dashboard application that lists all features that are activated in SoftRelease and let us manage them.

Dashboard

TBD