CloudServices/SoftRelease: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
Line 56: Line 56:
</pre>
</pre>


= Server implementation details =
= Dashboard =


For the simple yes/no answer, we can use a DNS-based system.
To manage the responses, the service provides a dashboard where admins can:
XXX


== Ramp up ==
* add or remove a feature name
* list the different possible responses that can be returned to a user
* configure a policy for the server to decide which response to return


A feature in Firefox has a set of prefs under a namespace. For example, Hello uses "loop". A feature that whishes to use the soft release mechanism has two pref set to "true": <name>.enabled and <name>.throttled.
We provide 3 policies


Upon '''''first''''' startup, each client will select a random number in the range of 1 to 2<sup>24</sup>-2 and write it into the "<name>.soft_start_ticket_number" pref. Then, upon this and every subsequent startup, each client will check the value of the "<name>.throttled" pref. If set to true, then the client checks the value of a DNS A record (tentatively "<name>.sofstart.services.mozilla.com" -- which is required to be in the range 127.0.0.0 - 127.255.255.255.  
* '''Weighted''': defines a percentage for each possible response.
* '''Geolocation''': associates regions to possible responses.
* '''User-Agent''': associates User-Agents to possible responses.


If the record is outside this range, or if there is an error retrieving the A record, then the client does not activate the feature.  
Policies can be combined.


If the record is successfully retrieved, then the low 24 bits of the address are treated as a "now serving" number, and compared to the value stored in "<name>.soft_start_ticket_number". If the value is strictly greater than the selected ticket number, then the feature is activated, and the "<name>.throttled" pref is set to false (which will bypass this procedure for all subsequent startups).
== Example ==


This allows us to increase load on the system very gradually after launch. The recommended handling of this number is as follows:
We want to try out a new donation campaign UI for French users. In their case, we want to set the color
# Ensure that the TTL for the DNS record is set to a relatively short value, so as to allow changes to propagate through the system rapidly. recommended value is in the range of 600 to 3600 seconds (10 minutes to an hour).
of the button Green. For other users, we want a Blue color.
# When initially launched, set the load level to 10%. Leave it at that level for at least 24 hours and observe server load.
# If server utilization is sufficiently low, increase the load level incrementally, waiting at least 24 hours between each change to ensure that server load can settle.
# Once server load is ramped all the way to 100%, file a bug to remove the throttling logic from the Loop feature.


For easy reference, the following table calculates the IP address values for loads from 0% to 100%, in 5% increments:
steps:


{| class="wikitable"
1. We add a new "campaign-button-color" feature via the dashboard
|-
2. we add the two possible responses:
! Load (%) !! Load<br/>(24-bit integer) !! IP Address
|-
|0% || 0 || 127.0.0.0
|-
|5% || 838860 || 127.12.204.204
|-
|'''10%''' || '''1677721''' || '''127.25.153.153'''
|-
|15% || 2516582 || 127.38.102.102
|-
|20% || 3355443 || 127.51.51.51
|-
|25% || 4194303 || 127.63.255.255
|-
|30% || 5033164 || 127.76.204.204
|-
|35% || 5872025 || 127.89.153.153
|-
|40% || 6710886 || 127.102.102.102
|-
|45% || 7549746 || 127.115.51.50
|-
|50% || 8388607 || 127.127.255.255
|-
|55% || 9227468 || 127.140.204.204
|-
|60% || 10066329 || 127.153.153.153
|-
|65% || 10905189 || 127.166.102.101
|-
|70% || 11744050 || 127.179.51.50
|-
|75% || 12582911 || 127.191.255.255
|-
|80% || 13421772 || 127.204.204.204
|-
|85% || 14260632 || 127.217.153.152
|-
|90% || 15099493 || 127.230.102.101
|-
|95% || 15938354 || 127.243.51.50
|-
|100% || 16777215 || 127.255.255.255
|}


== GeoLoc ==
<pre>
Green => {'color': 'green'}
</pre>


The Geoloc mode allows a feature to be activated in specific locations. Example of locations: Mozilla VPN, Europe, France, California, etc.
<pre>
Blue => {'color': 'blue'}
</pre>


A feature that whishes to use the geoloc release mechanism has two pref set to "true": <name>.enabled and <name>.geoloc.
3. we associate each response to a Geolocation Policy


Upon every startup, each client will check the value of the "<name>.geoloc" pref. If set to true, then the client checks the value of a DNS A record (tentatively "<name>.geoloc.services.mozilla.com" -- which is required to be 127.0.0.0 or 127.255.255.255. If the value is 127.0.0.0, the feature is activated. If it's 127.255.255.255, it's not activated.
<pre>
France: Green
</pre>


If the record is different from these two values, or if there is an error retrieving the A record, then the client does not activate the feature.
<pre>
 
default: Blue
== Server Side Architecture ==
</pre>
 
The SoftRelease service is composed of two components:
 
* Amazon's Route 53 DNS system
* A custom Dashboard application that lists all features that are activated in SoftRelease and let us manage them.


= Dashboard =  
= Dashboard =  


TBD
TBD

Revision as of 16:32, 11 December 2014

The SoftRelease service offers a way to activate/degrade a new feature to a subset of Firefox OS / Firefox users.

Use cases examples:

  • ramping up Firefox Hello for our user base by making it accessible to 10% of the user base and growing it to 100% once we are confident that the server infrastructure works well.
  • activating a feature for specific regions, like a new FxA Avatar system only for France.
  • making small UI variations like what they're doing at the Foundation for end of year campaign, see https://fundraising.mozilla.org/testing-testing-and-more-testing/

General Principle

When the client starts (Firefox OS, Firefox or a WebApp), it sends a request to the SoftRelease service to ask if a feature has to be activated or not.

The proposed API is a single HTTP endpoint that contains the name of the feature:

GET https://features.services.mozilla.com/<feature_name>

{'enabled': true}

The server analyzes the client IP and specific headers like the User Agent, and returns a JSON mapping containing the answer.

When the enabled key is sent back, the client gets a YES/NO answer and acts upon it. For example, for Firefox HELLO, the decision to display the Hello button or not could be done by this call:

GET https://features.services.mozilla.com/hello

In some other cases, the feature is activated but we want to make different versions. it's preferrable to let the client decide what to do, given a list of options values sent back by the server.

For example, in a campaign page, an UI has a button with two options that may vary amongst users: its color and its text. The client can call the server to get back the values to use:

GET https://features.services.mozilla.com/campaign-2015

{'color': '#ff0000',
 'value': 'Click Here'}

When the client wants to get several features at once, it can batch its requests by calling the root endpoint:

GET https://features.services.mozilla.com?features=campaign-2015,hello

{
 'hello': {'enabled': true}, 
 'campaign-2015': {'color': '#ff0000', 'value': 'Click Here'}
}

Dashboard

To manage the responses, the service provides a dashboard where admins can:

  • add or remove a feature name
  • list the different possible responses that can be returned to a user
  • configure a policy for the server to decide which response to return

We provide 3 policies

  • Weighted: defines a percentage for each possible response.
  • Geolocation: associates regions to possible responses.
  • User-Agent: associates User-Agents to possible responses.

Policies can be combined.

Example

We want to try out a new donation campaign UI for French users. In their case, we want to set the color of the button Green. For other users, we want a Blue color.

steps:

1. We add a new "campaign-button-color" feature via the dashboard 2. we add the two possible responses:

Green => {'color': 'green'}
Blue => {'color': 'blue'}

3. we associate each response to a Geolocation Policy

France: Green
default: Blue

Dashboard

TBD