Test Pilot/2016Q4: Difference between revisions
Jump to navigation
Jump to search
mNo edit summary |
mNo edit summary |
||
| Line 16: | Line 16: | ||
| 30% improvement in Test Pilot experiment enable rate for new users | | 30% improvement in Test Pilot experiment enable rate for new users | ||
| <span style="color:orange;font-weight:bold;">0.50</span> | | <span style="color:orange;font-weight:bold;">0.50</span> | ||
| Starting rate is 8.02% which makes our target 10.42%. On Dec 31 the average for the quarter was 8.27% (an increase of 3.1%). However, | | Starting rate is 8.02% which makes our target 10.42%. On Dec 31 the average for the quarter was 8.27% (an increase of 3.1%). However, the average for just December was 12.60% (an increase of 57.1%) so we definitely did some good, but that success is weighed down by the short period we had to measure after we pushed changes. Instead of a score of 0.03 I'd like to call this 0.5 for the positive trend going into Q1 (and a note about being able to measure within a quarter). | ||
|- | |- | ||
| KR2 | | KR2 | ||
Revision as of 21:58, 4 January 2017
Test Pilot 2016 Q4 Objectives and Key Results
| Final Status (Dec 31) | Comments | ||
|---|---|---|---|
| Objective 1 | Grow the Test Pilot user base. | ||
| KR1 | 30% improvement in Test Pilot experiment enable rate for new users | 0.50 | Starting rate is 8.02% which makes our target 10.42%. On Dec 31 the average for the quarter was 8.27% (an increase of 3.1%). However, the average for just December was 12.60% (an increase of 57.1%) so we definitely did some good, but that success is weighed down by the short period we had to measure after we pushed changes. Instead of a score of 0.03 I'd like to call this 0.5 for the positive trend going into Q1 (and a note about being able to measure within a quarter). |
| KR2 | Reduce ratio of Test Pilot users (DAU) with no experiments installed by 30% | At start of quarter is 57.4%, so this moves the target to 75% | |
| KR3 | At least 100 different accounts have commented on the Test Pilot Discourse forum over the course of a month. | 0.27 | We had 27 unique folks who were active in the month from mid-oct to mid-nov. |
| Objective 2 | Provide evidence that Test Pilot is useful in helping to make product decisions | .44 | |
| KR1 | Publish conclusions from three experiments and graduate them from Test Pilot | 0.33 | Universal Search |
| KR2 | Launch two new experiments | 0.0 | Closest was Cliqz, but that got bumped to Jan. Snoozetabs was functional, but quite a bit left on polish. |
| KR3 | Build an offboarding process | 1.0 | Universal Search graduated. |
| Objective 3 | Have the ability to run experiments outside of the US Market | .5 | |
| KR1 | Add localization support to Test Pilot and get 80% of users using Test Pilot in their first choice language. | 1.0 | Averaged over Q4 en-US is 85% of requests so calling this green. Note that at the end of December en-US is down to 74.71% of requests. |
| KR2 | Launch at least one locale-specific experiment in one non-English locale | 0.0 | This was Cliqz which got bumped to Jan. |
Updates
- Mid-quarter review: https://discourse.mozilla-community.org/t/2016q4-okr-mid-quarter-review/12161