Test Pilot/2017Q3: Difference between revisions
Jump to navigation
Jump to search
mNo edit summary |
mNo edit summary |
||
| Line 10: | Line 10: | ||
! Objective 1 | ! Objective 1 | ||
! Improve storytelling so that we support Firefox and grow community | ! Improve storytelling so that we support Firefox and grow community | ||
! | ! <span style="color:red;font-weight:bold;">.48</span> | ||
! | ! | ||
|- | |- | ||
| KR1 | | KR1 | ||
| Bi-weekly updates for each new experiment published on the blog or experiment pages. | | Bi-weekly updates for each new experiment published on the blog or experiment pages. | ||
| | | <span style="color:orange;font-weight:bold;">.66</span> | ||
| | | We didn't publish updates at this cadence for every experiment, but we did publish a lot of things, including specific posts on the experiments. We decided this was about 2/3rds the spirit of this KR. | ||
|- | |- | ||
| KR2 | | KR2 | ||
| | | Get 1000 combined reads on all blog posts | ||
| | | <span style="color:green;font-weight:bold;">1</span> | ||
| | | So many reads. Over 6000 near the end of September. | ||
|- | |- | ||
| KR3 | | KR3 | ||
| | | Have 15 people participate in user research on design prototypes | ||
| | | <span style="color:red;font-weight:bold;">.27</span> | ||
| | | Only 4 participants in the Notes optimization study | ||
|- | |- | ||
| KR4 | | KR4 | ||
| 30 attendees at a Brown Bag about storyboarding | | 30 attendees at a Brown Bag about storyboarding | ||
| | | <span style="color:red;font-weight:bold;">0</span> | ||
| | | This was planned for Taipei but that trip was cancelled. We've made a new plan to host a meeting in Austin with the ET team. | ||
|- | |- | ||
| | | | ||
| Line 45: | Line 40: | ||
! Objective 2 | ! Objective 2 | ||
! Grow Screenshots usage and expand offered features so that people find Screenshots to be a useful tool. | ! Grow Screenshots usage and expand offered features so that people find Screenshots to be a useful tool. | ||
! | ! <span style="color:black;font-weight:bold;">??</span> | ||
! | ! | ||
|- | |- | ||
| KR1 | | KR1 | ||
| Ship screenshots to 100% of release users | | Ship screenshots to 100% of release users | ||
| | | <span style="color:green;font-weight:bold;">1</span> | ||
| | | 🎉 Shipping by default in 56 | ||
|- | |- | ||
| KR2 | | KR2 | ||
| 10% of Screenshots users sign in to FxA | | 10% of Screenshots users sign in to FxA | ||
| | | <span style="color:red;font-weight:bold;">0</span> | ||
| | | A patch to integrate FxA exists, but we didn't ship it to production. | ||
|- | |- | ||
| KR3 | | KR3 | ||
| 5% of non-owner, non-Firefox visitors click a branded CTA | | 5% of non-owner, non-Firefox visitors click a branded CTA | ||
| | | <span style="color:black;font-weight:bold;">??</span> | ||
| | | Need to get this number from John | ||
|- | |- | ||
| KR4 | | KR4 | ||
| All pages and API endpoints load in under 2s (minus network time) | | All pages and API endpoints load in under 2s (minus network time) | ||
| | | <span style="color:green;font-weight:bold;">.9</span> | ||
| | | All the endpoints are well under 2s now, except we found the "delete all data" endpoint near the end and it's still pretty slow. | ||
|- | |- | ||
| | | | ||
| Line 75: | Line 70: | ||
! Objective 3 | ! Objective 3 | ||
! Improve Test Pilot so that the Test Pilot program is a pleasure to use, conforms to the Firefox 57 plan, and improves engagement with experiments. | ! Improve Test Pilot so that the Test Pilot program is a pleasure to use, conforms to the Firefox 57 plan, and improves engagement with experiments. | ||
! | ! <span style="color:green;font-weight:bold;">1</span> | ||
! | ! | ||
|- | |- | ||
| KR1 | | KR1 | ||
| Ship SDK-free Test Pilot to 100% of users | | Ship SDK-free Test Pilot to 100% of users | ||
| | | <span style="color:green;font-weight:bold;">1</span> | ||
| | | | ||
|- | |- | ||
| KR2 | | KR2 | ||
| Use storybook for all newly implemented feature cards | | Use storybook for all newly implemented feature cards | ||
| | | <span style="color:green;font-weight:bold;">1</span> | ||
| | | | ||
|- | |- | ||
| KR3 | | KR3 | ||
| Implement 4 Test Pilot feature cards | | Implement 4 Test Pilot feature cards | ||
| | | <span style="color:green;font-weight:bold;">1</span> | ||
| | | 2 shipped, 2 waiting on L10n freeze and will ship next sprint | ||
|- | |- | ||
| KR4 | | KR4 | ||
| Decrease home page load time by 25% | | Decrease home page load time by 25% | ||
| | | <span style="color:green;font-weight:bold;">1</span> | ||
| | | Beat this by a decent amount, but need to get the actual numbers from Donovan. | ||
|- | |- | ||
| KR5 | | KR5 | ||
| Send a notification through the Test Pilot add-on using a WebExtension experiment API | | Send a notification through the Test Pilot add-on using a WebExtension experiment API | ||
| | | <span style="color:green;font-weight:bold;">1</span> | ||
| | | This code exists and works, but didn't ship to production yet. | ||
|- | |- | ||
| | | | ||
| Line 110: | Line 105: | ||
! Objective 4 | ! Objective 4 | ||
! Find, guide, and graduate ideas and experiments into features so that Test Pilot is a proven valuable input to the future direction of Firefox | ! Find, guide, and graduate ideas and experiments into features so that Test Pilot is a proven valuable input to the future direction of Firefox | ||
! | ! <span style="color:red;font-weight:bold;">.17</span> | ||
! | ! | ||
|- | |- | ||
| KR1 | | KR1 | ||
| Publish conclusions from three A/B tests for Firefox Send | | Publish conclusions from three A/B tests for Firefox Send | ||
| | | <span style="color:red;font-weight:bold;">.33</span> | ||
| | | We shipped one A/B test and [https://medium.com/firefox-test-pilot/kicking-the-tires-on-a-b-testing-bc52939dec52 published the results]. | ||
|- | |- | ||
| KR2 | | KR2 | ||
| Create a document with a plan to ship a partner-repack experiment | | Create a document with a plan to ship a partner-repack experiment | ||
| | | <span style="color:red;font-weight:bold;">0</span> | ||
| | | | ||
|} | |} | ||
Revision as of 21:00, 3 October 2017
Test Pilot 2017 Q3 Objectives and Key Results
| Final Status (Sep 30) | Comments | ||
|---|---|---|---|
| Objective 1 | Improve storytelling so that we support Firefox and grow community | .48 | |
| KR1 | Bi-weekly updates for each new experiment published on the blog or experiment pages. | .66 | We didn't publish updates at this cadence for every experiment, but we did publish a lot of things, including specific posts on the experiments. We decided this was about 2/3rds the spirit of this KR. |
| KR2 | Get 1000 combined reads on all blog posts | 1 | So many reads. Over 6000 near the end of September. |
| KR3 | Have 15 people participate in user research on design prototypes | .27 | Only 4 participants in the Notes optimization study |
| KR4 | 30 attendees at a Brown Bag about storyboarding | 0 | This was planned for Taipei but that trip was cancelled. We've made a new plan to host a meeting in Austin with the ET team. |
| Objective 2 | Grow Screenshots usage and expand offered features so that people find Screenshots to be a useful tool. | ?? | |
| KR1 | Ship screenshots to 100% of release users | 1 | 🎉 Shipping by default in 56 |
| KR2 | 10% of Screenshots users sign in to FxA | 0 | A patch to integrate FxA exists, but we didn't ship it to production. |
| KR3 | 5% of non-owner, non-Firefox visitors click a branded CTA | ?? | Need to get this number from John |
| KR4 | All pages and API endpoints load in under 2s (minus network time) | .9 | All the endpoints are well under 2s now, except we found the "delete all data" endpoint near the end and it's still pretty slow. |
| Objective 3 | Improve Test Pilot so that the Test Pilot program is a pleasure to use, conforms to the Firefox 57 plan, and improves engagement with experiments. | 1 | |
| KR1 | Ship SDK-free Test Pilot to 100% of users | 1 | |
| KR2 | Use storybook for all newly implemented feature cards | 1 | |
| KR3 | Implement 4 Test Pilot feature cards | 1 | 2 shipped, 2 waiting on L10n freeze and will ship next sprint |
| KR4 | Decrease home page load time by 25% | 1 | Beat this by a decent amount, but need to get the actual numbers from Donovan. |
| KR5 | Send a notification through the Test Pilot add-on using a WebExtension experiment API | 1 | This code exists and works, but didn't ship to production yet. |
| Objective 4 | Find, guide, and graduate ideas and experiments into features so that Test Pilot is a proven valuable input to the future direction of Firefox | .17 | |
| KR1 | Publish conclusions from three A/B tests for Firefox Send | .33 | We shipped one A/B test and published the results. |
| KR2 | Create a document with a plan to ship a partner-repack experiment | 0 |