Test Pilot/2017Q1

Test Pilot 2017 Q1 Objectives and Key Results

		Final Status (Mar 31)	Comments
Objective 1	Demonstrate Test Pilot experiments can result in changes to Firefox	0.5
KR1	Two former Test Pilot experiments in active SHIELD studies	0.5	This was supposed to be Activity Stream and Tab Center. Tab Center hit a snag landing the system add-on and is delayed until early/mid Q2. In retrospect, we shouldn't have had this on our goal list since we don't own it.
Objective 2	Be company role models for community involvement	0.59
KR1	Test 3 contributor value hypotheses and publish the results	0.3	Lots of planning towards this. Three finalists: Voting up/down possible new experiments to help us choose and help people feel more involved Writing up ideas for other users on how they are using experiments (e.g. snooze tabs seems to be something that people could develop productivity tips around) Do a better job telling the story of Test Pilot. There's something here about content that TP is generating in terms of an approach to ideation and experimenting
KR2	Publish conclusions from three experiments as they graduate from Test Pilot	0.67	No More 404s and Tracking Protection
KR3	Create a draft curriculum for an upcoming class with the Open Innovation team	0.8	Lots of planning with the Ravensbourne group. We have two Design Prompts written which will be the foundation of the course.
Objective 3	Grow Test Pilot's user base and capabilities	0.45
KR1	At the end of Q1 double the percentage of new sessions enabling an experiment on intial Test Pilot install	0.61	Q4 2016 was 6.6% . source; page 3, "% of New Sessions Enabling an Exp". There is a new dashboard which puts this number at 10.9%, with some qualifiers: If you look at the performance by traffic medium, you'll see 65% of our traffic was from snippets, but snippets have the lowest percentage of sessions enabling experiments (7.4%). That means any time we run a snippet campaign we tank this number, so basing it off of a percentage of new visitors was a bad idea and not a great OKR. The proper way to do this would have been an A/B test. .61 credit for the increase in the number, just realize that it's heavily weighted by other forces.
KR2	Improve Smoothed Test Pilot Engagement Ratio by 6%	0.17	Average of the entirety of Q4 is .53 so our target was .59. Average for Q1 turned out to be .54, up 1% and 1/6th of our goal. From: source; second graph down "TxP Active Users / TxP Smoothed Engagement Ratio". Click View Source, change the dates to >90 days to include the whole quarter, download the CSV, ensure a quarter's worth of data for TxP add-on, run Average() on the smoothed ratio.
KR3	Plan a mobile GTM for 2017	0.0	This deliverable is a document outlining the plan.
KR4	Create 4 feature cards ready for UX inspired by user testing reports	1.0	Cards on trello

Updates

• On google drive
• Mid-Quarter we got the request for Firefox Screenshots (formerly Page Shot) to be landed directly in Firefox for our June launch. This made some waves in our schedule and disrupted some OKRs. Overall, we still have a lot to be proud of that isn't represented above: First experiment landing in Firefox (Activity Stream landing soon after), first web extension as a system add-on landing in Firefox, first use of Google Analytics in Firefox, coordination for Screenshots between a dozen teams (and still looking like it's on time), 2/3rds of our successfully graduated experiments happened this quarter, the Test Pilot codebase continues to be simplified and improved (added Flow)