Confirmed users
174
edits
Test Pilot/Metrics: Difference between revisions
m
no edit summary
(added data) |
mNo edit summary |
||
| Line 1: | Line 1: | ||
Test Pilot is an opt-in platform that allows us to perform controlled tests of new high-visibility product concepts in the general release channel of Firefox. It is not intended to replace trains for most features, nor is it a test bed for concepts we do not believe have a strong chance of shipping in general release. Rather, it is reserved for features that require user-feedback, testing, and tuning before they ship with the browser. | |||
=Metrics for individual tests= | =Metrics for individual tests= | ||
| Line 10: | Line 12: | ||
== Net Promoter Score == | === Net Promoter Score === | ||
'''Answers the Question:''' Is this the quality of experience users are looking for? | '''Answers the Question:''' Is this the quality of experience users are looking for? | ||
| Line 16: | Line 18: | ||
'''How we measure it:''' NPS is measured using surveys managed by the Test Pilot add-on that will fire at intervals over the life of your test. | '''How we measure it:''' NPS is measured using surveys managed by the Test Pilot add-on that will fire at intervals over the life of your test. | ||
== Usage Rate per User == | === Usage Rate per User === | ||
'''Answers the Question:''' Is this type of experience people are looking for? | '''Answers the Question:''' Is this type of experience people are looking for? | ||
| Line 40: | Line 42: | ||
|} | |} | ||
== Engagement == | === Engagement === | ||
'''Answers the Question:''' Do people believe this is the kind of thing they want? | '''Answers the Question:''' Do people believe this is the kind of thing they want? | ||
| Line 52: | Line 54: | ||
'''Note:''' Some tests may begin with set installation caps. In this case, engagement isn’t really a useful metric, since you will be testing on a fixed-size cohort. | '''Note:''' Some tests may begin with set installation caps. In this case, engagement isn’t really a useful metric, since you will be testing on a fixed-size cohort. | ||
== Usage Retention & Churn & Installation Retention & Churn == | === Usage Retention & Churn & Installation Retention & Churn === | ||
'''Answers the question:''' Is this idea sticky? Does it work for users? | '''Answers the question:''' Is this idea sticky? Does it work for users? | ||
| Line 67: | Line 69: | ||
Churn on installation doesn’t say much about the overall utility of the idea being tested, but it does indicate that the UX is deficient, annoying, or blocking on core browsing tasks. | Churn on installation doesn’t say much about the overall utility of the idea being tested, but it does indicate that the UX is deficient, annoying, or blocking on core browsing tasks. | ||
== Time to Churn == | === Time to Churn === | ||
'''How we measure it:''' If users leave your test, how long does it take them? | '''How we measure it:''' If users leave your test, how long does it take them? | ||
| Line 76: | Line 78: | ||
Your product is likely failing at some fundamental level. Is it taxing browser performance or flagrantly getting in the way of users’ normal workflows? Dive into data to find out. | Your product is likely failing at some fundamental level. Is it taxing browser performance or flagrantly getting in the way of users’ normal workflows? Dive into data to find out. | ||
== Churn Rationale Percentages == | === Churn Rationale Percentages === | ||
| Line 85: | Line 87: | ||
'''What to do with this data:''' Use it to help you make informed decisions about high-impact changes to make to your test. | '''What to do with this data:''' Use it to help you make informed decisions about high-impact changes to make to your test. | ||
== Bounce rate from Idea Details page == | === Bounce rate from Idea Details page === | ||
'''Answers the question:''' Is the description you give of your test enticing to users? Do users identify with your test. | '''Answers the question:''' Is the description you give of your test enticing to users? Do users identify with your test. | ||
| Line 92: | Line 94: | ||
A users bounces of they come to your test details page without the test installed and then leave the page without installing your test. | A users bounces of they come to your test details page without the test installed and then leave the page without installing your test. | ||
== Adoption Curve == | === Adoption Curve === | ||
'''Answers the question:''' Do specific events in Test Pilot (promotions etc) or changes to the packaging of my Add-on move the needle? This metric is useful to the Test Pilot team to help us understand how to better draw users to your idea (and to future ideas). | '''Answers the question:''' Do specific events in Test Pilot (promotions etc) or changes to the packaging of my Add-on move the needle? This metric is useful to the Test Pilot team to help us understand how to better draw users to your idea (and to future ideas). | ||
| Line 98: | Line 100: | ||
=General for Test Pilot= | =General for Test Pilot= | ||
== Core Metrics == | == Core Metrics == | ||
== % of Users Engaged in 0,1,2,3...n tests over time == | === % of Users Engaged in 0,1,2,3...n tests over time === | ||
'''Answers the question:''' Does Test Pilot promote engagement? Are the current batch of ideas useful overall? Are some ideas more useful than others? | '''Answers the question:''' Does Test Pilot promote engagement? Are the current batch of ideas useful overall? Are some ideas more useful than others? | ||
== Average NPS response score across active tests == | === Average NPS response score across active tests === | ||
'''Answers the question:''' Does Test Pilot make Firefox a more recommendable product? Are the current batch of ideas attractive overall? | '''Answers the question:''' Does Test Pilot make Firefox a more recommendable product? Are the current batch of ideas attractive overall? | ||
== Secondary Metrics == | == Secondary Metrics == | ||
== Average Idea Duration == | === Average Idea Duration === | ||
'''Answers the question:''' How long should devs plan to be in Test Pilot? | '''Answers the question:''' How long should devs plan to be in Test Pilot? | ||
== Overall NPS response rate across active tests == | === Overall NPS response rate across active tests === | ||
'''Answers the question:''' Do the current batch of tests promote engagement? | '''Answers the question:''' Do the current batch of tests promote engagement? | ||
== Average Engaged Retention & Churn == | === Average Engaged Retention & Churn === | ||
'''Answers the question:''' Are current tests working for users? | '''Answers the question:''' Are current tests working for users? | ||
''' | ''' | ||
How we measure it:''' This is aggregate churn data of all active tests. | How we measure it:''' This is aggregate churn data of all active tests. | ||
== Retention & Churn == | === Retention & Churn === | ||
'''Answers the question:''' Is Test Pilot working for users? Is the service itself healthy. | '''Answers the question:''' Is Test Pilot working for users? Is the service itself healthy. | ||
'''How we measure it''': We define a churned user in this case as a user who has uninstalled Test Pilot or who has not interacted with Test Pilot UI in one month. | '''How we measure it''': We define a churned user in this case as a user who has uninstalled Test Pilot or who has not interacted with Test Pilot UI in one month. | ||
== Bounce Rate through Sign Up and Install == | === Bounce Rate through Sign Up and Install === | ||
'''Answers the question:''' Is the Test Pilot flow effective | '''Answers the question:''' Is the Test Pilot flow effective | ||
== MAUs v. Total Sign Ups == | === MAUs v. Total Sign Ups === | ||
This may be a vanity metric, but it will help give us a sense of the scale of our population and give us insight into the statistical significance of our data. | This may be a vanity metric, but it will help give us a sense of the scale of our population and give us insight into the statistical significance of our data. | ||
== Average number of ideas installed == | === Average number of ideas installed === | ||
Detail concerning engagement. May not have a clear value from launch, except that it might help us define user typologies down the road, and gives a bit more insight into the ways in which users are making use of Test Pilot. | Detail concerning engagement. May not have a clear value from launch, except that it might help us define user typologies down the road, and gives a bit more insight into the ways in which users are making use of Test Pilot. | ||