MozillaWiki

Foundation/Metrics/Contributor Dashboard: Difference between revisions

Page Discussion

Foundation/Metrics/Contributor Dashboard (view source)

Revision as of 09:19, 15 September 2014

912 bytes added ,  15 September 2014
→‎Timetable
 
(43 intermediate revisions by 2 users not shown)
Line 1: Line 1:
The ‘Contributors Dashboard’ is the Mozilla Foundation’s top metrics priority for the start of 2014. This page documents our plan.
{{admon/note|View the [[Foundation/Metrics/Contributor_Dashboard/Status | Status Tracking Working Document]] to follow progress on this project. This planning document will not be updated regularly, though it's a useful record. }}


== What is a contributor dashboard? ==
The ‘Contributors Dashboard’ is the Mozilla Foundation’s top metrics priority for the start of 2014.
* This will be a single screen of information showing us how many people are actively contributing to Mozilla Foundation projects.
 
* There will be numbers, and a graph, and it should be easy for anyone to read at a glance.  
This is the first published version of this plan, so please feedback and challenge what's here: adam AT mozillafoundation DOT org
 
== What's a contributor dashboard? ==
* A single screen of information showing us how many people are actively contributing to Mozilla Foundation projects over time.
* This will be a publicly accessible document.
* This will be a publicly accessible document.


== What is a contributor? ==
== What is a contributor? ==
For anyone reading this document who is not as close to the Mozilla terminology, ‘Contributor’ is the name given to people who go ‘above and beyond’ in the actions they take to support Mozilla. These are the people who make the Mozilla community what it is. Contribution is not a single thing, and there are infinite levels of possible ways to contribute to Mozilla, but for the sake of metrics, we’re going to count the number of people who take one of an agreed list of actions. In metrics terms, we are treating contribution as a threshold people can pass, though in practice we know it is much more complex than this.
For anyone reading this document who is not embedded Mozilla terminology, ‘Contributor’ is the name given to people who take significant actions to support our projects. These are the people who make the Mozilla community what it is.  
 
Contribution is not a single thing, and there are infinite levels of possible ways to contribute to Mozilla, but for the sake of metrics, we’re going to count the number of people who take one of an agreed list of actions as Contributors.  
 
In metrics terms, we are treating contribution as a threshold people can pass, though in practice we know it is much more complex than this.
 
We believe that counting the number of people to cross this threshold is the most useful metric for us to work with to focus our efforts to build the community.


For some background on the label ‘Contributor’ and other Mozilla terminology see these blog posts:  
For some background on the label ‘Contributor’ and other Mozilla terminology see these blog posts:  
http://davidwboswell.wordpress.com/2011/06/07/who-is-in-the-mozilla-community/
* http://davidwboswell.wordpress.com/2011/06/07/who-is-in-the-mozilla-community/
http://davidwboswell.wordpress.com/2013/11/06/what-does-mozillian-mean/
* http://davidwboswell.wordpress.com/2013/11/06/what-does-mozillian-mean/


== Why do we need a contributors dashboard? ==
For the Mozilla Foundation dashboard that this plan relates too, our Contributor criteria is aligned with the Active Contributor and Core Contributor levels of engagement in David Boswell's blog posts. We are not sub-dividing types of Contributor (at least not for now).
We will measure, test and optimise all sorts of metrics in order to understand and improve the impact of our actions, but the agreed number that matters above all others for the Mozilla Foundation is ‘Number of Active Contributors’.


Our key assumption is that the number of active contributors will only grow if we are ‘doing things right’ across the board.
See the "sniff-test" later in this document for specific criteria.


This is a powerful proxy KPI to measure the combined effectiveness of all the different things we work on, and how well we collaborate and embrace the wider Mozillian community.
== Why do we need a contributor dashboard? ==
We will measure, test and optimize all sorts of metrics across our products and programs in order to improve our impact, but the agreed number that matters above all others for the Mozilla Foundation is ‘Number of Active Contributors’.
 
* Our key assumption is that the number of active contributors will only grow if we are ‘doing things right’ across the board.
 
This is a proxy KPI to measure the combined effectiveness of all the different things we work on. Our products, our processes, our story, our mission and our community building all have to align for this number to go up.
 
It will continually remind us to ask the question:
* Are we working in a way that best enables the community?


=== The Mozilla Foundation’s specific dashboard needs ===
=== The Mozilla Foundation’s specific dashboard needs ===
The Mozilla Foundation has a specific goal for 2014 to ship* 10,000 Contributors. There are also targets within individual teams to break this target down further. We need a means of measuring how well we are doing against this target, and we need this information early in the year so we can see if we are trending towards the target, and which of our activities are most effective at recruiting Contributors. Joining up our data and making this visible across the Foundation will help us to make better decisions.


* “Ship” is useful internal parlance to focus our activities, but “work with”/“recruit”/“engage”/“enable” are a reasonable fit to explain what we are doing. I’m cautious with the language around this which can sometimes make our community sound more passive than they are.  
* The Mozilla Foundation has a specific goal for 2014 to engage 10,000 Contributors.
* There are also targets within individual teams to break this target down further.
 
We need:
* to see how well we are doing against this target
* to see this early enough in the year to spot trends and make changes
* to know which of our activities are most effective at recruiting new Contributors
 
Making this information visible can help us make more effective decisions.


== What is the end goal? ==
== What is the end goal? ==
We want a dashboard showing the number of unique people who contribute in any way across any of our products or programmes:
We want a dashboard showing the number of unique people who contribute in any way across any of our products or programmes:
* On or off-line
* On or off-line;
* De-duped
* De-duped;
* Sharing a single data source with the Mozilla Corporation contribution metrics
* Sharing a single data source with the Mozilla Corporation contribution metrics;
* That includes every level of foundation contribution data in one place
* That includes every level of foundation contribution data in one place;
* And is effectively real-time (e.g. 95% of the data is accurate to the nearest hour)
* Is effectively real-time (e.g. 95% of the data is accurate to the nearest hour, remaining 5% to the nearest day);
* And of course is fully compliant with our privacy policy
* And (of course) is fully compliant with our privacy policy.


== What are the complications? ==
== What are the complications? ==
Line 41: Line 64:
== How are we going to tackle this? ==
== How are we going to tackle this? ==


=== 1. Find the ‘single source of truth’ ===
=== PLAN A: Find the ‘single source of truth’ ===
* Work with MoCo teams who are tackling the same problem to find a single place to join up this data. The most likely candidates at this time are:
* Work with MoCo teams who are tackling the same problem to find a single place to join up this data. The most likely candidates at this time are:
*# Mozilla Contributor Badges
** Mozilla Contributor Badges
*** If all contribution activity is acknowledged with a badge, we potentially have a central place to count contribution activity
*** If all contribution activity is acknowledged with a badge, we potentially have a central place to count contribution activity
*# Blackhole (Final name TBC) & MoCo BI Data Warehouse
** Project Baloo
*** These existing projects have already invested time in tackling the data problem and are merging in some form. Further info here: https://wiki.mozilla.org/Contribute/Systems
*** is a collaborative effort between the Data and BI Services and Community Building team to create a contribution tracking system for Mozilla: https://wiki.mozilla.org/Baloo


We will actively support both of these projects, and anticipate that further down the line that one of these projects, or something similar will form the ‘single source of truth’ database that will power our Contributor Dashboard.  
We will actively support both of these projects, and anticipate that further down the line that one of these projects, or something similar will form the ‘single source of truth’ database that will power our Contributor Dashboard.  


However, we are not going to wait for the perfect end-goal datastore before building out the other components that will be needed for our dashboard. We are going to start work right away on the front-end of our dashboard, and also the queries that will extract contribution activity from our existing scattered databases. These components will be built with the intention of re-use with the end-goal datastore, but can also be wired up now to form an interim solution that will give the Foundation partial visibility of Contributor numbers earlier in the year.
=== PLAN B: Interim dashboard ===


=== Components to build now ===
Rather than wait for the perfect end-goal datastore (plan A), we have started hooking together things that will give us insights and trends about Contribution numbers in the meantime.


==== 2. Front-end(s) ====
These components will be built with the intention of re-use with Plan A.
* Usable for cross mofo and at mofo team level
 
* Single HTML page for screens (also suitable for display on screens in offices)
=== Interim dashboard components ===
* Responsive for mobile
 
* D3 for graphs
==== 1. Front-end(s) ====
* A route for external viewers to find out what we mean by contributor and the story behind the dashboard
STATUS: ([http://adamlofting.github.io/mofo-contributors-dashboard/ Front-end V1]) is ready.
* Will be used with the final datastore
 
* Adam to design and build these
* Usable for all MoFo and also at MoFo team level dashboards
* Could be re-used with Plan A
* Source code is here: https://github.com/adamlofting/mofo-contributors-dashboard
 
==== 2. API Spec for buckets of data ====
STATUS: ([https://docs.google.com/a/mozillafoundation.org/document/d/16Sas-dbBzSftWqacYhFRojjXLCkAXu6h2XxbkfALG-Q/edit# API Spec V1]) is ready


==== 3. Data queries ====
* Responsibility for extracting contributor counts from systems will sit with each of the teams in the Foundation
* Responsibility for extracting contributor counts from systems will sit with each of the teams in the Foundation
** Cross-team collaboration will save time where there are shared datastores
* Adam to work on datastores shared between teams (github, transifex, etc)
* We can work through these in priority of where the biggest number of contributors sit
* We are working through these in priority of where the biggest number of contributors sit
* These queries will extract contributor counts for now, but should be re-usable later to extract raw contribution data and to feed into the ‘single source of truth’ datastore
* These queries will extract contributor counts for now, but should be re-usable later to extract raw contribution data and to feed into Plan A
* API spec for this to follow next


==== 4. Datastore for aggregate numbers ====
==== 3. Datastore for aggregate numbers ====
* A place to join up the counts from our existing data sources
STATUS: ([http://aggredash.herokuapp.com/api/mofo/2014 Aggredash V1]) is ready, and feeding the dashboard front-end
 
* This is the back-end to the interim dashboard
* It joins up the counts from our existing data sources
* Stores aggregate numbers, not raw contribution activity
* Stores aggregate numbers, not raw contribution activity
* This will be replaced by the ‘single source of truth’ datastore in the long run
* No re-use value for Plan A, but key to making the interim dashboard work
* Expose the numbers via an API to use with the Front-end, and to enable others to build on top of this
* The long game is to replace this with Plan A
 
* Source code is here: https://github.com/adamlofting/aggredash
==== 5. Datastore for ad-hoc activities (TBC) ====
* I suspect across the teams there may be ad-hoc contribution activity we want to count that isn’t currently logged somewhere
* If this is true, and these numbers are small, we can build a simple single place to log these cross-team that can feed into the dashboard numbers
* Imagining something at the “log these in a spreadsheet” level of input (but addressing our PII requirements)
* Adam to identify if this is needed, and implement
 
== Querying our data ==


We want to report on the Number of Active Contributors over time, along with Number of New Contributors in a given time.
==== 4. Datastore for ad-hoc activities ====
STATUS: Built but needs security review before use ([http://adhoctribution.herokuapp.com Preview V1])


The query will be:
* Across the MoFo teams there is ad-hoc contribution activity we want to count that isn’t currently logged somewhere
* Many of these numbers are small (5 people here, 10 people there, etc)
* This will be simple single cross-team place to log these that can feed into the dashboard numbers


* For date X, give me:
==== 5. Extract and reformat Github data ====
** The number of unique contributors in the past 12 months
STATUS: ([http://gitribution.herokuapp.com/api/?date=2013-7-20&team=webmaker Gitribution V1]) is ready
*** The number of those who contributed for the first time in the last 7 days


These will be logged in the aggregate datastore against:  
* Source code: https://github.com/adamlofting/gitribution
* a mofo team name (e.g. “OpenNews”)
* an activity description (e.g. “Hosted a maker party event”)
* an activity bucket (e.g. “Translation” or “Code”)
** we will use these to group up similar activity types for use in the dashboard


== MoFo contribution criteria ==
== MoFo contribution criteria ==
What is or isn’t counted as contribution activity will be decided individually by each team.  
What is or isn’t counted as contribution activity will be decided individually by each team. UPDATE: here is the [https://wiki.mozilla.org/Foundation/Metrics/Contributor_Dashboard/Who list of contributor actions from each team].


But, in order to normalise this across MoFo so that our dashboard is meaningful, we have defined this generic set of guidelines. These are not hard-rules, but form a good ‘sniff-test’ as to whether an activity should be counted:
But, in order to normalise this across MoFo so that our dashboard is meaningful, we have defined this generic set of guidelines. These are not hard-rules, but form a good ‘sniff-test’ as to whether an activity should be counted:


=== Sniff test ===
=== Sniff test ===
* Creating rather than consuming
The activity is:
* Doing rather than talking about doing
 
* At least 15 minutes of effort
* '''Knowingly furthering the work of MoFo''', or one of our programs
** Clicking ‘like’ or ‘share’ is support rather than contribution
* '''Creating''' rather than consuming
* ‘Badge worthy’
* '''Doing''' something to contribute to Mozilla, rather than saying 'I support Mozilla'
* ‘'''Badge worthy'''’
** If it’s not worth 'badging', was it a meaningful contribution?
** If it’s not worth 'badging', was it a meaningful contribution?
** That’s not to say we can’t count things until we badge them, but ask if it’s ‘badge worthy’
** That’s not to say we can’t count things until we badge them, but ask ourselves if it’s ‘badge worthy’
* Bot-proof / impossible to spam
* '''Bot-proof''' / impossible to spam
** This is where moderation / human approval of content might add a quality filter to some activity (especially around content creation)
** This is where moderation / human approval of content might add a quality filter to some activity (especially around content creation goals)
* Involves interaction with other members of the community  
* Involves '''interaction with other members of the community'''
** This includes talking to staff to get code reviewed
** This includes talking to staff to get code reviewed
* (Optional) at least '''15 minutes of effort'''
** Make a judgement call on this, but try to filter out very quick casual support (e.g. a retweet is helpful, but it's not on a par with the contributions we are counting here).


If you find yourself breaking these, lets revisit them and challenge the assumptions.
These guidelines can be broken, if we think it's justified. But if you find yourself breaking these often, lets revisit them and challenge the assumptions.
 
==== Climbing the engagement ladder ====
 
The threshold to becoming a contributor is the specific set of actions that push our work forward. We can separately count the steps that lead up to contribution, but this is a supporting metric, and not part of this dashboard plan. For example, forking a repository is an indication that someone is considering contribution, but submitting a pull-request is the point when they contribute.


=== Other definitions of contributor? ===
=== Other definitions of contributor? ===
There is a bigger piece of work happening across Mozilla to define contribution engagement ladders for all our activity in a normalised way. The ‘sniff-test’ here is not to replace that bigger piece of work, but is our quick means of sense checking the data going into our quickest solution to building a dashboard. The contributor conversion points project can be found here: https://wiki.mozilla.org/Contribute/Conversion_points
There is a bigger piece of work happening across Mozilla to define contribution engagement ladders for all our activity in a normalised way. The ‘sniff-test’ here is not to replace that bigger piece of work, but is our quick means of sense checking the data going into our immediate solution for building a dashboard.  
 
The contributor conversion points project can be found here: https://wiki.mozilla.org/Contribute/Conversion_points
Adamlofting
Confirmed users
402

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/927543...1014755"