Foundation/Metrics/Contributor Dashboard

View the Status Tracking Working Document to follow progress on this project. This planning document will not be updated regularly, though it's a useful record.

The ‘Contributors Dashboard’ is the Mozilla Foundation’s top metrics priority for the start of 2014.

This is the first published version of this plan, so please feedback and challenge what's here: adam AT mozillafoundation DOT org

What's a contributor dashboard?

• A single screen of information showing us how many people are actively contributing to Mozilla Foundation projects over time.
• This will be a publicly accessible document.

What is a contributor?

For anyone reading this document who is not embedded Mozilla terminology, ‘Contributor’ is the name given to people who take significant actions to support our projects. These are the people who make the Mozilla community what it is.

Contribution is not a single thing, and there are infinite levels of possible ways to contribute to Mozilla, but for the sake of metrics, we’re going to count the number of people who take one of an agreed list of actions as Contributors.

In metrics terms, we are treating contribution as a threshold people can pass, though in practice we know it is much more complex than this.

We believe that counting the number of people to cross this threshold is the most useful metric for us to work with to focus our efforts to build the community.

For some background on the label ‘Contributor’ and other Mozilla terminology see these blog posts:

• http://davidwboswell.wordpress.com/2011/06/07/who-is-in-the-mozilla-community/
• http://davidwboswell.wordpress.com/2013/11/06/what-does-mozillian-mean/

For the Mozilla Foundation dashboard that this plan relates too, our Contributor criteria is aligned with the Active Contributor and Core Contributor levels of engagement in David Boswell's blog posts. We are not sub-dividing types of Contributor (at least not for now).

See the "sniff-test" later in this document for specific criteria.

Why do we need a contributor dashboard?

We will measure, test and optimize all sorts of metrics across our products and programs in order to improve our impact, but the agreed number that matters above all others for the Mozilla Foundation is ‘Number of Active Contributors’.

• Our key assumption is that the number of active contributors will only grow if we are ‘doing things right’ across the board.

This is a proxy KPI to measure the combined effectiveness of all the different things we work on. Our products, our processes, our story, our mission and our community building all have to align for this number to go up.

It will continually remind us to ask the question:

• Are we working in a way that best enables the community?

The Mozilla Foundation’s specific dashboard needs

• The Mozilla Foundation has a specific goal for 2014 to engage 10,000 Contributors.
• There are also targets within individual teams to break this target down further.

We need:

• to see how well we are doing against this target
• to see this early enough in the year to spot trends and make changes
• to know which of our activities are most effective at recruiting new Contributors

Making this information visible can help us make more effective decisions.

What is the end goal?

We want a dashboard showing the number of unique people who contribute in any way across any of our products or programmes:

• On or off-line;
• De-duped;
• Sharing a single data source with the Mozilla Corporation contribution metrics;
• That includes every level of foundation contribution data in one place;
• Is effectively real-time (e.g. 95% of the data is accurate to the nearest hour, remaining 5% to the nearest day);
• And (of course) is fully compliant with our privacy policy.

What are the complications?

• Joining up lots of data is a complex process
• There are several projects already working on this, and we don’t want to start another from scratch
• The Foundation want to start using this dashboard by the end of March

How are we going to tackle this?

PLAN A: Find the ‘single source of truth’

• Work with MoCo teams who are tackling the same problem to find a single place to join up this data. The most likely candidates at this time are:
  • Mozilla Contributor Badges
    • If all contribution activity is acknowledged with a badge, we potentially have a central place to count contribution activity
  • Project Baloo
    • is a collaborative effort between the Data and BI Services and Community Building team to create a contribution tracking system for Mozilla: https://wiki.mozilla.org/Baloo

We will actively support both of these projects, and anticipate that further down the line that one of these projects, or something similar will form the ‘single source of truth’ database that will power our Contributor Dashboard.

PLAN B: Interim dashboard

Rather than wait for the perfect end-goal datastore (plan A), we have started hooking together things that will give us insights and trends about Contribution numbers in the meantime.

These components will be built with the intention of re-use with Plan A.

Interim dashboard components

1. Front-end(s)

STATUS: (Front-end V1) is ready.

• Usable for all MoFo and also at MoFo team level dashboards
• Could be re-used with Plan A
• Source code is here: https://github.com/adamlofting/mofo-contributors-dashboard

2. API Spec for buckets of data

STATUS: (API Spec V1) is ready

• Responsibility for extracting contributor counts from systems will sit with each of the teams in the Foundation
• Adam to work on datastores shared between teams (github, transifex, etc)
• We are working through these in priority of where the biggest number of contributors sit
• These queries will extract contributor counts for now, but should be re-usable later to extract raw contribution data and to feed into Plan A

3. Datastore for aggregate numbers

STATUS: (Aggredash V1) is ready, and feeding the dashboard front-end

• This is the back-end to the interim dashboard
• It joins up the counts from our existing data sources
• Stores aggregate numbers, not raw contribution activity
• No re-use value for Plan A, but key to making the interim dashboard work
• The long game is to replace this with Plan A
• Source code is here: https://github.com/adamlofting/aggredash

4. Datastore for ad-hoc activities

STATUS: Built but needs security review before use (Preview V1)

• Across the MoFo teams there is ad-hoc contribution activity we want to count that isn’t currently logged somewhere
• Many of these numbers are small (5 people here, 10 people there, etc)
• This will be simple single cross-team place to log these that can feed into the dashboard numbers

5. Extract and reformat Github data

STATUS: (Gitribution V1) is ready

• Source code: https://github.com/adamlofting/gitribution

MoFo contribution criteria

What is or isn’t counted as contribution activity will be decided individually by each team. UPDATE: here is the list of contributor actions from each team.

But, in order to normalise this across MoFo so that our dashboard is meaningful, we have defined this generic set of guidelines. These are not hard-rules, but form a good ‘sniff-test’ as to whether an activity should be counted:

Sniff test

The activity is:

• Knowingly furthering the work of MoFo, or one of our programs
• Creating rather than consuming
• Doing something to contribute to Mozilla, rather than saying 'I support Mozilla'
• ‘Badge worthy’
  • If it’s not worth 'badging', was it a meaningful contribution?
  • That’s not to say we can’t count things until we badge them, but ask ourselves if it’s ‘badge worthy’
• Bot-proof / impossible to spam
  • This is where moderation / human approval of content might add a quality filter to some activity (especially around content creation goals)
• Involves interaction with other members of the community
  • This includes talking to staff to get code reviewed
• (Optional) at least 15 minutes of effort
  • Make a judgement call on this, but try to filter out very quick casual support (e.g. a retweet is helpful, but it's not on a par with the contributions we are counting here).

These guidelines can be broken, if we think it's justified. But if you find yourself breaking these often, lets revisit them and challenge the assumptions.

Climbing the engagement ladder

The threshold to becoming a contributor is the specific set of actions that push our work forward. We can separately count the steps that lead up to contribution, but this is a supporting metric, and not part of this dashboard plan. For example, forking a repository is an indication that someone is considering contribution, but submitting a pull-request is the point when they contribute.

Other definitions of contributor?

There is a bigger piece of work happening across Mozilla to define contribution engagement ladders for all our activity in a normalised way. The ‘sniff-test’ here is not to replace that bigger piece of work, but is our quick means of sense checking the data going into our immediate solution for building a dashboard.

The contributor conversion points project can be found here: https://wiki.mozilla.org/Contribute/Conversion_points