Foundation/Metrics/Contributor Dashboard

The ‘Contributors Dashboard’ is the Mozilla Foundation’s top metrics priority for the start of 2014.

This is the first published version of this plan, so please feedback and challenge what's here: adam AT mozillafoundation DOT org

What is a contributor dashboard?

• This will be a single screen of information showing us how many people are actively contributing to Mozilla Foundation projects.
• There will be numbers, and a graph, and it should be easy for anyone to read at a glance.
• This will be a publicly accessible document.

What is a contributor?

For anyone reading this document who is not as close to the Mozilla terminology, ‘Contributor’ is the name given to people who go ‘above and beyond’ in the actions they take to support Mozilla. These are the people who make the Mozilla community what it is. Contribution is not a single thing, and there are infinite levels of possible ways to contribute to Mozilla, but for the sake of metrics, we’re going to count the number of people who take one of an agreed list of actions.

In metrics terms, we are treating contribution as a threshold people can pass, though in practice we know it is much more complex than this.

For some background on the label ‘Contributor’ and other Mozilla terminology see these blog posts:

• http://davidwboswell.wordpress.com/2011/06/07/who-is-in-the-mozilla-community/
• http://davidwboswell.wordpress.com/2013/11/06/what-does-mozillian-mean/

Why do we need a contributors dashboard?

We will measure, test and optimise all sorts of metrics in order to understand and improve the impact of our actions, but the agreed number that matters above all others for the Mozilla Foundation is ‘Number of Active Contributors’.

• Our key assumption is that the number of active contributors will only grow if we are ‘doing things right’ across the board.

This is a powerful proxy KPI to measure the combined effectiveness of all the different things we work on, and how well we collaborate and embrace the wider Mozillian community.

It will continually remind us to ask the question:

• Are we working in a way that best enables the community, and in turn supports the mission?

The Mozilla Foundation’s specific dashboard needs

The Mozilla Foundation has a specific goal for 2014 to ship* 10,000 Contributors. There are also targets within individual teams to break this target down further. We need a means of measuring how well we are doing against this target, and we need this information early in the year so we can see if we are trending towards the target, and which of our activities are most effective at recruiting Contributors. Joining up our data and making this visible across the Foundation will help us to make better decisions.

* “Ship” is useful internal parlance to focus our activities, but “work with”/“recruit”/“engage”/“enable” are a reasonable fit to explain what we are doing. I’m cautious with the language around this which can sometimes make our community sound more passive than they are.

What is the end goal?

We want a dashboard showing the number of unique people who contribute in any way across any of our products or programmes:

• On or off-line;
• De-duped;
• Sharing a single data source with the Mozilla Corporation contribution metrics;
• That includes every level of foundation contribution data in one place;
• Is effectively real-time (e.g. 95% of the data is accurate to the nearest hour, remaining 5% to the nearest day);
• And (of course) is fully compliant with our privacy policy.

What are the complications?

• Joining up lots of data is a complex process
• There are several projects already working on this, and we don’t want to start another from scratch
• The Foundation want to start using this dashboard by the end of March

How are we going to tackle this?

1. Find the ‘single source of truth’

• Work with MoCo teams who are tackling the same problem to find a single place to join up this data. The most likely candidates at this time are:
  • Mozilla Contributor Badges
    • If all contribution activity is acknowledged with a badge, we potentially have a central place to count contribution activity
  • Blackhole (final name TBC) & MoCo BI Data Warehouse
    • These existing projects have already invested time in tackling the data problem and are merging in some form. Further info here: https://wiki.mozilla.org/Contribute/Systems

We will actively support both of these projects, and anticipate that further down the line that one of these projects, or something similar will form the ‘single source of truth’ database that will power our Contributor Dashboard.

However, we are not going to wait for the perfect end-goal datastore before building out the other components that will be needed for our dashboard. We are going to start work right away on the front-end of our dashboard, and also the queries that will extract contribution activity from our existing scattered databases. These components will be built with the intention of re-use with the end-goal datastore, but can also be wired up now to form an interim solution that will give the Foundation partial visibility of Contributor numbers earlier in the year.

Components to build now

2. Front-end(s)

• Usable for cross mofo and at mofo team level dashboards
• Single HTML page for screens (also suitable for display on screens in offices)
• Responsive for mobile (though not single screen)
• D3 for graphs
• A route for external viewers to find out what we mean by contributor and the story behind the dashboard
• Will be used with the final datastore
• Adam to design and build these

3. Data queries

• Responsibility for extracting contributor counts from systems will sit with each of the teams in the Foundation
  • Cross-team collaboration will save time where there are shared datastores
• We can work through these in priority of where the biggest number of contributors sit
• These queries will extract contributor counts for now, but should be re-usable later to extract raw contribution data and to feed into the ‘single source of truth’ datastore
• API spec for this to follow next

4. Datastore for aggregate numbers

• This will be the short-term back-end to our dashboards
• A place to join up the counts from our existing data sources
• Stores aggregate numbers, not raw contribution activity
• This will be replaced by the ‘single source of truth’ datastore in the long run
• Expose the numbers via an API to use with the Front-end, and to enable others to build on top of this

5. Datastore for ad-hoc activities (TBC)

• I suspect across the teams there may be ad-hoc contribution activity we want to count that isn’t currently logged somewhere
• If this is true, and these numbers are small, we can build a simple single place to log these cross-team that can feed into the dashboard numbers
• I'm imagining something at the “log these in a spreadsheet” level of input (but addressing our PII requirements)
• Adam to identify if this is needed, and implement

Querying our data

We want to report on the Number of Active Contributors over time, along with Number of New Contributors in a given time.

The query will be:

• For date X, give me:
  • The number of unique contributors in the past 12 months
    • The number of those who contributed for the first time in the last 7 days

These will be logged in the aggregate datastore against:

• a mofo team name (e.g. “OpenNews”)
• an activity description (e.g. “Hosted a maker party event”)
• an activity bucket (e.g. “Translation” or “Code”)
  • we will use these to group up similar activity types for use in the dashboard

MoFo contribution criteria

What is or isn’t counted as contribution activity will be decided individually by each team.

But, in order to normalise this across MoFo so that our dashboard is meaningful, we have defined this generic set of guidelines. These are not hard-rules, but form a good ‘sniff-test’ as to whether an activity should be counted:

Sniff test

The activity is:

• Creating rather than consuming
• Doing rather than talking about doing
• ‘Badge worthy’
  • If it’s not worth 'badging', was it a meaningful contribution?
  • That’s not to say we can’t count things until we badge them, but ask ourselves if it’s ‘badge worthy’
• Bot-proof / impossible to spam
  • This is where moderation / human approval of content might add a quality filter to some activity (especially around content creation goals)
• Involves interaction with other members of the community
  • This includes talking to staff to get code reviewed
• (Optional) at least 15 minutes of effort
  • Make a judgement call on this, but try to filter out very quick casual support (e.g. a retweet is helpful, but it's not on a par with the contributions we are counting here).

These guidelines can be broken, if we think it's justified. But if you find yourself breaking these often, lets revisit them and challenge the assumptions.

The threshold to becoming a contributor is the specific set of actions that push our work forward. We can separately count the steps that lead up to contribution, but this is a supporting metric, and not part of this dashboard plan. For example, forking a repository is an indication that someone is considering contribution, but submitting a pull-request is the point when they contribute.

Other definitions of contributor?

There is a bigger piece of work happening across Mozilla to define contribution engagement ladders for all our activity in a normalised way. The ‘sniff-test’ here is not to replace that bigger piece of work, but is our quick means of sense checking the data going into our immediate solution for building a dashboard.

The contributor conversion points project can be found here: https://wiki.mozilla.org/Contribute/Conversion_points

Timetable

End of Feb

• Working front end demo with dummy data
• Ready for Mar board meeting

End of Mar

• Significant chunk of contributor data feeding live into the dashboard
• This will not be de-deduped
• Potential margin-of-error to be clearly displayed on dashboard