Baloo

From MozillaWiki
Jump to: navigation, search

Intro

Project Baloo, is a collaborative effort between the Data and BI Services and Community Building team to create a contribution tracking system for Mozilla.


Roadmap

File a Baloo bug

Integration Status

System Area Priority Schema Update Conversion points Import of data Building Aggregations On 10x dashboard
Support.mozilla.org SUMO P1 Yes 980385 1010197 Done Done
bugzilla.mozilla.org QA P1 990667 1019689 1010205 Done Done
Bugzilla and Github Coding P1 1010187 1010190 1010209 Done 1025952
reps.mozilla.org Mozilla Reps P1 1010180 Done 1010183 1010207 Done
developer.mozilla.org MDN P1 see Q2 above 1050315 see Q2 above see Q2 above see Q2 above
addons.mozilla.org AMMO P1 1050317
marketplace.mozilla.org AMMO P1 1050317
github.com WebDev P2 Done Done Done (MoFo DB) 1056771 Done
location.services.mozilla.com Location Services P1 1073054
webcompat.com Web Compatibility P2 1050320
moztrap.mozilla.org QA P2 1042101 Done 1079804 1079803
oneanddone.mozilla.org QA P1 1049696 1050321 1069250 1069250
mofointegration DB Webmaker P1 Schema Done 1047299 1047309
mofointegration DB Science Lab P2 Schema Done 1047299 1047313
mofointegration DB Open News P2 Schema Done mofo bug mofo bug
mofointegration DB MoCo Ad-hoc P1 Schema 1047299 1047319
mofointegration DB MozFest P2 Schema Done 1047299 1047315
Done In process Blocked No action yet

Q1 2014

  • Establish Baloo documentation and wiki pages -- DONE
  • Creation of appropriate bagheera entry points -- DONE
  • Tryout backporting of historical data from a major contribution system (SuMo?) -- DONE

Q2 2014

  • Start production backporting of historical data from all major contribution systems -- DONE
  • Creation of first dashboard based on 2013 historical data -- DONE

Q3 2014

ID Summary Priority Status
1042101 [Baloo] [QA] Schema update to include Moztrap -- RESOLVED
1047299 Import data from mofointegration to Baloo -- NEW
1047309 Build Baloo aggregations for Webmaker -- NEW
1047313 Build Baloo aggregations for Science Lab -- NEW
1047315 Build Baloo aggregations for MozFest -- NEW
1047319 Build Baloo aggregations for MoCo teams using Ad-hoc logger -- NEW
1049696 [Baloo] [QA] Schema update to include One and Done -- RESOLVED
1050315 [Baloo] Create MDN conversion points -- NEW
1050317 [Baloo] Create AMMO conversion points -- NEW
1050320 [Baloo] Create Webcompat conversion points -- NEW
1050321 [Baloo] Create One and Done conversion points -- RESOLVED
1053149 Simplify transfer of Baloo data from Tableau to Dashboard -- RESOLVED
1053643 [Baloo] Auto updating of Baloo data exports -- NEW
1056771 [Baloo] Automating github aggregations from MoFo integration DB -- RESOLVED
1056909 [Baloo] Update Schema to include github issues -- RESOLVED
1059885 Create a requirements document for areweamillionyet.org -- RESOLVED
1069250 [Baloo] Build and automate aggregations for One and Done -- NEW
1073054 [Baloo] Create Location Services conversion points -- NEW

18 Total; 11 Open (61.11%); 7 Resolved (38.89%); 0 Verified (0%);


Q4 2014

ID Summary Priority Status
1047299 Import data from mofointegration to Baloo -- NEW
1047309 Build Baloo aggregations for Webmaker -- NEW
1047313 Build Baloo aggregations for Science Lab -- NEW
1047315 Build Baloo aggregations for MozFest -- NEW
1047319 Build Baloo aggregations for MoCo teams using Ad-hoc logger -- NEW
1050315 [Baloo] Create MDN conversion points -- NEW
1050317 [Baloo] Create AMMO conversion points -- NEW
1050320 [Baloo] Create Webcompat conversion points -- NEW
1053643 [Baloo] Auto updating of Baloo data exports -- NEW
1059885 Create a requirements document for areweamillionyet.org -- RESOLVED
1069250 [Baloo] Build and automate aggregations for One and Done -- NEW
1073054 [Baloo] Create Location Services conversion points -- NEW
1079803 [Baloo] Build and automate aggregations for Moztrap -- NEW
1079804 [Baloo] Import data from moztrap to Baloo -- NEW

14 Total; 13 Open (92.86%); 1 Resolved (7.14%); 0 Verified (0%);


Technical Documentation

Baloo is re-using Data and BI Services team technological stack re-purposed to suit the Contributor tracking needs. Have a look on the data flow diagram below to understand the inner workings.

Stack

Baloo Data Flow.png

Schema

Baloo deals with Contribution Data. Contribution Data are defined using the established Schema of the Community Building Team.

Adding a new system

Note.png
Tracking contributions without adding a new system
Some contribution areas don't have contribution systems or their systems are not eligible for Baloo integration because they track a low volume of contributors (such as Creative, Legal and Policy). Check Ad-Hoc Baloo solution for an alternative way to track contributions in those areas

Several steps are needed in order to add a new system in Baloo. They can be broken down to two different sections. Porting historical data, and implementing live pushes

Baloo Implementation Workflow.png

Adding Historical Data

Once a system is identified holding historical contribution data, the following should be done to import those to Baloo. Note that this should be an one off procedure given that live-pushes are also implemented in a system.

  1. Identify *what* do you want to track in your system. Write down all possible contributions that are happening in your area.
  2. Talk to your principal webdev or technical people, on how this info is stored in your database.
  3. Tweak/amend the Contribution Schema to fit your tracking needs. (Contact Pierros or join Systems and Data WG meeting for help)
  4. Open a new bug on Mozilla Metrics::Data/Backend Reports describing the location of the data and the schema/mapping your are about to use
  5. Data and BI team will be following up with you on imports

Adding Live Pushes

  1. Make sure you follow steps 1,2 and 3 above.
  2. Coordinate with your principal webdev or technical people on implementing the code on your system side that produces and pushes the JSON contribution tracking payloads.
  3. Open a new bug on Mozilla Metrics::Data/Backend Reports so we can track the implementation process.
  4. Tryout staging the code using the following
    curl -v -X POST -d '{"test":"hello"}' http://10.22.28.101:8080/submit/baloo/
    If the operation successful we will return 201.
  5. Once you are ready for production you may use the production url: https://data.mozilla.com/submit/baloo/
Note.png
All data is in raw format till this point
If any or both of the above is implemented for your system, you have raw data in our HBASE. Although this is available for crunching, you still need to establish your contribution area conversion points so we can start tracking active contributors

Tracking active contributors

Once one or both of the processes above are complete we now have raw contribution data about your area. Raw data is cool, but we need to start making sense out of them. For that reason we need to establish Conversion Points to understand the type of contributions and identify Active contributors.

  1. Have a meeting with your CBT contact person to identify (if not already done) your contribution pathway.
  2. Navigate to our Conversion Points wiki page and add/edit your contribution area with all relevant info.
  3. Once the conversion points are up to date, open a bug on Mozilla Metrics::Data/Backend Reports so we can create the aggregation scenarios and start crunching the raw data.
  4. Magical numbers will flow in, and become available in Tableau (and APIs!) for usage!

Meetings

You can also see meeting notes on the master etherpad

Contact

Pierros and Adam serve as the main contact people at this point. Reach out to them for further info and instructions on how to add your system to Baloo.

You may also find us on #baloo IRC channel in irc.mozilla.org