BMO/ElasticSearch

From MozillaWiki
< BMO
Jump to: navigation, search

A Public ElasticSearch Cluster with Bugzilla Data

Objective

Provide the public with an easy-to-query historical view of Bugzilla bugs.

Justification

The BMO/ES cluster is important for managing many of Mozilla's projects: It is an active data system that underlies a multitude of dashboards to track various project metrics. It is capable of providing both high level aggregates over the million+ issues, providing detailed historical views into those issues, and it provides that information fast.

Let's Use It!

BMO/ES is a service, which means you can use it right now! You probably already have your browser open, so let's send it queries:

Examples

Minimum required to hit ES cluster on Github
Minimum required to pull comments on Github
Simple Bug Count on Github
Can you reach the private cluster? on Github

Basic walk-through

https://github.com/klahnakoski/Qb/blob/master/docs/BZ_Tutorial.md

Code

Backend ETL Code: https://github.com/klahnakoski/Bugzilla-ETL
Frontend Examples: https://github.com/klahnakoski/MoDevMetrics/tree/dev

Hacking/Contributors

This project needs your help! If you love data munging and dealing with complexity; and you get satisfaction that people will build atop your hard work every day, then this project is for you!

Contacts

  • IRC: ekyle@irc.mozilla.org (in #ateam and #bteam)
  • email: klahnakoski@mozilla.com

Good First Bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Big bugs

Full Query
ID Summary Status Resolution Assigned to
922905 Add code to trigger full ETL NEW
927494 Review flag from ? to +/- updates the requestee , this is wrong [2 of 3] NEW
945854 https://bugzilla.mozilla.org/show_bug.cgi?id=749568 does not clear review flag properly NEW
945862 Add BZ Patch Metadata to ES NEW
959668 Add support for userstory/abstract field [3 of 3] NEW
961588 Alias Analysis to put less preasure on BZ Database NEW
962882 Getting *real* bug history is hard NEW
979302 ETL should merge bugs_activity entries, like BZ does [3 of 3] NEW
986508 Alias analysis should use ES as datastore, not file NEW
987706 Inconsistent value fields not handled well enough NEW
993830 Add recorded_ts to help replication slaves keep up to date [2 of 3] NEW
997228 Renaming flags will destroy bug history for those flag values NEW
1028277 ETL should write "remaining_time" as number [2 of 3] NEW
1028950 Find metric to detect imminent failure of ES cluster NEW
1063125 Unstable SQL connection causes ETL corruption NEW
1067395 Use 'X' instead of 'D' to mark cleared flags NEW
1094366 Mentors field not captured by ETL [2 of 3] NEW
1102884 esFrontLine should track more stats [2 of 3] NEW
1110978 Add permission model to esFrontLine, using Persona [3 of 3] NEW
1125904 Blocked/blocked_by/depends_on is too confusing NEW
1160411 Send alert when ETL stops NEW
1182063 Find home for some daily scripts that use the BMO ES cluster NEW
1183445 Parse emails as lowercase NEW
1197813 [BZ ETL] Tracking flags changed corrupt bug history NEW
1201064 cf_blocking_loop dropped (renamed), not reflected in history NEW
1356329 Private cluster needs https NEW
1429372 bzETL Upgrade to ES version 6 (meta) NEW

27 Total; 27 Open (100%); 0 Resolved (0%); 0 Verified (0%);


Other Reading