Confirmed users
9,511
edits
EngineeringProductivity/Projects/ActiveData: Difference between revisions
EngineeringProductivity/Projects/ActiveData (view source)
Revision as of 05:34, 19 September 2019
, 19 September 2019→Overview
Klahnakoski (talk | contribs) (Add list of stuff NOT indexed) |
|||
| (20 intermediate revisions by one other user not shown) | |||
| Line 2: | Line 2: | ||
= Overview = | = Overview = | ||
ActiveData is a | ActiveData is a collection of about 8 billion records (Feb 2016) covering unit tests, Buildbot jobs, performance data, and mercurial. This collection is publicly available, and can be queried directly, similar to any database. | ||
ActiveData is built on top of ElasticSearch, a fast, distributed, redundant document store. ActiveData provides the benefits of familiar and succinct SQL by translating SQL-like queries to ElasticSearch queries. | |||
== Problem == | == Problem == | ||
In order to improve our testing infrastructure we require data on how that infrastructure is performing. That information can be extracted from the raw logs, but that requires downloading samples, parsing data, insertion into a database (or worse, writing queries in an imperative language, like Python). When we are done an analysis we have effectively built an ETL pipeline that does not scale, and is too specific to be reused elsewhere. The next project does this work all over again. | In order to improve our testing infrastructure we require data on how that infrastructure is performing. That information can be extracted from the raw logs, but that requires downloading samples, parsing data, insertion into a database (or worse, writing queries in an imperative language, like Python). When we are done an analysis we have effectively built an ETL pipeline that does not scale, and is too specific to be reused elsewhere. The next project does this work all over again. | ||
== Solution== | == Solution== | ||
ActiveData will serve as a reusable ETL pipeline; annotating the test results with as much relevant data as possible. It also provides a query service to explore and aggregate the data, so there is minimal setup required to access this data. | ActiveData will serve as a reusable ETL pipeline; annotating the test results with as much relevant data as possible. It also provides a query service to explore and aggregate the data, so there is minimal setup required to access this data. | ||
= Redash = | |||
ActiveData has a Redash connector for [https://sql.telemetry.mozilla.org/ stmo] and it accepts SQL. [https://github.com/mozilla/ActiveData/blob/dev/docs/redash.md see main doc for details] | |||
= Charts = | |||
ActiveData is fast enough to support dashboards. | |||
== Build times == | |||
{| class="wikitable" | |||
|- | |||
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/MoBuildbotTimings/End-to-End.html End to End Times] | |||
|| Shows overall time from when a build is first requested to the time tests on that build are complete. | |||
|- | |||
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/MoBuildbotTimings/Builds-Overview.html Build Times] | |||
|| Time series view of build times by platform and build type. Click on a bar to get a scatter plot view. | |||
|- | |||
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/MoBuildbotTimings/Builds-Detail.html Detailed Build Times] | |||
|| Scatter plot of build times. Use the left navigation panel to choose a combination. Click on a data point to see the Buildbot step times and Mozharness step times. | |||
|- | |||
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/MoBuildbotTimings/Buildbot-Simulator.html Buildbot Simulator]<br><span style="color:red">incomplete</span> | |||
|| An incomplete Buildbot scheduling simulator. It can be used to see past wait times, queue size, and inter-job delays. | |||
|- | |||
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/MoBuildbotTimings/index.html Test Runtimes] | |||
|| Choose test suite and machine pool to get an average run time for each of the buildbot steps, and Mozharness steps. | |||
|} | |||
==Unit Test Visualization== | |||
With all unit test results in ActiveData, we can get accurate estimates of "failure rate"; and be able to focus on the most-failing tests. | |||
{| class="wikitable" | |||
|- | |||
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/testfailures/index.html Top Intermittent Failures] | |||
|| List of top 30 most-failing unit tests, and list of top 30 most-recent failing tests. Click on the link to get a scatterplot. | |||
|- | |||
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/testfailures/test.html Find Test Results] | |||
|| Use the search bar to find a test. A list of matching tests, and platform combinations will show the unit test failures and durations. | |||
|- | |||
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/NeglectedOranges/index.html Neglected Oranges] | |||
|| Cross reference OrangeFactor and Bugzilla to give a list of frequent intermittents that have no bug activity. | |||
|} | |||
= Design = | = Design = | ||
ActiveData attempts to provide the benefits of an available database to the public; except larger and faster. | ActiveData attempts to provide the benefits of an available database to the public; except larger and faster. | ||
| Line 22: | Line 68: | ||
== Limitations == | == Limitations == | ||
The unittest data is limited to those test suites that generate structured logs. Currently ( | The unittest data is limited to those test suites that generate structured logs. Currently (Feb, 2016) the following do NOT have structured logs, and are NOT in ActiveData: | ||
* cppunittest | * cppunittest | ||
* and any of the js based gaia suites (e.g Gij) | * and any of the js based gaia suites (e.g Gij) | ||
Specifically, you can see if a structured log is being generated: In | Specifically, you can see if a structured log is being generated: In Treeherder, click a job. Under the "Job details" pane at the bottom, look for a line similar to: | ||
<blockquote> ''artifact uploaded: <suite>_raw.log'' | <blockquote> ''artifact uploaded: <suite>_raw.log'' | ||
| Line 51: | Line 91: | ||
ActiveData is not meant to replace an application database. Applications often track significantly more data related to good interface design, process sequences, complex relations, and object life cycles. | ActiveData is not meant to replace an application database. Applications often track significantly more data related to good interface design, process sequences, complex relations, and object life cycles. | ||
ActiveData's simple model makes it difficult to track object life cycles and impossible to model many-to-many relations. | ActiveData's simple model makes it difficult to track object life cycles and impossible to model many-to-many relations. | ||
Data is not live, and | Data is not live, and definitely does not track "pending jobs" like TreeHerder or TaskCluster do. Test results may take a day, or more, to be indexed. | ||
= Dependencies / Who will use this = | = Dependencies / Who will use this = | ||
== Dependencies == | == Dependencies == | ||
ActiveData's ETL pipeline ingests data from a variety of sources: | |||
* Structured Logs from Unittests | |||
* Task Cluster tasks | |||
** including mozharness timings | |||
* PerfHerder at a per-replicate level | |||
* Treeherder | |||
* hg.mozilla.org branches and revisions | |||
* code coverage (all tests, all files, all lines) | |||
* various logs from ETL pipeline | |||
* all firefox files and related components | |||
* firefox testing results | |||
* old buildbot jobs | |||
== Users == | == Users == | ||
ActiveData's primary goal is to support dashboards that give Mozilla useful perspectives into the large amount of data: | |||
* [https://github.com/mozilla/active-data-recipes/tree/master/adr/recipes ActiveData Recipes has a variety of use cases] | |||
* Individual unit test results | |||
* Task cluster test timing | |||
* Firefox compile times | |||
* Recently new, removed, and disabled tests | |||
* Buildbot wait times | |||
* CodeCoverage aggregates and per-file detail | |||
= Let's Use It! = | = Let's Use It! = | ||
The service listens at http://activedata.allizom.org/query and accepts queries in [https://github.com/klahnakoski/ | The service listens at http://activedata.allizom.org/query and accepts queries in [https://github.com/klahnakoski/ActiveData/blob/dev/docs/jx_tutorial.md JSON Query Expression format]. | ||
curl -XPOST -d "{\"from\":\"unittest\"}" http://activedata.allizom.org/query | |||
== The Query Tool == | == The Query Tool == | ||
The ActiveData service is intended for use by automated clients, not humans. The [http://activedata.allizom.org/tools/query.html Query Tool] is a minimal web page for humans to do some | The ActiveData service is intended for use by automated clients, not humans. The [http://activedata.allizom.org/tools/query.html Query Tool] is a minimal web page for humans to do some exploration, and to test phrasing queries. | ||
* [http://activedata.allizom.org/tools/query.html ActiveData QueryTool] | * [http://activedata.allizom.org/tools/query.html ActiveData QueryTool] | ||
== Documentation == | == Documentation == | ||
* [https://github.com/klahnakoski/ActiveData/blob/ | * [https://github.com/klahnakoski/ActiveData/blob/dev/docs/jx_tutorial.md tutorial] | ||
* [https://github.com/klahnakoski/ActiveData/blob/ | * [https://github.com/klahnakoski/ActiveData/blob/dev/docs/Unittest%20Schema.md Unit test results schema] | ||
* [https://github.com/mozilla/ActiveData/blob/dev/docs/schemas_190227.md Listing of current schemas] | |||
= Code = | = Code = | ||
Development is still in the early stages, setting up your own service | Development is still in the early stages, setting up your own service | ||
* Github: https://github.com/klahnakoski/ActiveData | * Github: https://github.com/klahnakoski/ActiveData | ||
= Contact = | = Contact = | ||