Confirm
513
edits
Changes
it has been 3 months, total rewrite
= Overview =
== Goals ==
An active data instance distinguishes itself from a static resource, or database, or big data solution, by delivering a particular set of features:
* '''A service, open to third party clients''' - By providing the service, clients don't need to setup their own datastore
* '''Fast filtering''' - Sub-second filtering over the contents of the whole datastore, independent of size, saves the application developer from declaring and managing indexes that do the same: There is sufficient information in the queries to determine which indexes should be built to deliver a quick response.
* '''Metadata on dimensions and measures''' - ActiveData also provides context to the data it holds. It serves the purpose to allow exploration and discovery by third parties; by describing unit-of-measure, how dimensions relate to others, and provide human descriptions of the columns stored. This metadata is also invaluable in automating the orientation and formatting of dashboard charts: Knowing the domain of an axis allows code to decide the best (default) chart form, and provides logically reasonable aggregate options.
* '''Has a security model''' - Simpler applications can avoid the complications of a security model if it is baked into the ActiveData solution. If ActiveData is to become mainstream it is important that it can manage sensitive data and PII.
== Limitations ==
ActiveData makes specific tradeoffs to achieve its goals. It has the following limitations:
* large memory requirements
* low add/update/remove speeds
* strict data model (snowflake schema, hierarchical relations only)
* non-relational
* ETL work required to de-normalize data
* ETL work required to provide dimension metadata
== Non Goals ==
ActiveData is not meant to replace an application database. Applications often track significantly more data related to good interface design, process sequences, complex relations, and object life cycles.
ActiveData's simple model makes it difficult to track object life cycles and impossible to model many-to-many relations.
Data is not live, and definitly does not track "pending jobs" like TreeHerder or TaskCluster do. Test results may take a day, or more, to be indexed.
= Dependencies / Who will use this =
== Dependencies ==
== Users ==
= Let's Use It! =
The service listens at http://activedata.allizom.org/query and accepts queries in [https://github.com/klahnakoski/Qb/blob/master/docs/Qb_Reference.md Qb format]. == The Query Tool ==The ActiveData service is intended for use by automated clients, not humans. A tutorial The [http://activedata.allizom.org/tools/query.html Query Tool] is sorely neededa minimal web page for humans to do some exporation, and work is in progressto test phrasing queries.* [http://activedata.allizom.org/tools/query.html ActiveData QueryTool] == Documentation ==* [https://github.com/klahnakoski/ActiveData/blob/master/docs/Qb_Tutorial. In the meantime, please read the md Simple tutorial]* [https://github.com/klahnakoski/ActiveData/blob/master/testsdocs/test_unittestsUnittest%20Schema.py testing code which sends some unittest-specific queriesmd Unittest results schema]
= Code =
Development is still in the early stages, setting up your own service
* Github: https://github.com/klahnakoski/ActiveData
== Tests ==
* [https://github.com/klahnakoski/ActiveData/blob/master/tests/test_unittests.py testing code which sends some unittest-specific queries]
== Bugs ==
Bug are tracked in [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=activedata Bugzilla]. The open issues are shown here:
<bugzilla>
{
"quicksearch":"activedata"
}
</bugzilla>
= Contact =
=More Context=Mostly rambling, optional reading.
== Problem Inspiration==This project is inspired by the data warehouse and data mart technology that is common inside large corporations. These warehouses are useful because they are "active" services: This means the data is not only available, but it can be explored interactively by large audience using a query language.
== General Problem ==
A significant portion of any application is the backend database/datastore, which include:
* Managing resources and machines to support the datastore
The manual effort put toward these features becomes significant as the amount of data grows in size and complexity. More importantly, this effort is being spent over and over on a multitude of applications, each a trivial variation of the next.
==General Solution==
Abstractly, we desire to reduce this redundant workload by adding a layer of abstraction called ActiveData: Clients using ActiveData benefit from the features it provides and avoid the datastore management complexities. While the ActiveData implementers can focus on these common issues while being given a simpler data model, and simpler query language, upon which to calculate optimizations.
Applications that leverage an active data warehouse can forgo significant server side development, if not all, and put the logic on the client side.
