Confirm
125
edits
Changes
no edit summary
* Message processing system must be able to distinguish between different message types, so the various types can be routed to the appropriate back end(s) for effective analysis and reporting.
* Service app owners must have access to an interface (or interfaces) that will provide reporting and querying capabilities appropriate to the various types of messages that have been sent into the system.
== Proposed Architecture ==
The proposed Services Metrics architecture will consist of 3 layers:
; generator : The generator portion of the system is the actual service application that is generating the data that is to be sent into the system. We will provide libraries (described below) that app authors can use to easily plug in. The libraries will take messages generated by the applications, serialize them, and then send them out "fire and forget" style over UDP. The metrics generating apps that need to be supported initially are based on the following platforms:
* Mozilla Services team's Python app framework (sync, reg, sreg, message queue, etc.)
* Node.js (BrowserID).
; router : The router is what will be listening for the UDP packets sent out by the provided libraries. It will deserialize these messages and examine the metadata to determine the appropriate back end(s) to which the message should be delivered. The format and protocol for delivering these messages to the endpoints will vary from back end to back end. We plan on using [http://logstash.net/ logstash] as the message router, because it is already planned to be installed on every services server machine, and it is built specifically for this type of event-based messager routing.
; endpoints : Different types of messages lend themselves to different types of presentation, processing, and analytics. We will start with a small selection of back end destinations, but we will be able to add to this over time as we generate more types of metrics data and we spin up more presentation and query layers. Proposed back ends are as follows:
* [https://github.com/etsy/statsd statsd]: '''(Phase 1)''' statsd is already in the pipeline to be running on every Services machine
* [https://github.com/mozilla-metrics/bagheera Bagheera]: '''(Phase 1)''' Bagheera is a REST service provided by the Mozilla Metrics team that will insert data into the Metrics team's Hadoop infrastructure, available for later processing.
* [https://github.com/dcramer/django-sentry Sentry]: '''(Phase 2)''' Sentry is an exception logging infrastructure that provides useful debugging tools to service app developers. Sentry is not yet planned on being provided by any Mozilla operations team, using it would require buy-in from and coordination with a Mozilla internal service provider (probably the Services Ops team).
== Proposed API ==
The atomic unit for the Services Metrics system is the "message". The structure of a message is inspired by that of the well known syslog message standard, with some slight extensions to allow for arbitrary more rich metadata. Each message will consist of the following fields:
* timestamp: Time at which the message is generated.
=== Ad-Hoc service app metrics gathering ===
Any service app will have the ability to easily generate arbitrary message data and metadata for delivery into the services metrics system. Any messages not specifically recognized as being intended for statsd or sentry will be delivered to a Hadoop cluster provided by the Metrics team, allowing for later analysis via custom map-reduce jobs or [https://hive.apache.org/ Hive ] queries.
