Confirm
125
edits
Changes
no edit summary
The proposed Services Metrics architecture will consist of 3 layers:
; generator : The generator portion of the system is the actual service application that is generating the data that is to be sent into the system. We will provide libraries (described below) that app authors can use to easily plug in. The libraries will take messages generated by the applications, serialize them, and then send them out "fire and forget" style over UDP (zeromq?using ZeroMQ as the transport, by default). The metrics generating apps that need to be supported initially are based on the following platforms:
* Mozilla Services team's Python app framework (sync, reg, sreg, message queue, etc.)
* Node.js (BrowserID).
; router : The router is what will be listening for the UDP packets messages sent out by the provided libraries. It will deserialize these messages and examine the metadata to determine the appropriate back end(s) to which the message should be delivered. The format and protocol for delivering these messages to the endpoints will vary from back end to back end. We plan on using [http://logstash.net/ logstash] as the message router, because it is already planned to be installed on every services server machine, and it is built specifically for this type of event-based messager routing.
; endpoints : Different types of messages lend themselves to different types of presentation, processing, and analytics. We will start with a small selection of back end destinations, but we will be able to add to this over time as we generate more types of metrics data and we spin up more presentation and query layers. Proposed back ends are as follows:
* ''tags'': Arbitrary set of key/value pairs that includes any additional data that may be useful for back end reporting or analysis.
We will provide a "metlog" library that will both ease generation of these messages and that will handle packaging them up and delivering them (via UDP) into the message processing infrastructure. Implementations of this library will likely be available in both Python and Javascript, but the Python library will be available first and this document will, for now, only describe the Python API. The Javascript API will be similar, modulo syntactic sugar that is available in Python but not in JS (e.g. decorators, context managers), and will be documented in detail in the future. The proposed Python API is as follows:
; '''MetlogClient(bindstrs, logger="", severity=6)''' : Primary metlog client class which can accept metlog messages and will deliver them to the message processor.
