Confirm
125
edits
Changes
no edit summary
* Rob Miller
* Victor Ng
== User Requirements ==
The first version of the Metrics system will focus on providing an easy mechanism for the [[Services/Sync| Sync]] and [https://browserid.org/ BrowserID] projects (and any other internal Mozilla services) to efficiently send profiling data and any other arbitrary metrics information that may be desired into one or more backend storage locations. Once the data has made it to its final destination, there should be available to those w/ appropriate access the ability to do analytics queries and report generation on the accumulated data.
Requirements:
* Service Apps using primary Services Python framework apps should be provided an easy to use API that will get certain pre-defined allow them to send arbitrary text data into the metrics captured for them "for free" (iand reporting infrastructure.e. w* Processing and I/o having to explicitly request them). Configuration options will O load generated by the API calls made by the services apps must be provided extremely small to prevent some or all of this data collection from happening allow for minimal impact on app performance even when this there is a requirement for user privacy and/or security reasonsvery high volume of messages being passed.* Service Apps will be provided API should provide a simple mechanism for inserting arbitrary data points into the metrics systemmetadata to be attached to every message payload.* All inserted data will Overall system should provide a sensible set of message categories so that commonly generated types of messages can be transparently (to labeled as such, and so that the service app) processed processing and passed into reporting functionality can easily distinguish between the appropriate back end storage and analytics destinationvarious types of message payloads.* Service app owners have access Message taxonomy must be easily extendable to an interface (or interfaces) where they can perform arbitrary queries against the data points support message types that have been capturedare not defined up front. === Phase 2 === The second phase of will focus on improving the back end reporting infrastructure. Once data has started flowing and there is an opportunity * Message processing system must be able to assess which reports and graphs would be most generally usefuldistinguish between different message types, so the goal will to various types can be routed to make it as easy as possible the appropriate back end(s) for existing effective analysis and new service app owners to get to their informationreporting. Requirements: * Service app owners must have access to an interface (or interfaces) where they can access a set that will provide reporting and querying capabilities appropriate to the various types of predefined reports and/or graphs & charts displaying useful information based on captured data points (e.gmessages that have been sent into the system. unique daily users, average time elapsed handling requests).
== Proposed API ==
The full atomic unit for the Services Metrics infrastructure will consist of a couple of different APIssystem is the "message". The first will be structure of a mechanism for sending performance- and ops-related datamessage is inspired by that of the well known syslog message standard, limited with some slight extensions to increment counters and timers (i.e. time elapsed allow for completion of a certain operation), into a [https://github.com/etsy/statsd statsd] setup, which will ultimately feed into a [http://graphite.wikidot.com/ graphite] installationarbitrary metadata. The second Each message will provide a way to capture arbitrary text data, analogous to syslog-style log entries, with each record accompanied by a set consist of string tokens that will identify the type of payload the record contains as well as any other metadata that may be useful for analytics and/or processing.following fields:
'''set_metlog_dest(host, port)'''
call themselves.
'''metlogset_default_logger(tokens, msglogger)''' Sends a single log message to the previously specified metlog listener. ''tokens'' should be a sequence of string tokens containing any metadata required to identify and classify the message, while ''msg'' should contain the main data payload. This will be serialized into a simple format and sent via UDP to the listener, "fire and forget"-style for minimal performance impact on the calling application.
'''set_message_flavor(flavor_name, metadata)'''
The first iteration of this solution will not require metadata for a great deal of engineering given message can be used to implement, as it will leverage lots of infrastructure label and categorize that is already in place message. The statsd client will be configured to talk to the statsd services that are already planned to be running on every Services hostThis function expects a string value ''flavor_name'' and a dictionary ''metadata''. The stats gathered from the various hosts will flavor name value can be sent on passed in as a ''flavor'' to subsequent ''metlog'' calls as shorthand for including the Services Ops graphite installation, which will aggregate and graph specified metadata in the stats for developer consumptionoutgoing message.
Sends a single log message to the previously specified metlog listener.
Most of the arguments correspond to the message fields described above.
None of them are strictly required, but most of them will be populated by
reasonable defaults if they aren't provided:
