Confirmed users
142
edits
Auto-tools/Projects/Pulse: Difference between revisions
Added link to Go (golang) pulse client library and made headings bigger
(Remove outdated sections, add some updates) |
(Added link to Go (golang) pulse client library and made headings bigger) |
||
| Line 1: | Line 1: | ||
= Introducing Pulse = | |||
https://pulse.mozilla.org/ | https://pulse.mozilla.org/ | ||
| Line 11: | Line 13: | ||
Also see the [https://tools.taskcluster.net/pulse-inspector/ Pulse Inspector] web app, which displays Pulse messages in real time, and the (manually updated) [[/Exchanges|list of Pulse exchanges]]. | Also see the [https://tools.taskcluster.net/pulse-inspector/ Pulse Inspector] web app, which displays Pulse messages in real time, and the (manually updated) [[/Exchanges|list of Pulse exchanges]]. | ||
= System Description = | |||
Pulse isn't any one thing. At its heart, it is a RabbitMQ system with a particular configuration and a set of conventions for using it along with a management tool, [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]], to make Pulse as automated and self-serve as possible. Pulse follows the pub-sub pattern, in which publishers send messages to topic exchanges, and consumers create queues bound to these exchanges in order to subscribe to the publishers' messages. | Pulse isn't any one thing. At its heart, it is a RabbitMQ system with a particular configuration and a set of conventions for using it along with a management tool, [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]], to make Pulse as automated and self-serve as possible. Pulse follows the pub-sub pattern, in which publishers send messages to topic exchanges, and consumers create queues bound to these exchanges in order to subscribe to the publishers' messages. | ||
= Specification = | |||
Pulse is a managed [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1] | Pulse is a managed [https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf AMQP 0-9-1] | ||
| Line 22: | Line 24: | ||
use to integrate and extend Mozilla infrastructure. | use to integrate and extend Mozilla infrastructure. | ||
== Authentication == | |||
Pulse credentials are managed and issued by [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]], | Pulse credentials are managed and issued by [[Auto-tools/Projects/Pulse/PulseGuardian|PulseGuardian]], | ||
| Line 33: | Line 35: | ||
''username'' and ''password'', respectively. | ''username'' and ''password'', respectively. | ||
== Authorized Users == | |||
Pulse is intended to be open to all Mozillians who want to | Pulse is intended to be open to all Mozillians who want to | ||
| Line 39: | Line 41: | ||
abuse PulseGuardian users MUST authenticate via Persona. PulseGuardian SHOULD verify that users have a vouched Mozillians profile. | abuse PulseGuardian users MUST authenticate via Persona. PulseGuardian SHOULD verify that users have a vouched Mozillians profile. | ||
== Publishers == | |||
Publishers MUST name ''exchanges'' in the form | Publishers MUST name ''exchanges'' in the form | ||
| Line 63: | Line 65: | ||
given exchange do not exhibit deliver ''at-least-once'' semantics. | given exchange do not exhibit deliver ''at-least-once'' semantics. | ||
== Subscribers == | |||
Subscribers MUST name ''queues'' in the form | Subscribers MUST name ''queues'' in the form | ||
| Line 82: | Line 84: | ||
for deliver-''at-least-once'' semantics. | for deliver-''at-least-once'' semantics. | ||
== Appendix A: Everything in Bullet Points == | |||
This is a summary of the above. | This is a summary of the above. | ||
| Line 122: | Line 124: | ||
* MUST not grow unbounded | * MUST not grow unbounded | ||
= Let's Use It = | |||
There are currently two pulse clients available. Please note that you can also connect to pulse in other languages, provided you have an AMQP 0.9.1 library that will let you interact with AMQP exchanges. See https://github.com/rabbitmq/rabbitmq-tutorials#languages for example. | |||
* Python Pulse client library: the [https://pypi.python.org/pypi/MozillaPulse mozillapulse] Python package provides classes for existing publishers, consumers, and messages so you can quickly build Pulse applications. | |||
* Go (golang) Pulse client library: http://petemoore.github.io/pulse-go/ | |||
= Contributing = | |||
To set up a local system for development, see the [https://hg.mozilla.org/automation/mozillapulse/file/tip/HACKING.md HACKING.md] file included in the mozillapulse source. | To set up a local system for development, see the [https://hg.mozilla.org/automation/mozillapulse/file/tip/HACKING.md HACKING.md] file included in the mozillapulse source. | ||
| Line 155: | Line 160: | ||
For mentored bugs, we use the User Story to provide a link back to this page, as well as any extra information for contributors, such as required knowledge/learnings. The basic text for mentored bugs should be "This is a mentored Pulse bug. For general information on Pulse, see https://wiki.mozilla.org/Auto-tools/Projects/Pulse, which includes a section on Contributing." An example of extra text is "This bug also requires you to have a working mail server." | For mentored bugs, we use the User Story to provide a link back to this page, as well as any extra information for contributors, such as required knowledge/learnings. The basic text for mentored bugs should be "This is a mentored Pulse bug. For general information on Pulse, see https://wiki.mozilla.org/Auto-tools/Projects/Pulse, which includes a section on Contributing." An example of extra text is "This bug also requires you to have a working mail server." | ||
= Consuming Buildbot messages = | |||
There are two ways to consume messages published by Buildbot. The most direct way, which requires the most knowledge about Buildbot, is using the BuildConsumer in [http://hg.mozilla.org/automation/mozillapulse mozillapulse]. This consumer has access to all the native Buildbot messages, and therefore offers the most flexibility. | There are two ways to consume messages published by Buildbot. The most direct way, which requires the most knowledge about Buildbot, is using the BuildConsumer in [http://hg.mozilla.org/automation/mozillapulse mozillapulse]. This consumer has access to all the native Buildbot messages, and therefore offers the most flexibility. | ||
| Line 167: | Line 172: | ||
Generally speaking, consumers that wish to be notified when specific build or test jobs are completed should use the NormalizedBuildConsumer; consumers that need direct access to the Buildbot pulse stream or are looking for non-specific jobs (such as all jobs belonging to a particular commit) should probably use the BuildConsumer. | Generally speaking, consumers that wish to be notified when specific build or test jobs are completed should use the NormalizedBuildConsumer; consumers that need direct access to the Buildbot pulse stream or are looking for non-specific jobs (such as all jobs belonging to a particular commit) should probably use the BuildConsumer. | ||
= Road Map = | |||
See the [http://mzl.la/1pc2F3M prioritized bug list] for all open issues and feature requests. | See the [http://mzl.la/1pc2F3M prioritized bug list] for all open issues and feature requests. | ||
= Security Model = | |||
This is summarized in the formal Pulse specification above. What follows is the rationale and some technical implementation notes. | This is summarized in the formal Pulse specification above. What follows is the rationale and some technical implementation notes. | ||
| Line 188: | Line 193: | ||
With this security model, we technically don't really need vhosts, since the names of the queues and exchanges the users can use are so specific. There may still be a benefit in allowing apps to use the same queue name for different exchanges, though, which would be possible if each exchange had its own vhost. The downside is that you cannot specify "all vhosts" when setting a user's permissions, so they would either have to list all vhosts they want to use when creating the user in PulseGuardian, and be able to update that list later, or PulseGuardian or some other app would have to automatically add new permissions to all users when a vhost is created. | With this security model, we technically don't really need vhosts, since the names of the queues and exchanges the users can use are so specific. There may still be a benefit in allowing apps to use the same queue name for different exchanges, though, which would be possible if each exchange had its own vhost. The downside is that you cannot specify "all vhosts" when setting a user's permissions, so they would either have to list all vhosts they want to use when creating the user in PulseGuardian, and be able to update that list later, or PulseGuardian or some other app would have to automatically add new permissions to all users when a vhost is created. | ||
= Admin Procedures = | |||
* PulseGuardian should be deleting queues that are too long. If you need to manually delete a queue, use the Management UI. Try to ping the queue owner first before killing if possible. | * PulseGuardian should be deleting queues that are too long. If you need to manually delete a queue, use the Management UI. Try to ping the queue owner first before killing if possible. | ||
| Line 194: | Line 199: | ||
* logparser service, used by [http://brasstacks.mozilla.com/orangefactor/ Orange Factor], runs on orangefactor1.dmz.phx1.mozilla.com | * logparser service, used by [http://brasstacks.mozilla.com/orangefactor/ Orange Factor], runs on orangefactor1.dmz.phx1.mozilla.com | ||
= More reading = | |||
* [http://slides.com/mcote/pulse Slides] from a presentation on Pulse. | * [http://slides.com/mcote/pulse Slides] from a presentation on Pulse. | ||