MozillaWiki

Auto-tools/Projects/Pulse: Difference between revisions

Page Discussion

Auto-tools/Projects/Pulse (view source)

Revision as of 15:04, 24 February 2020

1,038 bytes removed ,  24 February 2020
→‎To upgrade a ssl certificate on pulse.mozilla.org
m (clarify that clientId part of exchange matches the user binding to the server)
 
(14 intermediate revisions by 3 users not shown)
Line 127: Line 127:
== Python Pulse client library ==
== Python Pulse client library ==


The [https://hg.mozilla.org/automation/mozillapulse/ mozillapulse] Python package provides classes for existing publishers, consumers, and messages so you can quickly build Pulse applications.  See the [https://hg.mozilla.org/automation/mozillapulse/file/tip/README.md README] to get started (although note that the test publisher is currently offline; see {{bug|1218976}}.  You can use another consumer, e.g. BuildConsumer, to verify your setup.).
The [https://github.com/mozilla-services/mozillapulse mozillapulse] Python package provides classes for existing publishers, consumers, and messages so you can quickly build Pulse applications.  See the [https://github.com/mozilla-services/mozillapulse/blob/master/README.md README] to get started (although note that the test publisher is currently offline; see {{bug|1218976}}.  You can use another consumer, e.g. BuildConsumer, to verify your setup.).


This library is somewhat inflexible, however, and should be rewritten. One idea is to turn TaskCluster's Python client into a standalone package.
This library is somewhat inflexible, however, and should be rewritten. One idea is to turn TaskCluster's Python client into a standalone package.
Line 141: Line 141:
= Contributing =
= Contributing =


To set up a local system for development, see the [https://hg.mozilla.org/automation/mozillapulse/file/tip/HACKING.md HACKING.md] file included in the mozillapulse source.
To set up a local system for development, see the [https://github.com/mozilla-services/mozillapulse/blob/master/HACKING.md HACKING.md] file included in the mozillapulse source.


The main Pulse library (mozillapulse) and publisher shims (pulseshims) are written in Python, although there is also a Go library as mentioned in the section above.  We also want to provide a canonical JavaScript library at some point.  To hack on the main Pulse library, you should be comfortable in Python, and it's helpful to understand the basics of AMQP.  Knowledge of kombu is also useful.
The main Pulse library (mozillapulse) and publisher shims (pulseshims) are written in Python, although there is also a Go library as mentioned in the section above.  We also want to provide a canonical JavaScript library at some point.  To hack on the main Pulse library, you should be comfortable in Python, and it's helpful to understand the basics of AMQP.  Knowledge of kombu is also useful.
Line 173: Line 173:


For mentored bugs, we use the User Story to provide a link back to this page, as well as any extra information for contributors, such as required knowledge or tools.  The basic text for mentored bugs should be "This is a mentored Pulse bug.  For general information on Pulse, see https://wiki.mozilla.org/Auto-tools/Projects/Pulse, which includes a section on Contributing."  An example of extra text is "This bug also requires you to have a working mail server."
For mentored bugs, we use the User Story to provide a link back to this page, as well as any extra information for contributors, such as required knowledge or tools.  The basic text for mentored bugs should be "This is a mentored Pulse bug.  For general information on Pulse, see https://wiki.mozilla.org/Auto-tools/Projects/Pulse, which includes a section on Contributing."  An example of extra text is "This bug also requires you to have a working mail server."
= Consuming Buildbot messages =
There are two ways to consume messages published by Buildbot.  The most direct way, which requires the most knowledge about Buildbot, is using the BuildConsumer in [http://hg.mozilla.org/automation/mozillapulse mozillapulse].  This consumer has access to all the native Buildbot messages, and therefore offers the most flexibility.
The disadvantage of using the BuildConsumer is that you need to spend time understanding what messages Buildbot publishes to Pulse, and how these can vary, and associate particular messages with what you're trying to accomplish.  The format of Buildbot messages is undocumented, and can change without warning, which makes services based on the BuildConsumer potentially fragile.
To address some of these disadvantages, a translator is run against the BuildConsumer (the [https://github.com/mozilla/pulsetranslator pulsetranslator]) which re-publishes a subset of Buildbot messages to a NormalizedBuild exchange, which are available using the NormalizedBuildConsumer of mozillapulse.  The content of these messages is simplified and normalized, making it easier to consume without the need to have a thorough understanding of how Buildbot publishes messages to Pulse.  The re-published messages also protect consumers against some changes to the Pulse stream, although significant enough changes will likely break pulsetranslator as well as direct users of BuildConsumer.
Another advantage of the NormalizedBuildConsumer is that it will only publish messages for a given build or test job after the logs for that job are available; using the BuildConsumer directly can result in the reception of messages for a build before the build artifacts are available, which can cause problems in consumers if they don't explicitly guard against this.
Generally speaking, consumers that wish to be notified when specific build or test jobs are completed should use the NormalizedBuildConsumer; consumers that need direct access to the Buildbot Pulse stream or are looking for non-specific jobs (such as all jobs belonging to a particular commit) should probably use the BuildConsumer.


= Road Map =
= Road Map =
Line 209: Line 197:
= Admin Procedures =
= Admin Procedures =


jgriffin and mcote have access to the Pulse cluster on CloudAMQP and the following related services:
dustin and the taskcluster team have access to the Pulse cluster on CloudAMQP and the following related services:


* PulseGuardian should be deleting queues that are too long. If you need to manually delete a queue, use the Management UI. Try to ping the queue owner first before killing if possible.
* PulseGuardian should be deleting queues that are too long. If you need to manually delete a queue, use the Management UI. Try to ping the queue owner first before killing if possible.
* pulsetranslator service, which normalizes Buildbot messages, is currently running on Heroku and may need to be reset from time to time.
 
== To upgrade a ssl certificate on pulse.mozilla.org ==
 
Open a bug with IT to generate a new certificate https://bugzilla.mozilla.org/enter_bug.cgi?product=Infrastructure%20%26%20Operations&component=SSL%20Certificates
See {{bug|1532325}} for an example.
 
IT needs to email support@cloudamqp.com with the new cert.  The cloudampq support team will install it on all our of cloudampq nodes.  After it has been installed, you can login to the administrative start the nodes one by one which will not result any downtime.  (Ensure you wait for the node to restart before restarting another one.)  Verify that the certs are installed on the nodes
 
* https://sslanalyzer.comodoca.com/?url=pulse.mozilla.org
* https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com
* https://sslanalyzer.comodoca.com/?url=orange-antelope-02.rmq.cloudamqp.com
* https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com
 
This should show the dates for the new certificate and that the cert is trusted by Mozilla and Microsoft.
 
Cloudampq updated their web page since we last did this so that you should be able to upload the cert yourself and have it propagate.  See the admin console under "Certificate".


= More reading =
= More reading =
Glob
Confirmed users
1,201

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1128603...1224163"