Auto-tools/Projects/Pulse
Mozilla Pulse
Mozilla currently has a ton of different systems that are inter-connected via polling, screen scraping, email, and other brittle methods. To make their lives easier community members often build tools on top of this house of cards, adding yet another level of scraping and polling. Many systems don't even export important data for others to scrape and use, preventing better tools from being written.
The goal of "Mozilla Pulse" is to eliminate polling and add visibility into all aspects of Mozilla and its systems. This allows more robust, dynamic, and informative tools.
File bugs: https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=Pulse
Status
At the moment, only buildbot (build and test) messages are being published to Pulse. The other publishers have been disabled for various reasons. We can investigate turning them back on once we have completed some important fixes and features.
Technology used
- The message broker used is RabbitMQ.
- Protocol used to talk to the broker is AMQP.
- Messages are sent by default in JSON, though if you know all the producers and consumers will be in python you can change it to pickle.
- For python, the underlying library currently used to talk AMQP is Kombu.
- Celery is used to run the scrapers that feed data into the broker.
- pubsub plugins should be installed soon
- The STOMP plugin is live, but needs to be documented.
- Need to document how to connect via STOMP to orbit so that you can get push messages via javascript in the browser.
Task List
- Currently a security hole - anyone can post anything
- Currently all exchanges on one vhost
- Partition Pulse into multiple vhosts based on function. This is easier to lock down as well since permissions are based on vhost.
- need more vhosts with password accounts
- Enable the use of SSL certificates for all clients.
- Get IT to provide and maintain the certificates similar to other Mozilla sites.
- mozillapulse needs to be able to handle network fluctations better.
- Move to buildbot to own vhost
- Pulsetranslator currently uses a durable queue
- Find out if having a staging or development environment for Pulse is possible or even needed.
- May be able to accomplish this with setting up testing vhosts.
- Look into what would be needed to update RabbitMQ to latest 3.x version to pull in enhancements/bugfixes.
Admin Procedures
- When a queue becomes stuck, you can use the Admin UI to kill it. Try to ping the queue owner first before killing if possible.
- More than half of the queues are QA related (whimboo)
- pulsetranslator service, which normalizes buildbot messages, is currently running on pulsetranslator.ateam.phx1.mozilla.com and may need to be reset from time to time.
- logparser service, used by Orange Factor, runs on orangefactor1.dmz.phx1.mozilla.com
More reading
LegNeato wrote several blog posts on Pulse as he was building it. They contain some more background if you're really interested. They are linked below, in chronological order.
- Mozilla Pulse and RabbitMQ
- Push notifications for Bugzilla
- This idea has been shelved for the time being, both due to security issues and performance. See BMO/ChangeNotificationSystem for an alternative.
- bugzilla-amqp is now buzgilla-push, supports the STOMP protocol
- See above note. Just keeping these links for interest's sake.
- What’s up with Pulse?
- GitHub AMQP integration service hook live!
- Tangentially related to Pulse.
- Lots of Pulse changes, out of prototype mode this week!
- Another update on Pulse