ReleaseEngineering/Queue directories: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Added link to the mana page for the command queue Nagios alert)
(deleting obsolete page)
 
Line 1: Line 1:
Our buildbot masters make use of several queuedirs to perform out of process tasks such as pushing events to pulse or uploading logs.


== Queuedirs ==
A queuedir is a simple directory structure on disk where individual jobs are stored in files. The files are moved between directories depending on what state they're in.
* <tt>'''tmp'''</tt>: write out new job files here before moving into <tt>'''new'''</tt>.
* <tt>'''new'''</tt>: when job files are moved into here, the queue processors will pick them up and move them into <tt>'''cur'''</tt> to indicate they're currently being processed.
* <tt>'''cur'''</tt>: jobs currently in progress are here.
* <tt>'''logs'''</tt>: output and debugging information for current and finished jobs are here. Logs older than 5 minutes get deleted.
* <tt>'''dead'''</tt>: failed jobs go here. this is bad.
We currently have two queuedirs: '''/dev/shm/queue/commands''' and '''/dev/shm/queue/pulse'''
== Processors ==
We currently have two processors: '''command_runner.py''' and '''pulse_publisher.py'''. These are run as services, start on boot, and are managed by puppet. They both run out of a virtualenv in /builds/buildbot/queue.
We have nagios checks in place to ensure that the queue processors are running, and that there are no dead jobs.
== Troubleshooting ==
If a processor isn't running, they can be restarted via /etc/init.d/command_runner or /etc/init.d/pulse_publisher
If there are dead jobs, you can read the per job log files in the dead directory. After resolving the issue, job files should be moved back into <tt>'''new'''</tt> to be retried, or deleted.
* ''Note: there is a <tt>retry_dead_queue</tt> sub-command for [http://hg.mozilla.org/build/tools/file/5053b0ea4564/buildfarm/maintenance/manage_masters.py manage_masters.py]'' (Or, for ansible lovers <tt>deadqueue.yml</tt> in [https://github.com/mozilla/build-ansible build-ansible]. Example invocation: <tt>ansible-playbook -i master-inventory.py deadqueue.yml</tt>)
== Implementation ==
http://hg.mozilla.org/build/tools/file/739018ba9ff1/lib/python/buildtools/queuedir.py
http://hg.mozilla.org/build/tools/file/739018ba9ff1/buildbot-helpers/command_runner.py
http://hg.mozilla.org/build/tools/file/739018ba9ff1/buildbot-helpers/pulse_publisher.py
http://hg.mozilla.org/build/puppet-manifests/file/0deb57fc17ae/modules/buildmaster/manifests/queue.pp
== See also ==
https://mana.mozilla.org/wiki/display/NAGIOS/Command+Queue
{{Release Engineering How To|Queue Directories}}

Latest revision as of 21:45, 19 November 2018

Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/Queue_directories&oldid=1204125"