ReleaseEngineering/Queue directories: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Created page with "Our buildbot masters make use of several queuedirs to perform out of process tasks such as pushing events to pulse or uploading logs. == queuedirs == A queuedir is a simple dire...")
 
(deleting obsolete page)
 
(9 intermediate revisions by 6 users not shown)
Line 1: Line 1:
Our buildbot masters make use of several queuedirs to perform out of process tasks such as pushing events to pulse or uploading logs.


== queuedirs ==
A queuedir is a simple directory structure on disk where individual jobs are stored in files. The files are moved between directories depending on what state they're in.
* <tt>'''tmp'''</tt>: write out new job files here before moving into <tt>'''new'''</tt>.
* <tt>'''new'''</tt>: when job files are moved into here, the queue processors will pick them up and move them into <tt>'''cur'''</tt> to indicate they're currently being processed.
* <tt>'''cur'''</tt>: jobs currently in progress are here.
* <tt>'''logs'''</tt>: output and debugging information for current and finished jobs are here. Logs older than 5 minutes get deleted.
* <tt>'''dead'''</tt>: failed jobs go here. this is bad.
We currently have two queuedirs: /dev/shm/queue/commands, and /dev/shm/queue/pulse
== processors ==
We currently have two processors: command_runner.py and pulse_publisher.py. These are run as services, start on boot, and are managed by puppet. They both run out of a virtualenv in /builds/buildbot/queue.
We have nagios checks in place to ensure that the queue processors are running, and that there are no dead jobs. '''NB not finished yet - {{bug|685980}}'''
== implementation ==
http://hg.mozilla.org/build/tools/file/739018ba9ff1/lib/python/buildtools/queuedir.py
http://hg.mozilla.org/build/tools/file/739018ba9ff1/buildbot-helpers/command_runner.py
http://hg.mozilla.org/build/tools/file/739018ba9ff1/buildbot-helpers/pulse_publisher.py

Latest revision as of 21:45, 19 November 2018

Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/Queue_directories&oldid=1204125"