View source for Flume ElasticSearch WOO Maintenance Page

Flume WOO project facilitates realtime buildbot log ingestion inside HDFS/Hive and ElasticSearch via Flume. This page describes the different machines, installed software, and steps to restart services. 

'''ElasticSearch cluster: '''<br>elasticsearch1.metrics.sjc1.mozilla.com (master) elasticsearch2.metrics.sjc1.mozilla.com (slave) elasticsearch3.metrics.sjc1.mozilla.com (slave) 

'''Symptom''': Nagios ElasticSearch alert indicates one (or many) machines are down. <br>'''Fix''': Login to the relevant machine/s. Kill all running elasticsearch processes (ps ax|grep elasticsearch)<br> If problem persists, we will need to stop and start elasticsearch service for the entire cluster.&nbsp;Login to each machine and kill all running elasticsearch processes.<br>Restart the services in following order ('''elasticsearch1, elasticsearch2, elasticsearch3''') <br>'''Restart command:''' /usr/lib/es/bin/elasticsearch 

Please email '''aphadke@mozilla.com, desinspanjer@mozilla.com''' if problem persists. 

<br>'''Flume cluster: '''<br>elasticsearch3.metrics.sjc1.mozilla.com (master) <br>elasticsearch4.metrics.sjc1.mozilla.com (node-collector) <br>elasticsearch5.metrics.sjc1.mozilla.com (node-agent) 

'''Symptom:''' Nagios Flume alert indicates a given machine is down. <br>'''Hostname:''' elasticsearch4.metrics.sjc1.mozilla.com '''<br>Stop Flume''': /usr/lib/flume/bin/flume-daemon.sh stop <br>Confirm flume has stopped (ps ax|grep flume) else kill -9 the pid <br>'''Start Flume:''' /usr/lib/flume/bin/flume-daemon.sh start node_nowatch -n elasticsearch4.metrics.sjc1.mozilla.com 

'''Symptom''': Nagios Flume alert indicates a given machine is down. <br>'''Hostname:''' elasticsearch5.metrics.sjc1.mozilla.com <br>'''Stop Flume:''' /usr/lib/flume/bin/flume-daemon.sh stop Confirm flume has stopped (ps ax|grep flume) else kill -9 the pid <br>'''Start Flume:''' /usr/lib/flume/bin/flume-daemon.sh start node -n elasticsearch5.metrics.sjc1.mozilla.com 

<br> '''Symptom''': Nagios Flume alert indicates a given machine is down. <br>'''Hostname:''' elasticsearch3.metrics.sjc1.mozilla.com <br>'''Resolution''': Please email '''aphadke@mozilla.com (213-509-0575)''' or deinspanjer@mozilla.com. While we can restart Flume master, a master going down might indicate deeper problems. Given the infancy nature of flume, its best to investigate further before just restarting it. <br>'''Restart instructions for reference.'''<br>'''Stop Flume:''' /usr/lib/flume/bin/flume-daemon.sh stop Confirm flume has stopped (ps ax|grep flume) else kill -9 the pid <br>'''Start Flume:''' /usr/lib/flume/bin/flume-daemon.sh start master