Auto-tools/Projects/ActiveData/Support

ActiveData Support Documentation

Each part in ActiveData is described in five properties

• Manager: In some cases, another automation can rebuild a part.
• Monitor: The automation responsible for restarting the part if it misbehaves
• Host: Where the part can be found when running
• Configuration: The configuration file
• Code: Pointer to the repo, branch, and main entry point for part

Overall Architecture

This page is best viewed beside the Reference Diagram

Machines

ETL Machine (spot instance)

• Manager: SpotManager
• Monitor:
• Host: AWS
• Configuration: https://github.com/klahnakoski/SpotManager/blob/master/examples/config/etl_settings.json
• Code: https://github.com/klahnakoski/SpotManager/blob/master/examples/etl.py

ES Machine (spot)

• Manager: SpotManager
• Monitor:
• Host: AWS
• Configuration: https://github.com/klahnakoski/SpotManager/blob/master/examples/config/es_settings.json
• Code: https://github.com/klahnakoski/SpotManager/blob/master/examples/es.py

ES Primary Machine

• Manager: Manual Setup
• Monitor: None
• Host: AWS
• Configuration:
• Code: https://github.com/klahnakoski/TestLog-ETL/blob/dev/resources/elasticsearch/setup_es_primary_r3_xlarge.sh

ActiveData Machine

• Manager: Manual Setup
• Monitor:
• Host: AWS
• Configuration:
• Code: https://github.com/klahnakoski/ActiveData/blob/dev/resources/scripts/install_active_data_r3_xlarge.sh

Pulse Logger Machine

• Manager: Manual Setup
• Monitor:
• Host: AWS
• Configuration:
• Code: https://github.com/klahnakoski/TestLog-ETL/blob/dev/resources/scripts/setup_pulse_logger.sh

Beta

• Manager:
• Monitor:
• Host: AWS
• Configuration:
• Code:

Daemons and Services

ETL

• Manager: SpotManager
• Monitor: Supervisor+Cron
• Host: ETL Machine (spot)
• Configuration: https://github.com/klahnakoski/TestLog-ETL/blob/dev/resources/settings/staging/etl.json
• Code: https://github.com/klahnakoski/TestLog-ETL/blob/dev/testlog_etl/etl.py

ES (spot)

• Manager: SpotManager
• Monitor: Supervisor+Cron
• Host: ES Machine (spot)
• Configuration:
• Code:

ES (Primary)

• Manager:
• Monitor: Supervisor+Cron
• Host: ES Primary Machine
• Configuration: https://github.com/klahnakoski/TestLog-ETL/blob/dev/resources/elasticsearch/elasticsearch_primary.yml
• Code:

ES (Coordinator)

Smaller ES node, made to be master, but with no data, on the spot half of the cluster. Intended to coerce primary shards to reside on the spot side.

• Manager:
• Monitor:
• Host: activedata.allizom.org
• Configuration: https://github.com/klahnakoski/ActiveData/blob/dev/resources/config/elasticsearch.yml
• Code:

Spot Manager

• Manager:
• Monitor: Supervisor+Cron
• Host: Kyle's Machine
• Configuration:
• Code: https://github.com/klahnakoski/SpotManager/tree/master

Supervisor+Cron

This special version of Supervisor is used to perform standard monitoring, plus run periodic jobs for more sophisticated checks.

• Manager:
• Monitor:
• Host: All
• Configuration: */supervisord.conf
• Code: https://github.com/klahnakoski/supervisor-plus-cron

ActiveData Service

• Manager:
• Monitor: Supervisor+Cron
• Host: ActiveData Machine
• Configuration: https://github.com/klahnakoski/ActiveData/blob/dev/resources/config/staging_settings.json
• Code: https://github.com/klahnakoski/ActiveData/blob/dev/active_data/app.py

Pulse Logger

The Pulse Logger simply records the pulse messages coming from the pulse translator. They are put into S3 in blocks of 100. If there is a failure, there is only two hours before the queue is full and Pulse deletes the queue, resulting in permanent data loss.

There should be two, each in a different zone, listening on the same queue, to minimize the chance of failure. Loss of order that results will not impact the remaining ETL.

• Manager:
• Monitor:
• Host: Pulse Logger Machine
• Configuration: https://github.com/klahnakoski/TestLog-ETL/blob/dev/resources/settings/staging/pulse_logger.json
• Code: https://github.com/klahnakoski/TestLog-ETL/blob/dev/testlog_etl/pulse_logger.py

Push to ES

• Manager: SpotManager
• Monitor: Supervisor+Cron
• Host: ES Machine (spot)
• Configuration: https://github.com/klahnakoski/TestLog-ETL/blob/dev/resources/settings/staging/push_to_es.json
• Code: https://github.com/klahnakoski/TestLog-ETL/blob/dev/testlog_etl/push_to_es.py

Beta

The Beta configuration is meant to put all services on minimal hardware. It will not keep up with the data volume, but it is a reasonable test of code before it goes to staging.