TestEngineering/Services/TSVerifierSyncTestEnvironments

Tokenserver, Verifier, and Sync Production Environments

• TokenServer: https://token.services.mozilla.com
• Sync 1.5 Nodes (for now in AWS):
  • sync-1-us-east-2.sync.services.mozilla.com
  • sync-2-us-east-2.sync.services.mozilla.com
    • There are many more than this now...

• Monitoring
  • Heka shared:
    • https://heka.shared.us-west-2.prod.mozaws.net/
    • https://heka.shared.us-west-2.prod.mozaws.net/#health
    • https://heka.shared.us-west-2.prod.mozaws.net/#sandboxes
  • Kibana shared: https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/file/default.json
  • Others: see the "load" folder/pull-down
  • StackDriver:
    • Main: https://app.stackdriver.com/
    • https://app.stackdriver.com/groups/4208/prod-persona
    • https://app.stackdriver.com/groups/4251/production-all
    • https://app.stackdriver.com/groups/3879/prod-sync-15
    • https://app.stackdriver.com/groups/3828/prod-tokenserver

• QA Access: not allowed

• Pointing Fx Nightly to Production FxA/Sync:
  • Set/Verify the following Firefox configs:
    • services.sync.clusterURL = (should get automatically set by the TokenServer)
    • services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
    • services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
    • services.sync.log.appender.file.logOnError = Yes
    • services.sync.log.appender.file.logOnSuccess = Yes
    • services.sync.log.appender.file.level = Trace
    • identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
    • identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...

TokenServer Stage Environment

• URLs
  • TokenServer: https://token.stage.mozaws.net

• Version:

TokenServer:
rpm -qa | grep token
Example: tokenserver-svcops <latest version>

• AWS
  • Shared:
    • shared-elasticsearch
    • shared-elasticsearch
    • shared-elasticsearch
    • shared-elasticsearch
    • shared-rabbitmq
    • shared-rabbitmq
    • shared-bastion
    • shared-heka

• • tokenserver app server (2 m3.medium instances behind a CF stack and ELB)
  • tokenserver db (1 large DB instance behind RDS)

• Files
  • /data/tokenserver/*
• Processes
  • tokenserver app server:
    • Search for token, circus, nginx, gunicorn

• Logs
  • /media/ephemeral0/logs/
  • /media/ephemeral0/logs/nginx/access.log
  • /media/ephemeral0/logs/nginx/error.log
  • /media/ephemeral0/logs/tokenserver/token.error.log
  • /media/ephemeral0/logs/tokenserver/token.log.*
  • /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
  • /media/ephemeral0/logs/tokenserver/process_account_deletions.log
  • /media/ephemeral0/squid/access.log
  • /var/log/hekad/tokenserver.stdout.log
  • /var/log/hekad/tokenserver.stderr.log

• Hekad
  • /etc/puppet/modules/hekad

• QA Access via a Bastion Host
  • SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

• Monitoring
  • Kibana
    • https://kibana.shared.us-east-1.stage.mozaws.net/
    • https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json
    • https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/elasticsearch

/APK%20Signer%20Log%20Search

• • • https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Token%20App%20Logs%20POC
  • Heka
    • https://heka.shared.us-east-1.stage.mozaws.net/
    • https://heka.shared.us-east-1.stage.mozaws.net/#health
    • https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
  • Stackdriver:
    • Stage main: https://app.stackdriver.com
    • https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
    • https://app.stackdriver.com/instances/<AWS instance>

• Firefox Configs
  • services.sync.clusterURL should get automatically set by the TokenServer
  • services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

Verifier Stage Environment

• URLs
  • Verifier: https://verifier.stage.mozaws.net
  • IdP: https://mockmyid.s3-us-west-2.amazonaws.com/
  • OLD IdP: https://mockmyid.com/

• Version:

Verifier:
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>

• AWS
  • Shared:
    • shared-elasticsearch
    • shared-elasticsearch
    • shared-elasticsearch
    • shared-elasticsearch
    • shared-rabbitmq
    • shared-rabbitmq
    • shared-bastion
    • shared-heka

• • fxa-browserid-verifier (2 c3.large instances behind a CF stack and ELB)

• Files
  • /data/fxa-browserid-verifier
  • /data/hekad
  • /etc/puppet
  • /etc/heka.d
  • /media/ephemeral0/*

• Processes
  • fxa-browserid-verifer:
    • Search for node, heka, nginx, circus

• Logs
  • /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
  • /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
  • /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
  • /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
  • /media/ephemeral0/nginx/logs/squid/access.log
  • /var/log/circus.log
  • /var/log/hekad/fxa-browserid_verifier.stderr.log
  • /var/log/hekad/fxa-browserid_verifier.stdout.log
  • /var/log/nginx/access.log
  • /var/log/nginx/error.log

• Hekad
  • /etc/puppet/modules/hekad

• QA Access via a Bastion Host
  • SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

• Monitoring
  • Kibana
    • https://kibana.shared.us-east-1.stage.mozaws.net/
    • https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json
    • https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/elasticsearch

/APK%20Signer%20Log%20Search

• • Heka
    • https://heka.shared.us-east-1.stage.mozaws.net/
    • https://heka.shared.us-east-1.stage.mozaws.net/#health
    • https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
  • Stackdriver:
    • Stage main: https://app.stackdriver.com
    • https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
    • https://app.stackdriver.com/instances/<AWS instance>

• Firefox Configs
  • services.sync.clusterURL should get automatically set by the TokenServer
  • services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

Sync 1.5 Stage Environment

• This is a work in progress. Right now, there are only a small number of sync nodes (instances) in AWS.
• URLs
  • https://sync-1-us-east-1.stage.mozaws.net
  • https://sync-2-us-east-1.stage.mozaws.net
  • https://sync-3-us-east-1.stage.mozaws.net
  • https://sync-4-us-east-1.stage.mozaws.net
  • https://sync-5-us-east-1.stage.mozaws.net
  • https://sync-6-us-east-1.stage.mozaws.net

• Versions

Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

• AWS
  • Search for sync node instances in US East: "stage-sync-node-X"
  • Each node is a specific large instance (mixed m3 and c3)
• Files
  • /data/server-syncstorage/*
  • /var/log/nginx
• Processes
  • Search for sync, mysql, circusd, hekad, nginx, memcached
• Logs
  • /media/ephemeral0/logs (most important)
    • nginx/access.log
    • (what about error.log?)
    • sync/sync.err
    • sync/sync.log
  • /var/log/nginx
  • /var/log/circus.log
• Hekad
  • /etc/heka.d/sync_1_5.toml
• QA Access via a Bastion Host
  • SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.
• Monitoring
  • Kibana
    • https://kibana.shared.us-east-1.stage.mozaws.net/
    • https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json
    • https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync-Stage-Dash
    • https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/elasticsearch/APK%20Signer%20Log%20Search
    • NOTE: The above 2 are currently broken.
    • See https://github.com/mozilla-services/puppet-config/issues/278
    • and https://github.com/mozilla-services/puppet-config/issues/281
  • Heka
    • https://heka.shared.us-east-1.stage.mozaws.net/
    • https://heka.shared.us-east-1.stage.mozaws.net/#health
    • https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
  • Stackdriver:
    • Stage main: https://app.stackdriver.com
    • https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
    • https://app.stackdriver.com/instances/<AWS instance>
• Firefox Configs
  • services.sync.clusterURL should get automatically set by the TokenServer

• Note: There is no longer a Sync 1.1 Stage environment.

Loads Services Cluster Environment

• URLs
  • http://loads.services.mozilla.com/
  • or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com/

• Versions

Loads Cluster/Broker/Agents:
$ cd /home/ubuntu/loads/bin
$ ./loads-runner --version

• AWS in US West
  • loads-master (broker and agent processes)
  • loads-slave-1 (agent processes)
  • loads-slave-2 (agent processes)
  • NOTE: there is no stack or ELB for this cluster

• Files
  • /home/ubuntu
    • loads
    • loads-aws
    • loads-web
• Processes
  • Search for processes owned by ubuntu, loads, nginx, circus
• Logs
  • /var/log/redis
  • /var/log/nginx
• QA access
  • You need special access to be able to SSH into these devices
  • You need to make some changes to your .ssh/config file
• Monitoring
  • Web Interface: http://loads.services.mozilla.com/
  • StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster
• Cluster status
  • Check directly from the Loads Cluster dashboard:

Agents statuses
Launch a health check on all agents

• Links
  • http://loads.readthedocs.org/en/latest/
  • https://github.com/mozilla-services/loads
  • https://github.com/mozilla-services/loads-aws

TokenServer and Sync 1.5 Dev Environments

• URLs
  • https://sync1.dev.lcip.org/
  • https://token.dev.lcip.org/

• Versions

TokenServer:
rpm -qa | grep token
Example: tokenserver-svcops <latest version>

Verifier:
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>

Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

• AWS
  • This is part of the dev-lcip-org CloudFormation stack
    • TokenServer: Search for the fxa-tokenserver instance
    • SyncServer2: Search for the fxa-syncstorage instance
    • SyncServer1: Search for the fxa-syncstorage instance
    • LogServer: Search for the fxa-logbox instance
• Files
  • TBD
• Processes
  • TBD
• Logs
  • TBD
• QA Access
  • SSH with AWS keys to the various instances
• Firefox configs:
  • services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
  • services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5

OPs Mana and GitHub Pages

• NOTE: Talk to OPs for the links to Mana.
• Puppet Config: https://github.com/mozilla-services/puppet-config
• Cloud Formation: https://github.com/mozilla-services/svcops
  • and https://github.com/mozilla-services/svcops/tree/master/cloudformations/sync
  • and https://github.com/mozilla-services/svcops/tree/master/cloudformations/token