TestEngineering/Services/TSVerifierSyncTestEnvironments

• NOTE: We currently have two Verifier stacks in Stage (and probably Production):
  • The standalone Browser_ID Verifier stack: See that Verifier Stage section below...
  • A Tokenserver+Verifier stack: See the TokenServer Stage section below...

Tokenserver, Verifier, and Sync Production Environments

• TokenServer: https://token.services.mozilla.com
• Verifier: https://verifier.accounts.firefox.com
• Sync 1.5 Nodes (for now in AWS):
  • sync-1-us-east-2.sync.services.mozilla.com
  • sync-2-us-east-2.sync.services.mozilla.com
    • There are many more than this now...

• QA Access: not allowed

• Pointing Fx Nightly to Production FxA/Sync:
  • Set/Verify the following Firefox configs:
    • services.sync.clusterURL = (should get automatically set by the TokenServer)
    • services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
    • services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
    • services.sync.log.appender.file.logOnError = Yes
    • services.sync.log.appender.file.logOnSuccess = Yes
    • services.sync.log.appender.file.level = Trace
    • identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
    • identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...

TokenServer+Verifier Stage Environment

• NOTE: In an upcoming deployent, the Verifier functionality will be rolled into the Tokenserver stack of services.

• URLs
  • TokenServer: https://token.stage.mozaws.net
  • Verifier: https://verifier.stage.mozaws.net
  • IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

• Version:

TokenServer:
rpm -qa | grep -E 'token|browser'

• AWS
  • tokenserver app server or Token app server or similar
    • (1 or more instances behind a CF stack and ELB)
  • tokenserver db (1 large DB instance behind RDS)

• Files
  • /opt/aws
  • /opt/ec2
  • /opt/openresty
  • /opt/rh
  • /opt/stackdriver
  • /data/tokenserver
  • /data/fxa-browserid-verifier
    • esp. /data/fxa-browserid-verifier/config/settings.json
  • /data/hekad
  • /etc/heka.d
  • /etc/puppet
    • esp. /etc/puppet/yaml/app
    • fxa.stage.yaml
    • fxa.yaml
    • tokenserver.stage.yaml
    • tokenserver.yaml

• Processes
  • tokenserver app server:
    • Search for token, circus, nginx, gunicorn, python, hekad, node

• Logs
  • /media/ephemeral0/logs/tokenserver/token.error.log
  • /media/ephemeral0/logs/tokenserver/token.log.*
  • /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
  • /media/ephemeral0/logs/tokenserver/process_account_deletions.log
  • /media/ephemeral0/logs/tokenserver/purge_old_records.log
  • /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
  • /media/ephemeral0/nginx/logs/default.access.log (not in use)
  • /media/ephemeral0/nginx/logs/default.error.log (not in use)
  • /media/ephemeral0/nginx/logs/tokenserver.access.log
  • /media/ephemeral0/nginx/logs/tokenserver.error.log
  • /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
  • /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
  • /var/log/circus.log
  • /var/log/hekad/tokenserver.stdout.log
  • /var/log/hekad/tokenserver.stderr.log

• Hekad
  • /etc/puppet/modules/hekad

• QA Access via a Bastion Host
  • Old Dev IAM
    • SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance.
  • New Dev IAM
    • SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance.

• Firefox Configs
  • services.sync.clusterURL should get automatically set by the TokenServer
  • services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

Verifier Stage Environment

• NOTE: In an upcoming deployment, the Verifier functionality will be rolled into the Tokenserver stack of services.

• URLs
  • Verifier: https://verifier.stage.mozaws.net
  • IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

• Version:
  • rpm -qa | grep verifier

• AWS
  • Shared:
    • shared-elasticsearch
    • shared-elasticsearch
    • shared-elasticsearch
    • shared-elasticsearch
    • shared-rabbitmq
    • shared-rabbitmq
    • shared-bastion
    • shared-heka

• • fxa-browserid_verifier or fxa-bv-stage (usually a single instance)

• Files
  • /data/fxa-browserid-verifier
    • esp. /data/fxa-browserid-verifier/config/settings.json
  • /data/hekad
  • /opt/aws
  • /opt/ec2
  • /opt/openresty
  • /opt/rh
  • /opt/stackdriver
  • /etc/heka.d
  • /etc/puppet
    • esp. the /etc/puppet/yaml/app files
    • /etc/puppet/yaml/app/fxa.stage.yaml
    • /etc/puppet/yaml/app/fxa.yaml

• Processes
  • Search for node, heka, nginx, circus

• Logs
  • /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
  • /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
  • /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
  • /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
  • /media/ephemeral0/nginx/logs/default.access.log (not in use)
  • /media/ephemeral0/nginx/logs/default.error.log (not in use)
  • /media/ephemeral0/squid/access.log
  • /var/log/circus.log
  • /var/log/hekad/fxa-browserid_verifier.stderr.log
  • /var/log/hekad/fxa-browserid_verifier.stdout.log

• Hekad
  • /etc/puppet/modules/hekad
  • /data/hekad

• QA Access via a Bastion Host
  • Old Dev IAM
    • SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance.
  • New Dev IAM
    • SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance.

• Firefox Configs
  • services.sync.clusterURL should get automatically set by the TokenServer
  • services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

• Quick verifications

Browser: https://verifier.stage.mozaws.net/
returns "Method Not Allowed"

$ curl https://verifier.stage.mozaws.net
returns "Method Not Allowed"

$ curl -I https://verifier.stage.mozaws.net
HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache, max-age=0
Content-length: 18
Content-Type: text/plain
Date: Wed, 23 Jul 2014 20:19:40 GMT
Connection: keep-alive

Sync 1.5 Stage Environment

• This is a work in progress. Right now, there are only a small number of sync nodes (instances) in AWS.
• URLs
  • https://sync-1-us-east-1.stage.mozaws.net
  • https://sync-2-us-east-1.stage.mozaws.net
  • https://sync-3-us-east-1.stage.mozaws.net
  • https://sync-4-us-east-1.stage.mozaws.net
  • and sometimes
  • https://sync-5-us-east-1.stage.mozaws.net
  • https://sync-6-us-east-1.stage.mozaws.net

• Versions

Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

• AWS
  • Search for sync node instances in US East: "stage-sync-node-X"
  • Each node is a specific large instance (mixed m3 and c3)

• Files
  • /data/server-syncstorage/*
  • /var/log/nginx
  • /etc/puppet

• Processes
  • Search for sync, mysql, circusd, hekad, nginx, memcached

• Logs
  • /media/ephemeral0/logs/
  • /media/ephemeral0/logs/nginx/access.log
  • /media/ephemeral0/logs/nginx/error.log
  • /media/ephemeral0/logs/sync/sync.err
  • /media/ephemeral0/logs/sync/sync.log
  • /var/log/circus.log
  • /var/log/hekad/sync_1_5.stderr.log
  • /var/log/hekad/sync_1_5.stdout.log

• Hekad
  • /etc/heka.d/sync_1_5.toml

• QA Access via a Bastion Host
  • SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

• Firefox Configs
  • services.sync.clusterURL should get automatically set by the TokenServer

• Note: There is no longer a Sync 1.1 Stage environment.

Loads V1 Services Cluster Environment

• Details on the Loads V1 cluster are available at this link:
  • https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

Monitoring the Stage Environment

• Stackdriver:
  • Stage main: https://app.stackdriver.com
  • https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
  • https://app.stackdriver.com/groups/4389/stage-services-tag-sync15/stage-services-sync15/fxa-verifier
  • https://app.stackdriver.com/groups/4391/stage-services-tag-sync15/stage-services-tag-sync15/sync-storage
  • https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
  • https://app.stackdriver.com/monitoring/3827/tokenserver
  • https://app.stackdriver.com/instances/<AWS instance>
  • Also, make use of the Services, Groups, and Dashboards for more specific links
    • Example: Services > https://app.stackdriver.com/services/nginx
    • Example: Groups > https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver

• Graphite:
  • https://graphite.shared.us-east-1.stage.mozaws.net

• Kibana
  • https://kibana.shared.us-east-1.stage.mozaws.net/
  • https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
  • https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_errors.json
  • https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
  • https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_app_logs.json
  • https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/tokenserver_http_status.json

• Heka
  • https://heka.shared.us-east-1.stage.mozaws.net/
  • or https://heka.shared.us-east-1.stage.mozaws.net/#health
  • Filters
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/TokenServerHTTPStatus
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
  • Outputs
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxErrorFileOutput
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-SlowQueryFileOutput
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerLocalVerifierFileOutput
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-FileOutput
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxFileOutput
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerFileOutput
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerNginxFileOutput
  • Encoders
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-FileOutput-ProtobufEncoder
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerNginxFileOutput-ProtobufEncoder
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-SlowQueryFileOutput-ProtobufEncoder
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerLocalVerifierFileOutput-ProtobufEncoder
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerFileOutput-ProtobufEncoder
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxFileOutput-ProtobufEncoder
    • https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxErrorFileOutput-ProtobufEncoder
  • Sandboxes
    • https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
    • https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
    • https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagemetaglobal.cbuf

Monitoring the Production Environment

• Heka shared:
  • Main: https://heka.shared.us-west-2.prod.mozaws.net/
  • https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
  • https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
  • https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
  • https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
  • https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
  • https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/TokenServerHTTPStatus

• Kibana shared:
  • https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/file/default.json

• StackDriver:
  • Main: https://app.stackdriver.com/

TokenServer, Verifier, and Sync 1.5 Dev Environments

• URLs
  • https://sync1.dev.lcip.org/
  • https://token.dev.lcip.org/

• Versions

TokenServer:
rpm -qa | grep token
Example: tokenserver-svcops <latest version>

Verifier:
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>

Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

• AWS
  • This is part of the dev-lcip-org CloudFormation stack
    • TokenServer: Search for the fxa-tokenserver instance
    • SyncServer2: Search for the fxa-syncstorage instance
    • SyncServer1: Search for the fxa-syncstorage instance
    • LogServer: Search for the fxa-logbox instance
• Files
  • TBD
• Processes
  • TBD
• Logs
  • TBD
• QA Access
  • SSH with AWS keys to the various instances
• Firefox configs:
  • services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
  • services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5

OPs Mana and GitHub Pages

• NOTE: Talk to OPs for the links to Mana.
• Puppet Config: https://github.com/mozilla-services/puppet-config
• Cloud Formation: https://github.com/mozilla-services/svcops
  • and https://github.com/mozilla-services/svcops/tree/master/cloudformations/sync
  • and https://github.com/mozilla-services/svcops/tree/master/cloudformations/token