TestEngineering/Services/TSVerifierSyncTestEnvironments: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 232: Line 232:


== Monitoring the Stage Environment ==
== Monitoring the Stage Environment ==
* Stackdriver:
** Stage main: https://app.stackdriver.com
** https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** https://app.stackdriver.com/groups/4389/stage-services-tag-sync15/stage-services-sync15/fxa-verifier
** https://app.stackdriver.com/groups/4391/stage-services-tag-sync15/stage-services-tag-sync15/sync-storage
** https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
** https://app.stackdriver.com/monitoring/3827/tokenserver
** https://app.stackdriver.com/instances/<AWS instance>
** Also, make use of the Services, Groups, and Dashboards for more specific links
*** Example: Services > https://app.stackdriver.com/services/nginx
*** Example: Groups > https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
* Graphite:
** https://graphite.shared.us-east-1.stage.mozaws.net
* Kibana
* Kibana
** https://kibana.shared.us-east-1.stage.mozaws.net/
** https://kibana.shared.us-east-1.stage.mozaws.net/
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Token%20App%20Logs%20POC
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20Nginx%20Errors
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_errors.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20App%20Logs
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_app_logs.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/tokenserver_http_status.json


* Heka
* Heka
Line 250: Line 265:
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagebookmarks.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagebookmarks.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storageforms.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storageforms.cbuf
* Stackdriver:
** Stage main: https://app.stackdriver.com
** https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** https://app.stackdriver.com/instances/<AWS instance>
** Also, make use of the Services, Groups, and Dashboards for more specific links
*** Example: Groups > https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
*** Example: Dashboards > https://app.stackdriver.com/monitoring/3827/tokenserver
* Graphite:
** https://graphite.shared.us-east-1.stage.mozaws.net


== Monitoring the Production Environment ==
== Monitoring the Production Environment ==

Revision as of 02:33, 18 September 2014

  • NOTE: We currently have two Verifier stacks in Stage (and probably Production):
    • The standalone Browser_ID Verifier stack: See that Verifier Stage section below...
    • A Tokenserver+Verifier stack: See the TokenServer Stage section below...

Tokenserver, Verifier, and Sync Production Environments

  • QA Access: not allowed

TokenServer+Verifier Stage Environment

  • NOTE: In an upcoming deployent, the Verifier functionality will be rolled into the Tokenserver stack of services.
  • Version:
TokenServer:
rpm -qa | grep -E 'token|browser'
  • AWS
    • Shared:
      • shared-elasticsearch
      • shared-elasticsearch
      • shared-elasticsearch
      • shared-elasticsearch
      • shared-rabbitmq
      • shared-rabbitmq
      • shared-bastion
      • shared-heka
    • tokenserver app server (1 or more instances behind a CF stack and ELB)
    • tokenserver db (1 large DB instance behind RDS)
  • Files
    • /opt/aws
    • /opt/ec2
    • /opt/openresty
    • /opt/rh
    • /opt/stackdriver
    • /data/tokenserver
      • esp. /data/fxa-browserid-verifier/config/settings.json
    • /data/hekad
    • /data/fxa-browserid-verifier
    • /etc/heka.d
    • /etc/puppet
      • esp. /etc/puppet/yaml/app
      • /etc/puppet/yaml/app/fxa.stage.yaml
      • /etc/puppet/yaml/app/fxa_verifier.stage.yaml
      • /etc/puppet/yaml/app/tokenserver.stage.yaml
      • /etc/puppet/yaml/app/tokenserver.yaml
  • Processes
    • tokenserver app server:
      • Search for token, circus, nginx, gunicorn, python, hekad, node
  • Logs
    • /media/ephemeral0/logs/tokenserver/token.error.log
    • /media/ephemeral0/logs/tokenserver/token.log.*
    • /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
    • /media/ephemeral0/logs/tokenserver/process_account_deletions.log
    • /media/ephemeral0/logs/tokenserver/purge_old_records.log
    • /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
    • /media/ephemeral0/nginx/logs/default.access.log (not in use)
    • /media/ephemeral0/nginx/logs/default.error.log (not in use)
    • /media/ephemeral0/nginx/logs/tokenserver.access.log
    • /media/ephemeral0/nginx/logs/tokenserver.error.log
    • /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
    • /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
    • /var/log/circus.log
    • /var/log/hekad/tokenserver.stdout.log
    • /var/log/hekad/tokenserver.stderr.log
  • Hekad
    • /etc/puppet/modules/hekad
  • QA Access via a Bastion Host
    • SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

Verifier Stage Environment

  • NOTE: In an upcoming deployment, the Verifier functionality will be rolled into the Tokenserver stack of services.
  • Version:
    • rpm -qa | grep verifier
  • AWS
    • Shared:
      • shared-elasticsearch
      • shared-elasticsearch
      • shared-elasticsearch
      • shared-elasticsearch
      • shared-rabbitmq
      • shared-rabbitmq
      • shared-bastion
      • shared-heka
    • fxa-browserid_verifier or fxa-bv-stage (usually a single instance)
  • Files
    • /data/fxa-browserid-verifier
      • esp. /data/fxa-browserid-verifier/config/settings.json
    • /data/hekad
    • /opt/aws
    • /opt/ec2
    • /opt/openresty
    • /opt/rh
    • /opt/stackdriver
    • /etc/heka.d
    • /etc/puppet
      • esp. the /etc/puppet/yaml/app files
      • /etc/puppet/yaml/app/fxa.stage.yaml
      • /etc/puppet/yaml/app/fxa.yaml
  • Processes
    • Search for node, heka, nginx, circus
  • Logs
    • /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
    • /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
    • /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
    • /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
    • /media/ephemeral0/nginx/logs/default.access.log (not in use)
    • /media/ephemeral0/nginx/logs/default.error.log (not in use)
    • /media/ephemeral0/squid/access.log
    • /var/log/circus.log
    • /var/log/hekad/fxa-browserid_verifier.stderr.log
    • /var/log/hekad/fxa-browserid_verifier.stdout.log
  • Hekad
    • /etc/puppet/modules/hekad
    • /data/hekad
  • QA Access via a Bastion Host
    • SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.
  • Quick verifications
Browser: https://verifier.stage.mozaws.net/
returns "Method Not Allowed"

$ curl https://verifier.stage.mozaws.net
returns "Method Not Allowed"

$ curl -I https://verifier.stage.mozaws.net
HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache, max-age=0
Content-length: 18
Content-Type: text/plain
Date: Wed, 23 Jul 2014 20:19:40 GMT
Connection: keep-alive

Sync 1.5 Stage Environment

  • Versions
Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>
  • AWS
    • Search for sync node instances in US East: "stage-sync-node-X"
    • Each node is a specific large instance (mixed m3 and c3)
  • Files
    • /data/server-syncstorage/*
    • /var/log/nginx
    • /etc/puppet
  • Processes
    • Search for sync, mysql, circusd, hekad, nginx, memcached
  • Logs
    • /media/ephemeral0/logs/
    • /media/ephemeral0/logs/nginx/access.log
    • /media/ephemeral0/logs/nginx/error.log
    • /media/ephemeral0/logs/sync/sync.err
    • /media/ephemeral0/logs/sync/sync.log
    • /var/log/circus.log
    • /var/log/hekad/sync_1_5.stderr.log
    • /var/log/hekad/sync_1_5.stdout.log
  • Hekad
    • /etc/heka.d/sync_1_5.toml
  • QA Access via a Bastion Host
    • SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.
  • Firefox Configs
    • services.sync.clusterURL should get automatically set by the TokenServer
  • Note: There is no longer a Sync 1.1 Stage environment.

Loads V1 Services Cluster Environment

Monitoring the Stage Environment

https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_app_logs.json

Monitoring the Production Environment

TokenServer, Verifier, and Sync 1.5 Dev Environments

  • Versions
TokenServer:
rpm -qa | grep token
Example: tokenserver-svcops <latest version>

Verifier:
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>

Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>
  • AWS
    • This is part of the dev-lcip-org CloudFormation stack
      • TokenServer: Search for the fxa-tokenserver instance
      • SyncServer2: Search for the fxa-syncstorage instance
      • SyncServer1: Search for the fxa-syncstorage instance
      • LogServer: Search for the fxa-logbox instance
  • Files
    • TBD
  • Processes
    • TBD
  • Logs
    • TBD
  • QA Access
    • SSH with AWS keys to the various instances
  • Firefox configs:

OPs Mana and GitHub Pages

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/TSVerifierSyncTestEnvironments&oldid=1016139"