MozillaWiki

TestEngineering/Services/TSVerifierSyncTestEnvironments: Difference between revisions

Page Discussion

TestEngineering/Services/TSVerifierSyncTestEnvironments (view source)

Revision as of 20:05, 26 August 2016

5,523 bytes added ,  26 August 2016
m
StuartPhilp moved page QA/Services/TSVerifierSyncTestEnvironments to TestEngineering/Services/TSVerifierSyncTestEnvironments
No edit summary
 
(34 intermediate revisions by one other user not shown)
Line 1: Line 1:
* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier Stage section below...
** A Tokenserver+Verifier stack: See the TokenServer Stage section below...
== Tokenserver, Verifier, and Sync Production Environments ==
== Tokenserver, Verifier, and Sync Production Environments ==
* TokenServer: https://token.services.mozilla.com
* TokenServer: https://token.services.mozilla.com
* Verifier: https://verifier.accounts.firefox.com
* Sync 1.5 Nodes (for now in AWS):
* Sync 1.5 Nodes (for now in AWS):
** sync-1-us-east-2.sync.services.mozilla.com
** sync-1-us-east-2.sync.services.mozilla.com
Line 19: Line 24:
*** identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...
*** identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...


== TokenServer Stage Environment ==
== TokenServer+Verifier Stage Environment ==
* NOTE: In an upcoming deployent, the Verifier functionality will be rolled into the Tokenserver stack of services.
 
* URLs
* URLs
** TokenServer: https://token.stage.mozaws.net
** TokenServer: https://token.stage.mozaws.net
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com


* Version:
* Version:
  TokenServer:
  TokenServer:
  rpm -qa | grep token
  rpm -qa | grep -E 'token|browser'
Example: tokenserver-svcops <latest version>


* AWS
* AWS
** Shared:
** tokenserver app server or Token app server or similar
*** shared-elasticsearch
*** (1 or more instances behind a CF stack and ELB)
*** shared-elasticsearch
** tokenserver db (1 large DB instance behind RDS)
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq 
*** shared-bastion
*** shared-heka


** tokenserver app server          (2 m3.medium instances behind a CF stack and ELB)
* Files
** tokenserver db                  (1 large DB instance behind RDS)
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/tokenserver
** /data/fxa-browserid-verifier
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /etc/heka.d
** /etc/puppet
*** esp. /etc/puppet/yaml/app
*** fxa.stage.yaml
*** fxa.yaml
*** tokenserver.stage.yaml
*** tokenserver.yaml


* Files
** /data/tokenserver/*
* Processes
* Processes
** tokenserver app server:
** tokenserver app server:
*** Search for token, circus, nginx, gunicorn
*** Search for token, circus, nginx, gunicorn, python, hekad, node


* Logs
* Logs
** /media/ephemeral0/logs/
** /media/ephemeral0/logs/nginx/access.log
** /media/ephemeral0/logs/nginx/error.log
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/squid/access.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log
** /var/log/hekad/tokenserver.stderr.log
Line 64: Line 84:


* QA Access via a Bastion Host
* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.
** Old Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance.
** New Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance.


* Firefox Configs
* Firefox Configs
Line 71: Line 94:


== Verifier Stage Environment ==
== Verifier Stage Environment ==
* NOTE: In an upcoming deployment, the Verifier functionality will be rolled into the Tokenserver stack of services.
* URLs
* URLs
** Verifier: https://verifier.stage.mozaws.net
** Verifier: https://verifier.stage.mozaws.net
** IdP: https://mockmyid.s3-us-west-2.amazonaws.com/
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com
** OLD IdP: https://mockmyid.com/


* Version:  
* Version:
Verifier:
** rpm -qa | grep verifier
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>


* AWS
* AWS
Line 92: Line 114:
*** shared-heka
*** shared-heka


** fxa-browserid-verifier          (2 c3.large instances behind a CF stack and ELB)
** fxa-browserid_verifier or fxa-bv-stage (usually a single instance)


* Files
* Files
** /data/fxa-browserid-verifier
** /data/fxa-browserid-verifier
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/heka.d
** /etc/puppet
** /etc/puppet
** /etc/heka.d
*** esp. the /etc/puppet/yaml/app files
** /media/ephemeral0/*
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa.yaml


* Processes
* Processes
** fxa-browserid-verifer:
** Search for node, heka, nginx, circus
*** Search for node, heka, nginx, circus


* Logs
* Logs
Line 110: Line 139:
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/squid/access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log
** /var/log/nginx/access.log
** /var/log/nginx/error.log


* Hekad
* Hekad
** /etc/puppet/modules/hekad
** /etc/puppet/modules/hekad
** /data/hekad


* QA Access via a Bastion Host
* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.
** Old Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance.
** New Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance.


* Firefox Configs
* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5
* Quick verifications
Browser: https://verifier.stage.mozaws.net/
returns "Method Not Allowed"
$ curl https://verifier.stage.mozaws.net
returns "Method Not Allowed"
$ curl -I https://verifier.stage.mozaws.net
HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache, max-age=0
Content-length: 18
Content-Type: text/plain
Date: Wed, 23 Jul 2014 20:19:40 GMT
Connection: keep-alive


== Sync 1.5 Stage Environment ==
== Sync 1.5 Stage Environment ==
Line 134: Line 182:
** https://sync-3-us-east-1.stage.mozaws.net
** https://sync-3-us-east-1.stage.mozaws.net
** https://sync-4-us-east-1.stage.mozaws.net
** https://sync-4-us-east-1.stage.mozaws.net
** and sometimes
** https://sync-5-us-east-1.stage.mozaws.net
** https://sync-5-us-east-1.stage.mozaws.net
** https://sync-6-us-east-1.stage.mozaws.net
** https://sync-6-us-east-1.stage.mozaws.net
Line 145: Line 194:
** Search for sync node instances in US East: "stage-sync-node-X"
** Search for sync node instances in US East: "stage-sync-node-X"
** Each node is a specific large instance (mixed m3 and c3)
** Each node is a specific large instance (mixed m3 and c3)
* Files
* Files
** /data/server-syncstorage/*
** /data/server-syncstorage/*
** /var/log/nginx
** /var/log/nginx
** /etc/puppet
* Processes
* Processes
** Search for sync, mysql, circusd, hekad, nginx, memcached
** Search for sync, mysql, circusd, hekad, nginx, memcached
* Logs
* Logs
** /media/ephemeral0/logs (most important)
** /media/ephemeral0/logs/
*** nginx/access.log
** /media/ephemeral0/logs/nginx/access.log
*** (what about error.log?)
** /media/ephemeral0/logs/nginx/error.log
*** sync/sync.err
** /media/ephemeral0/logs/sync/sync.err
*** sync/sync.log
** /media/ephemeral0/logs/sync/sync.log
** /var/log/nginx
** /var/log/circus.log
** /var/log/circus.log
** /var/log/hekad/sync_1_5.stderr.log
** /var/log/hekad/sync_1_5.stdout.log
* Hekad
* Hekad
** /etc/heka.d/sync_1_5.toml
** /etc/heka.d/sync_1_5.toml
* QA Access via a Bastion Host
* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.
Line 168: Line 224:
* Note: There is no longer a Sync 1.1 Stage environment.
* Note: There is no longer a Sync 1.1 Stage environment.


== Loads Services Cluster Environment ==
== Loads V1 Services Cluster Environment ==
* URLs
* Details on the Loads V1 cluster are available at this link:
** http://loads.services.mozilla.com/
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com/


* Versions
== Monitoring the Stage Environment ==
Loads Cluster/Broker/Agents:
* Stackdriver:
$ cd /home/ubuntu/loads/bin
** Stage main: https://app.stackdriver.com
$ ./loads-runner --version
** https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** https://app.stackdriver.com/groups/4389/stage-services-tag-sync15/stage-services-sync15/fxa-verifier
** https://app.stackdriver.com/groups/4391/stage-services-tag-sync15/stage-services-tag-sync15/sync-storage
** https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
** https://app.stackdriver.com/monitoring/3827/tokenserver
** https://app.stackdriver.com/instances/<AWS instance>
** Also, make use of the Services, Groups, and Dashboards for more specific links
*** Example: Services > https://app.stackdriver.com/services/nginx
*** Example: Groups > https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver


* AWS in US West
* Graphite:
** loads-master (broker and agent processes)
** https://graphite.shared.us-east-1.stage.mozaws.net
** loads-slave-1 (agent processes)
** loads-slave-2 (agent processes)
** NOTE: there is no stack or ELB for this cluster
 
* Files
** /home/ubuntu
*** loads
*** loads-aws
*** loads-web
* Processes
** Search for processes owned by ubuntu, loads, nginx, circus
* Logs
** /var/log/redis
** /var/log/nginx
* QA access
** You need special access to be able to SSH into these devices
** You need to make some changes to your .ssh/config file
* Monitoring
** Web Interface: http://loads.services.mozilla.com/
** StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* Links
** http://loads.readthedocs.org/en/latest/
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
 
== Monitoring the Stage Environment ==


* Kibana
** https://kibana.shared.us-east-1.stage.mozaws.net/
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_errors.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_app_logs.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/tokenserver_http_status.json


* Heka
** https://heka.shared.us-east-1.stage.mozaws.net/
** or https://heka.shared.us-east-1.stage.mozaws.net/#health
** Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/TokenServerHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-SlowQueryFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerLocalVerifierFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-FileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerNginxFileOutput
** Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-FileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-SlowQueryFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerLocalVerifierFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxErrorFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagemetaglobal.cbuf


== Monitoring the Production Environment ==
== Monitoring the Production Environment ==
Monitoring
* Heka shared:
* Heka shared:
** https://heka.shared.us-west-2.prod.mozaws.net/
** Main: https://heka.shared.us-west-2.prod.mozaws.net/
** See the OPs Mana page for more page details
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/TokenServerHTTPStatus


* Kibana shared:  
* Kibana shared:  
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/file/default.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/file/default.json
** See the OPs Mana page for more page details


* StackDriver:
* StackDriver:
** Main: https://app.stackdriver.com/
** Main: https://app.stackdriver.com/
** See the OPs Mana page for more page details


== TokenServer and Sync 1.5 Dev Environments ==
== TokenServer, Verifier, and Sync 1.5 Dev Environments ==
* URLs
* URLs
** https://sync1.dev.lcip.org/
** https://sync1.dev.lcip.org/
StuartPhilp
81

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/998111...1145614"