TestEngineering/Services/TSVerifierSyncTestEnvironments: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 253: Line 253:
** https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** https://app.stackdriver.com/instances/<AWS instance>
** https://app.stackdriver.com/instances/<AWS instance>
** Also, make use of the Services, Groups, and Dashboards for more specific links
*** Example: Groups > https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
*** Example: Dashboards > https://app.stackdriver.com/monitoring/3827/tokenserver


* Graphite:
* Graphite:

Revision as of 17:35, 12 August 2014

Tokenserver, Verifier, and Sync Production Environments

  • QA Access: not allowed

TokenServer Stage Environment

  • Version:
TokenServer:
rpm -qa | grep token
Example: tokenserver-svcops <latest version>
  • AWS
    • Shared:
      • shared-elasticsearch
      • shared-elasticsearch
      • shared-elasticsearch
      • shared-elasticsearch
      • shared-rabbitmq
      • shared-rabbitmq
      • shared-bastion
      • shared-heka
    • tokenserver app server (2 m3.medium instances behind a CF stack and ELB)
    • tokenserver db (1 large DB instance behind RDS)
  • Files
    • /data/tokenserver/*
  • Processes
    • tokenserver app server:
      • Search for token, circus, nginx, gunicorn
  • Logs
    • /media/ephemeral0/logs/
    • /media/ephemeral0/logs/nginx/access.log
    • /media/ephemeral0/logs/nginx/error.log
    • /media/ephemeral0/logs/tokenserver/token.error.log
    • /media/ephemeral0/logs/tokenserver/token.log.*
    • /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
    • /media/ephemeral0/logs/tokenserver/process_account_deletions.log
    • /media/ephemeral0/squid/access.log
    • /var/log/hekad/tokenserver.stdout.log
    • /var/log/hekad/tokenserver.stderr.log
  • Hekad
    • /etc/puppet/modules/hekad
  • QA Access via a Bastion Host
    • SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

Verifier Stage Environment

  • Version:
Verifier:
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>
  • AWS
    • Shared:
      • shared-elasticsearch
      • shared-elasticsearch
      • shared-elasticsearch
      • shared-elasticsearch
      • shared-rabbitmq
      • shared-rabbitmq
      • shared-bastion
      • shared-heka
    • fxa-browserid_verifier (usually a single instance)
  • Files
    • /data/fxa-browserid-verifier
    • /data/hekad
    • /etc/puppet
    • /etc/heka.d
    • /media/ephemeral0/*
    • /opt/aws
    • /opt/ec2
    • /opt/openresty
    • /opt/rh
    • /opt/stackdriver
  • Processes
    • Search for node, heka, nginx, circus
  • Logs
    • /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
    • /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
    • /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
    • /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
    • /media/ephemeral0/nginx/logs/default.access.log (not in use)
    • /media/ephemeral0/nginx/logs/default.error.log (not in use)
    • /media/ephemeral0/squid/access.log
    • /var/log/circus.log
    • /var/log/hekad/fxa-browserid_verifier.stderr.log
    • /var/log/hekad/fxa-browserid_verifier.stdout.log
  • Hekad
    • /etc/puppet/modules/hekad
    • /data/hekad
  • QA Access via a Bastion Host
    • SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.
  • Quick verifications
Browser: https://verifier.stage.mozaws.net/
returns "Method Not Allowed"

$ curl https://verifier.stage.mozaws.net
returns "Method Not Allowed"

$ curl -I https://verifier.stage.mozaws.net
HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache, max-age=0
Content-length: 18
Content-Type: text/plain
Date: Wed, 23 Jul 2014 20:19:40 GMT
Connection: keep-alive

Sync 1.5 Stage Environment

  • Versions
Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>
  • AWS
    • Search for sync node instances in US East: "stage-sync-node-X"
    • Each node is a specific large instance (mixed m3 and c3)
  • Files
    • /data/server-syncstorage/*
    • /var/log/nginx
  • Processes
    • Search for sync, mysql, circusd, hekad, nginx, memcached
  • Logs
    • /media/ephemeral0/logs (most important)
      • nginx/access.log
      • (what about error.log?)
      • sync/sync.err
      • sync/sync.log
    • /var/log/nginx
    • /var/log/circus.log
  • Hekad
    • /etc/heka.d/sync_1_5.toml
  • QA Access via a Bastion Host
    • SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.
  • Firefox Configs
    • services.sync.clusterURL should get automatically set by the TokenServer
  • Note: There is no longer a Sync 1.1 Stage environment.

Loads Services Cluster Environment

  • Versions
Loads Cluster/Broker/Agents:
$ cd /home/ubuntu/loads/bin
$ ./loads-runner --version
  • AWS in US West
    • loads-master (broker and agent processes)
    • loads-slave-1 (agent processes)
    • loads-slave-2 (agent processes)
    • NOTE: there is no stack or ELB for this cluster
  • Files
    • /home/ubuntu
      • loads
      • loads-aws
      • loads-web
  • Processes
    • Search for processes owned by ubuntu, loads, nginx, circus
  • Logs
    • /var/log/redis
    • /var/log/nginx
  • QA access
    • You need special access to be able to SSH into these devices
    • You need to make some changes to your .ssh/config file
  • Monitoring
  • Cluster status
    • Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents

Monitoring the Stage Environment

Monitoring the Production Environment

TokenServer and Sync 1.5 Dev Environments

  • Versions
TokenServer:
rpm -qa | grep token
Example: tokenserver-svcops <latest version>

Verifier:
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>

Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>
  • AWS
    • This is part of the dev-lcip-org CloudFormation stack
      • TokenServer: Search for the fxa-tokenserver instance
      • SyncServer2: Search for the fxa-syncstorage instance
      • SyncServer1: Search for the fxa-syncstorage instance
      • LogServer: Search for the fxa-logbox instance
  • Files
    • TBD
  • Processes
    • TBD
  • Logs
    • TBD
  • QA Access
    • SSH with AWS keys to the various instances
  • Firefox configs:

OPs Mana and GitHub Pages

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/TSVerifierSyncTestEnvironments&oldid=1005036"