MozillaWiki - User contributions [en]

TestEngineering/Services/TSVerifierSyncTestEnvironments

2014-10-29T21:15:53Z

Jbonacci: /* TokenServer+Verifier Stage Environment */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier Stage section below...
** A Tokenserver+Verifier stack: See the TokenServer Stage section below...

== Tokenserver, Verifier, and Sync Production Environments ==
* TokenServer: https://token.services.mozilla.com
* Verifier: https://verifier.accounts.firefox.com
* Sync 1.5 Nodes (for now in AWS):
** sync-1-us-east-2.sync.services.mozilla.com
** sync-2-us-east-2.sync.services.mozilla.com
*** There are many more than this now...

* QA Access: not allowed

* Pointing Fx Nightly to Production FxA/Sync:
** Set/Verify the following Firefox configs:
*** services.sync.clusterURL = (should get automatically set by the TokenServer)
*** services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
*** services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
*** services.sync.log.appender.file.logOnError = Yes
*** services.sync.log.appender.file.logOnSuccess = Yes
*** services.sync.log.appender.file.level = Trace
*** identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
*** identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...

== TokenServer+Verifier Stage Environment ==
* NOTE: In an upcoming deployent, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** TokenServer: https://token.stage.mozaws.net
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
TokenServer:
rpm -qa | grep -E 'token|browser'

* AWS
** tokenserver app server or Token app server or similar
*** (1 or more instances behind a CF stack and ELB)
** tokenserver db (1 large DB instance behind RDS)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/tokenserver
** /data/fxa-browserid-verifier
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /etc/heka.d
** /etc/puppet
*** esp. /etc/puppet/yaml/app
*** fxa.stage.yaml
*** fxa.yaml
*** tokenserver.stage.yaml
*** tokenserver.yaml

* Processes
** tokenserver app server:
*** Search for token, circus, nginx, gunicorn, python, hekad, node

* Logs
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* Hekad
** /etc/puppet/modules/hekad

* QA Access via a Bastion Host
** Old Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance.
** New Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

== Verifier Stage Environment ==
* NOTE: In an upcoming deployment, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
** rpm -qa | grep verifier

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** fxa-browserid_verifier or fxa-bv-stage (usually a single instance)

* Files
** /data/fxa-browserid-verifier
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/heka.d
** /etc/puppet
*** esp. the /etc/puppet/yaml/app files
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa.yaml

* Processes
** Search for node, heka, nginx, circus

* Logs
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* Hekad
** /etc/puppet/modules/hekad
** /data/hekad

* QA Access via a Bastion Host
** Old Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance.
** New Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

* Quick verifications
Browser: https://verifier.stage.mozaws.net/
returns "Method Not Allowed"

$ curl https://verifier.stage.mozaws.net
returns "Method Not Allowed"

$ curl -I https://verifier.stage.mozaws.net
HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache, max-age=0
Content-length: 18
Content-Type: text/plain
Date: Wed, 23 Jul 2014 20:19:40 GMT
Connection: keep-alive

== Sync 1.5 Stage Environment ==
* This is a work in progress. Right now, there are only a small number of sync nodes (instances) in AWS.
* URLs
** https://sync-1-us-east-1.stage.mozaws.net
** https://sync-2-us-east-1.stage.mozaws.net
** https://sync-3-us-east-1.stage.mozaws.net
** https://sync-4-us-east-1.stage.mozaws.net
** and sometimes
** https://sync-5-us-east-1.stage.mozaws.net
** https://sync-6-us-east-1.stage.mozaws.net

* Versions
Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** Search for sync node instances in US East: "stage-sync-node-X"
** Each node is a specific large instance (mixed m3 and c3)

* Files
** /data/server-syncstorage/*
** /var/log/nginx
** /etc/puppet

* Processes
** Search for sync, mysql, circusd, hekad, nginx, memcached

* Logs
** /media/ephemeral0/logs/
** /media/ephemeral0/logs/nginx/access.log
** /media/ephemeral0/logs/nginx/error.log
** /media/ephemeral0/logs/sync/sync.err
** /media/ephemeral0/logs/sync/sync.log
** /var/log/circus.log
** /var/log/hekad/sync_1_5.stderr.log
** /var/log/hekad/sync_1_5.stdout.log

* Hekad
** /etc/heka.d/sync_1_5.toml

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer

* Note: There is no longer a Sync 1.1 Stage environment.

== Loads V1 Services Cluster Environment ==
* Details on the Loads V1 cluster are available at this link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Monitoring the Stage Environment ==
* Stackdriver:
** Stage main: https://app.stackdriver.com
** https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** https://app.stackdriver.com/groups/4389/stage-services-tag-sync15/stage-services-sync15/fxa-verifier
** https://app.stackdriver.com/groups/4391/stage-services-tag-sync15/stage-services-tag-sync15/sync-storage
** https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
** https://app.stackdriver.com/monitoring/3827/tokenserver
** https://app.stackdriver.com/instances/<AWS instance>
** Also, make use of the Services, Groups, and Dashboards for more specific links
*** Example: Services > https://app.stackdriver.com/services/nginx
*** Example: Groups > https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver

* Graphite:
** https://graphite.shared.us-east-1.stage.mozaws.net

* Kibana
** https://kibana.shared.us-east-1.stage.mozaws.net/
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_errors.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_app_logs.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/tokenserver_http_status.json

* Heka
** https://heka.shared.us-east-1.stage.mozaws.net/
** or https://heka.shared.us-east-1.stage.mozaws.net/#health
** Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/TokenServerHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-SlowQueryFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerLocalVerifierFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-FileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerNginxFileOutput
** Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-FileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-SlowQueryFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerLocalVerifierFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxErrorFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagemetaglobal.cbuf

== Monitoring the Production Environment ==
* Heka shared:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/TokenServerHTTPStatus

* Kibana shared:
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/file/default.json

* StackDriver:
** Main: https://app.stackdriver.com/

== TokenServer, Verifier, and Sync 1.5 Dev Environments ==
* URLs
** https://sync1.dev.lcip.org/
** https://token.dev.lcip.org/

* Versions
TokenServer:
rpm -qa | grep token
Example: tokenserver-svcops <latest version>

Verifier:
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>

Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** This is part of the dev-lcip-org CloudFormation stack
*** TokenServer: Search for the fxa-tokenserver instance
*** SyncServer2: Search for the fxa-syncstorage instance
*** SyncServer1: Search for the fxa-syncstorage instance
*** LogServer: Search for the fxa-logbox instance
* Files
** TBD
* Processes
** TBD
* Logs
** TBD
* QA Access
** SSH with AWS keys to the various instances
* Firefox configs:
** services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
** services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/sync
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/token

TestEngineering/Services/LoopTestEnvironments

2014-10-29T20:11:40Z

Jbonacci: /* MSISDN Mock Server Stage Details */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

In the browser: https://loop.services.mozilla.com/__heartbeat__
{"storage":true,"provider":true}

In the browser: https://loop.services.mozilla.com/push-server-config
{"pushServerURI":"wss://push.services.mozilla.com/"}

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Firefox Hello web client.

Browser: https://call.mozilla.com/config.js
now returns all of this info:
var loop = loop || {};
loop.config = {
bunch of loop-client specific info
};

Browser: https://call.mozilla.com/VERSION.txt

TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
In the browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD
** loop-redis-stage - need the details
** Loop-Client: https://call.stage.mozaws.net
** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net
** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

In the browser: https://loop.stage.mozaws.net/__heartbeat__

In the browser: https://loop.stage.mozaws.net/push-server-config

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for four loop-client-* stacks
** ELB: loopstage1-loo-ELB-<some value>
** One loop-client instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4 or 6.5

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad/loop-client.stderr.log
** /var/log/hekad/loop-client.stderr.log
** /media/ephemeral0/nginx/logs
*** default.access.log
*** default.error.log
*** loop-client.access.log
*** loop-client.error.log

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net
Welcome to the Firefox Hello web client.

Check the version: https://call.stage.mozaws.net/VERSION.txt

Check the TOS: https://call.stage.mozaws.net/legal/terms/

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {
serverUrl: 'https://loop.stage.mozaws.net',
and a bunch of other loop-client specific stuff
};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa | grep msisdn

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage or msisdnstage1
** ELB: msisdn-gateway-ELB-<SOME VALUE> or msisdnstage1-m-ELB-<SOME VALUE>
** Currently one m3.medium instance, look for msisdn-gateway

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway
** /etc/puppet/yaml/app, especially
*** msisdn_gateway.dev.yaml
*** msisdn_gateway.prod.yaml
*** msisdn_gateway.stage.yaml
*** msisdn_gateway.yaml

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/msisdn-gateway.access.log
** /media/ephemeral0/nginx/logs/msisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

In the browser: https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is in the new Dev IAM
** This is one t2.small
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** http://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* Old Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".
* New Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** https://app.stackdriver.com/groups/11042/loop-server/stage-loop

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Stackdriver
** https://app.stackdriver.com/groups/11044/loop-server/prod-loop

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopServerLoadTesting

2014-10-29T16:37:20Z

Jbonacci: /* Quick Verification Of Stage Deployments */

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
* Loop Server
Check the following in the browser:
https://loop.stage.mozaws.net
https://loop.stage.mozaws.net/__heartbeat__
https://loop.stage.mozaws.net/push-server-config
Or:
curl https://loop.stage.mozaws.net
curl -I https://loop.stage.mozaws.net
For now, just run a quick loadtest 'make test'
cd loop-server
cd loadtests
make test SERVER_URL=https://loop.stage.mozaws.net

* Loop Client
Check https://call.stage.mozaws.net/config.js
Should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.stage.mozaws.net'};
and
WIP from the client team: end to end tests
Also:
https://call.stage.mozaws.net
curl https://call.stage.mozaws.net
curl -I https://call.stage.mozaws.net

and now the TOS: https://call.stage.mozaws.net/legal/terms/

* MSISDN Gateway
In the browser: https://msisdn.stage.mozaws.net
or do the following from a command line:
curl https://msisdn.stage.mozaws.net
curl -I https://msisdn.stage.mozaws.net
and
For now, just run a quick loadtest 'make test'
cd msisdn-gateway
cd loadtests
make test SERVER_URL=https://msisdn.stage.mozaws.net
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Quick Verification of Production Deployments ==
* This is a quick sanity test of the environment for after each Production deployment.
* Loop Server
In the browser:
https://loop.services.mozilla.com
https://loop.services.mozilla.com/__heartbeat__
https://loop.services.mozilla.com/push-server-config
or do the following from a command line:
curl https://loop.services.mozilla.com
curl -I https://loop.services.mozilla.com

Then run a few 'make test' commands from the loadtests folder:
make test SERVER_URL=https://loop.services.mozilla.com
Note: this does hit a live third-party server

Then perform actual loop testing via desktop (Aurora/Nightly so far) and FxOS (2.1)
Verify that requests and strings point to Production environments

* Loop Client
In the browser: https://call.mozilla.com
should return "Welcome to the Loop web client."

In the browser: https://call.mozilla.com/config.js
should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

In the browser: https://call.mozilla.com/VERSION.txt
should return the version and build string info

and now the TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com
curl -I https://call.mozilla.com

Quick end-to-end tests:
Desktop: browser to browser
Desktop to FxOS
FxOS to Desktop
Two FxOS devices

* MSISDN Gateway
In the browser: https://msisdn.services.mozilla.com
or do the following from a command line:
curl https://msisdn.services.mozilla.com
curl -I https://msisdn.services.mozilla.com
Or
Run a single 'make test' command from the loadtests folder:
make test SERVER_URL=https://msisdn.services.mozilla.com
Note: this does hit a live third-party, so limit the check to a single run.
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Installing Loop-Server and the Loads tool on Localhost or AWS ==
* Installation:
git clone https://github.com/mozilla-services/loop-server.git
cd loop-server
Note: You may want to install a specific branch for testing vs defaulting to Master
npm install
ulimit -S -n 2048
npm test *
cd loadtests
make build
make test

Coverage report can be found here:
/loop-server/coverage/lcov-report/index.html

* This step requires the redis server to be installed and running:
Mac:
brew install redis
redis-server /usr/local/etc/redis.conf

Ubuntu Linux:
sudo apt-get install redis-server
sudo /usr/bin/redis-server /etc/redis/redis.conf
sudo tail -f /var/log/redis/redis-server.log

RHEL Linux:
Install redis from here: http://download.redis.io/releases
then
/usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
or similar

* Note: This will install a local copy of the Loads tool for use with the Loop-Server.

== Running the load test against the Loop-Server in Stage ==
* Stage environment:
$ cd loop-server/loadtests
$ make test
or
$ make test SERVER_URL=https://loop.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://loop.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the Loop-Server in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://loop.stage.mozaws.net

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing MSISDN-Gateway and the Loads tool on Localhost or AWS ==
* Installation:
** Install gmp, gmp-dev or gmp-devel
** Install ruby (very latest), ruby-dev or ruby-devel
** Install gem (required for fake_dynamo)
** Verify that gem is in your path
** Install redis-server to run the unit tests

* To install gmp
sudo yum -y install gmp, gmp-devel
or for Ubuntu
$ wget https://ftp.gnu.org/gnu/gmp/gmp-6.0.0a.tar.bz2
$ tar xvjf gmp-6.0.0a.tar.bz2
$ cd gmp-6.0.0
$ ./configure --prefix=/usr
$ make
$ make check
$ sudo make install

* To install ruby:
sudo yum -y install ruby, ruby-devel
or
sudo apt-get install ruby, ruby-dev

If this does not get you 1.9.3 or newer, then install manually:
Example:
$ wget http://cache.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p547.tar.gz
$ ./configure --prefix=/usr
$ make
$ sudo make install
(because for rhel, the default ruby version is 1.8.x.)
REF:
Main: https://www.ruby-lang.org/en/downloads/
Dev Tools: http://rubyinstaller.org/downloads/

* To install gem:
Grab rubygems from here: http://rubygems.org/pages/download
cd to rubygems directory
$ sudo ruby setup.rb

* To install fake_dynamo:
You should not have to install fake_dynamo since it is now part of the repo installer.
But if you do:
$ sudo gem install fake_dynamo
REF: https://github.com/ananthakumaran/fake_dynamo

* Install the msisdn-gateway repo:
$ git clone https://github.com/mozilla-services/msisdn-gateway.git
$ cd msisdn-gateway
Note: You may want to install a specific branch for testing vs defaulting to Master
$ sudo make install
(There is a bug open about the requirement to install with 'sudo')

* Note: This will install a local copy of the Loads tool for use with MSISDN-Gateway.

* Unit testing
Get redis-server installed
Start the server in a separate terminal or in the background with logging active
$ make test
The coverage report is here: msisdn-gateway/coverage/lcov-report/index.html

* Bugs:
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036158
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036174
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048451
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048456

== Running the load test against MSISDN-Gateway in Stage ==
* Building the load tests
$ cd loadtests
$ make build

* To load test the Stage environment:
$ make test SERVER_URL=https://msisdn.stage.mozaws.net
$ make bench SERVER_URL=https://msisdn.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the MSISDN-Gateway ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://msisdn.stage.mozaws.net

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (Loop-Server and MSISDN-Gateway):
*** Number of hits
*** Number of concurrent users

** For make bench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test

** For make megabench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* Loop-Server load test code
** The Loop-Server load test can not currently be configured in the code

* MSISDN-Gateway load test code
** The MSISDN-Gateway load test can be configured in the code - see the following lines:
** Test server URL: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
** Error percentages: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22

* General REFs:
** https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* Loop-Server
** TBD

* MSISDN-Gateway
** TBD

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Loop-Server Custom Metrics
** Opened web sockets
** Total web sockets
** Bytes/websockets
** addFailure (from the loads tool/client)

* MSISDN-Gateway Custom Metrics
** mt-flow
** ask-for-certificate
** try-wrong-code
** try-right-code
** momt-flow
** omxen-message-collision
** register
** unregister
** addFailure (from the loads tool/client)

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for Loop-Server (per server)
** /var/log/circus.log
** /var/log/loop_err.log
** /var/log/loop_out.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /var/log/nginx/access.log
** /var/log/nginx/error.log

* 2. Important logs for MSISDN-Gateway (per server)
** TBD

* Acceptable/Unacceptable Loop-Server errors:
hekad loop.stderr.log
The following are acceptable:
Decoder 'LoopServer-LoopServerDecoder' error: Failed parsing
Plugin 'AggregatorOutput' error: writing to heka.shared....

nginx logs:
The 200s (good stuff) and 101s (websockets) are acceptable.
The 499s are an artifact of the current load testing tool (V1).
You should only see them at the end of the load test.
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
Any percentage of 405s, 502s, or 503s is not acceptable.

Application logs:
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
See the loop_server.out.log file.

/var/log/loop_err.log
The following are acceptable: connect: res.on("header"): use on-headers module directly

In the Loads Cluster dashboard, watch for the following errors/failures:
string indices must be integers
OR
No JSON object could be decoded
OR
'hawk-session-token'

* Acceptable/Unacceptable MSISDN-Gateway errors:
The updated load test does generate a certain percentage of errors:
https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
So, expect to see a predefined percentage of 204s and 400s, along with the usual 200s in the nginx access logs.
The msisdn-gateway app logs should be clean with just msisdn and test data.

== Monitoring Loop Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver:
** https://app.stackdriver.com/groups/6664/stage-loads-cluster
** https://app.stackdriver.com/groups/11042/loop-server/stage-loop

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Loop Server: Check the following repos:
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla-services/docs/issues
** https://github.com/mozilla/loop-client
** https://github.com/mozilla-services/msisdn-gateway/issues

* Bugzilla
** and in Bugzilla for non-Verified bugs: http://mzl.la/1qPxizw

* OPs and Infra
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues

== References ==
* Repositories and Documentation
** Check here: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#References

* The QA Test Environments
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments

* Test Plans and other QA
** https://docs.google.com/a/mozilla.com/document/d/1xYZijHnJ1JeJhONm2AYvxSnD2tWztC16WGFyD5U_d6g/
** https://wiki.mozilla.org/Services/Loop/Try_Loop
** https://wiki.mozilla.org/Loop/QA
** https://wiki.mozilla.org/Loop/QA#Server
** https://intranet.mozilla.org/QA/Loop/MVP

* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/LoopTestEnvironments

2014-10-29T16:35:14Z

Jbonacci: /* Loop Server Stage Details */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

In the browser: https://loop.services.mozilla.com/__heartbeat__
{"storage":true,"provider":true}

In the browser: https://loop.services.mozilla.com/push-server-config
{"pushServerURI":"wss://push.services.mozilla.com/"}

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Firefox Hello web client.

Browser: https://call.mozilla.com/config.js
now returns all of this info:
var loop = loop || {};
loop.config = {
bunch of loop-client specific info
};

Browser: https://call.mozilla.com/VERSION.txt

TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
In the browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD
** loop-redis-stage - need the details
** Loop-Client: https://call.stage.mozaws.net
** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net
** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

In the browser: https://loop.stage.mozaws.net/__heartbeat__

In the browser: https://loop.stage.mozaws.net/push-server-config

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for four loop-client-* stacks
** ELB: loopstage1-loo-ELB-<some value>
** One loop-client instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4 or 6.5

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad/loop-client.stderr.log
** /var/log/hekad/loop-client.stderr.log
** /media/ephemeral0/nginx/logs
*** default.access.log
*** default.error.log
*** loop-client.access.log
*** loop-client.error.log

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net
Welcome to the Firefox Hello web client.

Check the version: https://call.stage.mozaws.net/VERSION.txt

Check the TOS: https://call.stage.mozaws.net/legal/terms/

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {
serverUrl: 'https://loop.stage.mozaws.net',
and a bunch of other loop-client specific stuff
};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa | grep msisdn

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage or msisdnstage1
** ELB: msisdn-gateway-ELB-<SOME VALUE> or msisdnstage1-m-ELB-<SOME VALUE>
** Currently one m3.medium instance, look for msisdn-gateway

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway
** /etc/puppet/yaml/app, especially
*** msisdn_gateway.dev.yaml
*** msisdn_gateway.prod.yaml
*** msisdn_gateway.stage.yaml
*** msisdn_gateway.yaml

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/msisdn-gateway.access.log
** /media/ephemeral0/nginx/logs/msisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

In the browser: https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** http://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* Old Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".
* New Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** https://app.stackdriver.com/groups/11042/loop-server/stage-loop

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Stackdriver
** https://app.stackdriver.com/groups/11044/loop-server/prod-loop

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopServerLoadTesting

2014-10-29T16:34:27Z

Jbonacci: /* Quick Verification of Production Deployments */

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
* Loop Server
For now, just run a quick loadtest 'make test'
cd loop-server
cd loadtests
make test SERVER_URL=https://loop.stage.mozaws.net

* Loop Client
Check https://call.stage.mozaws.net/config.js
Should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.stage.mozaws.net'};
and
WIP from the client team: end to end tests
Also:
https://call.stage.mozaws.net
curl https://call.stage.mozaws.net
curl -I https://call.stage.mozaws.net

and now the TOS: https://call.stage.mozaws.net/legal/terms/

* MSISDN Gateway
In the browser: https://msisdn.stage.mozaws.net
or do the following from a command line:
curl https://msisdn.stage.mozaws.net
curl -I https://msisdn.stage.mozaws.net
and
For now, just run a quick loadtest 'make test'
cd msisdn-gateway
cd loadtests
make test SERVER_URL=https://msisdn.stage.mozaws.net
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Quick Verification of Production Deployments ==
* This is a quick sanity test of the environment for after each Production deployment.
* Loop Server
In the browser:
https://loop.services.mozilla.com
https://loop.services.mozilla.com/__heartbeat__
https://loop.services.mozilla.com/push-server-config
or do the following from a command line:
curl https://loop.services.mozilla.com
curl -I https://loop.services.mozilla.com

Then run a few 'make test' commands from the loadtests folder:
make test SERVER_URL=https://loop.services.mozilla.com
Note: this does hit a live third-party server

Then perform actual loop testing via desktop (Aurora/Nightly so far) and FxOS (2.1)
Verify that requests and strings point to Production environments

* Loop Client
In the browser: https://call.mozilla.com
should return "Welcome to the Loop web client."

In the browser: https://call.mozilla.com/config.js
should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

In the browser: https://call.mozilla.com/VERSION.txt
should return the version and build string info

and now the TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com
curl -I https://call.mozilla.com

Quick end-to-end tests:
Desktop: browser to browser
Desktop to FxOS
FxOS to Desktop
Two FxOS devices

* MSISDN Gateway
In the browser: https://msisdn.services.mozilla.com
or do the following from a command line:
curl https://msisdn.services.mozilla.com
curl -I https://msisdn.services.mozilla.com
Or
Run a single 'make test' command from the loadtests folder:
make test SERVER_URL=https://msisdn.services.mozilla.com
Note: this does hit a live third-party, so limit the check to a single run.
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Installing Loop-Server and the Loads tool on Localhost or AWS ==
* Installation:
git clone https://github.com/mozilla-services/loop-server.git
cd loop-server
Note: You may want to install a specific branch for testing vs defaulting to Master
npm install
ulimit -S -n 2048
npm test *
cd loadtests
make build
make test

Coverage report can be found here:
/loop-server/coverage/lcov-report/index.html

* This step requires the redis server to be installed and running:
Mac:
brew install redis
redis-server /usr/local/etc/redis.conf

Ubuntu Linux:
sudo apt-get install redis-server
sudo /usr/bin/redis-server /etc/redis/redis.conf
sudo tail -f /var/log/redis/redis-server.log

RHEL Linux:
Install redis from here: http://download.redis.io/releases
then
/usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
or similar

* Note: This will install a local copy of the Loads tool for use with the Loop-Server.

== Running the load test against the Loop-Server in Stage ==
* Stage environment:
$ cd loop-server/loadtests
$ make test
or
$ make test SERVER_URL=https://loop.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://loop.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the Loop-Server in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://loop.stage.mozaws.net

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing MSISDN-Gateway and the Loads tool on Localhost or AWS ==
* Installation:
** Install gmp, gmp-dev or gmp-devel
** Install ruby (very latest), ruby-dev or ruby-devel
** Install gem (required for fake_dynamo)
** Verify that gem is in your path
** Install redis-server to run the unit tests

* To install gmp
sudo yum -y install gmp, gmp-devel
or for Ubuntu
$ wget https://ftp.gnu.org/gnu/gmp/gmp-6.0.0a.tar.bz2
$ tar xvjf gmp-6.0.0a.tar.bz2
$ cd gmp-6.0.0
$ ./configure --prefix=/usr
$ make
$ make check
$ sudo make install

* To install ruby:
sudo yum -y install ruby, ruby-devel
or
sudo apt-get install ruby, ruby-dev

If this does not get you 1.9.3 or newer, then install manually:
Example:
$ wget http://cache.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p547.tar.gz
$ ./configure --prefix=/usr
$ make
$ sudo make install
(because for rhel, the default ruby version is 1.8.x.)
REF:
Main: https://www.ruby-lang.org/en/downloads/
Dev Tools: http://rubyinstaller.org/downloads/

* To install gem:
Grab rubygems from here: http://rubygems.org/pages/download
cd to rubygems directory
$ sudo ruby setup.rb

* To install fake_dynamo:
You should not have to install fake_dynamo since it is now part of the repo installer.
But if you do:
$ sudo gem install fake_dynamo
REF: https://github.com/ananthakumaran/fake_dynamo

* Install the msisdn-gateway repo:
$ git clone https://github.com/mozilla-services/msisdn-gateway.git
$ cd msisdn-gateway
Note: You may want to install a specific branch for testing vs defaulting to Master
$ sudo make install
(There is a bug open about the requirement to install with 'sudo')

* Note: This will install a local copy of the Loads tool for use with MSISDN-Gateway.

* Unit testing
Get redis-server installed
Start the server in a separate terminal or in the background with logging active
$ make test
The coverage report is here: msisdn-gateway/coverage/lcov-report/index.html

* Bugs:
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036158
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036174
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048451
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048456

== Running the load test against MSISDN-Gateway in Stage ==
* Building the load tests
$ cd loadtests
$ make build

* To load test the Stage environment:
$ make test SERVER_URL=https://msisdn.stage.mozaws.net
$ make bench SERVER_URL=https://msisdn.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the MSISDN-Gateway ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://msisdn.stage.mozaws.net

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (Loop-Server and MSISDN-Gateway):
*** Number of hits
*** Number of concurrent users

** For make bench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test

** For make megabench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* Loop-Server load test code
** The Loop-Server load test can not currently be configured in the code

* MSISDN-Gateway load test code
** The MSISDN-Gateway load test can be configured in the code - see the following lines:
** Test server URL: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
** Error percentages: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22

* General REFs:
** https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* Loop-Server
** TBD

* MSISDN-Gateway
** TBD

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Loop-Server Custom Metrics
** Opened web sockets
** Total web sockets
** Bytes/websockets
** addFailure (from the loads tool/client)

* MSISDN-Gateway Custom Metrics
** mt-flow
** ask-for-certificate
** try-wrong-code
** try-right-code
** momt-flow
** omxen-message-collision
** register
** unregister
** addFailure (from the loads tool/client)

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for Loop-Server (per server)
** /var/log/circus.log
** /var/log/loop_err.log
** /var/log/loop_out.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /var/log/nginx/access.log
** /var/log/nginx/error.log

* 2. Important logs for MSISDN-Gateway (per server)
** TBD

* Acceptable/Unacceptable Loop-Server errors:
hekad loop.stderr.log
The following are acceptable:
Decoder 'LoopServer-LoopServerDecoder' error: Failed parsing
Plugin 'AggregatorOutput' error: writing to heka.shared....

nginx logs:
The 200s (good stuff) and 101s (websockets) are acceptable.
The 499s are an artifact of the current load testing tool (V1).
You should only see them at the end of the load test.
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
Any percentage of 405s, 502s, or 503s is not acceptable.

Application logs:
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
See the loop_server.out.log file.

/var/log/loop_err.log
The following are acceptable: connect: res.on("header"): use on-headers module directly

In the Loads Cluster dashboard, watch for the following errors/failures:
string indices must be integers
OR
No JSON object could be decoded
OR
'hawk-session-token'

* Acceptable/Unacceptable MSISDN-Gateway errors:
The updated load test does generate a certain percentage of errors:
https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
So, expect to see a predefined percentage of 204s and 400s, along with the usual 200s in the nginx access logs.
The msisdn-gateway app logs should be clean with just msisdn and test data.

== Monitoring Loop Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver:
** https://app.stackdriver.com/groups/6664/stage-loads-cluster
** https://app.stackdriver.com/groups/11042/loop-server/stage-loop

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Loop Server: Check the following repos:
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla-services/docs/issues
** https://github.com/mozilla/loop-client
** https://github.com/mozilla-services/msisdn-gateway/issues

* Bugzilla
** and in Bugzilla for non-Verified bugs: http://mzl.la/1qPxizw

* OPs and Infra
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues

== References ==
* Repositories and Documentation
** Check here: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#References

* The QA Test Environments
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments

* Test Plans and other QA
** https://docs.google.com/a/mozilla.com/document/d/1xYZijHnJ1JeJhONm2AYvxSnD2tWztC16WGFyD5U_d6g/
** https://wiki.mozilla.org/Services/Loop/Try_Loop
** https://wiki.mozilla.org/Loop/QA
** https://wiki.mozilla.org/Loop/QA#Server
** https://intranet.mozilla.org/QA/Loop/MVP

* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/LoopTestEnvironments

2014-10-29T16:33:28Z

Jbonacci: /* Loop Server Production Environments */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

In the browser: https://loop.services.mozilla.com/__heartbeat__
{"storage":true,"provider":true}

In the browser: https://loop.services.mozilla.com/push-server-config
{"pushServerURI":"wss://push.services.mozilla.com/"}

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Firefox Hello web client.

Browser: https://call.mozilla.com/config.js
now returns all of this info:
var loop = loop || {};
loop.config = {
bunch of loop-client specific info
};

Browser: https://call.mozilla.com/VERSION.txt

TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
In the browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD
** loop-redis-stage - need the details
** Loop-Client: https://call.stage.mozaws.net
** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net
** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

In the browser: https://loop.stage.mozaws.net/__heartbeat__

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for four loop-client-* stacks
** ELB: loopstage1-loo-ELB-<some value>
** One loop-client instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4 or 6.5

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad/loop-client.stderr.log
** /var/log/hekad/loop-client.stderr.log
** /media/ephemeral0/nginx/logs
*** default.access.log
*** default.error.log
*** loop-client.access.log
*** loop-client.error.log

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net
Welcome to the Firefox Hello web client.

Check the version: https://call.stage.mozaws.net/VERSION.txt

Check the TOS: https://call.stage.mozaws.net/legal/terms/

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {
serverUrl: 'https://loop.stage.mozaws.net',
and a bunch of other loop-client specific stuff
};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa | grep msisdn

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage or msisdnstage1
** ELB: msisdn-gateway-ELB-<SOME VALUE> or msisdnstage1-m-ELB-<SOME VALUE>
** Currently one m3.medium instance, look for msisdn-gateway

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway
** /etc/puppet/yaml/app, especially
*** msisdn_gateway.dev.yaml
*** msisdn_gateway.prod.yaml
*** msisdn_gateway.stage.yaml
*** msisdn_gateway.yaml

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/msisdn-gateway.access.log
** /media/ephemeral0/nginx/logs/msisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

In the browser: https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** http://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* Old Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".
* New Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** https://app.stackdriver.com/groups/11042/loop-server/stage-loop

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Stackdriver
** https://app.stackdriver.com/groups/11044/loop-server/prod-loop

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopTestEnvironments

2014-10-29T00:38:10Z

Jbonacci: /* Loop Server Production Environments */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

In the browser: https://loop.services.mozilla.com/__heartbeat__
{"storage":true,"provider":true}

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Firefox Hello web client.

Browser: https://call.mozilla.com/config.js
now returns all of this info:
var loop = loop || {};
loop.config = {
bunch of loop-client specific info
};

Browser: https://call.mozilla.com/VERSION.txt

TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
In the browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD
** loop-redis-stage - need the details
** Loop-Client: https://call.stage.mozaws.net
** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net
** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

In the browser: https://loop.stage.mozaws.net/__heartbeat__

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for four loop-client-* stacks
** ELB: loopstage1-loo-ELB-<some value>
** One loop-client instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4 or 6.5

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad/loop-client.stderr.log
** /var/log/hekad/loop-client.stderr.log
** /media/ephemeral0/nginx/logs
*** default.access.log
*** default.error.log
*** loop-client.access.log
*** loop-client.error.log

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net
Welcome to the Firefox Hello web client.

Check the version: https://call.stage.mozaws.net/VERSION.txt

Check the TOS: https://call.stage.mozaws.net/legal/terms/

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {
serverUrl: 'https://loop.stage.mozaws.net',
and a bunch of other loop-client specific stuff
};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa | grep msisdn

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage or msisdnstage1
** ELB: msisdn-gateway-ELB-<SOME VALUE> or msisdnstage1-m-ELB-<SOME VALUE>
** Currently one m3.medium instance, look for msisdn-gateway

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway
** /etc/puppet/yaml/app, especially
*** msisdn_gateway.dev.yaml
*** msisdn_gateway.prod.yaml
*** msisdn_gateway.stage.yaml
*** msisdn_gateway.yaml

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/msisdn-gateway.access.log
** /media/ephemeral0/nginx/logs/msisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

In the browser: https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** http://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* Old Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".
* New Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** https://app.stackdriver.com/groups/11042/loop-server/stage-loop

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Stackdriver
** https://app.stackdriver.com/groups/11044/loop-server/prod-loop

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopTestEnvironments

2014-10-29T00:25:39Z

Jbonacci: /* MSISDN Gateway Server Stage Details */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Firefox Hello web client.

Browser: https://call.mozilla.com/config.js
now returns all of this info:
var loop = loop || {};
loop.config = {
bunch of loop-client specific info
};

Browser: https://call.mozilla.com/VERSION.txt

TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD
** loop-redis-stage - need the details
** Loop-Client: https://call.stage.mozaws.net
** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net
** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

In the browser: https://loop.stage.mozaws.net/__heartbeat__

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for four loop-client-* stacks
** ELB: loopstage1-loo-ELB-<some value>
** One loop-client instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4 or 6.5

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad/loop-client.stderr.log
** /var/log/hekad/loop-client.stderr.log
** /media/ephemeral0/nginx/logs
*** default.access.log
*** default.error.log
*** loop-client.access.log
*** loop-client.error.log

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net
Welcome to the Firefox Hello web client.

Check the version: https://call.stage.mozaws.net/VERSION.txt

Check the TOS: https://call.stage.mozaws.net/legal/terms/

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {
serverUrl: 'https://loop.stage.mozaws.net',
and a bunch of other loop-client specific stuff
};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa | grep msisdn

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage or msisdnstage1
** ELB: msisdn-gateway-ELB-<SOME VALUE> or msisdnstage1-m-ELB-<SOME VALUE>
** Currently one m3.medium instance, look for msisdn-gateway

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway
** /etc/puppet/yaml/app, especially
*** msisdn_gateway.dev.yaml
*** msisdn_gateway.prod.yaml
*** msisdn_gateway.stage.yaml
*** msisdn_gateway.yaml

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/msisdn-gateway.access.log
** /media/ephemeral0/nginx/logs/msisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

In the browser: https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** http://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* Old Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".
* New Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** https://app.stackdriver.com/groups/11042/loop-server/stage-loop

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Stackdriver
** https://app.stackdriver.com/groups/11044/loop-server/prod-loop

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopTestEnvironments

2014-10-29T00:13:50Z

Jbonacci: /* Loop Client Stage Details */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Firefox Hello web client.

Browser: https://call.mozilla.com/config.js
now returns all of this info:
var loop = loop || {};
loop.config = {
bunch of loop-client specific info
};

Browser: https://call.mozilla.com/VERSION.txt

TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD
** loop-redis-stage - need the details
** Loop-Client: https://call.stage.mozaws.net
** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net
** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

In the browser: https://loop.stage.mozaws.net/__heartbeat__

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for four loop-client-* stacks
** ELB: loopstage1-loo-ELB-<some value>
** One loop-client instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4 or 6.5

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad/loop-client.stderr.log
** /var/log/hekad/loop-client.stderr.log
** /media/ephemeral0/nginx/logs
*** default.access.log
*** default.error.log
*** loop-client.access.log
*** loop-client.error.log

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net
Welcome to the Firefox Hello web client.

Check the version: https://call.stage.mozaws.net/VERSION.txt

Check the TOS: https://call.stage.mozaws.net/legal/terms/

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {
serverUrl: 'https://loop.stage.mozaws.net',
and a bunch of other loop-client specific stuff
};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** http://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* Old Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".
* New Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** https://app.stackdriver.com/groups/11042/loop-server/stage-loop

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Stackdriver
** https://app.stackdriver.com/groups/11044/loop-server/prod-loop

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopTestEnvironments

2014-10-29T00:10:01Z

Jbonacci: /* Loop Server Stage Details */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Firefox Hello web client.

Browser: https://call.mozilla.com/config.js
now returns all of this info:
var loop = loop || {};
loop.config = {
bunch of loop-client specific info
};

Browser: https://call.mozilla.com/VERSION.txt

TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD
** loop-redis-stage - need the details
** Loop-Client: https://call.stage.mozaws.net
** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net
** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

In the browser: https://loop.stage.mozaws.net/__heartbeat__

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4 or 6.5

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad/loop-client.stderr.log
** /var/log/hekad/loop-client.stderr.log
** /media/ephemeral0/nginx/logs
*** default.access.log
*** default.error.log
*** loop-client.access.log
*** loop-client.error.log

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net
Welcome to the Firefox Hello web client.

Check the version: https://call.stage.mozaws.net/VERSION.txt

Check the TOS: https://call.stage.mozaws.net/legal/terms/

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {
serverUrl: 'https://loop.stage.mozaws.net',
and a bunch of other loop-client specific stuff
};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** http://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* Old Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".
* New Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** https://app.stackdriver.com/groups/11042/loop-server/stage-loop

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Stackdriver
** https://app.stackdriver.com/groups/11044/loop-server/prod-loop

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopTestEnvironments

2014-10-29T00:06:49Z

Jbonacci: /* Loop Server Stage Environment */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Firefox Hello web client.

Browser: https://call.mozilla.com/config.js
now returns all of this info:
var loop = loop || {};
loop.config = {
bunch of loop-client specific info
};

Browser: https://call.mozilla.com/VERSION.txt

TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD
** loop-redis-stage - need the details
** Loop-Client: https://call.stage.mozaws.net
** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net
** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4 or 6.5

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad/loop-client.stderr.log
** /var/log/hekad/loop-client.stderr.log
** /media/ephemeral0/nginx/logs
*** default.access.log
*** default.error.log
*** loop-client.access.log
*** loop-client.error.log

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net
Welcome to the Firefox Hello web client.

Check the version: https://call.stage.mozaws.net/VERSION.txt

Check the TOS: https://call.stage.mozaws.net/legal/terms/

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {
serverUrl: 'https://loop.stage.mozaws.net',
and a bunch of other loop-client specific stuff
};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** http://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* Old Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".
* New Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** https://app.stackdriver.com/groups/11042/loop-server/stage-loop

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Stackdriver
** https://app.stackdriver.com/groups/11044/loop-server/prod-loop

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/TSVerifierSyncTestEnvironments

2014-10-28T23:53:39Z

Jbonacci: /* Verifier Stage Environment */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier Stage section below...
** A Tokenserver+Verifier stack: See the TokenServer Stage section below...

== Tokenserver, Verifier, and Sync Production Environments ==
* TokenServer: https://token.services.mozilla.com
* Verifier: https://verifier.accounts.firefox.com
* Sync 1.5 Nodes (for now in AWS):
** sync-1-us-east-2.sync.services.mozilla.com
** sync-2-us-east-2.sync.services.mozilla.com
*** There are many more than this now...

* QA Access: not allowed

* Pointing Fx Nightly to Production FxA/Sync:
** Set/Verify the following Firefox configs:
*** services.sync.clusterURL = (should get automatically set by the TokenServer)
*** services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
*** services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
*** services.sync.log.appender.file.logOnError = Yes
*** services.sync.log.appender.file.logOnSuccess = Yes
*** services.sync.log.appender.file.level = Trace
*** identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
*** identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...

== TokenServer+Verifier Stage Environment ==
* NOTE: In an upcoming deployent, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** TokenServer: https://token.stage.mozaws.net
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
TokenServer:
rpm -qa | grep -E 'token|browser'

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** tokenserver app server (1 or more instances behind a CF stack and ELB)
** tokenserver db (1 large DB instance behind RDS)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/tokenserver
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /data/fxa-browserid-verifier
** /etc/heka.d
** /etc/puppet
*** esp. /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa_verifier.stage.yaml
*** /etc/puppet/yaml/app/tokenserver.stage.yaml
*** /etc/puppet/yaml/app/tokenserver.yaml

* Processes
** tokenserver app server:
*** Search for token, circus, nginx, gunicorn, python, hekad, node

* Logs
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* Hekad
** /etc/puppet/modules/hekad

* QA Access via a Bastion Host
** Old Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance.
** New Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

== Verifier Stage Environment ==
* NOTE: In an upcoming deployment, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
** rpm -qa | grep verifier

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** fxa-browserid_verifier or fxa-bv-stage (usually a single instance)

* Files
** /data/fxa-browserid-verifier
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/heka.d
** /etc/puppet
*** esp. the /etc/puppet/yaml/app files
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa.yaml

* Processes
** Search for node, heka, nginx, circus

* Logs
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* Hekad
** /etc/puppet/modules/hekad
** /data/hekad

* QA Access via a Bastion Host
** Old Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance.
** New Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

* Quick verifications
Browser: https://verifier.stage.mozaws.net/
returns "Method Not Allowed"

$ curl https://verifier.stage.mozaws.net
returns "Method Not Allowed"

$ curl -I https://verifier.stage.mozaws.net
HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache, max-age=0
Content-length: 18
Content-Type: text/plain
Date: Wed, 23 Jul 2014 20:19:40 GMT
Connection: keep-alive

== Sync 1.5 Stage Environment ==
* This is a work in progress. Right now, there are only a small number of sync nodes (instances) in AWS.
* URLs
** https://sync-1-us-east-1.stage.mozaws.net
** https://sync-2-us-east-1.stage.mozaws.net
** https://sync-3-us-east-1.stage.mozaws.net
** https://sync-4-us-east-1.stage.mozaws.net
** and sometimes
** https://sync-5-us-east-1.stage.mozaws.net
** https://sync-6-us-east-1.stage.mozaws.net

* Versions
Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** Search for sync node instances in US East: "stage-sync-node-X"
** Each node is a specific large instance (mixed m3 and c3)

* Files
** /data/server-syncstorage/*
** /var/log/nginx
** /etc/puppet

* Processes
** Search for sync, mysql, circusd, hekad, nginx, memcached

* Logs
** /media/ephemeral0/logs/
** /media/ephemeral0/logs/nginx/access.log
** /media/ephemeral0/logs/nginx/error.log
** /media/ephemeral0/logs/sync/sync.err
** /media/ephemeral0/logs/sync/sync.log
** /var/log/circus.log
** /var/log/hekad/sync_1_5.stderr.log
** /var/log/hekad/sync_1_5.stdout.log

* Hekad
** /etc/heka.d/sync_1_5.toml

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer

* Note: There is no longer a Sync 1.1 Stage environment.

== Loads V1 Services Cluster Environment ==
* Details on the Loads V1 cluster are available at this link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Monitoring the Stage Environment ==
* Stackdriver:
** Stage main: https://app.stackdriver.com
** https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** https://app.stackdriver.com/groups/4389/stage-services-tag-sync15/stage-services-sync15/fxa-verifier
** https://app.stackdriver.com/groups/4391/stage-services-tag-sync15/stage-services-tag-sync15/sync-storage
** https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
** https://app.stackdriver.com/monitoring/3827/tokenserver
** https://app.stackdriver.com/instances/<AWS instance>
** Also, make use of the Services, Groups, and Dashboards for more specific links
*** Example: Services > https://app.stackdriver.com/services/nginx
*** Example: Groups > https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver

* Graphite:
** https://graphite.shared.us-east-1.stage.mozaws.net

* Kibana
** https://kibana.shared.us-east-1.stage.mozaws.net/
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_errors.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_app_logs.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/tokenserver_http_status.json

* Heka
** https://heka.shared.us-east-1.stage.mozaws.net/
** or https://heka.shared.us-east-1.stage.mozaws.net/#health
** Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/TokenServerHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-SlowQueryFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerLocalVerifierFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-FileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerNginxFileOutput
** Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-FileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-SlowQueryFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerLocalVerifierFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxErrorFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagemetaglobal.cbuf

== Monitoring the Production Environment ==
* Heka shared:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/TokenServerHTTPStatus

* Kibana shared:
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/file/default.json

* StackDriver:
** Main: https://app.stackdriver.com/

== TokenServer, Verifier, and Sync 1.5 Dev Environments ==
* URLs
** https://sync1.dev.lcip.org/
** https://token.dev.lcip.org/

* Versions
TokenServer:
rpm -qa | grep token
Example: tokenserver-svcops <latest version>

Verifier:
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>

Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** This is part of the dev-lcip-org CloudFormation stack
*** TokenServer: Search for the fxa-tokenserver instance
*** SyncServer2: Search for the fxa-syncstorage instance
*** SyncServer1: Search for the fxa-syncstorage instance
*** LogServer: Search for the fxa-logbox instance
* Files
** TBD
* Processes
** TBD
* Logs
** TBD
* QA Access
** SSH with AWS keys to the various instances
* Firefox configs:
** services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
** services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/sync
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/token

TestEngineering/Services/TSVerifierSyncTestEnvironments

2014-10-28T23:53:11Z

Jbonacci: /* TokenServer+Verifier Stage Environment */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier Stage section below...
** A Tokenserver+Verifier stack: See the TokenServer Stage section below...

== Tokenserver, Verifier, and Sync Production Environments ==
* TokenServer: https://token.services.mozilla.com
* Verifier: https://verifier.accounts.firefox.com
* Sync 1.5 Nodes (for now in AWS):
** sync-1-us-east-2.sync.services.mozilla.com
** sync-2-us-east-2.sync.services.mozilla.com
*** There are many more than this now...

* QA Access: not allowed

* Pointing Fx Nightly to Production FxA/Sync:
** Set/Verify the following Firefox configs:
*** services.sync.clusterURL = (should get automatically set by the TokenServer)
*** services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
*** services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
*** services.sync.log.appender.file.logOnError = Yes
*** services.sync.log.appender.file.logOnSuccess = Yes
*** services.sync.log.appender.file.level = Trace
*** identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
*** identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...

== TokenServer+Verifier Stage Environment ==
* NOTE: In an upcoming deployent, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** TokenServer: https://token.stage.mozaws.net
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
TokenServer:
rpm -qa | grep -E 'token|browser'

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** tokenserver app server (1 or more instances behind a CF stack and ELB)
** tokenserver db (1 large DB instance behind RDS)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/tokenserver
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /data/fxa-browserid-verifier
** /etc/heka.d
** /etc/puppet
*** esp. /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa_verifier.stage.yaml
*** /etc/puppet/yaml/app/tokenserver.stage.yaml
*** /etc/puppet/yaml/app/tokenserver.yaml

* Processes
** tokenserver app server:
*** Search for token, circus, nginx, gunicorn, python, hekad, node

* Logs
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* Hekad
** /etc/puppet/modules/hekad

* QA Access via a Bastion Host
** Old Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance.
** New Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

== Verifier Stage Environment ==
* NOTE: In an upcoming deployment, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
** rpm -qa | grep verifier

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** fxa-browserid_verifier or fxa-bv-stage (usually a single instance)

* Files
** /data/fxa-browserid-verifier
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/heka.d
** /etc/puppet
*** esp. the /etc/puppet/yaml/app files
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa.yaml

* Processes
** Search for node, heka, nginx, circus

* Logs
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* Hekad
** /etc/puppet/modules/hekad
** /data/hekad

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

* Quick verifications
Browser: https://verifier.stage.mozaws.net/
returns "Method Not Allowed"

$ curl https://verifier.stage.mozaws.net
returns "Method Not Allowed"

$ curl -I https://verifier.stage.mozaws.net
HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache, max-age=0
Content-length: 18
Content-Type: text/plain
Date: Wed, 23 Jul 2014 20:19:40 GMT
Connection: keep-alive

== Sync 1.5 Stage Environment ==
* This is a work in progress. Right now, there are only a small number of sync nodes (instances) in AWS.
* URLs
** https://sync-1-us-east-1.stage.mozaws.net
** https://sync-2-us-east-1.stage.mozaws.net
** https://sync-3-us-east-1.stage.mozaws.net
** https://sync-4-us-east-1.stage.mozaws.net
** and sometimes
** https://sync-5-us-east-1.stage.mozaws.net
** https://sync-6-us-east-1.stage.mozaws.net

* Versions
Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** Search for sync node instances in US East: "stage-sync-node-X"
** Each node is a specific large instance (mixed m3 and c3)

* Files
** /data/server-syncstorage/*
** /var/log/nginx
** /etc/puppet

* Processes
** Search for sync, mysql, circusd, hekad, nginx, memcached

* Logs
** /media/ephemeral0/logs/
** /media/ephemeral0/logs/nginx/access.log
** /media/ephemeral0/logs/nginx/error.log
** /media/ephemeral0/logs/sync/sync.err
** /media/ephemeral0/logs/sync/sync.log
** /var/log/circus.log
** /var/log/hekad/sync_1_5.stderr.log
** /var/log/hekad/sync_1_5.stdout.log

* Hekad
** /etc/heka.d/sync_1_5.toml

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer

* Note: There is no longer a Sync 1.1 Stage environment.

== Loads V1 Services Cluster Environment ==
* Details on the Loads V1 cluster are available at this link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Monitoring the Stage Environment ==
* Stackdriver:
** Stage main: https://app.stackdriver.com
** https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** https://app.stackdriver.com/groups/4389/stage-services-tag-sync15/stage-services-sync15/fxa-verifier
** https://app.stackdriver.com/groups/4391/stage-services-tag-sync15/stage-services-tag-sync15/sync-storage
** https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
** https://app.stackdriver.com/monitoring/3827/tokenserver
** https://app.stackdriver.com/instances/<AWS instance>
** Also, make use of the Services, Groups, and Dashboards for more specific links
*** Example: Services > https://app.stackdriver.com/services/nginx
*** Example: Groups > https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver

* Graphite:
** https://graphite.shared.us-east-1.stage.mozaws.net

* Kibana
** https://kibana.shared.us-east-1.stage.mozaws.net/
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_errors.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_app_logs.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/tokenserver_http_status.json

* Heka
** https://heka.shared.us-east-1.stage.mozaws.net/
** or https://heka.shared.us-east-1.stage.mozaws.net/#health
** Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/TokenServerHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-SlowQueryFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerLocalVerifierFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-FileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerNginxFileOutput
** Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-FileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-SlowQueryFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerLocalVerifierFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxErrorFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagemetaglobal.cbuf

== Monitoring the Production Environment ==
* Heka shared:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/TokenServerHTTPStatus

* Kibana shared:
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/file/default.json

* StackDriver:
** Main: https://app.stackdriver.com/

== TokenServer, Verifier, and Sync 1.5 Dev Environments ==
* URLs
** https://sync1.dev.lcip.org/
** https://token.dev.lcip.org/

* Versions
TokenServer:
rpm -qa | grep token
Example: tokenserver-svcops <latest version>

Verifier:
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>

Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** This is part of the dev-lcip-org CloudFormation stack
*** TokenServer: Search for the fxa-tokenserver instance
*** SyncServer2: Search for the fxa-syncstorage instance
*** SyncServer1: Search for the fxa-syncstorage instance
*** LogServer: Search for the fxa-logbox instance
* Files
** TBD
* Processes
** TBD
* Logs
** TBD
* QA Access
** SSH with AWS keys to the various instances
* Firefox configs:
** services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
** services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/sync
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/token

TestEngineering/Services/FxATestEnvironments

2014-10-28T23:50:54Z

Jbonacci: /* FxA Stage Environment */

NOTE: This site needs an update for all the extra Dev environments and the new environments - OAuth and Profile, etc...

== FxA Production Environments ==
* URLs
** Content server: https://accounts.firefox.com
** Auth server: https://api.accounts.firefox.com
** Verifier: https://verifier.accounts.firefox.com (talks to Auth Server via TokenServer)
** OAuth server: https://oauth.accounts.firefox.com
** Profile server: https://profile.accounts.firefox.com

* Versions:
** FxA-Auth-Server: curl https://api.accounts.firefox.com/; echo
** FxA-Content-Server: curl https://accounts.firefox.com/ver.json; echo

* QA Access: not allowed

* Pointing Fx Nightly to Production FxA/Sync:
** Set/Verify the following Firefox configs:
*** services.sync.clusterURL = (should get automatically set by the TokenServer)
*** services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
*** services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
*** services.sync.log.appender.file.logOnError = Yes
*** services.sync.log.appender.file.logOnSuccess = Yes
*** services.sync.log.appender.file.level = Trace
*** identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
*** identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...

== FxA Stage Environment ==
* URLs
** Content Server: https://accounts.stage.mozaws.net/
** Auth Server: https://api-accounts.stage.mozaws.net/
** Verifier: https://verifier.stage.mozaws.net (talks to Auth Server via TokenServer)
** OAuth server: https://oauth.stage.mozaws.net
** Profile server: https://profile.stage.mozaws.net

* Versions:
** FxA-Auth-Server: curl https://api-accounts.stage.mozaws.net/; echo
** FxA-Content-Server: curl https://accounts.stage.mozaws.net/ver.json; echo

* AWS
** New FxA-shared services:
*** svcops-fxa-stage-heka
*** svcops-fxa-stage-amqp
*** svcops-fxa-stage-elasticsearch
*** svcops-fxa-stage-asg-sns-topic
** Other shared Stage Services:
*** svcops-stage-heka (combined heka+kibana behind an ELB)
*** svcops-stage-amqp (shared RabbitMQ cluster)
*** svcops-stage-elasticsearch ( Elasticsearch cluster)
*** svcops-stage-asg-sns-topic
*** svcops-stage-bastion-servers (Bastion hosts for Stage)
** Load Balancers:
*** ContentServerELB
*** AuthServerELB
*** BastionELB
*** BastionExternalFQDN
** DB Instances (RDS):
*** RDSInstance = fxa-rds-stage
** All instances: Search in AWS Console using the Env column to find "stage"
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka
*** fxa-elasticsearch
*** fxa-rabbitmq
*** fxa-heka
*** fxa-logbox
*** fxa-auth (2 instances, m3.large)
*** fxa-content_server (1 small instance)
* Files
** /data
** /etc/puppet
* Processes
** fxa-auth-server:
*** Look for processes owned by app, nginx, circus
** fxa-content-server:
*** Look for processes owned by node, heka, nginx, circus
* Logs
** /var/log/nginx
** /var/log/circus.log
** fxa-auth-server
*** /media/ephemeral0/fxa-auth-server/auth_err.log.*
*** /media/ephemeral0/fxa-auth-server/auth_out.log
*** /media/ephemeral0/heka/hekad_err.log
*** /media/ephemeral0/heka/hekad_out.log
*** /media/ephemeral0/nginx/logs/access.log
*** /media/ephemeral0/nginx/logs/error.log
** fxa-content-server
*** /media/ephemeral0/fxa-content-server/content_err.log
*** /media/ephemeral0/fxa-content-server/content_out.log
*** /media/ephemeral0/heka/hekad_err.log
*** /media/ephemeral0/heka/hekad_out.log
*** /media/ephemeral0/nginx/access.log
*** /media/ephemeral0/nginx/logs/error.log

* Log Aggregation
** Look at /data/hekad/hekad.toml
** Check here: [aggregator-output]
** hekad --> rabbitmq --> elasticsearch --> kibana (since Kibana is the UI on top of ES)

* Stack code
** https://github.com/mozilla-services/svcops/tree/master/cloudformations/firefox-accounts
** https://github.com/mozilla-services/puppet-config/tree/master/fxa
** https://github.com/mozilla-services/puppet-config/tree/master/shared

* How to find the stack instance size and number of instances
** https://github.com/mozilla-services/svcops/tree/master/cloudformations/firefox-accounts
** Environment, number of hosts, and host/instance size deployed by default (per stack/service):
*** Look in the json files for the CF stacks in github:
fxa-auth-server.json:
defaults for "Environment", "AuthServerInstanceType", and "AuthServerDesiredCapacity"
fxa-content-server.json:
defaults for "Environment", "InstanceType", and "NumServers"
fxa-db-rds.json:
defaults for "Environment" and "DBInstanceType"
Compare this to what is actually live in AWS (via the Console or awsboxen)

* QA Access via a Bastion Host
** Old Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance.
** New Dev IAM
*** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance.

* Pointing Fx Nightly to Stage FxA/Sync:
** Set/Verify the following Firefox configs:
*** services.sync.clusterURL = (should get automatically set by the TokenServer)
*** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5
*** services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
*** services.sync.log.appender.file.logOnError = Yes
*** services.sync.log.appender.file.logOnSuccess = Yes
*** services.sync.log.appender.file.level = Trace
*** identity.fxaccounts.auth.uri = https://api-accounts.stage.mozaws.net/v1
*** identity.fxaccounts.remote.uri = https://accounts.stage.mozaws.net/?service=sync

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Monitoring the Production Environment ==
* Heka FxA:
** https://heka.fxa.us-west-2.prod.mozaws.net/
** https://heka.fxa.us-west-2.prod.mozaws.net/#health
** https://heka.fxa.us-west-2.prod.mozaws.net/#sandboxes

* Kibana FxA:
** Kibana FxA: https://kibana.fxa.us-west-2.prod.mozaws.net/index.html#/dashboard/file/weblogs.json
** Kibana FxA errors: https://kibana.fxa.us-west-2.prod.mozaws.net/index.html#/dashboard/elasticsearch/Endpoint%20Errors
** Others: see the "load" folder/pull-down

* StackDriver:
** Main: https://app.stackdriver.com/
** https://app.stackdriver.com/groups/4208/prod-persona
** https://app.stackdriver.com/groups/4251/production-all
** https://app.stackdriver.com/groups/4337/prod-fxa

== Monitoring the Stage Environment ==
* Kibana:
** Kibana: https://kibana.fxa.us-east-1.stage.mozaws.net/#/dashboard
** Kibana: https://kibana.fxa.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json

* Heka:
** https://heka.fxa.us-east-1.stage.mozaws.net/
** https://heka.fxa.us-east-1.stage.mozaws.net/#health
** https://heka.fxa.us-east-1.stage.mozaws.net/#sandboxes

* StackDriver:
** https://app.stackdriver.com/
** https://app.stackdriver.com/groups/4393/stage-fxa

== FxA Development Environments - partial ==
* NOTE: This is marked partial, because there are a number of more recent, ongoing Dev deploys for development and testing. They should get recorded here, soon...

* QA Access
** SSH with AWS keys to the various instances

* Content Server
** URLs
*** https://accounts.dev.lcip.org/
**** non-auto-updating environment (AWS - US East)
*** https://accounts-latest.dev.lcip.org/
**** auto-updated environment (AWS - US East)
** AWS
*** Search for the following in US East:
**** rfk's awsbox deployment (accounts-latest.dev.lcip.org)
**** rfk's awsbox deployment (accounts.dev.lcip.org)
** Versions
*** https://accounts-latest.dev.lcip.org/ver.txt
*** https://api-accounts-latest.dev.lcip.org/ver.txt
*** https://accounts.dev.lcip.org/ver.txt
*** https://api-accounts.dev.lcip.org/ver.txt
** Files
*** /home/app
*** /home/proxy
** Processes
*** Look at any process owned by root, mysql, app, proxy
** Logs
*** /var/log/nginx/access.log and error.log
*** /home/app/var/log/browserid-certifier.js.log
*** /home/app/var/log/firefox_account_bridge.js.log

* Auth Server
** URLs
*** https://api-accounts.dev.lcip.org
*** https://api-accounts-latest.dev.lcip.org
** AWS
*** Search for the following in US East:
**** rfk's awsbox deployment (api-accounts-latest.dev.lcip.org)
**** rfk's awsbox deployment (api-accounts.dev.lcip.org)
** Versions
*** https://api-accounts.dev.lcip.org/ver.txt
*** https://api-accounts-latest.dev.lcip.org/ver.txt
** Files
*** /home/app
** Processes
** Search for any process owned by app, root, mysql, proxy
** Logs
*** /home/app/hekad/*.log
*** /home/app/var/log/*.log
*** /home/proxy/var/log/nginx

* Log Aggregation
** http://logs.dev.lcip.org

* Mail
** https://restmail.lcip.org

* Monitoring
** http://logs.dev.lcip.org/kibana/index.html#/dashboard/file/weblogs.json

* Default Firefox configs for Nightly:
** services.sync.clusterURL = http://db1.oldsync.dev.lcip.org/ (should get automatically set by the TokenServer)
** services.sync.tokenServerURI = http://auth.oldsync.dev.lcip.org/1.0/sync/1.1
** firefox.accounts.remoteUrl = https://accounts.dev.lcip.org/?service=sync
*** identity.fxaccounts.enabled=true (NO LONGER NEEDED, BUT STILL CAN BE DEFINED)
*** services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
** services.sync.log.appender.file.logOnError = Yes
** services.sync.log.appender.file.logOnSuccess = Yes
** services.sync.log.appender.file.level = Trace
** identity.fxaccounts.auth.uri = https://api-accounts.dev.lcip.org/v1
** And to hit the TokenServer/Sync 1.5 Dev services, change the following:
*** services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
*** services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5
** And to hit the TokenServer/Sync 1.5 Stage services, change the following:
*** services.sync.clusterURL = https://sync-X-us-east-1.stage.mozaws.net/ (should get automatically set by the TokenServer)
*** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5
** And to hit the TokenServer/Sync 1.5 Prod services, change the following:
*** services.sync.clusterURL = https://sync-X-us-east-2.sync.services.mozilla.com/ (should get automatically set by the TokenServer)
*** services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5

* Other Dev Deployments:
** AWS: api-accounts-legacy.dev.lcip.org
** AWS: accounts-legacy.dev.lcip.org
** AWS: api-accounts-onepw.dev.lcip.org

== OAuth Dev Environment ==
* NOTE: TBD

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
** and https://github.com/mozilla-services/puppet-config/tree/master/fxa
* Cloud Formation: https://github.com/mozilla-services/svcops
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/firefox-accounts

TestEngineering/Services/LoopTestEnvironments

2014-10-28T23:46:30Z

Jbonacci: /* QA Access via a Bastion Host */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Firefox Hello web client.

Browser: https://call.mozilla.com/config.js
now returns all of this info:
var loop = loop || {};
loop.config = {
bunch of loop-client specific info
};

Browser: https://call.mozilla.com/VERSION.txt

TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD

** loop-redis-stage - need the details

** Loop-Client: https://call.stage.mozaws.net

** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net

** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4 or 6.5

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad/loop-client.stderr.log
** /var/log/hekad/loop-client.stderr.log
** /media/ephemeral0/nginx/logs
*** default.access.log
*** default.error.log
*** loop-client.access.log
*** loop-client.error.log

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net
Welcome to the Firefox Hello web client.

Check the version: https://call.stage.mozaws.net/VERSION.txt

Check the TOS: https://call.stage.mozaws.net/legal/terms/

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {
serverUrl: 'https://loop.stage.mozaws.net',
and a bunch of other loop-client specific stuff
};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** http://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* Old Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".
* New Dev IAM
** SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** https://app.stackdriver.com/groups/11042/loop-server/stage-loop

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Stackdriver
** https://app.stackdriver.com/groups/11044/loop-server/prod-loop

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopServerLoadTesting

2014-10-28T00:10:27Z

Jbonacci: /* Monitoring Loop Stage */

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
* Loop Server
For now, just run a quick loadtest 'make test'
cd loop-server
cd loadtests
make test SERVER_URL=https://loop.stage.mozaws.net

* Loop Client
Check https://call.stage.mozaws.net/config.js
Should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.stage.mozaws.net'};
and
WIP from the client team: end to end tests
Also:
https://call.stage.mozaws.net
curl https://call.stage.mozaws.net
curl -I https://call.stage.mozaws.net

and now the TOS: https://call.stage.mozaws.net/legal/terms/

* MSISDN Gateway
In the browser: https://msisdn.stage.mozaws.net
or do the following from a command line:
curl https://msisdn.stage.mozaws.net
curl -I https://msisdn.stage.mozaws.net
and
For now, just run a quick loadtest 'make test'
cd msisdn-gateway
cd loadtests
make test SERVER_URL=https://msisdn.stage.mozaws.net
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Quick Verification of Production Deployments ==
* This is a quick sanity test of the environment for after each Production deployment.
* Loop Server
In the browser: https://loop.services.mozilla.com
or do the following from a command line:
curl https://loop.services.mozilla.com
curl -I https://loop.services.mozilla.com

Then run a few 'make test' commands from the loadtests folder:
make test SERVER_URL=https://loop.services.mozilla.com
Note: this does hit a live third-party server

Then perform actual loop testing via desktop (Aurora/Nightly so far) and FxOS (2.1)
Verify that requests and strings point to Production environments

* Loop Client
In the browser: https://call.mozilla.com
should return "Welcome to the Loop web client."

In the browser: https://call.mozilla.com/config.js
should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

In the browser: https://call.mozilla.com/VERSION.txt
should return the version and build string info

and now the TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com
curl -I https://call.mozilla.com

Quick end-to-end tests:
Desktop: browser to browser
Desktop to FxOS
FxOS to Desktop
Two FxOS devices

* MSISDN Gateway
In the browser: https://msisdn.services.mozilla.com
or do the following from a command line:
curl https://msisdn.services.mozilla.com
curl -I https://msisdn.services.mozilla.com
Or
Run a single 'make test' command from the loadtests folder:
make test SERVER_URL=https://msisdn.services.mozilla.com
Note: this does hit a live third-party, so limit the check to a single run.
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Installing Loop-Server and the Loads tool on Localhost or AWS ==
* Installation:
git clone https://github.com/mozilla-services/loop-server.git
cd loop-server
Note: You may want to install a specific branch for testing vs defaulting to Master
npm install
ulimit -S -n 2048
npm test *
cd loadtests
make build
make test

Coverage report can be found here:
/loop-server/coverage/lcov-report/index.html

* This step requires the redis server to be installed and running:
Mac:
brew install redis
redis-server /usr/local/etc/redis.conf

Ubuntu Linux:
sudo apt-get install redis-server
sudo /usr/bin/redis-server /etc/redis/redis.conf
sudo tail -f /var/log/redis/redis-server.log

RHEL Linux:
Install redis from here: http://download.redis.io/releases
then
/usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
or similar

* Note: This will install a local copy of the Loads tool for use with the Loop-Server.

== Running the load test against the Loop-Server in Stage ==
* Stage environment:
$ cd loop-server/loadtests
$ make test
or
$ make test SERVER_URL=https://loop.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://loop.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the Loop-Server in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://loop.stage.mozaws.net

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing MSISDN-Gateway and the Loads tool on Localhost or AWS ==
* Installation:
** Install gmp, gmp-dev or gmp-devel
** Install ruby (very latest), ruby-dev or ruby-devel
** Install gem (required for fake_dynamo)
** Verify that gem is in your path
** Install redis-server to run the unit tests

* To install gmp
sudo yum -y install gmp, gmp-devel
or for Ubuntu
$ wget https://ftp.gnu.org/gnu/gmp/gmp-6.0.0a.tar.bz2
$ tar xvjf gmp-6.0.0a.tar.bz2
$ cd gmp-6.0.0
$ ./configure --prefix=/usr
$ make
$ make check
$ sudo make install

* To install ruby:
sudo yum -y install ruby, ruby-devel
or
sudo apt-get install ruby, ruby-dev

If this does not get you 1.9.3 or newer, then install manually:
Example:
$ wget http://cache.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p547.tar.gz
$ ./configure --prefix=/usr
$ make
$ sudo make install
(because for rhel, the default ruby version is 1.8.x.)
REF:
Main: https://www.ruby-lang.org/en/downloads/
Dev Tools: http://rubyinstaller.org/downloads/

* To install gem:
Grab rubygems from here: http://rubygems.org/pages/download
cd to rubygems directory
$ sudo ruby setup.rb

* To install fake_dynamo:
You should not have to install fake_dynamo since it is now part of the repo installer.
But if you do:
$ sudo gem install fake_dynamo
REF: https://github.com/ananthakumaran/fake_dynamo

* Install the msisdn-gateway repo:
$ git clone https://github.com/mozilla-services/msisdn-gateway.git
$ cd msisdn-gateway
Note: You may want to install a specific branch for testing vs defaulting to Master
$ sudo make install
(There is a bug open about the requirement to install with 'sudo')

* Note: This will install a local copy of the Loads tool for use with MSISDN-Gateway.

* Unit testing
Get redis-server installed
Start the server in a separate terminal or in the background with logging active
$ make test
The coverage report is here: msisdn-gateway/coverage/lcov-report/index.html

* Bugs:
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036158
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036174
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048451
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048456

== Running the load test against MSISDN-Gateway in Stage ==
* Building the load tests
$ cd loadtests
$ make build

* To load test the Stage environment:
$ make test SERVER_URL=https://msisdn.stage.mozaws.net
$ make bench SERVER_URL=https://msisdn.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the MSISDN-Gateway ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://msisdn.stage.mozaws.net

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (Loop-Server and MSISDN-Gateway):
*** Number of hits
*** Number of concurrent users

** For make bench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test

** For make megabench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* Loop-Server load test code
** The Loop-Server load test can not currently be configured in the code

* MSISDN-Gateway load test code
** The MSISDN-Gateway load test can be configured in the code - see the following lines:
** Test server URL: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
** Error percentages: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22

* General REFs:
** https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* Loop-Server
** TBD

* MSISDN-Gateway
** TBD

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Loop-Server Custom Metrics
** Opened web sockets
** Total web sockets
** Bytes/websockets
** addFailure (from the loads tool/client)

* MSISDN-Gateway Custom Metrics
** mt-flow
** ask-for-certificate
** try-wrong-code
** try-right-code
** momt-flow
** omxen-message-collision
** register
** unregister
** addFailure (from the loads tool/client)

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for Loop-Server (per server)
** /var/log/circus.log
** /var/log/loop_err.log
** /var/log/loop_out.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /var/log/nginx/access.log
** /var/log/nginx/error.log

* 2. Important logs for MSISDN-Gateway (per server)
** TBD

* Acceptable/Unacceptable Loop-Server errors:
hekad loop.stderr.log
The following are acceptable:
Decoder 'LoopServer-LoopServerDecoder' error: Failed parsing
Plugin 'AggregatorOutput' error: writing to heka.shared....

nginx logs:
The 200s (good stuff) and 101s (websockets) are acceptable.
The 499s are an artifact of the current load testing tool (V1).
You should only see them at the end of the load test.
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
Any percentage of 405s, 502s, or 503s is not acceptable.

Application logs:
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
See the loop_server.out.log file.

/var/log/loop_err.log
The following are acceptable: connect: res.on("header"): use on-headers module directly

In the Loads Cluster dashboard, watch for the following errors/failures:
string indices must be integers
OR
No JSON object could be decoded
OR
'hawk-session-token'

* Acceptable/Unacceptable MSISDN-Gateway errors:
The updated load test does generate a certain percentage of errors:
https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
So, expect to see a predefined percentage of 204s and 400s, along with the usual 200s in the nginx access logs.
The msisdn-gateway app logs should be clean with just msisdn and test data.

== Monitoring Loop Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver:
** https://app.stackdriver.com/groups/6664/stage-loads-cluster
** https://app.stackdriver.com/groups/11042/loop-server/stage-loop

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Loop Server: Check the following repos:
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla-services/docs/issues
** https://github.com/mozilla/loop-client
** https://github.com/mozilla-services/msisdn-gateway/issues

* Bugzilla
** and in Bugzilla for non-Verified bugs: http://mzl.la/1qPxizw

* OPs and Infra
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues

== References ==
* Repositories and Documentation
** Check here: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#References

* The QA Test Environments
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments

* Test Plans and other QA
** https://docs.google.com/a/mozilla.com/document/d/1xYZijHnJ1JeJhONm2AYvxSnD2tWztC16WGFyD5U_d6g/
** https://wiki.mozilla.org/Services/Loop/Try_Loop
** https://wiki.mozilla.org/Loop/QA
** https://wiki.mozilla.org/Loop/QA#Server
** https://intranet.mozilla.org/QA/Loop/MVP

* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/LoopTestEnvironments

2014-10-28T00:10:03Z

Jbonacci: /* Monitoring the Stage Environment */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Firefox Hello web client.

Browser: https://call.mozilla.com/config.js
now returns all of this info:
var loop = loop || {};
loop.config = {
bunch of loop-client specific info
};

Browser: https://call.mozilla.com/VERSION.txt

TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD

** loop-redis-stage - need the details

** Loop-Client: https://call.stage.mozaws.net

** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net

** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4 or 6.5

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad/loop-client.stderr.log
** /var/log/hekad/loop-client.stderr.log
** /media/ephemeral0/nginx/logs
*** default.access.log
*** default.error.log
*** loop-client.access.log
*** loop-client.error.log

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net
Welcome to the Firefox Hello web client.

Check the version: https://call.stage.mozaws.net/VERSION.txt

Check the TOS: https://call.stage.mozaws.net/legal/terms/

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {
serverUrl: 'https://loop.stage.mozaws.net',
and a bunch of other loop-client specific stuff
};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** http://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** https://app.stackdriver.com/groups/11042/loop-server/stage-loop

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Stackdriver
** https://app.stackdriver.com/groups/11044/loop-server/prod-loop

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopTestEnvironments

2014-10-28T00:09:40Z

Jbonacci: /* Monitoring the Production Environment */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Firefox Hello web client.

Browser: https://call.mozilla.com/config.js
now returns all of this info:
var loop = loop || {};
loop.config = {
bunch of loop-client specific info
};

Browser: https://call.mozilla.com/VERSION.txt

TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD

** loop-redis-stage - need the details

** Loop-Client: https://call.stage.mozaws.net

** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net

** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4 or 6.5

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad/loop-client.stderr.log
** /var/log/hekad/loop-client.stderr.log
** /media/ephemeral0/nginx/logs
*** default.access.log
*** default.error.log
*** loop-client.access.log
*** loop-client.error.log

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net
Welcome to the Firefox Hello web client.

Check the version: https://call.stage.mozaws.net/VERSION.txt

Check the TOS: https://call.stage.mozaws.net/legal/terms/

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {
serverUrl: 'https://loop.stage.mozaws.net',
and a bunch of other loop-client specific stuff
};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** http://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** None at this time. See https://bugzilla.mozilla.org/show_bug.cgi?id=1063134

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Stackdriver
** https://app.stackdriver.com/groups/11044/loop-server/prod-loop

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopTestEnvironments

2014-10-27T18:23:13Z

Jbonacci: /* Loop Client Stage Details */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Firefox Hello web client.

Browser: https://call.mozilla.com/config.js
now returns all of this info:
var loop = loop || {};
loop.config = {
bunch of loop-client specific info
};

Browser: https://call.mozilla.com/VERSION.txt

TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD

** loop-redis-stage - need the details

** Loop-Client: https://call.stage.mozaws.net

** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net

** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4 or 6.5

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad/loop-client.stderr.log
** /var/log/hekad/loop-client.stderr.log
** /media/ephemeral0/nginx/logs
*** default.access.log
*** default.error.log
*** loop-client.access.log
*** loop-client.error.log

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net
Welcome to the Firefox Hello web client.

Check the version: https://call.stage.mozaws.net/VERSION.txt

Check the TOS: https://call.stage.mozaws.net/legal/terms/

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {
serverUrl: 'https://loop.stage.mozaws.net',
and a bunch of other loop-client specific stuff
};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** http://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** None at this time. See https://bugzilla.mozilla.org/show_bug.cgi?id=1063134

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopTestEnvironments

2014-10-27T18:18:17Z

Jbonacci: /* Loop Server Production Environments */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Firefox Hello web client.

Browser: https://call.mozilla.com/config.js
now returns all of this info:
var loop = loop || {};
loop.config = {
bunch of loop-client specific info
};

Browser: https://call.mozilla.com/VERSION.txt

TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD

** loop-redis-stage - need the details

** Loop-Client: https://call.stage.mozaws.net

** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net

** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad
** /media/ephemeral0/nginx/logs

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net

Check the version: https://call.stage.mozaws.net/VERSION.txt

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {serverUrl: 'https://https://loop.stage.mozaws.net'};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** http://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** None at this time. See https://bugzilla.mozilla.org/show_bug.cgi?id=1063134

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopServerLoadTesting

2014-10-16T18:24:04Z

Jbonacci: /* Installing MSISDN-Gateway and the Loads tool on Localhost or AWS */

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
* Loop Server
For now, just run a quick loadtest 'make test'
cd loop-server
cd loadtests
make test SERVER_URL=https://loop.stage.mozaws.net

* Loop Client
Check https://call.stage.mozaws.net/config.js
Should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.stage.mozaws.net'};
and
WIP from the client team: end to end tests
Also:
https://call.stage.mozaws.net
curl https://call.stage.mozaws.net
curl -I https://call.stage.mozaws.net

and now the TOS: https://call.stage.mozaws.net/legal/terms/

* MSISDN Gateway
In the browser: https://msisdn.stage.mozaws.net
or do the following from a command line:
curl https://msisdn.stage.mozaws.net
curl -I https://msisdn.stage.mozaws.net
and
For now, just run a quick loadtest 'make test'
cd msisdn-gateway
cd loadtests
make test SERVER_URL=https://msisdn.stage.mozaws.net
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Quick Verification of Production Deployments ==
* This is a quick sanity test of the environment for after each Production deployment.
* Loop Server
In the browser: https://loop.services.mozilla.com
or do the following from a command line:
curl https://loop.services.mozilla.com
curl -I https://loop.services.mozilla.com

Then run a few 'make test' commands from the loadtests folder:
make test SERVER_URL=https://loop.services.mozilla.com
Note: this does hit a live third-party server

Then perform actual loop testing via desktop (Aurora/Nightly so far) and FxOS (2.1)
Verify that requests and strings point to Production environments

* Loop Client
In the browser: https://call.mozilla.com
should return "Welcome to the Loop web client."

In the browser: https://call.mozilla.com/config.js
should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

In the browser: https://call.mozilla.com/VERSION.txt
should return the version and build string info

and now the TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com
curl -I https://call.mozilla.com

Quick end-to-end tests:
Desktop: browser to browser
Desktop to FxOS
FxOS to Desktop
Two FxOS devices

* MSISDN Gateway
In the browser: https://msisdn.services.mozilla.com
or do the following from a command line:
curl https://msisdn.services.mozilla.com
curl -I https://msisdn.services.mozilla.com
Or
Run a single 'make test' command from the loadtests folder:
make test SERVER_URL=https://msisdn.services.mozilla.com
Note: this does hit a live third-party, so limit the check to a single run.
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Installing Loop-Server and the Loads tool on Localhost or AWS ==
* Installation:
git clone https://github.com/mozilla-services/loop-server.git
cd loop-server
Note: You may want to install a specific branch for testing vs defaulting to Master
npm install
ulimit -S -n 2048
npm test *
cd loadtests
make build
make test

Coverage report can be found here:
/loop-server/coverage/lcov-report/index.html

* This step requires the redis server to be installed and running:
Mac:
brew install redis
redis-server /usr/local/etc/redis.conf

Ubuntu Linux:
sudo apt-get install redis-server
sudo /usr/bin/redis-server /etc/redis/redis.conf
sudo tail -f /var/log/redis/redis-server.log

RHEL Linux:
Install redis from here: http://download.redis.io/releases
then
/usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
or similar

* Note: This will install a local copy of the Loads tool for use with the Loop-Server.

== Running the load test against the Loop-Server in Stage ==
* Stage environment:
$ cd loop-server/loadtests
$ make test
or
$ make test SERVER_URL=https://loop.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://loop.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the Loop-Server in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://loop.stage.mozaws.net

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing MSISDN-Gateway and the Loads tool on Localhost or AWS ==
* Installation:
** Install gmp, gmp-dev or gmp-devel
** Install ruby (very latest), ruby-dev or ruby-devel
** Install gem (required for fake_dynamo)
** Verify that gem is in your path
** Install redis-server to run the unit tests

* To install gmp
sudo yum -y install gmp, gmp-devel
or for Ubuntu
$ wget https://ftp.gnu.org/gnu/gmp/gmp-6.0.0a.tar.bz2
$ tar xvjf gmp-6.0.0a.tar.bz2
$ cd gmp-6.0.0
$ ./configure --prefix=/usr
$ make
$ make check
$ sudo make install

* To install ruby:
sudo yum -y install ruby, ruby-devel
or
sudo apt-get install ruby, ruby-dev

If this does not get you 1.9.3 or newer, then install manually:
Example:
$ wget http://cache.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p547.tar.gz
$ ./configure --prefix=/usr
$ make
$ sudo make install
(because for rhel, the default ruby version is 1.8.x.)
REF:
Main: https://www.ruby-lang.org/en/downloads/
Dev Tools: http://rubyinstaller.org/downloads/

* To install gem:
Grab rubygems from here: http://rubygems.org/pages/download
cd to rubygems directory
$ sudo ruby setup.rb

* To install fake_dynamo:
You should not have to install fake_dynamo since it is now part of the repo installer.
But if you do:
$ sudo gem install fake_dynamo
REF: https://github.com/ananthakumaran/fake_dynamo

* Install the msisdn-gateway repo:
$ git clone https://github.com/mozilla-services/msisdn-gateway.git
$ cd msisdn-gateway
Note: You may want to install a specific branch for testing vs defaulting to Master
$ sudo make install
(There is a bug open about the requirement to install with 'sudo')

* Note: This will install a local copy of the Loads tool for use with MSISDN-Gateway.

* Unit testing
Get redis-server installed
Start the server in a separate terminal or in the background with logging active
$ make test
The coverage report is here: msisdn-gateway/coverage/lcov-report/index.html

* Bugs:
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036158
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036174
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048451
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048456

== Running the load test against MSISDN-Gateway in Stage ==
* Building the load tests
$ cd loadtests
$ make build

* To load test the Stage environment:
$ make test SERVER_URL=https://msisdn.stage.mozaws.net
$ make bench SERVER_URL=https://msisdn.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the MSISDN-Gateway ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://msisdn.stage.mozaws.net

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (Loop-Server and MSISDN-Gateway):
*** Number of hits
*** Number of concurrent users

** For make bench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test

** For make megabench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* Loop-Server load test code
** The Loop-Server load test can not currently be configured in the code

* MSISDN-Gateway load test code
** The MSISDN-Gateway load test can be configured in the code - see the following lines:
** Test server URL: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
** Error percentages: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22

* General REFs:
** https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* Loop-Server
** TBD

* MSISDN-Gateway
** TBD

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Loop-Server Custom Metrics
** Opened web sockets
** Total web sockets
** Bytes/websockets
** addFailure (from the loads tool/client)

* MSISDN-Gateway Custom Metrics
** mt-flow
** ask-for-certificate
** try-wrong-code
** try-right-code
** momt-flow
** omxen-message-collision
** register
** unregister
** addFailure (from the loads tool/client)

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for Loop-Server (per server)
** /var/log/circus.log
** /var/log/loop_err.log
** /var/log/loop_out.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /var/log/nginx/access.log
** /var/log/nginx/error.log

* 2. Important logs for MSISDN-Gateway (per server)
** TBD

* Acceptable/Unacceptable Loop-Server errors:
hekad loop.stderr.log
The following are acceptable:
Decoder 'LoopServer-LoopServerDecoder' error: Failed parsing
Plugin 'AggregatorOutput' error: writing to heka.shared....

nginx logs:
The 200s (good stuff) and 101s (websockets) are acceptable.
The 499s are an artifact of the current load testing tool (V1).
You should only see them at the end of the load test.
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
Any percentage of 405s, 502s, or 503s is not acceptable.

Application logs:
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
See the loop_server.out.log file.

/var/log/loop_err.log
The following are acceptable: connect: res.on("header"): use on-headers module directly

In the Loads Cluster dashboard, watch for the following errors/failures:
string indices must be integers
OR
No JSON object could be decoded
OR
'hawk-session-token'

* Acceptable/Unacceptable MSISDN-Gateway errors:
The updated load test does generate a certain percentage of errors:
https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
So, expect to see a predefined percentage of 204s and 400s, along with the usual 200s in the nginx access logs.
The msisdn-gateway app logs should be clean with just msisdn and test data.

== Monitoring Loop Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Loop Server: Check the following repos:
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla-services/docs/issues
** https://github.com/mozilla/loop-client
** https://github.com/mozilla-services/msisdn-gateway/issues

* Bugzilla
** and in Bugzilla for non-Verified bugs: http://mzl.la/1qPxizw

* OPs and Infra
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues

== References ==
* Repositories and Documentation
** Check here: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#References

* The QA Test Environments
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments

* Test Plans and other QA
** https://docs.google.com/a/mozilla.com/document/d/1xYZijHnJ1JeJhONm2AYvxSnD2tWztC16WGFyD5U_d6g/
** https://wiki.mozilla.org/Services/Loop/Try_Loop
** https://wiki.mozilla.org/Loop/QA
** https://wiki.mozilla.org/Loop/QA#Server
** https://intranet.mozilla.org/QA/Loop/MVP

* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/LoopServerLoadTesting

2014-10-16T18:23:16Z

Jbonacci: /* Installing Loop-Server and the Loads tool on Localhost or AWS */

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
* Loop Server
For now, just run a quick loadtest 'make test'
cd loop-server
cd loadtests
make test SERVER_URL=https://loop.stage.mozaws.net

* Loop Client
Check https://call.stage.mozaws.net/config.js
Should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.stage.mozaws.net'};
and
WIP from the client team: end to end tests
Also:
https://call.stage.mozaws.net
curl https://call.stage.mozaws.net
curl -I https://call.stage.mozaws.net

and now the TOS: https://call.stage.mozaws.net/legal/terms/

* MSISDN Gateway
In the browser: https://msisdn.stage.mozaws.net
or do the following from a command line:
curl https://msisdn.stage.mozaws.net
curl -I https://msisdn.stage.mozaws.net
and
For now, just run a quick loadtest 'make test'
cd msisdn-gateway
cd loadtests
make test SERVER_URL=https://msisdn.stage.mozaws.net
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Quick Verification of Production Deployments ==
* This is a quick sanity test of the environment for after each Production deployment.
* Loop Server
In the browser: https://loop.services.mozilla.com
or do the following from a command line:
curl https://loop.services.mozilla.com
curl -I https://loop.services.mozilla.com

Then run a few 'make test' commands from the loadtests folder:
make test SERVER_URL=https://loop.services.mozilla.com
Note: this does hit a live third-party server

Then perform actual loop testing via desktop (Aurora/Nightly so far) and FxOS (2.1)
Verify that requests and strings point to Production environments

* Loop Client
In the browser: https://call.mozilla.com
should return "Welcome to the Loop web client."

In the browser: https://call.mozilla.com/config.js
should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

In the browser: https://call.mozilla.com/VERSION.txt
should return the version and build string info

and now the TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com
curl -I https://call.mozilla.com

Quick end-to-end tests:
Desktop: browser to browser
Desktop to FxOS
FxOS to Desktop
Two FxOS devices

* MSISDN Gateway
In the browser: https://msisdn.services.mozilla.com
or do the following from a command line:
curl https://msisdn.services.mozilla.com
curl -I https://msisdn.services.mozilla.com
Or
Run a single 'make test' command from the loadtests folder:
make test SERVER_URL=https://msisdn.services.mozilla.com
Note: this does hit a live third-party, so limit the check to a single run.
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Installing Loop-Server and the Loads tool on Localhost or AWS ==
* Installation:
git clone https://github.com/mozilla-services/loop-server.git
cd loop-server
Note: You may want to install a specific branch for testing vs defaulting to Master
npm install
ulimit -S -n 2048
npm test *
cd loadtests
make build
make test

Coverage report can be found here:
/loop-server/coverage/lcov-report/index.html

* This step requires the redis server to be installed and running:
Mac:
brew install redis
redis-server /usr/local/etc/redis.conf

Ubuntu Linux:
sudo apt-get install redis-server
sudo /usr/bin/redis-server /etc/redis/redis.conf
sudo tail -f /var/log/redis/redis-server.log

RHEL Linux:
Install redis from here: http://download.redis.io/releases
then
/usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
or similar

* Note: This will install a local copy of the Loads tool for use with the Loop-Server.

== Running the load test against the Loop-Server in Stage ==
* Stage environment:
$ cd loop-server/loadtests
$ make test
or
$ make test SERVER_URL=https://loop.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://loop.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the Loop-Server in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://loop.stage.mozaws.net

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing MSISDN-Gateway and the Loads tool on Localhost or AWS ==
* Installation:
** Install gmp, gmp-dev or gmp-devel
** Install ruby (very latest), ruby-dev or ruby-devel
** Install gem (required for fake_dynamo)
** Verify that gem is in your path
** Install redis-server to run the unit tests

* To install gmp
sudo yum -y install gmp, gmp-devel
or for Ubuntu
$ wget https://ftp.gnu.org/gnu/gmp/gmp-6.0.0a.tar.bz2
$ tar xvjf gmp-6.0.0a.tar.bz2
$ cd gmp-6.0.0
$ ./configure --prefix=/usr
$ make
$ make check
$ sudo make install

* To install ruby:
sudo yum -y install ruby, ruby-devel
or
sudo apt-get install ruby, ruby-dev

If this does not get you 1.9.3 or newer, then install manually:
Example:
$ wget http://cache.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p547.tar.gz
$ ./configure --prefix=/usr
$ make
$ sudo make install
(because for rhel, the default ruby version is 1.8.x.)
REF:
Main: https://www.ruby-lang.org/en/downloads/
Dev Tools: http://rubyinstaller.org/downloads/

* To install gem:
Grab rubygems from here: http://rubygems.org/pages/download
cd to rubygems directory
$ sudo ruby setup.rb

* To install fake_dynamo:
You should not have to install fake_dynamo since it is now part of the repo installer.
But if you do:
$ sudo gem install fake_dynamo
REF: https://github.com/ananthakumaran/fake_dynamo

* Install the msisdn-gateway repo:
$ git clone https://github.com/mozilla-services/msisdn-gateway.git
$ cd msisdn-gateway
$ sudo make install
(There is a bug open about the requirement to install with 'sudo')

* Note: This will install a local copy of the Loads tool for use with MSISDN-Gateway.

* Unit testing
Get redis-server installed
Start the server in a separate terminal or in the background with logging active
$ make test
The coverage report is here: msisdn-gateway/coverage/lcov-report/index.html

* Bugs:
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036158
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036174
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048451
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048456

== Running the load test against MSISDN-Gateway in Stage ==
* Building the load tests
$ cd loadtests
$ make build

* To load test the Stage environment:
$ make test SERVER_URL=https://msisdn.stage.mozaws.net
$ make bench SERVER_URL=https://msisdn.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the MSISDN-Gateway ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://msisdn.stage.mozaws.net

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (Loop-Server and MSISDN-Gateway):
*** Number of hits
*** Number of concurrent users

** For make bench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test

** For make megabench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* Loop-Server load test code
** The Loop-Server load test can not currently be configured in the code

* MSISDN-Gateway load test code
** The MSISDN-Gateway load test can be configured in the code - see the following lines:
** Test server URL: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
** Error percentages: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22

* General REFs:
** https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* Loop-Server
** TBD

* MSISDN-Gateway
** TBD

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Loop-Server Custom Metrics
** Opened web sockets
** Total web sockets
** Bytes/websockets
** addFailure (from the loads tool/client)

* MSISDN-Gateway Custom Metrics
** mt-flow
** ask-for-certificate
** try-wrong-code
** try-right-code
** momt-flow
** omxen-message-collision
** register
** unregister
** addFailure (from the loads tool/client)

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for Loop-Server (per server)
** /var/log/circus.log
** /var/log/loop_err.log
** /var/log/loop_out.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /var/log/nginx/access.log
** /var/log/nginx/error.log

* 2. Important logs for MSISDN-Gateway (per server)
** TBD

* Acceptable/Unacceptable Loop-Server errors:
hekad loop.stderr.log
The following are acceptable:
Decoder 'LoopServer-LoopServerDecoder' error: Failed parsing
Plugin 'AggregatorOutput' error: writing to heka.shared....

nginx logs:
The 200s (good stuff) and 101s (websockets) are acceptable.
The 499s are an artifact of the current load testing tool (V1).
You should only see them at the end of the load test.
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
Any percentage of 405s, 502s, or 503s is not acceptable.

Application logs:
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
See the loop_server.out.log file.

/var/log/loop_err.log
The following are acceptable: connect: res.on("header"): use on-headers module directly

In the Loads Cluster dashboard, watch for the following errors/failures:
string indices must be integers
OR
No JSON object could be decoded
OR
'hawk-session-token'

* Acceptable/Unacceptable MSISDN-Gateway errors:
The updated load test does generate a certain percentage of errors:
https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
So, expect to see a predefined percentage of 204s and 400s, along with the usual 200s in the nginx access logs.
The msisdn-gateway app logs should be clean with just msisdn and test data.

== Monitoring Loop Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Loop Server: Check the following repos:
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla-services/docs/issues
** https://github.com/mozilla/loop-client
** https://github.com/mozilla-services/msisdn-gateway/issues

* Bugzilla
** and in Bugzilla for non-Verified bugs: http://mzl.la/1qPxizw

* OPs and Infra
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues

== References ==
* Repositories and Documentation
** Check here: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#References

* The QA Test Environments
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments

* Test Plans and other QA
** https://docs.google.com/a/mozilla.com/document/d/1xYZijHnJ1JeJhONm2AYvxSnD2tWztC16WGFyD5U_d6g/
** https://wiki.mozilla.org/Services/Loop/Try_Loop
** https://wiki.mozilla.org/Loop/QA
** https://wiki.mozilla.org/Loop/QA#Server
** https://intranet.mozilla.org/QA/Loop/MVP

* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/TokenServerAndSyncLoadTesting

2014-10-16T18:22:43Z

Jbonacci: /* Installing Sync 1.5 and the Loads tool on Localhost or AWS */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier sections below...
** A Tokenserver+Verifier stack: See the TokenServer sections below...

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.

* TokenServer+Verifier Stage environment:
From the browser: https://token.stage.mozaws.net
curl https://token.stage.mozaws.net
curl -I https://token.stage.mozaws.net

Use the simple "make test" command from an install of tokenserver on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://token.stage.mozaws.net

Alternate method:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Stage:
bin/sync-cli.js -e EMAIL -p PASSWORD --env stage -t COLLECTION
where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms

* Verifier Stage environment:
In the browser: https://verifier.stage.mozaws.net/
curl https://verifier.stage.mozaws.net
curl -I https://verifier.stage.mozaws.net

Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.stage.mozaws.net

* Sync Server Stage environment:
Install server-syncstorage to the local host or AWS instance (see below)
$ cd server-syncstorage
Quick test against the TokenServer
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server <Stage TokenServer>
Current example:
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server
https://token.stage.mozaws.net/1.0/sync/1.5
Quick tests against the Sync nodes
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py <Stage Sync Node>#<Node Secret>
Current examples:
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
Get the Node Secret information from OPs

* Using TPS
** The TPS FxA/Sync automated tests can be used as well, but the following file will have to be edited to add Stage environment configuration parameters: https://github.com/mozilla/gecko-dev/blob/master/testing/tps/tps/testrunner.py
** See the following wiki page for more information: https://wiki.mozilla.org/User_Services/Sync/Run_TPS
** See also: https://bugzilla.mozilla.org/show_bug.cgi?id=1006675

== Quick Verification Of Production Deployments ==
* This is a quick sanity test of the environment after a new deployment.

* Tokenserver+Verifier Production Environment
In the browser: https://token.services.mozilla.com
curl https://token.services.mozilla.com
curl -I https://token.services.mozilla.com
Then:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Production:
bin/sync-cli.js -e PROD-EMAIL -p PASSWORD -t COLLECTION
where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms

* Verifier Production Environment
In the browser: https://verifier.accounts.firefox.com
curl https://verifier.accounts.firefox.com
curl -I https://verifier.accounts.firefox.com
Then:
Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.accounts.firefox.com

* Sync Server Stage environment
Sign in with a known FxA account and sync data with a current Production account (sync node).
Create a new FxA account and set up sync.

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment

== Installing BrowserID-Verifier and the Loads tool on Localhost or AWS ==
* Installation:
$ git clone git://github.com/mozilla/browserid-verifier
$ cd browserid-verifier
Note: You may want to install a specific branch for testing vs defaulting to Master
$ npm install
$ npm test
$ cd loadtest
$ make build
Note: This should hit Stage by default: SERVER_URL=https://verifier.stage.mozaws.net

* Note: This will install a local copy of the Loads tool for use with the verifier.

== Running the load test against the Verifier in Stage ==
* Stage environment:
$ make test
or
$ make test SERVER_URL=https://verifier.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://verifier.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Verifier hits the Stage mockmyid server

* Production environment:
$ make test SERVER_URL=https://verifier.accounts.firefox.com
$ make bench SERVER_URL=https://verifier.accounts.firefox.com

== Using the Loads V1 Services Cluster for the Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://verifier.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=TBD

* Production environment:
$ make megabench SERVER_URL=https://verifier.accounts.firefox.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing TokenServer+Verifier and the Loads tool on Localhost or AWS ==
* Installation:
$ git clone https://github.com/mozilla-services/tokenserver
$ cd tokenserver
Note: You may want to install a specific branch for testing vs defaulting to Master
$ make build
$ make test
Note: This is for local testing only
$ cd loadtest
$ make build
Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com

* Note: This will install a local copy of the Loads tool for use with TokenServer+Verifier.

== Running the load test against TokenServer+Verifier in Stage ==
* Stage environment:
$ make test SERVER_URL=https://token.stage.mozaws.net
$ make bench SERVER_URL=https://token.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: This also hits the Stage Verifier, which in turns hits the Stage mockmyid server

* And while we are at it...
* Dev environment:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org

* Production environment:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com

== Using the Loads V1 Services Cluster for TokenServer+Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://token.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=https://token.dev.lcip.org

* Production environment:
$ make megabench SERVER_URL=https://token.services.mozilla.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest

== Installing Sync 1.5 and the Loads tool on Localhost or AWS ==
Installation:
$ git clone https://github.com/mozilla-services/server-syncstorage/
$ cd server-syncstorage
Note: You may want to install a specific branch for testing vs defaulting to Master
$ make build
$ make test
$ cd loadtest
$ make build

* Note: This will install a local copy of the Loads tool for use with Sync 1.5.

== Running the load test against Sync 1.5 in Stage ==
* Loads against specific Sync nodes in Stage
$ make test SERVER_URL=https://your.storagenode.here#SECRET
$ make bench SERVER_URL=https://your.storagenode.here#SECRET
Sync Stage nodes:
https://sync-1-us-east-1.stage.mozaws.net
https://sync-2-us-east-1.stage.mozaws.net
https://sync-3-us-east-1.stage.mozaws.net
...etc...

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.

== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://your.storagenode.here#SECRET

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest

== Running a combined load test against TokenServer+Verifier and Sync 1.5 in Stage ==
* A combined loads test against TokenServer and Sync 1.5 in Stage
* This is done via the server-syncstorage directory that was cloned and built above
$ cd server-syncstorage
$ cd loadtest
$ make test SERVER_URL=https://your.tokenserver.here
$ make bench SERVER_URL=https://your.tokenserver.here

Examples for Stage:
$ make test SERVER_URL=https://token.stage.mozaws.net
$ make bench SERVER_URL=https://token.stage.mozaws.net
See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer.2BVerifier_Stage_Environment

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Tokenserver hits the Stage Verifier, which, in turn, hits the mockmyid server.

* And while we are at it...
Dev environment:
Examples:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org

Prod environment:
Examples:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com

See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA.2C_TokenServer.2C_and_Sync_Production_Environments
and https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer_and_Sync_1.5_Dev_Environments

== Using the Loads V1 Services Cluster for a combined load test in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://token.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=https://token.dev.lcip.org

* Prod environment:
$ make megabench SERVER_URL=https://token.services.mozilla.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of hits
*** Number of concurrent users

** For make bench (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of concurrent users
*** Duration of test

** For make megabench (using the LoadsCluster with BrowserID-Verifier, TokenSerer, Sync, Combined):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)
*** SSH (the user account needed to SSH into the loads cluster - the default is ubuntu)

* Tokenserver load test code
** The Tokenserver load test can be configured - see the following lines:
** Basic Settings: https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** MockMyID: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L19-L36
** Percentages: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L39-L51

* Verifier load test code
** The Verifier load test can be configured - see the following lines:
** Various settings: https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py#L13-L53

* Sync Server load test code
** The Sync Server load test can be configured - see the following lines:
** Setting MockMyID: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L26-L45
** Setting test distributions: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L48-L83

* REFs:
** https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py
** https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py
** https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* TokenServer
** File location: tokenserver/loadtest/loadtest.py
** Inside NoteAssignmentTest, test_realistic is the main load test; the others are for specific behaviors
** The test runs as following:
95% ask for assertions on existing users (on a DB filled by test_single_token_exchange)
4% ask for assertion on a new use
1% ask for a bad assertion
** A bug has been filed to get the following additional coverage for the load test:
*** generation numbers in assertion
*** client state string
** A bug has been filed to get some integration tests written:
*** to cover the edge/error cases not in the load test
*** to be pointed at a remote server

* Sync
** File location: server-syncstorage/loadtest/stress.py
** This is the Sync 2.0 load test that has been back-ported for Sync 1.5.
** The stress.py file is fully configurable for the following:
*** client probability
*** client distribution
*** collections
** A bug has been filed to add support for load testing tabs
*** The tab collection it uses memcache; we need to figure out a way to test it without overloading the server
** There are currently no constants to define how to select percentages per collection type
** Right now, we need to manually configure the collections list in stress.py:
*** collections = ['bookmarks', 'forms', 'passwords', 'history', 'prefs']
*** Basically, you can add more entries of each type, since the load test (per user/again/hit/pass) picks randomly from the list for any given request...

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Tokenserver Custom Metrics
** addFailure

* Verifier Custom Metrics
** addFailure

* Sync Custom Metrics
** addFailure

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for TokenServer (per server)
** /media/ephemeral0/logs/
** /media/ephemeral0/nginx/logs/default.access.log
** /media/ephemeral0/nginx/logs/default.error.log
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* 2. Important logs for Verifier (per server)
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* 3. Important error logs for Sync (per Sync node)
** /media/ephemeral0/logs/
** /media/ephemeral0/nginx/access.log
** /media/ephemeral0/error.log
** /media/ephemeral0/sync/sync.err
** /media/ephemeral0/sync/sync.log

* Acceptable TokenServer errors:
1% - 2% failures (as the following)
token.log:
"name": "token.assertion.invalid_signature_error"
"name": "token.assertion.verify_failure"
nginx access.log:
401s
NOTE: Values can be tweaked here:
https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L58-L60

The following types of errors are known:
/media/ephemeral0/logs/tokenserver/token.error.log
Exception KeyError: KeyError(49564400,) in <module 'threading'...
/media/ephemeral0/logs/tokenserver/token.log
..."Starting new HTTP connection (9): 127.0.0.1", "hostname": ...
{"error": "StopIteration()", "traceback": "Uncaught exception:\n
File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
..."Connection pool is full, discarding connection: 127.0.0.1", "...
Also, any 499s are probably an artifact of the current (V1) load test.
REF:
https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
https://bugzilla.mozilla.org/show_bug.cgi?id=1040397

OLD: Also, it may be the case that the following errors are "acceptable" if TS Stage is larger than Verifier Stage:
/media/ephemeral0/logs/tokenserver/token.error.log
Verifier-related errors of these types:
"HttpConnectionPool is full, discarding connection: verifier.stage.mozaws.net"
"Resetting dropped connection: verifier.stage.mozaws.net"
"Starting new HTTPS connection (179): verifier.stage.mozaws.net"

* Acceptable Verifier errors:
The verifier_out.log will show errors of the following types:
result: 'failure',\n reason: 'untrusted issuer...'
result: 'failure',\n reason: 'expired'
result: 'failure',\n reason: 'algorithms do not match'
result: 'failure',\n reason: 'audience mismatch: scheme mismatch'
Also, any 499s in the nginx logs are probably an artifact of the current (V1) load test.

* Acceptable Sync node errors:
In the nginx access.log files:
We will see some percentage of 404s. Right now we see the following:
14% 404s (compared to the total count of 200s)
with the config set up as follows:
users = 20
duration = 1800
agents = 5
Ideally, the overall percentage of 404s should drop the longer the load test.
Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
although they may show up in the logs after running the remote integration tests.
Also, any 499s are probably an artifact of the current (V1) load test.

In /var/log/hekad/sync_1_5.stderr.log
You may see some Decoder 'Sync-1_5-SlowQuery-MySqlSlowQueryDecoder' error: Failed parsing
and a lot of BSO INSERTs

In /media/ephemeral0/logs/sync/sync.err
You should see expected skew and QueuePool messages and Deprecation warnings
Also, these are known
Exception SystemExit
Exception KeyError
This is probably https://bugzilla.mozilla.org/show_bug.cgi?id=1040397

== Monitoring TS and Sync Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Tokenserver:
** Repo: https://github.com/mozilla-services/tokenserver/issues
** Bugzilla: http://mzl.la/1s4qZKn

* BrowserID-Verifier:
** Repo: https://github.com/mozilla/browserid-verifier/issues
** Bugzilla: no specific cateogory

* Sync:
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Bugzilla: http://mzl.la/VUrYQ5

* OPs and Infrastructure
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-aws/issues
** https://github.com/mozilla-services/loads-web/issues

== References ==
* Other URLs
** http://restmail.net
** https://mockmyid.com
* Repositories
** https://github.com/mozilla-services/tokenserver
** https://github.com/mozilla-services/server-syncstorage
** https://github.com/mozilla/browserid-verifier
** https://github.com/mozilla/browserid-local-verify
** https://github.com/callahad/mockmyid
** https://github.com/mozilla/restmail.net
* Documentation
** https://docs.services.mozilla.com
** https://docs.services.mozilla.com/token/
** https://wiki.mozilla.org/Services/Sagrada/TokenServer
** https://docs.services.mozilla.com/sync/
* The QA Test Environments:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
* Deploying the FxA Load Test environment for broker/agents usage:
** https://github.com/mozilla/fxa-deployment
* Sync 1.5 protocol, documentation, etc.
** https://github.com/mozilla-services/docs
** https://docs.services.mozilla.com/#how-to
** https://docs.services.mozilla.com/howtos/run-fxa.html
** https://docs.services.mozilla.com/token/apis.html
** https://docs.services.mozilla.com/storage/apis-1.5.html
** https://docs.services.mozilla.com/howtos/run-sync-1.5.html
** https://docs.services.mozilla.com/howtos/run-sync-1.5.html
** https://github.com/mozilla-services/syncserver
* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/TokenServerAndSyncLoadTesting

2014-10-16T18:22:25Z

Jbonacci: /* Installing TokenServer+Verifier and the Loads tool on Localhost or AWS */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier sections below...
** A Tokenserver+Verifier stack: See the TokenServer sections below...

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.

* TokenServer+Verifier Stage environment:
From the browser: https://token.stage.mozaws.net
curl https://token.stage.mozaws.net
curl -I https://token.stage.mozaws.net

Use the simple "make test" command from an install of tokenserver on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://token.stage.mozaws.net

Alternate method:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Stage:
bin/sync-cli.js -e EMAIL -p PASSWORD --env stage -t COLLECTION
where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms

* Verifier Stage environment:
In the browser: https://verifier.stage.mozaws.net/
curl https://verifier.stage.mozaws.net
curl -I https://verifier.stage.mozaws.net

Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.stage.mozaws.net

* Sync Server Stage environment:
Install server-syncstorage to the local host or AWS instance (see below)
$ cd server-syncstorage
Quick test against the TokenServer
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server <Stage TokenServer>
Current example:
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server
https://token.stage.mozaws.net/1.0/sync/1.5
Quick tests against the Sync nodes
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py <Stage Sync Node>#<Node Secret>
Current examples:
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
Get the Node Secret information from OPs

* Using TPS
** The TPS FxA/Sync automated tests can be used as well, but the following file will have to be edited to add Stage environment configuration parameters: https://github.com/mozilla/gecko-dev/blob/master/testing/tps/tps/testrunner.py
** See the following wiki page for more information: https://wiki.mozilla.org/User_Services/Sync/Run_TPS
** See also: https://bugzilla.mozilla.org/show_bug.cgi?id=1006675

== Quick Verification Of Production Deployments ==
* This is a quick sanity test of the environment after a new deployment.

* Tokenserver+Verifier Production Environment
In the browser: https://token.services.mozilla.com
curl https://token.services.mozilla.com
curl -I https://token.services.mozilla.com
Then:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Production:
bin/sync-cli.js -e PROD-EMAIL -p PASSWORD -t COLLECTION
where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms

* Verifier Production Environment
In the browser: https://verifier.accounts.firefox.com
curl https://verifier.accounts.firefox.com
curl -I https://verifier.accounts.firefox.com
Then:
Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.accounts.firefox.com

* Sync Server Stage environment
Sign in with a known FxA account and sync data with a current Production account (sync node).
Create a new FxA account and set up sync.

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment

== Installing BrowserID-Verifier and the Loads tool on Localhost or AWS ==
* Installation:
$ git clone git://github.com/mozilla/browserid-verifier
$ cd browserid-verifier
Note: You may want to install a specific branch for testing vs defaulting to Master
$ npm install
$ npm test
$ cd loadtest
$ make build
Note: This should hit Stage by default: SERVER_URL=https://verifier.stage.mozaws.net

* Note: This will install a local copy of the Loads tool for use with the verifier.

== Running the load test against the Verifier in Stage ==
* Stage environment:
$ make test
or
$ make test SERVER_URL=https://verifier.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://verifier.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Verifier hits the Stage mockmyid server

* Production environment:
$ make test SERVER_URL=https://verifier.accounts.firefox.com
$ make bench SERVER_URL=https://verifier.accounts.firefox.com

== Using the Loads V1 Services Cluster for the Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://verifier.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=TBD

* Production environment:
$ make megabench SERVER_URL=https://verifier.accounts.firefox.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing TokenServer+Verifier and the Loads tool on Localhost or AWS ==
* Installation:
$ git clone https://github.com/mozilla-services/tokenserver
$ cd tokenserver
Note: You may want to install a specific branch for testing vs defaulting to Master
$ make build
$ make test
Note: This is for local testing only
$ cd loadtest
$ make build
Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com

* Note: This will install a local copy of the Loads tool for use with TokenServer+Verifier.

== Running the load test against TokenServer+Verifier in Stage ==
* Stage environment:
$ make test SERVER_URL=https://token.stage.mozaws.net
$ make bench SERVER_URL=https://token.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: This also hits the Stage Verifier, which in turns hits the Stage mockmyid server

* And while we are at it...
* Dev environment:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org

* Production environment:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com

== Using the Loads V1 Services Cluster for TokenServer+Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://token.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=https://token.dev.lcip.org

* Production environment:
$ make megabench SERVER_URL=https://token.services.mozilla.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest

== Installing Sync 1.5 and the Loads tool on Localhost or AWS ==
Installation:
$ git clone https://github.com/mozilla-services/server-syncstorage/
$ cd server-syncstorage
$ make build
$ make test
$ cd loadtest
$ make build

* Note: This will install a local copy of the Loads tool for use with Sync 1.5.

== Running the load test against Sync 1.5 in Stage ==
* Loads against specific Sync nodes in Stage
$ make test SERVER_URL=https://your.storagenode.here#SECRET
$ make bench SERVER_URL=https://your.storagenode.here#SECRET
Sync Stage nodes:
https://sync-1-us-east-1.stage.mozaws.net
https://sync-2-us-east-1.stage.mozaws.net
https://sync-3-us-east-1.stage.mozaws.net
...etc...

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.

== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://your.storagenode.here#SECRET

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest

== Running a combined load test against TokenServer+Verifier and Sync 1.5 in Stage ==
* A combined loads test against TokenServer and Sync 1.5 in Stage
* This is done via the server-syncstorage directory that was cloned and built above
$ cd server-syncstorage
$ cd loadtest
$ make test SERVER_URL=https://your.tokenserver.here
$ make bench SERVER_URL=https://your.tokenserver.here

Examples for Stage:
$ make test SERVER_URL=https://token.stage.mozaws.net
$ make bench SERVER_URL=https://token.stage.mozaws.net
See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer.2BVerifier_Stage_Environment

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Tokenserver hits the Stage Verifier, which, in turn, hits the mockmyid server.

* And while we are at it...
Dev environment:
Examples:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org

Prod environment:
Examples:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com

See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA.2C_TokenServer.2C_and_Sync_Production_Environments
and https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer_and_Sync_1.5_Dev_Environments

== Using the Loads V1 Services Cluster for a combined load test in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://token.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=https://token.dev.lcip.org

* Prod environment:
$ make megabench SERVER_URL=https://token.services.mozilla.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of hits
*** Number of concurrent users

** For make bench (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of concurrent users
*** Duration of test

** For make megabench (using the LoadsCluster with BrowserID-Verifier, TokenSerer, Sync, Combined):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)
*** SSH (the user account needed to SSH into the loads cluster - the default is ubuntu)

* Tokenserver load test code
** The Tokenserver load test can be configured - see the following lines:
** Basic Settings: https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** MockMyID: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L19-L36
** Percentages: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L39-L51

* Verifier load test code
** The Verifier load test can be configured - see the following lines:
** Various settings: https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py#L13-L53

* Sync Server load test code
** The Sync Server load test can be configured - see the following lines:
** Setting MockMyID: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L26-L45
** Setting test distributions: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L48-L83

* REFs:
** https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py
** https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py
** https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* TokenServer
** File location: tokenserver/loadtest/loadtest.py
** Inside NoteAssignmentTest, test_realistic is the main load test; the others are for specific behaviors
** The test runs as following:
95% ask for assertions on existing users (on a DB filled by test_single_token_exchange)
4% ask for assertion on a new use
1% ask for a bad assertion
** A bug has been filed to get the following additional coverage for the load test:
*** generation numbers in assertion
*** client state string
** A bug has been filed to get some integration tests written:
*** to cover the edge/error cases not in the load test
*** to be pointed at a remote server

* Sync
** File location: server-syncstorage/loadtest/stress.py
** This is the Sync 2.0 load test that has been back-ported for Sync 1.5.
** The stress.py file is fully configurable for the following:
*** client probability
*** client distribution
*** collections
** A bug has been filed to add support for load testing tabs
*** The tab collection it uses memcache; we need to figure out a way to test it without overloading the server
** There are currently no constants to define how to select percentages per collection type
** Right now, we need to manually configure the collections list in stress.py:
*** collections = ['bookmarks', 'forms', 'passwords', 'history', 'prefs']
*** Basically, you can add more entries of each type, since the load test (per user/again/hit/pass) picks randomly from the list for any given request...

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Tokenserver Custom Metrics
** addFailure

* Verifier Custom Metrics
** addFailure

* Sync Custom Metrics
** addFailure

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for TokenServer (per server)
** /media/ephemeral0/logs/
** /media/ephemeral0/nginx/logs/default.access.log
** /media/ephemeral0/nginx/logs/default.error.log
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* 2. Important logs for Verifier (per server)
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* 3. Important error logs for Sync (per Sync node)
** /media/ephemeral0/logs/
** /media/ephemeral0/nginx/access.log
** /media/ephemeral0/error.log
** /media/ephemeral0/sync/sync.err
** /media/ephemeral0/sync/sync.log

* Acceptable TokenServer errors:
1% - 2% failures (as the following)
token.log:
"name": "token.assertion.invalid_signature_error"
"name": "token.assertion.verify_failure"
nginx access.log:
401s
NOTE: Values can be tweaked here:
https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L58-L60

The following types of errors are known:
/media/ephemeral0/logs/tokenserver/token.error.log
Exception KeyError: KeyError(49564400,) in <module 'threading'...
/media/ephemeral0/logs/tokenserver/token.log
..."Starting new HTTP connection (9): 127.0.0.1", "hostname": ...
{"error": "StopIteration()", "traceback": "Uncaught exception:\n
File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
..."Connection pool is full, discarding connection: 127.0.0.1", "...
Also, any 499s are probably an artifact of the current (V1) load test.
REF:
https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
https://bugzilla.mozilla.org/show_bug.cgi?id=1040397

OLD: Also, it may be the case that the following errors are "acceptable" if TS Stage is larger than Verifier Stage:
/media/ephemeral0/logs/tokenserver/token.error.log
Verifier-related errors of these types:
"HttpConnectionPool is full, discarding connection: verifier.stage.mozaws.net"
"Resetting dropped connection: verifier.stage.mozaws.net"
"Starting new HTTPS connection (179): verifier.stage.mozaws.net"

* Acceptable Verifier errors:
The verifier_out.log will show errors of the following types:
result: 'failure',\n reason: 'untrusted issuer...'
result: 'failure',\n reason: 'expired'
result: 'failure',\n reason: 'algorithms do not match'
result: 'failure',\n reason: 'audience mismatch: scheme mismatch'
Also, any 499s in the nginx logs are probably an artifact of the current (V1) load test.

* Acceptable Sync node errors:
In the nginx access.log files:
We will see some percentage of 404s. Right now we see the following:
14% 404s (compared to the total count of 200s)
with the config set up as follows:
users = 20
duration = 1800
agents = 5
Ideally, the overall percentage of 404s should drop the longer the load test.
Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
although they may show up in the logs after running the remote integration tests.
Also, any 499s are probably an artifact of the current (V1) load test.

In /var/log/hekad/sync_1_5.stderr.log
You may see some Decoder 'Sync-1_5-SlowQuery-MySqlSlowQueryDecoder' error: Failed parsing
and a lot of BSO INSERTs

In /media/ephemeral0/logs/sync/sync.err
You should see expected skew and QueuePool messages and Deprecation warnings
Also, these are known
Exception SystemExit
Exception KeyError
This is probably https://bugzilla.mozilla.org/show_bug.cgi?id=1040397

== Monitoring TS and Sync Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Tokenserver:
** Repo: https://github.com/mozilla-services/tokenserver/issues
** Bugzilla: http://mzl.la/1s4qZKn

* BrowserID-Verifier:
** Repo: https://github.com/mozilla/browserid-verifier/issues
** Bugzilla: no specific cateogory

* Sync:
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Bugzilla: http://mzl.la/VUrYQ5

* OPs and Infrastructure
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-aws/issues
** https://github.com/mozilla-services/loads-web/issues

== References ==
* Other URLs
** http://restmail.net
** https://mockmyid.com
* Repositories
** https://github.com/mozilla-services/tokenserver
** https://github.com/mozilla-services/server-syncstorage
** https://github.com/mozilla/browserid-verifier
** https://github.com/mozilla/browserid-local-verify
** https://github.com/callahad/mockmyid
** https://github.com/mozilla/restmail.net
* Documentation
** https://docs.services.mozilla.com
** https://docs.services.mozilla.com/token/
** https://wiki.mozilla.org/Services/Sagrada/TokenServer
** https://docs.services.mozilla.com/sync/
* The QA Test Environments:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
* Deploying the FxA Load Test environment for broker/agents usage:
** https://github.com/mozilla/fxa-deployment
* Sync 1.5 protocol, documentation, etc.
** https://github.com/mozilla-services/docs
** https://docs.services.mozilla.com/#how-to
** https://docs.services.mozilla.com/howtos/run-fxa.html
** https://docs.services.mozilla.com/token/apis.html
** https://docs.services.mozilla.com/storage/apis-1.5.html
** https://docs.services.mozilla.com/howtos/run-sync-1.5.html
** https://docs.services.mozilla.com/howtos/run-sync-1.5.html
** https://github.com/mozilla-services/syncserver
* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/TokenServerAndSyncLoadTesting

2014-10-16T18:22:04Z

Jbonacci: /* Installing BrowserID-Verifier and the Loads tool on Localhost or AWS */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier sections below...
** A Tokenserver+Verifier stack: See the TokenServer sections below...

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.

* TokenServer+Verifier Stage environment:
From the browser: https://token.stage.mozaws.net
curl https://token.stage.mozaws.net
curl -I https://token.stage.mozaws.net

Use the simple "make test" command from an install of tokenserver on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://token.stage.mozaws.net

Alternate method:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Stage:
bin/sync-cli.js -e EMAIL -p PASSWORD --env stage -t COLLECTION
where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms

* Verifier Stage environment:
In the browser: https://verifier.stage.mozaws.net/
curl https://verifier.stage.mozaws.net
curl -I https://verifier.stage.mozaws.net

Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.stage.mozaws.net

* Sync Server Stage environment:
Install server-syncstorage to the local host or AWS instance (see below)
$ cd server-syncstorage
Quick test against the TokenServer
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server <Stage TokenServer>
Current example:
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server
https://token.stage.mozaws.net/1.0/sync/1.5
Quick tests against the Sync nodes
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py <Stage Sync Node>#<Node Secret>
Current examples:
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
Get the Node Secret information from OPs

* Using TPS
** The TPS FxA/Sync automated tests can be used as well, but the following file will have to be edited to add Stage environment configuration parameters: https://github.com/mozilla/gecko-dev/blob/master/testing/tps/tps/testrunner.py
** See the following wiki page for more information: https://wiki.mozilla.org/User_Services/Sync/Run_TPS
** See also: https://bugzilla.mozilla.org/show_bug.cgi?id=1006675

== Quick Verification Of Production Deployments ==
* This is a quick sanity test of the environment after a new deployment.

* Tokenserver+Verifier Production Environment
In the browser: https://token.services.mozilla.com
curl https://token.services.mozilla.com
curl -I https://token.services.mozilla.com
Then:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Production:
bin/sync-cli.js -e PROD-EMAIL -p PASSWORD -t COLLECTION
where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms

* Verifier Production Environment
In the browser: https://verifier.accounts.firefox.com
curl https://verifier.accounts.firefox.com
curl -I https://verifier.accounts.firefox.com
Then:
Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.accounts.firefox.com

* Sync Server Stage environment
Sign in with a known FxA account and sync data with a current Production account (sync node).
Create a new FxA account and set up sync.

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment

== Installing BrowserID-Verifier and the Loads tool on Localhost or AWS ==
* Installation:
$ git clone git://github.com/mozilla/browserid-verifier
$ cd browserid-verifier
Note: You may want to install a specific branch for testing vs defaulting to Master
$ npm install
$ npm test
$ cd loadtest
$ make build
Note: This should hit Stage by default: SERVER_URL=https://verifier.stage.mozaws.net

* Note: This will install a local copy of the Loads tool for use with the verifier.

== Running the load test against the Verifier in Stage ==
* Stage environment:
$ make test
or
$ make test SERVER_URL=https://verifier.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://verifier.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Verifier hits the Stage mockmyid server

* Production environment:
$ make test SERVER_URL=https://verifier.accounts.firefox.com
$ make bench SERVER_URL=https://verifier.accounts.firefox.com

== Using the Loads V1 Services Cluster for the Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://verifier.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=TBD

* Production environment:
$ make megabench SERVER_URL=https://verifier.accounts.firefox.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing TokenServer+Verifier and the Loads tool on Localhost or AWS ==
* Installation:
$ git clone https://github.com/mozilla-services/tokenserver
$ cd tokenserver
$ make build
$ make test
Note: This is for local testing only
$ cd loadtest
$ make build
Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com

* Note: This will install a local copy of the Loads tool for use with TokenServer+Verifier.

== Running the load test against TokenServer+Verifier in Stage ==
* Stage environment:
$ make test SERVER_URL=https://token.stage.mozaws.net
$ make bench SERVER_URL=https://token.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: This also hits the Stage Verifier, which in turns hits the Stage mockmyid server

* And while we are at it...
* Dev environment:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org

* Production environment:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com

== Using the Loads V1 Services Cluster for TokenServer+Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://token.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=https://token.dev.lcip.org

* Production environment:
$ make megabench SERVER_URL=https://token.services.mozilla.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest

== Installing Sync 1.5 and the Loads tool on Localhost or AWS ==
Installation:
$ git clone https://github.com/mozilla-services/server-syncstorage/
$ cd server-syncstorage
$ make build
$ make test
$ cd loadtest
$ make build

* Note: This will install a local copy of the Loads tool for use with Sync 1.5.

== Running the load test against Sync 1.5 in Stage ==
* Loads against specific Sync nodes in Stage
$ make test SERVER_URL=https://your.storagenode.here#SECRET
$ make bench SERVER_URL=https://your.storagenode.here#SECRET
Sync Stage nodes:
https://sync-1-us-east-1.stage.mozaws.net
https://sync-2-us-east-1.stage.mozaws.net
https://sync-3-us-east-1.stage.mozaws.net
...etc...

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.

== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://your.storagenode.here#SECRET

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest

== Running a combined load test against TokenServer+Verifier and Sync 1.5 in Stage ==
* A combined loads test against TokenServer and Sync 1.5 in Stage
* This is done via the server-syncstorage directory that was cloned and built above
$ cd server-syncstorage
$ cd loadtest
$ make test SERVER_URL=https://your.tokenserver.here
$ make bench SERVER_URL=https://your.tokenserver.here

Examples for Stage:
$ make test SERVER_URL=https://token.stage.mozaws.net
$ make bench SERVER_URL=https://token.stage.mozaws.net
See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer.2BVerifier_Stage_Environment

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Tokenserver hits the Stage Verifier, which, in turn, hits the mockmyid server.

* And while we are at it...
Dev environment:
Examples:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org

Prod environment:
Examples:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com

See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA.2C_TokenServer.2C_and_Sync_Production_Environments
and https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer_and_Sync_1.5_Dev_Environments

== Using the Loads V1 Services Cluster for a combined load test in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://token.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=https://token.dev.lcip.org

* Prod environment:
$ make megabench SERVER_URL=https://token.services.mozilla.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of hits
*** Number of concurrent users

** For make bench (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of concurrent users
*** Duration of test

** For make megabench (using the LoadsCluster with BrowserID-Verifier, TokenSerer, Sync, Combined):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)
*** SSH (the user account needed to SSH into the loads cluster - the default is ubuntu)

* Tokenserver load test code
** The Tokenserver load test can be configured - see the following lines:
** Basic Settings: https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** MockMyID: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L19-L36
** Percentages: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L39-L51

* Verifier load test code
** The Verifier load test can be configured - see the following lines:
** Various settings: https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py#L13-L53

* Sync Server load test code
** The Sync Server load test can be configured - see the following lines:
** Setting MockMyID: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L26-L45
** Setting test distributions: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L48-L83

* REFs:
** https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py
** https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py
** https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* TokenServer
** File location: tokenserver/loadtest/loadtest.py
** Inside NoteAssignmentTest, test_realistic is the main load test; the others are for specific behaviors
** The test runs as following:
95% ask for assertions on existing users (on a DB filled by test_single_token_exchange)
4% ask for assertion on a new use
1% ask for a bad assertion
** A bug has been filed to get the following additional coverage for the load test:
*** generation numbers in assertion
*** client state string
** A bug has been filed to get some integration tests written:
*** to cover the edge/error cases not in the load test
*** to be pointed at a remote server

* Sync
** File location: server-syncstorage/loadtest/stress.py
** This is the Sync 2.0 load test that has been back-ported for Sync 1.5.
** The stress.py file is fully configurable for the following:
*** client probability
*** client distribution
*** collections
** A bug has been filed to add support for load testing tabs
*** The tab collection it uses memcache; we need to figure out a way to test it without overloading the server
** There are currently no constants to define how to select percentages per collection type
** Right now, we need to manually configure the collections list in stress.py:
*** collections = ['bookmarks', 'forms', 'passwords', 'history', 'prefs']
*** Basically, you can add more entries of each type, since the load test (per user/again/hit/pass) picks randomly from the list for any given request...

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Tokenserver Custom Metrics
** addFailure

* Verifier Custom Metrics
** addFailure

* Sync Custom Metrics
** addFailure

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for TokenServer (per server)
** /media/ephemeral0/logs/
** /media/ephemeral0/nginx/logs/default.access.log
** /media/ephemeral0/nginx/logs/default.error.log
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* 2. Important logs for Verifier (per server)
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* 3. Important error logs for Sync (per Sync node)
** /media/ephemeral0/logs/
** /media/ephemeral0/nginx/access.log
** /media/ephemeral0/error.log
** /media/ephemeral0/sync/sync.err
** /media/ephemeral0/sync/sync.log

* Acceptable TokenServer errors:
1% - 2% failures (as the following)
token.log:
"name": "token.assertion.invalid_signature_error"
"name": "token.assertion.verify_failure"
nginx access.log:
401s
NOTE: Values can be tweaked here:
https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L58-L60

The following types of errors are known:
/media/ephemeral0/logs/tokenserver/token.error.log
Exception KeyError: KeyError(49564400,) in <module 'threading'...
/media/ephemeral0/logs/tokenserver/token.log
..."Starting new HTTP connection (9): 127.0.0.1", "hostname": ...
{"error": "StopIteration()", "traceback": "Uncaught exception:\n
File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
..."Connection pool is full, discarding connection: 127.0.0.1", "...
Also, any 499s are probably an artifact of the current (V1) load test.
REF:
https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
https://bugzilla.mozilla.org/show_bug.cgi?id=1040397

OLD: Also, it may be the case that the following errors are "acceptable" if TS Stage is larger than Verifier Stage:
/media/ephemeral0/logs/tokenserver/token.error.log
Verifier-related errors of these types:
"HttpConnectionPool is full, discarding connection: verifier.stage.mozaws.net"
"Resetting dropped connection: verifier.stage.mozaws.net"
"Starting new HTTPS connection (179): verifier.stage.mozaws.net"

* Acceptable Verifier errors:
The verifier_out.log will show errors of the following types:
result: 'failure',\n reason: 'untrusted issuer...'
result: 'failure',\n reason: 'expired'
result: 'failure',\n reason: 'algorithms do not match'
result: 'failure',\n reason: 'audience mismatch: scheme mismatch'
Also, any 499s in the nginx logs are probably an artifact of the current (V1) load test.

* Acceptable Sync node errors:
In the nginx access.log files:
We will see some percentage of 404s. Right now we see the following:
14% 404s (compared to the total count of 200s)
with the config set up as follows:
users = 20
duration = 1800
agents = 5
Ideally, the overall percentage of 404s should drop the longer the load test.
Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
although they may show up in the logs after running the remote integration tests.
Also, any 499s are probably an artifact of the current (V1) load test.

In /var/log/hekad/sync_1_5.stderr.log
You may see some Decoder 'Sync-1_5-SlowQuery-MySqlSlowQueryDecoder' error: Failed parsing
and a lot of BSO INSERTs

In /media/ephemeral0/logs/sync/sync.err
You should see expected skew and QueuePool messages and Deprecation warnings
Also, these are known
Exception SystemExit
Exception KeyError
This is probably https://bugzilla.mozilla.org/show_bug.cgi?id=1040397

== Monitoring TS and Sync Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Tokenserver:
** Repo: https://github.com/mozilla-services/tokenserver/issues
** Bugzilla: http://mzl.la/1s4qZKn

* BrowserID-Verifier:
** Repo: https://github.com/mozilla/browserid-verifier/issues
** Bugzilla: no specific cateogory

* Sync:
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Bugzilla: http://mzl.la/VUrYQ5

* OPs and Infrastructure
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-aws/issues
** https://github.com/mozilla-services/loads-web/issues

== References ==
* Other URLs
** http://restmail.net
** https://mockmyid.com
* Repositories
** https://github.com/mozilla-services/tokenserver
** https://github.com/mozilla-services/server-syncstorage
** https://github.com/mozilla/browserid-verifier
** https://github.com/mozilla/browserid-local-verify
** https://github.com/callahad/mockmyid
** https://github.com/mozilla/restmail.net
* Documentation
** https://docs.services.mozilla.com
** https://docs.services.mozilla.com/token/
** https://wiki.mozilla.org/Services/Sagrada/TokenServer
** https://docs.services.mozilla.com/sync/
* The QA Test Environments:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
* Deploying the FxA Load Test environment for broker/agents usage:
** https://github.com/mozilla/fxa-deployment
* Sync 1.5 protocol, documentation, etc.
** https://github.com/mozilla-services/docs
** https://docs.services.mozilla.com/#how-to
** https://docs.services.mozilla.com/howtos/run-fxa.html
** https://docs.services.mozilla.com/token/apis.html
** https://docs.services.mozilla.com/storage/apis-1.5.html
** https://docs.services.mozilla.com/howtos/run-sync-1.5.html
** https://docs.services.mozilla.com/howtos/run-sync-1.5.html
** https://github.com/mozilla-services/syncserver
* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/FxALoadTesting

2014-10-16T18:21:21Z

Jbonacci: /* Installing FxA-Auth-Server and the Loads tool on Localhost or AWS */

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
Install FxA-Auth-Server to a local host or an AWS instance (see below)
$ cd fxa-auth-server
Run the integration tests against the remote Stage server (load balancer)
$ PUBLIC_URL=<FxA Stage> npm run test-remote
Current example:
$ PUBLIC_URL=https://api-accounts.stage.mozaws.net npm run test-remote
* NOTE: Make sure to install and test from the same branch that is deployed to Stage (ie do not use Master for running the tests against Stage or Production).

* Using TPS
** The TPS FxA/Sync automated tests can be used as well, but the following file will have to be edited to add Stage environment configuration parameters: https://github.com/mozilla/gecko-dev/blob/master/testing/tps/tps/testrunner.py
** See the following wiki page for more information: https://wiki.mozilla.org/User_Services/Sync/Run_TPS
** See also: https://bugzilla.mozilla.org/show_bug.cgi?id=1006675

== Quick Verification Of Production Deployments ==
* This is a quick sanity test of the environment after each new deployment. There are other verifications that can be run as well
Install FxA-Auth-Server to a local host or an AWS instance (see below)
$ cd fxa-auth-server
Run the integration tests against the remote Stage server (load balancer)
$ PUBLIC_URL=<FxA Prod> npm run test-remote
Current example:
$ PUBLIC_URL=https://api.accounts.firefox.com npm run test-remote
* NOTE: Make sure to install and test from the same branch that is deployed to Production.

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Installing FxA-Auth-Server and the Loads tool on Localhost or AWS ==
Installation:
$ git clone https://github.com/mozilla/fxa-auth-server.git
$ cd ./fxa-auth-server
Note: You may want to install a specific branch for testing vs defaulting to Master
$ npm install
$ npm test
$ cd ./test/load
$ make build

* Note: 'npm install' may need to be run now as root.
* Note: This will install a local copy of the Loads tool for use with FxA-Auth-Server.

== Running the Loads tool against FxA Stage ==
* The basic load test can be run as follows
$ make test SERVER_URL=https://api-accounts.stage.mozaws.net

* The full, default load test can be run as follows
$ make bench SERVER_URL=https://api-accounts.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* Configuring the bench load test - config folder:
** The test.ini file (for make test) can be configured for the following:
*** Number of hits
*** Number of concurrent users
** The bench.ini file (for make bench) can be configured for the following:
*** Number of concurrent users
*** Duration of test

* For both tests, start with the defaults, then tweak the duration. Users and Agents are optional tweaks/changes. Also, we can configure the bench load test to run in detached mode with an appropriate loads detach and observer settings.

* REF: https://github.com/mozilla/fxa-auth-server/tree/master/loadtest/config

== Running the Loads tool against FxA Development or Production ==
* This can be done if we are comparing Stage vs. some other environment and have access to the AWS logs in Dev or Production:

* Dev:
$ make test SERVER_URL=https://accounts.dev.lcip.org
$ make bench SERVER_URL=https://accounts.dev.lcip.org

* Prod:
$ make test SERVER_URL=https://api.accounts.firefox.com
$ make bench SERVER_URL=https://api.accounts.firefox.com

* The same optional configuration changes apply here.

== Using the Loads V1 Services Cluster ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).

* Testing against the Stage environment:
$ make megabench SERVER_URL=https://api-accounts.stage.mozaws.net

* Testing against the Dev environment:
$ make megabench SERVER_URL=https://api-accounts.dev.lcip.org

* Testing against the Prod enviornment:
$ make megabench SERVER_URL=https://api.accounts.firefox.com

* Configuring the megabench load test - config folder:
** The megabench.ini file (for make megabench) can be configured for the following:
*** Number of concurrent users
*** Duration of test
*** Include file (leave as defined for now)
*** Python dependencies (leave as defined for now)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* REF: https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
* REF: https://github.com/mozilla/fxa-auth-server/tree/master/loadtest/config

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test:
*** Number of hits
*** Number of concurrent users

** For make bench:
*** Number of concurrent users
*** Duration of test

** For make megabench:
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* Load Test code: loadtests.py
** The load test can be configured in the code - see the following lines:
** https://github.com/mozilla/fxa-auth-server/blob/master/test/load/loadtests.py#L17-L39

* General REFs:
** https://github.com/mozilla/fxa-auth-server/blob/master/test/load/loadtests.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* Location fxa-auth-server/loadtest/loadtests.py
* The following items are covered in the load test
** test_auth_server is the main entry point in the loadtests.py file
*** account creation
*** session creation
** account deletion
** session deletion
* Integration tests
** These are designed to cover the edge/error cases that are not applicable to the load test
** The tests can be run against a remote server

== Analyzing the Results ==
* TBD

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for FxA-Auth-Server (per server)
** /media/ephemeral0/fxa-auth-server/auth_err.log.*
** /media/ephemeral0/fxa-auth-server/auth_out.log
** /media/ephemeral0/heka/hekad_err.log
** /media/ephemeral0/heka/hekad_out.log
** /media/ephemeral0/nginx/logs/access.log
** /media/ephemeral0/nginx/logs/error.log

* Acceptable FxA-Auth-Server errors
503s: especially of this type - /v1/certificate/sign - are usually a sign that we are overloading the hosts

400s: we should never see these in the logs, especially if the "errno" value is 105.
Check the fxa-auth-server/auth_err.log
400s: "errno" values of 101, 102 are ok. These can be expected during a load test.

ELB issues: we may see 503s and corresponding "err":"cannot enqueue work: maximum backlog exceeded (30)"
messages if one or more of the hosts behind the ELB is receiving most of the load traffic.
REF: https://github.com/mozilla/fxa-auth-server/issues/647

== Monitoring FxA Stage ==
* Loads dashboard:
** http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com/
** or http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* FxA
** https://github.com/mozilla/fxa-auth-server/issues
** https://github.com/mozilla/fxa-content-server/issues
** and several others

* Bugzilla
** No specific category

* Infrastructure
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and clusters
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues

== Capacity Planning Stage and Production ==
* QA is tasked with providing some capacity requirements and constraints based on repeated load testing of the FxA-Auth-Server Stage environment.
* The goal is to be able to work with OPs to develop a realistic plan for deploying and maintaining the production environment at a level expected for projected user traffic, etc.

* Brainstorming the QA role:
** QA needs to get some realistic numbers from the Product team. This could be as simple as traffic flow (number of users per day or per segments of the day - peaks and valleys) or more detailed:
*** Traffic flow - QPS/RPS
*** Average number of users per time segment
*** Average and peak latency
*** Error percentages and thresholds
*** etc
** QA gets help from OPs to learn how to measure those required numbers/values using StackDriver or other tools (or to get data from OPsView). If those numbers can not be measured then we either need to
*** get a different set of data points from the Product team
*** enhance the current tools to track and measure the required data
** QA does repeated, scheduled, well-defined load tests in Stage while actively monitoring the results, logs, data, etc.
** QA finds a stable configuration that - when scaled - would
*** match the needs of Product when we release
*** match the realisitc capacity planning that OPs normally does

* Dependencies
** Realistic traffic/user numbers from the FxA Product team
** Timely training on monitoring tools from the OPs team
** Regular and realistic scaling/testing of deployments to Stage by QA given our current pre-release and post-release schedules

== References ==
* Repository: https://github.com/mozilla/fxa-auth-server
* The QA Test Environments:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
* Deploying the FxA Load Test environment for broker/agents usage:
** https://github.com/mozilla/fxa-deployment
* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/LoopTestEnvironments

2014-10-14T21:03:58Z

Jbonacci: /* Sentry Server Stage Details */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Loop web client.

Browser: https://call.mozilla.com/config.js
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

Browser: Browser: https://call.mozilla.com/VERSION.txt

and now the TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD

** loop-redis-stage - need the details

** Loop-Client: https://call.stage.mozaws.net

** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net

** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad
** /media/ephemeral0/nginx/logs

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net

Check the version: https://call.stage.mozaws.net/VERSION.txt

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {serverUrl: 'https://https://loop.stage.mozaws.net'};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** http://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** None at this time. See https://bugzilla.mozilla.org/show_bug.cgi?id=1063134

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopTestEnvironments

2014-10-14T21:03:44Z

Jbonacci: /* Loop Server Stage Environment */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Loop web client.

Browser: https://call.mozilla.com/config.js
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

Browser: Browser: https://call.mozilla.com/VERSION.txt

and now the TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD

** loop-redis-stage - need the details

** Loop-Client: https://call.stage.mozaws.net

** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net

** Sentry: http://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad
** /media/ephemeral0/nginx/logs

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net

Check the version: https://call.stage.mozaws.net/VERSION.txt

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {serverUrl: 'https://https://loop.stage.mozaws.net'};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** https://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** None at this time. See https://bugzilla.mozilla.org/show_bug.cgi?id=1063134

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopTestEnvironments

2014-09-25T21:52:50Z

Jbonacci: /* MSISDN Gateway Server Stage Details */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Loop web client.

Browser: https://call.mozilla.com/config.js
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

Browser: Browser: https://call.mozilla.com/VERSION.txt

and now the TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD

** loop-redis-stage - need the details

** Loop-Client: https://call.stage.mozaws.net

** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net

** Sentry: https://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad
** /media/ephemeral0/nginx/logs

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net

Check the version: https://call.stage.mozaws.net/VERSION.txt

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {serverUrl: 'https://https://loop.stage.mozaws.net'};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* Version string
** /data/msisdn-gateway/VERSION.txt

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** https://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** None at this time. See https://bugzilla.mozilla.org/show_bug.cgi?id=1063134

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopServerLoadTesting

2014-09-24T19:37:06Z

Jbonacci: /* Quick Verification of Production Deployments */

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
* Loop Server
For now, just run a quick loadtest 'make test'
cd loop-server
cd loadtests
make test SERVER_URL=https://loop.stage.mozaws.net

* Loop Client
Check https://call.stage.mozaws.net/config.js
Should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.stage.mozaws.net'};
and
WIP from the client team: end to end tests
Also:
https://call.stage.mozaws.net
curl https://call.stage.mozaws.net
curl -I https://call.stage.mozaws.net

and now the TOS: https://call.stage.mozaws.net/legal/terms/

* MSISDN Gateway
In the browser: https://msisdn.stage.mozaws.net
or do the following from a command line:
curl https://msisdn.stage.mozaws.net
curl -I https://msisdn.stage.mozaws.net
and
For now, just run a quick loadtest 'make test'
cd msisdn-gateway
cd loadtests
make test SERVER_URL=https://msisdn.stage.mozaws.net
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Quick Verification of Production Deployments ==
* This is a quick sanity test of the environment for after each Production deployment.
* Loop Server
In the browser: https://loop.services.mozilla.com
or do the following from a command line:
curl https://loop.services.mozilla.com
curl -I https://loop.services.mozilla.com

Then run a few 'make test' commands from the loadtests folder:
make test SERVER_URL=https://loop.services.mozilla.com
Note: this does hit a live third-party server

Then perform actual loop testing via desktop (Aurora/Nightly so far) and FxOS (2.1)
Verify that requests and strings point to Production environments

* Loop Client
In the browser: https://call.mozilla.com
should return "Welcome to the Loop web client."

In the browser: https://call.mozilla.com/config.js
should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

In the browser: https://call.mozilla.com/VERSION.txt
should return the version and build string info

and now the TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com
curl -I https://call.mozilla.com

Quick end-to-end tests:
Desktop: browser to browser
Desktop to FxOS
FxOS to Desktop
Two FxOS devices

* MSISDN Gateway
In the browser: https://msisdn.services.mozilla.com
or do the following from a command line:
curl https://msisdn.services.mozilla.com
curl -I https://msisdn.services.mozilla.com
Or
Run a single 'make test' command from the loadtests folder:
make test SERVER_URL=https://msisdn.services.mozilla.com
Note: this does hit a live third-party, so limit the check to a single run.
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Installing Loop-Server and the Loads tool on Localhost or AWS ==
* Installation:
git clone https://github.com/mozilla-services/loop-server.git
cd loop-server
npm install
ulimit -S -n 2048
npm test *
cd loadtests
make build
make test

Coverage report can be found here:
/loop-server/coverage/lcov-report/index.html

* This step requires the redis server to be installed and running:
Mac:
brew install redis
redis-server /usr/local/etc/redis.conf

Ubuntu Linux:
sudo apt-get install redis-server
sudo /usr/bin/redis-server /etc/redis/redis.conf
sudo tail -f /var/log/redis/redis-server.log

RHEL Linux:
Install redis from here: http://download.redis.io/releases
then
/usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
or similar

* Note: This will install a local copy of the Loads tool for use with the Loop-Server.

== Running the load test against the Loop-Server in Stage ==
* Stage environment:
$ cd loop-server/loadtests
$ make test
or
$ make test SERVER_URL=https://loop.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://loop.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the Loop-Server in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://loop.stage.mozaws.net

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing MSISDN-Gateway and the Loads tool on Localhost or AWS ==
* Installation:
** Install gmp, gmp-dev or gmp-devel
** Install ruby (very latest), ruby-dev or ruby-devel
** Install gem (required for fake_dynamo)
** Verify that gem is in your path
** Install redis-server to run the unit tests

* To install gmp
sudo yum -y install gmp, gmp-devel
or for Ubuntu
$ wget https://ftp.gnu.org/gnu/gmp/gmp-6.0.0a.tar.bz2
$ tar xvjf gmp-6.0.0a.tar.bz2
$ cd gmp-6.0.0
$ ./configure --prefix=/usr
$ make
$ make check
$ sudo make install

* To install ruby:
sudo yum -y install ruby, ruby-devel
or
sudo apt-get install ruby, ruby-dev

If this does not get you 1.9.3 or newer, then install manually:
Example:
$ wget http://cache.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p547.tar.gz
$ ./configure --prefix=/usr
$ make
$ sudo make install
(because for rhel, the default ruby version is 1.8.x.)
REF:
Main: https://www.ruby-lang.org/en/downloads/
Dev Tools: http://rubyinstaller.org/downloads/

* To install gem:
Grab rubygems from here: http://rubygems.org/pages/download
cd to rubygems directory
$ sudo ruby setup.rb

* To install fake_dynamo:
You should not have to install fake_dynamo since it is now part of the repo installer.
But if you do:
$ sudo gem install fake_dynamo
REF: https://github.com/ananthakumaran/fake_dynamo

* Install the msisdn-gateway repo:
$ git clone https://github.com/mozilla-services/msisdn-gateway.git
$ cd msisdn-gateway
$ sudo make install
(There is a bug open about the requirement to install with 'sudo')

* Note: This will install a local copy of the Loads tool for use with MSISDN-Gateway.

* Unit testing
Get redis-server installed
Start the server in a separate terminal or in the background with logging active
$ make test
The coverage report is here: msisdn-gateway/coverage/lcov-report/index.html

* Bugs:
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036158
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036174
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048451
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048456

== Running the load test against MSISDN-Gateway in Stage ==
* Building the load tests
$ cd loadtests
$ make build

* To load test the Stage environment:
$ make test SERVER_URL=https://msisdn.stage.mozaws.net
$ make bench SERVER_URL=https://msisdn.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the MSISDN-Gateway ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://msisdn.stage.mozaws.net

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (Loop-Server and MSISDN-Gateway):
*** Number of hits
*** Number of concurrent users

** For make bench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test

** For make megabench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* Loop-Server load test code
** The Loop-Server load test can not currently be configured in the code

* MSISDN-Gateway load test code
** The MSISDN-Gateway load test can be configured in the code - see the following lines:
** Test server URL: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
** Error percentages: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22

* General REFs:
** https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* Loop-Server
** TBD

* MSISDN-Gateway
** TBD

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Loop-Server Custom Metrics
** Opened web sockets
** Total web sockets
** Bytes/websockets
** addFailure (from the loads tool/client)

* MSISDN-Gateway Custom Metrics
** mt-flow
** ask-for-certificate
** try-wrong-code
** try-right-code
** momt-flow
** omxen-message-collision
** register
** unregister
** addFailure (from the loads tool/client)

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for Loop-Server (per server)
** /var/log/circus.log
** /var/log/loop_err.log
** /var/log/loop_out.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /var/log/nginx/access.log
** /var/log/nginx/error.log

* 2. Important logs for MSISDN-Gateway (per server)
** TBD

* Acceptable/Unacceptable Loop-Server errors:
hekad loop.stderr.log
The following are acceptable:
Decoder 'LoopServer-LoopServerDecoder' error: Failed parsing
Plugin 'AggregatorOutput' error: writing to heka.shared....

nginx logs:
The 200s (good stuff) and 101s (websockets) are acceptable.
The 499s are an artifact of the current load testing tool (V1).
You should only see them at the end of the load test.
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
Any percentage of 405s, 502s, or 503s is not acceptable.

Application logs:
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
See the loop_server.out.log file.

/var/log/loop_err.log
The following are acceptable: connect: res.on("header"): use on-headers module directly

In the Loads Cluster dashboard, watch for the following errors/failures:
string indices must be integers
OR
No JSON object could be decoded
OR
'hawk-session-token'

* Acceptable/Unacceptable MSISDN-Gateway errors:
The updated load test does generate a certain percentage of errors:
https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
So, expect to see a predefined percentage of 204s and 400s, along with the usual 200s in the nginx access logs.
The msisdn-gateway app logs should be clean with just msisdn and test data.

== Monitoring Loop Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Loop Server: Check the following repos:
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla-services/docs/issues
** https://github.com/mozilla/loop-client
** https://github.com/mozilla-services/msisdn-gateway/issues

* Bugzilla
** and in Bugzilla for non-Verified bugs: http://mzl.la/1qPxizw

* OPs and Infra
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues

== References ==
* Repositories and Documentation
** Check here: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#References

* The QA Test Environments
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments

* Test Plans and other QA
** https://docs.google.com/a/mozilla.com/document/d/1xYZijHnJ1JeJhONm2AYvxSnD2tWztC16WGFyD5U_d6g/
** https://wiki.mozilla.org/Services/Loop/Try_Loop
** https://wiki.mozilla.org/Loop/QA
** https://wiki.mozilla.org/Loop/QA#Server
** https://intranet.mozilla.org/QA/Loop/MVP

* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/LoopTestEnvironments

2014-09-24T19:36:26Z

Jbonacci: /* Loop Server Production Environments */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Loop web client.

Browser: https://call.mozilla.com/config.js
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

Browser: Browser: https://call.mozilla.com/VERSION.txt

and now the TOS: https://call.mozilla.com/legal/terms/

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD

** loop-redis-stage - need the details

** Loop-Client: https://call.stage.mozaws.net

** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net

** Sentry: https://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad
** /media/ephemeral0/nginx/logs

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net

Check the version: https://call.stage.mozaws.net/VERSION.txt

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {serverUrl: 'https://https://loop.stage.mozaws.net'};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** https://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** None at this time. See https://bugzilla.mozilla.org/show_bug.cgi?id=1063134

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopServerLoadTesting

2014-09-24T19:34:16Z

Jbonacci: /* Quick Verification Of Stage Deployments */

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
* Loop Server
For now, just run a quick loadtest 'make test'
cd loop-server
cd loadtests
make test SERVER_URL=https://loop.stage.mozaws.net

* Loop Client
Check https://call.stage.mozaws.net/config.js
Should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.stage.mozaws.net'};
and
WIP from the client team: end to end tests
Also:
https://call.stage.mozaws.net
curl https://call.stage.mozaws.net
curl -I https://call.stage.mozaws.net

and now the TOS: https://call.stage.mozaws.net/legal/terms/

* MSISDN Gateway
In the browser: https://msisdn.stage.mozaws.net
or do the following from a command line:
curl https://msisdn.stage.mozaws.net
curl -I https://msisdn.stage.mozaws.net
and
For now, just run a quick loadtest 'make test'
cd msisdn-gateway
cd loadtests
make test SERVER_URL=https://msisdn.stage.mozaws.net
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Quick Verification of Production Deployments ==
* This is a quick sanity test of the environment for after each Production deployment.
* Loop Server
In the browser: https://loop.services.mozilla.com
or do the following from a command line:
curl https://loop.services.mozilla.com
curl -I https://loop.services.mozilla.com

Then run a few 'make test' commands from the loadtests folder:
make test SERVER_URL=https://loop.services.mozilla.com
Note: this does hit a live third-party server

Then perform actual loop testing via desktop (Aurora/Nightly so far) and FxOS (2.1)
Verify that requests and strings point to Production environments

* Loop Client
In the browser: https://call.mozilla.com
should return "Welcome to the Loop web client."

In the browser: https://call.mozilla.com/config.js
should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

In the browser: https://call.mozilla.com/VERSION.txt
should return the version and build string info

curl https://call.mozilla.com
curl -I https://call.mozilla.com

Quick end-to-end tests:
Desktop: browser to browser
Desktop to FxOS
FxOS to Desktop
Two FxOS devices

* MSISDN Gateway
In the browser: https://msisdn.services.mozilla.com
or do the following from a command line:
curl https://msisdn.services.mozilla.com
curl -I https://msisdn.services.mozilla.com
Or
Run a single 'make test' command from the loadtests folder:
make test SERVER_URL=https://msisdn.services.mozilla.com
Note: this does hit a live third-party, so limit the check to a single run.
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Installing Loop-Server and the Loads tool on Localhost or AWS ==
* Installation:
git clone https://github.com/mozilla-services/loop-server.git
cd loop-server
npm install
ulimit -S -n 2048
npm test *
cd loadtests
make build
make test

Coverage report can be found here:
/loop-server/coverage/lcov-report/index.html

* This step requires the redis server to be installed and running:
Mac:
brew install redis
redis-server /usr/local/etc/redis.conf

Ubuntu Linux:
sudo apt-get install redis-server
sudo /usr/bin/redis-server /etc/redis/redis.conf
sudo tail -f /var/log/redis/redis-server.log

RHEL Linux:
Install redis from here: http://download.redis.io/releases
then
/usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
or similar

* Note: This will install a local copy of the Loads tool for use with the Loop-Server.

== Running the load test against the Loop-Server in Stage ==
* Stage environment:
$ cd loop-server/loadtests
$ make test
or
$ make test SERVER_URL=https://loop.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://loop.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the Loop-Server in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://loop.stage.mozaws.net

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing MSISDN-Gateway and the Loads tool on Localhost or AWS ==
* Installation:
** Install gmp, gmp-dev or gmp-devel
** Install ruby (very latest), ruby-dev or ruby-devel
** Install gem (required for fake_dynamo)
** Verify that gem is in your path
** Install redis-server to run the unit tests

* To install gmp
sudo yum -y install gmp, gmp-devel
or for Ubuntu
$ wget https://ftp.gnu.org/gnu/gmp/gmp-6.0.0a.tar.bz2
$ tar xvjf gmp-6.0.0a.tar.bz2
$ cd gmp-6.0.0
$ ./configure --prefix=/usr
$ make
$ make check
$ sudo make install

* To install ruby:
sudo yum -y install ruby, ruby-devel
or
sudo apt-get install ruby, ruby-dev

If this does not get you 1.9.3 or newer, then install manually:
Example:
$ wget http://cache.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p547.tar.gz
$ ./configure --prefix=/usr
$ make
$ sudo make install
(because for rhel, the default ruby version is 1.8.x.)
REF:
Main: https://www.ruby-lang.org/en/downloads/
Dev Tools: http://rubyinstaller.org/downloads/

* To install gem:
Grab rubygems from here: http://rubygems.org/pages/download
cd to rubygems directory
$ sudo ruby setup.rb

* To install fake_dynamo:
You should not have to install fake_dynamo since it is now part of the repo installer.
But if you do:
$ sudo gem install fake_dynamo
REF: https://github.com/ananthakumaran/fake_dynamo

* Install the msisdn-gateway repo:
$ git clone https://github.com/mozilla-services/msisdn-gateway.git
$ cd msisdn-gateway
$ sudo make install
(There is a bug open about the requirement to install with 'sudo')

* Note: This will install a local copy of the Loads tool for use with MSISDN-Gateway.

* Unit testing
Get redis-server installed
Start the server in a separate terminal or in the background with logging active
$ make test
The coverage report is here: msisdn-gateway/coverage/lcov-report/index.html

* Bugs:
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036158
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036174
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048451
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048456

== Running the load test against MSISDN-Gateway in Stage ==
* Building the load tests
$ cd loadtests
$ make build

* To load test the Stage environment:
$ make test SERVER_URL=https://msisdn.stage.mozaws.net
$ make bench SERVER_URL=https://msisdn.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the MSISDN-Gateway ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://msisdn.stage.mozaws.net

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (Loop-Server and MSISDN-Gateway):
*** Number of hits
*** Number of concurrent users

** For make bench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test

** For make megabench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* Loop-Server load test code
** The Loop-Server load test can not currently be configured in the code

* MSISDN-Gateway load test code
** The MSISDN-Gateway load test can be configured in the code - see the following lines:
** Test server URL: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
** Error percentages: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22

* General REFs:
** https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* Loop-Server
** TBD

* MSISDN-Gateway
** TBD

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Loop-Server Custom Metrics
** Opened web sockets
** Total web sockets
** Bytes/websockets
** addFailure (from the loads tool/client)

* MSISDN-Gateway Custom Metrics
** mt-flow
** ask-for-certificate
** try-wrong-code
** try-right-code
** momt-flow
** omxen-message-collision
** register
** unregister
** addFailure (from the loads tool/client)

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for Loop-Server (per server)
** /var/log/circus.log
** /var/log/loop_err.log
** /var/log/loop_out.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /var/log/nginx/access.log
** /var/log/nginx/error.log

* 2. Important logs for MSISDN-Gateway (per server)
** TBD

* Acceptable/Unacceptable Loop-Server errors:
hekad loop.stderr.log
The following are acceptable:
Decoder 'LoopServer-LoopServerDecoder' error: Failed parsing
Plugin 'AggregatorOutput' error: writing to heka.shared....

nginx logs:
The 200s (good stuff) and 101s (websockets) are acceptable.
The 499s are an artifact of the current load testing tool (V1).
You should only see them at the end of the load test.
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
Any percentage of 405s, 502s, or 503s is not acceptable.

Application logs:
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
See the loop_server.out.log file.

/var/log/loop_err.log
The following are acceptable: connect: res.on("header"): use on-headers module directly

In the Loads Cluster dashboard, watch for the following errors/failures:
string indices must be integers
OR
No JSON object could be decoded
OR
'hawk-session-token'

* Acceptable/Unacceptable MSISDN-Gateway errors:
The updated load test does generate a certain percentage of errors:
https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
So, expect to see a predefined percentage of 204s and 400s, along with the usual 200s in the nginx access logs.
The msisdn-gateway app logs should be clean with just msisdn and test data.

== Monitoring Loop Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Loop Server: Check the following repos:
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla-services/docs/issues
** https://github.com/mozilla/loop-client
** https://github.com/mozilla-services/msisdn-gateway/issues

* Bugzilla
** and in Bugzilla for non-Verified bugs: http://mzl.la/1qPxizw

* OPs and Infra
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues

== References ==
* Repositories and Documentation
** Check here: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#References

* The QA Test Environments
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments

* Test Plans and other QA
** https://docs.google.com/a/mozilla.com/document/d/1xYZijHnJ1JeJhONm2AYvxSnD2tWztC16WGFyD5U_d6g/
** https://wiki.mozilla.org/Services/Loop/Try_Loop
** https://wiki.mozilla.org/Loop/QA
** https://wiki.mozilla.org/Loop/QA#Server
** https://intranet.mozilla.org/QA/Loop/MVP

* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/LoopServerLoadTesting

2014-09-24T19:33:22Z

Jbonacci: /* Quick Verification Of Stage Deployments */

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
* Loop Server
For now, just run a quick loadtest 'make test'
cd loop-server
cd loadtests
make test SERVER_URL=https://loop.stage.mozaws.net

* Loop Client
Check https://call.stage.mozaws.net/config.js
Should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.stage.mozaws.net'};
and
WIP from the client team: end to end tests
Also:
https://call.stage.mozaws.net
curl https://call.stage.mozaws.net
curl -I https://call.stage.mozaws.net

and now the TOS: https://call.mozilla.com/legal/terms

* MSISDN Gateway
In the browser: https://msisdn.stage.mozaws.net
or do the following from a command line:
curl https://msisdn.stage.mozaws.net
curl -I https://msisdn.stage.mozaws.net
and
For now, just run a quick loadtest 'make test'
cd msisdn-gateway
cd loadtests
make test SERVER_URL=https://msisdn.stage.mozaws.net
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Quick Verification of Production Deployments ==
* This is a quick sanity test of the environment for after each Production deployment.
* Loop Server
In the browser: https://loop.services.mozilla.com
or do the following from a command line:
curl https://loop.services.mozilla.com
curl -I https://loop.services.mozilla.com

Then run a few 'make test' commands from the loadtests folder:
make test SERVER_URL=https://loop.services.mozilla.com
Note: this does hit a live third-party server

Then perform actual loop testing via desktop (Aurora/Nightly so far) and FxOS (2.1)
Verify that requests and strings point to Production environments

* Loop Client
In the browser: https://call.mozilla.com
should return "Welcome to the Loop web client."

In the browser: https://call.mozilla.com/config.js
should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

In the browser: https://call.mozilla.com/VERSION.txt
should return the version and build string info

curl https://call.mozilla.com
curl -I https://call.mozilla.com

Quick end-to-end tests:
Desktop: browser to browser
Desktop to FxOS
FxOS to Desktop
Two FxOS devices

* MSISDN Gateway
In the browser: https://msisdn.services.mozilla.com
or do the following from a command line:
curl https://msisdn.services.mozilla.com
curl -I https://msisdn.services.mozilla.com
Or
Run a single 'make test' command from the loadtests folder:
make test SERVER_URL=https://msisdn.services.mozilla.com
Note: this does hit a live third-party, so limit the check to a single run.
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Installing Loop-Server and the Loads tool on Localhost or AWS ==
* Installation:
git clone https://github.com/mozilla-services/loop-server.git
cd loop-server
npm install
ulimit -S -n 2048
npm test *
cd loadtests
make build
make test

Coverage report can be found here:
/loop-server/coverage/lcov-report/index.html

* This step requires the redis server to be installed and running:
Mac:
brew install redis
redis-server /usr/local/etc/redis.conf

Ubuntu Linux:
sudo apt-get install redis-server
sudo /usr/bin/redis-server /etc/redis/redis.conf
sudo tail -f /var/log/redis/redis-server.log

RHEL Linux:
Install redis from here: http://download.redis.io/releases
then
/usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
or similar

* Note: This will install a local copy of the Loads tool for use with the Loop-Server.

== Running the load test against the Loop-Server in Stage ==
* Stage environment:
$ cd loop-server/loadtests
$ make test
or
$ make test SERVER_URL=https://loop.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://loop.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the Loop-Server in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://loop.stage.mozaws.net

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing MSISDN-Gateway and the Loads tool on Localhost or AWS ==
* Installation:
** Install gmp, gmp-dev or gmp-devel
** Install ruby (very latest), ruby-dev or ruby-devel
** Install gem (required for fake_dynamo)
** Verify that gem is in your path
** Install redis-server to run the unit tests

* To install gmp
sudo yum -y install gmp, gmp-devel
or for Ubuntu
$ wget https://ftp.gnu.org/gnu/gmp/gmp-6.0.0a.tar.bz2
$ tar xvjf gmp-6.0.0a.tar.bz2
$ cd gmp-6.0.0
$ ./configure --prefix=/usr
$ make
$ make check
$ sudo make install

* To install ruby:
sudo yum -y install ruby, ruby-devel
or
sudo apt-get install ruby, ruby-dev

If this does not get you 1.9.3 or newer, then install manually:
Example:
$ wget http://cache.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p547.tar.gz
$ ./configure --prefix=/usr
$ make
$ sudo make install
(because for rhel, the default ruby version is 1.8.x.)
REF:
Main: https://www.ruby-lang.org/en/downloads/
Dev Tools: http://rubyinstaller.org/downloads/

* To install gem:
Grab rubygems from here: http://rubygems.org/pages/download
cd to rubygems directory
$ sudo ruby setup.rb

* To install fake_dynamo:
You should not have to install fake_dynamo since it is now part of the repo installer.
But if you do:
$ sudo gem install fake_dynamo
REF: https://github.com/ananthakumaran/fake_dynamo

* Install the msisdn-gateway repo:
$ git clone https://github.com/mozilla-services/msisdn-gateway.git
$ cd msisdn-gateway
$ sudo make install
(There is a bug open about the requirement to install with 'sudo')

* Note: This will install a local copy of the Loads tool for use with MSISDN-Gateway.

* Unit testing
Get redis-server installed
Start the server in a separate terminal or in the background with logging active
$ make test
The coverage report is here: msisdn-gateway/coverage/lcov-report/index.html

* Bugs:
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036158
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036174
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048451
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048456

== Running the load test against MSISDN-Gateway in Stage ==
* Building the load tests
$ cd loadtests
$ make build

* To load test the Stage environment:
$ make test SERVER_URL=https://msisdn.stage.mozaws.net
$ make bench SERVER_URL=https://msisdn.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the MSISDN-Gateway ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://msisdn.stage.mozaws.net

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (Loop-Server and MSISDN-Gateway):
*** Number of hits
*** Number of concurrent users

** For make bench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test

** For make megabench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* Loop-Server load test code
** The Loop-Server load test can not currently be configured in the code

* MSISDN-Gateway load test code
** The MSISDN-Gateway load test can be configured in the code - see the following lines:
** Test server URL: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
** Error percentages: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22

* General REFs:
** https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* Loop-Server
** TBD

* MSISDN-Gateway
** TBD

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Loop-Server Custom Metrics
** Opened web sockets
** Total web sockets
** Bytes/websockets
** addFailure (from the loads tool/client)

* MSISDN-Gateway Custom Metrics
** mt-flow
** ask-for-certificate
** try-wrong-code
** try-right-code
** momt-flow
** omxen-message-collision
** register
** unregister
** addFailure (from the loads tool/client)

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for Loop-Server (per server)
** /var/log/circus.log
** /var/log/loop_err.log
** /var/log/loop_out.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /var/log/nginx/access.log
** /var/log/nginx/error.log

* 2. Important logs for MSISDN-Gateway (per server)
** TBD

* Acceptable/Unacceptable Loop-Server errors:
hekad loop.stderr.log
The following are acceptable:
Decoder 'LoopServer-LoopServerDecoder' error: Failed parsing
Plugin 'AggregatorOutput' error: writing to heka.shared....

nginx logs:
The 200s (good stuff) and 101s (websockets) are acceptable.
The 499s are an artifact of the current load testing tool (V1).
You should only see them at the end of the load test.
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
Any percentage of 405s, 502s, or 503s is not acceptable.

Application logs:
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
See the loop_server.out.log file.

/var/log/loop_err.log
The following are acceptable: connect: res.on("header"): use on-headers module directly

In the Loads Cluster dashboard, watch for the following errors/failures:
string indices must be integers
OR
No JSON object could be decoded
OR
'hawk-session-token'

* Acceptable/Unacceptable MSISDN-Gateway errors:
The updated load test does generate a certain percentage of errors:
https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
So, expect to see a predefined percentage of 204s and 400s, along with the usual 200s in the nginx access logs.
The msisdn-gateway app logs should be clean with just msisdn and test data.

== Monitoring Loop Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Loop Server: Check the following repos:
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla-services/docs/issues
** https://github.com/mozilla/loop-client
** https://github.com/mozilla-services/msisdn-gateway/issues

* Bugzilla
** and in Bugzilla for non-Verified bugs: http://mzl.la/1qPxizw

* OPs and Infra
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues

== References ==
* Repositories and Documentation
** Check here: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#References

* The QA Test Environments
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments

* Test Plans and other QA
** https://docs.google.com/a/mozilla.com/document/d/1xYZijHnJ1JeJhONm2AYvxSnD2tWztC16WGFyD5U_d6g/
** https://wiki.mozilla.org/Services/Loop/Try_Loop
** https://wiki.mozilla.org/Loop/QA
** https://wiki.mozilla.org/Loop/QA#Server
** https://intranet.mozilla.org/QA/Loop/MVP

* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/LoopTestEnvironments

2014-09-24T19:32:56Z

Jbonacci: /* Loop Server Production Environments */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Loop web client.

Browser: https://call.mozilla.com/config.js
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

Browser: Browser: https://call.mozilla.com/VERSION.txt

and now the TOS: https://call.mozilla.com/legal/terms

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD

** loop-redis-stage - need the details

** Loop-Client: https://call.stage.mozaws.net

** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net

** Sentry: https://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad
** /media/ephemeral0/nginx/logs

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net

Check the version: https://call.stage.mozaws.net/VERSION.txt

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {serverUrl: 'https://https://loop.stage.mozaws.net'};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** https://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** None at this time. See https://bugzilla.mozilla.org/show_bug.cgi?id=1063134

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopTestEnvironments

2014-09-24T19:30:23Z

Jbonacci: /* Loop Server Production Environments */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Loop web client.

Browser: https://call.mozilla.com/config.js
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

Browser: Browser: https://call.mozilla.com/VERSION.txt

and now the TOS: https://call.mozilla.com/legal/terms

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD

** loop-redis-stage - need the details

** Loop-Client: https://call.stage.mozaws.net

** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net

** Sentry: https://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad
** /media/ephemeral0/nginx/logs

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net

Check the version: https://call.stage.mozaws.net/VERSION.txt

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {serverUrl: 'https://https://loop.stage.mozaws.net'};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** https://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** None at this time. See https://bugzilla.mozilla.org/show_bug.cgi?id=1063134

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopTestEnvironments

2014-09-24T19:20:17Z

Jbonacci: /* Loop Client Stage Details */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Loop web client.

Browser: https://call.mozilla.com/config.js
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

Browser: Browser: https://call.mozilla.com/VERSION.txt

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD

** loop-redis-stage - need the details

** Loop-Client: https://call.stage.mozaws.net

** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net

** Sentry: https://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** especially /data/loop-client/content/config.js
*** Note: this does not contain any app-specific config file in the config directory
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad
** /media/ephemeral0/nginx/logs

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net

Check the version: https://call.stage.mozaws.net/VERSION.txt

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {serverUrl: 'https://https://loop.stage.mozaws.net'};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** https://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** None at this time. See https://bugzilla.mozilla.org/show_bug.cgi?id=1063134

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/LoopTestEnvironments

2014-09-24T19:15:45Z

Jbonacci: /* Loop Client Stage Details */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Loop web client.

Browser: https://call.mozilla.com/config.js
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

Browser: Browser: https://call.mozilla.com/VERSION.txt

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD

** loop-redis-stage - need the details

** Loop-Client: https://call.stage.mozaws.net

** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net

** Sentry: https://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** Note: this does not contain any app-specific config file
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad
** /media/ephemeral0/nginx/logs

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net

Check the version: https://call.stage.mozaws.net/VERSION.txt

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {serverUrl: 'https://https://loop.stage.mozaws.net'};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** https://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** None at this time. See https://bugzilla.mozilla.org/show_bug.cgi?id=1063134

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/TokenServerAndSyncLoadTesting

2014-09-19T22:56:52Z

Jbonacci: /* Debugging the Issues */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier sections below...
** A Tokenserver+Verifier stack: See the TokenServer sections below...

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.

* TokenServer+Verifier Stage environment:
From the browser: https://token.stage.mozaws.net
curl https://token.stage.mozaws.net
curl -I https://token.stage.mozaws.net

Use the simple "make test" command from an install of tokenserver on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://token.stage.mozaws.net

Alternate method:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Stage:
bin/sync-cli.js -e EMAIL -p PASSWORD --env stage -t COLLECTION
where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms

* Verifier Stage environment:
In the browser: https://verifier.stage.mozaws.net/
curl https://verifier.stage.mozaws.net
curl -I https://verifier.stage.mozaws.net

Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.stage.mozaws.net

* Sync Server Stage environment:
Install server-syncstorage to the local host or AWS instance (see below)
$ cd server-syncstorage
Quick test against the TokenServer
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server <Stage TokenServer>
Current example:
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server
https://token.stage.mozaws.net/1.0/sync/1.5
Quick tests against the Sync nodes
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py <Stage Sync Node>#<Node Secret>
Current examples:
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
Get the Node Secret information from OPs

* Using TPS
** The TPS FxA/Sync automated tests can be used as well, but the following file will have to be edited to add Stage environment configuration parameters: https://github.com/mozilla/gecko-dev/blob/master/testing/tps/tps/testrunner.py
** See the following wiki page for more information: https://wiki.mozilla.org/User_Services/Sync/Run_TPS
** See also: https://bugzilla.mozilla.org/show_bug.cgi?id=1006675

== Quick Verification Of Production Deployments ==
* This is a quick sanity test of the environment after a new deployment.

* Tokenserver+Verifier Production Environment
In the browser: https://token.services.mozilla.com
curl https://token.services.mozilla.com
curl -I https://token.services.mozilla.com
Then:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Production:
bin/sync-cli.js -e PROD-EMAIL -p PASSWORD -t COLLECTION
where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms

* Verifier Production Environment
In the browser: https://verifier.accounts.firefox.com
curl https://verifier.accounts.firefox.com
curl -I https://verifier.accounts.firefox.com
Then:
Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.accounts.firefox.com

* Sync Server Stage environment
Sign in with a known FxA account and sync data with a current Production account (sync node).
Create a new FxA account and set up sync.

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment

== Installing BrowserID-Verifier and the Loads tool on Localhost or AWS ==
* Installation:
$ git clone git://github.com/mozilla/browserid-verifier
$ cd browserid-verifier
$ npm install
$ npm test
$ cd loadtest
$ make build
Note: This should hit Stage by default: SERVER_URL=https://verifier.stage.mozaws.net

* Note: This will install a local copy of the Loads tool for use with the verifier.

== Running the load test against the Verifier in Stage ==
* Stage environment:
$ make test
or
$ make test SERVER_URL=https://verifier.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://verifier.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Verifier hits the Stage mockmyid server

* Production environment:
$ make test SERVER_URL=https://verifier.accounts.firefox.com
$ make bench SERVER_URL=https://verifier.accounts.firefox.com

== Using the Loads V1 Services Cluster for the Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://verifier.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=TBD

* Production environment:
$ make megabench SERVER_URL=https://verifier.accounts.firefox.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing TokenServer+Verifier and the Loads tool on Localhost or AWS ==
* Installation:
$ git clone https://github.com/mozilla-services/tokenserver
$ cd tokenserver
$ make build
$ make test
Note: This is for local testing only
$ cd loadtest
$ make build
Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com

* Note: This will install a local copy of the Loads tool for use with TokenServer+Verifier.

== Running the load test against TokenServer+Verifier in Stage ==
* Stage environment:
$ make test SERVER_URL=https://token.stage.mozaws.net
$ make bench SERVER_URL=https://token.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: This also hits the Stage Verifier, which in turns hits the Stage mockmyid server

* And while we are at it...
* Dev environment:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org

* Production environment:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com

== Using the Loads V1 Services Cluster for TokenServer+Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://token.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=https://token.dev.lcip.org

* Production environment:
$ make megabench SERVER_URL=https://token.services.mozilla.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest

== Installing Sync 1.5 and the Loads tool on Localhost or AWS ==
Installation:
$ git clone https://github.com/mozilla-services/server-syncstorage/
$ cd server-syncstorage
$ make build
$ make test
$ cd loadtest
$ make build

* Note: This will install a local copy of the Loads tool for use with Sync 1.5.

== Running the load test against Sync 1.5 in Stage ==
* Loads against specific Sync nodes in Stage
$ make test SERVER_URL=https://your.storagenode.here#SECRET
$ make bench SERVER_URL=https://your.storagenode.here#SECRET
Sync Stage nodes:
https://sync-1-us-east-1.stage.mozaws.net
https://sync-2-us-east-1.stage.mozaws.net
https://sync-3-us-east-1.stage.mozaws.net
...etc...

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.

== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://your.storagenode.here#SECRET

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest

== Running a combined load test against TokenServer+Verifier and Sync 1.5 in Stage ==
* A combined loads test against TokenServer and Sync 1.5 in Stage
* This is done via the server-syncstorage directory that was cloned and built above
$ cd server-syncstorage
$ cd loadtest
$ make test SERVER_URL=https://your.tokenserver.here
$ make bench SERVER_URL=https://your.tokenserver.here

Examples for Stage:
$ make test SERVER_URL=https://token.stage.mozaws.net
$ make bench SERVER_URL=https://token.stage.mozaws.net
See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer.2BVerifier_Stage_Environment

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Tokenserver hits the Stage Verifier, which, in turn, hits the mockmyid server.

* And while we are at it...
Dev environment:
Examples:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org

Prod environment:
Examples:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com

See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA.2C_TokenServer.2C_and_Sync_Production_Environments
and https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer_and_Sync_1.5_Dev_Environments

== Using the Loads V1 Services Cluster for a combined load test in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://token.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=https://token.dev.lcip.org

* Prod environment:
$ make megabench SERVER_URL=https://token.services.mozilla.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of hits
*** Number of concurrent users

** For make bench (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of concurrent users
*** Duration of test

** For make megabench (using the LoadsCluster with BrowserID-Verifier, TokenSerer, Sync, Combined):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)
*** SSH (the user account needed to SSH into the loads cluster - the default is ubuntu)

* Tokenserver load test code
** The Tokenserver load test can be configured - see the following lines:
** Basic Settings: https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** MockMyID: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L19-L36
** Percentages: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L39-L51

* Verifier load test code
** The Verifier load test can be configured - see the following lines:
** Various settings: https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py#L13-L53

* Sync Server load test code
** The Sync Server load test can be configured - see the following lines:
** Setting MockMyID: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L26-L45
** Setting test distributions: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L48-L83

* REFs:
** https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py
** https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py
** https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* TokenServer
** File location: tokenserver/loadtest/loadtest.py
** Inside NoteAssignmentTest, test_realistic is the main load test; the others are for specific behaviors
** The test runs as following:
95% ask for assertions on existing users (on a DB filled by test_single_token_exchange)
4% ask for assertion on a new use
1% ask for a bad assertion
** A bug has been filed to get the following additional coverage for the load test:
*** generation numbers in assertion
*** client state string
** A bug has been filed to get some integration tests written:
*** to cover the edge/error cases not in the load test
*** to be pointed at a remote server

* Sync
** File location: server-syncstorage/loadtest/stress.py
** This is the Sync 2.0 load test that has been back-ported for Sync 1.5.
** The stress.py file is fully configurable for the following:
*** client probability
*** client distribution
*** collections
** A bug has been filed to add support for load testing tabs
*** The tab collection it uses memcache; we need to figure out a way to test it without overloading the server
** There are currently no constants to define how to select percentages per collection type
** Right now, we need to manually configure the collections list in stress.py:
*** collections = ['bookmarks', 'forms', 'passwords', 'history', 'prefs']
*** Basically, you can add more entries of each type, since the load test (per user/again/hit/pass) picks randomly from the list for any given request...

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Tokenserver Custom Metrics
** addFailure

* Verifier Custom Metrics
** addFailure

* Sync Custom Metrics
** addFailure

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for TokenServer (per server)
** /media/ephemeral0/logs/
** /media/ephemeral0/nginx/logs/default.access.log
** /media/ephemeral0/nginx/logs/default.error.log
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* 2. Important logs for Verifier (per server)
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* 3. Important error logs for Sync (per Sync node)
** /media/ephemeral0/logs/
** /media/ephemeral0/nginx/access.log
** /media/ephemeral0/error.log
** /media/ephemeral0/sync/sync.err
** /media/ephemeral0/sync/sync.log

* Acceptable TokenServer errors:
1% - 2% failures (as the following)
token.log:
"name": "token.assertion.invalid_signature_error"
"name": "token.assertion.verify_failure"
nginx access.log:
401s
NOTE: Values can be tweaked here:
https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L58-L60

The following types of errors are known:
/media/ephemeral0/logs/tokenserver/token.error.log
Exception KeyError: KeyError(49564400,) in <module 'threading'...
/media/ephemeral0/logs/tokenserver/token.log
..."Starting new HTTP connection (9): 127.0.0.1", "hostname": ...
{"error": "StopIteration()", "traceback": "Uncaught exception:\n
File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
..."Connection pool is full, discarding connection: 127.0.0.1", "...
Also, any 499s are probably an artifact of the current (V1) load test.
REF:
https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
https://bugzilla.mozilla.org/show_bug.cgi?id=1040397

OLD: Also, it may be the case that the following errors are "acceptable" if TS Stage is larger than Verifier Stage:
/media/ephemeral0/logs/tokenserver/token.error.log
Verifier-related errors of these types:
"HttpConnectionPool is full, discarding connection: verifier.stage.mozaws.net"
"Resetting dropped connection: verifier.stage.mozaws.net"
"Starting new HTTPS connection (179): verifier.stage.mozaws.net"

* Acceptable Verifier errors:
The verifier_out.log will show errors of the following types:
result: 'failure',\n reason: 'untrusted issuer...'
result: 'failure',\n reason: 'expired'
result: 'failure',\n reason: 'algorithms do not match'
result: 'failure',\n reason: 'audience mismatch: scheme mismatch'
Also, any 499s in the nginx logs are probably an artifact of the current (V1) load test.

* Acceptable Sync node errors:
In the nginx access.log files:
We will see some percentage of 404s. Right now we see the following:
14% 404s (compared to the total count of 200s)
with the config set up as follows:
users = 20
duration = 1800
agents = 5
Ideally, the overall percentage of 404s should drop the longer the load test.
Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
although they may show up in the logs after running the remote integration tests.
Also, any 499s are probably an artifact of the current (V1) load test.

In /var/log/hekad/sync_1_5.stderr.log
You may see some Decoder 'Sync-1_5-SlowQuery-MySqlSlowQueryDecoder' error: Failed parsing
and a lot of BSO INSERTs

In /media/ephemeral0/logs/sync/sync.err
You should see expected skew and QueuePool messages and Deprecation warnings
Also, these are known
Exception SystemExit
Exception KeyError
This is probably https://bugzilla.mozilla.org/show_bug.cgi?id=1040397

== Monitoring TS and Sync Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Tokenserver:
** Repo: https://github.com/mozilla-services/tokenserver/issues
** Bugzilla: http://mzl.la/1s4qZKn

* BrowserID-Verifier:
** Repo: https://github.com/mozilla/browserid-verifier/issues
** Bugzilla: no specific cateogory

* Sync:
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Bugzilla: http://mzl.la/VUrYQ5

* OPs and Infrastructure
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-aws/issues
** https://github.com/mozilla-services/loads-web/issues

== References ==
* Other URLs
** http://restmail.net
** https://mockmyid.com
* Repositories
** https://github.com/mozilla-services/tokenserver
** https://github.com/mozilla-services/server-syncstorage
** https://github.com/mozilla/browserid-verifier
** https://github.com/mozilla/browserid-local-verify
** https://github.com/callahad/mockmyid
** https://github.com/mozilla/restmail.net
* Documentation
** https://docs.services.mozilla.com
** https://docs.services.mozilla.com/token/
** https://wiki.mozilla.org/Services/Sagrada/TokenServer
** https://docs.services.mozilla.com/sync/
* The QA Test Environments:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
* Deploying the FxA Load Test environment for broker/agents usage:
** https://github.com/mozilla/fxa-deployment
* Sync 1.5 protocol, documentation, etc.
** https://github.com/mozilla-services/docs
** https://docs.services.mozilla.com/#how-to
** https://docs.services.mozilla.com/howtos/run-fxa.html
** https://docs.services.mozilla.com/token/apis.html
** https://docs.services.mozilla.com/storage/apis-1.5.html
** https://docs.services.mozilla.com/howtos/run-sync-1.5.html
** https://docs.services.mozilla.com/howtos/run-sync-1.5.html
** https://github.com/mozilla-services/syncserver
* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/LoopServerLoadTesting

2014-09-19T22:30:05Z

Jbonacci: /* Debugging the Issues */

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
* Loop Server
For now, just run a quick loadtest 'make test'
cd loop-server
cd loadtests
make test SERVER_URL=https://loop.stage.mozaws.net

* Loop Client
Check https://call.stage.mozaws.net/config.js
Should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.stage.mozaws.net'};
and
WIP from the client team: end to end tests
Also:
https://call.stage.mozaws.net
curl https://call.stage.mozaws.net
curl -I https://call.stage.mozaws.net

* MSISDN Gateway
In the browser: https://msisdn.stage.mozaws.net
or do the following from a command line:
curl https://msisdn.stage.mozaws.net
curl -I https://msisdn.stage.mozaws.net
and
For now, just run a quick loadtest 'make test'
cd msisdn-gateway
cd loadtests
make test SERVER_URL=https://msisdn.stage.mozaws.net
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Quick Verification of Production Deployments ==
* This is a quick sanity test of the environment for after each Production deployment.
* Loop Server
In the browser: https://loop.services.mozilla.com
or do the following from a command line:
curl https://loop.services.mozilla.com
curl -I https://loop.services.mozilla.com

Then run a few 'make test' commands from the loadtests folder:
make test SERVER_URL=https://loop.services.mozilla.com
Note: this does hit a live third-party server

Then perform actual loop testing via desktop (Aurora/Nightly so far) and FxOS (2.1)
Verify that requests and strings point to Production environments

* Loop Client
In the browser: https://call.mozilla.com
should return "Welcome to the Loop web client."

In the browser: https://call.mozilla.com/config.js
should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

In the browser: https://call.mozilla.com/VERSION.txt
should return the version and build string info

curl https://call.mozilla.com
curl -I https://call.mozilla.com

Quick end-to-end tests:
Desktop: browser to browser
Desktop to FxOS
FxOS to Desktop
Two FxOS devices

* MSISDN Gateway
In the browser: https://msisdn.services.mozilla.com
or do the following from a command line:
curl https://msisdn.services.mozilla.com
curl -I https://msisdn.services.mozilla.com
Or
Run a single 'make test' command from the loadtests folder:
make test SERVER_URL=https://msisdn.services.mozilla.com
Note: this does hit a live third-party, so limit the check to a single run.
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Installing Loop-Server and the Loads tool on Localhost or AWS ==
* Installation:
git clone https://github.com/mozilla-services/loop-server.git
cd loop-server
npm install
ulimit -S -n 2048
npm test *
cd loadtests
make build
make test

Coverage report can be found here:
/loop-server/coverage/lcov-report/index.html

* This step requires the redis server to be installed and running:
Mac:
brew install redis
redis-server /usr/local/etc/redis.conf

Ubuntu Linux:
sudo apt-get install redis-server
sudo /usr/bin/redis-server /etc/redis/redis.conf
sudo tail -f /var/log/redis/redis-server.log

RHEL Linux:
Install redis from here: http://download.redis.io/releases
then
/usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
or similar

* Note: This will install a local copy of the Loads tool for use with the Loop-Server.

== Running the load test against the Loop-Server in Stage ==
* Stage environment:
$ cd loop-server/loadtests
$ make test
or
$ make test SERVER_URL=https://loop.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://loop.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the Loop-Server in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://loop.stage.mozaws.net

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing MSISDN-Gateway and the Loads tool on Localhost or AWS ==
* Installation:
** Install gmp, gmp-dev or gmp-devel
** Install ruby (very latest), ruby-dev or ruby-devel
** Install gem (required for fake_dynamo)
** Verify that gem is in your path
** Install redis-server to run the unit tests

* To install gmp
sudo yum -y install gmp, gmp-devel
or for Ubuntu
$ wget https://ftp.gnu.org/gnu/gmp/gmp-6.0.0a.tar.bz2
$ tar xvjf gmp-6.0.0a.tar.bz2
$ cd gmp-6.0.0
$ ./configure --prefix=/usr
$ make
$ make check
$ sudo make install

* To install ruby:
sudo yum -y install ruby, ruby-devel
or
sudo apt-get install ruby, ruby-dev

If this does not get you 1.9.3 or newer, then install manually:
Example:
$ wget http://cache.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p547.tar.gz
$ ./configure --prefix=/usr
$ make
$ sudo make install
(because for rhel, the default ruby version is 1.8.x.)
REF:
Main: https://www.ruby-lang.org/en/downloads/
Dev Tools: http://rubyinstaller.org/downloads/

* To install gem:
Grab rubygems from here: http://rubygems.org/pages/download
cd to rubygems directory
$ sudo ruby setup.rb

* To install fake_dynamo:
You should not have to install fake_dynamo since it is now part of the repo installer.
But if you do:
$ sudo gem install fake_dynamo
REF: https://github.com/ananthakumaran/fake_dynamo

* Install the msisdn-gateway repo:
$ git clone https://github.com/mozilla-services/msisdn-gateway.git
$ cd msisdn-gateway
$ sudo make install
(There is a bug open about the requirement to install with 'sudo')

* Note: This will install a local copy of the Loads tool for use with MSISDN-Gateway.

* Unit testing
Get redis-server installed
Start the server in a separate terminal or in the background with logging active
$ make test
The coverage report is here: msisdn-gateway/coverage/lcov-report/index.html

* Bugs:
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036158
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036174
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048451
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048456

== Running the load test against MSISDN-Gateway in Stage ==
* Building the load tests
$ cd loadtests
$ make build

* To load test the Stage environment:
$ make test SERVER_URL=https://msisdn.stage.mozaws.net
$ make bench SERVER_URL=https://msisdn.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the MSISDN-Gateway ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://msisdn.stage.mozaws.net

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (Loop-Server and MSISDN-Gateway):
*** Number of hits
*** Number of concurrent users

** For make bench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test

** For make megabench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* Loop-Server load test code
** The Loop-Server load test can not currently be configured in the code

* MSISDN-Gateway load test code
** The MSISDN-Gateway load test can be configured in the code - see the following lines:
** Test server URL: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
** Error percentages: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22

* General REFs:
** https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* Loop-Server
** TBD

* MSISDN-Gateway
** TBD

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Loop-Server Custom Metrics
** Opened web sockets
** Total web sockets
** Bytes/websockets
** addFailure (from the loads tool/client)

* MSISDN-Gateway Custom Metrics
** mt-flow
** ask-for-certificate
** try-wrong-code
** try-right-code
** momt-flow
** omxen-message-collision
** register
** unregister
** addFailure (from the loads tool/client)

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for Loop-Server (per server)
** /var/log/circus.log
** /var/log/loop_err.log
** /var/log/loop_out.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /var/log/nginx/access.log
** /var/log/nginx/error.log

* 2. Important logs for MSISDN-Gateway (per server)
** TBD

* Acceptable/Unacceptable Loop-Server errors:
hekad loop.stderr.log
The following are acceptable:
Decoder 'LoopServer-LoopServerDecoder' error: Failed parsing
Plugin 'AggregatorOutput' error: writing to heka.shared....

nginx logs:
The 200s (good stuff) and 101s (websockets) are acceptable.
The 499s are an artifact of the current load testing tool (V1).
You should only see them at the end of the load test.
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
Any percentage of 405s, 502s, or 503s is not acceptable.

Application logs:
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
See the loop_server.out.log file.

/var/log/loop_err.log
The following are acceptable: connect: res.on("header"): use on-headers module directly

In the Loads Cluster dashboard, watch for the following errors/failures:
string indices must be integers
OR
No JSON object could be decoded
OR
'hawk-session-token'

* Acceptable/Unacceptable MSISDN-Gateway errors:
The updated load test does generate a certain percentage of errors:
https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
So, expect to see a predefined percentage of 204s and 400s, along with the usual 200s in the nginx access logs.
The msisdn-gateway app logs should be clean with just msisdn and test data.

== Monitoring Loop Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Loop Server: Check the following repos:
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla-services/docs/issues
** https://github.com/mozilla/loop-client
** https://github.com/mozilla-services/msisdn-gateway/issues

* Bugzilla
** and in Bugzilla for non-Verified bugs: http://mzl.la/1qPxizw

* OPs and Infra
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues

== References ==
* Repositories and Documentation
** Check here: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#References

* The QA Test Environments
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments

* Test Plans and other QA
** https://docs.google.com/a/mozilla.com/document/d/1xYZijHnJ1JeJhONm2AYvxSnD2tWztC16WGFyD5U_d6g/
** https://wiki.mozilla.org/Services/Loop/Try_Loop
** https://wiki.mozilla.org/Loop/QA
** https://wiki.mozilla.org/Loop/QA#Server
** https://intranet.mozilla.org/QA/Loop/MVP

* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/LoopServerLoadTesting

2014-09-18T02:50:51Z

Jbonacci: /* Summary for Loop Server, Loop Client, MSISDN Gateway */

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
* Loop Server
For now, just run a quick loadtest 'make test'
cd loop-server
cd loadtests
make test SERVER_URL=https://loop.stage.mozaws.net

* Loop Client
Check https://call.stage.mozaws.net/config.js
Should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.stage.mozaws.net'};
and
WIP from the client team: end to end tests
Also:
https://call.stage.mozaws.net
curl https://call.stage.mozaws.net
curl -I https://call.stage.mozaws.net

* MSISDN Gateway
In the browser: https://msisdn.stage.mozaws.net
or do the following from a command line:
curl https://msisdn.stage.mozaws.net
curl -I https://msisdn.stage.mozaws.net
and
For now, just run a quick loadtest 'make test'
cd msisdn-gateway
cd loadtests
make test SERVER_URL=https://msisdn.stage.mozaws.net
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Quick Verification of Production Deployments ==
* This is a quick sanity test of the environment for after each Production deployment.
* Loop Server
In the browser: https://loop.services.mozilla.com
or do the following from a command line:
curl https://loop.services.mozilla.com
curl -I https://loop.services.mozilla.com

Then run a few 'make test' commands from the loadtests folder:
make test SERVER_URL=https://loop.services.mozilla.com
Note: this does hit a live third-party server

Then perform actual loop testing via desktop (Aurora/Nightly so far) and FxOS (2.1)
Verify that requests and strings point to Production environments

* Loop Client
In the browser: https://call.mozilla.com
should return "Welcome to the Loop web client."

In the browser: https://call.mozilla.com/config.js
should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

In the browser: https://call.mozilla.com/VERSION.txt
should return the version and build string info

curl https://call.mozilla.com
curl -I https://call.mozilla.com

Quick end-to-end tests:
Desktop: browser to browser
Desktop to FxOS
FxOS to Desktop
Two FxOS devices

* MSISDN Gateway
In the browser: https://msisdn.services.mozilla.com
or do the following from a command line:
curl https://msisdn.services.mozilla.com
curl -I https://msisdn.services.mozilla.com
Or
Run a single 'make test' command from the loadtests folder:
make test SERVER_URL=https://msisdn.services.mozilla.com
Note: this does hit a live third-party, so limit the check to a single run.
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Installing Loop-Server and the Loads tool on Localhost or AWS ==
* Installation:
git clone https://github.com/mozilla-services/loop-server.git
cd loop-server
npm install
ulimit -S -n 2048
npm test *
cd loadtests
make build
make test

Coverage report can be found here:
/loop-server/coverage/lcov-report/index.html

* This step requires the redis server to be installed and running:
Mac:
brew install redis
redis-server /usr/local/etc/redis.conf

Ubuntu Linux:
sudo apt-get install redis-server
sudo /usr/bin/redis-server /etc/redis/redis.conf
sudo tail -f /var/log/redis/redis-server.log

RHEL Linux:
Install redis from here: http://download.redis.io/releases
then
/usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
or similar

* Note: This will install a local copy of the Loads tool for use with the Loop-Server.

== Running the load test against the Loop-Server in Stage ==
* Stage environment:
$ cd loop-server/loadtests
$ make test
or
$ make test SERVER_URL=https://loop.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://loop.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the Loop-Server in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://loop.stage.mozaws.net

* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/

* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json

* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.

* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing MSISDN-Gateway and the Loads tool on Localhost or AWS ==
* Installation:
** Install gmp, gmp-dev or gmp-devel
** Install ruby (very latest), ruby-dev or ruby-devel
** Install gem (required for fake_dynamo)
** Verify that gem is in your path
** Install redis-server to run the unit tests

* To install gmp
sudo yum -y install gmp, gmp-devel
or for Ubuntu
$ wget https://ftp.gnu.org/gnu/gmp/gmp-6.0.0a.tar.bz2
$ tar xvjf gmp-6.0.0a.tar.bz2
$ cd gmp-6.0.0
$ ./configure --prefix=/usr
$ make
$ make check
$ sudo make install

* To install ruby:
sudo yum -y install ruby, ruby-devel
or
sudo apt-get install ruby, ruby-dev

If this does not get you 1.9.3 or newer, then install manually:
Example:
$ wget http://cache.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p547.tar.gz
$ ./configure --prefix=/usr
$ make
$ sudo make install
(because for rhel, the default ruby version is 1.8.x.)
REF:
Main: https://www.ruby-lang.org/en/downloads/
Dev Tools: http://rubyinstaller.org/downloads/

* To install gem:
Grab rubygems from here: http://rubygems.org/pages/download
cd to rubygems directory
$ sudo ruby setup.rb

* To install fake_dynamo:
You should not have to install fake_dynamo since it is now part of the repo installer.
But if you do:
$ sudo gem install fake_dynamo
REF: https://github.com/ananthakumaran/fake_dynamo

* Install the msisdn-gateway repo:
$ git clone https://github.com/mozilla-services/msisdn-gateway.git
$ cd msisdn-gateway
$ sudo make install
(There is a bug open about the requirement to install with 'sudo')

* Note: This will install a local copy of the Loads tool for use with MSISDN-Gateway.

* Unit testing
Get redis-server installed
Start the server in a separate terminal or in the background with logging active
$ make test
The coverage report is here: msisdn-gateway/coverage/lcov-report/index.html

* Bugs:
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036158
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036174
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048451
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048456

== Running the load test against MSISDN-Gateway in Stage ==
* Building the load tests
$ cd loadtests
$ make build

* To load test the Stage environment:
$ make test SERVER_URL=https://msisdn.stage.mozaws.net
$ make bench SERVER_URL=https://msisdn.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details

== Using the Loads V1 Services Cluster for the MSISDN-Gateway ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://msisdn.stage.mozaws.net

* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json

* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (Loop-Server and MSISDN-Gateway):
*** Number of hits
*** Number of concurrent users

** For make bench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test

** For make megabench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* Loop-Server load test code
** The Loop-Server load test can not currently be configured in the code

* MSISDN-Gateway load test code
** The MSISDN-Gateway load test can be configured in the code - see the following lines:
** Test server URL: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
** Error percentages: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22

* General REFs:
** https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* Loop-Server
** TBD

* MSISDN-Gateway
** TBD

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Loop-Server Custom Metrics
** Opened web sockets
** Total web sockets
** Bytes/websockets
** addFailure (from the loads tool/client)

* MSISDN-Gateway Custom Metrics
** mt-flow
** ask-for-certificate
** try-wrong-code
** try-right-code
** momt-flow
** omxen-message-collision
** register
** unregister
** addFailure (from the loads tool/client)

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for Loop-Server (per server)
** /var/log/circus.log
** /var/log/loop_err.log
** /var/log/loop_out.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /var/log/nginx/access.log
** /var/log/nginx/error.log

* 2. Important logs for MSISDN-Gateway (per server)
** TBD

* Acceptable/Unacceptable Loop-Server errors:
hekad loop.stderr.log
The following are acceptable:
Decoder 'LoopServer-LoopServerDecoder' error: Failed parsing
Plugin 'AggregatorOutput' error: writing to heka.shared....

nginx logs:
Some percentage of 200s, 204s, and 404s is acceptable. Some of the 404s are actually bot/spam
activity in the /media/ephemeral0/nginx/logs/loop_server.access.log and
/media/ephemeral0/circus/loop_server/loop_server.out.log logs.
Any percentage of 405s, 502s, or 503s is not acceptable.

/var/log/loop_err.log
The following are acceptable: connect: res.on("header"): use on-headers module directly

In the Loads Cluster dashboard, watch for the following errors/failures:
string indices must be integers
OR
No JSON object could be decoded
OR
'hawk-session-token'

* Acceptable/Unacceptable MSISDN-Gateway errors:
The updated load test does generate a certain percentage of errors:
https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
So, expect to see a predefined percentage of 204s and 400s, along with the usual 200s in the nginx access logs.
The msisdn-gateway app logs should be clean with just msisdn and test data.

== Monitoring Loop Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Loop Server: Check the following repos:
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla-services/docs/issues
** https://github.com/mozilla/loop-client
** https://github.com/mozilla-services/msisdn-gateway/issues

* Bugzilla
** and in Bugzilla for non-Verified bugs: http://mzl.la/1qPxizw

* OPs and Infra
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues

== References ==
* Repositories and Documentation
** Check here: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#References

* The QA Test Environments
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments

* Test Plans and other QA
** https://docs.google.com/a/mozilla.com/document/d/1xYZijHnJ1JeJhONm2AYvxSnD2tWztC16WGFyD5U_d6g/
** https://wiki.mozilla.org/Services/Loop/Try_Loop
** https://wiki.mozilla.org/Loop/QA
** https://wiki.mozilla.org/Loop/QA#Server
** https://intranet.mozilla.org/QA/Loop/MVP

* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/TokenServerAndSyncLoadTesting

2014-09-18T02:50:38Z

Jbonacci: /* Summary for Tokenserver, Verifier, and Sync 1.5 */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier sections below...
** A Tokenserver+Verifier stack: See the TokenServer sections below...

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.

* TokenServer+Verifier Stage environment:
From the browser: https://token.stage.mozaws.net
curl https://token.stage.mozaws.net
curl -I https://token.stage.mozaws.net

Use the simple "make test" command from an install of tokenserver on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://token.stage.mozaws.net

Alternate method:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Stage:
bin/sync-cli.js -e EMAIL -p PASSWORD --env stage -t COLLECTION
where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms

* Verifier Stage environment:
In the browser: https://verifier.stage.mozaws.net/
curl https://verifier.stage.mozaws.net
curl -I https://verifier.stage.mozaws.net

Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.stage.mozaws.net

* Sync Server Stage environment:
Install server-syncstorage to the local host or AWS instance (see below)
$ cd server-syncstorage
Quick test against the TokenServer
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server <Stage TokenServer>
Current example:
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server
https://token.stage.mozaws.net/1.0/sync/1.5
Quick tests against the Sync nodes
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py <Stage Sync Node>#<Node Secret>
Current examples:
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
Get the Node Secret information from OPs

* Using TPS
** The TPS FxA/Sync automated tests can be used as well, but the following file will have to be edited to add Stage environment configuration parameters: https://github.com/mozilla/gecko-dev/blob/master/testing/tps/tps/testrunner.py
** See the following wiki page for more information: https://wiki.mozilla.org/User_Services/Sync/Run_TPS
** See also: https://bugzilla.mozilla.org/show_bug.cgi?id=1006675

== Quick Verification Of Production Deployments ==
* This is a quick sanity test of the environment after a new deployment.

* Tokenserver+Verifier Production Environment
In the browser: https://token.services.mozilla.com
curl https://token.services.mozilla.com
curl -I https://token.services.mozilla.com
Then:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Production:
bin/sync-cli.js -e PROD-EMAIL -p PASSWORD -t COLLECTION
where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms

* Verifier Production Environment
In the browser: https://verifier.accounts.firefox.com
curl https://verifier.accounts.firefox.com
curl -I https://verifier.accounts.firefox.com
Then:
Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.accounts.firefox.com

* Sync Server Stage environment
Sign in with a known FxA account and sync data with a current Production account (sync node).
Create a new FxA account and set up sync.

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment

== Installing BrowserID-Verifier and the Loads tool on Localhost or AWS ==
* Installation:
$ git clone git://github.com/mozilla/browserid-verifier
$ cd browserid-verifier
$ npm install
$ npm test
$ cd loadtest
$ make build
Note: This should hit Stage by default: SERVER_URL=https://verifier.stage.mozaws.net

* Note: This will install a local copy of the Loads tool for use with the verifier.

== Running the load test against the Verifier in Stage ==
* Stage environment:
$ make test
or
$ make test SERVER_URL=https://verifier.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://verifier.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Verifier hits the Stage mockmyid server

* Production environment:
$ make test SERVER_URL=https://verifier.accounts.firefox.com
$ make bench SERVER_URL=https://verifier.accounts.firefox.com

== Using the Loads V1 Services Cluster for the Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://verifier.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=TBD

* Production environment:
$ make megabench SERVER_URL=https://verifier.accounts.firefox.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing TokenServer+Verifier and the Loads tool on Localhost or AWS ==
* Installation:
$ git clone https://github.com/mozilla-services/tokenserver
$ cd tokenserver
$ make build
$ make test
Note: This is for local testing only
$ cd loadtest
$ make build
Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com

* Note: This will install a local copy of the Loads tool for use with TokenServer+Verifier.

== Running the load test against TokenServer+Verifier in Stage ==
* Stage environment:
$ make test SERVER_URL=https://token.stage.mozaws.net
$ make bench SERVER_URL=https://token.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: This also hits the Stage Verifier, which in turns hits the Stage mockmyid server

* And while we are at it...
* Dev environment:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org

* Production environment:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com

== Using the Loads V1 Services Cluster for TokenServer+Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://token.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=https://token.dev.lcip.org

* Production environment:
$ make megabench SERVER_URL=https://token.services.mozilla.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest

== Installing Sync 1.5 and the Loads tool on Localhost or AWS ==
Installation:
$ git clone https://github.com/mozilla-services/server-syncstorage/
$ cd server-syncstorage
$ make build
$ make test
$ cd loadtest
$ make build

* Note: This will install a local copy of the Loads tool for use with Sync 1.5.

== Running the load test against Sync 1.5 in Stage ==
* Loads against specific Sync nodes in Stage
$ make test SERVER_URL=https://your.storagenode.here#SECRET
$ make bench SERVER_URL=https://your.storagenode.here#SECRET
Sync Stage nodes:
https://sync-1-us-east-1.stage.mozaws.net
https://sync-2-us-east-1.stage.mozaws.net
https://sync-3-us-east-1.stage.mozaws.net
...etc...

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.

== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://your.storagenode.here#SECRET

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest

== Running a combined load test against TokenServer+Verifier and Sync 1.5 in Stage ==
* A combined loads test against TokenServer and Sync 1.5 in Stage
* This is done via the server-syncstorage directory that was cloned and built above
$ cd server-syncstorage
$ cd loadtest
$ make test SERVER_URL=https://your.tokenserver.here
$ make bench SERVER_URL=https://your.tokenserver.here

Examples for Stage:
$ make test SERVER_URL=https://token.stage.mozaws.net
$ make bench SERVER_URL=https://token.stage.mozaws.net
See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer.2BVerifier_Stage_Environment

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Tokenserver hits the Stage Verifier, which, in turn, hits the mockmyid server.

* And while we are at it...
Dev environment:
Examples:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org

Prod environment:
Examples:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com

See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA.2C_TokenServer.2C_and_Sync_Production_Environments
and https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer_and_Sync_1.5_Dev_Environments

== Using the Loads V1 Services Cluster for a combined load test in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://token.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=https://token.dev.lcip.org

* Prod environment:
$ make megabench SERVER_URL=https://token.services.mozilla.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of hits
*** Number of concurrent users

** For make bench (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of concurrent users
*** Duration of test

** For make megabench (using the LoadsCluster with BrowserID-Verifier, TokenSerer, Sync, Combined):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)
*** SSH (the user account needed to SSH into the loads cluster - the default is ubuntu)

* Tokenserver load test code
** The Tokenserver load test can be configured - see the following lines:
** Basic Settings: https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** MockMyID: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L19-L36
** Percentages: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L39-L51

* Verifier load test code
** The Verifier load test can be configured - see the following lines:
** Various settings: https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py#L13-L53

* Sync Server load test code
** The Sync Server load test can be configured - see the following lines:
** Setting MockMyID: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L26-L45
** Setting test distributions: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L48-L83

* REFs:
** https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py
** https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py
** https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* TokenServer
** File location: tokenserver/loadtest/loadtest.py
** Inside NoteAssignmentTest, test_realistic is the main load test; the others are for specific behaviors
** The test runs as following:
95% ask for assertions on existing users (on a DB filled by test_single_token_exchange)
4% ask for assertion on a new use
1% ask for a bad assertion
** A bug has been filed to get the following additional coverage for the load test:
*** generation numbers in assertion
*** client state string
** A bug has been filed to get some integration tests written:
*** to cover the edge/error cases not in the load test
*** to be pointed at a remote server

* Sync
** File location: server-syncstorage/loadtest/stress.py
** This is the Sync 2.0 load test that has been back-ported for Sync 1.5.
** The stress.py file is fully configurable for the following:
*** client probability
*** client distribution
*** collections
** A bug has been filed to add support for load testing tabs
*** The tab collection it uses memcache; we need to figure out a way to test it without overloading the server
** There are currently no constants to define how to select percentages per collection type
** Right now, we need to manually configure the collections list in stress.py:
*** collections = ['bookmarks', 'forms', 'passwords', 'history', 'prefs']
*** Basically, you can add more entries of each type, since the load test (per user/again/hit/pass) picks randomly from the list for any given request...

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Tokenserver Custom Metrics
** addFailure

* Verifier Custom Metrics
** addFailure

* Sync Custom Metrics
** addFailure

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for TokenServer (per server)
** /media/ephemeral0/logs/
** /media/ephemeral0/nginx/logs/default.access.log
** /media/ephemeral0/nginx/logs/default.error.log
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* 2. Important logs for Verifier (per server)
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* 3. Important error logs for Sync (per Sync node)
** /media/ephemeral0/logs/
** /media/ephemeral0/nginx/access.log
** /media/ephemeral0/error.log
** /media/ephemeral0/sync/sync.err
** /media/ephemeral0/sync/sync.log

* Acceptable TokenServer errors:
1% - 2% failures (as the following)
token.log:
"name": "token.assertion.invalid_signature_error"
"name": "token.assertion.verify_failure"
nginx access.log:
401s
NOTE: Values can be tweaked here:
https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L58-L60

The following types of errors are known:
/media/ephemeral0/logs/tokenserver/token.error.log
Exception KeyError: KeyError(49564400,) in <module 'threading'...
/media/ephemeral0/logs/tokenserver/token.log
..."Starting new HTTP connection (9): 127.0.0.1", "hostname": ...
{"error": "StopIteration()", "traceback": "Uncaught exception:\n
File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
..."Connection pool is full, discarding connection: 127.0.0.1", "...
REF:
https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
https://bugzilla.mozilla.org/show_bug.cgi?id=1040397

OLD: Also, it may be the case that the following errors are "acceptable" if TS Stage is larger than Verifier Stage:
/media/ephemeral0/logs/tokenserver/token.error.log
Verifier-related errors of these types:
"HttpConnectionPool is full, discarding connection: verifier.stage.mozaws.net"
"Resetting dropped connection: verifier.stage.mozaws.net"
"Starting new HTTPS connection (179): verifier.stage.mozaws.net"

* Acceptable Verifier errors:
The verifier_out.log will show errors of the following types:
result: 'failure',\n reason: 'untrusted issuer...'
result: 'failure',\n reason: 'expired'
result: 'failure',\n reason: 'algorithms do not match'
result: 'failure',\n reason: 'audience mismatch: scheme mismatch'

* Acceptable Sync node errors:
In the nginx access.log files:
We will see some percentage of 404s. Right now we see the following:
14% 404s (compared to the total count of 200s)
with the config set up as follows:
users = 20
duration = 1800
agents = 5
Ideally, the overall percentage of 404s should drop the longer the load test.
Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
although they may show up in the logs after running the remote integration tests.

In /var/log/hekad/sync_1_5.stderr.log
You may see some Decoder 'Sync-1_5-SlowQuery-MySqlSlowQueryDecoder' error: Failed parsing
and a lot of BSO INSERTs

In /media/ephemeral0/logs/sync/sync.err
You should see expected skew and QueuePool messages and Deprecation warnings
Also, these are known
Exception SystemExit
Exception KeyError
This is probably https://bugzilla.mozilla.org/show_bug.cgi?id=1040397

== Monitoring TS and Sync Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Tokenserver:
** Repo: https://github.com/mozilla-services/tokenserver/issues
** Bugzilla: http://mzl.la/1s4qZKn

* BrowserID-Verifier:
** Repo: https://github.com/mozilla/browserid-verifier/issues
** Bugzilla: no specific cateogory

* Sync:
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Bugzilla: http://mzl.la/VUrYQ5

* OPs and Infrastructure
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-aws/issues
** https://github.com/mozilla-services/loads-web/issues

== References ==
* Other URLs
** http://restmail.net
** https://mockmyid.com
* Repositories
** https://github.com/mozilla-services/tokenserver
** https://github.com/mozilla-services/server-syncstorage
** https://github.com/mozilla/browserid-verifier
** https://github.com/mozilla/browserid-local-verify
** https://github.com/callahad/mockmyid
** https://github.com/mozilla/restmail.net
* Documentation
** https://docs.services.mozilla.com
** https://docs.services.mozilla.com/token/
** https://wiki.mozilla.org/Services/Sagrada/TokenServer
** https://docs.services.mozilla.com/sync/
* The QA Test Environments:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
* Deploying the FxA Load Test environment for broker/agents usage:
** https://github.com/mozilla/fxa-deployment
* Sync 1.5 protocol, documentation, etc.
** https://github.com/mozilla-services/docs
** https://docs.services.mozilla.com/#how-to
** https://docs.services.mozilla.com/howtos/run-fxa.html
** https://docs.services.mozilla.com/token/apis.html
** https://docs.services.mozilla.com/storage/apis-1.5.html
** https://docs.services.mozilla.com/howtos/run-sync-1.5.html
** https://docs.services.mozilla.com/howtos/run-sync-1.5.html
** https://github.com/mozilla-services/syncserver
* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/FxALoadTesting

2014-09-18T02:50:23Z

Jbonacci: /* Summary for FxA-Auth-Server */

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
Install FxA-Auth-Server to a local host or an AWS instance (see below)
$ cd fxa-auth-server
Run the integration tests against the remote Stage server (load balancer)
$ PUBLIC_URL=<FxA Stage> npm run test-remote
Current example:
$ PUBLIC_URL=https://api-accounts.stage.mozaws.net npm run test-remote
* NOTE: Make sure to install and test from the same branch that is deployed to Stage (ie do not use Master for running the tests against Stage or Production).

* Using TPS
** The TPS FxA/Sync automated tests can be used as well, but the following file will have to be edited to add Stage environment configuration parameters: https://github.com/mozilla/gecko-dev/blob/master/testing/tps/tps/testrunner.py
** See the following wiki page for more information: https://wiki.mozilla.org/User_Services/Sync/Run_TPS
** See also: https://bugzilla.mozilla.org/show_bug.cgi?id=1006675

== Quick Verification Of Production Deployments ==
* This is a quick sanity test of the environment after each new deployment. There are other verifications that can be run as well
Install FxA-Auth-Server to a local host or an AWS instance (see below)
$ cd fxa-auth-server
Run the integration tests against the remote Stage server (load balancer)
$ PUBLIC_URL=<FxA Prod> npm run test-remote
Current example:
$ PUBLIC_URL=https://api.accounts.firefox.com npm run test-remote
* NOTE: Make sure to install and test from the same branch that is deployed to Production.

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Installing FxA-Auth-Server and the Loads tool on Localhost or AWS ==
Installation:
$ git clone https://github.com/mozilla/fxa-auth-server.git
$ cd ./fxa-auth-server
$ npm install
$ npm test
$ cd ./test/load
$ make build

* Note: 'npm install' may need to be run now as root.
* Note: This will install a local copy of the Loads tool for use with FxA-Auth-Server.

== Running the Loads tool against FxA Stage ==
* The basic load test can be run as follows
$ make test SERVER_URL=https://api-accounts.stage.mozaws.net

* The full, default load test can be run as follows
$ make bench SERVER_URL=https://api-accounts.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

* Configuring the bench load test - config folder:
** The test.ini file (for make test) can be configured for the following:
*** Number of hits
*** Number of concurrent users
** The bench.ini file (for make bench) can be configured for the following:
*** Number of concurrent users
*** Duration of test

* For both tests, start with the defaults, then tweak the duration. Users and Agents are optional tweaks/changes. Also, we can configure the bench load test to run in detached mode with an appropriate loads detach and observer settings.

* REF: https://github.com/mozilla/fxa-auth-server/tree/master/loadtest/config

== Running the Loads tool against FxA Development or Production ==
* This can be done if we are comparing Stage vs. some other environment and have access to the AWS logs in Dev or Production:

* Dev:
$ make test SERVER_URL=https://accounts.dev.lcip.org
$ make bench SERVER_URL=https://accounts.dev.lcip.org

* Prod:
$ make test SERVER_URL=https://api.accounts.firefox.com
$ make bench SERVER_URL=https://api.accounts.firefox.com

* The same optional configuration changes apply here.

== Using the Loads V1 Services Cluster ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).

* Testing against the Stage environment:
$ make megabench SERVER_URL=https://api-accounts.stage.mozaws.net

* Testing against the Dev environment:
$ make megabench SERVER_URL=https://api-accounts.dev.lcip.org

* Testing against the Prod enviornment:
$ make megabench SERVER_URL=https://api.accounts.firefox.com

* Configuring the megabench load test - config folder:
** The megabench.ini file (for make megabench) can be configured for the following:
*** Number of concurrent users
*** Duration of test
*** Include file (leave as defined for now)
*** Python dependencies (leave as defined for now)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* REF: https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
* REF: https://github.com/mozilla/fxa-auth-server/tree/master/loadtest/config

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test:
*** Number of hits
*** Number of concurrent users

** For make bench:
*** Number of concurrent users
*** Duration of test

** For make megabench:
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)

* Load Test code: loadtests.py
** The load test can be configured in the code - see the following lines:
** https://github.com/mozilla/fxa-auth-server/blob/master/test/load/loadtests.py#L17-L39

* General REFs:
** https://github.com/mozilla/fxa-auth-server/blob/master/test/load/loadtests.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* Location fxa-auth-server/loadtest/loadtests.py
* The following items are covered in the load test
** test_auth_server is the main entry point in the loadtests.py file
*** account creation
*** session creation
** account deletion
** session deletion
* Integration tests
** These are designed to cover the edge/error cases that are not applicable to the load test
** The tests can be run against a remote server

== Analyzing the Results ==
* TBD

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for FxA-Auth-Server (per server)
** /media/ephemeral0/fxa-auth-server/auth_err.log.*
** /media/ephemeral0/fxa-auth-server/auth_out.log
** /media/ephemeral0/heka/hekad_err.log
** /media/ephemeral0/heka/hekad_out.log
** /media/ephemeral0/nginx/logs/access.log
** /media/ephemeral0/nginx/logs/error.log

* Acceptable FxA-Auth-Server errors
503s: especially of this type - /v1/certificate/sign - are usually a sign that we are overloading the hosts

400s: we should never see these in the logs, especially if the "errno" value is 105.
Check the fxa-auth-server/auth_err.log
400s: "errno" values of 101, 102 are ok. These can be expected during a load test.

ELB issues: we may see 503s and corresponding "err":"cannot enqueue work: maximum backlog exceeded (30)"
messages if one or more of the hosts behind the ELB is receiving most of the load traffic.
REF: https://github.com/mozilla/fxa-auth-server/issues/647

== Monitoring FxA Stage ==
* Loads dashboard:
** http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com/
** or http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* FxA
** https://github.com/mozilla/fxa-auth-server/issues
** https://github.com/mozilla/fxa-content-server/issues
** and several others

* Bugzilla
** No specific category

* Infrastructure
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and clusters
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues

== Capacity Planning Stage and Production ==
* QA is tasked with providing some capacity requirements and constraints based on repeated load testing of the FxA-Auth-Server Stage environment.
* The goal is to be able to work with OPs to develop a realistic plan for deploying and maintaining the production environment at a level expected for projected user traffic, etc.

* Brainstorming the QA role:
** QA needs to get some realistic numbers from the Product team. This could be as simple as traffic flow (number of users per day or per segments of the day - peaks and valleys) or more detailed:
*** Traffic flow - QPS/RPS
*** Average number of users per time segment
*** Average and peak latency
*** Error percentages and thresholds
*** etc
** QA gets help from OPs to learn how to measure those required numbers/values using StackDriver or other tools (or to get data from OPsView). If those numbers can not be measured then we either need to
*** get a different set of data points from the Product team
*** enhance the current tools to track and measure the required data
** QA does repeated, scheduled, well-defined load tests in Stage while actively monitoring the results, logs, data, etc.
** QA finds a stable configuration that - when scaled - would
*** match the needs of Product when we release
*** match the realisitc capacity planning that OPs normally does

* Dependencies
** Realistic traffic/user numbers from the FxA Product team
** Timely training on monitoring tools from the OPs team
** Regular and realistic scaling/testing of deployments to Stage by QA given our current pre-release and post-release schedules

== References ==
* Repository: https://github.com/mozilla/fxa-auth-server
* The QA Test Environments:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
* Deploying the FxA Load Test environment for broker/agents usage:
** https://github.com/mozilla/fxa-deployment
* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/TSVerifierSyncTestEnvironments

2014-09-18T02:48:02Z

Jbonacci: /* Monitoring the Stage Environment */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier Stage section below...
** A Tokenserver+Verifier stack: See the TokenServer Stage section below...

== Tokenserver, Verifier, and Sync Production Environments ==
* TokenServer: https://token.services.mozilla.com
* Verifier: https://verifier.accounts.firefox.com
* Sync 1.5 Nodes (for now in AWS):
** sync-1-us-east-2.sync.services.mozilla.com
** sync-2-us-east-2.sync.services.mozilla.com
*** There are many more than this now...

* QA Access: not allowed

* Pointing Fx Nightly to Production FxA/Sync:
** Set/Verify the following Firefox configs:
*** services.sync.clusterURL = (should get automatically set by the TokenServer)
*** services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
*** services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
*** services.sync.log.appender.file.logOnError = Yes
*** services.sync.log.appender.file.logOnSuccess = Yes
*** services.sync.log.appender.file.level = Trace
*** identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
*** identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...

== TokenServer+Verifier Stage Environment ==
* NOTE: In an upcoming deployent, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** TokenServer: https://token.stage.mozaws.net
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
TokenServer:
rpm -qa | grep -E 'token|browser'

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** tokenserver app server (1 or more instances behind a CF stack and ELB)
** tokenserver db (1 large DB instance behind RDS)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/tokenserver
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /data/fxa-browserid-verifier
** /etc/heka.d
** /etc/puppet
*** esp. /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa_verifier.stage.yaml
*** /etc/puppet/yaml/app/tokenserver.stage.yaml
*** /etc/puppet/yaml/app/tokenserver.yaml

* Processes
** tokenserver app server:
*** Search for token, circus, nginx, gunicorn, python, hekad, node

* Logs
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* Hekad
** /etc/puppet/modules/hekad

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

== Verifier Stage Environment ==
* NOTE: In an upcoming deployment, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
** rpm -qa | grep verifier

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** fxa-browserid_verifier or fxa-bv-stage (usually a single instance)

* Files
** /data/fxa-browserid-verifier
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/heka.d
** /etc/puppet
*** esp. the /etc/puppet/yaml/app files
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa.yaml

* Processes
** Search for node, heka, nginx, circus

* Logs
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* Hekad
** /etc/puppet/modules/hekad
** /data/hekad

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

* Quick verifications
Browser: https://verifier.stage.mozaws.net/
returns "Method Not Allowed"

$ curl https://verifier.stage.mozaws.net
returns "Method Not Allowed"

$ curl -I https://verifier.stage.mozaws.net
HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache, max-age=0
Content-length: 18
Content-Type: text/plain
Date: Wed, 23 Jul 2014 20:19:40 GMT
Connection: keep-alive

== Sync 1.5 Stage Environment ==
* This is a work in progress. Right now, there are only a small number of sync nodes (instances) in AWS.
* URLs
** https://sync-1-us-east-1.stage.mozaws.net
** https://sync-2-us-east-1.stage.mozaws.net
** https://sync-3-us-east-1.stage.mozaws.net
** https://sync-4-us-east-1.stage.mozaws.net
** and sometimes
** https://sync-5-us-east-1.stage.mozaws.net
** https://sync-6-us-east-1.stage.mozaws.net

* Versions
Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** Search for sync node instances in US East: "stage-sync-node-X"
** Each node is a specific large instance (mixed m3 and c3)

* Files
** /data/server-syncstorage/*
** /var/log/nginx
** /etc/puppet

* Processes
** Search for sync, mysql, circusd, hekad, nginx, memcached

* Logs
** /media/ephemeral0/logs/
** /media/ephemeral0/logs/nginx/access.log
** /media/ephemeral0/logs/nginx/error.log
** /media/ephemeral0/logs/sync/sync.err
** /media/ephemeral0/logs/sync/sync.log
** /var/log/circus.log
** /var/log/hekad/sync_1_5.stderr.log
** /var/log/hekad/sync_1_5.stdout.log

* Hekad
** /etc/heka.d/sync_1_5.toml

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer

* Note: There is no longer a Sync 1.1 Stage environment.

== Loads V1 Services Cluster Environment ==
* Details on the Loads V1 cluster are available at this link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Monitoring the Stage Environment ==
* Stackdriver:
** Stage main: https://app.stackdriver.com
** https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** https://app.stackdriver.com/groups/4389/stage-services-tag-sync15/stage-services-sync15/fxa-verifier
** https://app.stackdriver.com/groups/4391/stage-services-tag-sync15/stage-services-tag-sync15/sync-storage
** https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
** https://app.stackdriver.com/monitoring/3827/tokenserver
** https://app.stackdriver.com/instances/<AWS instance>
** Also, make use of the Services, Groups, and Dashboards for more specific links
*** Example: Services > https://app.stackdriver.com/services/nginx
*** Example: Groups > https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver

* Graphite:
** https://graphite.shared.us-east-1.stage.mozaws.net

* Kibana
** https://kibana.shared.us-east-1.stage.mozaws.net/
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_errors.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_app_logs.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/tokenserver_http_status.json

* Heka
** https://heka.shared.us-east-1.stage.mozaws.net/
** or https://heka.shared.us-east-1.stage.mozaws.net/#health
** Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/TokenServerHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-SlowQueryFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerLocalVerifierFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-FileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerNginxFileOutput
** Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-FileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-SlowQueryFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerLocalVerifierFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxErrorFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagemetaglobal.cbuf

== Monitoring the Production Environment ==
* Heka shared:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/TokenServerHTTPStatus

* Kibana shared:
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/file/default.json

* StackDriver:
** Main: https://app.stackdriver.com/

== TokenServer, Verifier, and Sync 1.5 Dev Environments ==
* URLs
** https://sync1.dev.lcip.org/
** https://token.dev.lcip.org/

* Versions
TokenServer:
rpm -qa | grep token
Example: tokenserver-svcops <latest version>

Verifier:
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>

Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** This is part of the dev-lcip-org CloudFormation stack
*** TokenServer: Search for the fxa-tokenserver instance
*** SyncServer2: Search for the fxa-syncstorage instance
*** SyncServer1: Search for the fxa-syncstorage instance
*** LogServer: Search for the fxa-logbox instance
* Files
** TBD
* Processes
** TBD
* Logs
** TBD
* QA Access
** SSH with AWS keys to the various instances
* Firefox configs:
** services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
** services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/sync
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/token

TestEngineering/Services/TSVerifierSyncTestEnvironments

2014-09-18T02:47:30Z

Jbonacci: /* Monitoring the Stage Environment */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier Stage section below...
** A Tokenserver+Verifier stack: See the TokenServer Stage section below...

== Tokenserver, Verifier, and Sync Production Environments ==
* TokenServer: https://token.services.mozilla.com
* Verifier: https://verifier.accounts.firefox.com
* Sync 1.5 Nodes (for now in AWS):
** sync-1-us-east-2.sync.services.mozilla.com
** sync-2-us-east-2.sync.services.mozilla.com
*** There are many more than this now...

* QA Access: not allowed

* Pointing Fx Nightly to Production FxA/Sync:
** Set/Verify the following Firefox configs:
*** services.sync.clusterURL = (should get automatically set by the TokenServer)
*** services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
*** services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
*** services.sync.log.appender.file.logOnError = Yes
*** services.sync.log.appender.file.logOnSuccess = Yes
*** services.sync.log.appender.file.level = Trace
*** identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
*** identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...

== TokenServer+Verifier Stage Environment ==
* NOTE: In an upcoming deployent, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** TokenServer: https://token.stage.mozaws.net
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
TokenServer:
rpm -qa | grep -E 'token|browser'

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** tokenserver app server (1 or more instances behind a CF stack and ELB)
** tokenserver db (1 large DB instance behind RDS)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/tokenserver
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /data/fxa-browserid-verifier
** /etc/heka.d
** /etc/puppet
*** esp. /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa_verifier.stage.yaml
*** /etc/puppet/yaml/app/tokenserver.stage.yaml
*** /etc/puppet/yaml/app/tokenserver.yaml

* Processes
** tokenserver app server:
*** Search for token, circus, nginx, gunicorn, python, hekad, node

* Logs
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* Hekad
** /etc/puppet/modules/hekad

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

== Verifier Stage Environment ==
* NOTE: In an upcoming deployment, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
** rpm -qa | grep verifier

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** fxa-browserid_verifier or fxa-bv-stage (usually a single instance)

* Files
** /data/fxa-browserid-verifier
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/heka.d
** /etc/puppet
*** esp. the /etc/puppet/yaml/app files
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa.yaml

* Processes
** Search for node, heka, nginx, circus

* Logs
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* Hekad
** /etc/puppet/modules/hekad
** /data/hekad

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

* Quick verifications
Browser: https://verifier.stage.mozaws.net/
returns "Method Not Allowed"

$ curl https://verifier.stage.mozaws.net
returns "Method Not Allowed"

$ curl -I https://verifier.stage.mozaws.net
HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache, max-age=0
Content-length: 18
Content-Type: text/plain
Date: Wed, 23 Jul 2014 20:19:40 GMT
Connection: keep-alive

== Sync 1.5 Stage Environment ==
* This is a work in progress. Right now, there are only a small number of sync nodes (instances) in AWS.
* URLs
** https://sync-1-us-east-1.stage.mozaws.net
** https://sync-2-us-east-1.stage.mozaws.net
** https://sync-3-us-east-1.stage.mozaws.net
** https://sync-4-us-east-1.stage.mozaws.net
** and sometimes
** https://sync-5-us-east-1.stage.mozaws.net
** https://sync-6-us-east-1.stage.mozaws.net

* Versions
Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** Search for sync node instances in US East: "stage-sync-node-X"
** Each node is a specific large instance (mixed m3 and c3)

* Files
** /data/server-syncstorage/*
** /var/log/nginx
** /etc/puppet

* Processes
** Search for sync, mysql, circusd, hekad, nginx, memcached

* Logs
** /media/ephemeral0/logs/
** /media/ephemeral0/logs/nginx/access.log
** /media/ephemeral0/logs/nginx/error.log
** /media/ephemeral0/logs/sync/sync.err
** /media/ephemeral0/logs/sync/sync.log
** /var/log/circus.log
** /var/log/hekad/sync_1_5.stderr.log
** /var/log/hekad/sync_1_5.stdout.log

* Hekad
** /etc/heka.d/sync_1_5.toml

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer

* Note: There is no longer a Sync 1.1 Stage environment.

== Loads V1 Services Cluster Environment ==
* Details on the Loads V1 cluster are available at this link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Monitoring the Stage Environment ==
* Stackdriver:
** Stage main: https://app.stackdriver.com
** https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** https://app.stackdriver.com/groups/4389/stage-services-tag-sync15/stage-services-sync15/fxa-verifier
** https://app.stackdriver.com/groups/4391/stage-services-tag-sync15/stage-services-tag-sync15/sync-storage
** https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
** https://app.stackdriver.com/monitoring/3827/tokenserver
** https://app.stackdriver.com/instances/<AWS instance>
** Also, make use of the Services, Groups, and Dashboards for more specific links
*** Example: Services > https://app.stackdriver.com/services/nginx
*** Example: Groups > https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver

* Graphite:
** https://graphite.shared.us-east-1.stage.mozaws.net

* Kibana
** https://kibana.shared.us-east-1.stage.mozaws.net/
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_errors.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_app_logs.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/tokenserver_http_status.json

* Heka
** https://heka.shared.us-east-1.stage.mozaws.net/
** or https://heka.shared.us-east-1.stage.mozaws.net/#health
** Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/TokenServerHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-SlowQueryFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerLocalVerifierFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-FileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerNginxFileOutput
** Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-FileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-SlowQueryFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerLocalVerifierFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxErrorFileOutput-ProtobufEncoder
** Sandboxes
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagemetaglobal.cbuf

== Monitoring the Production Environment ==
* Heka shared:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/TokenServerHTTPStatus

* Kibana shared:
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/file/default.json

* StackDriver:
** Main: https://app.stackdriver.com/

== TokenServer, Verifier, and Sync 1.5 Dev Environments ==
* URLs
** https://sync1.dev.lcip.org/
** https://token.dev.lcip.org/

* Versions
TokenServer:
rpm -qa | grep token
Example: tokenserver-svcops <latest version>

Verifier:
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>

Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** This is part of the dev-lcip-org CloudFormation stack
*** TokenServer: Search for the fxa-tokenserver instance
*** SyncServer2: Search for the fxa-syncstorage instance
*** SyncServer1: Search for the fxa-syncstorage instance
*** LogServer: Search for the fxa-logbox instance
* Files
** TBD
* Processes
** TBD
* Logs
** TBD
* QA Access
** SSH with AWS keys to the various instances
* Firefox configs:
** services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
** services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/sync
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/token

TestEngineering/Services/TSVerifierSyncTestEnvironments

2014-09-18T02:42:07Z

Jbonacci: /* Monitoring the Stage Environment */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier Stage section below...
** A Tokenserver+Verifier stack: See the TokenServer Stage section below...

== Tokenserver, Verifier, and Sync Production Environments ==
* TokenServer: https://token.services.mozilla.com
* Verifier: https://verifier.accounts.firefox.com
* Sync 1.5 Nodes (for now in AWS):
** sync-1-us-east-2.sync.services.mozilla.com
** sync-2-us-east-2.sync.services.mozilla.com
*** There are many more than this now...

* QA Access: not allowed

* Pointing Fx Nightly to Production FxA/Sync:
** Set/Verify the following Firefox configs:
*** services.sync.clusterURL = (should get automatically set by the TokenServer)
*** services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
*** services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
*** services.sync.log.appender.file.logOnError = Yes
*** services.sync.log.appender.file.logOnSuccess = Yes
*** services.sync.log.appender.file.level = Trace
*** identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
*** identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...

== TokenServer+Verifier Stage Environment ==
* NOTE: In an upcoming deployent, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** TokenServer: https://token.stage.mozaws.net
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
TokenServer:
rpm -qa | grep -E 'token|browser'

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** tokenserver app server (1 or more instances behind a CF stack and ELB)
** tokenserver db (1 large DB instance behind RDS)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/tokenserver
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /data/fxa-browserid-verifier
** /etc/heka.d
** /etc/puppet
*** esp. /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa_verifier.stage.yaml
*** /etc/puppet/yaml/app/tokenserver.stage.yaml
*** /etc/puppet/yaml/app/tokenserver.yaml

* Processes
** tokenserver app server:
*** Search for token, circus, nginx, gunicorn, python, hekad, node

* Logs
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* Hekad
** /etc/puppet/modules/hekad

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

== Verifier Stage Environment ==
* NOTE: In an upcoming deployment, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
** rpm -qa | grep verifier

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** fxa-browserid_verifier or fxa-bv-stage (usually a single instance)

* Files
** /data/fxa-browserid-verifier
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/heka.d
** /etc/puppet
*** esp. the /etc/puppet/yaml/app files
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa.yaml

* Processes
** Search for node, heka, nginx, circus

* Logs
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* Hekad
** /etc/puppet/modules/hekad
** /data/hekad

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

* Quick verifications
Browser: https://verifier.stage.mozaws.net/
returns "Method Not Allowed"

$ curl https://verifier.stage.mozaws.net
returns "Method Not Allowed"

$ curl -I https://verifier.stage.mozaws.net
HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache, max-age=0
Content-length: 18
Content-Type: text/plain
Date: Wed, 23 Jul 2014 20:19:40 GMT
Connection: keep-alive

== Sync 1.5 Stage Environment ==
* This is a work in progress. Right now, there are only a small number of sync nodes (instances) in AWS.
* URLs
** https://sync-1-us-east-1.stage.mozaws.net
** https://sync-2-us-east-1.stage.mozaws.net
** https://sync-3-us-east-1.stage.mozaws.net
** https://sync-4-us-east-1.stage.mozaws.net
** and sometimes
** https://sync-5-us-east-1.stage.mozaws.net
** https://sync-6-us-east-1.stage.mozaws.net

* Versions
Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** Search for sync node instances in US East: "stage-sync-node-X"
** Each node is a specific large instance (mixed m3 and c3)

* Files
** /data/server-syncstorage/*
** /var/log/nginx
** /etc/puppet

* Processes
** Search for sync, mysql, circusd, hekad, nginx, memcached

* Logs
** /media/ephemeral0/logs/
** /media/ephemeral0/logs/nginx/access.log
** /media/ephemeral0/logs/nginx/error.log
** /media/ephemeral0/logs/sync/sync.err
** /media/ephemeral0/logs/sync/sync.log
** /var/log/circus.log
** /var/log/hekad/sync_1_5.stderr.log
** /var/log/hekad/sync_1_5.stdout.log

* Hekad
** /etc/heka.d/sync_1_5.toml

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer

* Note: There is no longer a Sync 1.1 Stage environment.

== Loads V1 Services Cluster Environment ==
* Details on the Loads V1 cluster are available at this link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Monitoring the Stage Environment ==
* Stackdriver:
** Stage main: https://app.stackdriver.com
** https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** https://app.stackdriver.com/groups/4389/stage-services-tag-sync15/stage-services-sync15/fxa-verifier
** https://app.stackdriver.com/groups/4391/stage-services-tag-sync15/stage-services-tag-sync15/sync-storage
** https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
** https://app.stackdriver.com/monitoring/3827/tokenserver
** https://app.stackdriver.com/instances/<AWS instance>
** Also, make use of the Services, Groups, and Dashboards for more specific links
*** Example: Services > https://app.stackdriver.com/services/nginx
*** Example: Groups > https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver

* Graphite:
** https://graphite.shared.us-east-1.stage.mozaws.net

* Kibana
** https://kibana.shared.us-east-1.stage.mozaws.net/
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_errors.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_app_logs.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/tokenserver_http_status.json

* Heka
** https://heka.shared.us-east-1.stage.mozaws.net/
** or https://heka.shared.us-east-1.stage.mozaws.net/#health
** Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/TokenServerHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-SlowQueryFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerLocalVerifierFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-FileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerNginxFileOutput
** Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-FileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-SlowQueryFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerLocalVerifierFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxErrorFileOutput-ProtobufEncoder
** Sandboxes
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-HTTPStatus/outputs/Sync-1_5-HTTPStatus.HTTPStatus.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagehistory.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagebookmarks.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storageforms.cbuf

== Monitoring the Production Environment ==
* Heka shared:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/TokenServerHTTPStatus

* Kibana shared:
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/file/default.json

* StackDriver:
** Main: https://app.stackdriver.com/

== TokenServer, Verifier, and Sync 1.5 Dev Environments ==
* URLs
** https://sync1.dev.lcip.org/
** https://token.dev.lcip.org/

* Versions
TokenServer:
rpm -qa | grep token
Example: tokenserver-svcops <latest version>

Verifier:
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>

Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** This is part of the dev-lcip-org CloudFormation stack
*** TokenServer: Search for the fxa-tokenserver instance
*** SyncServer2: Search for the fxa-syncstorage instance
*** SyncServer1: Search for the fxa-syncstorage instance
*** LogServer: Search for the fxa-logbox instance
* Files
** TBD
* Processes
** TBD
* Logs
** TBD
* QA Access
** SSH with AWS keys to the various instances
* Firefox configs:
** services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
** services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/sync
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/token

TestEngineering/Services/TSVerifierSyncTestEnvironments

2014-09-18T02:39:45Z

Jbonacci: /* Monitoring the Stage Environment */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier Stage section below...
** A Tokenserver+Verifier stack: See the TokenServer Stage section below...

== Tokenserver, Verifier, and Sync Production Environments ==
* TokenServer: https://token.services.mozilla.com
* Verifier: https://verifier.accounts.firefox.com
* Sync 1.5 Nodes (for now in AWS):
** sync-1-us-east-2.sync.services.mozilla.com
** sync-2-us-east-2.sync.services.mozilla.com
*** There are many more than this now...

* QA Access: not allowed

* Pointing Fx Nightly to Production FxA/Sync:
** Set/Verify the following Firefox configs:
*** services.sync.clusterURL = (should get automatically set by the TokenServer)
*** services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
*** services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
*** services.sync.log.appender.file.logOnError = Yes
*** services.sync.log.appender.file.logOnSuccess = Yes
*** services.sync.log.appender.file.level = Trace
*** identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
*** identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...

== TokenServer+Verifier Stage Environment ==
* NOTE: In an upcoming deployent, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** TokenServer: https://token.stage.mozaws.net
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
TokenServer:
rpm -qa | grep -E 'token|browser'

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** tokenserver app server (1 or more instances behind a CF stack and ELB)
** tokenserver db (1 large DB instance behind RDS)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/tokenserver
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /data/fxa-browserid-verifier
** /etc/heka.d
** /etc/puppet
*** esp. /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa_verifier.stage.yaml
*** /etc/puppet/yaml/app/tokenserver.stage.yaml
*** /etc/puppet/yaml/app/tokenserver.yaml

* Processes
** tokenserver app server:
*** Search for token, circus, nginx, gunicorn, python, hekad, node

* Logs
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* Hekad
** /etc/puppet/modules/hekad

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

== Verifier Stage Environment ==
* NOTE: In an upcoming deployment, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
** rpm -qa | grep verifier

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** fxa-browserid_verifier or fxa-bv-stage (usually a single instance)

* Files
** /data/fxa-browserid-verifier
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/heka.d
** /etc/puppet
*** esp. the /etc/puppet/yaml/app files
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa.yaml

* Processes
** Search for node, heka, nginx, circus

* Logs
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* Hekad
** /etc/puppet/modules/hekad
** /data/hekad

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

* Quick verifications
Browser: https://verifier.stage.mozaws.net/
returns "Method Not Allowed"

$ curl https://verifier.stage.mozaws.net
returns "Method Not Allowed"

$ curl -I https://verifier.stage.mozaws.net
HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache, max-age=0
Content-length: 18
Content-Type: text/plain
Date: Wed, 23 Jul 2014 20:19:40 GMT
Connection: keep-alive

== Sync 1.5 Stage Environment ==
* This is a work in progress. Right now, there are only a small number of sync nodes (instances) in AWS.
* URLs
** https://sync-1-us-east-1.stage.mozaws.net
** https://sync-2-us-east-1.stage.mozaws.net
** https://sync-3-us-east-1.stage.mozaws.net
** https://sync-4-us-east-1.stage.mozaws.net
** and sometimes
** https://sync-5-us-east-1.stage.mozaws.net
** https://sync-6-us-east-1.stage.mozaws.net

* Versions
Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** Search for sync node instances in US East: "stage-sync-node-X"
** Each node is a specific large instance (mixed m3 and c3)

* Files
** /data/server-syncstorage/*
** /var/log/nginx
** /etc/puppet

* Processes
** Search for sync, mysql, circusd, hekad, nginx, memcached

* Logs
** /media/ephemeral0/logs/
** /media/ephemeral0/logs/nginx/access.log
** /media/ephemeral0/logs/nginx/error.log
** /media/ephemeral0/logs/sync/sync.err
** /media/ephemeral0/logs/sync/sync.log
** /var/log/circus.log
** /var/log/hekad/sync_1_5.stderr.log
** /var/log/hekad/sync_1_5.stdout.log

* Hekad
** /etc/heka.d/sync_1_5.toml

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer

* Note: There is no longer a Sync 1.1 Stage environment.

== Loads V1 Services Cluster Environment ==
* Details on the Loads V1 cluster are available at this link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Monitoring the Stage Environment ==
* Stackdriver:
** Stage main: https://app.stackdriver.com
** https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** https://app.stackdriver.com/groups/4389/stage-services-tag-sync15/stage-services-sync15/fxa-verifier
** https://app.stackdriver.com/groups/4391/stage-services-tag-sync15/stage-services-tag-sync15/sync-storage
** https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
** https://app.stackdriver.com/monitoring/3827/tokenserver
** https://app.stackdriver.com/instances/<AWS instance>
** Also, make use of the Services, Groups, and Dashboards for more specific links
*** Example: Services > https://app.stackdriver.com/services/nginx
*** Example: Groups > https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver

* Graphite:
** https://graphite.shared.us-east-1.stage.mozaws.net

* Kibana
** https://kibana.shared.us-east-1.stage.mozaws.net/
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_errors.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_app_logs.json

** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/tokenserver_http_status.json

* Heka
** https://heka.shared.us-east-1.stage.mozaws.net/
** or https://heka.shared.us-east-1.stage.mozaws.net/#health
** Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/TokenServerHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator

** Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-SlowQueryFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerLocalVerifierFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-FileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerNginxFileOutput

** Encoders

** Sandboxes
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-HTTPStatus/outputs/Sync-1_5-HTTPStatus.HTTPStatus.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagehistory.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagebookmarks.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storageforms.cbuf

== Monitoring the Production Environment ==
* Heka shared:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/TokenServerHTTPStatus

* Kibana shared:
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/file/default.json

* StackDriver:
** Main: https://app.stackdriver.com/

== TokenServer, Verifier, and Sync 1.5 Dev Environments ==
* URLs
** https://sync1.dev.lcip.org/
** https://token.dev.lcip.org/

* Versions
TokenServer:
rpm -qa | grep token
Example: tokenserver-svcops <latest version>

Verifier:
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>

Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** This is part of the dev-lcip-org CloudFormation stack
*** TokenServer: Search for the fxa-tokenserver instance
*** SyncServer2: Search for the fxa-syncstorage instance
*** SyncServer1: Search for the fxa-syncstorage instance
*** LogServer: Search for the fxa-logbox instance
* Files
** TBD
* Processes
** TBD
* Logs
** TBD
* QA Access
** SSH with AWS keys to the various instances
* Firefox configs:
** services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
** services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/sync
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/token

TestEngineering/Services/TSVerifierSyncTestEnvironments

2014-09-18T02:33:17Z

Jbonacci: /* Monitoring the Stage Environment */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier Stage section below...
** A Tokenserver+Verifier stack: See the TokenServer Stage section below...

== Tokenserver, Verifier, and Sync Production Environments ==
* TokenServer: https://token.services.mozilla.com
* Verifier: https://verifier.accounts.firefox.com
* Sync 1.5 Nodes (for now in AWS):
** sync-1-us-east-2.sync.services.mozilla.com
** sync-2-us-east-2.sync.services.mozilla.com
*** There are many more than this now...

* QA Access: not allowed

* Pointing Fx Nightly to Production FxA/Sync:
** Set/Verify the following Firefox configs:
*** services.sync.clusterURL = (should get automatically set by the TokenServer)
*** services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
*** services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
*** services.sync.log.appender.file.logOnError = Yes
*** services.sync.log.appender.file.logOnSuccess = Yes
*** services.sync.log.appender.file.level = Trace
*** identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
*** identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...

== TokenServer+Verifier Stage Environment ==
* NOTE: In an upcoming deployent, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** TokenServer: https://token.stage.mozaws.net
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
TokenServer:
rpm -qa | grep -E 'token|browser'

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** tokenserver app server (1 or more instances behind a CF stack and ELB)
** tokenserver db (1 large DB instance behind RDS)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/tokenserver
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /data/fxa-browserid-verifier
** /etc/heka.d
** /etc/puppet
*** esp. /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa_verifier.stage.yaml
*** /etc/puppet/yaml/app/tokenserver.stage.yaml
*** /etc/puppet/yaml/app/tokenserver.yaml

* Processes
** tokenserver app server:
*** Search for token, circus, nginx, gunicorn, python, hekad, node

* Logs
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* Hekad
** /etc/puppet/modules/hekad

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

== Verifier Stage Environment ==
* NOTE: In an upcoming deployment, the Verifier functionality will be rolled into the Tokenserver stack of services.

* URLs
** Verifier: https://verifier.stage.mozaws.net
** IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com

* Version:
** rpm -qa | grep verifier

* AWS
** Shared:
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-elasticsearch
*** shared-rabbitmq
*** shared-rabbitmq
*** shared-bastion
*** shared-heka

** fxa-browserid_verifier or fxa-bv-stage (usually a single instance)

* Files
** /data/fxa-browserid-verifier
*** esp. /data/fxa-browserid-verifier/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/heka.d
** /etc/puppet
*** esp. the /etc/puppet/yaml/app files
*** /etc/puppet/yaml/app/fxa.stage.yaml
*** /etc/puppet/yaml/app/fxa.yaml

* Processes
** Search for node, heka, nginx, circus

* Logs
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* Hekad
** /etc/puppet/modules/hekad
** /data/hekad

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer
** services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5

* Quick verifications
Browser: https://verifier.stage.mozaws.net/
returns "Method Not Allowed"

$ curl https://verifier.stage.mozaws.net
returns "Method Not Allowed"

$ curl -I https://verifier.stage.mozaws.net
HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache, max-age=0
Content-length: 18
Content-Type: text/plain
Date: Wed, 23 Jul 2014 20:19:40 GMT
Connection: keep-alive

== Sync 1.5 Stage Environment ==
* This is a work in progress. Right now, there are only a small number of sync nodes (instances) in AWS.
* URLs
** https://sync-1-us-east-1.stage.mozaws.net
** https://sync-2-us-east-1.stage.mozaws.net
** https://sync-3-us-east-1.stage.mozaws.net
** https://sync-4-us-east-1.stage.mozaws.net
** and sometimes
** https://sync-5-us-east-1.stage.mozaws.net
** https://sync-6-us-east-1.stage.mozaws.net

* Versions
Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** Search for sync node instances in US East: "stage-sync-node-X"
** Each node is a specific large instance (mixed m3 and c3)

* Files
** /data/server-syncstorage/*
** /var/log/nginx
** /etc/puppet

* Processes
** Search for sync, mysql, circusd, hekad, nginx, memcached

* Logs
** /media/ephemeral0/logs/
** /media/ephemeral0/logs/nginx/access.log
** /media/ephemeral0/logs/nginx/error.log
** /media/ephemeral0/logs/sync/sync.err
** /media/ephemeral0/logs/sync/sync.log
** /var/log/circus.log
** /var/log/hekad/sync_1_5.stderr.log
** /var/log/hekad/sync_1_5.stdout.log

* Hekad
** /etc/heka.d/sync_1_5.toml

* QA Access via a Bastion Host
** SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

* Firefox Configs
** services.sync.clusterURL should get automatically set by the TokenServer

* Note: There is no longer a Sync 1.1 Stage environment.

== Loads V1 Services Cluster Environment ==
* Details on the Loads V1 cluster are available at this link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Monitoring the Stage Environment ==
* Stackdriver:
** Stage main: https://app.stackdriver.com
** https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** https://app.stackdriver.com/groups/4389/stage-services-tag-sync15/stage-services-sync15/fxa-verifier
** https://app.stackdriver.com/groups/4391/stage-services-tag-sync15/stage-services-tag-sync15/sync-storage
** https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
** https://app.stackdriver.com/monitoring/3827/tokenserver
** https://app.stackdriver.com/instances/<AWS instance>
** Also, make use of the Services, Groups, and Dashboards for more specific links
*** Example: Services > https://app.stackdriver.com/services/nginx
*** Example: Groups > https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver

* Graphite:
** https://graphite.shared.us-east-1.stage.mozaws.net

* Kibana
** https://kibana.shared.us-east-1.stage.mozaws.net/
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_errors.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_app_logs.json

** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/tokenserver_http_status.json

* Heka
** https://heka.shared.us-east-1.stage.mozaws.net/
** https://heka.shared.us-east-1.stage.mozaws.net/#health
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-HTTPStatus/outputs/Sync-1_5-HTTPStatus.HTTPStatus.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagehistory.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagebookmarks.cbuf
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storageforms.cbuf

== Monitoring the Production Environment ==
* Heka shared:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/TokenServerHTTPStatus

* Kibana shared:
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/file/default.json

* StackDriver:
** Main: https://app.stackdriver.com/

== TokenServer, Verifier, and Sync 1.5 Dev Environments ==
* URLs
** https://sync1.dev.lcip.org/
** https://token.dev.lcip.org/

* Versions
TokenServer:
rpm -qa | grep token
Example: tokenserver-svcops <latest version>

Verifier:
rpm -qa | grep verifier
Example: fxa-browserid-verifier-svcops <latest version>

Server-Syncstorage (sync node):
rpm -qa | grep syncstorage
Example: server-syncstorage <latest version>

* AWS
** This is part of the dev-lcip-org CloudFormation stack
*** TokenServer: Search for the fxa-tokenserver instance
*** SyncServer2: Search for the fxa-syncstorage instance
*** SyncServer1: Search for the fxa-syncstorage instance
*** LogServer: Search for the fxa-logbox instance
* Files
** TBD
* Processes
** TBD
* Logs
** TBD
* QA Access
** SSH with AWS keys to the various instances
* Firefox configs:
** services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
** services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/sync
** and https://github.com/mozilla-services/svcops/tree/master/cloudformations/token

TestEngineering/Services/LoopTestEnvironments

2014-09-17T22:30:49Z

Jbonacci: /* Loop Server Stage Details */

== Loop Server Production Environments ==
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com

* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment

* QA Access: not allowed

* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default

* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server

curl -I https://loop.services.mozilla.com

Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Loop web client.

Browser: https://call.mozilla.com/config.js
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};

Browser: Browser: https://call.mozilla.com/VERSION.txt

curl https://call.mozilla.com/
returns a page of HTML

curl -I https://call.mozilla.com/

MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}

curl -I https://msisdn.services.mozilla.com

== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD

** loop-redis-stage - need the details

** Loop-Client: https://call.stage.mozaws.net

** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net

** Sentry: https://sentry.shared.us-east-1.stage.mozaws.net (see below)

== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net

* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277

* VERSION.txt
** ./data/loop-server/VERSION.txt

* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"

* OS
** Running SciLinux 6.4

* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*

* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx

* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log

* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json

* Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details

* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml

== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net

* Versions
** rpm -qa | grep loop

* VERSION.txt
** /data/loop-client/content/VERSION.txt

* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"

* OS
** Scientific Linux release 6.4

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml/app/loop_client.dev.yaml
** /etc/puppet/yaml/app/loop_client.prod.yaml
** /etc/puppet/yaml/app/loop_client.stage.yaml
** /etc/puppet/yaml/app/loop_client.yaml
** /etc/puppet/yaml/app/loop_server.dev.yaml
** /etc/puppet/yaml/app/loop_server.prod.yaml
** /etc/puppet/yaml/app/loop_server.stage.yaml
** /etc/puppet/yaml/app/loop_server.yaml

* Processes
** look for stackdriver, nginx, python, circus, heka

* Logs
** /var/log/circus.log
** /var/log/hekad
** /media/ephemeral0/nginx/logs

* Heka
** /data/hekad
** /etc/heka.d

* Quick Verifications
In the browser: https://call.stage.mozaws.net

Check the version: https://call.stage.mozaws.net/VERSION.txt

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {serverUrl: 'https://https://loop.stage.mozaws.net'};

== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.

* URL
** Mock: https://loop-delayed-response.stage.mozaws.net

* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"

* OS
** Scientific Linux release 6.4

* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js

* Logs
** /var/log/circus.log

* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok

* Access
** Log in as "ec2-user"

== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net

* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723

* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance

* OS
** Scientific Linux release 6.4 (Carbon)

* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway

* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka

* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log

* Configuration File
** /data/msisdn-gateway/config/production.json

* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net

curl -I https://msisdn.stage.mozaws.net

== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send

* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net

*Access: SSH

* OS
** Amazon Linux AMI release 2014.03

* Versions
** TBD

* Processes
** Look for nginx, python, omxen

* Files
** /home/ec2-user/omxen
** /opt/aws

* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc

* Restarting
** TBD

* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net

== Sentry Server Stage Details ==
* URL
** https://sentry.shared.us-east-1.stage.mozaws.net

* Versions
** rpm -qa | grep sentry

* AWS
** TBD

* OS
** Scientific Linux release 6.3

* Files
** /opt/sentry

* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud

* Logs
** /var/log
** nothing in /media/ephemeral0

* Heka
** TBD

* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry

== QA Access via a Bastion Host ==
* SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team

* Stackdriver
** https://app.stackdriver.com/
** None at this time. See https://bugzilla.mozilla.org/show_bug.cgi?id=1063134

* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined

* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD

* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf

== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/

* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw

* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus

* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard

* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)

* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop

* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565

== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD

* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"

** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"

* OS
** TBD

* Files
** TBD

* Processes
** TBD

* Logs
** TBD

* Heka
** TBD

* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
"name": "mozilla-loop-server",
"description": "The Mozilla Loop (WebRTC App) server",
"version": "0.7.0",
"homepage": "https://github.com/mozilla-services/loop-server/",
"endpoint": "http://loop.dev.mozaws.net",
"fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true,
"provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
Loop
Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
"name": "mozilla-msisdn-gateway",
"description": "The Mozilla MSISDN Gateway",
"version": "0.3.0-DEV",
"homepage": "https://github.com/mozilla-services/msisdn-gateway/",
"endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
"storage": true
}
or similar

== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP

* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html

* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config

* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server

** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID

== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

TestEngineering/Services/TokenServerAndSyncLoadTesting

2014-09-17T00:41:59Z

Jbonacci: /* Known Bugs, Issues, and Tasks */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier sections below...
** A Tokenserver+Verifier stack: See the TokenServer sections below...

== Summary for Tokenserver, Verifier, and Sync 1.5 ==
* Latest Results
** Link to loads cluster: https://loads.services.mozilla.com/
*** Note: this now requires login privileges and a password
** Snapshots from StackDriver - TBD
** Snapshots from Kibana - TBD
* Latest Deployments
** TokenServer Stage: https://bugzilla.mozilla.org/show_bug.cgi?id=1014496
** TokenServer Prod: https://bugzilla.mozilla.org/show_bug.cgi?id=1027899
** Sync Server Stage: https://bugzilla.mozilla.org/show_bug.cgi?id=1026346
** Sync Server Prod: https://bugzilla.mozilla.org/show_bug.cgi?id=1026346
** Verifier Stage Deploy: ttps://bugzilla.mozilla.org/show_bug.cgi?id=1026644
** Verifier Prod Deploy: https://bugzilla.mozilla.org/show_bug.cgi?id=1027392
* In Progress
** Build out of Kibana dashboards
** Ongoing testing of Tokenserver, Verifier, and Sync releases
** Bug review and issue debug - there are a lot of issues to work on (see the long list near the bottom of the wiki)
* Bugs To Verify:
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=988095
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=1025767
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=1027444
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=1025735
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=735102
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=775395
* Planned
** Scaling for production traffic after release of Fx29
** Sync 1.5 migration work
** Operations readiness testing: See Bug 1006792
* Blockers
** none at this time
* Completed
** Pre-release load testing
** Previous load test results (short): http://loads.services.mozilla.com/
* Performance
** TBD

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.

* TokenServer+Verifier Stage environment:
From the browser: https://token.stage.mozaws.net
curl https://token.stage.mozaws.net
curl -I https://token.stage.mozaws.net

Use the simple "make test" command from an install of tokenserver on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://token.stage.mozaws.net

Alternate method:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Stage:
bin/sync-cli.js -e EMAIL -p PASSWORD --env stage -t COLLECTION
where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms

* Verifier Stage environment:
In the browser: https://verifier.stage.mozaws.net/
curl https://verifier.stage.mozaws.net
curl -I https://verifier.stage.mozaws.net

Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.stage.mozaws.net

* Sync Server Stage environment:
Install server-syncstorage to the local host or AWS instance (see below)
$ cd server-syncstorage
Quick test against the TokenServer
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server <Stage TokenServer>
Current example:
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server
https://token.stage.mozaws.net/1.0/sync/1.5
Quick tests against the Sync nodes
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py <Stage Sync Node>#<Node Secret>
Current examples:
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
Get the Node Secret information from OPs

* Using TPS
** The TPS FxA/Sync automated tests can be used as well, but the following file will have to be edited to add Stage environment configuration parameters: https://github.com/mozilla/gecko-dev/blob/master/testing/tps/tps/testrunner.py
** See the following wiki page for more information: https://wiki.mozilla.org/User_Services/Sync/Run_TPS
** See also: https://bugzilla.mozilla.org/show_bug.cgi?id=1006675

== Quick Verification Of Production Deployments ==
* This is a quick sanity test of the environment after a new deployment.

* Tokenserver+Verifier Production Environment
In the browser: https://token.services.mozilla.com
curl https://token.services.mozilla.com
curl -I https://token.services.mozilla.com
Then:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Production:
bin/sync-cli.js -e PROD-EMAIL -p PASSWORD -t COLLECTION
where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms

* Verifier Production Environment
In the browser: https://verifier.accounts.firefox.com
curl https://verifier.accounts.firefox.com
curl -I https://verifier.accounts.firefox.com
Then:
Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.accounts.firefox.com

* Sync Server Stage environment
Sign in with a known FxA account and sync data with a current Production account (sync node).
Create a new FxA account and set up sync.

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment

== Installing BrowserID-Verifier and the Loads tool on Localhost or AWS ==
* Installation:
$ git clone git://github.com/mozilla/browserid-verifier
$ cd browserid-verifier
$ npm install
$ npm test
$ cd loadtest
$ make build
Note: This should hit Stage by default: SERVER_URL=https://verifier.stage.mozaws.net

* Note: This will install a local copy of the Loads tool for use with the verifier.

== Running the load test against the Verifier in Stage ==
* Stage environment:
$ make test
or
$ make test SERVER_URL=https://verifier.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://verifier.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Verifier hits the Stage mockmyid server

* Production environment:
$ make test SERVER_URL=https://verifier.accounts.firefox.com
$ make bench SERVER_URL=https://verifier.accounts.firefox.com

== Using the Loads V1 Services Cluster for the Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://verifier.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=TBD

* Production environment:
$ make megabench SERVER_URL=https://verifier.accounts.firefox.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing TokenServer+Verifier and the Loads tool on Localhost or AWS ==
* Installation:
$ git clone https://github.com/mozilla-services/tokenserver
$ cd tokenserver
$ make build
$ make test
Note: This is for local testing only
$ cd loadtest
$ make build
Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com

* Note: This will install a local copy of the Loads tool for use with TokenServer+Verifier.

== Running the load test against TokenServer+Verifier in Stage ==
* Stage environment:
$ make test SERVER_URL=https://token.stage.mozaws.net
$ make bench SERVER_URL=https://token.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: This also hits the Stage Verifier, which in turns hits the Stage mockmyid server

* And while we are at it...
* Dev environment:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org

* Production environment:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com

== Using the Loads V1 Services Cluster for TokenServer+Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://token.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=https://token.dev.lcip.org

* Production environment:
$ make megabench SERVER_URL=https://token.services.mozilla.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest

== Installing Sync 1.5 and the Loads tool on Localhost or AWS ==
Installation:
$ git clone https://github.com/mozilla-services/server-syncstorage/
$ cd server-syncstorage
$ make build
$ make test
$ cd loadtest
$ make build

* Note: This will install a local copy of the Loads tool for use with Sync 1.5.

== Running the load test against Sync 1.5 in Stage ==
* Loads against specific Sync nodes in Stage
$ make test SERVER_URL=https://your.storagenode.here#SECRET
$ make bench SERVER_URL=https://your.storagenode.here#SECRET
Sync Stage nodes:
https://sync-1-us-east-1.stage.mozaws.net
https://sync-2-us-east-1.stage.mozaws.net
https://sync-3-us-east-1.stage.mozaws.net
...etc...

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.

== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://your.storagenode.here#SECRET

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest

== Running a combined load test against TokenServer+Verifier and Sync 1.5 in Stage ==
* A combined loads test against TokenServer and Sync 1.5 in Stage
* This is done via the server-syncstorage directory that was cloned and built above
$ cd server-syncstorage
$ cd loadtest
$ make test SERVER_URL=https://your.tokenserver.here
$ make bench SERVER_URL=https://your.tokenserver.here

Examples for Stage:
$ make test SERVER_URL=https://token.stage.mozaws.net
$ make bench SERVER_URL=https://token.stage.mozaws.net
See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer.2BVerifier_Stage_Environment

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Tokenserver hits the Stage Verifier, which, in turn, hits the mockmyid server.

* And while we are at it...
Dev environment:
Examples:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org

Prod environment:
Examples:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com

See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA.2C_TokenServer.2C_and_Sync_Production_Environments
and https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer_and_Sync_1.5_Dev_Environments

== Using the Loads V1 Services Cluster for a combined load test in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://token.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=https://token.dev.lcip.org

* Prod environment:
$ make megabench SERVER_URL=https://token.services.mozilla.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of hits
*** Number of concurrent users

** For make bench (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of concurrent users
*** Duration of test

** For make megabench (using the LoadsCluster with BrowserID-Verifier, TokenSerer, Sync, Combined):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)
*** SSH (the user account needed to SSH into the loads cluster - the default is ubuntu)

* Tokenserver load test code
** The Tokenserver load test can be configured - see the following lines:
** Basic Settings: https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** MockMyID: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L19-L36
** Percentages: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L39-L51

* Verifier load test code
** The Verifier load test can be configured - see the following lines:
** Various settings: https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py#L13-L53

* Sync Server load test code
** The Sync Server load test can be configured - see the following lines:
** Setting MockMyID: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L26-L45
** Setting test distributions: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L48-L83

* REFs:
** https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py
** https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py
** https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* TokenServer
** File location: tokenserver/loadtest/loadtest.py
** Inside NoteAssignmentTest, test_realistic is the main load test; the others are for specific behaviors
** The test runs as following:
95% ask for assertions on existing users (on a DB filled by test_single_token_exchange)
4% ask for assertion on a new use
1% ask for a bad assertion
** A bug has been filed to get the following additional coverage for the load test:
*** generation numbers in assertion
*** client state string
** A bug has been filed to get some integration tests written:
*** to cover the edge/error cases not in the load test
*** to be pointed at a remote server

* Sync
** File location: server-syncstorage/loadtest/stress.py
** This is the Sync 2.0 load test that has been back-ported for Sync 1.5.
** The stress.py file is fully configurable for the following:
*** client probability
*** client distribution
*** collections
** A bug has been filed to add support for load testing tabs
*** The tab collection it uses memcache; we need to figure out a way to test it without overloading the server
** There are currently no constants to define how to select percentages per collection type
** Right now, we need to manually configure the collections list in stress.py:
*** collections = ['bookmarks', 'forms', 'passwords', 'history', 'prefs']
*** Basically, you can add more entries of each type, since the load test (per user/again/hit/pass) picks randomly from the list for any given request...

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Tokenserver Custom Metrics
** addFailure

* Verifier Custom Metrics
** addFailure

* Sync Custom Metrics
** addFailure

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for TokenServer (per server)
** /media/ephemeral0/logs/
** /media/ephemeral0/nginx/logs/default.access.log
** /media/ephemeral0/nginx/logs/default.error.log
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* 2. Important logs for Verifier (per server)
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* 3. Important error logs for Sync (per Sync node)
** /media/ephemeral0/logs/
** /media/ephemeral0/nginx/access.log
** /media/ephemeral0/error.log
** /media/ephemeral0/sync/sync.err
** /media/ephemeral0/sync/sync.log

* Acceptable TokenServer errors:
1% - 2% failures (as the following)
token.log:
"name": "token.assertion.invalid_signature_error"
"name": "token.assertion.verify_failure"
nginx access.log:
401s
NOTE: Values can be tweaked here:
https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L58-L60

The following types of errors are known:
/media/ephemeral0/logs/tokenserver/token.error.log
Exception KeyError: KeyError(49564400,) in <module 'threading'...
/media/ephemeral0/logs/tokenserver/token.log
..."Starting new HTTP connection (9): 127.0.0.1", "hostname": ...
{"error": "StopIteration()", "traceback": "Uncaught exception:\n
File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
..."Connection pool is full, discarding connection: 127.0.0.1", "...
REF:
https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
https://bugzilla.mozilla.org/show_bug.cgi?id=1040397

OLD: Also, it may be the case that the following errors are "acceptable" if TS Stage is larger than Verifier Stage:
/media/ephemeral0/logs/tokenserver/token.error.log
Verifier-related errors of these types:
"HttpConnectionPool is full, discarding connection: verifier.stage.mozaws.net"
"Resetting dropped connection: verifier.stage.mozaws.net"
"Starting new HTTPS connection (179): verifier.stage.mozaws.net"

* Acceptable Verifier errors:
The verifier_out.log will show errors of the following types:
result: 'failure',\n reason: 'untrusted issuer...'
result: 'failure',\n reason: 'expired'
result: 'failure',\n reason: 'algorithms do not match'
result: 'failure',\n reason: 'audience mismatch: scheme mismatch'

* Acceptable Sync node errors:
In the nginx access.log files:
We will see some percentage of 404s. Right now we see the following:
14% 404s (compared to the total count of 200s)
with the config set up as follows:
users = 20
duration = 1800
agents = 5
Ideally, the overall percentage of 404s should drop the longer the load test.
Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
although they may show up in the logs after running the remote integration tests.

In /var/log/hekad/sync_1_5.stderr.log
You may see some Decoder 'Sync-1_5-SlowQuery-MySqlSlowQueryDecoder' error: Failed parsing
and a lot of BSO INSERTs

In /media/ephemeral0/logs/sync/sync.err
You should see expected skew and QueuePool messages and Deprecation warnings
Also, these are known
Exception SystemExit
Exception KeyError
This is probably https://bugzilla.mozilla.org/show_bug.cgi?id=1040397

== Monitoring TS and Sync Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Tokenserver:
** Repo: https://github.com/mozilla-services/tokenserver/issues
** Bugzilla: http://mzl.la/1s4qZKn

* BrowserID-Verifier:
** Repo: https://github.com/mozilla/browserid-verifier/issues
** Bugzilla: no specific cateogory

* Sync:
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Bugzilla: http://mzl.la/VUrYQ5

* OPs and Infrastructure
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-aws/issues
** https://github.com/mozilla-services/loads-web/issues

== References ==
* Other URLs
** http://restmail.net
** https://mockmyid.com
* Repositories
** https://github.com/mozilla-services/tokenserver
** https://github.com/mozilla-services/server-syncstorage
** https://github.com/mozilla/browserid-verifier
** https://github.com/mozilla/browserid-local-verify
** https://github.com/callahad/mockmyid
** https://github.com/mozilla/restmail.net
* Documentation
** https://docs.services.mozilla.com
** https://docs.services.mozilla.com/token/
** https://wiki.mozilla.org/Services/Sagrada/TokenServer
** https://docs.services.mozilla.com/sync/
* The QA Test Environments:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
* Deploying the FxA Load Test environment for broker/agents usage:
** https://github.com/mozilla/fxa-deployment
* Sync 1.5 protocol, documentation, etc.
** https://github.com/mozilla-services/docs
** https://docs.services.mozilla.com/#how-to
** https://docs.services.mozilla.com/howtos/run-fxa.html
** https://docs.services.mozilla.com/token/apis.html
** https://docs.services.mozilla.com/storage/apis-1.5.html
** https://docs.services.mozilla.com/howtos/run-sync-1.5.html
** https://docs.services.mozilla.com/howtos/run-sync-1.5.html
** https://github.com/mozilla-services/syncserver
* OPs pages for stats collection, logging, monitoring
** TBD

TestEngineering/Services/TokenServerAndSyncLoadTesting

2014-09-17T00:40:48Z

Jbonacci: /* Monitoring TS and Sync Stage */

* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
** The standalone Browser_ID Verifier stack: See that Verifier sections below...
** A Tokenserver+Verifier stack: See the TokenServer sections below...

== Summary for Tokenserver, Verifier, and Sync 1.5 ==
* Latest Results
** Link to loads cluster: https://loads.services.mozilla.com/
*** Note: this now requires login privileges and a password
** Snapshots from StackDriver - TBD
** Snapshots from Kibana - TBD
* Latest Deployments
** TokenServer Stage: https://bugzilla.mozilla.org/show_bug.cgi?id=1014496
** TokenServer Prod: https://bugzilla.mozilla.org/show_bug.cgi?id=1027899
** Sync Server Stage: https://bugzilla.mozilla.org/show_bug.cgi?id=1026346
** Sync Server Prod: https://bugzilla.mozilla.org/show_bug.cgi?id=1026346
** Verifier Stage Deploy: ttps://bugzilla.mozilla.org/show_bug.cgi?id=1026644
** Verifier Prod Deploy: https://bugzilla.mozilla.org/show_bug.cgi?id=1027392
* In Progress
** Build out of Kibana dashboards
** Ongoing testing of Tokenserver, Verifier, and Sync releases
** Bug review and issue debug - there are a lot of issues to work on (see the long list near the bottom of the wiki)
* Bugs To Verify:
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=988095
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=1025767
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=1027444
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=1025735
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=735102
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=775395
* Planned
** Scaling for production traffic after release of Fx29
** Sync 1.5 migration work
** Operations readiness testing: See Bug 1006792
* Blockers
** none at this time
* Completed
** Pre-release load testing
** Previous load test results (short): http://loads.services.mozilla.com/
* Performance
** TBD

== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.

* TokenServer+Verifier Stage environment:
From the browser: https://token.stage.mozaws.net
curl https://token.stage.mozaws.net
curl -I https://token.stage.mozaws.net

Use the simple "make test" command from an install of tokenserver on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://token.stage.mozaws.net

Alternate method:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Stage:
bin/sync-cli.js -e EMAIL -p PASSWORD --env stage -t COLLECTION
where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms

* Verifier Stage environment:
In the browser: https://verifier.stage.mozaws.net/
curl https://verifier.stage.mozaws.net
curl -I https://verifier.stage.mozaws.net

Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.stage.mozaws.net

* Sync Server Stage environment:
Install server-syncstorage to the local host or AWS instance (see below)
$ cd server-syncstorage
Quick test against the TokenServer
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server <Stage TokenServer>
Current example:
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server
https://token.stage.mozaws.net/1.0/sync/1.5
Quick tests against the Sync nodes
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py <Stage Sync Node>#<Node Secret>
Current examples:
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
$ ./local/bin/python ./syncstorage/tests/functional/test_storage.py
https://sync-1-us-east-1.stage.mozaws.net#<Node Secret>
Get the Node Secret information from OPs

* Using TPS
** The TPS FxA/Sync automated tests can be used as well, but the following file will have to be edited to add Stage environment configuration parameters: https://github.com/mozilla/gecko-dev/blob/master/testing/tps/tps/testrunner.py
** See the following wiki page for more information: https://wiki.mozilla.org/User_Services/Sync/Run_TPS
** See also: https://bugzilla.mozilla.org/show_bug.cgi?id=1006675

== Quick Verification Of Production Deployments ==
* This is a quick sanity test of the environment after a new deployment.

* Tokenserver+Verifier Production Environment
In the browser: https://token.services.mozilla.com
curl https://token.services.mozilla.com
curl -I https://token.services.mozilla.com
Then:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Production:
bin/sync-cli.js -e PROD-EMAIL -p PASSWORD -t COLLECTION
where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms

* Verifier Production Environment
In the browser: https://verifier.accounts.firefox.com
curl https://verifier.accounts.firefox.com
curl -I https://verifier.accounts.firefox.com
Then:
Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.accounts.firefox.com

* Sync Server Stage environment
Sign in with a known FxA account and sync data with a current Production account (sync node).
Create a new FxA account and set up sync.

== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment

== Installing BrowserID-Verifier and the Loads tool on Localhost or AWS ==
* Installation:
$ git clone git://github.com/mozilla/browserid-verifier
$ cd browserid-verifier
$ npm install
$ npm test
$ cd loadtest
$ make build
Note: This should hit Stage by default: SERVER_URL=https://verifier.stage.mozaws.net

* Note: This will install a local copy of the Loads tool for use with the verifier.

== Running the load test against the Verifier in Stage ==
* Stage environment:
$ make test
or
$ make test SERVER_URL=https://verifier.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://verifier.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Verifier hits the Stage mockmyid server

* Production environment:
$ make test SERVER_URL=https://verifier.accounts.firefox.com
$ make bench SERVER_URL=https://verifier.accounts.firefox.com

== Using the Loads V1 Services Cluster for the Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://verifier.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=TBD

* Production environment:
$ make megabench SERVER_URL=https://verifier.accounts.firefox.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest

== Installing TokenServer+Verifier and the Loads tool on Localhost or AWS ==
* Installation:
$ git clone https://github.com/mozilla-services/tokenserver
$ cd tokenserver
$ make build
$ make test
Note: This is for local testing only
$ cd loadtest
$ make build
Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com

* Note: This will install a local copy of the Loads tool for use with TokenServer+Verifier.

== Running the load test against TokenServer+Verifier in Stage ==
* Stage environment:
$ make test SERVER_URL=https://token.stage.mozaws.net
$ make bench SERVER_URL=https://token.stage.mozaws.net

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: This also hits the Stage Verifier, which in turns hits the Stage mockmyid server

* And while we are at it...
* Dev environment:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org

* Production environment:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com

== Using the Loads V1 Services Cluster for TokenServer+Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://token.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=https://token.dev.lcip.org

* Production environment:
$ make megabench SERVER_URL=https://token.services.mozilla.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest

== Installing Sync 1.5 and the Loads tool on Localhost or AWS ==
Installation:
$ git clone https://github.com/mozilla-services/server-syncstorage/
$ cd server-syncstorage
$ make build
$ make test
$ cd loadtest
$ make build

* Note: This will install a local copy of the Loads tool for use with Sync 1.5.

== Running the load test against Sync 1.5 in Stage ==
* Loads against specific Sync nodes in Stage
$ make test SERVER_URL=https://your.storagenode.here#SECRET
$ make bench SERVER_URL=https://your.storagenode.here#SECRET
Sync Stage nodes:
https://sync-1-us-east-1.stage.mozaws.net
https://sync-2-us-east-1.stage.mozaws.net
https://sync-3-us-east-1.stage.mozaws.net
...etc...

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...

NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.

== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://your.storagenode.here#SECRET

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest

== Running a combined load test against TokenServer+Verifier and Sync 1.5 in Stage ==
* A combined loads test against TokenServer and Sync 1.5 in Stage
* This is done via the server-syncstorage directory that was cloned and built above
$ cd server-syncstorage
$ cd loadtest
$ make test SERVER_URL=https://your.tokenserver.here
$ make bench SERVER_URL=https://your.tokenserver.here

Examples for Stage:
$ make test SERVER_URL=https://token.stage.mozaws.net
$ make bench SERVER_URL=https://token.stage.mozaws.net
See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer.2BVerifier_Stage_Environment

Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Tokenserver hits the Stage Verifier, which, in turn, hits the mockmyid server.

* And while we are at it...
Dev environment:
Examples:
$ make test SERVER_URL=https://token.dev.lcip.org
$ make bench SERVER_URL=https://token.dev.lcip.org

Prod environment:
Examples:
$ make test SERVER_URL=https://token.services.mozilla.com
$ make bench SERVER_URL=https://token.services.mozilla.com

See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA.2C_TokenServer.2C_and_Sync_Production_Environments
and https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer_and_Sync_1.5_Dev_Environments

== Using the Loads V1 Services Cluster for a combined load test in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://token.stage.mozaws.net

* Dev environment:
$ make megabench SERVER_URL=https://token.dev.lcip.org

* Prod environment:
$ make megabench SERVER_URL=https://token.services.mozilla.com

* REFs:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest

== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.

* Config files
** For make test (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of hits
*** Number of concurrent users

** For make bench (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of concurrent users
*** Duration of test

** For make megabench (using the LoadsCluster with BrowserID-Verifier, TokenSerer, Sync, Combined):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)
*** SSH (the user account needed to SSH into the loads cluster - the default is ubuntu)

* Tokenserver load test code
** The Tokenserver load test can be configured - see the following lines:
** Basic Settings: https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** MockMyID: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L19-L36
** Percentages: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L39-L51

* Verifier load test code
** The Verifier load test can be configured - see the following lines:
** Various settings: https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py#L13-L53

* Sync Server load test code
** The Sync Server load test can be configured - see the following lines:
** Setting MockMyID: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L26-L45
** Setting test distributions: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L48-L83

* REFs:
** https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py
** https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py
** https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py

== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations

* TokenServer
** File location: tokenserver/loadtest/loadtest.py
** Inside NoteAssignmentTest, test_realistic is the main load test; the others are for specific behaviors
** The test runs as following:
95% ask for assertions on existing users (on a DB filled by test_single_token_exchange)
4% ask for assertion on a new use
1% ask for a bad assertion
** A bug has been filed to get the following additional coverage for the load test:
*** generation numbers in assertion
*** client state string
** A bug has been filed to get some integration tests written:
*** to cover the edge/error cases not in the load test
*** to be pointed at a remote server

* Sync
** File location: server-syncstorage/loadtest/stress.py
** This is the Sync 2.0 load test that has been back-ported for Sync 1.5.
** The stress.py file is fully configurable for the following:
*** client probability
*** client distribution
*** collections
** A bug has been filed to add support for load testing tabs
*** The tab collection it uses memcache; we need to figure out a way to test it without overloading the server
** There are currently no constants to define how to select percentages per collection type
** Right now, we need to manually configure the collections list in stress.py:
*** collections = ['bookmarks', 'forms', 'passwords', 'history', 'prefs']
*** Basically, you can add more entries of each type, since the load test (per user/again/hit/pass) picks randomly from the list for any given request...

== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.

* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.

* Tokenserver Custom Metrics
** addFailure

* Verifier Custom Metrics
** addFailure

* Sync Custom Metrics
** addFailure

* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads

== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.

* 1. Important logs for TokenServer (per server)
** /media/ephemeral0/logs/
** /media/ephemeral0/nginx/logs/default.access.log
** /media/ephemeral0/nginx/logs/default.error.log
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log

* 2. Important logs for Verifier (per server)
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log

* 3. Important error logs for Sync (per Sync node)
** /media/ephemeral0/logs/
** /media/ephemeral0/nginx/access.log
** /media/ephemeral0/error.log
** /media/ephemeral0/sync/sync.err
** /media/ephemeral0/sync/sync.log

* Acceptable TokenServer errors:
1% - 2% failures (as the following)
token.log:
"name": "token.assertion.invalid_signature_error"
"name": "token.assertion.verify_failure"
nginx access.log:
401s
NOTE: Values can be tweaked here:
https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L58-L60

The following types of errors are known:
/media/ephemeral0/logs/tokenserver/token.error.log
Exception KeyError: KeyError(49564400,) in <module 'threading'...
/media/ephemeral0/logs/tokenserver/token.log
..."Starting new HTTP connection (9): 127.0.0.1", "hostname": ...
{"error": "StopIteration()", "traceback": "Uncaught exception:\n
File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
..."Connection pool is full, discarding connection: 127.0.0.1", "...
REF:
https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
https://bugzilla.mozilla.org/show_bug.cgi?id=1040397

OLD: Also, it may be the case that the following errors are "acceptable" if TS Stage is larger than Verifier Stage:
/media/ephemeral0/logs/tokenserver/token.error.log
Verifier-related errors of these types:
"HttpConnectionPool is full, discarding connection: verifier.stage.mozaws.net"
"Resetting dropped connection: verifier.stage.mozaws.net"
"Starting new HTTPS connection (179): verifier.stage.mozaws.net"

* Acceptable Verifier errors:
The verifier_out.log will show errors of the following types:
result: 'failure',\n reason: 'untrusted issuer...'
result: 'failure',\n reason: 'expired'
result: 'failure',\n reason: 'algorithms do not match'
result: 'failure',\n reason: 'audience mismatch: scheme mismatch'

* Acceptable Sync node errors:
In the nginx access.log files:
We will see some percentage of 404s. Right now we see the following:
14% 404s (compared to the total count of 200s)
with the config set up as follows:
users = 20
duration = 1800
agents = 5
Ideally, the overall percentage of 404s should drop the longer the load test.
Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
although they may show up in the logs after running the remote integration tests.

In /var/log/hekad/sync_1_5.stderr.log
You may see some Decoder 'Sync-1_5-SlowQuery-MySqlSlowQueryDecoder' error: Failed parsing
and a lot of BSO INSERTs

In /media/ephemeral0/logs/sync/sync.err
You should see expected skew and QueuePool messages and Deprecation warnings
Also, these are known
Exception SystemExit
Exception KeyError
This is probably https://bugzilla.mozilla.org/show_bug.cgi?id=1040397

== Monitoring TS and Sync Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster

* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Monitoring_the_Stage_Environment

== Performance Testing Information ==
* TBD

== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com

== Known Bugs, Issues, and Tasks ==
* Tokenserver, Verifier, and Sync
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=907479
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=1008066
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=1014411

** Tokenserver:
*** Repo: https://github.com/mozilla-services/tokenserver/issues
*** Bugzilla: http://mzl.la/1s4qZKn

** BrowserID-Verifier:
** Repo: https://github.com/mozilla/browserid-verifier/issues
** Bugzilla: no specific cateogory

** Sync:
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Bugzilla: http://mzl.la/VUrYQ5

* OPs and Infrastructure
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues

* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-aws/issues
** https://github.com/mozilla-services/loads-web/issues

== References ==
* Other URLs
** http://restmail.net
** https://mockmyid.com
* Repositories
** https://github.com/mozilla-services/tokenserver
** https://github.com/mozilla-services/server-syncstorage
** https://github.com/mozilla/browserid-verifier
** https://github.com/mozilla/browserid-local-verify
** https://github.com/callahad/mockmyid
** https://github.com/mozilla/restmail.net
* Documentation
** https://docs.services.mozilla.com
** https://docs.services.mozilla.com/token/
** https://wiki.mozilla.org/Services/Sagrada/TokenServer
** https://docs.services.mozilla.com/sync/
* The QA Test Environments:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
* Deploying the FxA Load Test environment for broker/agents usage:
** https://github.com/mozilla/fxa-deployment
* Sync 1.5 protocol, documentation, etc.
** https://github.com/mozilla-services/docs
** https://docs.services.mozilla.com/#how-to
** https://docs.services.mozilla.com/howtos/run-fxa.html
** https://docs.services.mozilla.com/token/apis.html
** https://docs.services.mozilla.com/storage/apis-1.5.html
** https://docs.services.mozilla.com/howtos/run-sync-1.5.html
** https://docs.services.mozilla.com/howtos/run-sync-1.5.html
** https://github.com/mozilla-services/syncserver
* OPs pages for stats collection, logging, monitoring
** TBD