TestEngineering/Services/LoopTestEnvironments

From MozillaWiki
Jump to navigation Jump to search

Loop Server Production Environments

  • QA Access: not allowed
  • Pointing Fx Nightly to Production
    • NOTE: Fx Nightly is already set to point to Production by default

Loop Server Stage Environment

    • loop-redis-stage - need the details

Loop Server Stage Details

  • Versions
    • rpm -qa | grep loop
    • Example:
      • loop-server-svcops 0.10.0-1 x86_64 20905864
      • puppet-config-loop 20140811172607-1 x86_64 14277
  • AWS
    • CF stack: loop-server-stage (or similar name)
    • ELB: loop-server-st-ELB-<some value>
    • One instance: look for "loop-server-stage"
  • OS
    • Running SciLinux 6.4
  • Files
    • /data/loop-server
    • /data/hekad
    • /opt/aws
    • /opt/ec2
    • /opt/openresty
    • /opt/rh
    • /opt/stackdriver
    • /etc/circus.ini
    • /etc/circus.d
    • /etc/circus.d/loopserver.ini
    • /etc/circus.d/hekad-loop.ini
    • /etc/puppet/*
      • especially, /etc/puppet/yaml/app
    • /var/log/*
  • Processes
    • node loop/index.js
    • stackdriver-collectd
    • circusd
    • heka.d
    • nginx
  • Logs
    • /var/log/circus.log
    • /var/log/hekad/loop.stdout.log
    • /var/log/hekad/loop.stderr.log
    • /media/ephemeral0/circus/loop_server/loop_server.err.log
    • /media/ephemeral0/circus/loop_server/loop_server.eout.log
    • /media/ephemeral0/nginx/logs/default.access.log (not in use)
    • /media/ephemeral0/nginx/logs/default.error.log (not in use)
    • /media/ephemeral0/nginx/logs/loop_server.access.log
    • /media/ephemeral0/nginx/logs/loop_server.error.log
  • Setting Stage configuration:
    • Toggling the config setting to use either a mock server or a third-party server
    • File: /data/loop-server/config/settings.json
  • Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

Loop Client and Server Configurations

  • Yaml Files
    • /etc/puppet/yaml/app
      • loop_client.dev.yaml
      • loop_client.prod.yaml
      • loop_client.stage.yaml
      • loop_client.yaml
      • loop_server.dev.yaml
      • loop_server.prod.yaml
      • loop_server.stage.yaml
      • loop_server.yaml

Loop Client Stage Details

  • Versions
    • rpm -qa | grep loop
  • AWS
    • CF Stack: look for loop-client-*
    • ELB: loop-client-st-ELB-<some value>
    • One instance: look for "loop-client app server"
  • OS
    • Scientific Linux release 6.4
  • Files
    • /opt/aws
    • /opt/ec2
    • /opt/openresty
    • /opt/rh
    • /opt/stackdriver
    • /data/loop-client
    • /data/hekad
    • /etc/heka.d
  • Processes
    • look for stackdriver, nginx, python, circus, heka
  • Logs
    • /var/log/circus.log
    • /var/log/hekad
    • /media/ephemeral0/nginx/logs
  • Heka
    • /data/hekad
    • /etc/heka.d
  • Quick Verifications
In the browser: https://call.stage.mozaws.net

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {serverUrl: 'https://https://loop.stage.mozaws.net'};

Loop Mock Server Stage Details

    • NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.
  • AWS
    • CF stack: loop-delayed-response (or similar)
    • ELB: loop-delayed-r-ELB-<some value>
    • R53AliasRecord: loop-delayed-response.stage.mozaws.net
    • Instances: 1 m1.small, search for "delayed HTTP server"
  • OS
    • Scientific Linux release 6.4
  • Files
    • /data/delayed
    • /data/delayed/delayed-master
    • /etc/circus.d/
    • /etc/circus.d/loopserver.ini
    • /etc/circus.d/hekad-loop.ini
    • /etc/circus.ini
  • Processes
    • root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
    • nobody node index.js
  • Logs
    • /var/log/circus.log
  • Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok
  • Access
    • Log in as "ec2-user"

MSISDN Gateway Server Stage Details

  • Versions
    • rpm -qa
    • Current:
      • msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
      • puppet-config-msisdn 20140811185304-1 x86_64 12723
  • AWS
    • CF Stack: msisdn-gateway-stage (or similar)
    • ELB: msisdn-gateway-ELB-<SOME VALUE>
    • Currently one m3.medium instance
  • OS
    • Scientific Linux release 6.4 (Carbon)
  • Files
    • /opt/aws
    • /opt/ec2
    • /opt/openresty
    • /opt/rh
    • /opt/stackdriver
    • /data/hekad
    • /data/msisdn-gateway
  • Processes
    • Look for stackdriver, python, circus, resty, nginx, node, heka
  • Logs
    • /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
    • /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
    • /media/ephemeral0/nginx/logsdefault.access.log (not in use)
    • /media/ephemeral0/nginx/logsdefault.error.log (not in use)
    • /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
    • /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
    • /var/log/circus.log
    • /var/log/hekad/msisdn_gateway.stderr.log
    • /var/log/hekad/msisdn_gateway.stdout.log
  • Configuration File
    • /data/msisdn-gateway/config/production.json
  • Quick Verifications
https://msisdn.stage.mozaws.net
returns
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN Gateway","version":"0.3.0- DEV",
"homepage":"https://github.com/mozilla-services/msisdn-gateway/","endpoint":"http://msisdn.stage.mozaws.net"}
(or similar)

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net
returns
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN Gateway","version":"0.3.0-DEV",
"homepage":"https://github.com/mozilla-services/msisdn-gateway/","endpoint":"http://msisdn.stage.mozaws.net"}
(or similar)

curl -I https://msisdn.stage.mozaws.net
returns
HTTP/1.1 200 OK
Content-length: 207
Content-Type: application/json; charset=utf-8
Date: Thu, 03 Jul 2014 00:38:55 GMT
ETag: W/"cf-2959630074"
Timestamp: 1404347935320
Connection: keep-alive
(or similar)

MSISDN Mock Server Stage Details

  • AWS
    • This is one m3.medium
    • Look for omxen.dev.mozaws.net
  • Access: SSH
  • OS
    • Amazon Linux AMI release 2014.03
  • Versions
    • TBD
  • Processes
    • Look for nginx, python, omxen
  • Files
    • /home/ec2-user/omxen
    • /opt/aws
  • Logs
    • /var/log/nginx
    • Use: sudo cat nginx/*.log, etc
  • Restarting
    • TBD
  • Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY

curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY

curl -I omxen.dev.mozaws.net
    HTTP/1.1 200 OK
    Server: nginx/1.4.7
    Date: Tue, 08 Jul 2014 23:10:10 GMT
    Content-Type: text/html; charset=UTF-8
    Content-Length: 0
    Connection: keep-alive

Sentry Server Stage Details

  • Versions
    • rpm -qa | grep sentry
  • AWS
    • TBD
  • OS
    • Scientific Linux release 6.3
  • Files
    • /opt/sentry
  • Processes
    • root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
    • apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud
  • Logs
    • /var/log
    • nothing in /media/ephemeral0
  • Heka
    • TBD
  • Quick Verifications
    • For now, just verify the service is running on the host: ps aux | grep sentry

QA Access via a Bastion Host

  • SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

Monitoring the Stage Environment

Monitoring the Production Environment

https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json

Loads Services Cluster Environment

  • Versions
Loads Cluster/Broker/Agents:
$ cd /home/ubuntu/loads/bin
$ ./loads-runner --version
  • AWS in US West
    • loads-master (broker and agent processes)
    • loads-slave-1 (agent processes)
    • loads-slave-2 (agent processes)
    • NOTE: there is no stack or ELB for this cluster
  • Files
    • /home/ubuntu
      • loads
      • loads-aws
      • loads-web
  • Processes
    • Search for processes owned by ubuntu, loads, nginx, circus
  • Logs
    • /var/log/redis
    • /var/log/nginx
  • QA access
    • You need special access to be able to SSH into these devices
    • You need to make some changes to your .ssh/config file
  • Monitoring
  • Cluster status
    • Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents

Dev Environment

  • AWS
    • CF Stack: ulfr-devstack-loop
    • ELB: ulfr-devstack-ELB-<some value>
    • One instance: search for "loop app server (ulfr-devstack-loop)"
    • CF Stack: bwong-msisdn-dev
    • ELB: bwong-msisdn-d-ELB-<some value>
    • One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"
  • OS
    • TBD
  • Files
    • TBD
  • Processes
    • TBD
  • Logs
    • TBD
  • Heka
    • TBD
  • Quick Verifications
http://loop.dev.mozaws.net
Returns
{
 "name": "mozilla-loop-server",
 "description": "The Mozilla Loop (WebRTC App) server",
 "version": "0.7.0",
 "homepage": "https://github.com/mozilla-services/loop-server/",
 "endpoint": "http://loop.dev.mozaws.net",
 "fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
 {
 "storage": true,
 "provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
    Loop
    Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
 "name": "mozilla-msisdn-gateway",
 "description": "The Mozilla MSISDN Gateway",
 "version": "0.3.0-DEV",
 "homepage": "https://github.com/mozilla-services/msisdn-gateway/",
 "endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
 "storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
 "name": "mozilla-msisdn-gateway",
 "description": "The Mozilla MSISDN Gateway",
 "version": "0.3.0-DEV",
 "homepage": "https://github.com/mozilla-services/msisdn-gateway/",
 "endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
 "storage": true
}
or similar

References

OPs Mana and GitHub Pages

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/LoopTestEnvironments&oldid=1005241"