TestEngineering/Services/LoopTestEnvironments: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 39: Line 39:
** Loop-Client: https://call.stage.mozaws.net
** Loop-Client: https://call.stage.mozaws.net


** MSISDN: http://msisdn.stage.mozaws.net (see below)
** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net
** Mock (load): http://omxen.dev.mozaws.net


Revision as of 17:40, 6 August 2014

Loop Server Production Environments

  • QA Access: not allowed
  • Pointing Fx Nightly to Production
    • NOTE: Fx Nightly is already set to point to Production by default
  • Quick Verifications
Loop-Server
https://loop.services.mozilla.com
curl https://loop.services.mozilla.com
curl -I https://loop.services.mozilla.com

Loop Client
https://call.mozilla.com
https://call.mozilla.com/config.js
curl https://call.mozilla.com
curl -I https://call.mozilla.com

MSISDN-Gateway
https://msisdn.services.mozilla.com
curl https://msisdn.services.mozilla.com
curl -I https://msisdn.services.mozilla.com

Loop Server Stage Environment

    • loop-redis-stage - need the details

Loop Server Stage Details

  • Versions
    • rpm -qa | grep loop
    • Example:
      • puppet-config-loop 20140709221704-1 x86_64 14798
      • loop-server-svcops 0.9.0-1 x86_64 20757626
  • AWS
    • CF stack: loop-server-stage (or similar name)
    • ELB: loop-server-st-ELB-<some value>
    • One instance: look for "loop-server-stage"
  • OS
    • Running SciLinux 6.4
  • Files
    • /data/loop-server
    • /data/hekad
    • /opt/aws
    • /opt/ec2
    • /opt/openresty
    • /opt/rh
    • /opt/stackdriver
    • /etc/circus.ini
    • /etc/circus.d
    • /etc/circus.d/loopserver.ini
    • /etc/circus.d/hekad-loop.ini
    • /etc/puppet/*
      • especially, /etc/puppet/yaml/app
    • /var/log/*
  • Processes
    • node loop/index.js
    • stackdriver-collectd
    • circusd
    • heka.d
    • nginx
  • Logs
    • /var/log/circus.log
    • /var/log/hekad/loop.stdout.log
    • /var/log/hekad/loop.stderr.log
    • /media/ephemeral0/circus/loop_server/loop_server.err.log
    • /media/ephemeral0/circus/loop_server/loop_server.eout.log
    • /media/ephemeral0/nginx/logs/default.access.log (not in use)
    • /media/ephemeral0/nginx/logs/default.error.log (not in use)
    • /media/ephemeral0/nginx/logs/loop_server.access.log
    • /media/ephemeral0/nginx/logs/loop_server.error.log
  • Setting Stage configuration:
    • Toggling the config setting to use either a mock server or a third-party server
    • File: /data/loop-server/config/settings.json
  • Quick Verifications
curl https://loop.stage.mozaws.net

curl -I https://loop.stage.mozaws.net

In the browser: https://loop.stage.mozaws.net/

Loop Client and Server Configurations

  • Yaml Files
    • /etc/puppet/yaml/app
      • loop_client.dev.yaml
      • loop_client.prod.yaml
      • loop_client.stage.yaml
      • loop_client.yaml
      • loop_server.dev.yaml
      • loop_server.prod.yaml
      • loop_server.stage.yaml
      • loop_server.yaml

Loop Client Stage Details

  • Versions
    • rpm -qa | grep loop
  • AWS
    • CF Stack: look for loop-client-*
    • ELB: loop-client-st-ELB-<some value>
    • One instance: look for "loop-client app server"
  • OS
    • Scientific Linux release 6.4
  • Files
    • /opt/aws
    • /opt/ec2
    • /opt/openresty
    • /opt/rh
    • /opt/stackdriver
    • /data/loop-client
    • /data/hekad
    • /etc/heka.d
  • Processes
    • look for stackdriver, nginx, python, circus, heka
  • Logs
    • /var/log/circus.log
    • /var/log/hekad
    • /media/ephemeral0/nginx/logs
  • Heka
    • /data/hekad
    • /etc/heka.d
  • Quick Verifications
In the browser: https://call.stage.mozaws.net

curl https://call.stage.mozaws.net

curl -I https://call.stage.mozaws.net

Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {serverUrl: 'https://https://loop.stage.mozaws.net'};

Loop Mock Server Stage Details

    • NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.
  • AWS
    • CF stack: loop-delayed-response (or similar)
    • ELB: loop-delayed-r-ELB-<some value>
    • R53AliasRecord: loop-delayed-response.stage.mozaws.net
    • Instances: 1 m1.small, search for "delayed HTTP server"
  • OS
    • Scientific Linux release 6.4
  • Files
    • /data/delayed
    • /data/delayed/delayed-master
    • /etc/circus.d/
    • /etc/circus.d/loopserver.ini
    • /etc/circus.d/hekad-loop.ini
    • /etc/circus.ini
  • Processes
    • root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
    • nobody node index.js
  • Logs
    • /var/log/circus.log
  • Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
Returns:
ok
  • Access
    • Log in as "ec2-user"

MSISDN Gateway Server Stage Details

  • Versions
    • rpm -qa
    • Current:
      • msisdn-gateway-svcops 0.3.0-0snap201407021023gite74cc9 x86_64 49842124
      • puppet-config-msisdn 20140702184605-1 x86_64 9669
  • AWS
    • CF Stack: msisdn-gateway-stage (or similar)
    • ELB: msisdn-gateway-ELB-<SOME VALUE>
    • Currently one m3.medium instance
  • OS
    • Scientific Linux release 6.4 (Carbon)
  • Files
    • /opt/aws
    • /opt/ec2
    • /opt/openresty
    • /opt/rh
    • /opt/stackdriver
    • /data/hekad
    • /data/msisdn-gateway
  • Processes
    • Look for stackdriver, python, circus, resty, nginx, node, heka
  • Logs
    • /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
    • /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
    • /media/ephemeral0/nginx/logsdefault.access.log (not in use)
    • /media/ephemeral0/nginx/logsdefault.error.log (not in use)
    • /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
    • /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
    • /var/log/circus.log
    • /var/log/hekad/msisdn_gateway.stderr.log
    • /var/log/hekad/msisdn_gateway.stdout.log
  • Configuration File
    • /data/msisdn-gateway/config/production.json
  • Quick Verifications
https://msisdn.stage.mozaws.net
returns
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN Gateway","version":"0.3.0- DEV",
"homepage":"https://github.com/mozilla-services/msisdn-gateway/","endpoint":"http://msisdn.stage.mozaws.net"}
(or similar)

https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}

curl https://msisdn.stage.mozaws.net
returns
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN Gateway","version":"0.3.0-DEV",
"homepage":"https://github.com/mozilla-services/msisdn-gateway/","endpoint":"http://msisdn.stage.mozaws.net"}
(or similar)

curl -I https://msisdn.stage.mozaws.net
returns
HTTP/1.1 200 OK
Content-length: 207
Content-Type: application/json; charset=utf-8
Date: Thu, 03 Jul 2014 00:38:55 GMT
ETag: W/"cf-2959630074"
Timestamp: 1404347935320
Connection: keep-alive
(or similar)

MSISDN Mock Server Stage Details

Sentry Server Stage Details

  • Versions
    • rpm -qa | grep sentry
  • AWS
    • TBD
  • OS
    • Scientific Linux release 6.3
  • Files
    • /opt/sentry
  • Processes
    • root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
    • apache /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud
  • Logs
    • /var/log
    • nothing in /media/ephemeral0
  • Heka
    • TBD
  • Quick Verifications
    • For now, just verify the service is running on the host: ps aux | grep sentry

QA Access via a Bastion Host

  • SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".

Monitoring the Stage Environment

Monitoring the Production Environment

https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json

Loads Services Cluster Environment

  • Versions
Loads Cluster/Broker/Agents:
$ cd /home/ubuntu/loads/bin
$ ./loads-runner --version
  • AWS in US West
    • loads-master (broker and agent processes)
    • loads-slave-1 (agent processes)
    • loads-slave-2 (agent processes)
    • NOTE: there is no stack or ELB for this cluster
  • Files
    • /home/ubuntu
      • loads
      • loads-aws
      • loads-web
  • Processes
    • Search for processes owned by ubuntu, loads, nginx, circus
  • Logs
    • /var/log/redis
    • /var/log/nginx
  • QA access
    • You need special access to be able to SSH into these devices
    • You need to make some changes to your .ssh/config file
  • Monitoring
  • Cluster status
    • Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents

Dev Environment

  • AWS
    • CF Stack: ulfr-devstack-loop
    • ELB: ulfr-devstack-ELB-<some value>
    • One instance: search for "loop app server (ulfr-devstack-loop)"
    • CF Stack: bwong-msisdn-dev
    • ELB: bwong-msisdn-d-ELB-<some value>
    • One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"
  • OS
    • TBD
  • Files
    • TBD
  • Processes
    • TBD
  • Logs
    • TBD
  • Heka
    • TBD
  • Quick Verifications
http://loop.dev.mozaws.net
Returns
{
 "name": "mozilla-loop-server",
 "description": "The Mozilla Loop (WebRTC App) server",
 "version": "0.7.0",
 "homepage": "https://github.com/mozilla-services/loop-server/",
 "endpoint": "http://loop.dev.mozaws.net",
 "fakeTokBox": false
}
or similar

http://loop.dev.mozaws.net/__heartbeat__
Returns
 {
 "storage": true,
 "provider": false
}
or similar

http://loop-webapp.dev.mozaws.net
Returns
    Loop
    Welcome to the Loop web client.

http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404

http://msisdn.dev.mozaws.net
Returns
{
 "name": "mozilla-msisdn-gateway",
 "description": "The Mozilla MSISDN Gateway",
 "version": "0.3.0-DEV",
 "homepage": "https://github.com/mozilla-services/msisdn-gateway/",
 "endpoint": "https://msisdn.dev.mozaws.net"
}
or similar

http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
 "storage": true
}
or similar

https://msisdn-dev.stage.mozaws.net
Returns
{
 "name": "mozilla-msisdn-gateway",
 "description": "The Mozilla MSISDN Gateway",
 "version": "0.3.0-DEV",
 "homepage": "https://github.com/mozilla-services/msisdn-gateway/",
 "endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar

https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
 "storage": true
}
or similar

References

OPs Mana and GitHub Pages

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/LoopTestEnvironments&oldid=1003374"