TestEngineering/Services/FxATestEnvironments

NOTE: This site needs an update for all the extra Dev environments and the new environments - OAuth and and Profile...

FxA Production Environments

Content server: https://accounts.firefox.com
Auth server: https://api.accounts.firefox.com
Verifier: https://verifier.accounts.firefox.com (talks to Auth Server via TokenServer)

Versions:
- FxA-Auth-Server: curl https://api.accounts.firefox.com/; echo
- FxA-Content-Server: curl https://accounts.firefox.com/ver.json; echo

Monitoring
- Heka FxA:
- Kibana FxA: https://kibana.fxa.us-west-2.prod.mozaws.net/index.html#/dashboard/file/weblogs.json
- Kibana FxA errors: https://kibana.fxa.us-west-2.prod.mozaws.net/index.html#/dashboard/elasticsearch/Endpoint%20Errors
- Others: see the "load" folder/pull-down
- StackDriver:

QA Access: not allowed

Pointing Fx Nightly to Production FxA/Sync:
- Set/Verify the following Firefox configs:
  - services.sync.clusterURL = (should get automatically set by the TokenServer)
  - services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
  - services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
  - services.sync.log.appender.file.logOnError = Yes
  - services.sync.log.appender.file.logOnSuccess = Yes
  - services.sync.log.appender.file.level = Trace
  - identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
  - identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...

FxA Stage Environment

URLs
- Content Server: https://accounts.stage.mozaws.net/
- Auth Server: https://api-accounts.stage.mozaws.net/
- Verifier: https://verifier.stage.mozaws.net (talks to Auth Server via TokenServer)

Versions:
- FxA-Auth-Server: curl https://api-accounts.stage.mozaws.net/; echo
- FxA-Content-Server: curl https://accounts.stage.mozaws.net/ver.json; echo

AWS
- New FxA-shared services:
  - svcops-fxa-stage-heka
  - svcops-fxa-stage-amqp
  - svcops-fxa-stage-elasticsearch
  - svcops-fxa-stage-asg-sns-topic
- Other shared Stage Services:
  - svcops-stage-heka (combined heka+kibana behind an ELB)
  - svcops-stage-amqp (shared RabbitMQ cluster)
  - svcops-stage-elasticsearch ( Elasticsearch cluster)
  - svcops-stage-asg-sns-topic
  - svcops-stage-bastion-servers (Bastion hosts for Stage)
- Load Balancers:
  - ContentServerELB
  - AuthServerELB
  - BastionELB
  - BastionExternalFQDN
- DB Instances (RDS):
  - RDSInstance = fxa-rds-stage
- All instances: Search in AWS Console using the Env column to find "stage"
  - shared-elasticsearch
  - shared-elasticsearch
  - shared-elasticsearch
  - shared-elasticsearch
  - shared-rabbitmq
  - shared-rabbitmq
  - shared-bastion
  - shared-heka
  - fxa-elasticsearch
  - fxa-rabbitmq
  - fxa-heka
  - fxa-logbox
  - fxa-auth (2 instances, m3.large)
  - fxa-content_server (1 small instance)
Files
- /data
- /etc/puppet
Processes
- fxa-auth-server:
  - Look for processes owned by app, nginx, circus
- fxa-content-server:
  - Look for processes owned by node, heka, nginx, circus
Logs
- /var/log/nginx
- /var/log/circus.log
- fxa-auth-server
  - /media/ephemeral0/fxa-auth-server/auth_err.log.*
  - /media/ephemeral0/fxa-auth-server/auth_out.log
  - /media/ephemeral0/heka/hekad_err.log
  - /media/ephemeral0/heka/hekad_out.log
  - /media/ephemeral0/nginx/logs/access.log
  - /media/ephemeral0/nginx/logs/error.log
- fxa-content-server
  - /media/ephemeral0/fxa-content-server/content_err.log
  - /media/ephemeral0/fxa-content-server/content_out.log
  - /media/ephemeral0/heka/hekad_err.log
  - /media/ephemeral0/heka/hekad_out.log
  - /media/ephemeral0/nginx/access.log
  - /media/ephemeral0/nginx/logs/error.log

Log Aggregation
- Look at /data/hekad/hekad.toml
- Check here: [aggregator-output]
- hekad --> rabbitmq --> elasticsearch --> kibana (since Kibana is the UI on top of ES)

Stack code

How to find the stack instance size and number of instances
- https://github.com/mozilla-services/svcops/tree/master/cloudformations/firefox-accounts
- Environment, number of hosts, and host/instance size deployed by default (per stack/service):
  - Look in the json files for the CF stacks in github:

fxa-auth-server.json:
    defaults for "Environment", "AuthServerInstanceType", and "AuthServerDesiredCapacity"
fxa-content-server.json:
    defaults for "Environment", "InstanceType", and "NumServers"
fxa-db-rds.json:
    defaults for "Environment" and "DBInstanceType"
Compare this to what is actually live in AWS (via the Console or awsboxen)

Monitoring:
- Kibana: https://kibana.fxa.us-east-1.stage.mozaws.net/#/dashboard
- Kibana: https://kibana.fxa.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json
- Heka:
- Note: Make sure to have the Mozilla Root Cert set up in your browser: https://wiki.mozilla.org/MozillaRootCertificate
- OPs has set up the following in StackDriver
  - https://app.stackdriver.com/
  - https://app.stackdriver.com/groups/4393/stage-fxa

QA Access via a Bastion Host
- SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

Pointing Fx Nightly to Stage FxA/Sync:
- Set/Verify the following Firefox configs:
  - services.sync.clusterURL = (should get automatically set by the TokenServer)
  - services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5
  - services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
  - services.sync.log.appender.file.logOnError = Yes
  - services.sync.log.appender.file.logOnSuccess = Yes
  - services.sync.log.appender.file.level = Trace
  - identity.fxaccounts.auth.uri = https://api-accounts.stage.mozaws.net/v1
  - identity.fxaccounts.remote.uri = https://accounts.stage.mozaws.net/?service=sync

OAuth Stage Environment

NOTE: TBD, see https://bugzilla.mozilla.org/show_bug.cgi?id=1022868

URLs
- OAuth: https://oauth.stage.mozaws.net
- Profile: https://profile.stage.mozaws.net
- Client: https://123done-stage.dev.lcip.org
- which talks to
  - https://api-accounts.stage.mozaws.net
  - https://accounts.stage.mozaws.net

Versions
- TBD

AWS
- TBD

Files
- TBD

Processes
- TBD

Logs
- TBD

Log Aggregation
- TBD

Monitoring
- TBD

QA Access via a Bastion Host
- SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.

Pointing Nightly to Stage OAuth
- TBD

FxA Load Test Environment

NOTE: Available in AWS only as needed.
URLs
AWS in US West
- CF: loads-lcip-org
- CF: loadtest-lcip-org
- LoadsBroker: broker.loads.lcip.org
- IdP: api-accounts.loadtest.lcip.org
- Logger: logs.loadtest.lcip.org
- Redis Server: redis.loadtest.lcip.org
- LoadsAgentLoadBalancer
- IdPLoadBalancer

Loads Services Cluster Environment

URLs
- http://loads.services.mozilla.com/
- or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com/

Versions

Loads Cluster/Broker/Agents:
$ cd /home/ubuntu/loads/bin
$ ./loads-runner --version

AWS in US West
- loads-master (broker and agent processes)
- loads-slave-1 (agent processes)
- loads-slave-2 (agent processes)
- NOTE: there is no stack or ELB for this cluster

Files
- /home/ubuntu
  - loads
  - loads-aws
  - loads-web
Processes
- Search for processes owned by ubuntu, loads, nginx, circus
Logs
- /var/log/redis
- /var/log/nginx
QA access
- You need special access to be able to SSH into these devices
- You need to make some changes to your .ssh/config file
Monitoring
- Web Interface: http://loads.services.mozilla.com/
- StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster
Cluster status
- Check directly from the Loads Cluster dashboard:

Agents statuses
Launch a health check on all agents

Links

FxA Development Environments - partial

NOTE: This is marked partial, because there are a number of more recent, ongoing Dev deploys for development and testing. They should get recorded here, soon...

QA Access
- SSH with AWS keys to the various instances

Content Server
- URLs
  - https://accounts.dev.lcip.org/
    - non-auto-updating environment (AWS - US East)
  - https://accounts-latest.dev.lcip.org/
    - auto-updated environment (AWS - US East)
- AWS
  - Search for the following in US East:
    - rfk's awsbox deployment (accounts-latest.dev.lcip.org)
    - rfk's awsbox deployment (accounts.dev.lcip.org)
- Versions
- Files
  - /home/app
  - /home/proxy
- Processes
  - Look at any process owned by root, mysql, app, proxy
- Logs
  - /var/log/nginx/access.log and error.log
  - /home/app/var/log/browserid-certifier.js.log
  - /home/app/var/log/firefox_account_bridge.js.log

Auth Server
- URLs
  - https://api-accounts.dev.lcip.org
  - https://api-accounts-latest.dev.lcip.org
- AWS
  - Search for the following in US East:
    - rfk's awsbox deployment (api-accounts-latest.dev.lcip.org)
    - rfk's awsbox deployment (api-accounts.dev.lcip.org)
- Versions
  - https://api-accounts.dev.lcip.org/ver.txt
  - https://api-accounts-latest.dev.lcip.org/ver.txt
- Files
  - /home/app
- Processes
- Search for any process owned by app, root, mysql, proxy
- Logs
  - /home/app/hekad/*.log
  - /home/app/var/log/*.log
  - /home/proxy/var/log/nginx

Log Aggregation
- http://logs.dev.lcip.org

Mail
- https://restmail.lcip.org

Monitoring
- http://logs.dev.lcip.org/kibana/index.html#/dashboard/file/weblogs.json

Default Firefox configs for Nightly:
- services.sync.clusterURL = http://db1.oldsync.dev.lcip.org/ (should get automatically set by the TokenServer)
- services.sync.tokenServerURI = http://auth.oldsync.dev.lcip.org/1.0/sync/1.1
- firefox.accounts.remoteUrl = https://accounts.dev.lcip.org/?service=sync
  - identity.fxaccounts.enabled=true (NO LONGER NEEDED, BUT STILL CAN BE DEFINED)
  - services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
- services.sync.log.appender.file.logOnError = Yes
- services.sync.log.appender.file.logOnSuccess = Yes
- services.sync.log.appender.file.level = Trace
- identity.fxaccounts.auth.uri = https://api-accounts.dev.lcip.org/v1
- And to hit the TokenServer/Sync 1.5 Dev services, change the following:
  - services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
  - services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5
- And to hit the TokenServer/Sync 1.5 Stage services, change the following:
  - services.sync.clusterURL = https://sync-X-us-east-1.stage.mozaws.net/ (should get automatically set by the TokenServer)
  - services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5
- And to hit the TokenServer/Sync 1.5 Prod services, change the following:
  - services.sync.clusterURL = https://sync-X-us-east-2.sync.services.mozilla.com/ (should get automatically set by the TokenServer)
  - services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5

Other Dev Deployments:
- AWS: api-accounts-legacy.dev.lcip.org
- AWS: accounts-legacy.dev.lcip.org
- AWS: api-accounts-onepw.dev.lcip.org

OAuth Dev Environment

NOTE: TBD

OPs Mana and GitHub Pages

NOTE: Talk to OPs for the links to Mana.
Puppet Config: https://github.com/mozilla-services/puppet-config
- and https://github.com/mozilla-services/puppet-config/tree/master/fxa
Cloud Formation: https://github.com/mozilla-services/svcops
- and https://github.com/mozilla-services/svcops/tree/master/cloudformations/firefox-accounts

TestEngineering/Services/FxATestEnvironments

Contents

FxA Production Environments

FxA Stage Environment

OAuth Stage Environment

FxA Load Test Environment

Loads Services Cluster Environment

FxA Development Environments - partial

OAuth Dev Environment

OPs Mana and GitHub Pages

Navigation menu

TestEngineering/Services/FxATestEnvironments

FxA Production Environments

FxA Stage Environment

OAuth Stage Environment

FxA Load Test Environment

Loads Services Cluster Environment

FxA Development Environments - partial

OAuth Dev Environment

OPs Mana and GitHub Pages

Navigation menu

Search