TestEngineering/Services/FxATestEnvironments

NOTE: This site needs an update for all the extra Dev environments and the new environments - OAuth and Profile, etc...

FxA Production Environments

URLs
- Content server: https://accounts.firefox.com
- Auth server: https://api.accounts.firefox.com
- Verifier: https://verifier.accounts.firefox.com (talks to Auth Server via TokenServer)
- OAuth server: https://oauth.accounts.firefox.com
- Profile server: https://profile.accounts.firefox.com

Versions:
- FxA-Auth-Server: curl https://api.accounts.firefox.com/; echo
- FxA-Content-Server: curl https://accounts.firefox.com/ver.json; echo

QA Access: not allowed

Pointing Fx Nightly to Production FxA/Sync:
- Set/Verify the following Firefox configs:
  - services.sync.clusterURL = (should get automatically set by the TokenServer)
  - services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
  - services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
  - services.sync.log.appender.file.logOnError = Yes
  - services.sync.log.appender.file.logOnSuccess = Yes
  - services.sync.log.appender.file.level = Trace
  - identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
  - identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...

FxA Stage Environment

URLs
- Content Server: https://accounts.stage.mozaws.net/
- Auth Server: https://api-accounts.stage.mozaws.net/
- Verifier: https://verifier.stage.mozaws.net (talks to Auth Server via TokenServer)
- OAuth server: https://oauth.stage.mozaws.net
- Profile server: https://profile.stage.mozaws.net

Versions:
- FxA-Auth-Server: curl https://api-accounts.stage.mozaws.net/; echo
- FxA-Content-Server: curl https://accounts.stage.mozaws.net/ver.json; echo

AWS
- New FxA-shared services:
  - svcops-fxa-stage-heka
  - svcops-fxa-stage-amqp
  - svcops-fxa-stage-elasticsearch
  - svcops-fxa-stage-asg-sns-topic
- Other shared Stage Services:
  - svcops-stage-heka (combined heka+kibana behind an ELB)
  - svcops-stage-amqp (shared RabbitMQ cluster)
  - svcops-stage-elasticsearch ( Elasticsearch cluster)
  - svcops-stage-asg-sns-topic
  - svcops-stage-bastion-servers (Bastion hosts for Stage)
- Load Balancers:
  - ContentServerELB
  - AuthServerELB
  - BastionELB
  - BastionExternalFQDN
- DB Instances (RDS):
  - RDSInstance = fxa-rds-stage
- All instances: Search in AWS Console using the Env column to find "stage"
  - shared-elasticsearch
  - shared-elasticsearch
  - shared-elasticsearch
  - shared-elasticsearch
  - shared-rabbitmq
  - shared-rabbitmq
  - shared-bastion
  - shared-heka
  - fxa-elasticsearch
  - fxa-rabbitmq
  - fxa-heka
  - fxa-logbox
  - fxa-auth (2 instances, m3.large)
  - fxa-content_server (1 small instance)
Files
- /data
- /etc/puppet
Processes
- fxa-auth-server:
  - Look for processes owned by app, nginx, circus
- fxa-content-server:
  - Look for processes owned by node, heka, nginx, circus
Logs
- /var/log/nginx
- /var/log/circus.log
- fxa-auth-server
  - /media/ephemeral0/fxa-auth-server/auth_err.log.*
  - /media/ephemeral0/fxa-auth-server/auth_out.log
  - /media/ephemeral0/heka/hekad_err.log
  - /media/ephemeral0/heka/hekad_out.log
  - /media/ephemeral0/nginx/logs/access.log
  - /media/ephemeral0/nginx/logs/error.log
- fxa-content-server
  - /media/ephemeral0/fxa-content-server/content_err.log
  - /media/ephemeral0/fxa-content-server/content_out.log
  - /media/ephemeral0/heka/hekad_err.log
  - /media/ephemeral0/heka/hekad_out.log
  - /media/ephemeral0/nginx/access.log
  - /media/ephemeral0/nginx/logs/error.log

Log Aggregation
- Look at /data/hekad/hekad.toml
- Check here: [aggregator-output]
- hekad --> rabbitmq --> elasticsearch --> kibana (since Kibana is the UI on top of ES)

Stack code

How to find the stack instance size and number of instances
- https://github.com/mozilla-services/svcops/tree/master/cloudformations/firefox-accounts
- Environment, number of hosts, and host/instance size deployed by default (per stack/service):
  - Look in the json files for the CF stacks in github:

fxa-auth-server.json:
    defaults for "Environment", "AuthServerInstanceType", and "AuthServerDesiredCapacity"
fxa-content-server.json:
    defaults for "Environment", "InstanceType", and "NumServers"
fxa-db-rds.json:
    defaults for "Environment" and "DBInstanceType"
Compare this to what is actually live in AWS (via the Console or awsboxen)

QA Access via a Bastion Host
- Old Dev IAM
  - SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance.
- New Dev IAM
  - SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance.

Pointing Fx Nightly to Stage FxA/Sync:
- Set/Verify the following Firefox configs:
  - services.sync.clusterURL = (should get automatically set by the TokenServer)
  - services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5
  - services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
  - services.sync.log.appender.file.logOnError = Yes
  - services.sync.log.appender.file.logOnSuccess = Yes
  - services.sync.log.appender.file.level = Trace
  - identity.fxaccounts.auth.uri = https://api-accounts.stage.mozaws.net/v1
  - identity.fxaccounts.remote.uri = https://accounts.stage.mozaws.net/?service=sync

Loads V1 Services Cluster Environment

For details on the Loads V1 cluster, see the following link:
- https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

Monitoring the Production Environment

Heka FxA:

Kibana FxA:
- Kibana FxA: https://kibana.fxa.us-west-2.prod.mozaws.net/index.html#/dashboard/file/weblogs.json
- Kibana FxA errors: https://kibana.fxa.us-west-2.prod.mozaws.net/index.html#/dashboard/elasticsearch/Endpoint%20Errors
- Others: see the "load" folder/pull-down

StackDriver:

Monitoring the Stage Environment

Kibana:
- Kibana: https://kibana.fxa.us-east-1.stage.mozaws.net/#/dashboard
- Kibana: https://kibana.fxa.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json

Heka:

StackDriver:
- https://app.stackdriver.com/
- https://app.stackdriver.com/groups/4393/stage-fxa

FxA Development Environments - partial

NOTE: This is marked partial, because there are a number of more recent, ongoing Dev deploys for development and testing. They should get recorded here, soon...

QA Access
- SSH with AWS keys to the various instances

Content Server
- URLs
  - https://accounts.dev.lcip.org/
    - non-auto-updating environment (AWS - US East)
  - https://accounts-latest.dev.lcip.org/
    - auto-updated environment (AWS - US East)
- AWS
  - Search for the following in US East:
    - rfk's awsbox deployment (accounts-latest.dev.lcip.org)
    - rfk's awsbox deployment (accounts.dev.lcip.org)
- Versions
- Files
  - /home/app
  - /home/proxy
- Processes
  - Look at any process owned by root, mysql, app, proxy
- Logs
  - /var/log/nginx/access.log and error.log
  - /home/app/var/log/browserid-certifier.js.log
  - /home/app/var/log/firefox_account_bridge.js.log

Auth Server
- URLs
  - https://api-accounts.dev.lcip.org
  - https://api-accounts-latest.dev.lcip.org
- AWS
  - Search for the following in US East:
    - rfk's awsbox deployment (api-accounts-latest.dev.lcip.org)
    - rfk's awsbox deployment (api-accounts.dev.lcip.org)
- Versions
  - https://api-accounts.dev.lcip.org/ver.txt
  - https://api-accounts-latest.dev.lcip.org/ver.txt
- Files
  - /home/app
- Processes
- Search for any process owned by app, root, mysql, proxy
- Logs
  - /home/app/hekad/*.log
  - /home/app/var/log/*.log
  - /home/proxy/var/log/nginx

Log Aggregation
- http://logs.dev.lcip.org

Mail
- https://restmail.lcip.org

Monitoring
- http://logs.dev.lcip.org/kibana/index.html#/dashboard/file/weblogs.json

Default Firefox configs for Nightly:
- services.sync.clusterURL = http://db1.oldsync.dev.lcip.org/ (should get automatically set by the TokenServer)
- services.sync.tokenServerURI = http://auth.oldsync.dev.lcip.org/1.0/sync/1.1
- firefox.accounts.remoteUrl = https://accounts.dev.lcip.org/?service=sync
  - identity.fxaccounts.enabled=true (NO LONGER NEEDED, BUT STILL CAN BE DEFINED)
  - services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
- services.sync.log.appender.file.logOnError = Yes
- services.sync.log.appender.file.logOnSuccess = Yes
- services.sync.log.appender.file.level = Trace
- identity.fxaccounts.auth.uri = https://api-accounts.dev.lcip.org/v1
- And to hit the TokenServer/Sync 1.5 Dev services, change the following:
  - services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
  - services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5
- And to hit the TokenServer/Sync 1.5 Stage services, change the following:
  - services.sync.clusterURL = https://sync-X-us-east-1.stage.mozaws.net/ (should get automatically set by the TokenServer)
  - services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5
- And to hit the TokenServer/Sync 1.5 Prod services, change the following:
  - services.sync.clusterURL = https://sync-X-us-east-2.sync.services.mozilla.com/ (should get automatically set by the TokenServer)
  - services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5

Other Dev Deployments:
- AWS: api-accounts-legacy.dev.lcip.org
- AWS: accounts-legacy.dev.lcip.org
- AWS: api-accounts-onepw.dev.lcip.org

OAuth Dev Environment

NOTE: TBD

OPs Mana and GitHub Pages

NOTE: Talk to OPs for the links to Mana.
Puppet Config: https://github.com/mozilla-services/puppet-config
- and https://github.com/mozilla-services/puppet-config/tree/master/fxa
Cloud Formation: https://github.com/mozilla-services/svcops
- and https://github.com/mozilla-services/svcops/tree/master/cloudformations/firefox-accounts

TestEngineering/Services/FxATestEnvironments

Contents

FxA Production Environments

FxA Stage Environment

Loads V1 Services Cluster Environment

Monitoring the Production Environment

Monitoring the Stage Environment

FxA Development Environments - partial

OAuth Dev Environment

OPs Mana and GitHub Pages

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

How to Contribute

MozillaWiki

Around Mozilla

Tools