TestEngineering/Services/FxATestEnvironments

NOTE: This site needs an update for all the extra Dev environments and the new environments - OAuth and Profile, etc...

FxA Production Environments

• URLs
  • Content server: https://accounts.firefox.com
  • Auth server: https://api.accounts.firefox.com
  • Verifier: https://verifier.accounts.firefox.com (talks to Auth Server via TokenServer)
  • OAuth server: https://oauth.accounts.firefox.com
  • Profile server: https://profile.accounts.firefox.com

• Versions:
  • FxA-Auth-Server: curl https://api.accounts.firefox.com/; echo
  • FxA-Content-Server: curl https://accounts.firefox.com/ver.json; echo

• QA Access: not allowed

• Pointing Fx Nightly to Production FxA/Sync:
  • Set/Verify the following Firefox configs:
    • services.sync.clusterURL = (should get automatically set by the TokenServer)
    • services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
    • services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
    • services.sync.log.appender.file.logOnError = Yes
    • services.sync.log.appender.file.logOnSuccess = Yes
    • services.sync.log.appender.file.level = Trace
    • identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
    • identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...

FxA Stage Environment

• URLs
  • Content Server: https://accounts.stage.mozaws.net/
  • Auth Server: https://api-accounts.stage.mozaws.net/
  • Verifier: https://verifier.stage.mozaws.net (talks to Auth Server via TokenServer)
  • OAuth server: https://oauth.stage.mozaws.net
  • Profile server: https://profile.stage.mozaws.net

• Versions:
  • FxA-Auth-Server: curl https://api-accounts.stage.mozaws.net/; echo
  • FxA-Content-Server: curl https://accounts.stage.mozaws.net/ver.json; echo

• AWS
  • New FxA-shared services:
    • svcops-fxa-stage-heka
    • svcops-fxa-stage-amqp
    • svcops-fxa-stage-elasticsearch
    • svcops-fxa-stage-asg-sns-topic
  • Other shared Stage Services:
    • svcops-stage-heka (combined heka+kibana behind an ELB)
    • svcops-stage-amqp (shared RabbitMQ cluster)
    • svcops-stage-elasticsearch ( Elasticsearch cluster)
    • svcops-stage-asg-sns-topic
    • svcops-stage-bastion-servers (Bastion hosts for Stage)
  • Load Balancers:
    • ContentServerELB
    • AuthServerELB
    • BastionELB
    • BastionExternalFQDN
  • DB Instances (RDS):
    • RDSInstance = fxa-rds-stage
  • All instances: Search in AWS Console using the Env column to find "stage"
    • shared-elasticsearch
    • shared-elasticsearch
    • shared-elasticsearch
    • shared-elasticsearch
    • shared-rabbitmq
    • shared-rabbitmq
    • shared-bastion
    • shared-heka
    • fxa-elasticsearch
    • fxa-rabbitmq
    • fxa-heka
    • fxa-logbox
    • fxa-auth (2 instances, m3.large)
    • fxa-content_server (1 small instance)
• Files
  • /data
  • /etc/puppet
• Processes
  • fxa-auth-server:
    • Look for processes owned by app, nginx, circus
  • fxa-content-server:
    • Look for processes owned by node, heka, nginx, circus
• Logs
  • /var/log/nginx
  • /var/log/circus.log
  • fxa-auth-server
    • /media/ephemeral0/fxa-auth-server/auth_err.log.*
    • /media/ephemeral0/fxa-auth-server/auth_out.log
    • /media/ephemeral0/heka/hekad_err.log
    • /media/ephemeral0/heka/hekad_out.log
    • /media/ephemeral0/nginx/logs/access.log
    • /media/ephemeral0/nginx/logs/error.log
  • fxa-content-server
    • /media/ephemeral0/fxa-content-server/content_err.log
    • /media/ephemeral0/fxa-content-server/content_out.log
    • /media/ephemeral0/heka/hekad_err.log
    • /media/ephemeral0/heka/hekad_out.log
    • /media/ephemeral0/nginx/access.log
    • /media/ephemeral0/nginx/logs/error.log

• Log Aggregation
  • Look at /data/hekad/hekad.toml
  • Check here: [aggregator-output]
  • hekad --> rabbitmq --> elasticsearch --> kibana (since Kibana is the UI on top of ES)

• Stack code
  • https://github.com/mozilla-services/svcops/tree/master/cloudformations/firefox-accounts
  • https://github.com/mozilla-services/puppet-config/tree/master/fxa
  • https://github.com/mozilla-services/puppet-config/tree/master/shared

• How to find the stack instance size and number of instances
  • https://github.com/mozilla-services/svcops/tree/master/cloudformations/firefox-accounts
  • Environment, number of hosts, and host/instance size deployed by default (per stack/service):
    • Look in the json files for the CF stacks in github:

fxa-auth-server.json:
    defaults for "Environment", "AuthServerInstanceType", and "AuthServerDesiredCapacity"
fxa-content-server.json:
    defaults for "Environment", "InstanceType", and "NumServers"
fxa-db-rds.json:
    defaults for "Environment" and "DBInstanceType"
Compare this to what is actually live in AWS (via the Console or awsboxen)

• QA Access via a Bastion Host
  • Old Dev IAM
    • SSH with AWS keys to the Stage bastion host in US-East-1. From there SSH directly into any instance.
  • New Dev IAM
    • SSH with AWS keys to the Stage bastion host in US-East-1 or EU-West-1. From there SSH directly into any instance.

• Pointing Fx Nightly to Stage FxA/Sync:
  • Set/Verify the following Firefox configs:
    • services.sync.clusterURL = (should get automatically set by the TokenServer)
    • services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5
    • services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
    • services.sync.log.appender.file.logOnError = Yes
    • services.sync.log.appender.file.logOnSuccess = Yes
    • services.sync.log.appender.file.level = Trace
    • identity.fxaccounts.auth.uri = https://api-accounts.stage.mozaws.net/v1
    • identity.fxaccounts.remote.uri = https://accounts.stage.mozaws.net/?service=sync

Loads V1 Services Cluster Environment

• For details on the Loads V1 cluster, see the following link:
  • https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1

Monitoring the Production Environment

• Heka FxA:
  • https://heka.fxa.us-west-2.prod.mozaws.net/
  • https://heka.fxa.us-west-2.prod.mozaws.net/#health
  • https://heka.fxa.us-west-2.prod.mozaws.net/#sandboxes

• Kibana FxA:
  • Kibana FxA: https://kibana.fxa.us-west-2.prod.mozaws.net/index.html#/dashboard/file/weblogs.json
  • Kibana FxA errors: https://kibana.fxa.us-west-2.prod.mozaws.net/index.html#/dashboard/elasticsearch/Endpoint%20Errors
  • Others: see the "load" folder/pull-down

• StackDriver:
  • Main: https://app.stackdriver.com/
  • https://app.stackdriver.com/groups/4208/prod-persona
  • https://app.stackdriver.com/groups/4251/production-all
  • https://app.stackdriver.com/groups/4337/prod-fxa

Monitoring the Stage Environment

• Kibana:
  • Kibana: https://kibana.fxa.us-east-1.stage.mozaws.net/#/dashboard
  • Kibana: https://kibana.fxa.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json

• Heka:
  • https://heka.fxa.us-east-1.stage.mozaws.net/
  • https://heka.fxa.us-east-1.stage.mozaws.net/#health
  • https://heka.fxa.us-east-1.stage.mozaws.net/#sandboxes

• StackDriver:
  • https://app.stackdriver.com/
  • https://app.stackdriver.com/groups/4393/stage-fxa

FxA Development Environments - partial

• NOTE: This is marked partial, because there are a number of more recent, ongoing Dev deploys for development and testing. They should get recorded here, soon...

• QA Access
  • SSH with AWS keys to the various instances

• Content Server
  • URLs
    • https://accounts.dev.lcip.org/
      • non-auto-updating environment (AWS - US East)
    • https://accounts-latest.dev.lcip.org/
      • auto-updated environment (AWS - US East)
  • AWS
    • Search for the following in US East:
      • rfk's awsbox deployment (accounts-latest.dev.lcip.org)
      • rfk's awsbox deployment (accounts.dev.lcip.org)
  • Versions
    • https://accounts-latest.dev.lcip.org/ver.txt
    • https://api-accounts-latest.dev.lcip.org/ver.txt
    • https://accounts.dev.lcip.org/ver.txt
    • https://api-accounts.dev.lcip.org/ver.txt
  • Files
    • /home/app
    • /home/proxy
  • Processes
    • Look at any process owned by root, mysql, app, proxy
  • Logs
    • /var/log/nginx/access.log and error.log
    • /home/app/var/log/browserid-certifier.js.log
    • /home/app/var/log/firefox_account_bridge.js.log

• Auth Server
  • URLs
    • https://api-accounts.dev.lcip.org
    • https://api-accounts-latest.dev.lcip.org
  • AWS
    • Search for the following in US East:
      • rfk's awsbox deployment (api-accounts-latest.dev.lcip.org)
      • rfk's awsbox deployment (api-accounts.dev.lcip.org)
  • Versions
    • https://api-accounts.dev.lcip.org/ver.txt
    • https://api-accounts-latest.dev.lcip.org/ver.txt
  • Files
    • /home/app
  • Processes
  • Search for any process owned by app, root, mysql, proxy
  • Logs
    • /home/app/hekad/*.log
    • /home/app/var/log/*.log
    • /home/proxy/var/log/nginx

• Log Aggregation
  • http://logs.dev.lcip.org

• Mail
  • https://restmail.lcip.org

• Monitoring
  • http://logs.dev.lcip.org/kibana/index.html#/dashboard/file/weblogs.json

• Default Firefox configs for Nightly:
  • services.sync.clusterURL = http://db1.oldsync.dev.lcip.org/ (should get automatically set by the TokenServer)
  • services.sync.tokenServerURI = http://auth.oldsync.dev.lcip.org/1.0/sync/1.1
  • firefox.accounts.remoteUrl = https://accounts.dev.lcip.org/?service=sync
    • identity.fxaccounts.enabled=true (NO LONGER NEEDED, BUT STILL CAN BE DEFINED)
    • services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
  • services.sync.log.appender.file.logOnError = Yes
  • services.sync.log.appender.file.logOnSuccess = Yes
  • services.sync.log.appender.file.level = Trace
  • identity.fxaccounts.auth.uri = https://api-accounts.dev.lcip.org/v1
  • And to hit the TokenServer/Sync 1.5 Dev services, change the following:
    • services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
    • services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5
  • And to hit the TokenServer/Sync 1.5 Stage services, change the following:
    • services.sync.clusterURL = https://sync-X-us-east-1.stage.mozaws.net/ (should get automatically set by the TokenServer)
    • services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5
  • And to hit the TokenServer/Sync 1.5 Prod services, change the following:
    • services.sync.clusterURL = https://sync-X-us-east-2.sync.services.mozilla.com/ (should get automatically set by the TokenServer)
    • services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5

• Other Dev Deployments:
  • AWS: api-accounts-legacy.dev.lcip.org
  • AWS: accounts-legacy.dev.lcip.org
  • AWS: api-accounts-onepw.dev.lcip.org

OAuth Dev Environment

• NOTE: TBD

OPs Mana and GitHub Pages

• NOTE: Talk to OPs for the links to Mana.
• Puppet Config: https://github.com/mozilla-services/puppet-config
  • and https://github.com/mozilla-services/puppet-config/tree/master/fxa
• Cloud Formation: https://github.com/mozilla-services/svcops
  • and https://github.com/mozilla-services/svcops/tree/master/cloudformations/firefox-accounts