MozillaWiki

TestEngineering/Services/TokenServerAndSyncLoadTesting: Difference between revisions

Page Discussion

TestEngineering/Services/TokenServerAndSyncLoadTesting (view source)

Revision as of 14:32, 1 March 2019

653 bytes removed ,  1 March 2019
→‎Using the Loads V1 Services Cluster for Sync 1.5 in Stage
 
(89 intermediate revisions by 3 users not shown)
Line 1: Line 1:
== Summary for Tokenserver, Verifier and Sync 1.5 ==
* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
* Latest Results
** The standalone Browser_ID Verifier stack: See that Verifier sections below...
** Link to loads cluster: https://loads.services.mozilla.com/
** A Tokenserver+Verifier stack: See the TokenServer sections below...
*** Note: this now requires login privileges and a password
 
** Snapshots from StackDriver - TBD
** Snapshots from Kibana - TBD
* Latest Deployments
** TokenServer Stage: https://bugzilla.mozilla.org/show_bug.cgi?id=1014496
** TokenServer Prod: TBD
** Sync Server Stage: https://bugzilla.mozilla.org/show_bug.cgi?id=1011755
** Sync Server Prod: https://bugzilla.mozilla.org/show_bug.cgi?id=1011755
** Verifier Stage Deploy: https://bugzilla.mozilla.org/show_bug.cgi?id=996763
** Verifier Prod Deploy: https://bugzilla.mozilla.org/show_bug.cgi?id=1009096
* In Progress
** Build out of Kibana dashboards
** Tokenserver 503s in Stage: https://github.com/mozilla-services/tokenserver/issues/59
** Verifier 503s in Stage:
*** https://github.com/mozilla/browserid-verifier/issues/48
*** https://github.com/mozilla-services/puppet-config/pull/477
** Bug review and issue debug - there are a lot of issues to work on (see the long list below)
* Bugs To Verify:
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=985784
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=969184
** Tokenserver: https://github.com/mozilla-services/tokenserver/issues/62
* Planned
** Scaling for production traffic after release of Fx29
** Debug of CPU usage during TokenServer/Verifier load test
** Debug of CPU usage during individual Sync node load test
* Blockers
** none at this time
* Completed
** Pre-release load testing
** Previous load test results (short): http://loads.services.mozilla.com/
* Performance
** TBD


== Quick Verification Of Stage Deployments ==
== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
* This is a quick sanity test of the environment before getting started on load tests.
* TokenServer Stage environment: TBD
 
  For now, just use the simple "make test" or "make bench" command from an install of tokenserver
* TokenServer+Verifier Stage environment:
on the localhost or AWS instance.
  From the browser: https://token.stage.mozaws.net
* Verifier Stage environment: TBD
curl https://token.stage.mozaws.net
  For now, just use the simple "make test" or "make bench" command from an install of browserid-verifier
curl -I https://token.stage.mozaws.net
on the localhost or AWS instance.
Use the simple "make test" command from an install of tokenserver on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://token.stage.mozaws.net
Alternate method:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Stage:
bin/sync-cli.js -e EMAIL -p PASSWORD --env stage -t COLLECTION
    where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms
 
* Verifier Stage environment:
  In the browser: https://verifier.stage.mozaws.net/
curl https://verifier.stage.mozaws.net
curl -I https://verifier.stage.mozaws.net
Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
make test SERVER_URL=https://verifier.stage.mozaws.net
 
* Sync Server Stage environment:
* Sync Server Stage environment:
  Install server-syncstorage to the local host or AWS instance (see below)
  Install server-syncstorage to the local host or AWS instance (see below)
Line 67: Line 55:
** See also: https://bugzilla.mozilla.org/show_bug.cgi?id=1006675
** See also: https://bugzilla.mozilla.org/show_bug.cgi?id=1006675


== Load Test Tool Client/Host ==
== Quick Verification Of Production Deployments ==
* It is always best to configure an AWS instance as the host for all load testing.
* This is a quick sanity test of the environment after a new deployment.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following link for more information: https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Loads_Services_Cluster_Environment
 
=== Creating a RHEL AWS instance ===
* Pick a Region then Create Instance > Launch Instance
* Follow the prompts to create a basic, RHEL-flavored instance
* Use of the QA/Dev key pairs that have been set up for this:
** US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
** US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
* Once the instance is running, log in as "ec2-user"


* The following apps, tools, and libs will need to be installed for use with various Services applications:
* Tokenserver+Verifier Production Environment
** gcc, gcc-c++
In the browser: https://token.services.mozilla.com
** hg
curl https://token.services.mozilla.com
** git
curl -I https://token.services.mozilla.com
** python-devel
Then:
** automake, autoconf, and libtool  (required for libzmq, for easy_install)
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
** pip
Install and check all collection types for a known account in Production:
** virtualenv
bin/sync-cli.js -e PROD-EMAIL -p PASSWORD -t COLLECTION
** node/npm
    where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms
** zeromq 3.X
** gmp, gmp-devel


* Also, general rhel updates:
* Verifier Production Environment
  $ sudo yum -y update
In the browser: https://verifier.accounts.firefox.com
  and/or
curl https://verifier.accounts.firefox.com
  $ sudo yum -y upgrade
  curl -I https://verifier.accounts.firefox.com
Then:
  Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
cd loadtest
  make test SERVER_URL=https://verifier.accounts.firefox.com


* Now, the instance should be ready for installing and using the Loads tool.
* Sync Server Stage environment
Sign in with a known FxA account and sync data with a current Production account (sync node).
Create a new FxA account and set up sync.


=== Creating an Ubuntu AWS instance ===
== Load Test Tool Client/Host ==
* Pick a Region then Create Instance > Launch Instance
* It is always best to configure an AWS instance as the host for all load testing.
* Follow the prompts to create a basic, Ubuntu-flavored instance
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:  
* Use of the QA/Dev key pairs that have been set up for this:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
* Once the instance is running, log in as "ubuntu"
 
* The following apps, tools, and libs will need to be installed for use with various Services applications:
** gcc, g++
** mercurial
** git
** python-setuptools, python-virtualenv, and python-dev
** automake, autoconf, libtool
** m4
** node/npm
** libzmq and zeromq 3.X
** gmp-5.1.3 or newer
 
* Also, general rhel updates:
$ sudo apt-get update
and/or
$ sudo apt-get upgrade
 
* Now, the instance should be ready for installing and using the Loads tool.


== Installing BrowserID-Verifier and the Loads tool on the AWS instance==
== Installing BrowserID-Verifier and the Loads tool on Localhost or AWS ==
* Installation:
* Installation:
  $ git clone git://github.com/mozilla/browserid-verifier
  $ git clone git://github.com/mozilla/browserid-verifier
  $ cd browserid-verifier
  $ cd browserid-verifier
Note: You may want to install a specific branch for testing vs defaulting to Master
  $ npm install
  $ npm install
  $ npm test
  $ npm test
Line 144: Line 108:
  or
  or
  $ make bench SERVER_URL=https://verifier.stage.mozaws.net
  $ make bench SERVER_URL=https://verifier.stage.mozaws.net
  NOTE: The URL for Stage environment will most likely change on a frequent basis.
   
  NOTE: NOTE: This also hits the Stage mockmyid server.
Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.  
 
  The recommendation is to use 'make test' and 'make megabench' instead (see below)...
* And while we are at it...
  Note: The Stage Verifier hits the Stage mockmyid server
* Dev environment:
$ make test SERVER_URL=TBD
$ make bench SERVER_URL=TBD


* Production environment:  
* Production environment:  
Line 156: Line 117:
  $ make bench SERVER_URL=https://verifier.accounts.firefox.com
  $ make bench SERVER_URL=https://verifier.accounts.firefox.com


== Using the Loads Services Cluster for the Verifier ==
== Using the Loads V1 Services Cluster for the Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
Line 169: Line 130:


* REFs:
* REFs:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Loads_Services_Cluster_Environment
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest


== Installing TokenServer and the Loads tool on the AWS instance ==
== Installing TokenServer+Verifier and the Loads tool on Localhost or AWS ==
* Installation:
* Installation:
  $ git clone https://github.com/mozilla-services/tokenserver
  $ git clone https://github.com/mozilla-services/tokenserver
  $ cd tokenserver
  $ cd tokenserver
Note: You may want to install a specific branch for testing vs defaulting to Master
  $ make build
  $ make build
  $ make test
  $ make test
Line 183: Line 145:
     Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com
     Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com


* Note: This will install a local copy of the Loads tool for use with TokenServer.
* Note: This will install a local copy of the Loads tool for use with TokenServer+Verifier.


== Running the load test against TokenServer in Stage ==
== Running the load test against TokenServer+Verifier in Stage ==
* Stage environment:
* Stage environment:
  $ make test SERVER_URL=https://token.stage.mozaws.net
  $ make test SERVER_URL=https://token.stage.mozaws.net
  $ make bench SERVER_URL=https://token.stage.mozaws.net
  $ make bench SERVER_URL=https://token.stage.mozaws.net
  NOTE: The URL for Stage environment will most likely change on a frequent basis.
   
  NOTE: This also hits the Stage Verifier, which in turns hits the Stage mockmyid server
Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.   
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
  Note: This also hits the Stage Verifier, which in turns hits the Stage mockmyid server


* And while we are at it...
* And while we are at it...
Line 201: Line 165:
  $ make bench SERVER_URL=https://token.services.mozilla.com
  $ make bench SERVER_URL=https://token.services.mozilla.com


== Using the Loads Services Cluster for TokenServer ==
== Using the Loads V1 Services Cluster for TokenServer+Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
Line 214: Line 178:


* REFs:
* REFs:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Loads_Services_Cluster_Environment
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest


== Installing Sync 1.5 and the Loads tool on the AWS instance ==
== Installing Sync and load testing on Localhost or AWS ==
  Installation:
  Installation:
  $ git clone https://github.com/mozilla-services/server-syncstorage/
  $ git clone https://github.com/mozilla-services/syncstorage-loadtest/
  $ cd server-syncstorage
  $ cd syncstorage-loadtest
$ make build
  Note: You may want to install a specific branch for testing vs defaulting to Master
$ make test
$ pip install -r requirements.txt
$ cd loadtest
  $ make build
 
* Note: This will install a local copy of the Loads tool for use with Sync 1.5.


== Running the load test against Sync 1.5 in Stage ==
== Running the load test against Sync 1.5 in Stage ==
* Loads against specific Sync nodes in Stage
* Loads against specific Sync nodes in Stage
  $ make test SERVER_URL=https://your.storagenode.here#SECRET
  $ export SERVER_URL=https://your.storagenode.here#SECRET
$ make bench SERVER_URL=https://your.storagenode.here#SECRET
  Sync Stage nodes:
  Sync Stage nodes:
     https://sync-1-us-east-1.stage.mozaws.net
     https://sync-1-us-east-1.stage.mozaws.net
     https://sync-2-us-east-1.stage.mozaws.net
     https://sync-2-us-east-1.stage.mozaws.net
     https://sync-3-us-east-1.stage.mozaws.net
     ...etc...
NOTE: The Stage sync nodes are likely to change frequently, so verify the URLs.
    See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Sync_1.5_Stage_Environment
   
   
  NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.
  NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.


== Using the Loads Services Cluster for Sync 1.5 in Stage ==
* Load testing with Molotov: https://molotov.readthedocs.io/en/stable/
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
  $ bin/molotov [commands] loadtest.py
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
  $ make megabench SERVER_URL=https://your.storagenode.here#SECRET


* REFs:
== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Loads_Services_Cluster_Environment
* loadtesting from server-syncstorage has been deprecated, please refer to mozilla-services/syncstorage-loadtest
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest


== Running a combined load test against TokenServer and Sync 1.5 in Stage ==
== Running a combined load test against TokenServer+Verifier and Sync 1.5 in Stage ==
* A combined loads test against TokenServer and Sync 1.5 in Stage
* A combined loads test against TokenServer and Sync 1.5 in Stage
* This is done via the server-syncstorage directory that was cloned and built above
* This is done via the server-syncstorage directory that was cloned and built above
Line 263: Line 216:
  $ make test SERVER_URL=https://token.stage.mozaws.net
  $ make test SERVER_URL=https://token.stage.mozaws.net
  $ make bench SERVER_URL=https://token.stage.mozaws.net
  $ make bench SERVER_URL=https://token.stage.mozaws.net
  See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#TokenServer_Stage_Environment
  See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer.2BVerifier_Stage_Environment
Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.   
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Tokenserver hits the Stage Verifier, which, in turn, hits the mockmyid server.


* And while we are at it...
* And while we are at it...
Line 277: Line 234:
   
   
  See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA.2C_TokenServer.2C_and_Sync_Production_Environments
  See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#FxA.2C_TokenServer.2C_and_Sync_Production_Environments
  and https://wiki.mozilla.org/QA/Services/FxATestEnvironments#TokenServer_and_Sync_1.5_Dev_Environments
  and https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer_and_Sync_1.5_Dev_Environments


== Using the Loads Services Cluster for a combined load test in Stage ==
== Using the Loads V1 Services Cluster for a combined load test in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
Line 292: Line 249:


* REFs:
* REFs:
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Loads_Services_Cluster_Environment
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest


== Configuring The Load Tests ==
== Configuring The Load Tests ==
* The TokenServer, Sync, and Combined load tests work with config files that can be edited to change how the load tests are run.
* Makefile
** The SERVER_URL constant can be changed.
 
* Config files
** For make test (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of hits
*** Number of concurrent users
 
** For make bench (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of concurrent users
*** Duration of test
 
** For make megabench (using the LoadsCluster with BrowserID-Verifier, TokenSerer, Sync, Combined):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)
*** SSH (the user account needed to SSH into the loads cluster - the default is ubuntu)


* For make test (TokenServer, Sync, Combined):
* Tokenserver load test code
** Number of hits
** The Tokenserver load test can be configured - see the following lines:
** Number of concurrent users
** Basic Settings: https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** MockMyID: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L19-L36
** Percentages: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L39-L51


* For make bench (TokenServer, Sync, Combined):
* Verifier load test code
** Number of concurrent users
** The Verifier load test can be configured - see the following lines:
** Duration of test
** Various settings: https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py#L13-L53


* For make megabench (using the LoadsCluster with TokenSerer, Sync, Combined):
* Sync Server load test code
** Number of concurrent users
** The Sync Server load test can be configured - see the following lines:
** Duration of test
** Setting MockMyID: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L26-L45
** Include file (this is code dependent)
** Setting test distributions: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L48-L83
** Python dependencies (this is code dependent)
** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
** Observer (this can be email or irc - the default is irc #services-dev channel)


* REF: https://github.com/mozilla-services/tokenserver/tree/master/loadtest/config
* REFs:  
* and https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest/config
** https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py
** https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py
** https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py


== Test Coverage and Stats ==
== Test Coverage and Stats ==
Line 362: Line 338:
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.
** If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads
 
* Tokenserver Custom Metrics
** addFailure
 
* Verifier Custom Metrics
** addFailure
 
* Sync Custom Metrics
** addFailure
 
* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads


== Debugging the Issues ==
== Debugging the Issues ==
Line 369: Line 355:
* 1. Important logs for TokenServer (per server)
* 1. Important logs for TokenServer (per server)
** /media/ephemeral0/logs/
** /media/ephemeral0/logs/
** /media/ephemeral0/logs/nginx/access.log
** /media/ephemeral0/nginx/logs/default.access.log
** /media/ephemeral0/logs/nginx/error.log
** /media/ephemeral0/nginx/logs/default.error.log
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log


* 2. Important logs for Verifier (per server)
* 2. Important logs for Verifier (per server)
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/heka/hekad_err.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/heka/hekad_out.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/error.log
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/squid/access.log
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log  


* 3. Important error logs for Sync (per Sync node)
* 3. Important error logs for Sync (per Sync node)
Line 400: Line 400:
  NOTE: Values can be tweaked here:
  NOTE: Values can be tweaked here:
     https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L58-L60
     https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L58-L60
 
  Also, it may be the case that the following errors are "acceptable" if TS Stage is larger than Verifier Stage:
The following types of errors are known:
/media/ephemeral0/logs/tokenserver/token.error.log
    Exception KeyError: KeyError(49564400,) in <module 'threading'...
/media/ephemeral0/logs/tokenserver/token.log
    ..."Starting new HTTP connection (9): 127.0.0.1", "hostname": ...
    {"error": "StopIteration()", "traceback": "Uncaught exception:\n 
    File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
    ..."Connection pool is full, discarding connection: 127.0.0.1", "...
Also, any 499s are probably an artifact of the current (V1) load test.
REF:
https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
https://bugzilla.mozilla.org/show_bug.cgi?id=1040397
  OLD: Also, it may be the case that the following errors are "acceptable" if TS Stage is larger than Verifier Stage:
  /media/ephemeral0/logs/tokenserver/token.error.log
  /media/ephemeral0/logs/tokenserver/token.error.log
  Verifier-related errors of these types:
  Verifier-related errors of these types:
Line 409: Line 422:


* Acceptable Verifier errors:
* Acceptable Verifier errors:
  In the verifier and squid logs:
  The verifier_out.log will show errors of the following types:
  References to mozilla.org and login.mozilla.org - part of the "invalid domain" tests
  result: 'failure',\n  reason: 'untrusted issuer...'
  In the verifier logs:
result: 'failure',\n  reason: 'expired'
  References to https://secret.mozilla.com, which are defined in the browserid-verifier load test
  result: 'failure',\n  reason: 'algorithms do not match'
https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py#L77 for example
  result: 'failure',\n  reason: 'audience mismatch: scheme mismatch'
Also, any 499s in the nginx logs are probably an artifact of the current (V1) load test.


* Acceptable Sync node errors:
* Acceptable Sync node errors:
Line 424: Line 438:
           agents = 5
           agents = 5
  Ideally, the overall percentage of 404s should drop the longer the load test.
  Ideally, the overall percentage of 404s should drop the longer the load test.
Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
although they may show up in the logs after running the remote integration tests.
Also, any 499s are probably an artifact of the current (V1) load test.
In /var/log/hekad/sync_1_5.stderr.log
You may see some Decoder 'Sync-1_5-SlowQuery-MySqlSlowQueryDecoder' error: Failed parsing
and a lot of BSO INSERTs
In /media/ephemeral0/logs/sync/sync.err
You should see expected skew and QueuePool messages and Deprecation warnings
Also, these are known
Exception SystemExit
Exception KeyError
This is probably https://bugzilla.mozilla.org/show_bug.cgi?id=1040397


== Monitoring TS and Sync Stage ==
== Monitoring TS and Sync Stage ==
Line 429: Line 457:
** http://loads.services.mozilla.com
** http://loads.services.mozilla.com
* Cluster status
* Cluster status
** Check from any loadtest folder or loads install:
** Check directly from the Loads Cluster dashboard:
  ./bin/loads-runner --ping-broker --ssh=ubuntu@loads.services.mozilla.com
  Agents statuses
  ./bin/loads-runner --check-cluster --ssh=ubuntu@loads.services.mozilla.com
  Launch a health check on all agents
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster


* Monitoring TS/Verifier/Sync Stage:
* For all other monitoring, see the following section:
** Stackdriver
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Monitoring_the_Stage_Environment
*** Stage TS + FxA + Sync 1.5 meta-dash: https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** Kibana
*** https://kibana.shared.us-east-1.stage.mozaws.net/
*** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20Web%20Logs
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Token%20App%20Logs%20POC
** Heka
*** https://heka.shared.us-east-1.stage.mozaws.net/
*** https://heka.shared.us-east-1.stage.mozaws.net/#health
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
 
** REF: https://mana.mozilla.org/wiki/display/SVCOPS/Sync+1.5+Dash+Boards


== Performance Testing Information ==
== Performance Testing Information ==
Line 463: Line 479:


== Known Bugs, Issues, and Tasks ==
== Known Bugs, Issues, and Tasks ==
* Tokenserver, Verifier, and Sync
* Tokenserver:  
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=907475
** Repo: https://github.com/mozilla-services/tokenserver/issues
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=907479
** Bugzilla: http://mzl.la/1s4qZKn
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=956217
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=1008066
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=1014411
 
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=988095
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=982412
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=982415
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=985794
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=996870
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=956615
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=1001485


** https://github.com/mozilla/browserid-verifier/issues/42
* BrowserID-Verifier:
** https://github.com/mozilla/browserid-verifier/issues/49
** Repo: https://github.com/mozilla/browserid-verifier/issues
** Bugzilla: no specific cateogory


** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=776777
* Sync:
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=799727
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=982417
** Bugzilla: http://mzl.la/VUrYQ5
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=989117
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=992420
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=959034
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=996819
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=1007987
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=1016722
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=1008802


* OPs and Infrastructure
* OPs and Infrastructure
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=981933
** https://github.com/mozilla-services/puppet-config/issues
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=982316
** https://github.com/mozilla-services/svcops/issues
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=982985
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=993033
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=996199
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=998050
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=1009389
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=1006031
** https://github.com/mozilla-services/puppet-config/issues/287
** https://github.com/mozilla-services/puppet-config/issues/292
** https://github.com/mozilla-services/puppet-config/issues/443
** https://github.com/mozilla-services/puppet-config/pull/546


* Loads Tool and Cluster
* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues/222
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads/issues/234
** https://github.com/mozilla-services/loads-aws/issues
** https://github.com/mozilla-services/loads/issues/235
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads/issues/251
** https://github.com/mozilla-services/loads/issues/257
** https://github.com/mozilla-services/loads/issues/259
** https://github.com/mozilla-services/loads/issues/265
** https://github.com/mozilla-services/loads/issues/266
** https://github.com/mozilla-services/loads-web/issues/24


== References ==
== References ==
Line 533: Line 516:
** https://wiki.mozilla.org/Services/Sagrada/TokenServer
** https://wiki.mozilla.org/Services/Sagrada/TokenServer
** https://docs.services.mozilla.com/sync/
** https://docs.services.mozilla.com/sync/
* The QA Test Environments: https://wiki.mozilla.org/QA/Services/FxATestEnvironments
* The QA Test Environments:  
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
* Deploying the FxA Load Test environment for broker/agents usage:
* Deploying the FxA Load Test environment for broker/agents usage:
** https://github.com/mozilla/fxa-deployment
** https://github.com/mozilla/fxa-deployment
Rbillings
Confirmed users
1,504

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/988979...1208327"