MozillaWiki

TestEngineering/Services/TokenServerAndSyncLoadTesting: Difference between revisions

Page Discussion

TestEngineering/Services/TokenServerAndSyncLoadTesting (view source)

Revision as of 14:32, 1 March 2019

3,934 bytes removed ,  1 March 2019
→‎Using the Loads V1 Services Cluster for Sync 1.5 in Stage
 
(62 intermediate revisions by 3 users not shown)
Line 1: Line 1:
== Summary for Tokenserver, Verifier and Sync 1.5 ==
* NOTE: We currently have two Verifier stacks in Stage (and probably Production):
* Latest Results
** The standalone Browser_ID Verifier stack: See that Verifier sections below...
** Link to loads cluster: https://loads.services.mozilla.com/
** A Tokenserver+Verifier stack: See the TokenServer sections below...
*** Note: this now requires login privileges and a password
 
** Snapshots from StackDriver - TBD
** Snapshots from Kibana - TBD
* Latest Deployments
** TokenServer Stage: https://bugzilla.mozilla.org/show_bug.cgi?id=1014496
** TokenServer Prod: https://bugzilla.mozilla.org/show_bug.cgi?id=1027899
** Sync Server Stage: https://bugzilla.mozilla.org/show_bug.cgi?id=1026346
** Sync Server Prod: https://bugzilla.mozilla.org/show_bug.cgi?id=1026346
** Verifier Stage Deploy: ttps://bugzilla.mozilla.org/show_bug.cgi?id=1026644
** Verifier Prod Deploy: https://bugzilla.mozilla.org/show_bug.cgi?id=1027392
* In Progress
** Build out of Kibana dashboards
** Ongoing testing of Tokenserver, Verifier, and Sync releases
** Bug review and issue debug - there are a lot of issues to work on (see the long list near the bottom of the wiki)
* Bugs To Verify:
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=988095
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=1025767
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=1027444
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=1025735
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=735102
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=775395
* Planned
** Scaling for production traffic after release of Fx29
** Sync 1.5 migration work
** Operations readiness testing: See Bug 1006792
* Blockers
** none at this time
* Completed
** Pre-release load testing
** Previous load test results (short): http://loads.services.mozilla.com/
* Performance
** TBD


== Quick Verification Of Stage Deployments ==
== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
* This is a quick sanity test of the environment before getting started on load tests.
* TokenServer Stage environment:
 
* TokenServer+Verifier Stage environment:
From the browser: https://token.stage.mozaws.net
curl https://token.stage.mozaws.net
curl -I https://token.stage.mozaws.net
  Use the simple "make test" command from an install of tokenserver on the localhost or AWS instance.
  Use the simple "make test" command from an install of tokenserver on the localhost or AWS instance.
  cd loadtest
  cd loadtest
  make test SERVER_URL=https://token.stage.mozaws.net
  make test SERVER_URL=https://token.stage.mozaws.net
Alternate method:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Stage:
bin/sync-cli.js -e EMAIL -p PASSWORD --env stage -t COLLECTION
    where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms
* Verifier Stage environment:
* Verifier Stage environment:
In the browser: https://verifier.stage.mozaws.net/
curl https://verifier.stage.mozaws.net
curl -I https://verifier.stage.mozaws.net
  Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
  Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
  cd loadtest
  cd loadtest
  make test SERVER_URL=https://verifier.stage.mozaws.net
  make test SERVER_URL=https://verifier.stage.mozaws.net
* Sync Server Stage environment:
* Sync Server Stage environment:
  Install server-syncstorage to the local host or AWS instance (see below)
  Install server-syncstorage to the local host or AWS instance (see below)
Line 71: Line 57:
== Quick Verification Of Production Deployments ==
== Quick Verification Of Production Deployments ==
* This is a quick sanity test of the environment after a new deployment.
* This is a quick sanity test of the environment after a new deployment.
* Tokenserver Production Environment
 
  TBD
* Tokenserver+Verifier Production Environment
  See https://bugzilla.mozilla.org/show_bug.cgi?id=1001485
  In the browser: https://token.services.mozilla.com
  curl https://token.services.mozilla.com
curl -I https://token.services.mozilla.com
Then:
Use the test tool from here: https://github.com/edmoz/fxa-sync-client
Install and check all collection types for a known account in Production:
bin/sync-cli.js -e PROD-EMAIL -p PASSWORD -t COLLECTION
    where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms
 
* Verifier Production Environment
* Verifier Production Environment
In the browser: https://verifier.accounts.firefox.com
curl https://verifier.accounts.firefox.com
curl -I https://verifier.accounts.firefox.com
Then:
  Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
  Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
  cd loadtest
  cd loadtest
  make test SERVER_URL=https://verifier.accounts.firefox.com
  make test SERVER_URL=https://verifier.accounts.firefox.com
* Sync Server Stage environment
* Sync Server Stage environment
  TBD
  Sign in with a known FxA account and sync data with a current Production account (sync node).
Create a new FxA account and set up sync.


== Load Test Tool Client/Host ==
== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following link for more information: https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:  
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment


=== Creating a RHEL AWS instance ===
== Installing BrowserID-Verifier and the Loads tool on Localhost or AWS ==
* Pick a Region then Create Instance > Launch Instance
* Follow the prompts to create a basic, RHEL-flavored instance
* Use of the QA/Dev key pairs that have been set up for this:
** US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
** US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
* Once the instance is running, log in as "ec2-user"
 
* The following apps, tools, and libs will need to be installed for use with various Services applications:
** gcc, gcc-c++
** hg
** git
** python-devel
** automake, autoconf, and libtool  (required for libzmq, for easy_install)
** pip
** virtualenv
** node/npm
** zeromq 3.X
** gmp, gmp-devel
 
* Also, general rhel updates:
$ sudo yum -y update
and/or
$ sudo yum -y upgrade
 
* Now, the instance should be ready for installing and using the Loads tool.
 
=== Creating an Ubuntu AWS instance ===
* Pick a Region then Create Instance > Launch Instance
* Follow the prompts to create a basic, Ubuntu-flavored instance
* Use of the QA/Dev key pairs that have been set up for this:
** US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
** US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
* Once the instance is running, log in as "ubuntu"
 
* The following apps, tools, and libs will need to be installed for use with various Services applications:
** gcc, g++
** mercurial
** git
** python-setuptools, python-virtualenv, and python-dev
** automake, autoconf, libtool
** m4
** node/npm
** libzmq and zeromq 3.X
** gmp-5.1.3 or newer
 
* Also, general rhel updates:
$ sudo apt-get update
and/or
$ sudo apt-get upgrade
 
* Now, the instance should be ready for installing and using the Loads tool.
 
== Installing BrowserID-Verifier and the Loads tool on the AWS instance==
* Installation:
* Installation:
  $ git clone git://github.com/mozilla/browserid-verifier
  $ git clone git://github.com/mozilla/browserid-verifier
  $ cd browserid-verifier
  $ cd browserid-verifier
Note: You may want to install a specific branch for testing vs defaulting to Master
  $ npm install
  $ npm install
  $ npm test
  $ npm test
Line 158: Line 108:
  or
  or
  $ make bench SERVER_URL=https://verifier.stage.mozaws.net
  $ make bench SERVER_URL=https://verifier.stage.mozaws.net
  NOTE: The URL for Stage environment will most likely change on a frequent basis.
   
  NOTE: NOTE: This also hits the Stage mockmyid server.
Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.  
 
  The recommendation is to use 'make test' and 'make megabench' instead (see below)...
* And while we are at it...
  Note: The Stage Verifier hits the Stage mockmyid server
* Dev environment:
$ make test SERVER_URL=TBD
$ make bench SERVER_URL=TBD


* Production environment:  
* Production environment:  
Line 170: Line 117:
  $ make bench SERVER_URL=https://verifier.accounts.firefox.com
  $ make bench SERVER_URL=https://verifier.accounts.firefox.com


== Using the Loads Services Cluster for the Verifier ==
== Using the Loads V1 Services Cluster for the Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
Line 183: Line 130:


* REFs:
* REFs:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest


== Installing TokenServer and the Loads tool on the AWS instance ==
== Installing TokenServer+Verifier and the Loads tool on Localhost or AWS ==
* Installation:
* Installation:
  $ git clone https://github.com/mozilla-services/tokenserver
  $ git clone https://github.com/mozilla-services/tokenserver
  $ cd tokenserver
  $ cd tokenserver
Note: You may want to install a specific branch for testing vs defaulting to Master
  $ make build
  $ make build
  $ make test
  $ make test
Line 197: Line 145:
     Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com
     Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com


* Note: This will install a local copy of the Loads tool for use with TokenServer.
* Note: This will install a local copy of the Loads tool for use with TokenServer+Verifier.


== Running the load test against TokenServer in Stage ==
== Running the load test against TokenServer+Verifier in Stage ==
* Stage environment:
* Stage environment:
  $ make test SERVER_URL=https://token.stage.mozaws.net
  $ make test SERVER_URL=https://token.stage.mozaws.net
  $ make bench SERVER_URL=https://token.stage.mozaws.net
  $ make bench SERVER_URL=https://token.stage.mozaws.net
  NOTE: The URL for Stage environment will most likely change on a frequent basis.
   
  NOTE: This also hits the Stage Verifier, which in turns hits the Stage mockmyid server
Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.   
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
  Note: This also hits the Stage Verifier, which in turns hits the Stage mockmyid server


* And while we are at it...
* And while we are at it...
Line 215: Line 165:
  $ make bench SERVER_URL=https://token.services.mozilla.com
  $ make bench SERVER_URL=https://token.services.mozilla.com


== Using the Loads Services Cluster for TokenServer ==
== Using the Loads V1 Services Cluster for TokenServer+Verifier ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
Line 228: Line 178:


* REFs:
* REFs:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest


== Installing Sync 1.5 and the Loads tool on the AWS instance ==
== Installing Sync and load testing on Localhost or AWS ==
  Installation:
  Installation:
  $ git clone https://github.com/mozilla-services/server-syncstorage/
  $ git clone https://github.com/mozilla-services/syncstorage-loadtest/
  $ cd server-syncstorage
  $ cd syncstorage-loadtest
$ make build
  Note: You may want to install a specific branch for testing vs defaulting to Master
$ make test
$ pip install -r requirements.txt
$ cd loadtest
  $ make build
 
* Note: This will install a local copy of the Loads tool for use with Sync 1.5.


== Running the load test against Sync 1.5 in Stage ==
== Running the load test against Sync 1.5 in Stage ==
* Loads against specific Sync nodes in Stage
* Loads against specific Sync nodes in Stage
  $ make test SERVER_URL=https://your.storagenode.here#SECRET
  $ export SERVER_URL=https://your.storagenode.here#SECRET
$ make bench SERVER_URL=https://your.storagenode.here#SECRET
  Sync Stage nodes:
  Sync Stage nodes:
     https://sync-1-us-east-1.stage.mozaws.net
     https://sync-1-us-east-1.stage.mozaws.net
     https://sync-2-us-east-1.stage.mozaws.net
     https://sync-2-us-east-1.stage.mozaws.net
     https://sync-3-us-east-1.stage.mozaws.net
     ...etc...
NOTE: The Stage sync nodes are likely to change frequently, so verify the URLs.
    See https://wiki.mozilla.org/QA/Services/FxATestEnvironments#Sync_1.5_Stage_Environment
   
   
  NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.
  NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.


== Using the Loads Services Cluster for Sync 1.5 in Stage ==
* Load testing with Molotov: https://molotov.readthedocs.io/en/stable/
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
  $ bin/molotov [commands] loadtest.py
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
  $ make megabench SERVER_URL=https://your.storagenode.here#SECRET


* REFs:
== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
* loadtesting from server-syncstorage has been deprecated, please refer to mozilla-services/syncstorage-loadtest
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest


== Running a combined load test against TokenServer and Sync 1.5 in Stage ==
== Running a combined load test against TokenServer+Verifier and Sync 1.5 in Stage ==
* A combined loads test against TokenServer and Sync 1.5 in Stage
* A combined loads test against TokenServer and Sync 1.5 in Stage
* This is done via the server-syncstorage directory that was cloned and built above
* This is done via the server-syncstorage directory that was cloned and built above
Line 277: Line 216:
  $ make test SERVER_URL=https://token.stage.mozaws.net
  $ make test SERVER_URL=https://token.stage.mozaws.net
  $ make bench SERVER_URL=https://token.stage.mozaws.net
  $ make bench SERVER_URL=https://token.stage.mozaws.net
  See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer_Stage_Environment
  See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer.2BVerifier_Stage_Environment
Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.   
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
Note: The Stage Tokenserver hits the Stage Verifier, which, in turn, hits the mockmyid server.


* And while we are at it...
* And while we are at it...
Line 293: Line 236:
  and https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer_and_Sync_1.5_Dev_Environments
  and https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer_and_Sync_1.5_Dev_Environments


== Using the Loads Services Cluster for a combined load test in Stage ==
== Using the Loads V1 Services Cluster for a combined load test in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
Line 306: Line 249:


* REFs:
* REFs:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest


Line 314: Line 257:


* Config files
* Config files
** For make test (TokenServer, Sync, Combined):
** For make test (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of hits
*** Number of hits
*** Number of concurrent users
*** Number of concurrent users


** For make bench (TokenServer, Sync, Combined):
** For make bench (BrowserID-Verifier, TokenServer, Sync, Combined):
*** Number of concurrent users
*** Number of concurrent users
*** Duration of test
*** Duration of test


** For make megabench (using the LoadsCluster with TokenSerer, Sync, Combined):
** For make megabench (using the LoadsCluster with BrowserID-Verifier, TokenSerer, Sync, Combined):
*** Number of concurrent users
*** Number of concurrent users
*** Duration of test
*** Duration of test
*** Include file (this is code dependent)
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)  
*** Observer (this can be email or irc - the default is irc #services-dev channel)  
*** SSH (the user account needed to SSH into the loads cluster - the default is ubuntu)


* In the code
* Tokenserver load test code
* The Tokenserver load test can be configured - see the following lines: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L14-L51
** The Tokenserver load test can be configured - see the following lines:
* The Verifier load test can be configured - see the following lines: https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py#L13-L53
** Basic Settings: https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
* The Sync Server load test can be configured - see the following lines: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L83
** MockMyID: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L19-L36
** Percentages: https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L39-L51
 
* Verifier load test code
** The Verifier load test can be configured - see the following lines:
** Various settings: https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py#L13-L53
 
* Sync Server load test code
** The Sync Server load test can be configured - see the following lines:
** Setting MockMyID: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L26-L45
** Setting test distributions: https://github.com/mozilla-services/server-syncstorage/blob/master/loadtest/stress.py#L48-L83


* REFs:  
* REFs:  
Line 402: Line 355:
* 1. Important logs for TokenServer (per server)
* 1. Important logs for TokenServer (per server)
** /media/ephemeral0/logs/
** /media/ephemeral0/logs/
** /media/ephemeral0/logs/nginx/access.log
** /media/ephemeral0/nginx/logs/default.access.log
** /media/ephemeral0/logs/nginx/error.log
** /media/ephemeral0/nginx/logs/default.error.log
** /media/ephemeral0/nginx/logs/tokenserver.access.log
** /media/ephemeral0/nginx/logs/tokenserver.error.log
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.error.log
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/token.log.*
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/logs/tokenserver/process_account_deletions.log
** /media/ephemeral0/squid/access.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.log
** /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /var/log/circus.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stdout.log
** /var/log/hekad/tokenserver.stderr.log  
** /var/log/hekad/tokenserver.stderr.log


* 2. Important logs for Verifier (per server)
* 2. Important logs for Verifier (per server)
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
** GONE: /media/ephemeral0/heka/hekad_err.log
** GONE: /media/ephemeral0/heka/hekad_out.log
** GONE: /media/ephemeral0/nginx/logs/access.log
** GONE: /media/ephemeral0/nginx/logs/error.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
** /media/ephemeral0/nginx/logs/squid/access.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/squid/access.log
** /var/log/circus.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stderr.log
** /var/log/hekad/fxa-browserid_verifier.stdout.log  
** /var/log/hekad/fxa-browserid_verifier.stdout.log  
Line 442: Line 400:
  NOTE: Values can be tweaked here:
  NOTE: Values can be tweaked here:
     https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L58-L60
     https://github.com/mozilla-services/tokenserver/blob/master/loadtest/loadtest.py#L58-L60
 
  Also, it may be the case that the following errors are "acceptable" if TS Stage is larger than Verifier Stage:
The following types of errors are known:
/media/ephemeral0/logs/tokenserver/token.error.log
    Exception KeyError: KeyError(49564400,) in <module 'threading'...
/media/ephemeral0/logs/tokenserver/token.log
    ..."Starting new HTTP connection (9): 127.0.0.1", "hostname": ...
    {"error": "StopIteration()", "traceback": "Uncaught exception:\n 
    File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
    ..."Connection pool is full, discarding connection: 127.0.0.1", "...
Also, any 499s are probably an artifact of the current (V1) load test.
REF:
https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
https://bugzilla.mozilla.org/show_bug.cgi?id=1040397
  OLD: Also, it may be the case that the following errors are "acceptable" if TS Stage is larger than Verifier Stage:
  /media/ephemeral0/logs/tokenserver/token.error.log
  /media/ephemeral0/logs/tokenserver/token.error.log
  Verifier-related errors of these types:
  Verifier-related errors of these types:
Line 451: Line 422:


* Acceptable Verifier errors:
* Acceptable Verifier errors:
  In the verifier and squid logs:
  The verifier_out.log will show errors of the following types:
  References to mozilla.org and login.mozilla.org - part of the "invalid domain" tests
  result: 'failure',\n  reason: 'untrusted issuer...'
  In the verifier logs:
result: 'failure',\n  reason: 'expired'
  References to https://secret.mozilla.com, which are defined in the browserid-verifier load test
  result: 'failure',\n  reason: 'algorithms do not match'
https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py#L77 for example
  result: 'failure',\n  reason: 'audience mismatch: scheme mismatch'
Also, any 499s in the nginx logs are probably an artifact of the current (V1) load test.


* Acceptable Sync node errors:
* Acceptable Sync node errors:
Line 466: Line 438:
           agents = 5
           agents = 5
  Ideally, the overall percentage of 404s should drop the longer the load test.
  Ideally, the overall percentage of 404s should drop the longer the load test.
Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
although they may show up in the logs after running the remote integration tests.
Also, any 499s are probably an artifact of the current (V1) load test.
In /var/log/hekad/sync_1_5.stderr.log
You may see some Decoder 'Sync-1_5-SlowQuery-MySqlSlowQueryDecoder' error: Failed parsing
and a lot of BSO INSERTs
In /media/ephemeral0/logs/sync/sync.err
You should see expected skew and QueuePool messages and Deprecation warnings
Also, these are known
Exception SystemExit
Exception KeyError
This is probably https://bugzilla.mozilla.org/show_bug.cgi?id=1040397


== Monitoring TS and Sync Stage ==
== Monitoring TS and Sync Stage ==
Line 476: Line 462:
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster


* Monitoring TS/Verifier/Sync Stage:
* For all other monitoring, see the following section:
** Stackdriver
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Monitoring_the_Stage_Environment
*** Stage TS + FxA + Sync 1.5 meta-dash: https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** Kibana
*** https://kibana.shared.us-east-1.stage.mozaws.net/
*** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20Web%20Logs
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Token%20App%20Logs%20POC
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20Nginx%20Errors
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20App%20Logs
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
 
** Heka
*** https://heka.shared.us-east-1.stage.mozaws.net/
*** https://heka.shared.us-east-1.stage.mozaws.net/#health
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-HTTPStatus/outputs/Sync-1_5-HTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagehistory.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagebookmarks.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storageforms.cbuf
 
** REF: https://mana.mozilla.org/wiki/display/SVCOPS/Sync+1.5+Dash+Boards


== Performance Testing Information ==
== Performance Testing Information ==
Line 515: Line 479:


== Known Bugs, Issues, and Tasks ==
== Known Bugs, Issues, and Tasks ==
* Tokenserver, Verifier, and Sync
* Tokenserver:  
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=907475
** Repo: https://github.com/mozilla-services/tokenserver/issues
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=907479
** Bugzilla: http://mzl.la/1s4qZKn
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=956217
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=1008066
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=1014411
 
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=988095
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=982412
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=982415
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=985794
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=996870
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=956615
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=1001485


** https://github.com/mozilla/browserid-verifier/issues/42
* BrowserID-Verifier:
** https://github.com/mozilla/browserid-verifier/issues/49
** Repo: https://github.com/mozilla/browserid-verifier/issues
** https://github.com/mozilla/browserid-verifier/issues/56
** Bugzilla: no specific cateogory
** https://github.com/mozilla-services/tokenserver/issues/66


** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=735102
* Sync:
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=776777
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=799727
** Bugzilla: http://mzl.la/VUrYQ5
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=982417
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=989117
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=992420
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=959034
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=996819
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=1007987
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=1016722
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=1008802
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=1025735
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=1031855


* OPs and Infrastructure
* OPs and Infrastructure
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=981933
** https://github.com/mozilla-services/puppet-config/issues
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=982316
** https://github.com/mozilla-services/svcops/issues
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=982985
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=993033
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=996199
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=998050
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=1009389
** OPs: https://bugzilla.mozilla.org/show_bug.cgi?id=1006031
** https://github.com/mozilla-services/puppet-config/issues/287
** https://github.com/mozilla-services/puppet-config/issues/292
** https://github.com/mozilla-services/puppet-config/issues/443
** https://github.com/mozilla-services/puppet-config/pull/546
** https://github.com/mozilla-services/puppet-config/pull/613


* Loads Tool and Cluster
* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues/222
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads/issues/234
** https://github.com/mozilla-services/loads-aws/issues
** https://github.com/mozilla-services/loads/issues/235
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads/issues/251
** https://github.com/mozilla-services/loads/issues/257
** https://github.com/mozilla-services/loads/issues/259
** https://github.com/mozilla-services/loads/issues/265
** https://github.com/mozilla-services/loads/issues/266
** https://github.com/mozilla-services/loads-web/issues/24


== References ==
== References ==
Rbillings
Confirmed users
1,504

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/995173...1208327"