MozillaWiki

TestEngineering/Services/TokenServerAndSyncLoadTesting: Difference between revisions

Page Discussion

TestEngineering/Services/TokenServerAndSyncLoadTesting (view source)

Revision as of 14:32, 1 March 2019

5,297 bytes removed ,  1 March 2019
→‎Using the Loads V1 Services Cluster for Sync 1.5 in Stage
 
(24 intermediate revisions by 3 users not shown)
Line 3: Line 3:
** A Tokenserver+Verifier stack: See the TokenServer sections below...
** A Tokenserver+Verifier stack: See the TokenServer sections below...


== Summary for Tokenserver, Verifier, and Sync 1.5 ==
* Latest Results
** Link to loads cluster: https://loads.services.mozilla.com/
*** Note: this now requires login privileges and a password
** Snapshots from StackDriver - TBD
** Snapshots from Kibana - TBD
* Latest Deployments
** TokenServer Stage: https://bugzilla.mozilla.org/show_bug.cgi?id=1014496
** TokenServer Prod: https://bugzilla.mozilla.org/show_bug.cgi?id=1027899
** Sync Server Stage: https://bugzilla.mozilla.org/show_bug.cgi?id=1026346
** Sync Server Prod: https://bugzilla.mozilla.org/show_bug.cgi?id=1026346
** Verifier Stage Deploy: ttps://bugzilla.mozilla.org/show_bug.cgi?id=1026644
** Verifier Prod Deploy: https://bugzilla.mozilla.org/show_bug.cgi?id=1027392
* In Progress
** Build out of Kibana dashboards
** Ongoing testing of Tokenserver, Verifier, and Sync releases
** Bug review and issue debug - there are a lot of issues to work on (see the long list near the bottom of the wiki)
* Bugs To Verify:
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=988095
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=1025767
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=1027444
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=1025735
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=735102
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=775395
* Planned
** Scaling for production traffic after release of Fx29
** Sync 1.5 migration work
** Operations readiness testing: See Bug 1006792
* Blockers
** none at this time
* Completed
** Pre-release load testing
** Previous load test results (short): http://loads.services.mozilla.com/
* Performance
** TBD


== Quick Verification Of Stage Deployments ==
== Quick Verification Of Stage Deployments ==
Line 44: Line 8:


* TokenServer+Verifier Stage environment:
* TokenServer+Verifier Stage environment:
From the browser: https://token.stage.mozaws.net
curl https://token.stage.mozaws.net
curl -I https://token.stage.mozaws.net
  Use the simple "make test" command from an install of tokenserver on the localhost or AWS instance.
  Use the simple "make test" command from an install of tokenserver on the localhost or AWS instance.
  cd loadtest
  cd loadtest
Line 53: Line 21:
  bin/sync-cli.js -e EMAIL -p PASSWORD --env stage -t COLLECTION
  bin/sync-cli.js -e EMAIL -p PASSWORD --env stage -t COLLECTION
     where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms
     where -t is one of bookmarks,history,passwords,tabs,addons,prefs,forms
* Verifier Stage environment:
* Verifier Stage environment:
In the browser: https://verifier.stage.mozaws.net/
curl https://verifier.stage.mozaws.net
curl -I https://verifier.stage.mozaws.net
  Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
  Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
  cd loadtest
  cd loadtest
  make test SERVER_URL=https://verifier.stage.mozaws.net
  make test SERVER_URL=https://verifier.stage.mozaws.net
* Sync Server Stage environment:
* Sync Server Stage environment:
  Install server-syncstorage to the local host or AWS instance (see below)
  Install server-syncstorage to the local host or AWS instance (see below)
Line 85: Line 59:


* Tokenserver+Verifier Production Environment
* Tokenserver+Verifier Production Environment
In the browser: https://token.services.mozilla.com
curl https://token.services.mozilla.com
curl -I https://token.services.mozilla.com
Then:
  Use the test tool from here: https://github.com/edmoz/fxa-sync-client
  Use the test tool from here: https://github.com/edmoz/fxa-sync-client
  Install and check all collection types for a known account in Production:
  Install and check all collection types for a known account in Production:
Line 91: Line 69:


* Verifier Production Environment
* Verifier Production Environment
In the browser: https://verifier.accounts.firefox.com
curl https://verifier.accounts.firefox.com
curl -I https://verifier.accounts.firefox.com
Then:
  Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
  Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
  cd loadtest
  cd loadtest
  make test SERVER_URL=https://verifier.accounts.firefox.com
  make test SERVER_URL=https://verifier.accounts.firefox.com
* Sync Server Stage environment
* Sync Server Stage environment
  Sign in with a known FxA account and sync data with a current Production account (sync node).
  Sign in with a known FxA account and sync data with a current Production account (sync node).
Line 100: Line 83:
== Load Test Tool Client/Host ==
== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following link for more information: https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:  
 
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
=== Creating a RHEL AWS instance ===
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
* Pick a Region then Create Instance > Launch Instance
* Follow the prompts to create a basic, RHEL-flavored instance
* Use of the QA/Dev key pairs that have been set up for this:
** US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
** US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
* Once the instance is running, log in as "ec2-user"
 
* The following apps, tools, and libs will need to be installed for use with various Services applications:
** gcc, gcc-c++
** hg
** git
** python-devel
** automake, autoconf, and libtool  (required for libzmq, for easy_install)
** pip
** virtualenv
** node/npm
** zeromq 3.X
** gmp, gmp-devel
 
* Also, general rhel updates:
$ sudo yum -y update
and/or
$ sudo yum -y upgrade
 
* Now, the instance should be ready for installing and using the Loads tool.
 
=== Creating an Ubuntu AWS instance ===
* Pick a Region then Create Instance > Launch Instance
* Follow the prompts to create a basic, Ubuntu-flavored instance
* Use of the QA/Dev key pairs that have been set up for this:
** US East Key Pair: QA-Dev-Share (created by jbonacci) for general use
** US West Key Pair: QA-dev-share (created by RaFromBRC) for general use
* Once the instance is running, log in as "ubuntu"
 
* The following apps, tools, and libs will need to be installed for use with various Services applications:
** gcc, g++
** mercurial
** git
** python-setuptools, python-virtualenv, and python-dev
** automake, autoconf, libtool
** m4
** node/npm
** libzmq and zeromq 3.X
** gmp-5.1.3 or newer
 
* Also, general rhel updates:
$ sudo apt-get update
and/or
$ sudo apt-get upgrade
 
* Now, the instance should be ready for installing and using the Loads tool.


== Installing BrowserID-Verifier and the Loads tool on Localhost or AWS ==
== Installing BrowserID-Verifier and the Loads tool on Localhost or AWS ==
Line 159: Line 91:
  $ git clone git://github.com/mozilla/browserid-verifier
  $ git clone git://github.com/mozilla/browserid-verifier
  $ cd browserid-verifier
  $ cd browserid-verifier
Note: You may want to install a specific branch for testing vs defaulting to Master
  $ npm install
  $ npm install
  $ npm test
  $ npm test
Line 197: Line 130:


* REFs:
* REFs:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest


Line 204: Line 137:
  $ git clone https://github.com/mozilla-services/tokenserver
  $ git clone https://github.com/mozilla-services/tokenserver
  $ cd tokenserver
  $ cd tokenserver
Note: You may want to install a specific branch for testing vs defaulting to Master
  $ make build
  $ make build
  $ make test
  $ make test
Line 211: Line 145:
     Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com
     Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com


* Note: This will install a local copy of the Loads tool for use with TokenServer.
* Note: This will install a local copy of the Loads tool for use with TokenServer+Verifier.


== Running the load test against TokenServer+Verifier in Stage ==
== Running the load test against TokenServer+Verifier in Stage ==
Line 244: Line 178:


* REFs:
* REFs:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest


== Installing Sync 1.5 and the Loads tool on the AWS instance ==
== Installing Sync and load testing on Localhost or AWS ==
  Installation:
  Installation:
  $ git clone https://github.com/mozilla-services/server-syncstorage/
  $ git clone https://github.com/mozilla-services/syncstorage-loadtest/
  $ cd server-syncstorage
  $ cd syncstorage-loadtest
$ make build
  Note: You may want to install a specific branch for testing vs defaulting to Master
$ make test
$ pip install -r requirements.txt
$ cd loadtest
  $ make build
 
* Note: This will install a local copy of the Loads tool for use with Sync 1.5.


== Running the load test against Sync 1.5 in Stage ==
== Running the load test against Sync 1.5 in Stage ==
* Loads against specific Sync nodes in Stage
* Loads against specific Sync nodes in Stage
  $ make test SERVER_URL=https://your.storagenode.here#SECRET
  $ export SERVER_URL=https://your.storagenode.here#SECRET
$ make bench SERVER_URL=https://your.storagenode.here#SECRET
  Sync Stage nodes:
  Sync Stage nodes:
     https://sync-1-us-east-1.stage.mozaws.net
     https://sync-1-us-east-1.stage.mozaws.net
     https://sync-2-us-east-1.stage.mozaws.net
     https://sync-2-us-east-1.stage.mozaws.net
    https://sync-3-us-east-1.stage.mozaws.net
     ...etc...
     ...etc...
Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.   
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
   
   
  NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.
  NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.
* Load testing with Molotov: https://molotov.readthedocs.io/en/stable/
$ bin/molotov [commands] loadtest.py


== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* loadtesting from server-syncstorage has been deprecated, please refer to mozilla-services/syncstorage-loadtest
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://your.storagenode.here#SECRET
 
* REFs:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest


Line 294: Line 216:
  $ make test SERVER_URL=https://token.stage.mozaws.net
  $ make test SERVER_URL=https://token.stage.mozaws.net
  $ make bench SERVER_URL=https://token.stage.mozaws.net
  $ make bench SERVER_URL=https://token.stage.mozaws.net
  See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer_Stage_Environment
  See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer.2BVerifier_Stage_Environment
   
   
  Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.     
  Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.     
Line 327: Line 249:


* REFs:
* REFs:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest


Line 487: Line 409:
     File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
     File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
     ..."Connection pool is full, discarding connection: 127.0.0.1", "...
     ..."Connection pool is full, discarding connection: 127.0.0.1", "...
Also, any 499s are probably an artifact of the current (V1) load test.
  REF:
  REF:
  https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
  https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
Line 499: Line 422:


* Acceptable Verifier errors:
* Acceptable Verifier errors:
  In the verifier and squid logs:
  The verifier_out.log will show errors of the following types:
  References to mozilla.org and login.mozilla.org - part of the "invalid domain" tests
  result: 'failure',\n  reason: 'untrusted issuer...'
  In the verifier logs:
  result: 'failure',\n  reason: 'expired'
  References to https://secret.mozilla.com, which are defined in the browserid-verifier load test
  result: 'failure',\n  reason: 'algorithms do not match'
  https://github.com/mozilla/browserid-verifier/blob/master/loadtest/loadtest.py#L77 for example
  result: 'failure',\n reason: 'audience mismatch: scheme mismatch'
   
  Also, any 499s in the nginx logs are probably an artifact of the current (V1) load test.
  Depending on the number of users and agents, there will be 503s in the nginx and app logs.
For a single instance deploy to Stage, a good starting configuration is 20 users, 1 agent.
See also https://github.com/mozilla/browserid-verifier/issues/58


* Acceptable Sync node errors:
* Acceptable Sync node errors:
Line 520: Line 440:
  Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
  Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
  although they may show up in the logs after running the remote integration tests.
  although they may show up in the logs after running the remote integration tests.
Also, any 499s are probably an artifact of the current (V1) load test.
   
   
  In /var/log/hekad/sync_1_5.stderr.log
  In /var/log/hekad/sync_1_5.stderr.log
Line 541: Line 462:
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster


* Monitoring TS/Verifier/Sync Stage:
* For all other monitoring, see the following section:
** Stackdriver
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Monitoring_the_Stage_Environment
*** Stage TS + FxA + Sync 1.5 meta-dash: https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** Kibana
*** https://kibana.shared.us-east-1.stage.mozaws.net/
*** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20Web%20Logs
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Token%20App%20Logs%20POC
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20Nginx%20Errors
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20App%20Logs
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
 
** Heka
*** https://heka.shared.us-east-1.stage.mozaws.net/
*** https://heka.shared.us-east-1.stage.mozaws.net/#health
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-HTTPStatus/outputs/Sync-1_5-HTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagehistory.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagebookmarks.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storageforms.cbuf
 
** REF: https://mana.mozilla.org/wiki/display/SVCOPS/Sync+1.5+Dash+Boards


== Performance Testing Information ==
== Performance Testing Information ==
Line 580: Line 479:


== Known Bugs, Issues, and Tasks ==
== Known Bugs, Issues, and Tasks ==
* Tokenserver, Verifier, and Sync
* Tokenserver:  
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=907479
** Repo: https://github.com/mozilla-services/tokenserver/issues  
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=1008066
** Bugzilla: http://mzl.la/1s4qZKn
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=1014411
 
** Tokenserver:
*** Repo: https://github.com/mozilla-services/tokenserver/issues  
*** Bugzilla: http://mzl.la/1s4qZKn


** BrowserID-Verifier:
* BrowserID-Verifier:
** Repo: https://github.com/mozilla/browserid-verifier/issues
** Repo: https://github.com/mozilla/browserid-verifier/issues
** Bugzilla: no specific cateogory
** Bugzilla: no specific cateogory


** Sync:
* Sync:
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Bugzilla: http://mzl.la/VUrYQ5
** Bugzilla: http://mzl.la/VUrYQ5
Rbillings
Confirmed users
1,504

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1014293...1208327"