MozillaWiki

TestEngineering/Services/TokenServerAndSyncLoadTesting: Difference between revisions

Page Discussion

TestEngineering/Services/TokenServerAndSyncLoadTesting (view source)

Revision as of 14:32, 1 March 2019

3,898 bytes removed ,  1 March 2019
→‎Using the Loads V1 Services Cluster for Sync 1.5 in Stage
 
(15 intermediate revisions by 3 users not shown)
Line 3: Line 3:
** A Tokenserver+Verifier stack: See the TokenServer sections below...
** A Tokenserver+Verifier stack: See the TokenServer sections below...


== Summary for Tokenserver, Verifier, and Sync 1.5 ==
* Latest Results
** Link to loads cluster: https://loads.services.mozilla.com/
*** Note: this now requires login privileges and a password
** Snapshots from StackDriver - TBD
** Snapshots from Kibana - TBD
* Latest Deployments
** TokenServer Stage: https://bugzilla.mozilla.org/show_bug.cgi?id=1014496
** TokenServer Prod: https://bugzilla.mozilla.org/show_bug.cgi?id=1027899
** Sync Server Stage: https://bugzilla.mozilla.org/show_bug.cgi?id=1026346
** Sync Server Prod: https://bugzilla.mozilla.org/show_bug.cgi?id=1026346
** Verifier Stage Deploy: ttps://bugzilla.mozilla.org/show_bug.cgi?id=1026644
** Verifier Prod Deploy: https://bugzilla.mozilla.org/show_bug.cgi?id=1027392
* In Progress
** Build out of Kibana dashboards
** Ongoing testing of Tokenserver, Verifier, and Sync releases
** Bug review and issue debug - there are a lot of issues to work on (see the long list near the bottom of the wiki)
* Bugs To Verify:
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=988095
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=1025767
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=1027444
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=1025735
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=735102
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=775395
* Planned
** Scaling for production traffic after release of Fx29
** Sync 1.5 migration work
** Operations readiness testing: See Bug 1006792
* Blockers
** none at this time
* Completed
** Pre-release load testing
** Previous load test results (short): http://loads.services.mozilla.com/
* Performance
** TBD


== Quick Verification Of Stage Deployments ==
== Quick Verification Of Stage Deployments ==
Line 127: Line 91:
  $ git clone git://github.com/mozilla/browserid-verifier
  $ git clone git://github.com/mozilla/browserid-verifier
  $ cd browserid-verifier
  $ cd browserid-verifier
Note: You may want to install a specific branch for testing vs defaulting to Master
  $ npm install
  $ npm install
  $ npm test
  $ npm test
Line 172: Line 137:
  $ git clone https://github.com/mozilla-services/tokenserver
  $ git clone https://github.com/mozilla-services/tokenserver
  $ cd tokenserver
  $ cd tokenserver
Note: You may want to install a specific branch for testing vs defaulting to Master
  $ make build
  $ make build
  $ make test
  $ make test
Line 179: Line 145:
     Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com
     Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com


* Note: This will install a local copy of the Loads tool for use with TokenServer.
* Note: This will install a local copy of the Loads tool for use with TokenServer+Verifier.


== Running the load test against TokenServer+Verifier in Stage ==
== Running the load test against TokenServer+Verifier in Stage ==
Line 212: Line 178:


* REFs:
* REFs:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest


== Installing Sync 1.5 and the Loads tool on Localhost or AWS ==
== Installing Sync and load testing on Localhost or AWS ==
  Installation:
  Installation:
  $ git clone https://github.com/mozilla-services/server-syncstorage/
  $ git clone https://github.com/mozilla-services/syncstorage-loadtest/
  $ cd server-syncstorage
  $ cd syncstorage-loadtest
$ make build
  Note: You may want to install a specific branch for testing vs defaulting to Master
$ make test
$ pip install -r requirements.txt
$ cd loadtest
  $ make build
 
* Note: This will install a local copy of the Loads tool for use with Sync 1.5.


== Running the load test against Sync 1.5 in Stage ==
== Running the load test against Sync 1.5 in Stage ==
* Loads against specific Sync nodes in Stage
* Loads against specific Sync nodes in Stage
  $ make test SERVER_URL=https://your.storagenode.here#SECRET
  $ export SERVER_URL=https://your.storagenode.here#SECRET
$ make bench SERVER_URL=https://your.storagenode.here#SECRET
  Sync Stage nodes:
  Sync Stage nodes:
     https://sync-1-us-east-1.stage.mozaws.net
     https://sync-1-us-east-1.stage.mozaws.net
     https://sync-2-us-east-1.stage.mozaws.net
     https://sync-2-us-east-1.stage.mozaws.net
    https://sync-3-us-east-1.stage.mozaws.net
     ...etc...
     ...etc...
Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.   
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
   
   
  NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.
  NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.
* Load testing with Molotov: https://molotov.readthedocs.io/en/stable/
$ bin/molotov [commands] loadtest.py


== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* loadtesting from server-syncstorage has been deprecated, please refer to mozilla-services/syncstorage-loadtest
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://your.storagenode.here#SECRET
 
* REFs:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest


Line 262: Line 216:
  $ make test SERVER_URL=https://token.stage.mozaws.net
  $ make test SERVER_URL=https://token.stage.mozaws.net
  $ make bench SERVER_URL=https://token.stage.mozaws.net
  $ make bench SERVER_URL=https://token.stage.mozaws.net
  See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer_Stage_Environment
  See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer.2BVerifier_Stage_Environment
   
   
  Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.     
  Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.     
Line 295: Line 249:


* REFs:
* REFs:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest


Line 455: Line 409:
     File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
     File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
     ..."Connection pool is full, discarding connection: 127.0.0.1", "...
     ..."Connection pool is full, discarding connection: 127.0.0.1", "...
Also, any 499s are probably an artifact of the current (V1) load test.
  REF:
  REF:
  https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
  https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
Line 472: Line 427:
  result: 'failure',\n  reason: 'algorithms do not match'
  result: 'failure',\n  reason: 'algorithms do not match'
  result: 'failure',\n  reason: 'audience mismatch: scheme mismatch'
  result: 'failure',\n  reason: 'audience mismatch: scheme mismatch'
Also, any 499s in the nginx logs are probably an artifact of the current (V1) load test.


* Acceptable Sync node errors:
* Acceptable Sync node errors:
Line 484: Line 440:
  Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
  Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
  although they may show up in the logs after running the remote integration tests.
  although they may show up in the logs after running the remote integration tests.
Also, any 499s are probably an artifact of the current (V1) load test.
   
   
  In /var/log/hekad/sync_1_5.stderr.log
  In /var/log/hekad/sync_1_5.stderr.log
Line 505: Line 462:
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster


* Monitoring TS/Verifier/Sync Stage:
* For all other monitoring, see the following section:
** Stackdriver
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Monitoring_the_Stage_Environment
*** Stage TS + FxA + Sync 1.5 meta-dash: https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
*** Per instance: https://app.stackdriver.com/instances/<INSTANCE>
** Kibana
*** https://kibana.shared.us-east-1.stage.mozaws.net/
*** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20Web%20Logs
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Token%20App%20Logs%20POC
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20Nginx%20Errors
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20App%20Logs
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
 
** Heka
*** https://heka.shared.us-east-1.stage.mozaws.net/
*** https://heka.shared.us-east-1.stage.mozaws.net/#health
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-HTTPStatus/outputs/Sync-1_5-HTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagehistory.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagebookmarks.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storageforms.cbuf
 
** REF: https://mana.mozilla.org/wiki/display/SVCOPS/Sync+1.5+Dash+Boards


== Performance Testing Information ==
== Performance Testing Information ==
Line 545: Line 479:


== Known Bugs, Issues, and Tasks ==
== Known Bugs, Issues, and Tasks ==
* Tokenserver, Verifier, and Sync
* Tokenserver:  
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=907479
** Repo: https://github.com/mozilla-services/tokenserver/issues  
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=1008066
** Bugzilla: http://mzl.la/1s4qZKn
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=1014411
 
** Tokenserver:
*** Repo: https://github.com/mozilla-services/tokenserver/issues  
*** Bugzilla: http://mzl.la/1s4qZKn


** BrowserID-Verifier:
* BrowserID-Verifier:
** Repo: https://github.com/mozilla/browserid-verifier/issues
** Repo: https://github.com/mozilla/browserid-verifier/issues
** Bugzilla: no specific cateogory
** Bugzilla: no specific cateogory


** Sync:
* Sync:
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Bugzilla: http://mzl.la/VUrYQ5
** Bugzilla: http://mzl.la/VUrYQ5
Rbillings
Confirmed users
1,504

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1015663...1208327"