MozillaWiki

TestEngineering/Services/TokenServerAndSyncLoadTesting: Difference between revisions

Page Discussion

TestEngineering/Services/TokenServerAndSyncLoadTesting (view source)

Revision as of 14:32, 1 March 2019

3,577 bytes removed ,  1 March 2019
→‎Using the Loads V1 Services Cluster for Sync 1.5 in Stage
 
(18 intermediate revisions by 3 users not shown)
Line 3: Line 3:
** A Tokenserver+Verifier stack: See the TokenServer sections below...
** A Tokenserver+Verifier stack: See the TokenServer sections below...


== Summary for Tokenserver, Verifier, and Sync 1.5 ==
* Latest Results
** Link to loads cluster: https://loads.services.mozilla.com/
*** Note: this now requires login privileges and a password
** Snapshots from StackDriver - TBD
** Snapshots from Kibana - TBD
* Latest Deployments
** TokenServer Stage: https://bugzilla.mozilla.org/show_bug.cgi?id=1014496
** TokenServer Prod: https://bugzilla.mozilla.org/show_bug.cgi?id=1027899
** Sync Server Stage: https://bugzilla.mozilla.org/show_bug.cgi?id=1026346
** Sync Server Prod: https://bugzilla.mozilla.org/show_bug.cgi?id=1026346
** Verifier Stage Deploy: ttps://bugzilla.mozilla.org/show_bug.cgi?id=1026644
** Verifier Prod Deploy: https://bugzilla.mozilla.org/show_bug.cgi?id=1027392
* In Progress
** Build out of Kibana dashboards
** Ongoing testing of Tokenserver, Verifier, and Sync releases
** Bug review and issue debug - there are a lot of issues to work on (see the long list near the bottom of the wiki)
* Bugs To Verify:
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=988095
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=1025767
** Tokenserver: https://bugzilla.mozilla.org/show_bug.cgi?id=1027444
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=1025735
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=735102
** Sync: https://bugzilla.mozilla.org/show_bug.cgi?id=775395
* Planned
** Scaling for production traffic after release of Fx29
** Sync 1.5 migration work
** Operations readiness testing: See Bug 1006792
* Blockers
** none at this time
* Completed
** Pre-release load testing
** Previous load test results (short): http://loads.services.mozilla.com/
* Performance
** TBD


== Quick Verification Of Stage Deployments ==
== Quick Verification Of Stage Deployments ==
Line 95: Line 59:


* Tokenserver+Verifier Production Environment
* Tokenserver+Verifier Production Environment
In the browser: https://token.services.mozilla.com
curl https://token.services.mozilla.com
curl -I https://token.services.mozilla.com
Then:
  Use the test tool from here: https://github.com/edmoz/fxa-sync-client
  Use the test tool from here: https://github.com/edmoz/fxa-sync-client
  Install and check all collection types for a known account in Production:
  Install and check all collection types for a known account in Production:
Line 101: Line 69:


* Verifier Production Environment
* Verifier Production Environment
In the browser: https://verifier.accounts.firefox.com
curl https://verifier.accounts.firefox.com
curl -I https://verifier.accounts.firefox.com
Then:
  Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
  Use the simple "make test" command from an install of browserid-verifier on the localhost or AWS instance.
  cd loadtest
  cd loadtest
  make test SERVER_URL=https://verifier.accounts.firefox.com
  make test SERVER_URL=https://verifier.accounts.firefox.com
* Sync Server Stage environment
* Sync Server Stage environment
  Sign in with a known FxA account and sync data with a current Production account (sync node).
  Sign in with a known FxA account and sync data with a current Production account (sync node).
Line 118: Line 91:
  $ git clone git://github.com/mozilla/browserid-verifier
  $ git clone git://github.com/mozilla/browserid-verifier
  $ cd browserid-verifier
  $ cd browserid-verifier
Note: You may want to install a specific branch for testing vs defaulting to Master
  $ npm install
  $ npm install
  $ npm test
  $ npm test
Line 156: Line 130:


* REFs:
* REFs:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest


Line 163: Line 137:
  $ git clone https://github.com/mozilla-services/tokenserver
  $ git clone https://github.com/mozilla-services/tokenserver
  $ cd tokenserver
  $ cd tokenserver
Note: You may want to install a specific branch for testing vs defaulting to Master
  $ make build
  $ make build
  $ make test
  $ make test
Line 170: Line 145:
     Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com
     Note: This should hit Prod by default: SERVER_URL=https://token.services.mozilla.com


* Note: This will install a local copy of the Loads tool for use with TokenServer.
* Note: This will install a local copy of the Loads tool for use with TokenServer+Verifier.


== Running the load test against TokenServer+Verifier in Stage ==
== Running the load test against TokenServer+Verifier in Stage ==
Line 203: Line 178:


* REFs:
* REFs:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest
** https://github.com/mozilla-services/tokenserver/tree/master/loadtest


== Installing Sync 1.5 and the Loads tool on Localhost or AWS ==
== Installing Sync and load testing on Localhost or AWS ==
  Installation:
  Installation:
  $ git clone https://github.com/mozilla-services/server-syncstorage/
  $ git clone https://github.com/mozilla-services/syncstorage-loadtest/
  $ cd server-syncstorage
  $ cd syncstorage-loadtest
$ make build
  Note: You may want to install a specific branch for testing vs defaulting to Master
$ make test
$ pip install -r requirements.txt
$ cd loadtest
  $ make build
 
* Note: This will install a local copy of the Loads tool for use with Sync 1.5.


== Running the load test against Sync 1.5 in Stage ==
== Running the load test against Sync 1.5 in Stage ==
* Loads against specific Sync nodes in Stage
* Loads against specific Sync nodes in Stage
  $ make test SERVER_URL=https://your.storagenode.here#SECRET
  $ export SERVER_URL=https://your.storagenode.here#SECRET
$ make bench SERVER_URL=https://your.storagenode.here#SECRET
  Sync Stage nodes:
  Sync Stage nodes:
     https://sync-1-us-east-1.stage.mozaws.net
     https://sync-1-us-east-1.stage.mozaws.net
     https://sync-2-us-east-1.stage.mozaws.net
     https://sync-2-us-east-1.stage.mozaws.net
    https://sync-3-us-east-1.stage.mozaws.net
     ...etc...
     ...etc...
Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.   
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
   
   
  NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.
  NOTE: The OPs team has the SECRET string for Stage. Get it from them before you start testing.
* Load testing with Molotov: https://molotov.readthedocs.io/en/stable/
$ bin/molotov [commands] loadtest.py


== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
== Using the Loads V1 Services Cluster for Sync 1.5 in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* loadtesting from server-syncstorage has been deprecated, please refer to mozilla-services/syncstorage-loadtest
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://your.storagenode.here#SECRET
 
* REFs:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest


Line 253: Line 216:
  $ make test SERVER_URL=https://token.stage.mozaws.net
  $ make test SERVER_URL=https://token.stage.mozaws.net
  $ make bench SERVER_URL=https://token.stage.mozaws.net
  $ make bench SERVER_URL=https://token.stage.mozaws.net
  See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer_Stage_Environment
  See https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#TokenServer.2BVerifier_Stage_Environment
   
   
  Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.     
  Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.     
Line 286: Line 249:


* REFs:
* REFs:
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Loads_Services_Cluster_Environment
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest
** https://github.com/mozilla-services/server-syncstorage/tree/master/loadtest


Line 446: Line 409:
     File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
     File \"/data/tokenserver/local/lib/python2.6/site-packages/gunicorn/workers/async.py\"...
     ..."Connection pool is full, discarding connection: 127.0.0.1", "...
     ..."Connection pool is full, discarding connection: 127.0.0.1", "...
Also, any 499s are probably an artifact of the current (V1) load test.
  REF:
  REF:
  https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
  https://bugzilla.mozilla.org/show_bug.cgi?id=1040396
Line 463: Line 427:
  result: 'failure',\n  reason: 'algorithms do not match'
  result: 'failure',\n  reason: 'algorithms do not match'
  result: 'failure',\n  reason: 'audience mismatch: scheme mismatch'
  result: 'failure',\n  reason: 'audience mismatch: scheme mismatch'
Also, any 499s in the nginx logs are probably an artifact of the current (V1) load test.


* Acceptable Sync node errors:
* Acceptable Sync node errors:
Line 475: Line 440:
  Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
  Usually, you will not see 304s, 400s, 412s, or 415s for a load test,
  although they may show up in the logs after running the remote integration tests.
  although they may show up in the logs after running the remote integration tests.
Also, any 499s are probably an artifact of the current (V1) load test.
   
   
  In /var/log/hekad/sync_1_5.stderr.log
  In /var/log/hekad/sync_1_5.stderr.log
Line 496: Line 462:
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster
* and also on StackDriver: https://app.stackdriver.com/groups/6664/stage-loads-cluster


* Monitoring TS/Verifier/Sync Stage:
* For all other monitoring, see the following section:
** Stackdriver
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments#Monitoring_the_Stage_Environment
*** Stage TS + FxA + Sync 1.5 meta-dash: https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
** Kibana
*** https://kibana.shared.us-east-1.stage.mozaws.net/
*** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20Web%20Logs
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Token%20App%20Logs%20POC
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20Nginx%20Errors
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync%20App%20Logs
*** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
 
** Heka
*** https://heka.shared.us-east-1.stage.mozaws.net/
*** https://heka.shared.us-east-1.stage.mozaws.net/#health
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-HTTPStatus/outputs/Sync-1_5-HTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagehistory.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagebookmarks.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storageforms.cbuf
 
** REF: https://mana.mozilla.org/wiki/display/SVCOPS/Sync+1.5+Dash+Boards


== Performance Testing Information ==
== Performance Testing Information ==
Line 535: Line 479:


== Known Bugs, Issues, and Tasks ==
== Known Bugs, Issues, and Tasks ==
* Tokenserver, Verifier, and Sync
* Tokenserver:  
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=907479
** Repo: https://github.com/mozilla-services/tokenserver/issues  
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=1008066
** Bugzilla: http://mzl.la/1s4qZKn
** Meta: https://bugzilla.mozilla.org/show_bug.cgi?id=1014411
 
** Tokenserver:
*** Repo: https://github.com/mozilla-services/tokenserver/issues  
*** Bugzilla: http://mzl.la/1s4qZKn


** BrowserID-Verifier:
* BrowserID-Verifier:
** Repo: https://github.com/mozilla/browserid-verifier/issues
** Repo: https://github.com/mozilla/browserid-verifier/issues
** Bugzilla: no specific cateogory
** Bugzilla: no specific cateogory


** Sync:
* Sync:
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Repo: https://github.com/mozilla-services/server-syncstorage/issues
** Bugzilla: http://mzl.la/VUrYQ5
** Bugzilla: http://mzl.la/VUrYQ5
Rbillings
Confirmed users
1,504

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1014389...1208327"