MozillaWiki

Loop/Test/Loadtesting: Difference between revisions

Page Discussion

Loop/Test/Loadtesting (view source)

Revision as of 16:15, 22 May 2015

7,204 bytes removed ,  22 May 2015
m
formatting
m (remove unnecessary content)
m (formatting)
 
(3 intermediate revisions by the same user not shown)
Line 77: Line 77:
Of the three tests available, <code>make megabench</code> will probably be the one you use most.
Of the three tests available, <code>make megabench</code> will probably be the one you use most.
<code>make test</code> is largely for a quick smoke test and <code>make bench</code> doesn't do anything you can't do with the other two.
<code>make test</code> is largely for a quick smoke test and <code>make bench</code> doesn't do anything you can't do with the other two.
For information about test monitoring:
[[Loop/Test/Monitoring]]




Line 203: Line 207:
- register : 4
- register : 4


</pre>
[[#toc]]
= Loadtest Monitoring =
== Monitoring Loop Stage ==
* Use the loadsv1 dashboard:
** http://loads.services.mozilla.com
[[#toc]]
== Monitoring Tools / Dashboards ==
== Monitoring ==
=== STAGE Environment ===
'''Sentry:'''<br />
* https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry STAGE - talk to the OPs team
'''Stackdriver:'''<br />
* https://app.stackdriver.com/
* https://app.stackdriver.com/groups/11042/loop-server/stage-loop
'''Graphite:'''<br />
* https://graphite.shared.us-east-1.stage.mozaws.net
** Graphite > stats > loop-activated-users
** Graphite > stats > loop-call-urls
** User Graphs > not currently defined
'''Kibana:'''<br />
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD
'''Heka'''<br />
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf
=== PRODUCTION Environment ===
'''Sentry:'''<br />
* http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/
'''Stackdriver:'''<br />
* https://app.stackdriver.com/groups/11044/loop-server/prod-loop
'''Kibana:'''<br />
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw
'''Heka:'''<br />
* Main: https://heka.shared.us-west-2.prod.mozaws.net/#
* https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
* https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
* https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus
'''loop-server Dashboard'''<br />
* https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard
'''Graphite:'''<br />
* https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)
'''Firefox Input Dashboard:'''<br />
* https://input.mozilla.org/en-US/?product=Loop
'''Pingdom:'''<br />
* loop-client: http://stats.pingdom.com/20dar76w4hmv/1304575
* loop-server: http://stats.pingdom.com/20dar76w4hmv/1304565
[[#top]]
== Log Monitoring ==
Important logs for monitoring.
For example, for loop-server:<br />
* /var/log/circus.log
* /var/log/loop_err.log
* /var/log/loop_out.log
* /var/log/hekad/loop.stdout.log
* /var/log/hekad/loop.stderr.log
* /var/log/nginx/access.log
* /var/log/nginx/error.log
[[#top]]
== HTTP Access Log - Parsing ==
'''unique REST activity with counts'''
<pre>
$ cat /media/ephemeral0/nginx/logs/loop_server.access.log | grep "HTTP/" | awk '{print $6" "$3" "}' | sort | uniq -c
</pre>
'''unique REST activity with counts (from a .gz file)'''
<pre>
$ zgrep -a "HTTP/"  /media/ephemeral0/nginx/logs/loop_server.access.log-20150114.gz |  awk '{print $6" "$3" "}' | sort | uniq -c
</pre>
'''return only the interesting stuff'''
<pre>
$ cat loop_server.access.log | grep -v '200 ' | grep -v '101 ' | grep -v '499 ' | grep -v '201 ' | grep -v '204 '
</pre>
'''Tail multiple logs'''
<pre>
$ cd /media/ephemeral0/nginx/logs
$ tail -f loop_server.access.log loop_server.error.log  default.access.log default.error.log
</pre>
</pre>


Line 388: Line 225:


= Reference =
= Reference =
== Monitoring ==
* [[Loop/Test/Monitoring]]




Line 396: Line 236:


== Loads Tool and Cluster ==
== Loads Tool and Cluster ==
** http://loads.services.mozilla.com
* http://loads.services.mozilla.com
** https://loads.readthedocs.org/en/latest
* https://loads.readthedocs.org/
** https://github.com/mozilla-services/loads/issues
* https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
* https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues
* https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
* https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
* https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
* https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
Rpappalardo
Confirmed users
487

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1075920...1076037"