Changes

=== Public app ==='''NOTE: ''' These instructions were written before [https://aws.amazon.com/blogs/aws/amazon-athena-interactive-sql-queries-for-data-in-amazon-s3/ Amazon Athena existed]. The next time we need production instance of Balrog publishes logs to do such analysis, it's probably worth giving it a try. Other techniques may be better too - the instructions below are just something we've done in the past.two different S3 buckets:* The ELB nginx access logs for (that contain all of the public-facing application update requests we receive) are replicated published to the '''balrog-us-west-2-elb-logs''' S3 bucket, located in us-west-2. Logs These logs are rotated very quicklylarge, and we end up with tens of thousands of separate files each day. Because of this, and the fact that S3 has a lot of overhead per-file, it can be tricky to do analysis on them. Youyou're unlikely to be able to download the logs locally in any reasonable amount of time (ie, less than a day), but mounting them on an EC2 instance in us-west-2 should provide you for local querying. The best way to work with reasonably quick accessthem is through Athena. Here's an example:* Launch EC2 instance (you probably a compute-optimized one, and at least 100GB The rest of storage).* Generate an access token for your CloudOps AWS account. If you donthe logs are published to 't have a CloudOps AWS account, talk to Ben Hearsum or Bensong Wong. Put the token in a plaintext file somewhere on the instance.** If you've chosen local storage, you'll probably need to format and mount volume.* Install s3fs by following the instructions on https://github.com/s3fsnet-fuse/s3fsmozaws-fuse.* Mount the bucket on your instance, eg: s3fs balrogprod-us-west-2-elblogging-logs balrog''', in the "firehose/media/bucket -o passwd_file=pws3" directory.txtWithin that there are subdirectories for different parts of Balrog:* Do some broad grepping directly on the S3 logs, and store it in a local file* balrog. This should speed up subsequent queriesadmin. Eg: grep '/Firefox/syslog.*WINNTadmin contains the admin wsgi app output.*/release/' /media/bucket/AWSLogs/361527076523/elasticloadbalancing/us-west-2/2016/09/17/* | gzip > /media/ephemeral0/sept-17-winnt-releasebalrog.admin.txtnginx.gz* Do additional queries on {access,error} contain the new logfileadmin access & error logs from nginx. === Admin app ===Nginx The access logs for are generally a subset of the admin wsgi app are available output (on which logs requests with a ~1 day time delaybit of extra detail) in the [https://console.aws** balrog.admin.syslog.amazonagent contains the agent app output.com/s3/buckets/net-mozaws-prod-us-west-2-logging-balrog/** balrog.admin.nginxsyslog.access/?region=us-west-2&tab=overview "net-mozaws-prod-us-west-2-logging-cron contains cronjob output (eg: the history cleanup and production database dump)** balrog" S3 bucket]. These logs are small enough web.syslog.web contains the public wsgi app output. Note that downloading and querying them locally this app does _not_ log requests, so this is generally largely warning/exception output. If you care about requests to the public app, use the most efficient thing to donginx access logs (see above).