canmove, Confirmed users
6,442
edits
Balrog: Difference between revisions
Jump to navigation
Jump to search
→ELB Logs
| Line 29: | Line 29: | ||
The ELB logs for the public-facing application are replicated to the '''balrog-us-west-2-elb-logs''' S3 bucket, located in us-west-2. Logs are rotated very quickly, and we end up with tens of thousands of separate files each day. Because of this, and the fact that S3 has a lot of overhead per-file, it can be tricky to do analysis on them. You're unlikely to be able to download the logs locally in any reasonable amount of time (ie, less than a day), but mounting them on an EC2 instance in us-west-2 should provide you with reasonably quick access. Here's an example: | The ELB logs for the public-facing application are replicated to the '''balrog-us-west-2-elb-logs''' S3 bucket, located in us-west-2. Logs are rotated very quickly, and we end up with tens of thousands of separate files each day. Because of this, and the fact that S3 has a lot of overhead per-file, it can be tricky to do analysis on them. You're unlikely to be able to download the logs locally in any reasonable amount of time (ie, less than a day), but mounting them on an EC2 instance in us-west-2 should provide you with reasonably quick access. Here's an example: | ||
* Launch EC2 instance (you probably a compute-optimized one, and at least 100GB of storage). | * Launch EC2 instance (you probably a compute-optimized one, and at least 100GB of storage). | ||
* Generate an access token for your CloudOps AWS account. If you don't have a CloudOps AWS account, talk to Ben Hearsum or Bensong Wong. Put the token in a plaintext file | * Generate an access token for your CloudOps AWS account. If you don't have a CloudOps AWS account, talk to Ben Hearsum or Bensong Wong. Put the token in a plaintext file somewhere on the instance. | ||
** If you've chosen local storage, you'll probably need to format and mount volume. | ** If you've chosen local storage, you'll probably need to format and mount volume. | ||
* Install s3fs by following the instructions on https://github.com/s3fs-fuse/s3fs-fuse. | * Install s3fs by following the instructions on https://github.com/s3fs-fuse/s3fs-fuse. | ||