Breakpad status meetings occur on Wed at 10:00am Pacific Time.
Vidyo: Stability 650-903-0800 x92 conf 98200# 800-707-2533 (pin 369) conf 98200#
IRC backchannel: #breakpad
Mountain View: Dancing Baby (3rd floor)
- ES cluster size is up 2 nodes in prod
- our business is uptime
- business is boomin'
- ES is great
- load testing environments exist now
- one that doesn't auto scale
- one that auto scales
- finishing antennae pipeline work
- every deploy has its own monitoring instance (newrelic)
- blue/green deployments
-- coming this week: ES5 cluster (maybe) (stretch goals)
Splitting out collector (Antenna)
- (miles, mbrandt, willkg, rpapa) Had a big meeting about load testing needs, plan and schedule.
- (miles) Set up a -stage-like environment for load tests. Working on redoing deployment pipeline.
- (mbrandt, rpapa) Working on load testing for the next week hopefully with results by Friday, February 24th.
- mbrandt is skipping our meeting today, I was able to schedule some time to run tests and analyze results with rpapa for this time slot.
Added some new stuff:
Lots still to do:
Next time we get together as a group we will want to whiteboard some things out. So far I'm still on uncontroversial porting of older code.
- (Adrian) solved the "matrix" issue
- (Adrian) down to only 3 failing tests, w00t!
- (Adrian) next step is to have all tests passing and create a mapping file to use during reindexing
- (Adrian) ES 5.2 went out recently, should we upgrade to it instead of 5.1?
- Crash Reports schema file is not updated on S3: https://bugzilla.mozilla.org/show_bug.cgi?id=1340105
- that file is pushed to s3 via a crontabber job, however that job runs on a long-running admin node and the code doesn't automatically get updated on it
- is there a written process for updating that file?
- are there complexities in updating the code on the admin nodes? is that written down somewhere?
- how do we want to deal with code deploys and long-running nodes?
- does new relic get frame data?
- is this a problem with running on antennae
- lets open a bug about it!
- we're going to try to get the ES5 cluster going (stretch)
- heading for antennae as prod collector
- load test could cause an iteration cycle
- let's try to set up so that if things go well we could ship this Q