Breakpad/Status Meetings/2016-08-03

« previous meeting — index – next week » ^create?

Meeting Info

Breakpad status meetings occur on Wed at 10:00am Pacific Time.

Conference numbers:

   Vidyo: Stability 
   650-903-0800 x92 conf 98200#
   800-707-2533 (pin 369) conf 98200# 

IRC backchannel: #breakpad
Mountain View: Dancing Baby (3rd floor)

Operations Updates

• focused on "not being on fire"
  • seems to be going well
  • pull request went up for changing the retention in ES
    • did it get merged?
    • it did _not_!
  • well then JP should have slept better last night
  • we had started a job that ran once a week to expire old indexes
  • we trimmed down how many indexes are kept
  • root cause of last weeks issues
    • configuration mismatch with the rest of the cluster
    • puppet missed putting the .yaml file in there
    • they defaulted to 2GB and when they exhausted themselves everything went to hell
    • we initially suspected that it was retention related
    • debated but didn't land a change that would lower retention temporarily
  • new pingdom accounts coming if you have one already

Project Updates

• Socorro::Middleware component to Graveyard
• Monitoring/healthcheck now checks for ES shards errors. In prod. Every minute.

Deployment Triage

• https://whatsdeployed.io/s-7M7

PR Triage

• http://prs.mozilla.io/mozilla/socorro

Major Projects

Migrating off of persona

• Deployed in stage
  • emails will be sent as soon as we put this in prod

Sending public data to parquet for reading from spark/re:dash

• Adrian and Peter has prototype to add another crash storage that sends to S3
  • https://github.com/peterbe/socorro/pull/17/files
• Mark's awareness of reprocessing (aka. primary keys)

Symbols service refactoring (snappy, somewhat tangental to us)

No update.

Splitting out collector

No update.