Breakpad/Status Meetings/2015-07-29

« previous meeting — index – next week » ^create?

Meeting Info

Breakpad status meetings occur on Wed at 11:00am Pacific Time.

Conference numbers:

   Vidyo: Stability 
   650-903-0800 x92 conf 98200#
   800-707-2533 (pin 369) conf 98200# 

IRC backchannel: #breakpad
Mountain View: Dancing Baby (3rd floor)

Operations Updates

Other

• Stage is busted
  • What's wrong with it?
    • stage has been missing ADI, and has fallen way behind on reports
  • What can we do to fix it?
    • peter landed a fix today, but we need manual intervention to get it going again
    • we can either fill in ADI from prod, or force crontabber to skip old jobs

Project Updates

Socorro Bug Tracker
this week's bugs

Deployment Triage

• http://mzl.la/1HxZ4Gu

PR Triage

• https://prs.paas.allizom.org/mozilla/socorro

other business

• adobe has had many problems uploading symbols
  • we had upload issues before whistler
  • they're packaging files strangely now
  • ted has been unavailable
  • kairo and rhelmer jumped in to help
  • seems like its working now
  • peter is working out documentation and a checker to validate symbols
• Friday's troubles
  • On Friday morning JP + Lars went to get the stage submitter up and running
  • Tested by having stage submit crashes to itself
  • AWS dropped and created a new NIC on the Rabbit MQ ELB
  • we accidentally shut down collectors for 10 minutes
  • Responded by pushing some configuration changes
    • modified the config from what was in stage before it went to prod
  • config changed how the submitter was changed to read from the main queue instead of its own
  • since the submitter is faster, we ended up putting all prod crashes to stage for a while
  • we learned:
    • need two people to fully understand all the changes
    • logging is important. we're trying loggly now
    • review is necessary for config changes and indeed any production changes