Breakpad/Status Meetings/2015-07-29

From MozillaWiki
Jump to: navigation, search

« previous meetingindexnext week » create?

Meeting Info

Breakpad status meetings occur on Wed at 11:00am Pacific Time.

Conference numbers:

   Vidyo: Stability 
   650-903-0800 x92 conf 98200#
   800-707-2533 (pin 369) conf 98200# 

IRC backchannel: #breakpad
Mountain View: Dancing Baby (3rd floor)

Operations Updates





  • Stage is busted
    • What's wrong with it?
      • stage has been missing ADI, and has fallen way behind on reports
    • What can we do to fix it?
      • peter landed a fix today, but we need manual intervention to get it going again
      • we can either fill in ADI from prod, or force crontabber to skip old jobs

Project Updates

Socorro Bug Tracker
this week's bugs

Deployment Triage

PR Triage

other business

  • adobe has had many problems uploading symbols
    • we had upload issues before whistler
    • they're packaging files strangely now
    • ted has been unavailable
    • kairo and rhelmer jumped in to help
    • seems like its working now
    • peter is working out documentation and a checker to validate symbols
  • Friday's troubles
    • On Friday morning JP + Lars went to get the stage submitter up and running
    • Tested by having stage submit crashes to itself
    • AWS dropped and created a new NIC on the Rabbit MQ ELB
    • we accidentally shut down collectors for 10 minutes
    • Responded by pushing some configuration changes
      • modified the config from what was in stage before it went to prod
    • config changed how the submitter was changed to read from the main queue instead of its own
    • since the submitter is faster, we ended up putting all prod crashes to stage for a while
    • we learned:
      • need two people to fully understand all the changes
      • logging is important. we're trying loggly now
      • review is necessary for config changes and indeed any production changes

Travel, etc