Breakpad/Status Meetings/2017-05-03

« previous meeting — index – next week » ^create?

Meeting Info

Breakpad status meetings occur on Wed at 10:00am Pacific Time.

Conference numbers:

   Vidyo: Stability 
   650-903-0800 x92 conf 98200#
   800-707-2533 (pin 369) conf 98200# 

IRC backchannel: #breakpad
Mountain View: Dancing Baby (3rd floor)

Operations Updates

• looking into Elasticsearch migration stuff. it's not going to be the Bahama cruise we were hoping for. problems with migrating data. looking for other options.
  • AWS doesn't support reindex from remote :(
  • instead we have to snapshot, bring up a 2.x cluster, load the snapshot, make a new snapshot, move to the 5.x cluster, load the snapshot
  • should have a talk about this
• having problems with existing Elasticsearch cluster--some nodes are hitting fielddata usage waterlines :(
  • seems like we're back to having to do maintenance every 3 weeks on the cluster
  • going to add nodes to the cluster later today -- we have 23 of which 19 are data
• working on symbols systemtest stuff--won't be done for days because low priority
• still need to update jenkins

Project Updates

Deployment Triage

• Socorro: https://whatsdeployed.io/s-7M7
• Antenna: https://whatsdeployed.io/s-Up2

PR Triage

• http://prs.mozilla.io/mozilla/socorro

Major Projects

Deprecation rampage

• discussed the impact of this on the processor rewrite with Will and Peter
  • it seems getting rid of the reports table entirely before processor rewrite happens would be good
  • I (Adrian) am going to try to make this happen in a timely manner

Processor rewrite

• (willkg) continued working on the processor plan: https://docs.google.com/document/d/1oHLJOmBT9c0-tiDyBWl0PQzBfqJLM1y2abvzq0bFTcs/edit#
• (willkg) worked on figuring out how the processor will run as a process in a docker container
• (willkg) worked on figuring out how raw and processed crash data will get to data storage

• the processor puts missing symbols in postgres, but we don't want to do that. maybe tecken should own maintaining the list of missing symbols?
  • ted's scripts are the only consumers of this data
  • we should work with Ted to figure out how to get that information from the processor to him
  • peterbe will write up a bug to figure this out, willkg will work on it

Upgrading elasticsearch

• (Adrian) no updates

Symbol Server 2017

• Tecken gets into the business of migrating S3 buckets from one AWS org to another
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1361503
  • The problem is that we have two S3 buckets in the webeng infrastructure that we need to move to the Ops infrastructure. That means we need to update configuration for the processor, webapp, and other things. But we also have permissions problems.
  • If we move Soccoro (and processors) to the new AWS infra, many ideas of this migration plan becomes moot.
  • http://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html
  • peterbe will follow up on this with miles

Other Business

• (willkg) Antenna is in -prod. It's doing fine and I think at this point we don't want to go back to the Socorro collector. Can we remove it from the deploy/infrastructure and dashboards? Do we also want to remove the Socorro collector code, tests, and docs from the Socorro repo?
  • We should add a big note to the docs and to the repo README saying Socorro as we know it is done. Maybe fork it as socorro-old, too.
  • Then remove collector code, nix the unit tests for it, nix the integration tests, and maybe flatten the filesystem crash storage based on socorro-collector repo to simplify it.

• (peterbe) What to do with Aurora in the webapp where it's going away, but we don't actually know when.
  • we're going to wait on landing this until things are clearer. mbrandt is working on related things and will know when things happen and will give us a headsup.

• (chartjes) Asked what the swagger status is. Adrian said he'd look into it. Will suggested we write up a bug.

Travel, etc

• Adrian has a French holiday on Monday, May 8th
• Miles might take Friday off.
• Daniel might get sick again.