Balrog/Meetings/CloudOps - January 18, 2017: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
Line 6: Line 6:
** mostlygeek will run on production with read only credentials when ready
** mostlygeek will run on production with read only credentials when ready
* Contract tests
* Contract tests
** chartjes working on them, wants to run as part of deployment
** chartjes working on them
** Will be set-up on QA jenkins and kicked off as part of deployment
* Athena/Log parsing
* Athena/Log parsing
** We have 1.6TB logs/day
** We have 1.6TB logs/day

Revision as of 19:41, 18 January 2017

Attendees

bhearsum, mostlygeek, chartjes, ckolos

Notes

  • Running new extraction script on prod db, or clone?
    • mostlygeek will run on production with read only credentials when ready
  • Contract tests
    • chartjes working on them
    • Will be set-up on QA jenkins and kicked off as part of deployment
  • Athena/Log parsing
    • We have 1.6TB logs/day
    • Not efficient to use Athena - ~$5/TB/query
    • Someone from ops is looking at converting raw logs into Apache parque(?) format
      • Probably will get ~99% improvement in efficiency
      • No loss of info or granularity
      • Should be able to enable Athena afterwards
Retrieved from "https://wiki.mozilla.org/index.php?title=Balrog/Meetings/CloudOps_-_January_18,_2017&oldid=1159919"