Socorro:Staging

Socorro currently has two staging boxes (PostgreSQL and Hadoop/HBase are not covered here):

collector + crashmover + webapp-php

• collector
  • https://crash-reports.stage.mozilla.com
  • receives crashes via HTTP POST
  • saves crashes to local disk

• crashmover
  • reads crashes from disk
  • inserts to PostgreSQL and HBase, queues up jobs for monitor (below)

• webapp-php
  • https://crash-stats.stage.mozilla.com
  • pulls crash info from PostgreSQL and middleware
  • rewrites URLs for rawdumps to middleware

middleware + monitor + processor + cronjobs

• middleware
  • TODO URL for raw dumps (rewritten by crash-stats)
  • mediates webapp-php and PostgreSQL/HBase

• monitor
  • checks queue for crashes to-be-processed, assigns to processor

• processor
  • pulls raw crash from HBase, runs minidump_stackwalk, inserts results to HBase and PostgreSQL

• cronjobs
  • cron_aggregates.sh - updates aggregate top_crash_by_signature table
  • cron_bugzilla.sh - search bugzilla for crash signatures
  • cron_builds.sh - search FTP for build info (e.g. changeset)
  • cron_tcurl.sh - updates aggregate top_crash_by_url table
  • cron_daily_reports.sh - daily CSV reports
  • cron_libraries.sh - daily CSV reports
  • cron_daily_crashes.sh - processes ADU data (?)
  • cron_status.sh - updates https://crash-stats.stage.mozilla.com/status
  • cron_create_partitions.sh - creates reports_ partitions