Socorro/Release Process

New release

• Socorro bugs are triaged and "Target Milestone" field in bugzilla is set
• As bugs are resolved, QA verifies them against the "master branch" staging server (crash-stats-dev.allizom.org)
• config or crontab changes that cannot be made directly in the code should be documented on release-specific pages such as https://wiki.mozilla.org/Socorro:Releases/2.4
  • changes should be made directly on crash-stats-dev and documented on this page, as dependent code is landed
• https://wiki.mozilla.org/Socorro:Releases is updated

Code freeze

• code is force-pushed to the "stage" branch
  • a seperate Jenkins job tracks the stage branch and will create a new build automatically (https://jenkins.mozilla.org/job/socorro-release)
• new build is loaded to the staging server by IT (crash-stats.allizom.org)
  • this same build, pending QA's signoff, will later be pushed production, for example https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
• Refer to release-specific page (e.g. https://wiki.mozilla.org/Socorro:Releases/2.4) and ensure that any system changes (config, crontab, etc) are made
  • system changes are made via puppet, involve IT if you do not have access to the socorro puppet repo
  • verify that all config changes are accounted for, e.g. "git diff v2.4.1 | grep '^diff' | grep config"
• staging data is refreshed and DB updates are run on stage by DBA
• https://wiki.mozilla.org/Socorro:Releases is updated and email sent to socorro-dev and QA

Verification

• QA verifies fixes against branch staging server
• Socorro devs verify logs on each server:
  • /var/log/socorro/*.log
  • /var/log/socorro/kohana/*.php
• Socorro devs or IT verify no pending puppet-managed config changes
  • "diff -x '.svn' -r prod/ stage/ | less" in puppet checkout
• https://wiki.mozilla.org/Socorro:Releases is updated

Push to production

• Ensure that the expected build is listed on https://wiki.mozilla.org/Socorro:Releases and https://crash-stats.allizom.org/status
• Get QA sign off
• Ask DBA to push any release-specific changes that are not already deployed
• file a push bug in Socorro/Infra
  • for example see: bug 735837
  • if there were UI changes, request that IT also kick memcached
• Socorro devs verify logs on each server:
  • /var/log/socorro/*.log
  • /var/log/socorro/kohana/*.php
• Devs can also verify from StatsD, Ganglia, and Status Page
• Schedule verification of cron job changes for after their next scheduled run
• https://wiki.mozilla.org/Socorro:Releases is updated
  • schema dump is checked into git (e.g. sql/schema/2.4/breakpad_schema.sql)
    • pg_dump -U $databaseUserName -h $databaseHost -s -T '*_201*' $databaseName > breakpad_schema.sql
    • also update breakpad_roles.sql if any roles changes have been made
  • release is tagged (GPG signed) and pushed
    • set up gpg (on OSX: `brew install gnupg; gpg --gen-key`)
    • add `signingkey = [your key]` to ~/.gitconfig under 'user'
    • `git tag -s -m "bug 733931 - push 2.5 release" v2.5`
    • `git push mozilla v2.5`
  • email sent to socorro-dev and tools-socorro lists

PostgreSQL Database Changes

Cumulative schema changes are stored in the various directories under /scripts/upgrades in the source code. Each directory is tied to a Socorro version (e.g. /scripts/upgrades/2.1/), and contains scripts to upgrade from the prior release version to the current version.

For most database changes, there will be a shell script upgrade.sh. This shell script will contain calls to one or more python scripts and/or SQL files contained in the same directory and will execute them in order. In some cases, there will be a different script for the development server, in which case there will be an upgrade-dev.sh script to be executed on development and stage. Unless otherwise noted in upgrade.sh, SQL database changes are to be deployed before application code changes.

Some SQL database changes require downtime or require large amounts of time to execute, or may require downtimes. These will be scheduled and executed in advance of the code upgrade. Their scripts will be present in upgrade.sh for reference, but commented out.

stageDB and production autosync the Socorro SVN repository trunk automatically every hour. As such, scripts should be available to execute on each database server in /data/socorro/applications/scripts/upgrades/

So, to sum up:

• Before deploying application code changes:

1. check /data/socorro/application/scripts/upgrades/ver.num/upgrade.sh for any notes
2. execute /data/socorro/application/scripts/upgrades/ver.num/upgrade.sh