Releases/End-to-end Automation

This page will track the project to enable end-to-end automation of the release process.

Goal

• For releases, a driver clicks a big red button on a webpage and some short time later the build is on mirrors, updates fully tested, the website is updated, and release announcements are sent
• During that time there is a dashboard to track status
  • Giving time estimates based on history as well
• No humans are involved in the process unless something goes wrong
• If some part goes wrong, the process doesn't need to start over completely

Why?

The release mechanics have (in the best case) clearly defined steps with expected inputs and outputs. This should be automated and humans should only spend their time dealing with exceptional cases and babysitting the automation

Benefits

1. Releases can be quicker
2. Employees no longer have to stay late for a release
3. Pipeline and front-load the process, teasing out implicit dependencies
4. Less human error potential
5. Better metrics across the whole release process, which can driver future improvements
6. Standard tools and processes across products

General Plan

1. Identify the current process, breaking it down into steps and their dependencies
2. Automate any step (changing websites, pushing websites, etc) that needs human intervention
3. Connect the steps in some way (likely Pulse, webhooks and email can do in a pinch)
4. Test on a release that is not time-critical (betas, etc)
5. Put into production
6. Make more robust, improve areas, etc

Meetings

TBD

Systems that need to interact

• hg.mozilla.org
  • build configs, metadata, etc
• RelEng's buildbot
• QA's update automation
• Mozmill
• bugzilla (if we need to file bugs for IT and tuff?)
• AMO
  • For new product versions, etc
• Socorro
  • For new product versions, current version, etc
• QA's results DB
• sentry
• CDN (might be the same as sentry)
• mozilla.com svn
  • Release notes
  • .htaccess redirects
  • MFSAs (?)
• product-details svn
  • history and details
• wordpress (for blog announcements)
  • Need to have queued and automation push live
• mailman / imap / email to announce list

Possible Messages

NOTE: This is a quick brain dump

• Build started message
• Build finished message
• Build posting started message
• Build posting finished message
• Snippet betatest posting started
• Snippet betatest posting finished
• Build ready for betatest testing message
• testing started message
• testing progress messages
• testing finished messages
• release to internal mirror message
• Write bug to turn on crash-stats throttling
• Different stages of mirror release messages
• Mirror uptake progress messages
• Mirror uptake sufficient message
• Snippet releasetest posting started
• Snippet releasetest posting finished
• Build ready for releasetest testing message
• p-d pushed to trunk message
• p-d pushed to staging message
• Blog post drafted message
• Pages pushed to trunk message
• Pages pushed to staging message
• Pages pushed to production message
• testing started message
• testing progress messages
• testing finished messages
• Snippet release posting started
• Snippet release posting finished
• Build ready for release testing message
• testing started message
• testing progress messages
• testing finished messages
• p-d pushed to production message
• announce email sent message
• blog post live message