ReleaseEngineering/Rapid Release Nightly Betas

From MozillaWiki
Jump to: navigation, search

Starting in July (7/17/2012) we will be doing Beta releases at the same frequency as Nightly and Aurora builds. That is, every night as long as there is changes. If we are to continue to use the Release Automation to do these, it needs to get to the point of being 100% automated. Let's figure out all we need to do to get there! To start, here's a list of things that are currently done manually. What's involved, and how hard would it be to automate these?

Live Notes

  • Configuration bumping
    • version bumps (beta1..beta42)
    • need a tool to do configuration bumps
      • might have some of this written as part of preprod releases
    • buildbot-configs, tools, buildbotcustom tagging
      • should we tag at all?
      • could clutter things up
      • maybe we can dump revs to a file (configs, custom, tools, en-US (mozilla-beta), etc.)
        • could be done in whatever replaces tagging+source factory
    • l10n changesets ! -- https://wiki.mozilla.org/Release_Management/Rapid_Betas#L10N Rel-Man will work with Axel to determine the 'new hotness' here
      • should be able to pull it from _somewhere_. questions around how sign offs work, not really relevant to release automation
  • Do we need to bump version number?
    • releases/17.0-betas/date-version
    • can use a static configuration file if we don't have to bump the version
  • Reconfiging the master
    • reconfigless!!
      • builds update to explicit revision right now
      • source builders require a reconfig
      • updates builder requires a reconfig
      • l10n verify requires a reconfig
        • let's just kill it?
        • maybe not useful for nightly betas, or at all
        • kill it!!!!!!! (bhearsum)
      • bouncer submission reconfig
    • slave shellcommand can workaround hanging builder issues
    • release specific master makes this less scary
    • might not need to reconfig if we don't bump version numbers
  • snippet explosion
    • maybe use nightly logic instead of explicit
    • run snippet optimization regularly
  • should be we build mobile every night?
    • unlikely that we'll be shipping it (user PAIN!)
    • frees up releng resources, because they won't have to drive weekly betas
    • want signing on demand (not a blocker though)
  • can find previous updates as long as we have a config that gets bumped every night
    • or we can do nightly-style and grep ftp
  • what to do about mirrors?
    • could use up a lot of space?
    • nthomas might have concerns here, not sure what they are
  • probably need bouncer entries that we update
    • need to check that tuxedo API supports it
  • Setting clobbers
  • Symlinking nightly directory to candidates
  • should we automate archiving old candidates dir?
  • Reserving slaves
    • Is this a concern for nightlies? load is lower (usually) than daytime releases, and time window is different (e2e time expectations) - can we treat like a normal nightly (no reserve)?
      • let's not do it, e2e time doesn't matter so much!
  • Metrics e-mails
    • Subject to how metrics will be tracking these rapid betas - may not need to email, but will put that on the list with Rel-Man investigation
    • also will be affected by whether we use version numbers or something else
  • Android signing
    • we'll only be doing one weekly mobile (avoiding update fatigue) so signing doesn't need to happen every night
  • Backupsnip in advance of push to beta
    • is it worthwhile?
      • maybe not
  • Pushsnip to beta
  • verification script rebuilds
    • not using l10n verify
    • do we need final verify?
      • need to test releasetest snippets somehow
      • maybe run update verify against releasetest?
        • need to move it to happen after internal mirrors
  • Pushing to Google Play

n/a, we'll be pushing mobile betas manually.(and weekly)

  • Do we need to do source? See Questions: check with legal
    • don't do it unless we have to

Questions

  • Are we really going to be tagging x4 every day?
    • Do we need to continue to tag at all?
      • not if we're reconfigless!
  • How do we deal with l10n changesets?
    • we'll get them from somewhere, see action item below
  • Who is on the hook for rescuing broken beta builds -- buildduty?
  • This could take up a lot of disk space
  • Check with Legal as to our responsibility to produce source package for every beta
  • Is it worthwhile, compared to using nightly logic?
    • probably should scope out using nightly logic for beta first

Action Items