ReleaseEngineering/Releaseduty/FAQ: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
m (Typo fix.)
(Update link to releasewarrior-2.0)
 
(14 intermediate revisions by 2 users not shown)
Line 1: Line 1:
0. '''Which IRC channels should I join?'''
This page has been moved to [https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/day1/ReleaseDuty-Day-1-Checklist-and-FAQ.md releasewarrior]
 
Good starters are: #releaseduty, #tbdrivers, #release-drivers. There are also mailinglist subscriptions to [https://groups.google.com/a/mozilla.com/forum/?hl=en#!forum/release-automation-notifications release-automation-notifications] as well as the <thunderbird-drivers@mozilla.org>, <release-automation-notifications-thunderbird@mozilla.org>,  <release-drivers@mozilla.org> and <release@mozilla.com>.
 
1. '''How does the Ship-it workflow work in terms of shipping a new release?'''
 
Release Manager (RelMan) submits a new release form, another RelMan reviews that and once it hits 'Ready' the release enters the 'Reviewed' section and wais to be run. Since there's a release-runner.sh script running in a loop on bm81, there's a max windows of (60 seconds-ish?) till the job gets its share. Following which it enters the 'Running/Complete' table where we can observe its state. The "Reviewed" tab goes to "No pending release" yet again. This drawing depicts more technical details on this topic, in the Release-runner / Ship-it section.
 
2. With respect to release-runner, I grasped the release-runner part in the graph. However the http://hg.mozilla.org/build/tools/file/tip/buildfarm/release/release-runner.py#l309 shows me a TC task is created.
* '''Where are those operations''' (patch, tag, sanity check, push &reconfig, trigger) '''being run'''?
* '''What does the scheduler.createTaskGraph mean and the mark_as_completed part'''?
 
<code>TODO answer</code>
 
3. '''What does release-promotion refer to within all this context'''?
 
'release promotion' is simply the idea that we take an already existing CI build from (e.g.) beta and 'promote' that to being the build we release/ship to users. Prior to this idea, we have always rebuilt Firefox at the start of each new release. The diff between the current CI builds and CI builds going forward is that they will have slightly different mozconfigs and bits that make them as ready-to-ship. The assumption is that each CI build is releasable without changing it, so the signatures are final. (yes, no more bumping versions/build configs/mozconfigs/signatures!!)
 
4. '''In a few steps, how does a beta generation process looks like?'''
 
Please address [[Release:Release_Automation_on_Mercurial:Preparation#Overall_beta_release|this]] schema for details.
 
5. Somebody on IRC said "we need to update several stuff till the beta goes out in the release channel". '''Are all these hotfixes for stuff that has been observed since the changes started riding the trains?'''
 
Yes! Since 2012 Mozilla moved to a fixed-schedule release model, otherwise known as the Train Model, in which we released Firefox every six weeks to get features and updates to users faster and move at the speed of the Web. Hence, every six weeks the following merges take place:
 
[http://hg.mozilla.org/mozilla-central/ mozilla-central] => [http://hg.mozilla.org/releases/mozilla-aurora/ mozilla-aurora]
 
[http://hg.mozilla.org/releases/mozilla-aurora/ mozilla-aurora] => [http://hg.mozilla.org/releases/mozilla-beta/ mozilla-beta]
 
[http://hg.mozilla.org/releases/mozilla-beta/ mozilla-beta] => [http://hg.mozilla.org/releases/mozilla-release/ mozilla-release]
 
So before we even ship a new release, there's a 6-8 weeks of betas in which we test things. Unexpected bugs can and usually come up in this time so there's need for hotfixes.
 
6. Overall rate vs update rate?
* '''What's the overall rate everyone's keep talking about in the release channel mtg?'''
(e.g. Overall rate: 1.0 - yellow (almost green), on the upper edge of what 42 data looked like)
* '''when they are talking about "update rate from 30 to 100" they are talking about throttling a release?'''
 
<code>TODO answer</code>
 
7. All of the following bugs are excerpt from a channel mtg. '''Are all these bugs noticed when 44 was already riding the trains in beta channel?'''
 
Overall rate: 1.1 - yellow, slightly up from last week of 43
* bug 1212133 (a11y::DocAccessible::RemoveDependentIDsFor) is 2.2% of 44.0b1 data
* bug 1233481 (js::AutoEnterOOMUnsafeRegion::crash) is 2.0%
* bug 1215970 (CondVar::Wait) is 1.5%, see release
* bug 1217135 (layers::ImageBridgeChild::EndTransaction) is 1.3%
* bug 1234170 (nsIWebSocketChannel::Serial) is 1.2%
* bug 1233962 (net::InterceptedChannelBase::DoNotifyController) is 1.1%
 
'''Therefore, are all of these hotfixes that need to get immediately in the release?'''
 
<code>TODO answer</code>
 
8. Scenario: Release-runner triggers a sendchange to buildbot. Buildbot builds. '''Is there any automatic testing chained up as soon as the build is ready?''' (as in, the same testing I see on treeherder for regular granular commits?)
 
<code>TODO answer</code>
 
9. Just a general overview, no need for micro-level details, '''how do the QA tools work'''?
* their testing is subsequent to automatic testing, right? (that is regular tests from treeherder)
* how do they test a build? Do they have specific tools?
* do they focus on verifying bugs that are "stated as fixed" for that specific release? (whatsnew page)
 
<code>TODO answer</code>
 
10. '''What is a partner repack change for FF?''' What does "partner" refer to in this context? (e.g. Bug 1231679)
 
<code>TODO answer</code>
 
11. During the release-drivers meeting, people often refer to conversations noticed on Twitter (as in complaints or positive feedback on something). How do people know what's on Twitter? '''Is there a specific handle or hashtag they're following or it's just the @mozilla and related conversations?'''
 
<code>TODO answer</code>
 
12. '''Is there a programmed calendar for the Thunderbird events?'''As it is for FF desktop or it's just something that is being setup in the #release-drivers and synced-up with TB folks?
 
<code>TODO answer</code>
 
13. I got confused by the order of the things happening.
 
As soon as the build is completed and available, the QE starts testing it? (manual, regresion, smoking, etc)
I ask that because earlier today I was debugging some firefox_antivirus and update_verify_release on win64 steps when I saw an email like "[desktop] Firefox 43.0.4 (build 3) - Sign Off on Manual Functional Testing".
Indeed the build was successful but there were many following steps afterwards that were not yet completed. '''How does it work, when does QE actually start testing?'''
 
It's easy! QE starts testing as soon as they put their hands on the candidate release. In order to better understand the main logic, please see [https://www.lucidchart.com/documents/view/b733c8be-e607-445f-824a-f3353c287294 this] drawing. To better correlate, please find the below list of emails that arrive (usually) chronologically (excerpt from releasing the 45.0b2):
 
* [release] Firefox 45.0b2 build1: tagging started for Firefox 45.0b2
* [release] Firefox 45.0b2 build1: completed firefox_reset_schedulers
* [release] Firefox 45.0b2 build1: Source Repo tagging complete
* [release] Firefox 45.0b2 build1: L10n Repo tagging complete
* [release] Firefox 45.0b2 build1: completed firefox_bouncer_submitter
* [release] Firefox 45.0b2 build1: completed firefox_source
* ''(depending on the platform, e.g. on macosx64 below)''
* [release] Firefox 45.0b2 build1: macosx64 en-US build available
* ''(for each platform, 10 emails tracking l10n repacks (e.g. Linux 10/10 repacks email)''
* [release] Firefox 45.0b2 build1: completed repack_10/10 on linux
* [release] Firefox 45.0b2 build1: All win64 builds now available
* [release] Firefox 45.0b2 build1: Updates available on beta-localtest
* [release] Fennec 45.0b2 build1: completed firefox_checksums
* [release] Firefox 45.0b2 release
* <font color="red">'''Once this watershed is being hit, QE has both the patient and the tools to pursue the surgery tests! Upon successful sign-off, we usually need to wait for the update verify tests to be run and the antivirus check.'''</font>
* [release] Firefox 45.0b2 build1: completed update_verify_beta_6/6 on linux64
* ''.... (6/6 emails for each platform)''
* [release] Firefox 45.0b2 build1: completed firefox_antivirus
* ''Upon pushing to Balrog from RelEng side, the following email announces final victory!''
* [release] Firefox 45.0b2 build1: Updates available on beta
 
14. How come the same RC ready-to-be-shipped (e.g. Desktop Firefox 44) results in bunch of errors (GTK3-related, binary found on diff, etc) on the update_verify_beta but completes successful on the update_verify_release?
 
<code>TODO answer</code>
 
15. How come Windows64 is not in the platform list for 38.6.0esr? http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/release-firefox-mozilla-esr38.py#l71
 
16. Email stages I've noticed for <release> and <beta>.
 
'''Are these stages correct?'''
 
'''release'''
 
* [desktop] Firefox 43.0.4 (build 3) - Sign Off on Manual Functional Testing by Softvision
assumption: QA performs manual and smoke testing following which it signs-off and hands to release-drivers
* [Desktop] Firefox 43.0.4 build3 signed off on the release-localtest channel by QE
assumption: release-drivers perform update testing on release-localtest channel following which they sign-off and hand to release-management
* [desktop] Please push Firefox 43.0.3 (build#1) to the release-cdntest channel
assumption: RelMan asks releng to push to release-cdntest (aka push to mirrors / push to cdn)
* releng pushes to mirrors
* [Desktop] Firefox 43.0.3 signed off on the release-cdntest channel by QE
assumption: QE performs the update tests on the release cdn-tests  following which they sign off and hand to releaase managers
* [desktop] Please push Firefox Desktop 43.0.3 (build #1) to the release channel - 100% update rate
assumption: release managers ask releng to push to release channel (aka push to balrog)
* releng pushes to release channel
* release-automation sends notification: "[release] Firefox 43.0.3 build1: Updates available on release"
* [Desktop] QE signing off on 43.0.3 release
assumptioon: release-drivers perform the updates tests on the release channel
* releng runs post-release for the release
* release-automation sends notification stating that: "[release] Firefox 43.0.3 build1: completed firefox_postrelease"
* done
 
'''betas'''
 
* [desktop] Firefox 44.0 Beta 4 (build 1) - Sign Off on Manual Functional Testing
assumption: QA performs manual, regression and smoke testing on beta-localchannel (??) following which it signs-off and hands to release-drivers
* [Desktop] Please push Firefox 44.0b4 build1 to the beta channel
assumption: QE and release-drivers perform the updates on the beta-cdntest channel and ask releng to push it to beta channel
* releng performs the 'Push to balrog' action
* release-automation sends notification stating: "[release] Firefox 44.0b4 build1: Updates available on beta"
* [Desktop] Firefox 44.0b4 updates on beta channel signed off by QE
assumption: QE performs update testing on the beta channel and ask releng for post-release step
* releng performs the 'post-release' action step
* release-automation sends notification stating that: "[release] Firefox 44.0b4 build1: completed firefox_postrelease"
* done
 
Questions:
* My explanation so far is that betas are auto when it comes to push-to-mirrors, while for releases this step is to be done manually. '''Is this correct?'''
* '''what does the "updates testing" refer to?''' I saw it performed at various steps during the release process. Is it automated / stage?
* '''for betas, is there a beta-localtest or is it the same thing with beta-cdntest'''?
 
<code>TODO answer</code>
 
17. '''Why don't I see update_verify_beta for dot releases?'''
 
From time to time, a handful of issues precipitate a dot release. When that happens, its behavior slightly varies from a normal release. A normal release (e.g. 43.0, 44.0, etc) has its RC shipped to beta channel first before making it to release channel - for testing purposes, update verify steps are taking place both ways, hence update_verify_release '''and''' update_verify_beta steps. Upon successful testing we ship the RC on the beta channel and then on the release channel, following which we merge the code for the next release cycle so that the beta release bumps its version. In the lights of this logic, a dot release (e.g. 43.0.1 or 44.0.1) happens a certain amount of time '''after''' the official release. For that reason, a dot release can't be tested in beta channel as the at-that-moment beta version is greater than the dot release version, hence the updater would refuse to downgrade. Therefore, there is only one cycle of update_verify for dot releases (update_verify_release == update_verify in this case).
 
18. '''As part of the release process, which step takes the user's Firefox to prompt the "There is a new update. Please click below [...]"?'''
 
The tl;dr is "Publish to Balrog". That is the magic step that makes the hot stuff available to all the users. However, there are certain bits to complete the equitation here. Once a release completes its deliverables and its signing is done - that release is available in the release-localtest channel. After the release passes by the push-to-mirrors step it arrives in the release-cdntest channel. Finally, when the release is being pushed to balrog, it reaches to its release channel. Normally, each of these steps require a QE sign-off (and, for some, a RelMan approval).
Balrog also has the so-called throttling rules which control the outcome population for each release. In the most common scenario, the throttling steps come before the release gets actual published in Balrog. That is - change/add/update the rules so that the reaching user population does not exceed a certain X% amount (0 <= X <= 100). Even though the windowing would be small, we wouldn't want to publish first and then update the rules. It's against the logic. However, there are certain scenarios in which this action can take place. Consider having a release (e.g. Firefox 122.0) that was found to having high crash rates and has been set to a 0% user-absorption rate. A follow-up dot release is imminent in this case (say Firefox 122.0.1). If we update the rates to some greater values before pushing the dot release to Balrog, the users will be pointed to the old-release (Firefox 122.0). That's why in this case, we firstly push the dot release to Balrog and only then change the rates to some greater values. Whenever we push a new release to Balrog, its buildbot-configs update configs use the ruleId to map to that given rule and update the Balrog database. Long story short, if it's a dot release, the rules are there already so you push it first and then slightly adjust the rules.
Upon successful push to Balrog, users across the globe will start getting their Firefox prompt asking them to update their version. So this happens after Push-to-Balrog, and NOT after post-release. However, not all the browsers will ask for that right away. Each Firefox process has an internal clock that runs to ask for updates in a given window timeframe of X hours (typically 24). It depends on each clock's measurements, sync and functionality on when to ask for that. However, we can be sure that after the completion of the entire time frame, all browsers across the world would have asked the question to the user.
Worth to mention here that there is a Ship-it option that enforces update with immediate or planned action. The option is controlled by RelMan whenever they trigger the release in Ship-it.

Latest revision as of 22:38, 12 July 2018

This page has been moved to releasewarrior

Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/Releaseduty/FAQ&oldid=1197210"