ReleaseEngineering/Maintenance: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(reconfig)
 
Line 1: Line 1:
This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.
This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.  


[[ReleaseEngineering:BuildbotBestPractices]] describes how we manage changes to our masters.
[[ReleaseEngineering:BuildbotBestPractices]] describes how we manage changes to our master buildbots.


= Relevant repositories =
= Relevant repositories =
* [http://hg.mozilla.org/build/buildbot/ buildbot]
* [https://hg.mozilla.org/build/buildbot/ buildbot]
* [http://hg.mozilla.org/build/buildbot-configs/ buildbot-configs]
* [https://hg.mozilla.org/build/buildbot-configs/ buildbot-configs]
* [http://hg.mozilla.org/build/buildbotcustom/ buildbotcustom]
* [https://hg.mozilla.org/build/buildbotcustom/ buildbotcustom]
* [http://hg.mozilla.org/build/tools/ tools]
* [https://hg.mozilla.org/build/tools/ tools]
* [http://mxr.mozilla.org/mozilla/source/testing/performance/talos/ talos]
* [https://hg.mozilla.org/build/mozharness/ mozharness]
* [https://hg.mozilla.org/build/puppet/ puppet]


= Ready to rollout =
= Non-releng repositories involved in the CI =
== [[Auto-tools/Projects/Treeherder|Treeherder]] ==
* [http://treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/history Chief self-deploy recent history] (needs VPN; more readable than raw logs, but is cleared if the Chief service is restarted)
* [treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/logs/ Chief self-deploy logs archive] (needs VPN; use if the history page is empty)


{| class="fullwidth-table sortable"
= Mozharness deployments =
| style="background:#cccccc" | '''Bug #'''
Merges to the <em>production</em> branch of [http://hg.mozilla.org/build/mozharness/graph the mozharness repository] are live immediately.  Please see that repo for push history.
| style="background:#cccccc" | '''Description'''
 
| style="background:#cccccc" | '''Master'''
= Puppet deployments =
| style="background:#cccccc" | '''Urgency'''
 
| style="background:#cccccc" | '''Status'''
Merges to the <em>production</em> branch of [http://hg.mozilla.org/build/puppet/graph the puppet repository] are automatically deployed by Puppet within a 1/2 hour.  Please see that repo for push history.
| style="background:#cccccc" | '''Need downtime?'''
 
| style="background:#cccccc" | '''Owner'''
Please note if you are applying a puppet change, '''unless you have good reason not to do so''' please merge it to production branch immediately. The reason for this is that we do not want puppet changes to "back up" on default, and then land ''en masse'' in production. It can be very difficult to untangle multiple puppet changes that arrive together.
|}
 
The reason we have a production branch at all, is to allow contributions on default branch from a larger group of people, than those responsible for maintaining the infrastructure controlled by the puppet repo - in other words, a smaller group of people can merge to production, than the group of people that can land on default.
 
= Reconfigs / Deployments =
This page is updated by the person who does a [[ReleaseEngineering:Buildduty:Reconfigs|reconfig]] on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as <i>mozharness</i> changes.)
 
Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.


= Upcoming changes =
Older changes are archived in: [[ReleaseEngineering/Changes/2016|2016]] | [[ReleaseEngineering/Changes/2015|2015]] | [[ReleaseEngineering/Changes/2014|2014]] | [[ReleaseEngineering/Changes/2013|2013]] | [[ReleaseEngineering/Changes/2012|2012]] | [[ReleaseEngineering/Changes/2011|2011]] | [[ReleaseEngineering/Changes/2010|2010]] | [[ReleaseEngineering/Changes/2009|2009]]


{| class="fullwidth-table sortable"
{| class="fullwidth-table wikitable sortable"
| style="background:#cccccc" | '''Bug #'''
| style="background:#cccccc" | '''Outcome'''
| style="background:#cccccc" | '''Description'''
| style="background:#cccccc" | '''When'''
| style="background:#cccccc" | '''Master'''
| style="background:#cccccc" | '''Bug #(s)''' - '''Description(s)'''
| style="background:#cccccc" | '''Urgency'''
| style="background:#cccccc" | '''Status'''
| style="background:#cccccc" | '''Need downtime?'''
| style="background:#cccccc" | '''Bug Owner'''
|-
|-
| unknown
| in production
| netapp firmware update (joint goal w/IT)
| 2017-09-21 07:51 PT
| unknown
| unknown
| unknown
| no
|
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1333792,1384626&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1384626}} - Bump central - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/300ee5bb7690 300ee5bb7690])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1333792}} - Try push confirmation emails for Taskcluster jobs should link to the Task inspector rather than the upload directory on archive.m.o - r=mtabara ([https://hg.mozilla.org/build/buildbotcustom/rev/994ad65b2abd 994ad65b2abd])
|-
|-
| {{bug|463262}} automate nonsse unittest, being done manually for now
| in production
| unknown
| 2017-09-14 03:55 PT
| unknown
| unknown
| unknown
| unknown
|
|
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1384626,1392488,1397721&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
</div>
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
* {{bug|1384626}} - m-r gecko version bump bbconfigs - r=jlorenzo ([https://hg.mozilla.org/build/buildbot-configs/rev/eabff7c418b6 eabff7c418b6])
* {{bug|1397721}} - Support cross channel l10n in buildbot, part 1 - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/618145e94126 618145e94126])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1392488}} - Skip commit created by try fuzzy for sensible commit title. r?catlee ([https://hg.mozilla.org/build/buildbotcustom/rev/45e90cfbfd5e 45e90cfbfd5e])
|-
|-
| unknown
| in production
| roll out ability to get crash dumps from frozen browser on winxp/vista
| 2017-09-06 10:14 PT
| unknown
| unknown
| unknown
| unknown
|
|
|}
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
 
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1384482&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
= Previous changes =
</div>
 
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.
* {{bug|1384482}} - Strip useless android-api-16 config - r=mtabara ([https://hg.mozilla.org/build/buildbot-configs/rev/8300468d9cce 8300468d9cce])
 
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
{| class="fullwidth-table sortable"
* {{bug|1384482}} - Keep using android-api-15 config - r=mtabara ([https://hg.mozilla.org/build/buildbotcustom/rev/b2fbd21e6e01 b2fbd21e6e01])
| style="background:#cccccc" | '''Outcome'''
| style="background:#cccccc" | '''When'''
| style="background:#cccccc" | '''Bug #'''
| style="background:#cccccc" | '''Description'''
| style="background:#cccccc" | '''Master'''
| style="background:#cccccc" | '''Urgency'''
| style="background:#cccccc" | '''Status'''
| style="background:#cccccc" | '''reconfig/restart required'''
| style="background:#cccccc" | '''Bug Owner'''
|-
|-
| in production
| in production
| 20100122 1600 PST
| 2017-08-31 06:50 PT
| {{bug|536188}}
|
| Created project branch "Lorentz"
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm02
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1384482&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| medium
</div>
| landed
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
| reconfig
* {{bug|1384482}} - part2: Activate api-16 on beta - r=mtabara ([https://hg.mozilla.org/build/buildbotcustom/rev/dfeb07934f0f dfeb07934f0f])
| lsblakk
|-
|-
| in production
| in production
| 20100122 0900 PST
| 2017-08-06 22:46 PT
| {{bug|541169}}
|
| UnittestPackagedBuildFactory hardcodes "firefox"
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm*
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1387878&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| low
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1387878}} - Buildbot DevEdition builds still running on mozilla-beta - r=mtabara ([https://hg.mozilla.org/build/buildbot-configs/rev/6bddeb6c48ea 6bddeb6c48ea])
| bhearsum/kairo
* No bug - Update release config for Thunderbird-56.0b1-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/f26ec86b000e f26ec86b000e])
|-
|-
| in production
| in production
| 20100122 0900 PST
| 2017-08-04 03:42 PT
| {{bug|541282}}
|
| Bump build space for win32 opt builds
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm*
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| low
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* No bug - Update release config for Thunderbird-56.0b1-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/663ad30ff27d 663ad30ff27d])
| bhearsum
* No bug - Relpro  mac and win builds index changes required for 56.0b1 - r=mtabara ([https://hg.mozilla.org/build/buildbot-configs/rev/3c7a94990487 3c7a94990487])
|-
|-
| in production
| in production
| 20100121 0930 PST
| 2017-08-02 00:27 PT
| {{bug|507024}}
|
| WinMo mar and snippet generation
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm*
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1369535,1385134,1385435&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| low
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1369535}} - m-b, m-c,comm-b gecko version bump bbconfigs - r=jlorenzo ([https://hg.mozilla.org/build/buildbot-configs/rev/e0963a01070a e0963a01070a])
| coop
* {{bug|1385134}} - Staging release: point to jlorenzo's disposable repo - r=mtabara ([https://hg.mozilla.org/build/buildbot-configs/rev/3738fcf37144 3738fcf37144])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1385435}} - Use python 2.7 to run client.py in comm-central L10N repacks - r=nthomas ([https://hg.mozilla.org/build/buildbotcustom/rev/c48ef0db803e c48ef0db803e])
|-
|-
| in production
| in production
| 20100121 0930 PST
| 2017-07-31 04:12 PT
| {{bug|540913}}
|
| Switch partner repack automation to use the 1.9.2 branch
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm*
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1369535&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| low
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1369535}} - m-r gecko version bump bbconfigs - r=jlorenzo ([https://hg.mozilla.org/build/buildbot-configs/rev/acbb8cf0729e acbb8cf0729e])
| coop
* No bug - No Bug - Staging release: Point to a disposable balrog instance - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/c700658593c4 c700658593c4])
* No bug - No Bug - Staging release: Point to rail's disposable repo - r=jlorenzo ([https://hg.mozilla.org/build/buildbot-configs/rev/749272eda177 749272eda177])
|-
|-
| in production
| in production
| 20100120 0930 PST
| 2017-06-12 03:31 PT
| {{bug|497156}}
|
| add partner repacks to release automation
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm*
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1319436&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| medium
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1319436}} - stabilizing branches merge day - r=jlorenzo ([https://hg.mozilla.org/build/buildbot-configs/rev/45a1996a9a6d 45a1996a9a6d])
| coop
|-
|-
| in production
| in production
| 20100120 0930 PST
| 2017-06-05 06:32 PT
| {{bug|540743}}
|
| bump build space for mac opt builds on mozilla-central and related branches
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm*
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1370182&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| medium
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1370182}} - add missing Win10 information in config seta - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/f3b2449ff762 f3b2449ff762])
| bhearsum
|-
|-
| in production
| in production
| 20100120 0918 PST
| 2017-06-05 02:12 PT
| {{bug|539938}}
|
| Hand a branch env var to generate-snippets-%
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm*
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1319436&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| low
</div>
| [http://hg.mozilla.org/build/buildbotcustom/rev/f3925d17cec9 landed]
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1319436}} - bump release to 54 - r=aki ([https://hg.mozilla.org/build/buildbot-configs/rev/748e07c05bb6 748e07c05bb6])
| KaiRo
|-
|-
| In production
| in production
| 20100120 0810 PST
| 2017-05-11 06:12 PT
| {{bug|511616}}
|
| use different l10n.ini for Fennec
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm02
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1358607&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| low
</div>
| [http://hg.mozilla.org/build/buildbot-configs/rev/bb59e181e10f landed]
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1358607}} - add DevEdition corresponding configs - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/1df37d26fe89 1df37d26fe89])
| armenzg
|-
|-
| In production
| in production
| 20100120 0810 PST
| 2017-03-16 07:25 PT
| {{bug|533780}}
|
| add repack-on-change for Win32 Desktop builds
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm02
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1346129,1346806&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| low
</div>
| [http://hg.mozilla.org/build/buildbot-configs/rev/8d4222ba4839 landed]
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1346129}} - use linux64's source mozconfig instead of release - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/09752149182b 09752149182b])
| armenzg
* {{bug|1346806}} - add bouncer submission bb-configs - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/26d66bf3bcbd 26d66bf3bcbd])
|-
|-
| in production
| in production
| 20100119 1631 PST
| 2017-03-10 15:02 PT
| {{bug|539011}}
|
| turn off nightly builds for project branches that don't need them
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm*
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1344229&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| low
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1344229}} - add bb configs for bump versioning and marking release as shipped - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/c7a0a2fee71e c7a0a2fee71e])
| lsblakk
|-
|-
| in production
| in production
| 20100119 1524 PST
| 2017-03-10 10:11 PT
| {{bug|393411}}
|
| reboot slaves after mobile builds
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm02
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1344225&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| low
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1344225}} - Add bb-configs for bouncer aliases - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/67085f3c07bc 67085f3c07bc])
| aki
|-
|-
| in production
| in production
| 20100119 1200 PST
| 2017-03-09 13:18 PT
| {{bug|525674}}
|
| Fennec repacks on change is trying to load incorrect all-locales URL
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm02
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1344202&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| medium
</div>
| [http://hg.mozilla.org/build/buildbotcustom/rev/3f8614275080 landed]
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1344202}} - Add bb-configs dedicated uptake monitoring platforms - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/8b3fffe1e92c 8b3fffe1e92c])
| armenzg
* No bug - Update release config for Thunderbird-52.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/3d2531eb564d 3d2531eb564d])
|-
|-
| in production
| in production
| 20100119 0820 PST
| 2017-03-06 11:57 PT
| {{bug|540249}}
|
| talos should fail more gracefully when it is unable to run 10.5 sdk based builds
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| talos-pool,talos-try (talos-master.m.o)
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1344768&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| medium
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1344768}} - drop universal mac support for CI in buildbot, clean up - r=mshal ([https://hg.mozilla.org/build/buildbot-configs/rev/b9c91fd3235e b9c91fd3235e])
| bhearsum
|-
|-
| in production
| in production
| 20100118 1000 PST
| 2017-03-06 11:49 PT
| {{bug|493237}}
|
| Enable crash stack display on leaktest builds
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm*
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1344768&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| low
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1344768}} - drop universal mac support for beta releases in buildbot - r=mshal ([https://hg.mozilla.org/build/buildbot-configs/rev/5a0e02cfe694 5a0e02cfe694])
| catlee/bhearsum
|-
|-
| in production
| in production
| 201018 1000 PST
| 2017-03-03 13:50 PT
| {{bug|519209}}
|
| "make upload" step does not fail if the server uploading to is full
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm*
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1343628&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| low
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1343628}} - Revert jamun priority. Backed out changeset 7b5c4a1b1242 ([https://hg.mozilla.org/build/buildbot-configs/rev/1477d8ed6c82 1477d8ed6c82])
| bhearsum
|-
|-
| In production
| in production
| 20100115 1225 PST
| 2017-03-03 12:53 PT
| {{bug|540000}}
|
| Firefox 3.6rc2 configs
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1339179,1343935&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| high
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1339179}} - enable talos jobs on pine - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/9125de07c8e0 9125de07c8e0])
| catlee
* {{bug|1343935}} - Buildbot specific changes to enable TC based linux release promotion - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/8e841643343f 8e841643343f])
* No bug - Update release config for Fennec-52.0-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/ca5dfc82938a ca5dfc82938a])
* No bug - Update release config for Fennec-52.0b11-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/76a2975698c2 76a2975698c2])
* No bug - Update release config for Fennec-52.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/43f2397f9ab7 43f2397f9ab7])
* No bug - Update release config for Fennec-52.0-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/dfcd3b0be55a dfcd3b0be55a])
|-
|-
| In production
| in production
| 20100115 1225 PST
| 2017-03-01 11:50 PT
| {{bug|507540}}
|
| Stop running logrefcnt builds on osx on mozilla-central and tracemonkey
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm,pm02
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1343628&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| medium
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1343628}} - Temporarily bump jamun priority - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/7b5c4a1b1242 7b5c4a1b1242])
| catlee
|-
|-
| Backed out
| in production
| 20100115 1232 PST
| 2017-02-27 19:34 PT
| {{bug|538699}}
|
| Fix maemo deb repacks after version change
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm02
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1343130&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| high
</div>
| landed
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
| reconfig
* {{bug|1343130}} - Normalize builder name - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/4272ee7d83bd 4272ee7d83bd])
| aki
|-
|-
| In production
| in production
| 20100114 0800 PST
| 2017-02-27 18:50 PT
| {{bug|452861}}
|
| Split up everythingelse
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm,pm02
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1343130&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| medium
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* No bug - Update release config for Fennec-52.0b10-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/edf97f3ab377 edf97f3ab377])
| catlee
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1343130}} - Enable partner repacks builder on mozilla-esr52 - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/9120977013a7 9120977013a7])
|-
|-
| In production
| in production
| 20100113 0730 PST
| 2017-02-27 13:40 PT
| {{bug|524176}}
|
| Turn off refcounting builds on tracemonkey
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm02
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1339151&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| medium
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1339151}} - Remove leadind "index." from routes - r=aki ([https://hg.mozilla.org/build/buildbot-configs/rev/931d94078878 931d94078878])
| catlee
|-
|-
| In production
| in production
| 20100112 1639 PST
| 2017-02-27 11:56 PT
| {{bug|528642}}
|
| maemo talos -- update for xres, get rid of extraneous trailing whitespace
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| talos pool
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1339151&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| medium
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| no
* {{bug|1339151}} - Adjust jamun en-US CI build routes - r=aki ([https://hg.mozilla.org/build/buildbot-configs/rev/de82546851ec de82546851ec])
| alice/aki
|-
|-
| In production
| in production
| 20100112 0730 PST
| 2017-02-24 13:02 PT
| {{bug|536764}}
|
| Clamp log sizes on try server
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| try
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1338232&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| medium
</div>
| landed
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
| reconfig
* {{bug|1338232}} - Use sha512 only for old release automation - r=aki ([https://hg.mozilla.org/build/buildbotcustom/rev/8598bfd05d49 8598bfd05d49])
| catlee
|-
|-
| In production
| in production
| 20100111 0930 PST
| 2017-02-23 06:27 PT
| {{bug|534684}}
|
| add a tryserver-staging to buildbot-configs  
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm*
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1339832&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| low
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1339832}} - Buildbot specific changes to enable mozilla-esr52 - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/c0795c1164ba c0795c1164ba])
| lsblakk
|-
|-
| In production
| in production
| 20100111 0930 PST
| 2017-02-17 09:50 PT
| {{bug|535896}}
|
| move release automation out of master-main.cfg
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm*
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1322402,1339151&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| low
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1322402}} - Use config from macosx64 instead macosx-universal - r=rail ([https://hg.mozilla.org/build/buildbot-configs/rev/a7218494e505 a7218494e505])
| lsblakk
* {{bug|1339151}} - Tweak jamun configs - r=aki ([https://hg.mozilla.org/build/buildbot-configs/rev/aaf7f4db57ff aaf7f4db57ff])
|-
|-
| In production
| in production
| 20100111 0930 PST
| 2017-02-16 13:27 PT
| {{bug|520903}}
|
| turn off nightly builds with flag in config.py
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm*
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1338610,1339151&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| low
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1338610}} - Split up mochitest-other into separate mochitest-a11y and mochitest-chrome jobs - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/4f814a83ba4f 4f814a83ba4f])
| lsblakk
* {{bug|1339151}} - buildbot configs to use different tasks for signed and unsigned bits in release promotion - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/fb545d14c97f fb545d14c97f])
|-
|-
| In production
| in production
| 20100111 0830 PST
| 2017-02-15 12:50 PT
| {{bug|534626}}
|
| fail when Win32 Desktop fails to package
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm02
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| med
</div>
| needs landing
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| armenzg
* No bug - backouts Bug No Bug - rev: c9b1f1f58974 ([https://hg.mozilla.org/build/buildbot-configs/rev/fbcd05687641 fbcd05687641])
* No bug - reverts 51.0.4 release fennec patch ([https://hg.mozilla.org/build/buildbot-configs/rev/7d7513aadda6 7d7513aadda6])
|-
|-
| in production
| in production
| 20100106 1010 PST
| 2017-02-08 13:10 PT
| {{bug|528287}}
|
| new darwin9 slaves
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm01,pm02
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1334538&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| medium
</div>
| landed
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1334538}} - Separate eme_free_repacks_platforms and sha1_repacks_platforms configs - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/3cb47357583d 3cb47357583d])
| bhearsum
|-
|-
| In production
| in production
| 20100105 1820 PST (started)
| 2017-02-08 12:19 PT
| {{bug|535567}}
|
| release config for fx3.6rc1 build1
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm01
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1163465,1336602,1337824&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| high
</div>
| [http://hg.mozilla.org/build/buildbot-configs/rev/8ab319b9c004 landed]
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1337824}} - Prepare jamun for 53 fennec/firefox reelases - r=Callek ([https://hg.mozilla.org/build/buildbot-configs/rev/38c9973328c8 38c9973328c8])
| nthomas
* {{bug|1337824}} - Prepare jamun for 53 fennec/firefox reelases - r=kmoir ([https://hg.mozilla.org/build/buildbot-configs/rev/bf3b180bb991 bf3b180bb991])
* No bug - Backed out changeset 38c9973328c8 ([https://hg.mozilla.org/build/buildbot-configs/rev/3aad44858b82 3aad44858b82])
[https://hg.mozilla.org/build/buildbotcustom buildbotcustom]
* {{bug|1163465}} - Remove HACK_first_released_version - r=jlund ([https://hg.mozilla.org/build/buildbotcustom/rev/fc664d42c36f fc664d42c36f])
* {{bug|1336602}} - Migrate http->https for all references to hg (or anything else that can be updated) - r=kmoir ([https://hg.mozilla.org/build/buildbotcustom/rev/a22012e2ee94 a22012e2ee94])
|-
|-
| In production
| in production
| 20100105 1200 PST
| 2017-01-23 03:51 PT
| [https://bugzilla.mozilla.org/attachment.cgi?id=420119&action=edit patch]
|
| only use mozconfig for WinCE and fix regression
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm*
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1305492&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| high
</div>
| [http://hg.mozilla.org/build/buildbotcustom/rev/dfa7408e2bd9 landed]
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1305492}} - bumps integration repos for migration - r=mtabara ([https://hg.mozilla.org/build/buildbot-configs/rev/01a37fa3cac7 01a37fa3cac7])
| armenzg
* No bug - Update release config for Fennec-51.0-build2 ([https://hg.mozilla.org/build/buildbot-configs/rev/4b674ed969e7 4b674ed969e7])
* No bug - Update release config for Fennec-51.0-build3 ([https://hg.mozilla.org/build/buildbot-configs/rev/6ca3cd6cefdb 6ca3cd6cefdb])
|-
|-
| In production
| in production
| 20100104 0810 PST
| 2017-01-05 04:33 PT
| {{bug|534626}}
|
| generate WinCE l10n nightly repackages
<div style="border: thin grey solid; background-color: lightgrey; float: right; text-align: right; text-size: 80%; padding-left: 5px; padding-right: 5px;">
| pm01
[https://bugzilla.mozilla.org/buglist.cgi?bug_id=1325726&query_format=advanced&order=bug_status%2Cbug_id&tweak=1 View list in Bugzilla]
| high
</div>
| [http://hg.mozilla.org/build/buildbotcustom/rev/9125d3fa086d][http://hg.mozilla.org/build/buildbot-configs/rev/596f22b540b4]
[https://hg.mozilla.org/build/buildbot-configs buildbot-configs]
| reconfig
* {{bug|1325726}} - Remove the old win32-st-an-debug buildbot job - r=jlund ([https://hg.mozilla.org/build/buildbot-configs/rev/c776af2d9fa0 c776af2d9fa0])
| armenzg
* No bug - Update release config for Fennec-51.0b11-build1 ([https://hg.mozilla.org/build/buildbot-configs/rev/6bedaca59794 6bedaca59794])
|}
|}
[[ReleaseEngineering:BuildbotMasterChanges:Archive | Older Changes]]

Latest revision as of 14:51, 21 September 2017

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our master buildbots.

Relevant repositories

Non-releng repositories involved in the CI

Treeherder

  • Chief self-deploy recent history (needs VPN; more readable than raw logs, but is cleared if the Chief service is restarted)
  • [treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/logs/ Chief self-deploy logs archive] (needs VPN; use if the history page is empty)

Mozharness deployments

Merges to the production branch of the mozharness repository are live immediately. Please see that repo for push history.

Puppet deployments

Merges to the production branch of the puppet repository are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Please note if you are applying a puppet change, unless you have good reason not to do so please merge it to production branch immediately. The reason for this is that we do not want puppet changes to "back up" on default, and then land en masse in production. It can be very difficult to untangle multiple puppet changes that arrive together.

The reason we have a production branch at all, is to allow contributions on default branch from a larger group of people, than those responsible for maintaining the infrastructure controlled by the puppet repo - in other words, a smaller group of people can merge to production, than the group of people that can land on default.

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as mozharness changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: 2016 | 2015 | 2014 | 2013 | 2012 | 2011 | 2010 | 2009

Outcome When Bug #(s) - Description(s)
in production 2017-09-21 07:51 PT

buildbot-configs

buildbotcustom

  • bug 1333792 - Try push confirmation emails for Taskcluster jobs should link to the Task inspector rather than the upload directory on archive.m.o - r=mtabara (994ad65b2abd)
in production 2017-09-14 03:55 PT

buildbot-configs

buildbotcustom

in production 2017-09-06 10:14 PT

buildbot-configs

buildbotcustom

in production 2017-08-31 06:50 PT

buildbotcustom

in production 2017-08-06 22:46 PT

buildbot-configs

in production 2017-08-04 03:42 PT

buildbot-configs

  • No bug - Update release config for Thunderbird-56.0b1-build1 (663ad30ff27d)
  • No bug - Relpro mac and win builds index changes required for 56.0b1 - r=mtabara (3c7a94990487)
in production 2017-08-02 00:27 PT

buildbot-configs

buildbotcustom

in production 2017-07-31 04:12 PT

buildbot-configs

  • bug 1369535 - m-r gecko version bump bbconfigs - r=jlorenzo (acbb8cf0729e)
  • No bug - No Bug - Staging release: Point to a disposable balrog instance - r=rail (c700658593c4)
  • No bug - No Bug - Staging release: Point to rail's disposable repo - r=jlorenzo (749272eda177)
in production 2017-06-12 03:31 PT

buildbot-configs

in production 2017-06-05 06:32 PT

buildbot-configs

in production 2017-06-05 02:12 PT

buildbot-configs

in production 2017-05-11 06:12 PT

buildbot-configs

in production 2017-03-16 07:25 PT

buildbot-configs

in production 2017-03-10 15:02 PT

buildbot-configs

in production 2017-03-10 10:11 PT

buildbot-configs

in production 2017-03-09 13:18 PT

buildbot-configs

in production 2017-03-06 11:57 PT

buildbot-configs

in production 2017-03-06 11:49 PT

buildbot-configs

in production 2017-03-03 13:50 PT

buildbot-configs

in production 2017-03-03 12:53 PT

buildbot-configs

in production 2017-03-01 11:50 PT

buildbot-configs

in production 2017-02-27 19:34 PT

buildbotcustom

in production 2017-02-27 18:50 PT

buildbot-configs

  • No bug - Update release config for Fennec-52.0b10-build1 (edf97f3ab377)

buildbotcustom

in production 2017-02-27 13:40 PT

buildbot-configs

in production 2017-02-27 11:56 PT

buildbot-configs

in production 2017-02-24 13:02 PT

buildbotcustom

in production 2017-02-23 06:27 PT

buildbot-configs

in production 2017-02-17 09:50 PT

buildbot-configs

in production 2017-02-16 13:27 PT

buildbot-configs

  • bug 1338610 - Split up mochitest-other into separate mochitest-a11y and mochitest-chrome jobs - r=jlund (4f814a83ba4f)
  • bug 1339151 - buildbot configs to use different tasks for signed and unsigned bits in release promotion - r=jlund (fb545d14c97f)
in production 2017-02-15 12:50 PT

buildbot-configs

  • No bug - backouts Bug No Bug - rev: c9b1f1f58974 (fbcd05687641)
  • No bug - reverts 51.0.4 release fennec patch (7d7513aadda6)
in production 2017-02-08 13:10 PT

buildbot-configs

in production 2017-02-08 12:19 PT

buildbot-configs

buildbotcustom

in production 2017-01-23 03:51 PT

buildbot-configs

in production 2017-01-05 04:33 PT

buildbot-configs

Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/Maintenance&oldid=1180895"