ReleaseEngineering/Maintenance

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our master buildbots.

Relevant repositories

• buildbot
• buildbot-configs
• buildbotcustom
• tools
• mozharness
• puppet

Non-releng repositories involved in the CI

Treeherder

• Chief self-deploy recent history (needs VPN; more readable than raw logs, but is cleared if the Chief service is restarted)
• [treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/logs/ Chief self-deploy logs archive] (needs VPN; use if the history page is empty)

Mozharness deployments

Merges to the production branch of the mozharness repository are live immediately. Please see that repo for push history.

Puppet deployments

Merges to the production branch of the puppet repository are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Please note if you are applying a puppet change, unless you have good reason not to do so please merge it to production branch immediately. The reason for this is that we do not want puppet changes to "back up" on default, and then land en masse in production. It can be very difficult to untangle multiple puppet changes that arrive together.

The reason we have a production branch at all, is to allow contributions on default branch from a larger group of people, than those responsible for maintaining the infrastructure controlled by the puppet repo - in other words, a smaller group of people can merge to production, than the group of people that can land on default.

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as mozharness changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: 2015 | 2014 | 2013 | 2012 | 2011 | 2010 | 2009

Outcome	When	Bug #(s) - Description(s)
in production	2016-03-16 10:35 PT	View list in Bugzilla buildbot-configs bug 1252292 - Don't run Ru-e10s on debug builds - r=me (04c8fe62d3a4) bug 1253341 - Enable talos on linux64 AWS machines on try - r=kmoir (2d6158d681e0) bug 1257230 - Incorrect branding for 45.0.1esr build - r=jlund (b5163b4a7814) No bug - Update release config for Fennec-46.0b2-build1 (216ac6505867) No bug - Update release config for Firefox-38.7.1esr-build1 (2b0b8272dda2) No bug - backouts release change for 38.7.1esr - rev: 2b0b8272dda2 (ea85259cf436) No bug - merges backout 2b0b8272dda2 -> default (7b0276c224b4) No bug - Update release config for Firefox-38.7.1esr-build1 (b735b52c0d5a) No bug - Update release config for Firefox-45.0.1-build1 (b3267390eb7d) No bug - Update release config for Firefox-45.0.1esr-build1 (1f7801999788) No bug - Update release config for Fennec-45.0.1-build1 (e759c66d311c) No bug - Backed out 2d6158d681e0 for suspicion of causing intermittent checkconfig (7c463958098d) buildbotcustom bug 1253341 - Allow running talos on AWS machines - r=kmoir (42184dbdf8f3) No bug - Backed out changeset 42184dbdf8f3 for suspicion of causing intermittent checkconfig failureis (046c46eb099d)
in production	2016-03-15 07:43 PT	View list in Bugzilla buildbot-configs bug 1249528 - releaserunner should handle RC channels - r=jlund (5235b4ceb15e) bug 1249528 - Bustage fix - r=me (806fd2074e58) bug 1249528 - backout - r=me (fc1e57afc013) bug 1252292 - Set Ash to the same branch priority as mozilla-central - r=rail (50c1fdd4ff3e) bug 1255812 - Move 10 XP test machines to the Windows 8 pool - r=kmoir (36eb260ec81c) bug 1256184 - Remove code that unconditionally enables Talos g2 irrespective of the branch's enable_talos setting - r=jmaher (ec3937ad8bd7)
in production	2016-03-11 09:50 PT	View list in Bugzilla buildbot-configs bug 1242682 - Separate dom/media into its own subsuite - r=kmoir (cc9e268a0036) bug 1245585 - turn off android talos tests which run on the panda boards - r=aselagea (771999c6f4ef) bug 1250999 - releng - remove api-9-10 from trunk - r=kmoir (1bbe091937b4) No bug - Update release config for Thunderbird-38.7.0-build1 (cf9b859c9d8c) buildbotcustom bug 1186617 - releng changes to decommission pandas - r=aselagea (695299094ba0) bug 1255078 - Android Beta tests broken: Found installer url ~/bouncer.apk - r=catlee (b4c6ed50c14c)
in production	2016-03-10 06:17 PT	View list in Bugzilla buildbot-configs bug 1245585 - turn off android talos tests which run on the panda boards - r=aselagea (c58e5674bae2) bug 1245585 - turn off android talos tests which run on the panda boards - r=aselagea (87168a6da9f1) bug 1253858 - Turn off mochitest-push on esr45 - r=rail (95e902ea4772) bug 1255088 - turn on talos e10s for mozilla-beta, remove some other misconfigurations - r=catlee (8a6d6e31623f)
in production	2016-03-09 06:14 PT	View list in Bugzilla buildbot-configs bug 1166420 - Add relengapi.tok to TB bb configs - r=rail (d7776ec49bb4) bug 1242682 - Separate dom/media into its own subsuite - r=armenzg (dd518a784a46) bug 1251351 - Add config file options for marionette-e10s on Windows - r=catlee (1ce5d30d8e7c) bug 1251477 - rename update verify configs - r=bhearsum (aed2ba7655a0) bug 1254902 - Add relengapi.tok to buildbot configs - r=rail (ddf4c824ca46)
in production	2016-03-07 19:31 PT	View list in Bugzilla buildbot-configs bug 1254337 - fix tuple typo for beta releasetasks items, bbotcfgs, r?rail (b9a2659ad8fe)
in production	2016-03-07 16:12 PT	View list in Bugzilla buildbot-configs bug 1251351 - Schedule Mn-e10s tests on win7 - r=catlee (8a111c694d59) bug 1251936 - Part 3: Add buildbot configs for Windows x86 static analysis buillds - r=jlund (ae056e1b3ad7) bug 1254337 - use release-signing on mozilla-beta and disable push2releases automatic, bbotcfgs, r?rail (75be1891c7f8) No bug - Update release config for Fennec-46.0b1-build1 (236b949b7359)
in production	2016-03-07 05:36 PT	View list in Bugzilla buildbot-configs bug 1243796 - Tracking bug for Mar-7-2016 migration work - r=rail (f03990b8f382)
in production	2016-03-04 13:23 PT	View list in Bugzilla buildbot-configs bug 1253492 - Add missing e10s defines to un-break Mn-e10s and M(gl) jobs - r=rail (7b8c9c7698ed) bug 1253508 - Let e10s jobs ride the trains - r=catlee (1e47404949d2) bug 1253710 - Enable OSX 10.10 opt e10s tests on Gecko 47 - r=rail (15dc4b9def85) bug 1253714 - Enable OSX 10.10 debug e10s browser-chrome and Marionette tests on Gecko 47 - r=rail (e5cc6f88dcf5)
in production	2016-03-04 12:26 PT	View list in Bugzilla buildbot-configs bug 1251477 - prep buildbot configs for release promotion in production on beta - r=rail (57c166d8109b)
in production	2016-03-04 09:03 PT	View list in Bugzilla buildbot-configs bug 1242682 - Separate dom/media into its own subsuite [fix e10s config-file] - r=kmoir (f0acc9241544) bug 1253410 - Enable ASAN builds/tests on ESR45 - r=rail (e254fe53e290) bug 1253437 - Split Linux debug/ASAN xpcshell into 4 chunks - r=rail (9a94815dc367) bug 1253492 - Enable more e10s test suites on Linux - r=rail (8fb9139950ba) bug 1253492 - Fix up duplicate builder bustage - r=rail (6535597727b5)
in production	2016-03-04 07:06 PT	View list in Bugzilla buildbot-configs bug 1237094 - tracking bug for build and release of Firefox 45.0 ESR - r=bhearsum (f65f8e566a68) bug 1242682 - Separate dom/media into its own subsuite - r=kmoir (ad171d57880d)
in production	2016-03-03 16:54 PT	View list in Bugzilla buildbot-configs bug 1192994 - Handle Linux64 data from SETA - r=kmoir (8382286ea52c) bug 1252248 - Add more tst-linux64, tst-emulator64 capacity in AWS - r=kmoir (bc958535ce5a) bug 1252809 - Only test pretty names on comm train branches, not comm-central - r=jlund (18c0c4c99064) bug 1253364 - Clean up list of machines that aren't in slavealloc - r=kmoir (b86333d2e140) bug 1253429 - production-ify release promotion configs, use existing bucket - r=rail (858343665d94) No bug - Update release config for Firefox-38.7.0esr-build1 (39239fcf0362) No bug - Update release config for Fennec-45.0-build2 (20518d5e75df) No bug - Update release config for Fennec-45.0-build3 (92ab47cbc53a) No bug - Update release config for Firefox-45.0-build2 (7f197028d1d9) No bug - Update release config for Firefox-45.0esr-build1 (08355f5f053b) No bug - Backed out 3 changesets on default to restart releases (436d072f5952) No bug - Update release config for Fennec-45.0-build3 (fc5ddedd1c1d) No bug - Update release config for Firefox-45.0-build2 (b51aad4f9e4b) No bug - Update release config for Firefox-45.0esr-build1 (13e413a55b5b)
in production	2016-03-02 13:51 PT	View list in Bugzilla buildbot-configs bug 1237094 - tracking bug for build and release of Firefox 45.0 ESR - r=nthomas (91b85a6e4c1e) bug 1251381 - Switch gtests to riding 46 - r=kmoir (2a06d32c80b9) bug 1252756 - Shut off periodic PGO on b2g-inbound - r=catlee (6c131bed7fae) No bug - No bug - shorten travis results to be less spammy - r=catlee (fb1af69edbec) No bug - Update release config for Fennec-45.0-build1 (38e4592e3d57) buildbotcustom No bug - No Bug - shorten travis results notification to be less spammy - r=catlee (4b96f5cb0d0f)
in production	2016-03-02 06:25 PT	View list in Bugzilla buildbot-configs bug 1237094 - tracking bug for Firefox 45.0 ESR, channel rules - r=rail (4b855f6b6eae) bug 1252258 - Move 8 w10 test machines to the Windows 7 pool - r=kmoir (25b7652be68b) bug 1252328 - Only run e10s tests on Gecko 46+ - r= (e9147a125ebc)
in production	2016-03-01 07:30 PT	View list in Bugzilla buildbot-configs bug 1242682 - Separate dom/media into its own subsuite - r=kmoir (bf7f47ae7e0f) bug 1249376 - Move 20 XP test machines to the Windows 7 pool - r=kmoir (dee25cbffdcd) No bug - Update release config for Fennec-45.0b11-build1 (b42fec100a37) No bug - Update release config for Firefox-45.0-build1 (e172a9923d43)
in production	2016-02-29 08:10 PT	View list in Bugzilla buildbotcustom bug 1249682 - Remove addon-sdk buildbot configs - r=Callek (01d5bda06fc9)
in production	2016-02-25 09:50 PT	View list in Bugzilla buildbot-configs bug 1237094 - tracking bug for build and release of Firefox 45.0 ESR - r=bhearsum (8712d19cff79) bug 1251144 - Don't merge release promotion builders - r=catlee (1668779e0c88) bug 1251285 - Disable alder project branch (81db79611afa) bug 1251285 - Disable alder project branch - r=bhearsum (0441563764d3) buildbotcustom bug 1251144 - Don't merge release promotion builders - r=catlee (0cbca385a7e2)
in production	2016-02-24 13:45 PT	View list in Bugzilla buildbot-configs bug 1250585 - adds push to mirrors automatic date, buildbot-configs - r=rail (8ff751a38618) No bug - Update release config for Fennec-45.0b9-build2 (fc63a5db2c0c) No bug - Update release config for Firefox-45.0b9-build2 (884497cab040)
in production	2016-02-23 13:53 PT	View list in Bugzilla buildbot-configs bug 1245559 - Enable wpt e10s on Windows 7 - r=kmoir (17cc02d08629) bug 1245559 - Enable wpt e10s on Windows 7 - r=kmoir (646145351982) bug 1250306 - Point date tools repo at rail's fork - r=rail (9bae3c5f95f2) No bug - Backed out changeset 17cc02d08629 (a06e78e32bcb) No bug - [mq]: tb.cypress.tests (4218b4cabec6)
in production	2016-02-22 13:17 PT	View list in Bugzilla buildbot-configs No bug - Update release config for Firefox-45.0b8-build1 (5f9a37958eaf) No bug - Update release config for Firefox-45.0b8-build1 (5f9a37958eaf) buildbotcustom bug 1248474 - Don't use a slash in buildername - r=rail (c87b90ed718d) bug 1248474 - Don't use a slash in buildername - r=rail (c87b90ed718d)
in production	2016-02-19 13:00 PT	View list in Bugzilla buildbotcustom bug 1248474 - [RelPromo] Run Mac/Windows update verification via BBB - r=rail (6035a0b57bfb)
in production	2016-02-19 07:03 PT	View list in Bugzilla buildbot-configs bug 1241955 - Enable tests on date - r=callek (cc561d788755) bug 1248474 - [RelPromo] Run Mac/Windows update verification via BBB - r=rail (437d703fad2b) bug 1248729 - Schedule e10s mochitest-push on Windows 7 patch by aselagea - r=jgriffin (2999a899e438) bug 1249383 - disable talos on b2g-inbound - r=Callek (279753fc211c) No bug - Update release config for Firefox-45.0b7-build1 (87671b77f024) No bug - Update release config for Thunderbird-45.0b2-build1 (2706c3b66654) buildbotcustom bug 1246538 - mochitest-o should trigger mochitest-a11y and mochitest-chrome-* patch by aselagea - r=kmoir (926a0351a0d7) bug 1248474 - [RelPromo] Run Mac/Windows update verification via BBB - r=rail (eb8290943fcf)
in production	2016-02-17 08:46 PT	View list in Bugzilla buildbot-configs bug 1246627 - Schedule e10s jsreftests on win7 patch by aselagea - r=jgriffin (cc37bdb15492) No bug - revert Bug 1246627 - Schedule e10s jsreftests on win7 (71b9ccb8abf1) No bug - Update release config for Fennec-45.0b6-build1 (69f84fb1f35b) No bug - Update release config for Firefox-45.0b6-build1 (df4ed20df968) buildbotcustom bug 1236954 - add standalone partner repack factory - r=jlund (a7bd1793a7ab) bug 1247078 - [tryparser] Don't schedule any jobs if no try syntax is detected - r=catlee (46fb74d54d79)
in production	2016-02-14 08:43 PT	View list in Bugzilla buildbot-configs bug 1245605 - Use 'dumb' source mozconfig for TB beta builds following bug 123448 - r=jlund (22f1e00d700c) bug 787208 - Followup, don't set mozilla_dir - r=no-reviewer (8c7677a7cdb6) buildbotcustom bug 1244510 - Add a try_parser alias for mochitest-e10s-(bc|browser-chrome) patch by aselagea - r=kmoir (c994ec64ba03)
in production	2016-02-12 07:43 PT	View list in Bugzilla buildbot-configs bug 1218589 - Nightly mozilla-central builds of graphene patch by aselagea - r=kmoir (f88d45c3fbe6) bug 1229532 - remove hg bundle builder config - r=bhearsum (f0c9c5b5a7b8) bug 1246627 - Schedule e10s jsreftests on win7 patch by aselagea - r=kmoir (66bc120b896c) bug 787208 - In support of merging comm-central and mozilla-central together, configure cypress as a Thunderbird branch. r=rail - r=Fallen f=jcranmer (b92aaf935173) No bug - Update release config for Firefox-38.6.1esr-build1 (33fe33d65478) No bug - Update release config for Firefox-44.0.2-build3 (09cd8415adb9) No bug - Update release config for Thunderbird-38.6.0-build1 (91cf6aa50268) No bug - Update release config for Firefox-45.0b5-build1 (6376fd9567c4) buildbotcustom bug 1229532 - remove hg bundle builders - r=bhearsum (1b880a68c991)
in production	2016-02-02 14:53 PT	View list in Bugzilla buildbot-configs bug 1242103 - Add browser-chrome-screenshots to m-c and other try - r=armenzg (63614b6564ee) No bug - Update release config for Fennec-45.0b2-build1 (7a121e3abe60) No bug - Update release config for Firefox-45.0b2-build1 (a70a9918a489) No bug - Update release config for Thunderbird-45.0b1-build1 (39642e4c3ba4)
in production	2016-01-26 20:05 PT	View list in Bugzilla buildbotcustom bug 1190860 - fix valgrind by setting up mock before running tooltool, r=jlund (28864afc6409) buildbot-configs bug 1190860 - Copy in tooltool.py, and run it from in-mock location (with TB too), r=jlund (10af9c3220aa)
in production	2016-01-26 18:05 PT	View list in Bugzilla buildbotcustom bug 1190860 - followup fix r=Callek over irc (a4bc25f76f2d) bug 1166464 - followup to fix reconfig, r=jlund (de981e55b123)
in production	2016-01-26 16:00 PT	View list in Bugzilla buildbotcustom bug 1242374 - set an sdkUrl to avoid interfering with packageUrl; r=bhearsum (34f988f27f10) bug 1166464 - Bump version after shipping release builds r=jlund (97febf14fe8d) bug 1242641 - backout 3b001e0b8438 (360f6f0cf623) bug 1242641 - backout 31ff0be74400 (9b9b645d00a1) bug 1190860 - Run tooltool inside mock, r=Callek (889eb503d966) buildbot-configs bug 1211889 - split mochitest-oth into- mochitest-a11y (a), mochitest-chrome (c1, c2, c3) for linux32 debug. r=armenzg (d9e027f12297) bug 1166464 - Bump version after shipping release builds r=jlund (bb2664d7bf15)
in production	2016-01-25 20:47 PT	View list in Bugzilla buildbotcustom bug 1242641 - followup to fix cwd for mock call, and use python from the virtualenv (3b001e0b8438)
in production	2016-01-25 14:54 PT	View list in Bugzilla buildbotcustom bug 1242641 - fix printconfigsetting.py calls for Firefox on m-b and higher, r=Callek (31ff0be74400)
in production	2016-01-25 05:50 PT	View list in Bugzilla buildbot-configs bug 1232329 - Tracking bug for Jan-25-2016 migration work - r=trivial (c38c7f7b76e9) No bug - Update release config for Firefox-44.0-build3 (3941c0a6daf6)
in production	2016-01-21 12:12 PT	View list in Bugzilla buildbot-configs bug 1235961 - set tools_repo_cache for mac platforms - r=bustage (fa16bb751b50)
in production	2016-01-21 10:56 PT	View list in Bugzilla buildbot-configs bug 1231408 - Add browser-chrome-screenshots to yosemite_r7 on Try - r=armenzg (b74388666f8e) bug 1233005 - Enable packageSDK in release configs - r=bhearsum (9f0795fa7184) No bug - Update release config for Fennec-44.0-build2 (2b71401d73e8) No bug - Update release config for Firefox-44.0-build2 (0154167166dd) No bug - Update release config for Firefox-38.6.0esr-build1 (fa10d55edf12) buildbotcustom bug 1233005 - Set packageSDK if enabled in the config - r=bhearsum (5d15ab1c5ddd) bug 1236951 - set tools_repo_cache for partner repacks - r=bustage (c06e740a09e4) bug 1241159 - handle mochites-e10s-devtools-chrome as an alias for the e10s devtools chunks - r=Callek (25315f9d0e2e) bug 1241481 - Get rid of script_revlink - r=rail (dc9b1cf7b9e1)
in production	2016-01-20 06:16 PT	View list in Bugzilla buildbot-configs No bug - Update release config for Fennec-44.0b11-build1 (370703910927) buildbotcustom bug 1236951 - use tools_repo_cache for toolsdir when set - r=kmoir (6368247d8c7a) bug 1236951 - use WithProperties in case we actually have a revision - r=bustage (b049ab7ef9a7)
in production	2016-01-19 07:36 PT	View list in Bugzilla buildbot-configs bug 1225899 - add beetmover candidates flag to date config - r=rail (fca43032761e) bug 1236951 - Update partner repack stanza in release templates - r=kmoir (2f6ad8dd5b7e) bug 1239785 - Move some XP test machines to the Windows 7 pool - r=kmoir (31b0f9f1195c) No bug - Update release config for Fennec-44.0b10-build1 (cfff00d1a9f5) No bug - Update release config for Firefox-44.0-build1 (8a87f5fe2b16) buildbotcustom bug 1236835 - Disable all B2G Desktop Builds & Tests on All trees. Cleanup - r=rail (e0ab5efe7082)
in production	2016-01-18 06:19 PT	View list in Bugzilla buildbot-configs bug 1232329 - Tracking bug for Jan-25-2016 migration work - r=mtabara (a0007d5cd72b) bug 1239025 - Turn on reftest for e10s on win7 - r=jlund (c257ac5cf785) mozharness bug 1151954 - move get_apk.sh to a mozharness file - r=sylvestre (15845edae9a1)
in production	2016-01-15 08:55 PT	View list in Bugzilla buildbot-configs No bug - Update release config for Fennec-44.0b9-build1 (15dc357fc5bd) No bug - Update release config for Firefox-44.0b9-build1 (df3ee52930d6) buildbotcustom bug 1236951 - Set script_repo_revision for ScriptFactory when archiver is used - r=kmoir (aa885f5cdbc3)
in production	2016-01-13 15:03 PT	View list in Bugzilla buildbot-configs bug 1226180 - decommission t-yosemite-r5 machines - r=kmoir (b05f8de5ec9d) bug 1236835 - Disable all B2G Desktop Builds & Tests on All trees - r=rail (66b14e07db81) bug 1236951 - Re-enable partner repacks on beta and release - r=kmoir (5bdbf1ac3f5f) bug 1239074 - Add trailing slash to ${MOZILLABUILD} so that PATH gets properly created - r=catlee (388b403428e2) buildbotcustom bug 1236951 - Replace PartnerRepackFactory with a SigningScriptFactory instance - r=kmoir (089b94bab05c)
in production	2016-01-12 09:04 PT	View list in Bugzilla buildbot-configs bug 1238947 - Disable b2g37_v2.2 automation - r=rail (4989f1200474) No bug - Update release config for Fennec-44.0b8-build1 (e8d645fd93cd) No bug - Update release config for Firefox-44.0b8-build1 (6196d2a86352) No bug - Update release config for Thunderbird-44.0b1-build1 (6580fa6c368b) buildbotcustom bug 1238947 - Disable b2g37_v2.2 automation - r=rail (2135d176b8e8) mozharness bug 1238947 - Disable b2g37_v2.2 automation - r=rail (bcdac5ca4d07)
in production	2016-01-11 08:10 PT	View list in Bugzilla buildbot-configs bug 1238083 - Clean up stage tasks - r=bhearsum (0d695ad18d3f) buildbotcustom bug 1238083 - Clean up stage tasks - r=bhearsum (65c073287d56)
in production	2016-01-11 07:41 PT	View list in Bugzilla buildbot-configs bug 1073442 - adjust chunks from 5 to 8 for debug mochitests - r=kmoir (5ae05a86c3ab) bug 1237985 - Disable b2g37_v2.2R automation - r=rail (00c4b51cb1fb) buildbotcustom bug 1237985 - Disable b2g37_v2.2R automation - r=rail (7eb568e273d8) mozharness bug 1237985 - Disable b2g37_v2.2R automation - r=rail (9effeabb1c18) bug 1237985 - Disable b2g37_v2.2R automation - r=rail (8da7cb911d70) No bug - Revert Bug 1237985 - Disable b2g37_v2.2R automation - r=rail (cdc0ebfcef83)
in production	2016-01-07 06:39 PT	View list in Bugzilla buildbot-configs bug 1211770 - Disable HSTS/HPKP updates on b2g 2.2 branches - r=kmoir (65d3648d9b27) bug 1221963 - Schedule media-youtube-tests on trunk branches - r=jlund (f7d9ee4eab35) bug 1229790 - add ec2 windows slaves to configs, restricted to try - r=kmoir (4a61c10820eb) bug 1232780 - Schedule e10s mochitest-plain on win7-opt - r=jgriffin (ea429699060a) No bug - Update release config for Firefox-43.0.4-build2 (1bc16691f520) No bug - Update release config for Firefox-43.0.4-build3 (4b67594abc24) buildbotcustom bug 1215204 - Add bouncer submitter to release promotion graph - r=rail (93196910e3df) bug 1215204 - Add bouncer submitter to release promotion graph - r=rail (4cda855c1404) No bug - revert Bug 1215204 - Add bouncer submitter to release promotion graph - r=rail (8dda9c1cce64)