ReleaseEngineering/Buildduty/StandupMeetingNotesQ12015

From MozillaWiki
Jump to: navigation, search

2015-03-31


2015-03-30

  • buildduty report cleanup <- lots


2015-03-27


2015-03-26

  • https://bugzil.la/1147853 - Widespread "InternalError: Starting video failed" failures across all trees on AWS-based test instances
  • Q1 is almost done
    • what do we need to document/update prior to buildduty hand-off next week?
  • testing out latest mh prod rev on cedar in a canary fashion :)
    • be better if releng stays under the radar for at least the rest of the day
  • jlund|buildduty> nagios-releng: downtime vcssync2.srv.releng.usw2.mozilla.com 1h "bug 1135266"
  • disabling two foopies for recovery


2015-03-25


2015-03-24


2015-03-23


2015-03-20

  • chemspill in progress, ***NO UNNECESSARY CHANGES***
  • coop going through "All dependencies resolved" section of buildduty report
    • doing all non-pandas first
    • will do a second, panda-only pass after


2015-03-19


2015-03-18


2015-03-17

  • https://bugzil.la/1143681 - Some AWS test slaves not being recycled as expected
    • found a way to track these down
      • in the AWS console, search for the instances in the Spot Requests tab. You can click on the instance ID to get more info.
      • e.g. for tst-linux64-spot-233, the instance has no name associated and is marked as "shutting-down"
  • https://bugzil.la/1144362 - massive spike in hg load
    • possibly due to new/rescued instances from the morning (re)cloning mh & tools
      • negative feedback loop?


2015-03-16


2015-03-13

  • buildbot DB "too many connections" again. (perhaps DBA's are able to increase the conn pool limits?)
  • need a button in slave health that automatically files a diagnostics bug for a given slave
    • should disable the slave if not already disabled
    • should do the bug linking automatically
    • should have a small text entry box for the description of the diagnostics bug, i.e. why are we asking for diagnostics
    • would hopefully prevent sheriffs from just taking slaves offline and waiting for us to perform the next step(s)
  • file https://bugzil.la/1143018 - Update runslave.py with current machine types and basedirs
    • we essentially guess at the builddir in most cases these days(!)
  • https://bugzil.la/1142825 - high windows test pending


2015-03-12

  • filed https://bugzil.la/1142493 - panda-relay-037 is down
  • Win7 test pending >2000 (unclear on why)
  • tree closure
    • caused by RyanVM


2015-03-11


2015-03-10


2015-03-09


2015-03-06


2015-03-05

  • never got to adding win64 m-a nightlies to jacuzzi https://bugzil.la/1139763
  • need to enable slaves from https://bugzil.la/1138672
  • end_to_end script tells bugs things for mozharness are live in production. this no longer is the case for all our build + test jobs (most things aside from vcs-sync, bumper, etc).
    • should we be still automatically updating bugs for mh after reconfig?
    • we need a way to roll out changes to mh on a regular cadence. right now it's up to the individual to update mozharness.json with a REV they want applied and consequently, whatever mh patches are in between are also applied...
    • coop to drop mozharness from end-to-end-reconfig script and email public list
  • added http://pypi.pvt.build.mozilla.org/pub/mozrunner-6.6.tar.gz
  • talked to catlee re: releng-try pipeline
    • fully supportive
    • one wrinkle: how to tackle release tagging
    • coop will get bugs filed today
  • add 4-repo view to slave health?


2015-03-04

  • https://bugzil.la/1138937 - Slave loan request for a t-w864-ix machine
  • reconfig in progress
  • buildduty report:
    • re-imaging a bunch of slaves to help with capacity
  • https://bugzil.la/1138672 - vlan request - move bld-lion-r5-[006-015] machines from prod build pool to try build pool (needs to be enabled)
  • test master upgrades (done)
  • (hwine) meeting with Linda (head of #moc)
    • make more specific requests from #moc
    • share top issues with #moc
    • when: next meeting is 13th
      • come up with prioritized list of releng needs by early next week
  • coop to file bugs re: releng-try improvements
    • add builderlists/dumpmasters diff to travis
    • switch RoR for key repos to github
      • reverse VCS sync flow
      • enable travis testing for forks - this is done on a per-fork basis by the owners of the forks. PR's will get travis jobs regardless.


upgrade test linux masters (https://bugzil.la/1136527):

  • bm51 (complete)
  • bm53 (complete)
  • bm117-tests1-linux64 (complete)
  • bm52-tests1-linux64 (complete)
  • bm54-tests1-linux64 (complete)
  • use1
    • bm67-tests1-linux64 (complete)
    • bm113-tests1-linux64 (complete)
    • bm114-tests1-linux64 (complete)
    • bm120-tests1-linux64 (complete)
    • bm121-tests1-linux64 (complete)
  • usw2
    • bm68-tests1-linux64 (complete)
    • bm115-tests1-linux64 (complete)
    • bm116-tests1-linux64 (complete)
    • bm118-tests1-linux64 (complete)
    • bm122-tests1-linux64 (complete)
    • bm123-tests1-linux64 (started)


add swap (https://bugzil.la/1135664):

  • bm53 (complete)
  • buildbot-master54 (complete)
  • use1
    • buildbot-master117 BAD
    • buildbot-master120 BAD (complete)
    • buildbot-master121 BAD (complete)
  • usw2
    • buildbot-master68 (complete)
    • buildbot-master115 (complete)
    • buildbot-master116 BAD (complete)
    • buildbot-master118 BAD (complete)
    • buildbot-master122 BAD (complete)
    • buildbot-master123 BAD


buildbot-master04 BAD
buildbot-master05 BAD
buildbot-master06 BAD
buildbot-master66 BAD
buildbot-master72 BAD
buildbot-master73 BAD
buildbot-master74 BAD
buildbot-master78 BAD
buildbot-master79 BAD
buildbot-master91 BAD



2015-03-03

  • https://bugzil.la/1138955 - Slow Builds and lagginess
    • tree closure due to backlog (10:00am ET)
    • *mostly* unexpected load (extra poorly-timed pushes to try), although a bunch of test instances not recycling properly
      • coop is investigating these


2015-03-02


2015-02-27

  • queue issues on build masters due to graphene jobs
    • should be resolved by reconfig this morning
  • re-imaging some 10.8 machines as 10.10
    • 10.10 will be running opt jobs on inbound, 10.8 debug on inbound + opt on release branches
    • sheriffs are understandably worried about capacity issues in both pools
  • re-re-imaging talos-linux32-ix-0[01,26]
    • may have an underlying issue with the re-imaging process for linux hw


2015-02-26


2015-02-25

  • things to circle back on today:
    • https://bugzil.la/1136195 - Frequent download timeouts across all trees
    • https://bugzil.la/1136465 - New: Spot instances failing with remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.spread.pb.PBConnectionLost'>:
    • [Bug 1041763] upgrade ec2 linux64 test masters from m3.medium to m3.large
  • https://bugzil.la/1136531- Slave loan request for a tst-linux64-spot vm


2015-02-24

  • https://bugzil.la/1136195 - Frequent download timeouts across all trees
    • related to release traffic?
  • release reconfigs don't log themselves
    • should probably reconfig everything not just build/scheduler masters
      • i think this takes care of itself once masters start updating themselves based on tag updates



10:35:13 <hwine> ah, I see coop already asked Usul about 0900PT
10:36:34 <hwine> ashlee: sounds like our theory of load isn't right - can someone check further, please? https://bugzil.la/1136195#c1
10:38:02 <•pir> hwine: check... what?
10:38:56 <hwine> ftp.m.o is timing out and has closed trees. Our guess was release day load, but that appears not to be it
10:39:33 <•pir> hwine: I can't see any timeouts in that link, I may be missing something.
10:39:48 <jlund> ashlee: hwine catlee-lunch we have bug 1130242#c4 to avail of now too it seems. might provide some insight to health or even a possible cause as to why we are hitting timeouts since the the change time lines up within the time of reported timeouts.
10:40:35 <jlund> pir: that's the bug tracking timeouts. there are timeouts across many of our continuous integration jobs: http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/1424790646/mozilla-inbound_ubuntu32_vm_test-jittest-1-bm01-tests1-linux32-build102.txt.gz
10:40:39 mbrandt → mbrandt|lunch
10:40:49 <•pir> hwine: we don't have a lot of visibility into how ftp.m.o works or is not working. This isn't a good situation, but sadly how it is.
10:41:28 <hwine> pir: right, my understanding is that you (moc) coordinates all the deeper dives for IT infrastructure (which ftp.m.o still is)
10:42:15 <•pir> hwine: To clarify, I don't think anyone has a lot of visibility into how ftp.m.o is working :(
10:42:19 <•pir> it's a mess
10:42:34 <•pir> ashlee: want to loop in C ?
10:43:00 <•pir> (and I think mixing continuous build traffic and release traffic is insane, personally)
10:43:26 <•pir> jlund: yes, that's what I was reading and not seeing anythig
10:43:43 <hwine> pir: system should handle it fine (has in the past) release traffic s/b minimal since we use CDNs
10:44:12 <•pir> hwine: should be. isn't.
10:44:18 <•ashlee> pir sure
10:47:53 <•pir> the load on the ftp servers is... minimal
10:48:27 <•fox2mike> jlund: may I ask where these timeouts are happening from?
10:49:15 <jlund> hm, so load may not be the issue. begs the question "what's changed"
10:49:17 <•pir> and what the timeouts actually are. I can't see anything timing out in the listed logs
10:49:37 <•pir> jlund: for ftp.m.o? nothing that I'm aware of
10:50:06 <cyliang> no bandwith alerts from zeus. looking at the load balancers to see if anything pops out.
10:50:09 <•ashish> from http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/1424790646/mozilla-inbound_ubuntu32_vm_test-jittest-1-bm01-tests1-linux32-build102.txt.gz
10:50:13 <•ashish> i see
10:50:14 <•ashish> 08:00:28 WARNING - Timed out accessing http://ftp.mozilla.org.proxxy1.srv.releng.use1.mozilla.com/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/1424790646/firefox-39.0a1.en-US.linux-i686.tests.zip: timed out
10:50:18 <•ashish> what is that server?
10:50:31 <•fox2mike> USE1
10:50:34 <•fox2mike> FUCK YEAH! :p
10:50:35 → agibson joined (agibson@moz-j04gi9.cable.virginm.net)
10:50:36 <•fox2mike> the cloud baby
10:50:55 <•fox2mike> jlund: I bet if you were to try this from other amazon regions, you might not his this
10:50:57 <•ashish> i don't see timeouts for http://ftp.mozilla.org/*
10:50:59 <cyliang> fox2mike: Is this the same timeout stuff as last time?
10:51:03 <•ashish> (in that log)
10:51:03 <•fox2mike> I'm guessing
10:51:06 <•fox2mike> cyliang: ^
10:51:17 <•fox2mike> because the last time we saw random issues
10:51:21 <•fox2mike> it was all us-east1
10:51:39 <•fox2mike> jlund: for reference - bug 1130386
10:52:11 <•fox2mike> our infra is the same, we can all save time by trying to see if you guys hit this from any other amazon region (if that's possible)
10:53:08 <jlund> proxxy is a host from aws but after failing to try that a few times, we poke ftp directly and timeout after 30 min:
10:53:12 <jlund> https://www.irccloud.com/pastebin/WmSehqzj

Plain Text • 8 lines raw | line numbers


10:53:13 <wesley> jlund's shortened url is http://tinyurl.com/q47zbvl
10:53:15 <•pir> yay cloud
10:54:36 <•pir> jlund: that download from ftp-ssl works fine from anywhere I have access to test it
10:55:06 <•fox2mike> jlund: where did that fail from?
10:55:40 <jlund> sure, and it doesn't always timeout, but of our thousands of jobs, a bunch have failed and timed out.
10:55:54 <unixfairy> jlund can you be more specific
10:56:10 <jlund> fox2mike: same log example: http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/1424790646/mozilla-inbound_ubuntu32_vm_test-jittest-1-bm01-tests1-linux32-build102.txt.gz
10:56:56 <jlund> sorry, I don't know exact failure rate numbers. RyanVM|sheriffduty may know more.
10:57:03 <•fox2mike> jlund: so
10:57:03 <•fox2mike> builder: mozilla-inbound_ubuntu32_vm_test-jittest-1
10:57:04 <•fox2mike> slave: tst-linux32-spot-105
10:57:10 <•fox2mike> that's from amazon again
10:57:18 <•fox2mike> tst-linux32-spot-105
10:57:23 <•fox2mike> that's a spot instance
10:57:34 <•pir> yep, master: http://buildbot-master01.bb.releng.use1.mozilla.com:8201/
10:57:40 <•fox2mike> us-east1
10:57:45 <•pir> so far the connection I see is use1 as fox2mike says
10:58:14 <•fox2mike> we've been through this before :)
10:58:17 <•fox2mike> is all I'm saying
10:59:02 <jlund> sure. let's make sure we can narrow it down to that. I'll see if I can track down more jobs that have hit the timeout where slaves are not in aws.
10:59:08 <jlund> thanks for your help so far.
10:59:49 <•fox2mike> jlund: aws is fine, anything that's a non use1 failure
10:59:55 <•fox2mike> before we go to non aws failure
11:00:06 <•fox2mike> but your case will narrow it down further
11:00:07 <•fox2mike> thanks!
11:00:11 <jlund> rgr
11:00:15 <RyanVM|sheriffduty> fox2mike: things have been quiet for a little while now
11:00:26 <RyanVM|sheriffduty> but we had a lull awhile ago too before another spike
11:00:36 <RyanVM|sheriffduty> so I'm not feeling overly inclined to say that things are resolved
11:00:55 jp-food → jp
11:01:00 <jlund> RyanVM|sheriffduty: have any mac or windows jobs hit this timeout?
11:01:08 <RyanVM|sheriffduty> yes
11:01:13 <RyanVM|sheriffduty> windows definitely
11:01:26 <jlund> k, fox2mike ^ we don't have any windows machines in the cloud
11:01:48 <RyanVM|sheriffduty> random example - https://treeherder.mozilla.org/logviewer.html#?job_id=6928327&repo=mozilla-inbound
11:01:54 <•ashish> are there logs from thoes machines?
11:01:55 <•ashish> ty
11:02:00 → KaiRo joined (robert@moz-dqe9u3.highway.telekom.at)
11:02:17 <RyanVM|sheriffduty> OSX - https://treeherder.mozilla.org/logviewer.html#?job_id=6924712&repo=mozilla-inbound
11:02:51 jlund → jlund|mtg
11:02:57 ⇐ agibson quit (agibson@moz-j04gi9.cable.virginm.net)
11:04:28 jlund|mtg → jlund
11:04:36 <KaiRo> who is the right contact for getting HTTP requests to a Mozilla-owned domain set up to redirect to a different website (another Mozilla-owned domain)?
11:04:50 <KaiRo> the case in question is bug 998793
11:05:27 → agibson joined (agibson@moz-j04gi9.cable.virginm.net)
11:06:48 <•ashish> KaiRo: looks like that IP is hosted/maintained by the community
11:07:05 <•pir> KaiRo: 173.5.47.78.in-addr.arpa domain name pointer static.173.5.47.78.clients.your-server.de.
11:07:09 <•pir> KaiRo: not ours
11:07:39 <jlund> so, it sounds like we have confirmed that this outside aws. for completeness, I'll see if I can find this happening on usw-2 instances too.
11:08:01 agibson → agibson|brb
11:08:23 <KaiRo> ashish: yes, the IP is right now not Mozilla-hosted (atopal, who does host it and actually is an employee nowadays, will be working on getting it moved to Mozilla in the next months) but the domains are both Mozilla-owned
11:09:02 <•pir> KaiRo: the server isn't, though, and you do redirects on server
11:09:54 <KaiRo> pir: well, what we want in that bug is to have mozilla.at point to the same IP as mozilla.de (or CNAME to it or whatever)
11:10:33 <•pir> KaiRo: ah, that's not the same question
11:11:17 <KaiRo> and the stuff hosted by atopal that I was referring to is actually the .de one - I have no idea what the .at one even points to
11:11:45 <•ashish> KaiRo: ok, file a bug with webops. they'll have to change nameservers, setup dns and then put up redirects as needed
11:12:06 <•pir> that
11:13:27 <KaiRo> ashish: OK, thanks!
11:13:56 <•pir> KaiRo: www.mozilla.de or www.mozilla.com/de/ ?
11:14:13 <•pir> KaiRo: the former is community, the latter is mozilla corp
New messages
11:17:04 agibson|brb → agibson
11:17:54 <KaiRo> pir: the former, we want both .at and .de point to the same community site
11:18:57 <•pir> KaiRo: then you need someone in corp to do the dns change and someone who runs the de community site to make sure their end is set up
11:20:08 <KaiRo> pir: sure
11:21:00 <KaiRo> pir: I was mostly concerned about who to contact for the crop piece, I know the community people, we just met this last weekend
11:21:35 <•pir> KaiRo: file a child bug into infra & ops :: moc: service requests
11:21:46 <•pir> KaiRo: if we can't do it directly then we can find someone who can
11:22:10 <KaiRo> pir: thanks, good to know
11:22:18 ⇐ agibson quit (agibson@moz-j04gi9.cable.virginm.net)
11:22:31 <•pir> KaiRo: I'd suggest asking for a CNAME from mozilla.at to mozilla.de so if the de site's IP changes it doesn't break
11:23:03 jlund → jlund|mtg
11:23:51 <KaiRo> pir: yes, that's what I would prefer as well, esp. given the plans to move that communitxy website from atopal's server to Mozilla Community IT
11:25:12 <•ashish> KaiRo: will mozilla.at always remain a direct? (in the near future, at least)
11:25:39 <KaiRo> ashish: in the near future for sure, yes
11:25:45 <•ashish> KaiRo: if so, we can have our static cluster handle the redirect
11:25:59 <•ashish> that migh save some resources for the community
11:26:18 <•pir> if it's ending up on the same server, how does that save resources?
11:27:03 <•ashish> if it's all the same server then yeah, not a huge benefit
11:27:20 Fallen|away → Fallen, hwine → hwine|mtg, catlee-lunch → catlee
11:37:07 <KaiRo> ashish, pir: thanks for your help, I filed bug 1136318 as a result, I hope that moves this forward :)
11:38:31 <•pir> np
11:38:46 <•ashish> KaiRo: yw
11:40:23 coop|lunch → coop|mtg
Tuesday, February 24th, 2015

2015-02-20

  • reimaging a bunch of linux talos machines that have sat idle for 6 months
    • talos-linux32-ix-001
    • talos-linux64-ix-[003,004,008,092]
  • https://bugzil.la/1095300
    • working on slaveapi code for "is this slave currently running a job?"
  • pending is up over 5000 again
    • mostly try
    • Callek: What caused this, just large amounts of pushing? What OS's were pending? etc.


2015-02-19

  • another massive gps push to try, another poorly-terminated json prop
    • https://bugzil.la/1134767
    • rows excised from db by jlund
    • jobs canceled by jlun/nthomas/gps
    • master exception logs cleaned up with:
      • python manage_masters.py -f production-masters.json -R scheduler -R try -j16 update_exception_timestamp


2015-02-18

  • filed https://bugzil.la/1134316 for tst-linux64-spot-341
  • been thinking about builder mappings since last night
    • simplest way may be to augment current allthethings.json output
      • need display names for slavepools
      • need list of regexps matched to language for each slavepool
      • this can be verified internally very easily: can run regexp against all builders in slavepool
      • external apps can pull down allthethings.json daily(?) and process file to strip out only what they need, e.g. slavepool -> builder regexp mapping
      • would be good to publish hash of allthethings.json so consumers can easily tell when it has updated


2015-02-17


2015-02-13

  • going through buildduty report


2015-02-12

  • buildbot db failover by sheeri (planned)
  • https://bugzil.la/1132469 - tree closure
    • lots of idle slaves connected to masters despite high pending counts
    • have rebooted some masters so far:
      • bm70, bm71, bm72, bm73, bm74, bm91, bm94
    • coop looking into windows builders
      • found 2 builders that hadn't run *any* jobs ever (since late sept at least)


2015-02-11

  • reconfig is needed. last one was on thurs. blocked on the 10th from planned reconfig
    • will kick off a reconfig at 10am ET
    • bm118 ended up with 2 reconfig procs running
      • disabled in slavealloc, initiated clean shutdown. Will restart when jobs drain.
  • went through aws_sanity_checker backlog
    • lots of unnamed hosts up for multiple days
      • I'm assuming this is mostly for Windows AWS work based on the platform of the image, but we should really push people to tag instances more rigorously, or expect them to get killed randomly
  • recovering "broken" slaves in slave health list
  • Currently from jacuzzi report, 28 pending windows builds (for non-try) that are not in a jacuzzi
    • 18 of them are disabled for varying reasons, should cull that list to see if any of them can/should be turned on.


2015-02-10


2015-02-09

  • STAT for jlund


2015-02-05

  • tree closures
    • [Bug 1130024] New: Extremely high Linux64 test backlog
      • chalking that one up to a 20% more push increase than what we had previously
    • [Bug 1130207] Several tests failing with "command timed out: 1800 seconds without output running" while downloading from ftp-ssl.mozilla.org
      • again, likely load related but nothing too obvious. worked with netops, suspect we were hitting load balancer issues (ZLB) since hg and ftp share balancers and hg was under heavy load today
      • dcurado will follow up
        • and his follow up: bug 1130242
  • two reconfigs
  • dev-stage01 was running low on disk space
  • loan for sfink



2015-02-04

  • treeherder master db node is getting rebooted for ghost patching
    • I asked mpressman to do it tomorrow and confirm with #treeherder folks first as there was not many on that were familiar with the system
  • puppet win 2008 slaves are ready for the big leagues (prod)!
    • I will be coordinating with markco the testing on that front
  • did a reconfig.lot's landed
  • investigated the 13 win builders that got upgraded RAM. 4 of them have been disabled for various issues
  • dustin ghost patched bm103 and signing5/6


2015-02-03

  • fallout from: Bug 1127482 - Make Windows B2G Desktop builds periodic
    • caused a ~dozen dead command items every 6 hours
    • patched: bug 1127482#c15
    • moved current dead items to my own special dir in case I need to poke them again
    • more dead items will come every 6 hours till above patch lands
  • arr/dustin ghost slave work
    • pod4 and 5 of pandas was completed today
      • 1 foopy failed to clone tools (/build/sut_tools/) on re-image
        • it was a timeout and puppet wasn't smart enough to re clone it without a removal first
    • try linux ec2 instances completed
  • maybe after ami / cloud-tools fallout we should have nagios alerts for when aws spins up instances and kils them right away
  • pro-tip when looking at ec2 graphs:
    • zoom in or out to a time you care about and click on individual colour headings in legend below graph
      • last night I did not click on individual moz-types under running graph and since there is so few bld-linux builders that run normally anyway, it was hard to notice any change



2015-02-02

  • https://bugzil.la/1088032 - Test slaves sometimes fail to start buildbot after a reboot
    • I thought the problem had solved itself, but philor has been rebooting windows slaves everyday which is why we haven't run out of windows slaves yet
    • may require some attention next week
  • panda pod round 2 and 3 started today
    • turns out disabling pandas in slavealloc can kill its current job
    • Calling fabric's stop command (disable.flg) on foopies kills its current job
    • This was a misunderstanding in terms of plan last week, but is what we did pods 1->3 with, and will be the continued plan for next sets
    • we'll inform sheriffs at start and end of each pod's work
  • reconfig is failing as masters won't update local repos
    • vcs error: 500 ISE
    • fallout from vcs issues. gps/hwine kicked a webhead and all is merry again
  • added a new report link to slave health for runner's dashboard
  • late night Tree Closures
    • Bug 1128780
      • test pending skyrocketed, builds not running, builder graphs broken
      • tests were just linux test capacity (with ~1600 pending in <3 hours)
      • graphs relating to running.html were just a fallout from dead-code removal
      • builds not running brought together mrrrgn dustin and catlee and determined it was fallout from dustin's AMI work with cent 6.5 causing earlier AMI's to get shut off automatically on us
    • generic.scl3 got rebooted, causing mozpool to die out and restart, leaving many panda jobs dead
  • B2G nightlies busted, unknown cause
    • Bug 1128826



2015-01-30

  • loan for markco: https://bugzil.la/1127411
  • GHOST
  • started reconfig 11:00 PT
  • fewer pandas decomm-ed than anticipated, will have final numbers today
  • https://bugzil.la/1109862 - re-assigned to relops for dll deployment
  • buildapi + new buildbot passwd: do we know what went wrong here?
    • catlee suspects he updated the wrong config
  • positive feedback from philor on Callek's jacuzzi changes


2015-01-29


2015-01-28


2015-01-27


2015-01-26

  • audited windows pool for RAM: bug 1122975#c6
  • 'over the weekend': small hiccup with bgp router swap bug: killed all of scl3 for ~10min not on purpose.
    • tl;dr - everything came back magically and I only had to clear up ~20 command queue jobs
  • which component is nagios bugs these days? seems like mozilla.org::Infrastructure & Operations bounced https://bugzil.la/1125218 back to releng::other. do we (releng) play with nagios now?
    • "MOC: Service Requests" - refreshed assurance per chat with MOC manager.(linda)
  • terminated loan with slaveapi: bug 1121319#c4
  • attempted reconfig but hit conflict in merge: bug 1110286#c13
  • catlee is changing buildapi r/o sql pw now (11:35 PT)
  • updated trychooser to fix bustage


2015-01-23

  • deployed new bbot r/o pw to aws-manager and 'million other non puppetized tools'
    • do we have a list? We should puppetize them *or* replace them
  • filed: Bug 1125218 - disk space nagios alerts are too aggressive for signing4.srv.releng.scl3.mozilla.com
  • investigated: Bug 1124200 - Android 4 L10n Nightly Broken
  • report-4hr hung at 10:42 - coop killed the cron task



2015-01-22

  • https://bugzil.la/1124705 - tree closure due to builds-4hr not updating
    • queries and replication blocked in db
    • sheeri flushed some tables, builds-4hr recovered
    • re-opened after 20min
  • https://bugzil.la/1121516 - sheeri initiated buildbot db failover after reconfig (per email)
  • philor complaining about panda state:
    • "I lied about the panda state looking totally normal - 129 broken then, fine, exactly 129 broken for all time, not so normal"
    • filed Bug 1124863 - more than 100 pandas have not taken a job since 2015-01-20 around reconfig
      • status: fixed
  • filed Bug 1124850 - slaveapi get_console error handling causes an exception when log formatting
    • status: wontfix but pinged callek before closing
  • filed Bug 1124843 - slaveapi cltbld creds are out of date
    • status: fixed, also improved root pw list order
  • did a non merge reconfig for armen/bustage
  • b2g37 fix for bustage I (jlund) caused. reconfiged https://bugzil.la/1055919


2015-01-21

  • landed fix for https://bugzil.la/1123395 - Add ability to reboot slaves in batch on the slavetype pag
  • many of our windows timeouts (2015-01-16) may be the result of not having enough RAM. Need to look into options like doubling page size: bug 1110236#c20


2015-01-20

  • reconfig, mostly to test IRC notifications
  • master
  • grabbed 2 bugs:
  • network flapping thoughout the day
  • b2g bumper bustage
  • rebooted ~100 pandas that stopped taking jobs after reconfig
  • Bug 1124059 - create a buildduty dashboard that highlights current infra health
  • TODO: Bug for "make it painfully obvious when slave_health testing mode is enabled, thus is displaying stale data"
    • hurt philor in #releng this evening when an old patch wih testing mode on deployed..
    • i have a precommit hook for this now, shouldn't happen again


2015-01-19

  • Filed bugs for issues discussed on Friday:
  • fixed slavealloc datacenter issue for some build/try linux instances - bug 1122582#c7
  • re-imaged b-2008-ix-0006, b-2008-ix-0020, b-2008-ix-0172
  • deployed 'terminate' to slaveapi and then broke slaveapi for bonus points
  • re-patched 'other' aws end points for slaveapi - deploying that today (20th)
  • fixed nical's troublesome loan


2015-01-16 (rollup of below scratchpad)

JLUND
sheriffs requested I investigate:

  • spike in win64 filesystem loops:
    • sheriffs suggested they have pinged many times recently and they will start disabling slaves if objdir nuking is not preferable
    • nuking b-2008-ix-0114 objdir of related builder
    • filed bug 1122746
  • Bug 916765 - Intermittent "command timed out: 600 seconds without output, attempting to kill" running expandlibs_exec.py in libgtest
    • diagnosis: bug 916765#c193
    • follow up: I will post a patch but it is not buildduty actionable from here on out IMO
  • Bug 1111137 - Intermittent test_user_agent_overrides.html | Navigator UA not overridden at step 1 - got Mozilla/5.0 (Android; Mobile; rv:37.0) Gecko/37.0 Firefox/37.0, expected DummyUserAgent
  • Bug 1110236 - Intermittent "mozmake.exe[6]: *** [xul.dll] Error 1318" after "fatal error LNK1318: Unexpected PDB error"
  • there was a common trend from the above 3 bugs with certain slaves


loans:



CALLEK
Puppet Issues:

  • Had a db_cleanup puppet failure on bm81, catlee fixed with http://hg.mozilla.org/build/puppet/rev/d88423d7223f
  • There is a MIG puppet issue blocking our golden AMI's from completing. Ulfr pinged in #releng and I told he has time to investigate (rather than asking for an immediate backout)


Tree Closure:

  • bug 1122582
  • Linux jobs, test and build were pending far too long
  • I (Callek) got frustrated trying to get assistance to find out what the problem is and while trying to get other releng assistance to look at the problem
  • Boils down to capacity issues, but was darn hard to pinpoint


Action Items

  • Find some way to identify we're at capacity in AWS easier (my jacuzzi slave health work should help with that, at least a bit)
  • Get <someone> to increase our AWS capacity or find out if/why we're not using existing capacity. If increasing we'll need more masters.




2015-01-16

arr
12:08:38 any of the eu folks around? looks like someone broke puppet last night.

mgerva
12:20:01 arr: i'm here

arr
12:21:10 mgerva: looks like a problem with someone who's trying to upgrade mig
12:21:32 mgerva: it's been sending out mail about failing hosts
12:21:39 wasn't sure if it was also taking them offline eventually
12:21:48 (so I think this is limited to linux)
12:32:43 mgerva is now known as mgerva|afk

pmoore
12:47:16 arr: mgerva|afk: since the sheriffs aren't complaining yet, we can probably leave this for build duty which should start in a couple of hours

arr
12:47:46 pmoore: okay!

pmoore
12:47:51 i don't think anyone is landing puppet changes at the moment, so hopefully it should affect anything… i hope!
12:48:02 *shouldn't*

I see two different errors impacting different types of machines:

  • Issues with mig: Puppet (err): Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install mig-agent=20150109+a160729.prod' returned 100
  • Issues with a different config file: Puppet (err): Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter user on File[/builds/buildbot/db_maint/config.ini] at /etc/puppet/production/modules/buildmaster/manifests/db_maintenance.pp:48


2015-01-15


2015-01-14

  • slave loan for tchou
  • started patch to reboot slaves that have not reported in X hours (slave health)
  • reconfig for catlee/ehsan
  • recovered 2 windows builders with circular directory structure


2015-01-13

  • reconfig for ehsan
  • https://bugzil.la/1121015 - dolphin non-eng nightlies busted after merge
    • bhearsum took it (fallout from retiring update.boot2gecko.org)
  • scheduler reconfig for fubar
  • https://bugzil.la/1117811 - continued master setup for Fallen
  • clearing buildduty report backlog


2015-01-12

  • recovering loaned slaves
  • setting up Tb test master for Fallen
    • already has one apparently, some commentary in bug 1117811
  • reconfig took almost 4hr (!)
  • some merge day fallout with split APK


2015-01-08

  • bug 1119447 - All buildbot-masters failing to connect to MySQL: Too many connections
    • caused 3-hour tree closure


2015-01-07


2015-01-06

  • bug 1117395 - Set RETRY on "timed out waiting for emulator to start" and "We have not been able to establish a telnet connection with the emulator"
  • reclaiming loaned machines based on responses to yesterday's notices


2015-01-05

  • sent reminder notices to people with loaned slaves


2014-12-30


2014-12-29

  • returning spot nodes disabled by philor
    • these terminate pretty quickly after being disabled (which is why he does it)
    • to re-enable en masse, run 'update slaves set enabled=1 where name like '%spot%' and enabled=0' in the slavealloc db
    • use the buildduty report, click on the 'View list in Bugzilla' button, and then close all the spot node bugs at once
  • started going throung bugs in the dependencies resolved section based on age. Here is a rundown of state:
    • b-2008-ix-0010: kicked off a re-image, but I did this before fire-and-forget in early Dec and it doesn't seem to have taken. will check back in later
      •  :markco using to debug Puppet on Windows issues
    • panda-0619: updated relay info, but unclear in bug whether there are further issues with panda or chassis


2014-12-14


2014-12-19

  • what did we accomplish?
    • vcssync, b2 bumper ready to hand-off to dev services(?)
    • increased windows test capacity
    • moved staging slaves to production
    • disabled 10.8 on try
      • PoC for further actions of this type
    • investigations with jmaher re: turning off "useless" tests
    • opening up releng for contributions:
      • new public distribution list
      • moved tests over to travis
      • mentored bugs
    • improved reconfigs
    • set up CI for b2g bumper
  • what do we need to accomplish next quarter?
    • self-serve slave loan
    • turn off "useless" tests
      • have a way to do this easily and regularly
    • better ability to correlate tree state changes with releng code changes
    • better platform change pipeline
      • proper staging env
    • task cluster tackles most of the above, therefore migration of jobs to task cluster should enable these as a consequence
  • what tools do we need?
    • self-serve slave loan
    • terminate AWS instances from slave health (slaveapi)
    • ability to correlate releng changes with tree state changes
      • e.g. linux tests started timing out at Thursday at 8:00am: what changed in releng repos around that time?
      • armen's work on pinning mozharness tackles the mozharness part - migrating to task cluster puts build configs in-tree, so is also solved mostly with task cluster move


2014-11-27

  • https://bugzil.la/1105826
    • trees closed most of the day due to Armen's try jobs run amok
    • reporting couldn't handle the volume of retried jobs, affected buildapi and builds-4hr
      • disabled buildapi cronjobs until solution found
    • db sync between master->slave lost for 5 hours
    • re-ran buildapi cronjobs incrementally by hand in order to warm the cache for build-4hr
    • all buildapi cronjobs re-enabled
    • catlee picked up https://bugzil.la/733663 for the long-term fix
    • didn't get to deploy https://bugzil.la/961279 as planned :(


2014-11-26

  • https://bugzil.la/961279 - Mercurial upgrade - how to proceed?
    • yes, we should have time to deploy it Thursday/Friday this week


2014-11-25


2014-11-24


2014-11-21

  • work on 10.10
    • running in staging
  • restarted bm84
  • reconfig for bhearsum/rail for pre-release changes for Fx34
  • setup foopy56 after returning from diagnostics


2014-11-20 a.k.a "BLACK THURSDAY"



2014-11-19


2014-11-18

  • bm82 - BAD REQUEST exceptions
    • gracefully shutdown and restarted to clear
  • updated tools on foopys to pick up Callek's patch to monitor for old pywebsocket processes
  • sent foopy56 for diagnostics
  • https://bugzil.la/1082852 - slaverebooter hangs
    • had been hung since Nov 14
    • threads aren't terminating, need to figure out why
    • have I mentioned how much i hate multi-threading?
  • https://bugzil.la/1094293 - 10.10 support
    • patches waiting for review


2014-11-17


2014-11-14

  •  ???


2014-11-13

  •  ???


2014-11-12

  •  ???


2014-11-11

  •  ???


2014-11-10

  • release day
    • ftp is melting under load; trees closed
      • dev edition went unthrottled
        • catlee throttled background updates to 25%
      • dev edition not on CDN


2014-11-07

  • shared mozharness checkout
  • jlund hg landings
  • b2g_bumper travis tests working
  • buildbot-master52
    • hanging on every reconfig
    • builder limits, hitting PB limits
    • split masters: Try + Everything Else?
    • graceful not working -> nuke it from orbit
    • structured logging in mozharness has landed
  • coop to write docs:
    • moving slaves from production to staging
    • dealing with bad slaves


2014-11-06

  • b2g_bumper issues
  • https://bugzil.la/1094922 - Widespread hg.mozilla.org unresponsiveness
  • buildduty report queue
  • some jobs pending for more than 4 hours
    • aws tools needed to have the new cltbld password added to their json file, idle instances not being reaped
    • need some monitoring here


2014-11-05

  • sorry for the last few days, something important came up and i've barely been able to focus on buildduty
  • https://bugzil.la/foopy56
    • hitting load spikes
  • https://bugzil.la/990173 - Move b2g bumper to a dedicated host
    • bm66 hitting load spikes
    • what is best solution: beefier instance? multiple instances?
  • PT - best practices for buildduty?
    • keep "Current" column accurate


2014-11-04

  • t-snow-r4-0002 hit an hdiutil error and is now unreachable
  • t-w864-ix-026 destoying jobs, disabled
  • bug 1093600
    • bugzilla api updates were failing, fixed now
    • affected reconfigs (script could not update bugzilla)
  • bug 947462
    • tree outage when this landed
    • backed it out
    • probably it can be relanded, just needs a clobber


2014-11-03


2014-10-31

  • valgrind busted on Try
    • only build masters reconfig-ed last night by nthomas
      • reconfig-ed try masters this morning


2014-10-30

  • how best to handle broken manifests?
    • difference of opinion w/ catlee
    • catlee does see the human cost of not fixing this properly
  • mapper docs
  • b2g bumper: log rotation
  • https://bugzil.la/1091707
    • Frequent FTP/proxxy timeouts across all trees
      • network blip?
  • https://bugzil.la/1091696
    • swap on fwunit1.private.releng.scl3.mozilla.com is CRITICAL: SWAP CRITICAL - 100% free (0 MB out of 0 MB)
    • these are dustin's firewall unit tests: ping him when we get these alerts
  • reconfig


2014-10-29

  • b2g bumper
    • b2g manifests
      • no try for manifests
  • All new w864 boxes have wrong resolution
  • started thread about disabling try testing on mtnlion by default


2014-10-28

  • testing new hg 3.1.2 GPO
  • cleaned up loaner list from yesterday
    • closed 2 bugs that we're unused
    • added 2 missing slavealloc notes
    • terminated 11 instances
    • removed many, many out-of-date names & hosts from ldapadmin
  • lots of bigger scope bugs getting filed under the buildduty category
    • most belong in general automation or tools IMO
    • I don't think buildduty bugs should have a scope bigger than what can be accomplished in a single day. thoughts?
  • reconfig to put new master (bm119) and new Windows test slaves into production
  • massive spike in pending jobs around 6pm ET
    • 2000->5000
    • closed trees
  • waded through the buildduty report a bit


2014-10-27

  • 19 *running* loan instances

dev-linux64-ec2-jlund2
dev-linux64-ec2-kmoir
dev-linux64-ec2-pmoore
dev-linux64-ec2-rchien
dev-linux64-ec2-sbruno
tst-linux32-ec2-evold
tst-linux64-ec2-evanxd
tst-linux64-ec2-gbrown
tst-linux64-ec2-gweng
tst-linux64-ec2-jesup
tst-linux64-ec2-jesup2
tst-linux64-ec2-jgilbert
tst-linux64-ec2-kchen
tst-linux64-ec2-kmoir
tst-linux64-ec2-mdas
tst-linux64-ec2-nchen
tst-linux64-ec2-rchien
tst-linux64-ec2-sbruno
tst-linux64-ec2-simone

27 open loan bugs:
http://mzl.la/1nJGtTw

We should reconcile. Should also cleanup entries in ldapadmin.


2014-10-24


2014-10-23

  • test slaves sometimes fail to start buildbot on reboot
  • re-imaging a bunch of w864 machines that were listed as only needing a re-image to be recovered:
    • t-w864-ix-0[04,33,51,76,77]
    • re-image didn't help any of these slaves
    • bug 1067062
  • investigated # of windows test masters required for arr
    • 500 windows test slaves, 4 existing windows test masters
  • OMG load
    • ~7000 pending builds at 4pm ET
    • KWierso killed off lots of try load: stuff that had already landed, stuff with followup patches
      • developer hygiene is terrible here


2014-10-22

  • many win8 machines "broken" in slave health
    • working theory is that 64-bit browser is causing them to hang somehow
    • https://bugzil.la/1080134
    • same for mtnlion
    • same for win7
    • we really need to find out why these slaves will simply fail to start buildbot and then sit waiting to be rebooted


2014-10-21

  • bug 1086564 Trees closed
    • alerted Fubar - he is working on it
  • bug 1084414 Windows loaner for ehsan
  • killing esr24 branch
  • https://bugzil.la/1066765 - disabling foopy64 for disk replacement
  • https://bugzil.la/1086620 - Migrate slave tools to bugzilla REST API
    • wrote patches and deployed to slavealloc, slave health
  • trimmed Maintenance page to Q4 only, moved older to 2014 page
  • filed https://bugzil.la/1087013 - Move slaves from staging to production
    • take some of slave logic out of configs, increase capacity in production
  • helped mconley in #build with a build config issue
  • https://bugzil.la/973274 - Install GStreamer 1.x on linux build and test slaves
    • this may have webrtc implications, will send mail to laura to check


2014-10-20

  • reconfig (jetpack fixes, alder l10n, holly e10s)
    • several liunx64 test masters hit the PB limit
      • put out a general call to disable branches, jobs
      • meanwhile, set masters to gracefully shutdown, and then restarted them. Took about 3 hours.
  • 64-bit Windows testing
    • clarity achieved!
      • testing 64-bit browser on 64-bit Windows 8, no 32-bit testing on Window 8 at all
      • this means we can divvy the incoming 100 machines between all three Windows test pools to improve capacity, rather than just beefing up the WIn8 platform and splitting it in 2


2014-10-17

  • blocklist changes for graphics (Sylvestre)
  • code for bug updating in reconfigs is done
    • review request coming today
  • new signing server is up
    • pete is testing, configuring masters to use it
  • some classes of slaves not reconnecting to masters after reboot
    • e.g. mtnlion
    • need to find a slave in this state and figure out why
      • puppet problem? runslave.py problem (connection to slavealloc)? runner issue (connection to hg)?
  • patch review for bug 1004617
  • clobbering m-i for rillian
  • helping Tomcat cherry-pick patches for m-r
  • reconfig for Alder + mac-signing


2014-10-16

  • Updated all windows builders with new ffxbld_rsa key
  • Patched reconfig code to publish to bugzilla - will test on next reconfig
  • Working on set up of mac v2 signing server
  • Fixed sftp.py script
  • Set up meeting the J Lal, H Wine, et al for vcs sync handover
  • lost my reconfig logs from yesterday in order to validate against https://hg.mozilla.org/build/tools/file/a8eb2cdbe82e/buildfarm/maintenance/watch_twistd_log.py#l132 - will do so with next reconfig
  • spoke to Amy about windows reimaging problem, and requested a single windows reimage to validate GPO setup
  • reconfig for alder and esr24 changes
  • rebooting mtnlion slaves that had been idle for 4 hours (9 of them)
    • this seems to be a common occurrence. If I can find a slave in this state today, I'll file a bug and dive in. Not sure why mahcine is rebooting and not launching buildbot.


2014-10-15


2014-10-14

  • bug 1081825 b2gbumper outage / mirroring problem - backed out - new mirroring request in bug bug 1082466
    • symptoms: b2g_bumper lock file is stale
    • should mirror new repos automatically rather than fail
      • bare minimum: report which repo is affected
  • bug 962863 rolling out l10n gecko and l10n gaia vcs sync - still to do: wait for first run to complete, update wiki, enable cron
  • bug 1061188 rolled out, and had to backout due to puppet changes not hitting spot instances yet, and gpo changes not hitting all windows slaves yet - for spot instances, just need to wait, for GPO i have a needinfo on :markco
    • need method to generate new golden AMIs on demand, e.g. when puppet changes land
  • mac signing servers unhappy - probably not unrelated to higher load due to tree closure - have downtimed in #buildduty for now due to extra load
    • backlog of builds on Mac
      • related to slaverebooter hang?
      • many were hung for 5+ hours trying to run signtool.py on repacks
        • not sure whether this was related to (cause? symptom?) of signing server issues
        • could also be related to reconfigs + key changes (ffxbld_rsa)
      • rebooted idle&hung mac builders by hand
      • bug 1082770 - getting another mac v2 signing machine into service
  • sprints for this week:
    • [pete] bug updates from reconfigs
    • [coop] password updates?
  • slaverebooter was hung but not alerting, *however* I did catch the failure mode: indefinitely looping waiting for an available worker thread
    • added a 30min timeout waiting for a worker, running locally on bm74
    • filed bug 1082852
  • put foppy64 back into service - bug 1066765
  • https://bugzil.la/1082818 - t-w864-ix loaner for Armen
  • https://bugzil.la/1082784 - tst-linux64-ec2 loaner for dburns
  • emptied buildduty bug queues


2014-10-13



2014-10-10:

  • kgrandon reported to getting updates for flame-kk
  • work


2014-10-09

  • db issue this morning all day
    • sheeri ran an errant command on the slave db that inadvertently propagated to the master db
    • trees closed for about 2 hours until jobs started
    • however, after the outage while the trees were still closed, we did a live fail over between the master and the slave without incident
    • later in the day, we tried to fail back over to the master db from the slave db, but we ended up with inconsistent data between the two databases. This resulted in a bunch of jobs not starting because they were in the wrong db.
    • fixed with a hot copy
    • filed https://bugzil.la/1080855 for RFO\
  • https://bugzil.la/1079396 - loaner win8 machine for :jrmuziel
  • https://bugzil.la/1075287 - loaner instance for :rchien
    • after some debugging over the course of the day, determined he needed a build instance after all
  • filed https://bugzil.la/1080951 - Add fabric action to reset the timestamp used by buildbot-master exception log reporting


2014-10-08


2014-10-07

  • bug 1079256 - B2G device image nightlies (non-eng only) constantly failing/retrying due to failure to upload to update.boot2gecko.org
    • cleaned up, now 5% free
    • fix is to stop creating/publishing/uploading b2g mars for all branches *except* 1.3 <- bug 1000217
  • fallout from PHX outage?
  • cleared buildduty report module open dependencies
  • filed https://bugzil.la/1079468 - [tracking][10.10] Continuous integration testing on OS X 10.10 Yosemite


2014-10-06

  • bug 1078300#c3
    • hg push showing on tbpl and treeherder with no associated builders generated
  • sprints for this week:
    • slaverebooter
      • [coop] determine why it sometimes hangs on exit
    • [pete] end_to_end_reconfig.sh
      • add bug updates


2014-10-03

  • slaverebooter hung
    • added some extra instrumentation locally to try to find out why, when removed the lockfile and restarted
    • hasn't failed again today, will see whether it fails around the same time tonight (~11pm PT)
  • https://bugzil.la/1077432 - Skip more unittests on capacity-starved platforms
    • now skipping opt tests for mtnlion/win8/win7/xp
  • reconfig
  • https://bugzil/la/1065677 - started rolling restarts of all master
    • done


2014-10-02

  • mozmill CI not receiving pulse messages
    • some coming through now
    • no logging around pulse notifications
  • mtnlion machines
    • lost half of pool last night, not sure why
    • dbs shutdown last night <- related?
    • reboot most, leave 2 for diagnosis
    • snow affected? yes! rebooted
    • also windows
      • rebooted XP and W8
      • can't ssh to w7
        • rebooted via IPMI (-mgmt via web)
  • pulse service - nthomas restarted pulse on masters
    • multiple instances running, not checking PID file
    • bug 1038006
  • bug 1074147 - increasing test load on Windows
    • coop wants to start using our skip-test functionality on windows (every 2) and mtnlion (every 3)
  • uploads timing out
  • https://bugzil.la/1069429 - Upload mozversion to internal pypi
  • https://bugzil.la/1076934 - Temporarily turn off OTA on FX OS Master branch
    • don't have a proper buildid to go on here, may have time to look up later
  • added report links to slave health: hgstats, smokepings


2014-10-01


2014-09-30

  • https://bugzil.la/1074827 - buildbot-configs_tests failing on Jenkins due to problem with pip install of master-pip.txt
    • non-frozen version of OpenSSL being used - rail fixing and trying again
  • https://bugzil.la/943932 - T-W864-IX-025 having blue jobs
    • root cause not known - maybe faulty disk - removed old mozharness checkout, now has a green job
  • https://bugzil.la/1072434 - balrog submitter doesn't set previous build number properly
    • this caused bustage with locale repacks - nick and massimo sorted it out
  • https://bugzil.la/1050808 - several desktop repack failures today - I proposed we apply patch in this bug
  • https://bugzil.la/1072872 - last machines rebooted
  • https://bugzil.la/1074655 - Requesting a loaner machine b2g_ubuntu64_vm to diagnose bug 1053703
  • going through buildduty report
    • filed new panda-recovery bug, added pandas to it
    • t-snow-r4-0075: reimaged, returned to production
    • talos-linux64-ix-027: reimaged, returned to production
    • emptied 3 sections (stopped listing the individual bugs)
  • reconfig
  • https://bugzil.la/1062465 - returned foopy64 and attached pandas to production
    • disk is truly failing on foopy64, undid all that work


2014-09-29

  • https://bugzil.la/1073653 - bash on OS X
    • dustin landed fix, watched for fallout
    • complications with signing code bhearsum landed
      • all Macs required nudging (manual puppet runs + reboot). Mercifully dustin and bhearsum took care of this.
  • https://bugzil.la/1072405 - Investigate why backfilled pandas haven't taken any jobs
    • checking failure logs for patterns
    • looks like mozpool is still trying to reboot using old relay info: needs re-sync from inventory?
    • tools checkout on foopies hadn't been updated, despite a reconfig on Saturday
    • enabled 610-612
    • each passed 2 tests in a row, re-enabled the rest
  • cleaned up resolved loans for bld-lion, snow, and mtnlion machines
  • https://bugzil.la/1074358 - Please loan OS X 10.8 Builder to dminor
  • https://bugzil.la/1074267 - Slave loan request for a talos-r4-snow machine
  • https://bugzil.la/1073417 - Requesting a loaner machine b2g_ubuntu64_vm to diagnose


2014-09-26

  • cleared dead pulse queue items after pulse publisher issues in the morning
  • https://bugzil.la/1072405 - Investigate why backfilled pandas haven't taken any jobs
    • updated devices.json
    • created panda dirs on floppies
    • had to re-image panda-0619 by hand
    • all still failing, need to investigate on Monday
  • https://bugzil.la/1073040 - loaner for mdas


Besides the handover notes for last week, which I received from pete, there are the following issues:

bug 1038063 Running out of space on dev-stage01:/builds
The root cause of the alerts was the addition of folder /builds/data/ftp/pub/firefox/releases/31.0b9 by Massimo in order to run some tests.
Nick did some further cleanup, the bug has been reopened this morning by pete, proposing to automate some of the steps nick did manually.

https://bugzilla.mozilla.org/show_bug.cgi?id=1036176 Some spot instances in us-east-1 are failing to connect to hg.mozilla.org
Some troubleshooting has been done by Nick, and case 222113071 has been opened with AWS

2014-07-07 to 2014-07-11

Hi Simone,

Open issues at end of week:

foopy117 is playing up (bug 1037441)
this is also affecting end_to_end_reconfig.sh (solution: comment out manage_foopies.py lines from this file and run manually)
Foopy 117 seems to be back and working normally

Major problems with pending queues (bug 1034055) - this should hopefully be fixed relatively soon. most notably linux64 in-house ix machines. Not a lot you can do about this - just be aware of it if people ask.
Hopefully this is solved after Kim's recent work

Two changes currently in queue for next reconfig: bug 1019962 (armenzg) and bug 1025322 (jford)

Some changes to update_maintenance_wiki.sh from aki will be landing when the review passes: (bug 1036573) - potential is to impact the wiki update in end_to_end_reconfig.sh as it has been refactored - be aware of this.

Currently no outstanding loaner requests at time of handover, but there are some that need to be checked or returned to the pool.

See the 18 open loan requests: https://bugzilla.mozilla.org/buglist.cgi?bug_id=989521%2C1036768%2C1035313%2C1035193%2C1036254%2C1006178%2C1035270%2C876013%2C818198%2C981095%2C977190%2C880893%2C1017303%2C1023856%2C1017046%2C1019135%2C974634%2C1015418&list_id=10700493


I've pinged all the people in this list (except for requests less than 5 days old) to ask for status.

Pete

2014-05-26 to 2014-05-30

  • Monday
  • Tuesday
    • reconfig
      • catlee's patch had bustage, and armenzg's had unintended consequences
      • they each reconfiged again for their own problems
    • buildduty report:
      • tackled bugs without dependencies
      • tackled bugs with all dependencies resolved
  • Wednesday
    • new nightly for Tarako
      • was actually a b2g code issue: Bug 1016157 - updated the version of vold
    • resurrecting tegras to deal with load
  • Thursday
    • AWS slave loan for ianconnoly
    • puppet patch for talos-linux64-ix-001 reclaim
    • resurrecting tegras
  • Friday
    • tegras continue to fall behind, ping Pete very late Thursday with symptoms. Filed https://bugzil.la/1018118
    • reconfig
      • chiefly to deploy https://bugzil.la/1017599 <- reduce # of test on tegras
      • fallout:
        • non-unified mozharness builds are failing in post_upload.py <- causing queue issues on masters
        • panda tests are retrying more than before
          • hitting "Caught Exception: Remote Device Error: unable to connect to panda-0402 after 5 attempts", but it *should be non-fatal, i.e. test runs fine afterwards but still gets flagged for retry
          • filed: https://bugzil.la/1018531
        • reported by sheriffs (RyanVM)

2014-05-05 to 2014-05-09

2014-04-21 to 2014-04-25

follow up:

buildduty report:
Bug 999930 - put tegras that were on loan back onto a foopy and into production

action items:
* Bug 1001518 - bld-centos6-hp-* slaves are running out of disk space

    • this pool had 4 machines run out of disk space all within the last week
    • I scrubbed a ton of space (bandaid) but the core issue will need to be addressed

load:
(high load keep an eye on) Bug 999558 - high pending for ubuntu64-vm try test jobs on Apr 22 morning PT
(keep an eye on) bug 997702


(jlund) reboot all these xp stuck slaves-https://bugzil.la/977341- XP machines out of action
**https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slavetype.html?class=test&type=t-xp32-ixTHERE ARE ONLY 4 MACHINES TODAY THAT ARE "BROKEN" AND ONLY HUNG TODAY.
pmoore: there is only 1 now

  • (jlund) iterate through old disabled slaves in these platform lists - https://bugzil.la/984915 - Improve slave health for disabled slaves <- THIS WAS NOT DONE. I ASKED PMOORE TO HELP
    • pmoore: i'm not entirely sure which platform lists this means, as the bug doesn't contain a list of platforms. So I am looking at all light blue numbers on https://secure.pub.build.mozilla.org/builddata/reports/slave_health/index.html (i.e. the disabled totals per platform). When jlund is online later i'll clarify with him.
    • jlund: thanks pete. Sorry I meant all platform lists I suppose starting with whatever platform held our worst wait times. I have started going through the disabled hosts looking for 'forgotten ones'

2014-04-10 to 2014-04-11 (Thursday and Friday)

bug 995060
Nasty nasty tree closure lasting several hours
b-c taking loooooong time and log files too large for buildbot to handle
Timeout for MOCHITEST_BC_3 increased from 4200s to 12000s
When Joel's patch has landed: bug 984930
then we should "undo" the changes from bug 995060 and put timeout back down to 4200s (was just a temporary workaround). Align with edmorley on this.

bug 975006
bug 938872
after much investigation, it turns out a monitor is attached to this slave - can you raise a bug to dc ops to get it removed?


Loaners returned:
https://bugzil.la/978054
https://bugzil.la/990722
https://bugzil.la/977711 (discovered this older one)

Loaners created:
bug 994283


bug 994321#c7
Still problems with 7 slaves that can't be rebooted:
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=test&type=talos-mtnlion-r5&name=talos-mtnlion-r5-005
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=test&type=talos-mtnlion-r5&name=talos-mtnlion-r5-006
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=test&type=talos-mtnlion-r5&name=talos-mtnlion-r5-061
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=test&type=talos-mtnlion-r5&name=talos-mtnlion-r5-065
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=test&type=talos-mtnlion-r5&name=talos-mtnlion-r5-074
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=test&type=talos-mtnlion-r5&name=talos-mtnlion-r5-086
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=test&type=talos-mtnlion-r5&name=talos-mtnlion-r5-089
Slave API has open bugs on all of these.


https://bugzil.la/977341
Stuck Win XP slaves - only one was stuck (t-xp32-ix-073). Rebooted.


Thanks Callek!

Week of 2014-04-05 to 2014-04-09 (thurs-wed)

Hiya pete

  • van has been working hard at troubleshooting winxp 085: bug 975006#c21
    • this needs to be put back into production along with 002 and reported back to van on findings
    • note this is a known failing machine. please try to catch it fail before sheriffs.
  • we should reconfig either thurs or by fri at latest
  • latest aws sanity check runthrough yielded better results than before. Very view long running lazy instances. Very few unattended loans. This should be checked again on Friday
  • there was a try push that broke a series of mtnlion machines this afternoon. Callek, nthomas, and Van worked hard at helping me diagnose and solve issue.
    • there are some slaves that failed to reboot via slaveapi. This is worth following up on especially since we barely have any 10.8 machines to begin with:
    • bug 994321#c7
  • on tues we started having github/vsync issues where sheriffs noticed that bumper bot wasn't keeping up with csets on github.
    • looks like things have been worked on and possibly fixed but just a heads up
    • bug 993632
  • as per: bug 991259#c1 I checked on these and the non green ones should be followed up on

tegra-063.tegra.releng.scl3.mozilla.com is alive <- up and green
tegra-050.tegra.releng.scl3.mozilla.com is alive <- up and green
tegra-028.tegra.releng.scl3.mozilla.com is alive <- up but not running jobs
tegra-141.tegra.releng.scl3.mozilla.com is alive <- up and green
tegra-117.tegra.releng.scl3.mozilla.com is alive <- up but not running jobs
tegra-187.tegra.releng.scl3.mozilla.com is alive <- up and green
tegra-087.tegra.releng.scl3.mozilla.com is alive <- up and green
tegra-299.tegra.releng.scl3.mozilla.com is alive <- up and green
tegra-309.tegra.releng.scl3.mozilla.com is alive <- up and green
tegra-335.tegra.releng.scl3.mozilla.com is alive <- up and green

as per jhopkins last week and reformatting: again the non green ones should be followed up on.
tegra-108 - bug 838425 - SD card reformat was successful <- cant write to sd
tegra-091 - bug 778886 - SD card reformat was successful <- sdcard issues again
tegra-073 - bug 771560 - SD card reformat was successful <- lockfile issues
tegra-210 - bug 890337 - SD card reformat was successful <- green in prod
tegra-129 - bug 838438 - SD card reformat was successful <- fail to connect to telnet
tegra-041 - bug 778813 - SD card reformat was successful <- sdcard issues again
tegra-035 - bug 772189 - SD card reformat was successful <- sdcard issues again
tegra-228 - bug 740440 - SD card reformat was successful <- fail to connect to telnet
tegra-133 - bug 778923 - SD card reformat was successful <- green in prod
tegra-223 - bug 740438 - SD card reformat was successful <- Unable to properly cleanup foopy processes
tegra-080 - bug 740426 - SD card reformat was successful <- green in prod
tegra-032 - bug 778899 - SD card reformat was successful <- sdcard issues again
tegra-047 - bug 778909 - SD card reformat was successful have not got past here
tegra-038 - bug 873677 - SD card reformat was successful
tegra-264 - bug 778841 - SD card reformat was successful
tegra-092 - bug 750835 - SD card reformat was successful
tegra-293 - bug 819669 - SD card reformat was successful

Week of 2014-03-31 to 2014-04-04

Wednesday:

Someone will need to follow up on how these tegras did since I reformatted their SD cards:

tegra-108 - bug 838425 - SD card reformat was successful
tegra-091 - bug 778886 - SD card reformat was successful
tegra-073 - bug 771560 - SD card reformat was successful
tegra-210 - bug 890337 - SD card reformat was successful
tegra-129 - bug 838438 - SD card reformat was successful
tegra-041 - bug 778813 - SD card reformat was successful
tegra-035 - bug 772189 - SD card reformat was successful
tegra-228 - bug 740440 - SD card reformat was successful
tegra-133 - bug 778923 - SD card reformat was successful
tegra-223 - bug 740438 - SD card reformat was successful
tegra-080 - bug 740426 - SD card reformat was successful
tegra-032 - bug 778899 - SD card reformat was successful
tegra-047 - bug 778909 - SD card reformat was successful
tegra-038 - bug 873677 - SD card reformat was successful
tegra-264 - bug 778841 - SD card reformat was successful
tegra-092 - bug 750835 - SD card reformat was successful
tegra-293 - bug 819669 - SD card reformat was successful


Week of 2014-03-17 to 2014-03-21
buildduty: armenzg

Monday

  • bugmail and deal with broken slaves
  • mergeday

Tuesday

  • reviewed aws sanity check
  • cleaned up and assigned some buildduty bugs
  • reconfig

TODO:

  • bug 984944
  • swipe through problem tracking bugs

Wednesday

Thursday

Friday

Week of 2014-01-20 to 2014-01-24
buildduty: armenzg

Monday

  • deal with space warnings
  • loan to dminor
  • terminated returned loan machines

Tuesday

  • loan win64 builder
  • Callek helped with the tegras

TODO

  • add more EC2 machines


Week of 2014-01-20 to 2014-01-24
buildduty: jhopkins

Bugs filed:

  • Bug 962269 (dupe) - DownloadFile step does not retry status 503 (server too busy)
  • Bug 962698 - Expose aws sanity report data via web interface in json format
  • Bug 963267 - aws_watch_pending.py should avoid region/instance combinations that lack capacity


Monday

  • Nightly updates are disabled (bug 908134 comment 51)
  • loan bug 961765


Tuesday


Wednesday

  • added AWS instance Tag "moz-used-by" to the nat-gateway instance to help with processing the long-running instances report
  • would be nice if we could get the aws sanity report data to be produced by slave api so it could be pulled by a web page and correlated with recent job history, for example
  • Bug 934938 - jakem switched to round-robin DNS (see bug 934938#c1519 for technical details) to avoid "thundering herd" problem.


Thursday

  • AWS lacking capacity and slowing down instance startup. Filed 963267.


Friday

  • missed some loan requests b/c I thought they were being included in the buildduty report (2 previous ones seemed to be). Can we add loans to the buildduty report?
  • some automated slave recovery not happening due to Bug 963171'- please allow buildbot-master65 to talk to production slaveapi



Week of 2014-01-13 to 2014-01-17
buildduty: bhearsum

Bugs filed (not a complete list):

  • Bug 960535 - Increase bouncerlatestchecks Nagios script timeout


Week of 2014-01-16 to 2014-01-10
buildduty: armenzg

Bugs filed:

Monday

  • loan machines
  • deal with some broken slaves

Tuesday

  • loan machines
  • deal with some broken slaves
  • reconfig
  • second reconfig for backout

Wednesday

  • enable VNC for a Mac loaner
  • check signing issues filed by Tomcat
  • mozharness merge
  • help RyanVM with some timeout

Thursday

  • do reconfig with jlund

Friday

  • restart redis
  • loan 2 machines
  • process problem tracking bugs

Week of 2013-12-16 to 2013-12-20
buildduty: jhopkins

Bugs filed:

  • Bug 950746 - Log aws_watch_pending.py operations to a machine-parseable log or database
  • Bug 950780 - Start AWS instances in parallel
  • Bug 950789 - MozpoolException should be retried
  • Bug 952129 - download_props step can hang indefinitely

* Bug 952517 - Run l10n repacks on a smaller EC2 instance type

Monday

  • several talos-r3-fed* machines have a date of 2001
  • adding hover-events to our slave health pages would be helpful to get quick access to recent job history
  • other interesting possibilities:
    • a page showing last 50-100 jobs for all slaves in a class
    • ability to filter on a certain builder to spot patterns/anomalies. eg. "robocop tests always fail on this slave but not the other slaves"

Wednesday

  • taking over 15 minutes for some changes to show as 'pending' in tbpl. Delay from scheduler master seeing the change in twistd.log
  • Bug 951558 - buildapi-web2 RabbitMQ queue is high

Friday

  • Bug 952448 - Integration Trees closed, high number of pending linux compile jobs
    • AWS instance 'start' requests returning "Error starting instances - insufficient capacity"
  • dustin has fixed Bug 951558 - buildapi-web2 RabbitMQ queue is high

Week of 2013-11-04 to 2013-11-08
buildduty: jhopkins

Tuesday

  • rev2 migrations going fairly smoothly (biggest issue is some IPMI interfaces being down and requiring a power cycle by DCOPs)

Wednesday

  • needs attention (per RyanVM): Bug 935246 - Graphserver doesn't know how to handle the talos results from non-PGO builds on the B2G release branches
  • IT's monitoring rollout happening today
  • request to build "B2G device image nightlies" non-obvious what the builders are or what masters they live on. No howto I could find. How do we automate this and keep it

Friday

  • RyanVM reports that pushing mozilla-beta to Try is a fairly normal thing to do but fails on the win64-rev2 build slaves. He has been helping with backporting the fixes in User:Jhopkins/win64rev2Uplift to get this addressed.
  • catlee's buildbot checkconfig improvement went into production but we need a restart on all the masters to get the full benefit. no urgency, however.

Week of 2013-10-14 to 2013-10-18
buildduty: armenzg

Monday

  • uploaded mozprocess
  • landed puppet change that made all Linux64 hosts get libvirt-bin get installed and made them fall to sync with puppet
    • I had to back out and land a patch to uninstall the package
    • we don't know why it got installed
  • redis issues
    • build-4hr issues
    • signing issues

Tuesday

  • returned some slaves to the pool
  • investigated some cronmail
  • uploaded talos.zip
  • reclaimed machines and requested reimages

Wednesday

  • put machines back into produciton
  • loan
  • process delays email


Week of 2013-10-14 to 2013-10-18

buildduty: coop (callek on monday)

Monday

Tuesday

  • meetings! At least one of them was about buildduty.

Wednesday

  • shutdown long-running AWS instance for hverschore: bug 910368
  • investigating disabled mtnlion slaves
    • many needed the next step taken: filing the IT bug for recovery

Thursday
*filed Bug 927951 - Request for smoketesting Windows builds from the cedar branch

  • filed Bug 927941 - Disable IRC alerts for issues with individual slaves
  • reconfig for new Windows in-house build master

Friday

Week of 2013-09-23 to 2013-09-27

buildduty: armenzg

Monday

  • help marcia debug some b2g nightly questions
  • meetings and meetings and distractions
  • started patches to transfer 11 win64 hosts to become try ones

Tuesday

  • run a reconfig
  • did a backout for buildbotcustom and run another reconfig
  • started work on moving win64 hosts from build pool to the try pool
  • analyzed a bug filed by sheriff wrt to clobberer
    • no need for buildduty to fix (moved to Platform Support)
    • asked people's input for proper fix
  • reviewed some patches for jmaher and kmoir
  • assist edmorley with clobberer issue
  • assist edmorley with git.m.o issue
  • assist RyanVM with git.m.o issue
  • put w64-ix-slave64 in the production pool
  • updated buildduty wiki page
  • updated wiki page to move machines from one pool to another

Wednesday

  • messy

Thursday

  • messy

Monday

  • messy

Week of 2013-09-16 to 2013-09-20

buildduty: Callek

Monday:

  • (pmoore) batch 1 of watcher update [Bug 914302]
  • MERGE DAY
    • We missed having a point person for merge day again, rectified (thanks armen/rail/aki)
  • 3-reconfigs or so,
    • Merge day
    • Attempted to fix talos-mozharness (broken by panda-mozharness landing)
    • Backout out talos-mozharness change for continued bustaged
    • Also backed out emulator-ics for in-tree (crossing-tree) bustage relating to name change.
      • Thanks to aki for helping while I had to slip out for a few minutes
  • Loaner bug poking/assigning
  • Did one high priority loaner needed for tree-closure which blocked MERGE DAY

Tuesday:

  • (pmoore) batch 2 of watcher update [Bug 914302]
  • Buildduty Meeting
  • Bug queue churn-through
  • Hgweb OOM: Bug 917668

Wednesday:

  • (pmoore) batch 3 of watcher update [Bug 914302]
  • Reconfig
  • Hgweb OOM continues (IT downtimed it, bkero is on PTO today, no easy answer)
    • Very Low visible tree impact at present
  • Bug queue churn-through
  • Discovered last-job-per-slave view of slave_health is out of date.
  • Discovered reboot-history is either out of date or reboots not running for tegras

Thursday

  • (pmoore) batch 4 [final] of watcher update [Bug 914302]
  • Hgweb OOM continues
  • Bug queue churn... focus on tegras today
  • Coop fixed last-job-per-slave generation, and slaves_needing_reboots
  • Downtime (and problems) for scl1 nameserver and scl3 zeus nodes
    • Caused tree closure due to buildapi01 being in scl1 and long delay


Week of 2013-09-09 to 2013-09-13

buildduty: coop

Monday:

  • meetings
  • kittenherder hanging on bld-centos - why?
    • multiple processes running, killed off (not sure if that's root cause)

Tuesday:

  • buildduty meeting
    • filed bug 913606 to stop running cronjobs to populate mobile-dashboard
  • wrote reboot_tegras.py quick-n-dirty script to kill buildbot processes and reboot tegras listed as hung

Wednesday:

  • re-enabled kittenherder rebooting of tegras
  • wrote bugzilla shared queries for releng-buildduty, releng-buildduty-tagged, and releng-buildduty-triage
  • playing with bztools/bzrest to try to get query that considers dependent bugs
  • meetings
  • deploying registry change for bug 897768 (fuzzing dumps)

Thursday:

  • broke up kittenherder rebooting of tegras into 4 batches to improve turnaround time
  • got basic buildduty query working with bztools
  • respun Android nightly
  • resurrecting as many Mac testers as possible to deal with load
  • filed bug 915766 to audit pdu2.r102-1.build.scl1
  • resurrected a bunch of talos-r3-fed machines that were not running buildbot

Friday:

  • AWS US-East-1 outage
  • reconfig for nexus4 changes
    • re-reconfig to backout kmoir's changes that closed the tree: bug 829211
  • Mac tester capacity

Week of 2013-09-02 to 2013-09-06
buildduty: bhearsum

Monday

  • US/Canada holiday


Tuesday

  • Bug 912225 - Intermittent B2G emulator image "command timed out: 14400 seconds elapsed, attempting to kill" or "command timed out: 3600 seconds without output, attempting to kill" during the upload step
    • Worked around by lowering priority of use1. Acute issue is fixed, we suspect there's still symptoms from time to time.
    • Windows disconnects may be the early warning sign.


Week of 2013-08-26 to 2013-08-30
buildduty: jhopkins

Monday

  • many talos-r3-w7 slaves have a broken session which prevents a new SSH login session (you can authenticate but it kicks you out right away). Needed to RDP in, open a terminal as Administrator, and delete the files in c:\program files\kts\log\ip-ban\* and active-sessions\*.
  • many other talos-r3-w7 slaves had just ip-ban\* files (no active-sessions\* files) which prevented kittenherder from managing the slave, since there are no IPMI or PDU mechanisms to manage these build slaves.
  • trying slaveapi
    • IPMI reboot failing (no netflow)


$curl -dwaittime=60 http://cruncher.srv.releng.scl3.mozilla.com:8000/slave/talos-r3-xp-076/action/reboot
{

"requestid": 46889168, 

"state": 0, 

"text": ""

}

  • slavealloc-managed devices are live (per Callek)
  • ec2 slave loan to eflores (909186)


Tuesday


Wednesday


Thursday

  • filed Bug 910818 - Please investigate cause of network disconnects 2013-08-29 10:22-10:24 Pacific
    • need to automate gathering of details for filing this type of bug
  • Bug 910662 - B2G Leo device image builds broken with "error: patch failed: include/hardware/hwcomposer.h:169" during application of B2G patches to android source


Friday



Week of 2013-08-19 to 2013-08-23
buildduty: armenzg

Monday

  • 40+ Windows builders had not been rebooted for several days
    • bug 906660
    • rebooted a bunch with csshX
    • I gave a couple to jhopkins to look into
      • cruncher was banned from being able to ssh
      • ipmi said that it was successful
      • more investigation happening
  • edmorley requested that I look into fixing the TMPDIR removal issue
    • bug 880003
    • ted to land a fix to disable the test that causes
    • filed a bug for IT to clean up the TMPDIR through puppet
    • cleaned up tmpdir manually on 2 hosts and put them back in production to check
  • do a reconfig for mihneadb
  • rebooted talos-r4-lion-041 upon philor's request due to hdutil
  • promote unagi
  • upload talos.zip for mobile


Tuesday

  • I see a bunch of these:
    • nagios-releng: Tue 06:04:58 PDT [4193] buildbot-master92.srv.releng.use1.mozilla.com:ntp time is CRITICAL: CHECK_NRPE: Socket timeout after 15 seconds. (http://m.allizom.org/ntp+time)
    • bug 907158
    • We disabled all use1 masters
    • We reduced use1's priority for host generation
    • Fixed Queue dirs for one of the masters by restarting the init service
      • had to kill the previous pid
  • rebooted remaining win64 machines
  • rebooted remaining bld-lion-r5 machines
  • buildapi issues
    • Callek took care of it and move it to Tools
  • Queue dir issues
    • it seems that the Amazon issues caused this
    • I have been stopping and starting the pulse_publisher
    • /etc/initd.d/pulse_publisher {stop|start}
    • the /dev/shm/queue/pulse/new will start decreasing
  • Re-enabled all aws-us-es-1 masters


Wednesday

  • one of the b2g repos has a 404 bundle and intermittent ISE 500
    • bug 907693
    • bhearsum has moved it to IT
    • fubar is looking into it
  • done a reconfig
    • graceful restart for buildbot-master69
      • I had to do a less graceful restart
    • graceful restart for buildbot-master70
    • graceful restart for buildbot-master71
  • loan t-xp32-ix006 in https://bugzil.la/904219
  • promoted b2g build
  • deploy graphs change for jmaher
  • vcs major alert
    • hwine to look into it


Thursday

  • disable production-opsi
  • reconfig
  • trying to cleanup nagios


Friday

  • a bunch of win64 machines are not taking jobs
    • deploy to all machines the fix
    • filed bug for IT to add to task sequence
  • some win64 imaging bugs filed
  • reconfig for hwine
  • Callek deployed his slavealloc change
  • merged mozharness
  • investigated a bunch of hosts that were down
  • we might be having some DNS tree-closing issues



Week of 2013-08-12 to 2013-08-16

buildduty: coop

Monday:


Tuesday:


Wednesday:

  • fixed buildfaster query to cover helix builds
  • set aside talos-r4-lion-001 for dustin: bug 902903#c16
  • promoted unagi build for dogfood: 20130812041203
  • closed bug 891880
  • investigated talos failures affecting (primarily) lion slaves: bug 739089


Thursday:


Friday:

  • investigation into bug 905350
    • basedir wrong in slavealloc
  • reconfigs for catlee, aki



Issues/Questions:

  • many/most of the tree closure reasons on treestatus don't have bug#'s. should we encourage sheriffs to enter bug#'s so others can follow along more easily?
  • uncertainty around the difference between mozpool-managed and non-mozpool-managed pandas. How do I take one offline - do they both use disabled.flg? A: yes, all pandas use disabled.flg on the foopy
    • when is it ok to use Lifeguard to for the state to "disabled"? per dustin: [it's] for testing, and working around issues like pandas that are still managed by old releng stuff. it's to save us loading up mysql and writing UPDATE queries
      • what's "old releng stuff"?
  • Windows test slaves
  • another example of PDU reboot not working correctly: bug 737408#c5 and bug 885969#c5. We need to automate power off,pause,power on to increase reliability.

Bustages:

  • bug 900273 landed and backed out
Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/Buildduty/StandupMeetingNotesQ12015&oldid=1066245"