ReleaseEngineering/Buildduty/SVMeetings/Sept28-Oct2

From MozillaWiki
Jump to: navigation, search

2015-09-28 - 2015-09-29

1. Started again the process of testing the jobs on yosemite-r7 , more details : http://dev-master2.bb.releng.use1.mozilla.com:8050/builders .

list of jobs that are failing 

mozilla-central talos dromaeojs (only on t-yosemite-r7-0001)

mozilla-central opt test gtest (on both of them t-yosemite-r7-0001 and t-yosemite-r5-0107) mozilla-central debug test gtest (only on t-yosemite-r5-0107) list of jobs with warnings: mozilla-central opt test web-platform-tests-2 mozilla-central opt test web-platform-tests-3 mozilla-central debug test marionette mozilla-central debug test mochitest-devtools-chrome-3 mozilla-central debug test web-platform-tests-7 mozilla-central debug test web-platform-tests-reftests

2. https://bugzilla.mozilla.org/show_bug.cgi?id=1190868

[alin]

The current state of the jobs that were run on http://dev-master2.bb.releng.use1.mozilla.com:8096 :

Failing jobs: 

   	-> mozilla-central opt test gtest
   	-> mozilla-central debug test gtest

Both of these are known failing jobs.

Jobs that finish with warnings: 

   	-> mozilla-central opt test mochitest-devtools-chrome-2
   	-> mozilla-central opt test mochitest-gl
   	-> mozilla-central opt test web-platform-tests-2
   	-> mozilla-central opt test web-platform-tests-3
   	-> mozilla-central opt test web-platform-tests-4
   	-> mozilla-central opt test web-platform-tests-reftests
   	-> mozilla-central debug test marionette
   	-> mozilla-central debug test mochitest-2
   	-> mozilla-central debug test mochitest-devtools-chrome-3
   	-> mozilla-central debug test mochitest-gl
   	-> mozilla-central debug test web-platform-tests-2
   	-> mozilla-central debug test web-platform-tests-3
   	-> mozilla-central debug test web-platform-tests-5
   	-> mozilla-central debug test web-platform-tests-7
   	-> mozilla-central debug test web-platform-tests-8

All the above jobs end with messages like the following: 

   INFO -  (blobuploader) - INFO - Uploading /builds/slave/test/build/blobber_upload_dir/mozilla-test-fail-screenshot_jhNkcL.png ...

INFO - (blobuploader) - INFO - Using https://blobupload.elasticbeanstalk.com INFO - (blobuploader) - INFO - Uploading, attempt #1. INFO - (blobuploader) - CRITICAL - Authentication failed! INFO - (blobuploader) - INFO - Blobserver returned 403, bailing... INFO - (blobuploader) - INFO - Done attempting.

--> invoked senchanges for one more revision: e8209bc90420340f73f58aeee2fe3a7e7377021c --> stopped the master, cleaned the pending jobs, updated Buildbot.py with new blobber credentials on my master and started the master again. Waiting to see the way the problematic jobs complete after this.

UPDATE: seems like we still have orange jobs + a couple of red jobs that did not fail before. Migrated all the slaves to the first master (8095), updated only Buildbot.py and invoked senchanges. Let's see how things go in this case.

2015-09-30 - 2015-10-01

copied from https://etherpad.mozilla.org/buildduty-notes


[alin]: link to the test master: http://dev-master2.bb.releng.use1.mozilla.com:8095/builders

repeated the tests during yesterday on the first master → got some errors on jobs that previously worked fine today I re-imaged all the slaves, removed the pending jobs from the master and invoked sendchanges in the same manner as show in the logs: → bin/buildbot sendchange --master dev-master2.bb.releng.use1.mozilla.com:9095 --username sendchange --branch mozilla-central-macosx64-talos -r e8209bc90420340f73f58aeee2fe3a7e7377021c --username cbook@mozilla.com --property buildid:20150929025224 --property pgo_build:False --property builduid:9abaeb45572c44d1b6a9240559dc0fe1 https://queue.taskcluster.net/v1/task/jlpVhNH_SdKuOE7BOpkKew/artifacts/public/build/firefox-44.0a1.en-US.mac.dmg

→ bin/buildbot sendchange --master dev-master2.bb.releng.use1.mozilla.com:9095 --username sendchange-unittest --branch mozilla-central-macosx64-opt-unittest -r e8209bc90420340f73f58aeee2fe3a7e7377021c --username cbook@mozilla.com --property buildid:20150929025224 --property pgo_build:False --property builduid:9abaeb45572c44d1b6a9240559dc0fe1 https://queue.taskcluster.net/v1/task/jlpVhNH_SdKuOE7BOpkKew/artifacts/public/build/firefox-44.0a1.en-US.mac.dmg https://queue.taskcluster.net/v1/task/jlpVhNH_SdKuOE7BOpkKew/artifacts/public/build/test_packages.json

→ bin/buildbot sendchange --master dev-master2.bb.releng.use1.mozilla.com:9095 --username sendchange-unittest --branch mozilla-central-macosx64-debug-unittest -r e8209bc90420340f73f58aeee2fe3a7e7377021c --username cbook@mozilla.com --property buildid:20150929025224 --property pgo_build:False --property builduid:9abaeb45572c44d1b6a9240559dc0fe1 https://queue.taskcluster.net/v1/task/PuefQGErQ9C5N5JTjtLXJg/artifacts/public/build/firefox-44.0a1.en-US.mac64.dmg https://queue.taskcluster.net/v1/task/PuefQGErQ9C5N5JTjtLXJg/artifacts/public/build/test_packages.json most web-platform jobs (opt+debug), but also some mochitest and marionette jobs still finish with warnings

kmoir How are the patches coming along to run on trunk? https://bugzilla.mozilla.org/show_bug.cgi?id=1203128 file new bug for greening up 10.10.5 tests needinfo jmaher get r7 machines running in try

[alin] https://bugzilla.mozilla.org/show_bug.cgi?id=1203128 Coop suggested this bug as a starting point: https://bugzilla.mozilla.org/show_bug.cgi?id=1122039 added a preliminary version of the patch (updated config.py from bb) need some suggestions for the next steps

Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/Buildduty/SVMeetings/Sept28-Oct2&oldid=1099145"