CIDuty: Difference between revisions

CIDuty (view source)

265 bytes removed , 27 May 2015

canmove, Confirmed users

2,850

edits

@@ Line 51: / Line 51: @@
 === Semi-Daily ===
 * '''Reconfigs'''
-** Run [[ReleaseEngineering/Buildduty/Reconfigs|reconfigs]] (every day or two days) for other relengers
+** Run [[ReleaseEngineering/Buildduty/Reconfigs|reconfigs]] (every day or two days) for other release engineers. Reconfigs are automated now, so this is usually just a matter of merging the relevant repositories to production.
-*'''Review "long running" and "lazy" AWS instances'''
+* '''Review logs in papertrail'''
-** ''When'': 2-3 times a week (eg: Mondays or after weekends/holidays, Wednesdays, and Fridays)
+** We have SNS alerts for most known issues. The alerts will appear in the #buildduty IRC channel. Buildduty should log into papertrail a few times a week (M/W/F) to check for new issues that are *not* covered by alerts. New alerts should be setup for these issues.
-** ''How'': use aws sanity check email (sent daily):
-*** Email filter => to: release+aws-sanity-check@mozilla.com, subject: [cron] aws sanity check
-*** for each host under heading "Long running instances", follow steps in dealing with [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves#Long_Running_Instances long running instances]
-*** for each host under heading "Lazy long running instances", figure out why they're still up and not taking jobs
-**** twistd.log, uptime, reboot history on the slave health page et al
 === Weekly ===