View source for CiDuty/actionable

== Intro ==
CiDuty is responsible for routine keep-the-lights-on (KTLO) waterline tasks. These include but are not limited to providing loaners to developers, triaging urgent break-fix issues, handling trees closures, and maintaining documentation. If any of these come up early in CiDuty morning, they take priority over anything else. CiDuty should continue performing checks on a regular basis throughout the day to make sure things are in a good state. 

== Continuously ==
'''1.''' Nagios and SNS Alerts - Monitor alerts from [https://nagios1.private.releng.mdc1.mozilla.com/releng-mdc1/ MDC1] / [https://nagios1.private.releng.mdc2.mozilla.com/releng-mdc2/ MDC2] Nagios instances and SNS alerts from [https://papertrailapp.com/dashboard/ papertrail] in the #ci IRC channel. Triage unacknowledged alerts and file/fix bugs as necessary according to the [https://wiki.mozilla.org/CIDuty/How_To CiDuty How-Tos]. Make sure that all CiDuty bugs have the correct priority set according to the [[ReleaseEngineering/Buildduty_actionable#Buildduty_Bugzilla_Priority_levels| priority list]] below. A few examples of such alerts include:
* High CI wait times
* CI pending job backlogs
* Unresponsive machines
* Disk/RAM/CPU issues
* Failed processes

'''2.''' Monitor the #ci, #platform-ops-alerts, #releaseduty and #taskcluster irc channels for requests/questions from developers and other ops teams.

'''3.''' Bug triaging - The [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=CiDuty&list_id=14513240 CiDuty report] should be your starting point for bug triaging. Click on the bug number link in the CiDuty report to see the specified bug in bugzilla.

== Daily ==
'''1.''' Review email from AWS looking for announced maintenance, degraded instances, etc. Resolve any issues with instances specified in the AWS email, and notify the appropriate groups of any planned maintenance.

'''2.''' Check for and terminate long-running/outdated AWS instances.

'''3.''' Sanity checks in [https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/ TaskCluster Provisioner]making sure that the workers are working as expected.

== Weekly ==
'''1.''' Review AWS instances. All those AWS that have 'Unknown State/Type' or have been 'stopped for a while'.
* check the AWS sanity logs in [https://papertrailapp.com/dashboard/ papertrail]
* for each host under heading "''[https://papertrailapp.com/systems/74950614/events?q=Unknown%20State Unknown State]''", "''[https://papertrailapp.com/systems/74950614/events?q=Unknown%20Type Unknown Type]''", follow this [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves#Unknown_Type_Or_State_Instances/ guide]
* for each host under heading "''[https://papertrailapp.com/systems/74950614/events?q=Stopped%20For%20A%20While Stopped For A While]''", follow this [https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_AWS_slaves#Stopped_For_A_While_Instances/ guide] 

'''2.''' Run the aws_manage_routingtables script to make sure that the AWS routing tables are up to date and error free. Fix any reported errors

'''3.''' Review the loaner email report and verify that loaners are still required. Reimage/terminate returned loaners as necessary

== CiDuty Bugzilla Priority levels ==

'''P1:''' Waterline KTLO work.
This includes developer loaners, urgent break-fix, trees closures, etc.

'''P2:''' Projects above waterline that take high priority and have hard deadlines.
This includes various fixes that we need to take care of once, not on a recurring basis. Examples of this are [https://bugzilla.mozilla.org/show_bug.cgi?id=1363897/ 1363897]  (legacy extensions disabling prior to 57) and [https://bugzilla.mozilla.org/show_bug.cgi?id=1315977/ 1315977]  (upgrading python to 2.7.6 on mac builders)

'''P3.''' Training/Daily Documentation.
This includes onboarding and runbook documentation. Creating/expanding this documentation will help us deliver faster on other bugs.

'''P4.''' Operational development.
This includes adding new scripts or improving existing scripts, e.g. to reduce running times, improve logging and notifications system, etc.