View source for IT/Maintenance/Quarterly

{| class="fullwidth-table"
|-
! Document Status
! Last Reviewed
|-
| <font color=red>DRAFT</font> || Aug 31 2012
|}

= Introduction =
To underline the procedures involved in upgrading Red Hat servers managed by IT at Mozilla.

We will refer to “full system upgrades” (that is, yum update), unless otherwise mentioned. Some group of servers require special attention and a full system upgrade is not recommended. This document will also apply to the firmware upgrades that require a system reboot (BIOS and RAID controller firmware upgrades).

= When =
Preparations, as defined in this document, should start no later than the first two weeks of a new quarter. The desired goal is to have all the systems upgraded within the first 8 weeks at maximum of the quarter.

= What = 
We will use inventory ( https://inventory.mozilla.org ) to get the list of the servers. The list of servers will be generated within the first two weeks of each quarter.
Systems that are marked in the inventory with the following status

* production
* building

will be added in a spreadsheet. The spreadsheet will have tabs for each datacenter/office location. Hostnames will have associated the following:

* owner (DBAs, WebOps, AMO, etc)
* full system upgrade? (Y/N). If “N”, it will be assumed just kernel upgrade is OK. If none are permitted, it should be noted in the “other” column.
* can be rebooted at any time? (Y/N)
* date and time for reboot - applicable if previous answer was “N”.
* other - specific notes and observations


= How and who = 
Mozilla SRE will be responsible for enforcing the quarterly upgrades.
If there is at least one “N” answer to the two questions in the spreadsheet, they will coordinate with the owners for what and when the upgrade can be performed.

For systems that are part of clusters, run a yum update and pastebin so that the owners have a look. This is to prevent accidentally upgrades of packages that are not supposed to be updated. If such packages are found, work with those system owners to have the excludes managed by puppet in /etc/yum.conf. One example is redis package on redisX.addons.phx1.
Sometimes, if only a kernel upgrade is permitted, check with Security Assurance/bug queue if there are other packages that need to be upgraded. For example, at the time this document was written, an openssl upgrade is advised in bug 747471. It will be applied along with the kernel upgrade.
If there are systems where not even a kernel upgrade is possible, let Security Assurance know about this.

The first round of upgrades will be performed on servers that serve as dev/stage environments. Wait one week before upgrading production and carefully monitor those servers performance.
For production services that are part of clusters, upgrades will be rolled in chunks.
Example: if there are 20 nodes in a cluster, upgrade a third of them, wait a couple of days, then upgrade the rest.

= Announcements = 
Mozilla SRE is also responsible to announce the work. Follow https://wiki.mozilla.org/IT/Maintenance#Communications_.26_Notifications for instructions.
Special case: HCI, which is described here: https://wiki.mozilla.org/IT/Maintenance/HCI