Schedule / Location / Call Information

Overview

HBase data sync not complete and verified until just a few days before the migration
Various issues through the last week with network configuration issues: Zeus slowness, bonding setup etc.
Getting backlog in was going to take a long time. We corrected our approach on Sunday night (and finished back processing before Monday a.m.), but we should have acted on this earlier.
Difficulties getting correct ADUS due to an unrelated problem with Vertica; poor timing here (SJC was broken in the same way as PHX)
Missed a cron job despite multiple audits (signatureProductDims)
Upgrade to RHEL6 coincided with hardware/software architecture change.
Checklist could have been followed more closely

Better communication with Netops:
- get a unified list of requested changes in order before requesting change
- get notice of concurrent changes in the network environment
Better communication to groups outside release-drivers. Actions bug 628318:
- Get Socorro on status.mozilla.org
- Set up a Socorro specific blog for maintenance/downtime information
- devs and ops have different bugzilla workflow (discussion versus unit-of-work action)
- file ops bugs as clear action items, spun off from dev bug or tracking bug
need more insight into network/systems changes
- cannot keep everyone apprised of every change
  - maybe better tools can help here
Need better coordination of all staff with timeline
- Need better tools than bugzilla + google spreadsheets -- a professional project management tool is called for
  - Mozilla is aware of and working on this, agreement that Bugzilla is not ideal
- Need to have a manager in charge for every day of the timeline, even when people are on vacation
- If we ever have a project this big again, consider hiring a professional project manager consultant.