Confirm, emeritus
101
edits
Changes
→IT
* Sunday 7/1 at 00:00 UTC we experienced a leap second that caused problems with any of our RHEL6 servers that were running java applications. This affected Zimbra, Elastic Search for many websites, HBase for crash-stats, and unrelated to java some mysql servers were also affected. Downtime to any single service was minimal as the servers were just experiencing high load and slowed down. This was covered in the following bugs, 769971 769973 769972 and in [http://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/ this IT blog post].
** The Mozilla bugs and our IT blog post have been referenced in numerous other write-ups on the issue, including [http://arstechnica.com/business/2012/07/one-day-later-the-leap-second-v-the-internet-scorecard/ Ars Technica] and [http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/ Wired], and we've seen tons of tweets and retweets about it.
** <b>huge thanks to the individuals who were up working this issue late Saturday night: aj, bear, cshields, cyliang, deinspanjer, ericz, fox2mike, gcox, justdave, lars, laura, lerxst, pedro, phong, sheeri, solarce, tmary</b>
** we pushed out a 'fix' to the entire infrastructure to catch any issues we were unaware of. That said, if there are any issues that you notice that may be related, please file an [https://bugzilla.mozilla.org/form:itrequest IT bug].
