WeeklyUpdates/2012-07-02: Difference between revisions

Jump to navigation Jump to search
Line 132: Line 132:


* Sunday 7/1 at 00:00 UTC we experienced a leap second that caused problems with any of our RHEL6 servers that were running java applications. This affected Zimbra, Elastic Search for many websites, HBase for crash-stats, and unrelated to java some mysql servers were also affected.  Downtime to any single service was minimal as the servers were just experiencing high load and slowed down.  This was covered in the following bugs, 769971 769973 769972 and in [http://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/ this IT blog post].  
* Sunday 7/1 at 00:00 UTC we experienced a leap second that caused problems with any of our RHEL6 servers that were running java applications. This affected Zimbra, Elastic Search for many websites, HBase for crash-stats, and unrelated to java some mysql servers were also affected.  Downtime to any single service was minimal as the servers were just experiencing high load and slowed down.  This was covered in the following bugs, 769971 769973 769972 and in [http://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/ this IT blog post].  
** The Mozilla bugs and our IT blog post have been referenced in numerous other write-ups on the issue, including [http://arstechnica.com/business/2012/07/one-day-later-the-leap-second-v-the-internet-scorecard/ Ars Technica] and [http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/ Wired], and we've seen tons of tweets and retweets about it.
** <b>huge thanks to the individuals who were up working this issue late Saturday night: aj, bear, cshields, cyliang, deinspanjer, ericz, fox2mike, gcox, justdave, lars, laura, lerxst, pedro, phong, sheeri, solarce, tmary</b>
** <b>huge thanks to the individuals who were up working this issue late Saturday night: aj, bear, cshields, cyliang, deinspanjer, ericz, fox2mike, gcox, justdave, lars, laura, lerxst, pedro, phong, sheeri, solarce, tmary</b>
** we pushed out a 'fix' to the entire infrastructure to catch any issues we were unaware of.  That said, if there are any issues that you notice that may be related, please file an [https://bugzilla.mozilla.org/form:itrequest IT bug].
** we pushed out a 'fix' to the entire infrastructure to catch any issues we were unaware of.  That said, if there are any issues that you notice that may be related, please file an [https://bugzilla.mozilla.org/form:itrequest IT bug].
Confirmed users, Bureaucrats and Sysops emeriti
101

edits

Navigation menu