BMO/Meetings/2014-03-11

Instrumentation waiting on installation of Perl module.

Memcache deployed; no noticeable difference in timings. :( We can still extend it further; gozer has been helping with analysis.

DST-related errors over the weekend. 21000 errors thrown regarding invalid timestamps. Issues lingered to Monday; got stack traces but they aren't helpful. Need to experiment with VMs to reproduce and understand these errors. No long-term problems.

Attachment ID crisis yesterday. Background: in November 2013 there was a trial fallback to PHX; both were accidentally live at the same time briefly, resulting in duplicate IDs. Had to fix manually, and there was typo resulting in the attachment ID autoinc being increased by a factor of 10. Thus we hit max much earlier than expected, although we still should have caught it earlier. Fix (moving to larger int type) took a long time, caused timeouts and other problems, including flags assigned to the wrong attachments, which required manual fixes. At the same time, we updated other columns where IDs were nearing maxes.

On-going replication lag. Main problem is using MyISAM instead of InnoDB (couldn't do full-text searches on InnoDB until recently). Full-text searches on MyISAM cause locks; huge searches locked for a long time, causing lag in replication. Moving to InnoDB soon. Also going to add new slave for long-running queries and for polling systems (ETL, push notifications).

Yahoo employee wants to unofficially mentor GSoC YUI3 project. Can put in 3 hours/week to help.

Git migration scheduled for 1:00 pm EDT today. After it completes, git.mozilla.org will host the repositories of record.