- Smoke, fire
- Improving back-off for smoke, fire
- Partial views/sync
- 450 inserts/sec this morning, 4x normal load, replication delay is ~ 6hours and worsening, but may be cresting.
- 0.6 causes full sync, monday morning probably caused a bunch of 0.6 updates.
- we're getting far more selects for 0.6.
- We are running slow, but we are being consistent.
- We are only seeing 100 inserts/sec.
- We are not sending 503s.
- Potential cures:
- Bring up a new cluster, ASAP.
- Seeing a lot more selects doesn't have any explanation.
- AI: Zandr will put a rush order on getting the new cluster up as soon as possible. Probably done by tomorrow morning.
- AI: We need a better plan for handling escalations like this in the future.
- AI: Let's set up a brainstorming/post-mortem session.
- We won't be doing l10n for 0.7, UI is still in flux. We will need to communicate this out to sethb.
- Concentrate on partial view/sync and about:weave.
- We need to take out the identity pieces or mark them as experimental. MConnor is a bit concerned that if we keep the bits in, they will be a distraction. But at the same time, we also don't want to stop shipping features (even if experimental) that users like.
- MConnor has a patch, but the topic is large enough that we need a WEP.
- AI: MConnor will author a WEP, using MConnor's patch as a baseline, but also include other cases. Dan will send him email with list of stuff.