There is no official spec for text-overflow. The notes in the thread starting here are probably the best reference we have: http://lists.w3.org/Archives/Public/www-style/2008Aug/0006.html
See also discussion here: http://lists.w3.org/Archives/Public/www-style/2009Nov/0219.html (including replies)
text-overflow can and should be implemented as something that happens at paint time only, and does not affect reflow at all.
My idea is to put the text-overflow logic mainly in nsBlockFrame::BuildDisplayList. After building display list items for all the block's children, if text-overflow is set, figure out what the clipping rect is for the block (by examining the overflow style of the block and its ancestors, I guess --- exactly how this is done depends on the spec). Then scan the display items that are in the display lists for the block's children; build a hash table that maps nsLineBox* to the associated nsBlockFrame* plus an array of nsDisplayItem* items that belong to that line. Also, compute the width of the text-overflow marker.
Then for each line, determine whether it needs a text-overflow marker. It needs a marker if it has any text that goes beyond the edge of the clipping rectangle in the text direction of the block. So for LTR blocks, a line needs a marker if there is any text beyond the right edge of its block. For RTL blocks, a line needs a marker if there's any text beyond the left edge of its block.
For each line that needs a marker, we need to hide content in the line that would overlap the marker. So go through the nsDisplayItems for the line. For LTR blocks, any non-text items that are not entirely to the left of the text-overflow marker must be completely hidden (removed from the display list). Any text items that are not entirely to the left of the text-overflow marker should have some clusters hidden; the easiest thing to do is probably to just store in nsDisplayText the text-overflow marker boundary and let nsTextFrame::PaintText figure out which clusters should be drawn.
Then the last thing to do is to have the block with text-overflow actually add new nsDisplayItems that draw the text-overflow markers for lines that need them. We'll probably want to use nsBidiPresUtils::ProcessText to draw the text here, so that mixed-direction text-overflow strings are handled. The painful thing here is that to draw with the right text style we'll need to replicate some logic from nsTextFrame to set the color, draw shadow, draw decorations etc (actually I'm not sure which effects should be applied). Maybe some code from nsTextFrame can be refactored and reused. nsHTMLContainerFrame has some code for drawing decorations at least. It's actually probably OK for an initial checkin to just draw text with the right color and font and not worry shadows etc.
We need to make sure that when you scroll, we don't just do a horizontal blit in cases where we normally would --- text that was hidden due to text-overflow may become visible, for example.
Fortunately scrolling uses display lists to figure out what can be blitted and what needs to be repainted. A simple option that would work is, when building a display list for scroll analysis and horizontal scrolling is occurring, for blocks with text-overflow that are moving to add an item to their display list that covers the entire block and reports true for IsVaryingForMovingContent.