Mobile/AsyncSubframePanning

From MozillaWiki
Jump to: navigation, search

Goals

Phase 1

  • land in 23 (not likely)
  • B2G subframe scrolling
  • Performance is
    • as good as main frame
  • subframes can't zoom
  • seamed scrolling

Phase 2a

  • seamless scrolling (needed for B2G dynamic toolbar)

Phase 2b

  • Fennec subframe scrolling

Not doing

  • scroll indicators are a non-goal
  • not fixing page drift
  • concurrently scrolling parents and ancestors
  • subframe zooming

Plan

To dos
  • merge B2G APZC into fennec
  • multiple display ports
  • bind multiple display ports
  • hit detection
  • transform equation cleanup
    • unit tests
  • tuning/profiling
  • concurrently scrolling subframes

Design

Step 1

Performed in Layout module, Main Gecko thread

  • (Bug 866232) Layout (FrameLayerBuilder et al.) builds an initial simple layer tree with the position of the subframes tagged to the best fitting layer
  • (See diagram for sample page & layer tree)

Step 2

Performed in the compositor's main thread

  • Layer tree update/transaction is received by the compositor.
  • Touch event is delivered at (6,51).
  • (Bug 775452) Hit detection is performed on a layer tree in reverse depth-first order and matches 1st subframe.
  • APZC decides of the appropriate buffer/pre-rendering and sends an IPC message to the main thread with an updated FrameMetrics/display port request.
  • (See diagram for sample page & layer tree)

Step 3

Performed in Layout module, Main Gecko thread

  • Display port is requested on the subframe in Layout
  • (Bug 864447) Layer tree is grown to make room for a layer of the 1st subframe.
  • Subframe is painted with a display port allowing Async scrolling. FrameMetrics are updated.
  • (See diagram for sample page & layer tree)

Step 4

Performed in the compositor's main thread

  • (Bug 866265) APZC can now perform Async panning on new sublayer.

Step 5

  • After timeout APZC resets FrameMetrics/display port on the subframe removing the display port, the layer and releasing the memory usage.

Notes and diagram

ApzcDesignPage1.png 9-.png B2g-input-flow.png

Bugs

Tracking bug for async subframe scrolling: bug 775452


Full Query
ID Priority Summary Status Assigned to
775459 -- Implement (more) hit-testing logic for async/synchronously scrolled frames on UI thread RESOLVED Benoit Girard (:BenWa)
833795 -- Content jitters when URL bar hidden RESOLVED Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)
864441 -- Add APZC tests RESOLVED Benoit Girard (:BenWa)
864447 -- Support drawing display port on nested APZC frames RESOLVED Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)
865735 -- Disambiguate between Point instances in different coordinate systems RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
866232 -- Support having multiple APZC instances attached to the layer tree RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
866265 -- Transformation equation cleanups RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
872911 -- Find a better solution for handling touch gestures with moving frames RESOLVED
883646 -- Distinguish between CSSPixel and "device pixel" for Fennec RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
890279 -- Create ScrollInfoLayers for frames we want APZC support RESOLVED Benoit Girard (:BenWa)
890280 -- APZC input event transformations should consist of the async transform relative to the correct coordinate system RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
891384 -- APZC Hit Testing doesn't support 3D transform RESOLVED
893872 -- Add TransformShadowTree unit tests RESOLVED Benoit Girard (:BenWa)
894288 -- nsLayoutUtils::FindContentFor aborts when loading any tab RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
894619 -- TransformShadowTree only transforms root APZC RESOLVED
896045 -- Do input events go to the right place with off-main-thread animations? RESOLVED
898443 -- Figure out a good focus model for input events to multi-APZC RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
898444 -- Figure out why overflow:scroll elements aren't scrolling properly RESOLVED Robert O'Callahan (:roc) (email my personal email if necessary)
900638 -- Fix event-fluffing to work with APZC RESOLVED Dale Harvey (:daleharvey)
903460 -- AsyncCompositionManager applies async transform and CSS transform in the wrong order RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
907754 -- Pages with iframes that are less than screen width are not rendering correctly RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
912144 -- Hit testing for subframes is sometimes not right RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
913659 -- Strange behaviour when zooming page with iframe a lot RESOLVED
916125 -- When displayports are set on elements, their layers are not always clipped correctly RESOLVED Timothy Nikkel (:tnikkel)

24 Total; 0 Open (0%); 24 Resolved (100%); 0 Verified (0%);


Tracking bug for allowing Fennec to use APZC: bug 776030


Full Query
ID Priority Summary Status Assigned to
777075 -- Extract an interface for the PanZoomController -> LayerController dependency RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
777351 -- Merge LayerController and GeckoLayerClient RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
777468 -- Extract an interface for dependencies on PanZoomController RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
777896 -- Break dependencies from PanZoomController -> other java code RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
839641 -- Wire up the AsyncPanZoomController to be usable from java RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
859929 -- AsyncPanZoomController doesn't play well with progressive tile painting RESOLVED Botond Ballo [:botond]
859935 -- AsyncPanZoomController assumes the "UI/controller" has a message loop RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
859939 -- AsyncPanZoomController has some type wrongness RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
859951 -- Refactor the code to convert from MOTION_EVENT to nsTouchEvent instances RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
859962 -- BrowserApp assumes LayerView.getPanZoomController never returns null RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
860162 -- AsyncPanZoomController doesn't apply proper scroll compensation if the root layer has a scale RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
860455 -- AsyncPanZoomController::WidgetSpaceToCompensatedViewportSpace is not fully implemented RESOLVED
860613 -- Add a way to bind fennec's APZC to the primary scrollable layer RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
860940 -- Replace AndroidGeckoEvent constructors with static factory methods RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
867269 -- SyncFrameMetrics override is disconnected and breaks compiles on gcc 4.7+ RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
880024 -- Create a local version of GeckoContentController RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
880578 -- APZC: Scroll events are only sent when repainting is needed RESOLVED Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)
880932 -- APZC: Page stops repainting and zoom/pan stops updating RESOLVED Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)
881451 -- ThebesLayerComposite::GetCompositionBounds() needs to be fixed up in various ways RESOLVED Chris Lord [:cwiiis]
890280 -- APZC input event transformations should consist of the async transform relative to the correct coordinate system RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
895904 -- NativePanZoomController.abortAnimation() needs to be hooked up RESOLVED Botond Ballo [:botond]
943557 -- Use modifier keys for click events generated by single-tap gestures in Android APZC code RESOLVED
1046344 -- Unbitrot APZ glue code for Fennec RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1049136 -- Hook up touch event flow for APZ usage on Fennec RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1055548 -- With native APZ enabled clicking on stuff sometimes doesn't work quite right RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1055557 -- Properly notify the APZ of root zoom constraints and changes to zoom constraints RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1137872 -- Make the urlbar hiding stuff work with native pan zoom controller RESOLVED
1146020 -- Make APZCCallbackHandler extend from ChromeProcessController RESOLVED Danilo Cesar Lemes de Paula
1146024 -- Fix up input routing for APZ on Fennec RESOLVED Danilo Cesar Lemes de Paula
1156392 -- Remove MOZ_ANDROID_APZ condition from ifdef in ScrollFrameHelper::WantAsyncScroll RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1156401 -- Fix ContextMenu handling when building Fennec with C++ APZ enabled RESOLVED Danilo Cesar Lemes de Paula
1158424 -- Clean up FrameMetrics::mIsRoot and related notions RESOLVED Botond Ballo [:botond]
1159405 -- Fix "Found a non-root APZ with no handoff parent" warning when root process has a root XUL document with scrollable subframes RESOLVED Danilo Cesar Lemes de Paula
1162701 -- Fix panning on Fennec when built with C++ APZ RESOLVED
1163572 -- Override RequestContentRepaint to detect root-frame updates RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1175282 -- [Fennec-APZ] Unable to scroll outside scrollable subframe after zooming on a non-scrollable document RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1180267 -- Update Fennec code to use MobileViewportManager instead of the big pile of browser.js code RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1188225 -- Implement ChromeProcessController::HandleDoubleTap RESOLVED Randall Barker [:rbarker]
1190979 -- Text is selected when pinch zooming RESOLVED
1190988 -- Hit testing appears to return incorrect positions when --enable-android-apz is specified. RESOLVED Randall Barker [:rbarker]
1195553 -- Double tap gesture fails on Fennec when C++ APZ is enabled. RESOLVED Randall Barker [:rbarker]
1197824 -- Update Fennec code to use ZoomConstraintsClient instead of the big pile of browser.js code RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1200393 -- Java APZ appears more responsive during page load compared to C++ APZ. RESOLVED
1200397 -- Pinch zoom gesture does not always appear to work when C++APZ is enabled. RESOLVED
1200399 -- The dynamic tool bar in Fennec will some times permanently hide RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1201100 -- Perma-checkerboarding when zoomed in with apz-fennec RESOLVED Randall Barker [:rbarker]
1201130 -- Switching tabs in Fennec-APZ sometimes doesn't update the metrics properly RESOLVED
1201217 -- Switching tabs in Fennec-APZ sometimes renders the page at an incorrect zoom RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1201529 -- When APZ is enabled, all browser.js codepaths that set a resolution or displayport should be disabled RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1201581 -- Fixed-position margins/clip are not updated properly in fennec-apz RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1202020 P5 Remove dependencies on async pan/zoom information from Java code RESOLVED
1206620 -- Zoom out does not work at double tap action in Landscape mode on Tablet devices RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1206872 -- [meta] Enable APZ on fennec on nightly builds RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1206874 -- [meta] Let APZ on Fennec ride the trains RESOLVED
1229349 -- Tapping on things in the zoomed view is broken with C++ APZ RESOLVED
1229439 -- Horizontal Scrolling of elements is not registered during page load or janky RESOLVED
1229462 -- Use Android Scroller class for fling animation RESOLVED Randall Barker [:rbarker]
1229840 -- Reader mode toolbar seems to show/hide a lot with C++ APZ RESOLVED
1230081 -- Scrolling on partially rendered pages causes judder and other issues RESOLVED
1230085 -- White checkerboarding visible (instead of page background) RESOLVED
1230176 P3 Weirdness at end of Fennec fling. Don't send duplicate DOM 'scroll' event for sub-pixel scrolling REOPENED
1240065 -- Reconsider viewport update policy when pinch zooming RESOLVED Randall Barker [:rbarker]
1248014 -- Allow horizontal scrolling from a pointer device when positioned over a horizontally scrolling layer. RESOLVED
1260534 P3 Checkerboarding in reader view is white instead of page background color RESOLVED
1260905 -- Fennec APZ flywheel physics result in very strange behavior when you do sequence of quick and short pans RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1262807 P1 Lots of repainting while scrolling around in Fennec RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1265372 -- Scrolling in reader mode enters/exits fullscreen mode too easily RESOLVED
1272478 -- Cannot scroll about:memory horizontally VERIFIED Randall Barker [:rbarker]
1274597 P1 Clicking on links sometimes doesn't work, long-pressing results in text selection instead of context menu RESOLVED Kartikaya Gupta (email:kats@mozilla.staktrace.com)
1278390 P2 Fennec doesn't display Gitter.im content, after scrolling up/down RESOLVED
1280666 P2 Use modified OverScroller to work around bugs RESOLVED James Willcox (:snorp) (jwillcox@mozilla.com) (he/him)
1281042 P3 Displayport expiry can result in perma-checkerboarding RESOLVED
1291678 P3 Some posts on GoT (gathering of tweakers)-forum are scrollable for fractions of pixels NEW
1297148 P3 Crash in mozilla::CrossProcessMutex::CrossProcessMutex RESOLVED
1302150 P2 Dynamic toolbar doesn't reappear after scrolling to bottom of long page VERIFIED Randall Barker [:rbarker]

75 Total; 2 Open (2.67%); 71 Resolved (94.67%); 2 Verified (2.67%);


Meetings

2013-04-26

  • initial meeting, created phases, goals and overall plan

2013-05-21

  • Chris will take the display port work

2013-05-28

  • Kats put up patch for disambiguated point classes
    • will continue propegating the use of the classes through the
    • should be up for review this week
  • Chris should start on display ports per scroll frame this week

2013-06-04

  • Kats
    • propagating type changes to frame metrics and APZC code
  • BenWa
    • unit tests
    • hit testing
  • ajones
    • multiple display ports
    • code already largely supports it
  • does metro need subframe scrolling?
  • next items:
    • hit testing (in progress, owned by BenWa)
    • creating and managing APZCs from the frame loader builder (need owner)
AsyncSubFrameSchedule.jpeg


2013-06-11

  • Getting Jim and Brian up to speed, explaining how things work:
    • Going through some of our "older" bugs, figuring out the history of how this came to be.
    • C++ version that is now being used in Android, almost working completely
    • In depth discussion on what the pieces are, how they work together
  • Jim & Brian will work on getting single APZC without subframes on Metro. Primarily Brian.
    • This will be based off C++ code which B2G uses, and there may be remaining issues in there for a desktop platform. We have to make sure this is tested as soon as it starts working so that we can flush out any remaining bugs.
  • The work that Kats, BenWa, AJones are doing should then just click in place and give us subframe APZC on Metro as well.

2013-06-18

  • Kats:
    • There is another coordinate system that we have to worry about, and B2G seems to be very broken on high dpi devices. This is what the code looks like as well.
    • We don't need high dpi for 1.1, so we probably don't need to uplift this. Need to test these changes, and then we should be OK.
  • Benoit:
    • On hit testing during the last week. More complicated test cases, with transforms, etc.
  • Multiple display ports support in the layout, but it probably already supports it. We just need to use those assumptions and proceed.

2013-06-25

  • BenWa:
    • Pushed the hit testing on try, failed, sorting it out now.
    • will finish hit testing, need to sort out gtest

2013-07-02

done
  • BenWa hit testing
    • layout assumed that root frames were container frames, which isn't correct
  • kats fixed high dpi event delivery
    • two code paths (with root APZC and w/o APZC)
    • seems like enforcing that there is always a root APZC would be the best route
next
  • BenWa will finish hit testing before going on PTO
    • when he gets back from PTO, hook up hit testing
  • Kats will make sure that there is no code assumeing that there is only one APZC
    • will require making APZC owned by layer tree not compositor parent, compositor parent will hold a reference to layer tree
  • Kats will defer continuing to work on input event transformation


2013-07-09

done
  • Kats
    • We can have multiple PZCs now
    • How do we (uniquely) identify scrollable layers? Need to use the layer ID and the scroll/frame ID, and presshell ID to guard against page navigation changes.
    • Threading problem is under control by creating a separate tree of APZC instances that shadows the layer tree that is walkable on the UI thread
    • This will tie in with dzbarsky's work
  • BenWa
    • Creating the scroll info layers is done, will be landed today
    • Not much else to do at this point, blocked on kats

2013-07-16

Need to discuss
  • Requirements for dynamic toolbar that roc/Cwiiis planned out - this requires controlling which nested frame gets the scroll events
    • roc/Cwiiis claim they depend on the APZC work. Unclear if they're blocked, and on which part.
    • Controlling which nested frame gets scroll events seems to be the item that is blocking the above work. This is what we called "Phase 2, seamless scrolling". Looks like this should be the first thing in the second phase.
  • What about Metro?
    • In good shape, mostly working
    • Scrolling properly
    • Waiting for us to finish
    • Brian was looking at axis locking
  • What's next?
    • Dynamic scrollbar
    • Seamless scrolling
    • B2G E-mail app (two frames)
Kats
  • Tree manager issues, timing of APZC being created. Even if message gets dropped, we seem to get the right information later, and will process it there.
  • Still not clean, as we're dropping messages, so we may be looking for a better solution in the future.
  • Porting hit testing patches from before, in progress. This is next. Hoping the input events aren't going to complicate things and that it should be as we had it before. The focus model needs to be sorted out as well.
BenWa
  • Transform shadow tree test, then sidetracked by a mobile bug. Will focus on the tests today.

2013-07-23

status
  • kats
    • has patch for APZC per layer
    • patch was expecting a layer tree, was actually getting indicator layers
    • plan is to on-demand convert layers to scrollable layers
    • currently working on making code in RenderFrameParent/TabParent/TabChild do the right thing in terms of scrolling the subframes when APZC asks it to
  • BenWa is at SIGRAPH
  • Botond
    • patch to make APZC not break progressive tile painting
    • on inbound for Android


next steps
  • Kats
    • wrap up patch
    • write more tests
  • Botond
    • making b2g APZC work with progressive tiles
notes
  • aiming to land everything next week. It'll be tight

2013-07-30

status
  • kats
    • got review from BenWa, waiting for mattwoodrow
    • landed a few follow up bugs
      • most important is focus
    • overscroll DIVs are broken, they're not being made scrollable layers
    • bug 890290 function to convert widget space in input events needs to be modified to take async into account
  • BenWa
    • away at sigraph
    • did reviews
next
  • get multi APZC landed
  • figure out focus model
  • figure out overscroll DIVs

2013-08-06

status
  • kats
    • landed multiple APZC patch
      • regression bug 900742 - input events don't always end up at the right place
      • regression bug 899810 - on B2G and Metro where the screen goes blank
next

2013-08-13

status
  • kats
    • input event targeting
    • discussed a central gesture detector
    • setting resolution on top level pres shell
  • benwa
    • (unrelated) b2g leo blocker
  • botond
  • metro apzc
next steps
  • kats
    • several patches up for review, need to get them landed
      • some should have some more testing before landing
    • continue working on input events
  • benwa
    • finish leo+ blockers
    • do reviews

2013-08-20

status
  • kats
  • benwa
    • camera crash bugs
      • has his head around the crashes, should have patches soon
  • botond
    • getting feedback on gaia patch
    • looking at seemless panning transition
next
  • kats
    • turn on async panning for all apps on b2g
    • investigate overflow div panning
need to look at
  • display port tuning
  • input events go through gecko thread
    • would be better if input events didn't

2013-08-27

status
  • kats
    • landed some focus model stuff (bug 898443, bug 906427, 907977)
    • have a patch for bug 907754 (incorrect scaling in subframes) which needs cleanup. Also landed bug 885023 which this fix required
    • looked at what things are like when APZC is turned on for all of gaia. Some weird issues but I will work through the big ones one at a time to see if they are hiding any other issues underneath
next
  • kats
    • going to work on tracking down the overflow:scroll div problem
    • work through the gaia issues I found (mostly using the contacts app as a test)


2013-09-03

status
  • kats
    • landed 907754 (zoom, overflow documents issues)
    • landing 910322 today after addressing some comments
    • overflow scroll sub-documents problems - probably bug in flattening code, waiting for mwoodrow and tn
  • BenWa
    • not working on this last week
  • botond
    • handing overscroll to the parent layer; have the first implementation going
    • will be looking at display port bugs
    • composition bounds of iframes, same as what jmathies was running at - have an r+ from tn
next steps
  • kats
    • may run out of stuff to look at - one bug to file, hit testing a bit off
    • blocked on waiting for info and reviews
notes
  • kats - issues with tiling on fennec? wrong resolution, wrong sizes?
  • benwa - you see progressive tile painting in the cases where you should not (e.g, updating existing content)

2013-09-10

status
  • kats
    • implementing parent APZC
    • talked to tn about overflow scroll problem
      • frames construction isn't correct (there's a nested scroll frame)
      • tn will think about it
      • milan will follow up with tn tomorrow
  • botond
    • patch for scroll hand off posted
      • issue with zoomed subframes
    • moving on to annotating the browser iframe
  • BenWa did nothing of relevance
next
  • botond to annotate the browser iframe

2013-09-24

status
  • kats
    • last week was mostly mobile work week
    • continued working on bug 912657. input events are currently not being sent to the APZC in the root process so that will need fixing. currently they are delivered from gonk/nsWindow.cpp directly to gecko, gecko does some hit testing, and if they are in the area of the content process, *then* they go to the APZC. instead nsWindow should send it directly to the APZC first. i will work on this, building off the work jimm has done in bug 915213.
  • botond
    • at a C++ standardization conference this week
    • last week he worked on bug 912666 and made good progress on special handling for the toolbar. the individual pieces seem to work but it depends on bug 912657 to tie it all together.