Accessibility/CacheTheWorld: Difference between revisions
Jump to navigation
Jump to search
Full Query
Full Query
Full Query
Mreschenberg (talk | contribs) (Add assignee to bug tracker) |
(Added Roadmap.) |
||
| Line 5: | Line 5: | ||
Jamie, Morgan, and Eitan meet weekly to discuss this project. You can find the meeting notes in [https://docs.google.com/document/d/1JpjAgOM793shYIR0vcFKmmE2RIpJ33RoLuX8NzNWPq4/edit?usp=sharing this google doc]. | Jamie, Morgan, and Eitan meet weekly to discuss this project. You can find the meeting notes in [https://docs.google.com/document/d/1JpjAgOM793shYIR0vcFKmmE2RIpJ33RoLuX8NzNWPq4/edit?usp=sharing this google doc]. | ||
== | == Roadmap == | ||
Given the large scope of the project, we are breaking the project down into quarterly milestones. Each milestone will aim to support a set of user scenarios. This roadmap is in the early stages and subject to significant change. It will be updated as milestones become clearer, with future milestones being less well defined than earlier ones. | |||
=== Milestone 0: December 2021 === | |||
Initial proof of concept. | |||
All testing will be performed with the NVDA screen reader, for two reasons: | |||
# Cache the World is all or nothing on Windows. That makes it easy to determine where we're at regarding real world usage. | |||
# The performance benefits are most necessary and noticeable on Windows. | |||
In milestone 0, the following capabilities will be provided: | |||
# Reading and navigating a page with text, links and headings. | |||
# Plain text editing: reading the line of text when focused; backspacing; moving the caret by line, word and character. | |||
# Access to formatting information: font, bold/italic, etc. | |||
# Access to screen coordinates on simple pages. | |||
# Loading very large pages will be at least 10x faster with the cache than without. | |||
==== Test Scenarios ==== | |||
# Do a Google search, navigate the results using heading navigation and follow a result link. | |||
# Fill out the form for a Google Advanced Search. | |||
# Go to https://www.reaper.fm. Check the formatting of the “This is REAPER.” text (which isn’t a heading even though it should be) and confirm that the font size is reported as bigger than the paragraph of text below it. | |||
# Build Gecko in the background. Do a Google search. Verify that the browser does not become unresponsive. | |||
# Load https://searchfox.org/mozilla-central/source/layout/base/nsCSSFrameConstructor.cpp. Page should take < 10 sec to be usable. | |||
# Open Gmail, find a message in the inbox, open it, read it, return to the inbox. | |||
# Compose a message in Gmail containing text, a link, a bulleted list and a block quote. Read back through the message. | |||
# Open Slack, use the quick switcher to switch to a channel, read some messages, write a message. | |||
==== Bugzilla ==== | |||
Note that the roadmap wasn't created until late in milestone 0, so many bugs are missing below. | |||
<bugzilla> | |||
{ | |||
"quicksearch": "ALL whiteboard:[ctw-m0]", | |||
"include_fields": "id, summary, assigned_to, status" | |||
} | |||
</bugzilla> | |||
=== Milestone 1: March 2022 === | |||
Android. | |||
The primary focus of this milestone is getting the cache working for Android. Mozilla aims to implement Fission for Android in 2022h1. Modifying the existing multi-process architecture to support Fission on Android will require significant engineering effort. Rather than investing in a solution which we will be throwing away once the cache is implemented, we will instead switch Android to use the cache and extend the cache to include functionality required by Android. | |||
In milestone 1, the following capabilities will be provided: | |||
# The cache will support GroupPosition. | |||
# TextLeafRange will support word end and line end boundaries, which are needed for Android text navigation. | |||
# Pivot will support navigating text using TextLeafRange, which is needed for Android text navigation. | |||
# Cached screen bounds will be updated appropriately when scrolling. | |||
# Android will use the cache for all functionality except hit testing. | |||
# As an interim solution, Android will use the existing async IPDL mechanism for hit testing, updated to target the call at the correct document to handle OOP iframes. (Synchronous hit testing in the core cache will take longer to implement and will be done in a future milestone.) | |||
==== Test Scenarios ==== | |||
# TBD: simple website reading. News site? | |||
# TBD: filling a form. | |||
# TBD: character/word/line navigation. | |||
# With Talkback, load https://www.nvaccess.org/. Navigate to the embedded video in four different ways: item navigation, explore by touch, controls navigation and Talkback search. Activate the Play button to play the video. | |||
==== Bugzilla ==== | |||
<bugzilla> | |||
{ | |||
"quicksearch": "ALL whiteboard:[ctw-m1]", | |||
"include_fields": "id, summary, assigned_to, status" | |||
} | |||
</bugzilla> | |||
=== Milestone 2: June 2022 === | |||
TBD. Opt-in user preview. Enable in Nightly? | |||
=== Milestone 3: September 2022 === | |||
TBD. Beta experiment? Release experiment? | |||
== Backlog == | |||
<bugzilla> | <bugzilla> | ||
{ | { | ||
" | "quicksearch": "ALL blocked:1694563,1733486,1735970,1733417 -whiteboard:[ctw-m", | ||
"include_fields": "id, summary, assigned_to, status" | |||
"include_fields": "id, summary, assigned_to, status" | |||
} | } | ||
</bugzilla> | </bugzilla> | ||
Revision as of 02:47, 22 December 2021
About
Firefox's current architecture for multi-process accessibility suffers from severe performance issues and is costly and difficult to maintain due to the massively different and specialised approaches necessary on different operating systems. In addition, it is currently impossible to support builtin Windows accessibility tools such as Narrator and Windows Speech Recognition. This project aims to re-architect our multi-process accessibility support to cache the entire accessibility trees for all content processes within the parent process.
Meeting Notes
Jamie, Morgan, and Eitan meet weekly to discuss this project. You can find the meeting notes in this google doc.
Roadmap
Given the large scope of the project, we are breaking the project down into quarterly milestones. Each milestone will aim to support a set of user scenarios. This roadmap is in the early stages and subject to significant change. It will be updated as milestones become clearer, with future milestones being less well defined than earlier ones.
Milestone 0: December 2021
Initial proof of concept.
All testing will be performed with the NVDA screen reader, for two reasons:
- Cache the World is all or nothing on Windows. That makes it easy to determine where we're at regarding real world usage.
- The performance benefits are most necessary and noticeable on Windows.
In milestone 0, the following capabilities will be provided:
- Reading and navigating a page with text, links and headings.
- Plain text editing: reading the line of text when focused; backspacing; moving the caret by line, word and character.
- Access to formatting information: font, bold/italic, etc.
- Access to screen coordinates on simple pages.
- Loading very large pages will be at least 10x faster with the cache than without.
Test Scenarios
- Do a Google search, navigate the results using heading navigation and follow a result link.
- Fill out the form for a Google Advanced Search.
- Go to https://www.reaper.fm. Check the formatting of the “This is REAPER.” text (which isn’t a heading even though it should be) and confirm that the font size is reported as bigger than the paragraph of text below it.
- Build Gecko in the background. Do a Google search. Verify that the browser does not become unresponsive.
- Load https://searchfox.org/mozilla-central/source/layout/base/nsCSSFrameConstructor.cpp. Page should take < 10 sec to be usable.
- Open Gmail, find a message in the inbox, open it, read it, return to the inbox.
- Compose a message in Gmail containing text, a link, a bulleted list and a block quote. Read back through the message.
- Open Slack, use the quick switcher to switch to a channel, read some messages, write a message.
Bugzilla
Note that the roadmap wasn't created until late in milestone 0, so many bugs are missing below.
| ID | Summary | Assigned to | Status |
|---|---|---|---|
| 1735955 | Cached bounds all 0s for many (most?) elements on Google search | James Teh [:Jamie] | RESOLVED |
| 1739050 | If the focused Accessible is moved, the RemoteAccessible is recreated but focus isn't fired on it (AKA broken Google Search box on Windows + CTW) | James Teh [:Jamie] | RESOLVED |
| 1741792 | Cache the caret | James Teh [:Jamie] | RESOLVED |
| 1742902 | Fix window emulation when the cache is enabled | James Teh [:Jamie] | RESOLVED |
| 1742915 | Cache tag object attribute | James Teh [:Jamie] | RESOLVED |
| 1742917 | Implement StartOffset for RemoteAccessible and LinkIndexAtOffset for HyperTextAccessibleBase | James Teh [:Jamie] | RESOLVED |
| 1746827 | Crash in [@ PLDHashTable::Search | mozilla::a11y::RemoteAccessibleBase<T>::MinValue] | James Teh [:Jamie] | RESOLVED |
7 Total; 0 Open (0%); 7 Resolved (100%); 0 Verified (0%);
Milestone 1: March 2022
Android.
The primary focus of this milestone is getting the cache working for Android. Mozilla aims to implement Fission for Android in 2022h1. Modifying the existing multi-process architecture to support Fission on Android will require significant engineering effort. Rather than investing in a solution which we will be throwing away once the cache is implemented, we will instead switch Android to use the cache and extend the cache to include functionality required by Android.
In milestone 1, the following capabilities will be provided:
- The cache will support GroupPosition.
- TextLeafRange will support word end and line end boundaries, which are needed for Android text navigation.
- Pivot will support navigating text using TextLeafRange, which is needed for Android text navigation.
- Cached screen bounds will be updated appropriately when scrolling.
- Android will use the cache for all functionality except hit testing.
- As an interim solution, Android will use the existing async IPDL mechanism for hit testing, updated to target the call at the correct document to handle OOP iframes. (Synchronous hit testing in the core cache will take longer to implement and will be done in a future milestone.)
Test Scenarios
- TBD: simple website reading. News site?
- TBD: filling a form.
- TBD: character/word/line navigation.
- With Talkback, load https://www.nvaccess.org/. Navigate to the embedded video in four different ways: item navigation, explore by touch, controls navigation and Talkback search. Activate the Play button to play the video.
Bugzilla
40 Total; 0 Open (0%); 40 Resolved (100%); 0 Verified (0%);
Milestone 2: June 2022
TBD. Opt-in user preview. Enable in Nightly?
Milestone 3: September 2022
TBD. Beta experiment? Release experiment?
Backlog
241 Total; 19 Open (7.88%); 204 Resolved (84.65%); 18 Verified (7.47%);