[libreoffice-accessibility] Re: [a11y] LibreOffice Calc exposes 2^31 children, freezes on `GetChildren`

Hi Michael,

On 2024-06-11 11:40, Michael Meeks wrote:
>      Some great questions and points here; forgive my intruding into the 
> space again =)

No need to apologize - thanks a lot for your valuable input! :-)

> > * How do other interfaces (like AT-SPI Table, TableCell and Selection) 
> > expose information? Does e.g. the table report it only has 50 rows and 
> > 30 columns if that's what's visible on screen? Does cell Q227 report a 
> > row and column index of 0 if it's the first one in the visible area?
>
>      I think exposing the whole thing through the Table interface may 
> make some sense; it is clearly a crazy set of cells - and think it's 
> reasonable to blame ATs if they use this interface for doing something 
> silly.

In the meantime, Joanmarie has added some further very useful 
information in tdf#156657 comment [1], also giving a reference to a grid 
example from the ARIA WG [2].

There, the Table interface also only exposes the same amount of cells as 
are exposed via the a11y tree.

(An even more advanced example also managing selection that that example 
references [3] currently doesn't exist yet, unfortunately. ("This 
example has not yet been developed."))

>      Right; so - I mentioned "near to the screen" - by near; I mean we 
> will probably want a number of things that are navigationally close: eg. 
> "next heading" or somesuch - to lurk around as real & tracked peers. The 
> content of the Navigator headings should prolly always be present in a 
> writer document's object hierarcy IMHO. That should let ATs very quickly 
> enumerate headings, jump focus to them with a simple API etc.

That sounds interesting, but in a way also like a rather strange tree to 
me if it contains elements of some type for the whole doc, but other 
parts of the document in between are missing.

From looking into the NVDA and Orca doc, other objects of interest are 
e.g. tables, lists, list items, or links. I suppose that including all 
of these (and more) in the tree could potentially again result in a 
large tree and the re-pagination update problems you've mentioned, for 
particular documents.

> > * How do screen readers implement features like "read the whole row"? 
>
>      This comes down to the navigation API I mentioned: having a good 
> API to allow continuous screen-reading of large data-sets - with caching 
> pre-loading & fetching along eg. a selection is really useful. Current 
> writer behavior is far from optimal since you need to do something odd 
> to get a simple navigation such as "next page" IIRC.
>
> > Do they just read the part of the row that's currently visible on 
> > screen and leave out the rest? Or do they somehow implement some extra 
> > logic
> > to  retrieve the remaining content?
>
>      Extra navigation / enumeration logic I think.

I see, the navigation API apparently is definitely an essential key 
aspect then.

If those need changes to the platform a11y APIs, we theoretically have 
more chances for these for some (the FLOSS ones: Linux/AT-SPI, 
Windows/IAccessible2, maybe Linux/Newton) than others 
(macOS/NSAccessibility, Windows/UIA - the latter not used by LO at this 
point in time).

> > > though of course it is then ideal to have some nice navigation API 
> > > support wrapped around that
> >
> > What kind of API does that refer to? Existing or new API on the
> > platform a11y level that LO (or the toolkits it uses) would then
> > implement, or something else? Do you have anything particular in mind?
>
>      I was actually somewhat optimistic about the UIA API Navigation API 
> conceptually:
>
> https://learn.microsoft.com/en-us/windows/win32/api/uiautomationcore/nf-uiautomationcore-irawelementproviderfragment-navigate

AT-SPI's flows-from and flows-to relations (and ARIA's aria-flowto) seem 
somewhat similar to the UIA Navigation API you mention.

If they allow consistent access to off-screen content (related: 
tdf#96492), they could potentially be used to retrieve the previous/next 
heading,...

IIUC, that would require retrieving one object at a time via the AT-SPI 
bus, though, as compared to using e.g. AT-SPI Collection (which can only 
take into account objects that are already part of the a11y tree).

>      Although - I'd really suggest that a11y doesn't work against the 
> application, and if navigating - it should allow the AT to scroll the 
> actual visible/view-port to match what is being interrogated.

Interesting thought, and maybe that could be part of the solution, if it 
becomes clearer what that can look like in practice.

E.g. it would seem odd to me if an AT starts scrolling through the 
document if a "go to next heading/list item" navigation command is 
triggered, and then e.g. goes back if it doesn't find anything, because 
it can't otherwise access the previously off-screen content to search 
for the item.

> > to look into at some point. My idea so far is to also expose pages on 
> > the a11y level, which should avoid the problem of a single object (the 
> > document) having an enormous amount of children due to that.
> > If there any general concerns about that, please raise them. :-)
>
>      I guess this moves the problem to re-pagination; where we can get 
> 300+ pages re-built for the sake of moving a single paragraph; then 
> again - I guess if we are notifying changes in position on large sets of 
> accessible peers we have a similar problem.

Good point! That could indeed be problematic performance-wise.


> > The feedback I've received from a11y experts so far is that off-screen 
> > doc content should *generally* be exposed on the a11y level, and 
> > limiting Calc to not do that with its huge amount of table cells is 
> > meant to be an exception to the rule in that regard (see e.g. the 
> > discussion in [2] and tdf#156657).
>
>      I really think that's a mistake that will ultimately hurt ATs 
> performance and that we should focus on the end-user use-cases we want 
> to succeed with - rather than having an abstract absolutist 
> pre-conception that we can expose everything in an efficient way =)

Sure - if there's a better way to properly make the AT use cases a 
reality, then let's go that route instead. :-)

>      Of course; I'm just one viewpoint. My strong feeling is that 
> focusing on things that make it easier to code fast, simple ATs that 
> meet the common use-cases people want is the vital thing; and I really 
> think that trying to re-build arbitrary document models and synchronize 
> them on the other end of a bus - particularly if we require lots of 
> synchronous round-trips to interrogate the content - is not going to fly.
>
> On 11/06/2024 09:49, Michael Weghorn wrote:
> > > Otherwise, as long as the underlying platform a11y protocols are 
> > > pull-based and given the input I've received up to this point, I tend 
> > > to think that ATs actively querying the tree are primarily 
> > > responsible for limiting that to a reasonable amount of information, 
> > > but I'm thankful for any guidance here...
>
>      Its a nice hope =) I'd want to create APIs that capture the common 
> things that ATs want to do, make them easy, and really hard to screw up.
>
>      But now I shut up ;-) we're working on the web side of this; 
> caching bits in the browser and adding another protocol latency there - 
> and I'm sure we want to be handling a reasonably bounded set of data 
> there =)

Is there an easy way to test COOL a11y web and impacts of potential changes?

(I just opened a sample Writer doc on nextcloud.documentfoundation.org 
and couldn't find the doc content via Accerciser in a quick test, but am 
also not very familiar with web content/browser a11y.)


As an additional note, one more potential source to get some interesting 
insights could be to check how NVDA's browse mode is currently 
implemented for MS Word, for example.


Thanks again for your thoughts!

Regards,
Michael


[1] https://bugs.documentfoundation.org/show_bug.cgi?id=156657#c11
[2] https://www.w3.org/WAI/ARIA/apg/patterns/grid/examples/data-grids/
[3] 
https://www.w3.org/WAI/ARIA/apg/patterns/grid/examples/advanced-data-grid/

--
To unsubscribe e-mail to: accessibility+unsubscribe@global.libreoffice.org
Problems? https://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: https://wiki.documentfoundation.org/Netiquette
List archive: https://listarchives.libreoffice.org/global/accessibility/
Privacy Policy: https://www.documentfoundation.org/privacy