Re: Extending subsequent tests with dogtail tests?

Hey,

let me at least add some comments.

On Thu, Mar 7, 2019 at 5:42 PM Jean-Philippe MENGUAL <
jean-philippe.mengual@libreoffice.org> wrote:

> Hi,
>
>
> Le 06/03/2019 à 20:52, Samuel Thibault a écrit :
> > Hello,
> >
> > Markus Mohrhard, le lun. 25 févr. 2019 00:34:33 +0800, a ecrit:
> > > On a slightly related note I think that we have already quite a few
> tests for
> > > the accessibility UNO layer but as that layer is full of bugs many of
> the tests
> > > are disabled. It might be a good idea to work on these tests before
> actually
> > > trying to implement more complex tests that depend on lower layers
> working
> > > correctly.
> >
> > I'm not sure which piece you are referring to.  Is that the AWB?  I
> > indeed see some source code in toolkit/test/accessibility but no
> > reference to it.

There should be accessibility UNO API tests in most top level modules. E.g.
from a quick git grep Accessible -- qadevOOo/Jar_OOoRunner.mk:

qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessible \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleAction \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleComponent \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleComponent \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleContext \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleEditableText \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleEventBroadcaster \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleExtendedComponent \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleImage \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleSelection \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleTable \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleText \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleValue \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_basctl/AccessibleShape \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/AccessibleEditableTextPara_HeaderFooter \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/AccessibleEditableTextPara_PreviewCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sch/AccessibleDocumentView \
qadevOOo/Jar_OOoRunner.mk:    qadevOOo/tests/java/mod/_sc/ScAccessibleCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleCsvCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleCsvGrid \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleCsvRuler \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleDocument \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleDocumentPagePreview \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePageHeader \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePageHeaderArea \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewHeaderCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewTable \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleSpreadsheet \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sd/AccessibleDrawDocumentView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sd/AccessibleOutlineView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sd/AccessibleSlideView \
qadevOOo/Jar_OOoRunner.mk:    qadevOOo/tests/java/mod/_sm/SmEditAccessible \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sm/SmGraphicAccessible \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBox \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxHeaderBar \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxHeaderCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxTable \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxTableCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleIconChoiceCtrl \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleIconChoiceCtrlEntry \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTabBar \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTabBarPage \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTabBarPageList \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTreeListBox \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTreeListBoxEntry \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svx/AccessibleControlShape \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svx/AccessibleEditableTextPara \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svx/AccessibleGraphicShape \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svx/AccessibleOLEShape \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svx/AccessiblePageShape \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svx/AccessiblePresentationGraphicShape \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svx/AccessiblePresentationOLEShape \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svx/AccessiblePresentationShape \
qadevOOo/Jar_OOoRunner.mk:    qadevOOo/tests/java/mod/_svx/AccessibleShape \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svx/SvxGraphCtrlAccessibleContext \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sw/SwAccessibleDocumentPageView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sw/SwAccessibleDocumentView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sw/SwAccessibleEndnoteView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sw/SwAccessibleFooterView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sw/SwAccessibleFootnoteView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sw/SwAccessibleHeaderView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sw/SwAccessiblePageView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sw/SwAccessibleParagraphView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sw/SwAccessibleTableCellView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sw/SwAccessibleTableView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sw/SwAccessibleTextEmbeddedObject \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sw/SwAccessibleTextFrameView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sw/SwAccessibleTextGraphicObject \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleButton \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleCheckBox \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleComboBox \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleDropDownComboBox \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleDropDownListBox \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleEdit \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleFixedText \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleList \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleListBox \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleListItem \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleMenu \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleMenuBar \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleMenuItem \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleMenuSeparator \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessiblePopupMenu \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleRadioButton \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleScrollBar \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleStatusBar \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleStatusBarItem \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleTabControl \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleTabPage \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleToolBox \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleToolBoxItem \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_toolkit/AccessibleWindow \


However you will quickly note that quite a lot of them are actually
disabled because they are broken, the underlying interfaces found bugs that
nobody cared or are just unreliable due to various issues. As a simple
example for sc:

sc/qa/unoapi/knownissues.xcl:sc.AccessibleEditableTextPara_PreviewCell::com::sun::star::accessibility::XAccessibleEditableText
sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleDocument::com::sun::star::accessibility::XAccessibleSelection
sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleCsvCell::com::sun::star::accessibility::XAccessibleText
sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleCsvRuler::com::sun::star::accessibility::XAccessibleText
sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleCsvGrid::com::sun::star::accessibility::XAccessibleTable
sc/qa/unoapi/knownissues.xcl:sc.AccessibleEditableTextPara_PreviewNote
sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleCsvRuler
sc/qa/unoapi/knownissues.xcl:sc.AccessibleEditableTextPara_HeaderFooter
sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleDocument
sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleCsvCell
sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleCsvGrid
sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleDocumentPagePreview
sc/qa/unoapi/knownissues.xcl:sc.AccessibleEditableTextPara_PreviewCell::com::sun::star::accessibility::XAccessibleComponent
sc/qa/unoapi/knownissues.xcl:sc.ScAccessiblePageHeaderArea::com::sun::star::accessibility::XAccessibleEventBroadcaster
sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleCell::com::sun::star::accessibility::XAccessibleText
sc/qa/unoapi/knownissues.xcl:sc.ScAccessiblePreviewCell::com::sun::star::accessibility::XAccessibleComponent
sc/qa/unoapi/sc_1.sce:# i84554 -o sc.AccessibleEditableTextPara_PreviewNote
sc/qa/unoapi/sc_1.sce:# i88241 -o sc.AccessibleEditableTextPara_HeaderFooter
sc/qa/unoapi/sc_1.sce:-o sc.AccessibleEditableTextPara_PreviewCell
sc/qa/unoapi/sc_1.sce:-o sc.ScAccessibleCell
sc/qa/unoapi/sc_1.sce:# i91044 -o sc.ScAccessibleCsvCell
sc/qa/unoapi/sc_1.sce:# i91044 -o sc.ScAccessibleCsvGrid
sc/qa/unoapi/sc_1.sce:# i84641 -o sc.ScAccessibleCsvRuler
sc/qa/unoapi/sc_1.sce:# i88330 -o sc.ScAccessibleDocument
sc/qa/unoapi/sc_1.sce:# i91045 -o sc.ScAccessibleDocumentPagePreview
sc/qa/unoapi/sc_1.sce:-o sc.ScAccessiblePageHeader
sc/qa/unoapi/sc_2.sce:# fdo#FOO -o sc.ScAccessiblePageHeaderArea
sc/qa/unoapi/sc_2.sce:-o sc.ScAccessiblePreviewCell
sc/qa/unoapi/sc_2.sce:-o sc.ScAccessiblePreviewHeaderCell
sc/qa/unoapi/sc_2.sce:-o sc.ScAccessiblePreviewTable
sc/qa/unoapi/sc_2.sce:# fdo#45337 -o sc.ScAccessibleSpreadsheet

All entries in "sc/qa/unoapi/knownissues.xcl" are disabled and all lines in
the *.sce file starting with # are also disabled which means that for
example for calc most accessibility tests are actually disabled.


> > > The focus handling can be easily integrated into the existing UI
> > > testing infrastructure and might benefit there from some of the
> concepts that
> > > should make them more stable
> >
> > Good :) So could you plan to work on it?

No. But I can point you to the corresponding code.

> How could such plan be scheduled in LO qa? Should we report a bug? Or
> open a wiki roadmap?

I think the only way is to actively do the work yourself. That means more
than just writing tests. At least in the beginning you will need to make
sure the framework is reliable and try to make using the test framework for
developers convenient enough that it is easy enough to reproduce failures
and debug them. Most importantly the framework needs to be reliable enough
that you can convince developers to actually care about test failures. If
your accessibility tests produce a noticable rate of false positives you
will quickly see all your tests being disabled and no developer paying
attention to any of your tests.

If you have a reliable test framework it can be integrated into our CI
infrastructure similar to how all the other test frameworks are executed by
our CI builds.


> The problem now is that it is not possible to fix accessibility bugs,
> fix regressions from 4.2, integrating in the code non-regression tests,
> and do the same for the three major programs in free software. It is
> less a will problem than a resource problem, because even with funds, we
> do not have enough persons to work on this with skills related to
> Libreoffice and accessibility in general. As you know more and more
> persons go the web or backend techno, less in programming for desktop
> software.
>
> So while we are ready to fix accessibility bugs, we need non-regression
> tests. And it is difficult to do both. The tool on which we are working,
> dogtail, is interesting because enables to test via the same framework
> different programs, without needing to know the code of each of one. I
> think gateway is possible between Libreoffice framework test and such
> tool. We also could imagine a dedicated machine with dogtail to test,
> but should be acceptd. Also, the thing is to know if LO is ready to
> prevent a release or a commit because introduces a regression in
> accessibility.

Most likely no. If we have tests for something it is more likely that it
will be fixed after it is discovered by in the end an accessibility
regression is similar to any other regression. That means that a test
started failing you are expected to inspect the test failure before
commiting but we will most likely not stop a release if a new test
discovers that something is broken. In general while we do much to avoid
regressions and having tests helps we don't treat regressions as complete
release blockers. There is always a case-by-case analysis neccessary.


> Well to sum up, beyond our effort, that we try general and
> cross-software, we would need to know how we can together set a kind of
> roadmap to implement accessibility in the existing frameworks, and add a
> layer to make common scenarios between our general tool and LO's one.
> Having that may be in the easy hacks to have contributions? Or b a
> specific TDF tender just like we did for labels?

Well, in the end the decision is mostly on you. I can only tell you based
on experience that the most important part is to implement a reliable
testing framework. You can start from scratch with an external tool or you
can base it on one of the existing frameworks that we already have. The
advantage of existing frameworks is that they have seen quite some use
already and are usually accepted as stable by most developers. For any new
framework you will need to convince existing developers about the value and
the reliability of the framework before they will accept the framework. You
can see with some parts of our UNO API tests, especially the accessibility
related UNO API tests, that unreliable tests just lead developers to
disable any failing test instead of spending time checking whether it was a
genuine failure or a false positive.

I wish you the best with your work on the accessibility testing.

Regards,
Markus

> Regards
>
> > Samuel
> > _______________________________________________
> > LibreOffice mailing list
> > LibreOffice@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/libreoffice
>
> --
> Logo Hypra      JEAN-PHILIPPE MENGUAL
> DIRECTEUR TECHNIQUE ET QUALITÉ
> 102, rue des poissonniers, 75018, Paris
> Tel : +331 84 73 06 61 <tel:+33184730661> Mob : +336 76 34 93 37
> <tel:+33676349337>
> jpmengual@hypra.fr <mailto:jpmengual@hypra.fr>
> www.hypra.fr <http://www.hypra.fr/>
> Facebook Hypra <https://www.facebook.com/hyprasoftware/> Twitter Hypra
> <https://twitter.com/Hypra_> Linkedin Jean-Philippe
> <https://fr.linkedin.com/in/jean-philippe-mengual-800133135>