Re: Extending subsequent tests with dogtail tests?

Hello,

Markus Mohrhard, le lun. 25 févr. 2019 00:34:33 +0800, a ecrit:
> On Sat, Feb 23, 2019 at 6:24 PM Samuel Thibault <[1]sthibault@hypra.fr> wrote:
>
> > That said, we could as well make tests work at both layers. Run them
> > along uitests, thus very frequently, and run them periodically through
> > the accessibility layer on a system which has it.
>
> I think you most likely want to test them independently if possible
> as I think you'll discover that combining them will lead to hard to diagnose
> problems.

That's why I mentioned trying with both layers.

> Additionally, the target of any test framework has to be that it
> minimizes the number of random test failures and requires as little adoption
> for unrelated code changes as possible.

Sure. That's why we're proposing to follow the blind user testcases,
which we usually don't want to change without some thinking.

> I noticed after quickly looking at your proposed tests is that you use UI
> strings in your tests which I think you want to avoid as much as possible.
> There are several problems with UI strings that make them a bad property during
> testing: they change often, often not even by developers and are localized.
> Especially the second point means that your tests suddenly only work in en-US
> which surely limits a bit the usefulness of the tests. Even more if you plan to
> generate test cases automatically as it means that the tests can only be
> generated with an en-US locale.

Sure, both testing and generating testcases needs to be done in the C
locale.

The fact that even the C UI strings may change is a concern indeed. We
however don't really have another way to identify widgets, do we? (we
don't want to identify them structurally, that'd be even less stable)

That was actually mentioned in

https://gitlab.gnome.org/GNOME/atk/issues/4

we'd need stable identifiers for the expected fields.

> On a slightly related note, please make sure that you are developing against
> python 3. I did not check if you are actually using any pure python2 features
> but at least "#!/usr/bin/python" will get you a python 2 interpreter on most
> systems.

Yes, that's on purpose because Debian doesn't have the python3 package
for dogtail yet :)

Samuel