Re: Testing/Working on PyUNO?

Hi,

On Wed, Feb 19, 2014 at 12:02:35PM +0100, Stephan Bergmann wrote:
> The idea would be that Kevin (and others) would fill this with PyUNO
> coding scenarios that cross their mind, discover errors in the PyUNO
> infrastructure, ideally distill tests for those specific errors out
> of the more general tests (that could then even go into more
> specific code modules like pyuno or testtools), and eventually prune
> test snippets again that are no longer useful.

That sounds very good to me. I think if the tests:

- serve as example code for Python users and thus attract more people around it 
- test PyUNO and the underlying product
- tests are reasonably selfcontained and do not introduce a huge framework on
  their own (hint: like unoapi did)

that would be huge winner. As telling Python users to write C++ tests instead
is of course nonsense -- and if our options are either getting Python tests or
none at all, I much prefer the first.

The tricky question will be to decide when to run these tests and what to do
with flaky tests -- aka tests that are flaky because of PyUNO and not the
underlying code. It would be unhelpful if a huge load of PyUNO failures turn up
at the application developers -- but that remains to be seen.

Ultimately, most tests should pass almost all the time. _If_ a test fails, it
should always be worth investigating. Part of that might be that the failing
tests gets rewritten in C++. So we would have a big set of Python tests (as
there are more people available to write them) and whenever one of those fails,
it migrates to C++. As you never know beforehand which one that will be, this
will help selecting the important ones.

Recent developments made more than obvious that no matter how much tests you
think you have, it will always be too few.

Best,

Bjoern