Re: Testing/Working on PyUNO?

On Wed, Feb 19, 2014 at 5:50 PM, Bjoern Michaelsen <
bjoern.michaelsen@canonical.com> wrote:

> On Wed, Feb 19, 2014 at 05:20:19PM +0100, Markus Mohrhard wrote:
> > I doubt that. Look at the situation with the java tests where I'm the
> only
> > one who rewrites failing tests in c++. Most people just disable the test
> > that is failing and go on.
>
> I just linked a bug that showed more 50 crash reports per day in the
> released
> product and was almost impossible to reproduce but was somewhat
> reproducable
> with the java tests. As for rewriting the Java tests -- the unoapi
> framework
> made that extra painful, though it should be somewhat saner for the complex
> tests. So given the choice, the Python tests should not imitate the unoapi
> test, to stay easily rewritable.

> Also not that as I said in the other mail, I wouldnt want those tests to
> be run
> like unittests (on every build), but rather on each tagged release an
> prerelease build (with more runs optional) and we should aim to get them
> in a
> reasonable state by release (not ASAP/next day as for unittests) and no
> they
> are not blockers for a release in any way.
So what you say is that you want to have python tests that test core
functionality but don't want to run them regularly. Somehow that can't
work. You have the two things you can do:

1) test core functionality with implementation details but that will rely
on implementation details and will bit rot if you don't execute them
2) test our stable API but then you should not try to avoid implementation
details which means you can't test most bugs

From what I see only point 2 makes sense for python tests especially if you
want to run them only before releases and don't make them fail the build.

Regards,
Markus