Re: [Libreoffice-qa] Minutes of ESC call: 2015-07-30

Hi,

On Sun, Aug 02, 2015 at 06:25:05PM +0200, Markus Mohrhard wrote:
> That is already possible with cppunit. Instead of using CPPUNIT_TEST use
> CPPUNIT_TEST_FAIL which tells cppunit that the test is expected to fail
> with a cppunit exception being thrown (it is extensively used in the
> cppunit internal unit tests). The test will start to fail when none of the
> asserts fail anymore. Keep in mind that it might be dangerous to use this
> with more than one assertion as an unexpected one might fail.
>
> Additionally there is CPPUNIT_TEST_EXCEPTION which expects as second
> parameter the expected exception. So this is a replacement for the
> following pattern that can sometimes be found in our code:

Yes, but for one we do not only have c++ tests, esp. since one of the aims is
to get bugreporters (aka a broad audience) to improve their reports by adding (failing) tests:
>        + writing a test needs some knowledge, couldn't we actually mentor the author to do a fix 
> too? (Kendy)
>            + can be much harder (Norbert, Bjoern)

Also there might be unstable tests, for which expecting reliable failure is not
what we want (rather we want to ignore unstable tests until they are stable and
non-failing). Thus:

>        + 'make -k stagingcheck' - for tests that do not have a fix yet (Bjoern)

That doesnt preclude using the cppunit features you describe for really
expected failures/exceptions, so thanks for the hints!

Best,

Bjoern