Re: [Libreoffice-qa] Minutes of ESC call: 2015-07-30

Hey,

On Sun, Aug 2, 2015 at 9:01 PM, Norbert Thiebaud <nthiebaud@gmail.com>
wrote:

> On Sun, Aug 2, 2015 at 11:25 AM, Markus Mohrhard
> <markus.mohrhard@googlemail.com> wrote:
>
>
> > > AI:     + will have a look at the CppUnit to implement 'expected
> failure'
> > > (Jan-Marek)
> > >             + Cpp logs are e.g. in
> > > workdir/CppunitTest/sal_rtl_math.test.log
> >
> >
> >
> > That is already possible with cppunit. Instead of using CPPUNIT_TEST use
> > CPPUNIT_TEST_FAIL which tells cppunit that the test is expected to fail
> with
> > a cppunit exception being thrown (it is extensively used in the cppunit
> > internal unit tests). The test will start to fail when none of the
> asserts
> > fail anymore. Keep in mind that it might be dangerous to use this with
> more
> > than one assertion as an unexpected one might fail.
>
> That is not exactly what tha aim is...
> the aim is to have a failure be repport as such and not stop
> everything.. when a test is tagged as _can fail_ or something
> this is when a test is added before the fix

So you want to have this as non-fatal tests. Ok, that is currently missing
from cppunit but at least with the CPPUNIT_TEST_FAIL you can add tests
before the bugfix and just change it from CPPUNIT_TEST_FAIL to CPPUNIT_TEST
when the fix has been implemented. So it is a compromise as it is already
there and we can have a look how difficult it would be to properly
implement non-fatal tests in cppunit.


> also the minutes claim that cppunit log exist.. it is true they do,
> but they are not exploitable.
> I would like to be able to repport at the end of the build a nice recap
>
> <module> <section> <nb_of_test> <#sucess> <#skipped> <#failed>
> <for each failed
> <space><testname> : FAILED (<optionally one line reason>)
>
> the minute point to
> workdir/CppunitTest/sal_rtl_math.test.log
> which contains:
> OK (3)
>
> but  writerperfect_writer.test.log fro instance
> contain 12027 lines.. most of it random trace messages, and I could
> not find any way to parse (at all, even less reliably) that thing to
> extract any useful information.
>
> maybe it is just a matter of using
>
> http://cppunit.sourceforge.net/doc/1.8.0/class_cpp_unit_1_1_xml_outputter.html
> and then adding a (optional) post processing step in the build to
> generate a nice summary....


It is even easier. You can add another TestListener in
sal/cppunittester/cppunittester.cxx that logs all failures, and executed
tests. Of course as there is no support for non-fatal tests right now you
can't log them. It would at least give you an information how many tests
are executed. I think there is no post-processing necessary as you get all
the information the the xml outputter has already in the listener and
process is directly there.

Regards,
Markus

> Norbert