Re: [Libreoffice-qa] Issues with 3.5.0 - ready for release ...

Hey,

2012/2/7 Caolán McNamara <caolanm@redhat.com>:
> On Tue, 2012-02-07 at 20:32 +0100, Dag Wieers wrote:
> > Do we
> > have tests doing automated QA (conversions) to test export filters and are
> > we using fuzzing techniques to test our import filters ?
>
> I've personally been to-date more interested in automated import filter
> tests, typically various binary file formats. You can find the existing
> tests in e.g. sw/qa/core/filters-test.cxx where the minimal
> always-tested test-cases are in e.g. sw/qa/core/data/*/
>
> The idea of these is that you use something like
> bin/get-bugzilla-attachments-by-mimetype to slurp down a pile of extra
> documents and shove them into sw/qa/core/data/*/indeterminate and just
> run "make" in sw to see if any of them triggers a crash (or a valgrind
> warning when export VALGRIND=memcheck make is used).
>
> Same basic "see if file format parser crashes" idea implemented for
> other modules spread around the place, e.g. sc, sd, svtools and stuff
> like .xls, .ppt .wmf and so forth.

With some modifications this will also work for odt. Odt is just a
special case because our internal code distinguishs odt and all other
filters and will fail if we don't provide all necessary information
for odt ( see sc/qa/unit/filters-test.cxx for the needed adjustments
). The more difficult part is then to find all necessary component
files and add them otherwise around 80% of the complex documents will
crash. I can port the sc adjustments to sw but someone else would need
to add the component files.

> calc has extended things a bit further for some extra tests to ensure
> that selected test documents have the expected calculation results and
> Markus demoed at FOSDEM the in-progress stuff for sd to ensure that
> selected test documents render as expected.

We also used Caolan's great script and concept to test all ods, xls
and xlsx files from OOo, fdo and redhat bugzilla for crashs. Testing
these files can only search for crashs not that documents are imported
because only around 60 to 70% of them can be imported at all. A lot of
them are either totally broken or contain passwords. I added a bit of
code to sc's filters-tesst and planned to improve that code for the
next release but if you plan to use this concept for writer or impress
I can integrate the remaining points from todo list.

But keep in mind that this is nothing that will be fast. I needed
around one week to check a bit more than 4000 files. Anyway it would
be great if you could help out doing the same for writer. This would
mainly involve finding the component files and running the test, I
would port all my changes to sw if needed.

Regards,
Markus