Re: Acceptable outcomes of SwarmSolverTest::testUnconstrained

On 28.02.2018 02:36, Tomaž Vajngerl wrote:
> On Tue, Feb 27, 2018 at 4:53 PM, Stephan Bergmann <sbergman@redhat.com> wrote:
> > SwarmSolverTest::testUnconstrained in sccomp/qa/unit/SwarmSolverTest.cxx has
> > already been weakened in the past,
> > <https://cgit.freedesktop.org/libreoffice/core/commit/?id=1fa761af825641da5c87f80c2a17135f92418960>
> > "Ridiculously large delta for SwarmSolverTest::testUnconstrained for now"
> > and
> > <https://cgit.freedesktop.org/libreoffice/core/commit/?id=0c3444c9bcee093ad5976af8948138e6f2a97706>
> > "Weaken SwarmSolverTest::testUnconstrained even further for now".  The first
> > one has the following in its commit message: "suggestion by Tomaž Vajngerl
> > was: 'Let's adapt the delta for now. Generally anything close to 3 should be
> > acceptable as the algorithm greatly depends on random values.'"
> >
> > Now <https://ci.libreoffice.org/job/lo_ubsan/833/console> failed with
> >
> > > /home/tdf/lode/jenkins/workspace/lo_ubsan/sccomp/qa/unit/SwarmSolverTest.cxx:106:(anonymous
> > > namespace)::SwarmSolverTest::testUnconstrained
> > > double equality assertion failed
> > > - Expected: 3
> > > - Actual  : 94.6605927051114
> > > - Delta   : 0.9
> >
> >
> > Is that also an acceptable outcome, or does it indicate a bug somewhere that
> > would need to be fixed?  What good is a test whose success criterion is the
> > result of ad-hoc guesswork, instead of being determined precisely up-front
> > when the test was written?
> > Can that test please be fixed properly, so that it would be actually useful?
>
> Well, it is neither - that's just the nature of stochastic algorithms.
> It is not the fault of the test - how it was defined at the beginning
> was the exact outcome we would expect (just like a global maximum of
> an function is exactly one value). The problem is that the algorithm
> itself doesn't guarantee to find that solution or comes as near to the
> solution in its allotted time, allotted number of generations or just
> gets stuck in some local extreme value, however this should usually
> happen with a small statistical probability in a normal run of the
> algorithm that has a fast enough CPU.

Then those qualities of the algorithm need to be taken into account when 
writing the test, I think.  A small probability of failure is apparently 
still a problem.  We need tests to be reliable.

> Maybe I'm wrong but I don't see this failing in tinderboxes or
> jenkins, so I wonder what ubsan does to make it fail. The algorithm
> has a time limit, could it be that the execution is slowed down so
> much that the result didn't develop enough (I didn't expect this to be
> so). Could we skip it for ubsan only?

Those ASan+UBSan tinderbox builds execute rather slowly, yes. 
(<http://clang.llvm.org/docs/AddressSanitizer.html> claims "Typical 
slowdown introduced by AddressSanitizer is 2x.")

But also as reported by others today on #libreoffice-dev:

> Feb 28 09:17:32 <buovjaga>        sberg: I got a swamsolver failure yesterday. Then I pulled later 
> and the next build went fine.
> Feb 28 09:19:03 <buovjaga>        After the failure, soffice refused to start. I don't have logs, 
> unfortunately