On 10/06/15 20:22, Norbert Thiebaud wrote:
- a test auto-induced one: when a test is unstable and produce random
failures based on circumstances... the infamous 'heisenbugs'
and heisenbug can be a systemic/design problem or can be a real bug
that is hard to trigger. either way these are not useful, and in fact
harmful in a ci context; because the human nature is 'If you can't
reproduce it is not a bug'
so the later category of real hard to trigger bug is always labeled
'systemic error' and ignored anyway... and it make people numb to
errors...
For automated testing, trust is paramount: heisenbug test failure are
the enemy, false non-failure is bad but actually less painful
Would it make sense to have a server dedicated to Heisenbugs? If a test
triggers a heisenbug, disable it on most of them but try to instrument
the heisenbug tester up the wazoo so that when it fails, there's a pile
of logs to try and work out what went wrong.
And of course, it doesn't spam failure reports generally unless someone
asks for them (the person to whom the bug is assigned should get them,
of course), but it does save those logs for forensic analysis.
Cheers,
Wol
Context
- Re: test infrastructure ideas appreciated ... (continued)
Privacy Policy |
Impressum (Legal Info) |
Copyright information: Unless otherwise specified, all text and images
on this website are licensed under the
Creative Commons Attribution-Share Alike 3.0 License.
This does not include the source code of LibreOffice, which is
licensed under the Mozilla Public License (
MPLv2).
"LibreOffice" and "The Document Foundation" are
registered trademarks of their corresponding registered owners or are
in actual use as trademarks in one or more countries. Their respective
logos and icons are also subject to international copyright laws. Use
thereof is explained in our
trademark policy.