Re: probabilistic approach to tinderboxing

On Tuesday 12 of June 2012, Bjoern Michaelsen wrote:
> Hi all,
>
> this is a proposal to get as much focused tinderboxing as possible with
> limited resources and unreliable build clients. This also assumes patches
> under consideration to be independant (that is: without conflicts) and not
> dependant on a specific order. gerrit allow us to have patches independant
> of order as they are not yet merged/picked to the branch. We can also
> easily exclude any patch on gerrit that conflicts against the existing
> patches under
> consideration: To do so we cherry-pick the new patch on the tip of the
> branch and then cherry-pick all patches under consideration on top of that.
> If it conflicts, we can report that back.(*)
>
> == Testcase ==
>
> Tinderboxes will be grouped in "testcases": All tinderboxes in belonging to
> one "testcase" are assumed to provide the same results for the same input.
> The simplest testcase is: "completing a build on platform foo" while more
> complex testcases include "completed a build on platform foo and run
> checks".

 Where are we going to get these tinderboxes? The page for master lists 17 
tinderboxes, out of which only about 11 build regularly. Out of which there 
are 4 linux-gcc tinderboxes, one linux-clang, 1(2?) macosx, 3 windows and 1 
mingw. That, not counting any further division like running or not running 
checks, gives 5 testcases if I understand it correctly, and building 
successfully in any of them does not guarantee that the rest builds as well.

 So, the thing I do not understand, how is this (complex, as far as I can say) 
setup supposed to improve anything? There does not seem to be any big 
difference between dedicating X tinderboxes to it or just 1-2 tinderboxes. If 
we put a number of tinderboxes on this, they still won't test the patches 
enough, while there won't be that many left for checking that the actual 
repository builds fine, which will be needed anyway.

 If patches in gerrit[*] need to be tested, it should work just as well to use 
one of the really fast tinderboxes for checking that it builds, which will 
cover a majority of possible problems, then maybe one more can build 
including checks, and if any problem gets through, it'll be caught by the 
normal tinderboxes, which are needed anyway (for direct commits, some other 
seemingly unrelated commit breaking the patch meanwhile, etc.).


[*] Speaking of which, when will we get told about this gerrit thing, besides 
random remarks, or have I missed it?

-- 
 Lubos Lunak
 l.lunak@suse.cz