Re: gerrit-buildbot master builds

Hi,

On Mon, Mar 25, 2013 at 04:43:49PM -0500, Norbert Thiebaud wrote:
> On Mon, Mar 25, 2013 at 3:45 PM, Bjoern Michaelsen
> <bjoern.michaelsen@canonical.com> wrote:
> > Hi,
> > On Mon, Mar 25, 2013 at 01:04:56PM -0500, Norbert Thiebaud wrote:
> > > 1/ I have been working on a new script that can monitor both in the
> > > same instance...
> > > iow do tinderbox and/or gerrit build as needed, in one script instance
> >
> > Yes, but I want to get rid of the legacy 'tinderbox builds' as much as
> > possible,
> why ?

To unify as much as possible on one build dispatch.

> Not a good idea. tinderbox build are progressive so you can take
> advantage of incremental build. things you cannot do with a gerrit
> build since the base of a gerrit patch can be all over the place...

There is no defined direction of time in the build system. Going forward is not
different from going backwards.

> and incremental build kind of work when moving progressively
> forward... but is not expected to work if you start to checkout random
> point in master

It would only break if the dependencies are wrong, and then it would break in
both directions.
 
> basic tinderbuild can be done without any kind of authorization or
> external setup... whereas gerrit based build need a registered builbot
> user in gerrit.
> iow a much higher barrier of entry.

Yes, but once a machine is on the buildbot queue, its already past that barrier
-- so no need to bother about that beyond that point.

> I have plan to use cron to 'trigger' build for stuff that we do not
> want to continually build.. but the cron job would just touch a
> semaphore file so that the one instance of tb that run detect that it
> is time to try _that_ kind of build.

Hmm, I dont quite understand how this is different from what I propose, which
is: whenever possible, builds should be triggered through the buildbot
scheduler queue as that allows for most coordination. 

> so for instance you can have a tb scrip that build gerrit patch +
> master continuously... and run a full-lang build once a week and a
> valgrind perf regression build once every 3 days... the later two
> being triggered by cron using a simple touch <tigger_file>  command

The first is different from the second and the third. The first can and should
notify committers about breakage (thus pushing back the issue on those
responsible), while the second and the third cant do that: Even if they would,
with build intervalls of more than 1 day, email notifications would be
cheerfully ignored. So in the latter case either the owner of the box has to
hunt down the issue himself, or the state has to be published at a conventient
place (like a jenkins page) showing the current state. And even then I wouldnt
be too optimistic about others than the owner to take much care for them.

As such, the first is what fits into the gerrit buildbot queue, while the
latter is a completely different beast. As such the first can and should be
piped through the buildbot queue, while the latter needs other way of
publishing. Integrating the first in the gerrit buildbot queue also centralises
the information there: E.g. it allow to check if a patch verification failed
because of the patch, or because of the master state it is based upon.

> gerrit patch verification are reduced to 1 trusted per platform for
> ressource resons...

That will change.

> and tinderbox should be able to be brought online/offline... reseted,
> changed etc. at the tb owner discretion...
> even be able to run without repporting to the tb web-site

Doesnt this apply to the gerrit-buildbot just the same? You take your builder
of the queue, it doesnt pickup anything out of the queue anymore. (Thus you
could even run those longer intervall checks mentioned above then -- although
we have no good ways to report the results for that yet:
tinderbox.libreoffice.org is not suited at all for that).

> well, 'non-registered' tinderbox are the rule not the exception...
> there are only 3 box today that are 'regsitered' in gerrit... the one
> that do gerrit buildpatch verification...
> that leave 20+ that are 'unregistered'...

If a tinderbox is sufficiently 'default' for a platform, it will be registered.
If its not, it will -- lets face that -- only be watched by the owner (of set
of owners) anyway. Actually I would argue that these boxes in the most cases
produce more harm than they do good. By the sheer amount of noise they generate
they are scary to new or casual contributors and the fact that -- because
the tinderboxes are uncoordinated -- by breaking the master once you can get
20+ tinderboxes yelling at you, possibly long after the issue is fixed isnt
helpful either.

>  but for the rest there is no need or benefit to deprive ourself of the
>  occasional/part-time tinderbox, or make it more complicated for people to
>  play with a tinderbox for a while on their own... without requiering any
>  approval, control or admin involvement.

Of course, running a tinderbox should be easy. But I actually very much like
the idea that there should be a barrier to entry for emailing huge sets of
committers.

> iow: Please explain what is the goal and justification to add
> complexity and fragility to something that is not broken.

There are two separate issues. One is patch verification and closely watching
master with quick turnarounds. As explained above these are closly related and
having the information in one place is hugely advantagous.

The second is slower checks and tests, like full-lang builds and valgrind perf
checks. Both of our current notification methods (email galore and
tinderbox.libreoffice.org) are not well suited for that scenario. So we would
need to find a better way to solve that, _but_ thats for another day as it is
mostly independant of the first issue.

Best,

Bjoern