Re: gerrit-buildbot master builds

On Mon, Mar 25, 2013 at 5:46 PM, Thorsten Behrens
<thb@documentfoundation.org> wrote:
> Norbert Thiebaud wrote:
> > > so I want all builds be scheduled through the queue with the builder
> > > being ignorant what kind of build he does. Make the client as dumb as possible.
> >
> > Not a good idea. tinderbox build are progressive so you can take
> > advantage of incremental build.
> The scheduler would know that, so it could instruct the builder to do
> the right thing?

no it would not... since it does not know what has happen on the
tinderbox and in which state it is...
the 'scheduler' has not memory and cannot keep track of the state of
the machine... for all it know I've just blown-up my git repo and
cloned it for scratch... or I did a manually build in the same
build_dir for whatever reason.... or anyone of dozen of scenario that
would change the state of the build directory...

> basic tinbuild should ideally come with nightlies IMO, so that is
> debatable.
No. no reason to upload nightly for every combinaisons. for instance
my Gentoo linux box never uploaded nightlies, because pacakging is
unreliable on that box due to rpm.
and there are legitimate use of tinderbox that do _not_ take the cost
of packaging, because their goal is to test something else...

> > so for instance you can have a tb scrip that build gerrit patch +
> > master continuously... and run a full-lang build once a week and a
> > valgrind perf regression build once every 3 days... the later two
> > being triggered by cron using a simple touch <tigger_file>  command
> Wouldn't that be much better handled from a central master scheduler,
> that has a much broader view on what is needed and when?

No because tinderbox machine belong to their owner... _they_ decide
what run on their box and when not a 'central master schedule'
The model for tinderbox is completely decentralize, and for gerrit
buildbot is client-server... not master slave.
If someone what to run a full bugzilla regression on a regualr basis
or on demand (doing the touch by themself when they feel like it)
the 'so-called' global scheduler has nothing to do with it...

in any case is is _much_ simpler to manage what a box schedule is on
each box than try to have a 'global scheduler'
How are you going to manage autogen.lastrun from a remote centralize
server... what happen when I run emerge --update
and suddently I need to adjust the autogen, because the system libs is
not compatbile anymore, etc....
No, tinderbox need to be 'managed' by their owner(s), and they should
have some autonomy to do that on their own, without having to involve
a gerrit admin to tweak/fix things.. think 'bazaar' not 'cathedral'.

> > box that will do gerrit patch verification will need to be
> > registered in gerrit indeed. and we want to have some level of
> > control and/or confidence in the operation of these boxes... iow we
> > would _rely_ on them to work well and work reliably...
> I would consider a not-so-well maintained tinderbox a nuisance in any
> case, if it spams committers with false positives.

1/ such box maybe 'unreliable' because they are not always online...
that does not necessarily mean that they are not-so-well maintained,
just not dedicated.
2/ it is quite possible not to spam people and still report build to
the web site, heck one can even, today, setup a box that only spam
oneself.

> > so yes the barrier of entry will be higher, and eventually I expect
> > most if not all of these to be TDF operated. but for the rest there
> > is no need or benefit to deprive ourself of the occasional/part-time
> > tinderbox, or make it more complicated for people to play with a
> > tinderbox for a while on their own... without requiering any
> > approval, control or admin involvement.
> That does not follow, if all that is needed is a ssh key on the
> builder?

no what is need is creating a special 'bot user' in gerrit in the
right group, and associate a ssh key with that user...
that is an admin task. (no cannot be done from the web... nor should
it be desirable)

> > The other consideration is that the tinderbox system today works
> > independently of gerrit... so if gerrit were to go down hard for a
> > significant amount of time... we could fairly easily fall back to a
> > git mirror and continue working without gerrit for some time...with
> > still tinderbox verification.
> That is the one point in your email I agree with. :)
>
> > iow: Please explain what is the goal and justification to add
> > complexity and fragility to something that is not broken.
> Not speaking for Bjoern of course - but my ideal outcome would be a
> set of distributed build slaves,

And here is a crux of the misunderstanding: neither the tinderboxes
not the gerrit-patch-verification system is master-slave.
Master-slave require the master to have control on the box... to
access it remotely and decide what run and when on the slave...
that is not a viable option for volunteer boxes.
If that where the chosen model then we would have to restrict
operation to dedicated TDF owned box.
There may be a argument to be made for such a master-slave system to
be setup to do 'official' tinderboxing... but that should not
'replace' the current
volunteer based tinderboxing... it should at most supplement it.


> that are more flexible than the
> current tinbuild system. You cannot shutdown a builder that is
> misbehaving, you cannot re-purpose it to 4-0-3 if somehow noone else
> builds that branch, you cannot have them build feature branches for
> specific setups, you cannot instruct them to bisect a breakage that
> occured just on their niche etc etc - and you have to poke their
> owners each time tinbuild2 requires an update. Well, you might
> consider that an advantage. ;)

I do, to the extent that that model allow me to use spare cycle on box
that would not otherwise be able to be part of a master slave scheme.
for instance I have built Mac tinderbox and release build on my main
machine, that I use for everyday purpose, for 2 years... (I just
changed things...)
and I would never have been able to do that in a master slave system,
as there is no way I'll give ssh access to that box to anyone. (heck
not even myself, since there is no incoming open port on that box at
all, not even ssh)

> Part of the underlying reason is also the question of *what exactly*
> is the canonical TDF build. Default configure, or release build, or
> some temporally varying mix? Or as of today, whatever the tinderbox
> sponsor deems valuable?

Again TDF owned and operated box _can_ be setup that way... although
I'd argue that the notion of a centralize scheduler is not a realistic
way to address the problem you mentioned earlier: if a box misbeave,
it is unlikely that a 'master scheduler' could diagnose what is wrong
and fix it... that will almost always require a person to log-on and
figure it out (otherwise if it was manageable by an automate, then why
did it mis-behave to start with, it should fix itself or not fail to
start with).
and we get into the realm of 'puppet' and other admin management... I
fail to see the wisdom in trying to built that in a gerrit plugin.

So, provided the necessary dedicated hardware, then sure, we could
have a whole scheme based on that model... but at that point the whole
plugin is pointless since you can then simply use vanilla gerrit and
jenkins to acheive that. the whole point of the gerrit plugin was to
allow the client-server scheme.

Norbert