Re: How are Jenkins builds killed exactly?

Hi Stephan, *,

On Sun, Dec 29, 2019 at 3:17 PM Stephan Bergmann <sbergman@redhat.com> wrote:
> Still trying to track down why sometimes zombie processes survive on the
> (Linux) Jenkins build machines […]
> […]
> That tb_slave_wrapper script contains
>
> > trap cleanup 1 2 3 6 15
> >
> > cleanup()
> > {
> >   echo "Caught Signal ... killing everything...."
> >   # kill everything in same process group (pseudo-pid 0)
> >   kill -9 0
> > }
>
> intended to kill all processes if the script itself receives any of
> SIGHUP/-INT/-QUIT/-ABRT/-TERM.
>
> But how does the tb_slave_wrapper script get terminated by whatever
> entity that starts it and prints out the
>
> > Build timed out (after 15 minutes). Marking the build as aborted.
> > Build was aborted
> > Finished: ABORTED
>
> mentioned above?

Regular abortion should be using
https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller

it looks for processes with the BUILD_ID environment variable that the
scripts inherit when running under jenkins.
that tries a term and kill - although our handler from the scirpt then
should take over.
https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/util/ProcessTree.java#L776

>  Could it be that the script itself gets killed with
> SIGKILL, so its cleanup() trap doesn't fire, and processes (indirectly)
> spawned from the script may stay alive?

Timeout based abortion is done via
https://github.com/jenkinsci/build-timeout-plugin and that triggers
interrupt for the thread.

ciao
Christian

ciao
Christian


-- 
Christian Lohmaier, Release Engineer
Tel: +49 30 5557992-60 | IRC: cloph on Freenode
The Document Foundation, Kurfürstendamm 188, 10707 Berlin, DE
Gemeinnützige rechtsfähige Stiftung des bürgerlichen Rechts
Legal details: http://www.documentfoundation.org/imprint