Clean way of launching a shell script in background from Jenkins

Question

What's the proper way to launch a script from jenkins, don't get the build hanging, and leave the process running? I can't seem to get it to work. Either the script doesn't run or the build hangs.

If I put in the build's "Execute shell" step bash relaunch.sh & or relaunch.sh > output.log & or nohup bash relaunch.sh &, nothing happens; build finishes, but the process doesn't run. I guess it can be related to Jenkins waiting for the error pipe to close.

If I do nohup bash relaunch.sh 2>&1 > output.log as suggested here, the output is properly redirected, but the build hangs (doesn't finish), and the process dies when I kill the build.

Adding export BUILD_ID=dontKillMe, as suggested here, here, and here, either to the "Execute shell" step or the script itself doesn't help either. The build hangs and the process dies when I kill the build. Needless to say, my knowledge of linux is very limited.

How do people do this in a clean way?

Solution 1

A convenient way to achieve that is to change the environment variable BUILD_ID under Execute shell which Jenkins's ProcessTreeKiller is looking for.

By doing,

BUILD_ID=dontKillMe nohup bash relaunch.sh &

Jenkins will assume that the background job is not spawned by the build and will not kill them after finishing the job.

Thanks to Joshua for his observation that you could also use JENKINS_NODE_COOKIE as

JENKINS_NODE_COOKIE=dontKillMe

Solution 2

I was having the exact same problem. I ended up fixing this by placing the following in the Jenkins execute shell box:

BUILD_ID=dontKillMe ./grid.sh

I moved the & inside the script file. Here's what the script looks like:

#!/bin/bash
java -jar selenium-server-standalone-3.0.1.jar -role hub &

Hopefully this helps someone!

Solution 3

Interesting solutions. Has anyone tried screen?

Here's how I run it (for RTL simulation using ModelSim-ASE):

screen -S $testname -d -m -L bash -c 'cd build_sim ; make; make sim'
echo Waiting for simulator ...

I then wait for our simulator to get to a waiting state by watching the screenlog.0

until ((`fgrep -c 'Ready for simulation' screenlog.0`)) ;do
    sleep 1
done

Then, when the other jobs that need the simulation are done running (or Jenkins detects a failure), in the post, run:

screen -S $testname -X kill

The only caveat is that if Jenkins somehow dies before the job cleans up, in theory you could have a simulator chewing up resources while waiting for something to happen.

To break it down a little: With screen, '-S' assigns a title to the session (or addresses a running session with that title), '-d' runs detached, '-m' spawn a fork first (which hides from Jenkins watchful eyes), and -L turns on logging (which is also handy in case something in the simulator breaks - you now have a log file artifact with the output). The rest is simply bash inside of the screen session.