Run a remote script/application in detached mode in Ansible

Question

I am having trouble running a remote application startup script "detachedly" from an Ansible playbook. The script will run, but I can't get it to get/stay detached. I am probably doing something wrong, but what?

Here is my reproducer.

1. My remote Java application in Test.java runs for 10 seconds:

   class Test {
       public static void main(String[] args) {
           for (int I = 1; I <= 10; i++) {
               System.out.println("Hello world " + I);
   
               try { Thread.sleep(1000);
               } catch (Exception e) { System.out.println("Exception caught: " + e);
               }
   } } }
   

Compiling this to Test.class (javac Test.java), then running that class with "java Test" works as expected (gives 10 output messages and then exits).

2. My executable shell script (as in chmod 755) running this application looks as follows:

   #!/bin/bash
   java Test &
   

Running this manually is also perfectly fine: Java app runs and generates the same standard output in my console, but shell script has exited and I am back in control of my bash session.

3. Now to run it through an ansible playbook from another server. I tried using the "command" module and the "shell" module in different ways but to no avail...

   ---
   - hosts: vagrant1
     gather_facts: false
   
     tasks:
     - debug: msg="Running test app through Ansible shell module..."
   
     - name: Start application
       shell: "/tmp/test.sh"
       args:
         chdir: "/tmp"
         executable: "/bin/bash"
   
     - debug: msg="Running test app through Ansible command module..."
   
     - name: Start application
       command: "nohup /tmp/test.sh &"
       args:
         chdir: "/tmp"
   

All of this runs just fine, i.e. the shell script runs, the Java application runs and does its thing (i.e. run for 10 seconds, generate output and quit). But ansible-playbook runs until the Java application has finished and then returns the output the Java application generated. Why won't it just detach the shell script and finish the playbook?

The ansible-playbook output is:

    monsterkill@monsterkill-ub-dt:~/playbooks$ ansible-playbook testrun.yaml -v

    PLAY [vagrant1] *************************************************************** 

    TASK: [debug msg="Running test app through Ansible shell module..."] ********** 
    ok: [vagrant1] => {
        "msg": "Running test app through Ansible shell module..."
    }

    TASK: [Start application] ***************************************************** 
    changed: [vagrant1] => {"changed": true, "cmd": " /tmp/test.sh ", "delta": "0:00:10.052927", "end": "2015-01-29 00:14:43.327418", "rc": 0, "start": "2015-01-29 00:14:33.274491", "stderr": "", "stdout": "Hello world 1\nHello world 2\nHello world 3\nHello world 4\nHello world 5\nHello world 6\nHello world 7\nHello world 8\nHello world 9\nHello world 10"}

    TASK: [debug msg="Running test app through Ansible command module..."] ******** 
    ok: [vagrant1] => {
        "msg": "Running test app through Ansible command module..."
    }

    TASK: [Start application] ***************************************************** 
    changed: [vagrant1] => {"changed": true, "cmd": ["nohup", "/tmp/test.sh", "&"], "delta": "0:00:10.045643", "end": "2015-01-29 00:14:53.557164", "rc": 0, "start": "2015-01-29 00:14:43.511521", "stderr": "nohup: ignoring input", "stdout": "Hello world 1\nHello world 2\nHello world 3\nHello world 4\nHello world 5\nHello world 6\nHello world 7\nHello world 8\nHello world 9\nHello world 10"}

    PLAY RECAP ******************************************************************** 
    vagrant1                   : ok=4    changed=2    unreachable=0    failed=0

Solution 1

Just use an asynchronous action with poll: 0:

- command: yourscript.sh
  async: 45
  poll: 0

These options are needed to prevent ansible from killing the child processes of that task with os.killpg.

Then in yourscript.sh:

java Test &
disown

disown removes the process from job table, so SIGHUP does not get sent to it. nohup does the same, but is not necessary.

Edit: Note that disown is not available in all shells (eg. dash)

Solution 2

Running your script as a Daemon with start-stop-daemon seems like a more elegant solution here. http://manpages.ubuntu.com/manpages/raring/man8/start-stop-daemon.8.html

- name: Start application
  shell: "start-stop-daemon --start --quiet --pidfile /var/run/test --exec /tmp/test.sh"
  args:
    executable: "/bin/bash"

Solution 3

The google servers must be smoking by now, but I found a solution. When I realized Ansible executes all its remote commands through separate ssh calls, I tested by doing that manually from the ansible server and found that it has to do with ssh functionality.

The answer is, apparently, to have the script that I called to do the double background trick, combined with a nohup to make it ignore hangup signals (SIGHUP) and combined with disconnecting the stdout and stderr streams. So, the remote startup script no longer does:

java Test &

... but instead now does:

( ( nohup java Test 1>/dev/null 2>&1 ) & )

which does what I want. The ansible-playbook now fires up the remote script which in turn fires up the java application, but then double-backgrounds and nohups it and disconnects the output streams.

This stackoverflow thread helped me.

Comments

[David Richerby]

Please give all the relevant information in your answer. External links are great for references and further reading but they shouldn't be the whole content of your answer.

[Mrunal Gosar]

Hi what should be done to run a command indefinitely in background and not block ansible? async option allows script to run only for 45 seconds and then terminates it i want to run the command indefinitely i am already using nohup command & option but the command terminates the moment async time is out i've set poll to 0

[Kassav']

Hi, i'm facing the same issue, and async tasks works well. However, i didn't find a clear explanation, also , i didn't found the code part of ansible that kills a child process. Any help ?