ssh fails to execute remote command when run from cron bash script - works from CLI

Question

I have a script, written in bash, which runs from cron. One of the first things it does is SSH to a remote host and retrieve a list of files in a directory. Everything works fine when run from the command line, but not from cron.

The section of the script originally looked like this:

FILES=$($SSH_BINARY -i $SSH_KEY $SSH_USER@$REMOTE_HOST "ls $REMOTE_DIRECTORY")
echo "Got files = $FILES"

I added an echo statement above that line (shown below) to prove it wasn't a path or variable issue:

echo "$SSH_BINARY -i $SSH_KEY $SSH_USER@$REMOTE_HOST \"ls $REMOTE_DIRECTORY\""

If I take the resultant output line and run it as the same user cron will (root), it works without issue.

Thinking it may have something to do with assignment to a variable, I modified the FILES= line to read (thus, putting the output directly into my last_run_output file):

$SSH_BINARY -vv -i $SSH_KEY $SSH_USER@$REMOTE_HOST "ls $REMOTE_DIRECTORY"

The cron entry looks like this:

34 * * * * /root/path/to/my/script/get_logs.sh > /root/path/to/last_run_output 2>&1

So, PATH, variable assignment, and permissions issues aside, I started using debug flags in ssh. I ran once from the command line, then from cron, and compared the outputs. Here are some highlights from the diff:

The - side is the unsuccessful attempt, the + side is the successful attempt, run outside of cron.

@@ -87,9 +77,7 @@
 debug1: Remote: X11 forwarding disabled.
 debug1: Remote: Forced command: /home/sshacs/acssshsink netstorageuser
 debug1: Authentication succeeded (publickey).
-debug2: fd 4 setting O_NONBLOCK
 debug2: fd 5 setting O_NONBLOCK
-debug2: fd 6 setting O_NONBLOCK
 debug1: channel 0: new [client-session]
 debug2: channel 0: send open
 debug1: Entering interactive session.

I can't explain why these extra file descriptors are mentioned in debug2 when run from cron, but it appears to be related (notice read<=0 rfd 4 len 0 line below):

@@ -100,20 +88,672 @@
 debug2: callback done
 debug2: channel 0: open confirm rwindow 0 rmax 32768
 debug2: channel 0: rcvd adjust 131072
-debug2: channel 0: read<=0 rfd 4 len 0
-debug2: channel 0: read failed
-debug2: channel 0: close_read
-debug2: channel 0: input open -> drain
-debug2: channel 0: ibuf empty
-debug2: channel 0: send eof
-debug2: channel 0: input drain -> closed
+  [[ Very large output of the ls command ]]
+debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
 debug2: channel 0: rcvd eof
 debug2: channel 0: output open -> drain
 debug2: channel 0: obuf empty
 debug2: channel 0: close_write
 debug2: channel 0: output drain -> closed
-debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
 debug2: channel 0: rcvd close
+debug2: channel 0: close_read
+debug2: channel 0: input open -> closed
 debug2: channel 0: almost dead
 debug2: channel 0: gc: notify user
 debug2: channel 0: gc: user detached

Any ideas are greatly appreciated.

Solution 1

Try adding -t -t to your SSH connection options. This will; force a pseudo-terminal to be allocated.

Multiple -t options force tty allocation, even if ssh has no local tty.

Solution 2

You should debug your script further. Two things to try:

Rwrite it to do file output directly instead of through output redirection. That is, in the script do this:

$SSH_BINARY -i $SSH_KEY $SSH_USER@$REMOTE_HOST "ls $REMOTE_DIRECTORY" > savefile

Run your script with bash -x inside cron and save result. Change first line of script to #!/bin/bash -x and try something like

34 * * * * /root/path/to/my/script/get_logs.sh > script_debug 2>&1

that should help you narrow down exactly what your script is doing. I suspect redirection of output in cron is not working correctly in this case.

Also, dumb question: have you verified that root can write to /root/path/to/last_run_output? Have you verified that you don't have noclobber set somehow when the script runs in cron?

EDIT: further troubleshooting ideas based on comments from OP.

So none of the above ideas seem to be working. What about saving the fielist in a file on the remote machine and scp'ing it back? That would eliminate any weird quoting or input redirection problems.

Comments

[ewindisch]

Is the remote directory the same when run from each? SSH command should be: $SSH_BINARY -vv -i $SSH_KEY $SSH_USER@$REMOTE_HOST ls "$REMOTE_DIRECTORY"

[Kyle Smith]

ewindisch, the command is exactly the same when run from each, as I use the output of the 'echo' line to determine what to run outside of the script. Also, what does '-w' do? My ssh binary doesn't appear to support it.

[SmallClanger]

One suggestion: Can you set your shebang to #!/bin/bash -ex (assuming you use bash), this will echo all executed commands to stdout (-x) before running them and exit if an undefined $VAR is referenced in the script (-e). Useful for debugging this stuff.

[Kyle Smith]

SmallClanger: Very good idea, unfortunately in this case I found nothing unexpected. Seems to do what I expect.. :( I will keep -x in mind in the future though.

[kenorb]

Related: SSH session never closes when executing “apt-get install”

[Kyle Smith]

Phil, thanks for the ideas. I had actually already tried eliminated the output redirection in the cron entry (thinking it may be related to fd4, or something) and it made no difference. I did also try redirecting at that command instead of in cron, same problem. At SmallClanger's suggestion (see OP comments), I tried a -x as well. Nothing unexpected. Also I have write access to that file, it gets replaced with useful debug at every run.

[Kyle Smith]

THANK YOU! We've been looking at this for far too long, we had actually tried the -t option but hadn't read to the part where 'multiple -t's will force a psuedo-terminal even if a local psuedo-terminal is absent'. Thanks again!