SSH Protocol mismatch

Question

I have a very strange problem. I have two servers, namely daytona, which serves as a storage server with a raid array. I WOL it when I want to back up to it. The second server is testarossa which runs my services. It is the latter that I want to backup daily using duplicity. Both machines run Ubuntu Server 14.04, fully up-to-date.

I have written a script to WOL the machine and then execute the duplicity backup each day on a fixed time.

The import part of the backupscript is shown below. The backup runs as user root on testarossa and backups over SSH via backupper on daytona. Then it shuts down via ssh using user christophe on daytona.

I have configured ssh keys on testarossa so I can ssh into daytona using backupper and christophe. I can execute the commands from the script just fine, and even execute the script in the shell as well (./script.sh).

I have added the script in the cronjobs using:

0 10  * * * /bin/bash /root/scripts/dailybackup >> /var/log/backup.daily.log 2>&1

Each time the cronjob runs I get the following error:

BackendException: ssh connection to [email protected]:22 failed: [Errno 111] Connection refused

I have, suggested on #ubuntu-server, tried echo "" | nc 192.168.1.120 22 and that returns the following error:

SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3
Protocol mismatch.

This led me to believe that I had to upgrade daytona which I did. There was an upgrade for the gnu-openssl package and then the cronjob ran fine. But now it doesn't anymore.

I am out of ideas on how to debug this. I have too little experience to fix it. Any pointers?

Script

serverip=192.168.1.120
servermac=14:DA:E9:4C:6E:17
attempts=50
sourcedir="/"
targetdir="sftp://[email protected]//mnt/raidarr0/backups/testarossa/duplicity/daily"
encryptkey="AC7A8F8C"
keep="1M"
sudouser="christophe" 
fullbackup=""

## Load in the passphrase file env variable
. /root/.passphrase
export PASSPHRASE

## Do the snapshot backup
if [ "$fullbackup" == "full" ]; then
    $(which duplicity) full --encrypt-key "$encryptkey" --exclude /srv --exclude /usr --exclude /cdrom --exclude /lib64 --exclude /bin --exclude /sbin --exclude /boot   --exclude /dev --exclude /proc --exclude /sys --exclude /tmp --exclude /run --exclude /mnt --exclude /media --exclude /lost+found "$sourcedir" "$targetdir"
else
    $(which duplicity)      --encrypt-key "$encryptkey" --exclude /srv --exclude /usr --exclude /cdrom --exclude /lib64 --exclude /bin --exclude /sbin --exclude /boot   --exclude /dev --exclude /proc --exclude /sys --exclude /tmp --exclude /run --exclude /mnt --exclude /media --exclude /lost+found "$sourcedir" "$targetdir"
fi

echo "Backup to target completed."

## Remove older backups. We only want to backup 30 days. 
# (We have a full every month)
$(which duplicity) remove-older-than "$keep" --force "$targetdir"
echo "Removal of stale backups completed"
## Shut down the machine using a sudo account. Expects the user to have a key installed for this.
ssh "$sudouser"@"$serverip" "sudo shutdown -h now"
echo "Shutdown command issued to remote machine"

Follow up:

1) The script has a function which waits for the host to be ping-able. So it only starts backing up when the host has fully booted. (This script ran fine for over a year on a different machine with Debian.)

2) The script runs fine in the shell of the root indeed.

3) And no, I do not have a proxy command in either setting files.

4) I have tried running the command using sudo /bin/bash /root/scripts/dailybackup and now, for some reason, it asks me to verify the authenticity of the host (with yes/no). So now it seems like the duplicity command is not using my known_hosts file?

Solution

The error "Protocol mismatch." is very simple: nc doesn't use the ssh protocol to connect to the remote address and port.

Regarding the actual problem: is backupper user connecting via ssh keys? If so, where are the keys kept?

What I think is happening is that you test your script with another user than root.

Try to run your script as root like this and see if it fails also:

sudo /bin/bash /root/scripts/dailybackup

Others had the same error because of IP conflict

Comments

[Stéphane Chazelas]

No, the protocol mismatch is just because "echo" doesn't speak ssh protocol (sshd is confused by that LF character echo sends to it). The fact that sshd outputs a banner is inconsistent with your "Connection refused" error though. Do you have a ProxyCommand configured in /etc/ssh_config or ~/.ssh/config?

[Lambert]

Is the script to fast? In other words, is the backup being performed before the host is fully started and initialized? A connection refused message may appear if the sshd service is not started/operational yet on the target host.

[Lambert]

That a machine is 'pingable' does not mean that network dependent services are initialized yet. To test if sshd is running, consider using nc -vz 192.168.1.120 22

[Christophe De Troyer]

I will temporarily test it with a sleep of 1 minute.

[Lambert]

Besides, if the backup host is not in the known_hosts for the backup user, copy the entry from the root user to the backup user or su into the context of the backup user and try to ssh to the backup host to get the entry into the known_hosts file of the backup user.

[Christophe De Troyer]

I have done all that before. So that is one of the reasons I'm assuming that duplicity has its own known_hosts file.

[Christophe De Troyer]

See follow up. The machine now asks me to verify the key. How odd. These should be in the known_hosts file..

[cristi]

add those options to your ssh command and check again "-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"

[Christophe De Troyer]

They keys are kept in the user's respective folder as one would expect. So root has his public key in /home/(christophe|backupper)/.ssh/authorized_keys . So yes, via ssh keys.

[cristi]

The last error is not about ssh keys, but because the system doesn't trust the ssh host. You either run once the command and reply with "y" when prompted, or use the above parameters for ssh to trust all hosts