known_hosts not updating, but ssh says it added

Question

When I log in from termserv to any other servers, ssh asks me to verify the host fingerprint. Than it warns me that it added it and moves on. If I immediately log out and log back in, this happens again. I tried blanking the known_hosts file, checked permissions and tried again. The known_hosts file stayed blank. There is a notable amount of pause between the report that it added the fingerprint to known hosts and being asked for my key passphrase.

me@termserv:$ ssh [email protected]
The authenticity of host '10.0.X.X (10.0.X.X)' can't be established.
ECDSA key fingerprint is d4:a2:cf:42:0b:01:xx:e5:xx:7a:xx:93:xx:53:xx:b4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.X.X' (ECDSA) to the list of known hosts.
Enter passphrase for key '/home/me/.ssh/id_rsa':
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-23-generic x86_64)
...
[email protected]:$

I checked drive space issues. On my primary partition (there is no special /home) I have only 3% usage. My syslog doesn't show anything related. If I run ssh with -vvv I don't get any extra information between it asking if I want to connect and warning me that it is permanently added to known hosts (which we've determined is a lie).

I've tried a search for some of those phrases in the OpenSSH webCVS repo, but didn't find it via google site: searching.

.ssh directory permissions are me:me drwx------

.ssh/known_hosts permissions have been me:me -rw-r--r-- and -rw-------

Here is an strace of what's going on. I only included the part between me saying yes to add it and it saying it had added it.

strace -o sshtrace ssh localhost
write(4, "The authenticity of host 'localh"..., 200) = 200
read(4, "y", 1)                         = 1
read(4, "e", 1)                         = 1
read(4, "s", 1)                         = 1
read(4, "\n", 1)                        = 1
rt_sigaction(SIGALRM, {SIG_DFL, [], SA_RESTORER, 0x7fc73374d4a0}, NULL, 8) = 0
rt_sigaction(SIGHUP, {SIG_DFL, [], SA_RESTORER, 0x7fc73374d4a0}, NULL, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fc73374d4a0}, NULL, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7fc73374d4a0}, NULL, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [], SA_RESTORER, 0x7fc73374d4a0}, NULL, 8) = 0
rt_sigaction(SIGTERM, {SIG_DFL, [], SA_RESTORER, 0x7fc73374d4a0}, NULL, 8) = 0
rt_sigaction(SIGTSTP, {SIG_DFL, [], SA_RESTORER, 0x7fc73374d4a0}, NULL, 8) = 0
rt_sigaction(SIGTTIN, {SIG_DFL, [], SA_RESTORER, 0x7fc73374d4a0}, NULL, 8) = 0
rt_sigaction(SIGTTOU, {SIG_DFL, [], SA_RESTORER, 0x7fc73374d4a0}, NULL, 8) = 0
close(4)                                = 0
open("/dev/null", O_WRONLY|O_CREAT|O_APPEND, 0666) = 4
fstat(4, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7ffff63c12c8) = -1 ENOTTY (Inappropriate ioctl for device)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc734b64000
fstat(4, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
lseek(4, 0, SEEK_END)                   = 0
write(4, "|1|qcJVRUE6IlxxxxxBvjBgHiiov4/8=|"..., 222) = 222
close(4)                                = 0
munmap(0x7fc734b64000, 4096)            = 0
write(2, "Warning: Permanently added 'loca"..., 76) = 76`

Solution

The SSH client thinks your known_hosts file's path is /dev/null:

open("/dev/null", O_WRONLY|O_CREAT|O_APPEND, 0666) = 4
[...]
write(4, "|1|qcJVRUE6IlxxxxxBvjBgHiiov4/8=|"..., 222) = 222

Check ~/.ssh/config and perhaps /etc/ssh/ssh_config or equivalent; it seems likely one or the other contains something bogus.

Comments

[Flup]

Can you try strace -o ssh [email protected] and post lines that relate to your known_hosts file?

[september]

Try to use -vvv: ssh -vvv [email protected]. It is more verbose (level 3) mode.

[flickerfly]

Even if I use -vvv it doesn't give any further input between the "Are you sure" and the "Warning:" line.

[flickerfly]

@Flup, I added the part of the strace I expect is relevant to the question.

[Kun]

In my case, it was because the write permission was set to 0 on the know_hosts file

[flickerfly]

Yep, turns out /etc/ssh/ssh_config has " UserKnownHostsFile /dev/null" in it. Why that changed, I don't yet know. It might have been when I was playing with a puppet module.

[alpha]

Never seen puppet do anything like that. Sounds more suspicious than anything else. If I didn't want a record of where I connected, that would be one way to do it. And outgoing connections aren't logged. (Incoming are though). You might look at your logs for weirdness involving ssh, sudo (to edit ssh_config).

[Kjeld Flarup]

Also check if your ssh command is aliased to use these options: alias ssh='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'