Differing SSH known_hosts formats

Question

I've been having issues with my CI server's deployment lately due to the client (CI) rejecting the remote's host key (despite it being present in known_hosts). I was stumped until today, when I realized that SSH was saving host keys in a format that the deployment plugin doesn't seem to be compatible with. For reference, the compatible format (still present on my personal machine) resembles this:

11.22.33.44 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkVf7rhfC7nLxbeIQRj2bWitUC+XLSAeQ0ap8r8rKObDXYfPdB97NZth9JCEt3OrBXuBeg4PaAEuPu2QF7WXoT60hgAP6etr0W4LqcH59yd/X0ogFP7Y7hIf6dz1txDKaW92wgUi5XShwH6vukf0gLvW6/ak1LTBuoy72gaoUvxZge4KZivz9XqvSQHNOG9KYNfh8U6cRM8YTQo5in7YD5d6REV/FUmXpvBzCa9kbVRSlQFGYEc1HidTnPnJDteas3A9y3na385O7WN64aAkg7TO8IFXKdDHSwji9ZyrCVPA5GEuyLKhDFanV8iJ7CNflHMP8TwG5FOT2bSkV0lPyl

While the format SSH is currently saving when accepting new host keys resembles this:

11.22.33.44 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJJEs165NgdEcD94Xg3ySFA/qgkfytxNCX1X3pB2SPgU/mHLGXCXM8+VqMBXocM8OMOq2L0fDGr5mI+nGqjhNU=

(Note: while I fudged the public keys a bit, they still don't resemble each other at all in their original form.)

Only the first format is compatible with the deployment plugin, while the second is unconditionally ignored. Can anyone explain this discrepancy?

Solution

These are not different formats of known_hosts, but different key types (ssh-rsa and ecdsa-sha2-nistp256 - well described on the manual page for sshd). The server usually has more host keys of different types to provide wider compatibility with different clients.

If you are on the server, you can find all the host keys and print their public keys using (but the line is not in the same format):

$ cat /etc/ssh/ssh_host_*.pub
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJJEs165NgdEcD94Xg3ySFA/qgkfytxNCX1X3pB2SPgU/mHLGXCXM8+VqMBXocM8OMOq2L0fDGr5mI+nGqjhNU= user@host

The format that is accepted by known_hosts file can be obtained using (from the server to achieve the authenticity of the keys):

$ ssh-keyscan 11.22.33.44
11.22.33.44 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJJEs165NgdEcD94Xg3ySFA/qgkfytxNCX1X3pB2SPgU/mHLGXCXM8+VqMBXocM8OMOq2L0fDGr5mI+nGqjhNU=
#[...]

This prints the format you can directly store in the client' known_hosts file.

For the whole picture (from the manual page):

Each line in these files contains the following fields: markers (optional), hostnames, keytype, base64-encoded key, comment. The fields are separated by spaces.

Comments

[TOOGAM]

The format I'm used to is: IP address, space, keytype, space, four letter As, and some other stuff (which often starts out similar, but eventually contains the raw key data and is different). So both of your examples are following the same format. With ecdsa-sha2-nistp256 I notice another AAAA and a later AAABBB seems common. Your solution: get the plug-in to be upgraded to support the newer key type. (You want your plug-in to be regularly updated, or else this may be an ongoing issue as new keytypes become common.)

[Jakuje]

@TOOGAM No, the first one has IP address prefixed. Key data has to be same if you want to have it working (and why to encode the same data in two different ways?).

[dave_thompson_085]

Late but silly-necroed: the format you quote (bits, exponent, modulus) was for protocol 1, and obsolete since about 2000. When this was posted the sshd manpage section on authorized_keys format correctly described both protocol 1 and 2 formats, but for known_hosts it only described 1. OpenSSH 7.6 in 2017 (finally) removed the protocol 1 logic, and fixed the manpage to describe protocol 2 in both places. Also ssh-keyscan by default doesn't include a DSA (aka ssh-dss) key, although the OpenSSH people consider DSA deprecated and since 7.0 in 2015 it is disabled by default.

[Jakuje]

@dave_thompson_085 yes, I know. Thanks for reminder. We already fixed that in the manual page to refer to current format so I will fix it in this answer too.