Unable to connect to the AWS EC2 instance - "Host key verification failed"

Question

I had set up a Ubuntu instance with a Rails package, deployed my app, and it is working fine.

But when I try to do SSH, it's not allowing me for the remote login and throws errors like: Host key verification failed.

The problem seem to be persistent. I have attached the Elastic IP to that instance and I am not able to see the public DNS.

My instance is running in Singapore region.

ssh debug output:

OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 46.137.253.231 [46.137.253.231] port 22.
debug1: Connection established.
debug1: identity file st.pem type -1
debug1: identity file st.pem-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.5p1 Debian-4ubuntu6
debug1: match: OpenSSH_5.5p1 Debian-4ubuntu6 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-7ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is.
Please contact your system administrator.
Add correct host key in /home/ubuntu/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/ubuntu/.ssh/known_hosts:1
  remove with: ssh-keygen -f "/home/ubuntu/.ssh/known_hosts" -R 46.137.253.231
RSA host key for 46.137.253.231 has changed and you have requested strict checking.
Host key verification failed.

Solution 1

When you connect to a ssh server your ssh client keeps a list of trusted hosts as key-value pairs of IP and ssh server finger print. With ec2 you often reuse the same IP with several server instances which causes conflict.

If you have connected to an earlier ec2 instance with this IP, and now connect to a new instance with the same IP your computer will complain of "Host verification failed" as its previously stored pair no longer matches the new pair.

The error message tells you how to fix it:

Offending RSA key in /home/ubuntu/.ssh/known_hosts:1
remove with: ssh-keygen -f "/home/ubuntu/.ssh/known_hosts" -R 46.137.253.231"

Alternative simply open /home/ubuntu/.ssh/known_hosts and delete line 1 (as indicated by the ":1").

You can now connect and receive a new host verification.

Please note usually ssh's known_hosts file usually have stored a second line pair for hostname or ip6 value so you might need to remove a couple of lines.

Warning: Host verification is important and it is a good reason why you get this warning. Make sure you are expecting host verification to fail. Do not remove the verification key-value pair if not certain.

Solution 2

@flurdy's answer is good as a one-off resolution.

But if you often:

• launch new EC2 instances,
• start and stop EC2 instances,

..without using Elastic IPs (permanently attached to your servers) then you deal with new/changing IPs/hostnames of your instances all the time.

If so then you may want to permanently stop SSH checking and storing server fingerprints for EC2 public hostnames.

To do that just add this to your ~/.ssh/config:

# AWS EC2 public hostnames (changing IPs)
Host *.compute.amazonaws.com 
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null

Please note that SSH will still say Warning: Permanently added (...) to the list of known hosts. when connecting but just means that it has added it to /dev/null...

SSH will however stop asking if you confirm the authenticity of host and just continue connecting.

So it is more convenient and you may avoid not always verbose enough SSH connection errors while using your EC2 instances.

I have to add that in theory this setting lowers security of your SSH connections but in the real life you probably wouldn't check the fingerprints of your one-off EC2 instances anyway.

Comments

[David Schwartz]

You have to tell us the exact actual errors you got. Telling us what one of the errors was like is not helpful.

[Najeeb]

I did that. Now I am getting this error: Permission denied (publickey). Any idea how to resolve this, because the public key file was good enough the last time I accessed my AWS server.

[enharmonic]

@Najeeb you're trying to login to an EC2 instance that isn't yours anymore, since the original IP address has been reallocated after the instance was stopped.