ssh connection refused only from my mac, my linux box connects without issue

Question

I am having a strange issue. I have an EC2 server (Arch Linux) that I am able to access (via ssh) from my local linux server without issue, however when I try to ssh into my EC2 server from my macbook, I get a connection refused.

$ ssh -vvv -i key.pem [email protected]
OpenSSH_5.6p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /etc/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to myserver.com 184.72.xxx.xx port 22.
debug1: connect to address 184.72.xxx.xx port 22: Connection refused
ssh: connect to host myserver.com port 22: Connection refused

I also have work clients using EC2 for some of their servers, and I have the exact same issue. I can log into those EC2 machines from other boxes, but not my macbook. I am able to ssh to other servers from my macbook, both locally and out over the network. Which means that while there may be some issue with my macbook, I am still able to ssh into other boxes. I am also able to visit websites that I am serving from my server on my macbook, so the server isn't blacklisted on my macbook, as far as I can tell. This is not the case for all EC2 boxes. I set up a test instance with the same key on my EC2 account, and I was able to ssh into that from my macbook.

Since I am able to connect to my EC2 machine from another box on my local network it rules out that ssh is not running on the server, that the port may be blocked, and that my ip might be blacklisted on the server side. If I run a tcpdump while trying to ssh or nc in from my macbook, I get nothing happening from my local IP address. It seems like the server is not even seeing my attempts. I also see no output in /var/log/auth.log for my macbook's attempts, while other attempts are logged.

I have created a new key on the server and copied the private key back to my macbook (tested elsewhere) and that failed to get me in. I have checked iptables (shut iptables down and tried to connect), /etc/hosts.deny (empty) and the security group, where ssh (port 22) is wide open. On my local network, I have swapped out my router since this issue began, but that didn't help. The issue seems to have happened around when I upgraded my mac to Lion and installed a new hard drive, keeping the same user directory. I am not sure if the problem lives on my mac, or on EC2's end, but I am pretty stuck at this point, and since there are two separate EC2 boxes that I can't seem to get into (one CentOS and one Arch Linux).

I have also tried connecting from my macbook while on another network; same results. I recompiled openssh and installed it in /opt/openssh, tried running it from that location with a couple different keys without luck. I am using ssh-agent, and have tried dropping all keys, and explicitly identifying the key that I am going to use; same results. If this was simply a bad key issue, I should get a permission denied message, or a 'too many attempts' message if it was trying to connect using too many different keys. I have tried the ip address directly, as well as the special address that amazon assigns, and neither of those work. Also, when I attempt to ssh to my server from my macbook, it lists the correct IP in the verbose output.

Here is the output of a telnet attempt to port 22:

telnet mysite.com 22
Trying 184.72.xx.xx...
telnet: connect to address 184.72.xx.xx: Connection refused
telnet: Unable to connect to remote host

Basically, I am completely out of ideas, and would appreciate any help. I feel like I have tried just about everything, though there must be something that I am missing. Is it possible that my macbook is blocking certain traffic without my knowledge? I checked the firewall settings and it is disabled, and ipfw is not running either (I don't think).

Update: I have attempted a traceroute to my server from my macbook, it fails saying, 'No route to host':

$ traceroute -I 184.72.xx.xx
traceroute to 184.72.xx.xx (184.72.220.0), 64 hops max, 72 byte packets
traceroute: sendto: No route to host
 1 traceroute: wrote 184.72.xx.xx 72 chars, ret=-1
 *traceroute: sendto: No route to host
traceroute: wrote 184.72.xx.xx 72 chars, ret=-1
 *traceroute: sendto: No route to host
traceroute: wrote 184.72.xx.xx 72 chars, ret=-1
 *
traceroute: sendto: No route to host
 2 traceroute: wrote 184.72.xx.xx 72 chars, ret=-1
 *traceroute: sendto: No route to host
traceroute: wrote 184.72.xx.xx 72 chars, ret=-1
 *traceroute: sendto: No route to host
traceroute: wrote 184.72.xx.xx 72 chars, ret=-1

From a linux box on my local network things look good:

# traceroute -I 184.72.xx.xx
traceroute to 184.72.xx.xx (184.72.xx.xx), 30 hops max, 60 byte packets
 1  192.168.1.1 (192.168.1.1)  0.190 ms  0.237 ms  0.282 ms
 2  10.1.10.1 (10.1.10.1)  0.946 ms  1.779 ms  2.138 ms
 3  76.109.128.1 (76.109.128.1)  16.581 ms  18.187 ms  32.675 ms
 4  te-9-2-ur02.delrayeast.fl.pompano.comcast.net (68.85.125.149)  17.810 ms  17.976 ms  18.077 ms
 5  te-8-1-ur01.bocaraton.fl.pompano.comcast.net (68.86.165.194)  18.325 ms  18.427 ms  18.521 ms
 6  te-3-4-ar01.stuart.fl.pompano.comcast.net (68.86.165.109)  19.430 ms  18.559 ms  18.645 ms
 7  te-0-4-0-5-ar03.northdade.fl.pompano.comcast.net (68.85.127.205)  24.839 ms  24.438 ms  24.525 ms
 8  pos-0-4-0-0-cr01.miami.fl.ibone.comcast.net (68.86.91.81)  23.113 ms  16.435 ms  24.480 ms
 9  xe-10-1-0.edge2.Miami1.Level3.net (64.156.8.9)  23.354 ms  23.544 ms  24.256 ms
10  ae-32-52.ebr2.Miami1.Level3.net (4.69.138.126)  30.777 ms  31.698 ms  31.878 ms
11  ae-2-2.ebr2.Atlanta2.Level3.net (4.69.140.142)  36.471 ms  37.461 ms  37.654 ms
12  ae-73-73.ebr3.Atlanta2.Level3.net (4.69.148.253)  37.825 ms  37.917 ms  38.013 ms
13  ae-2-2.ebr1.Washington1.Level3.net (4.69.132.86)  50.805 ms  42.708 ms  47.774 ms
14  ae-91-91.csw4.Washington1.Level3.net (4.69.134.142)  48.827 ms  49.018 ms  49.122 ms
15  ae-4-90.edge3.Washington1.Level3.net (4.69.149.209)  56.149 ms  113.159 ms  114.077 ms
16  AMAZON.COM.edge3.Washington1.Level3.net (4.59.144.94)  88.162 ms  47.429 ms  57.533 ms
17  72.21.220.131 (72.21.220.131)  68.472 ms  52.906 ms  57.836 ms
18  72.21.222.143 (72.21.222.143)  58.755 ms  43.988 ms  50.344 ms
19  216.182.224.53 (216.182.224.53)  51.369 ms  43.720 ms  48.007 ms
20  * * *
21  216.182.232.125 (216.182.232.125)  49.900 ms  46.469 ms  50.883 ms
22  * * *
23  * * *
24  mail.myserver.com (184.72.xx.xx)  48.432 ms  45.051 ms  49.796 ms

Coming back from the server, the results also look reasonable:

# traceroute -I 76.109.130.xx
traceroute to 76.109.130.xx (76.109.130.99), 30 hops max, 40 byte packets
 1  10.204.200.3 (10.204.200.3)  10.902 ms  4.576 ms  0.466 ms
 2  10.1.44.25 (10.1.44.25)  0.621 ms  0.634 ms  0.366 ms
 3  10.1.34.136 (10.1.34.136)  0.484 ms  0.804 ms  20.380 ms
 4  216.182.232.74 (216.182.232.74)  0.401 ms  0.457 ms  0.415 ms
 5  216.182.232.52 (216.182.232.52)  0.373 ms  0.458 ms  0.438 ms
 6  72.21.222.156 (72.21.222.156)  1.265 ms  1.280 ms  1.214 ms
 7  72.21.220.126 (72.21.220.126)  2.014 ms  2.079 ms  2.089 ms
 8  xe-4-0-0.edge3.Washington1.Level3.net (4.59.144.81)  1.369 ms  1.445 ms  1.477 ms
 9  vlan90.csw4.Washington1.Level3.net (4.69.149.254)  1.499 ms  1.503 ms  1.498 ms
10  ae-91-91.ebr1.Washington1.Level3.net (4.69.134.141)  2.367 ms  2.272 ms  2.453 ms
11  ae-2-2.ebr3.Atlanta2.Level3.net (4.69.132.85)  15.431 ms  15.273 ms  15.684 ms
12  ae-73-73.ebr2.Atlanta2.Level3.net (4.69.148.254)  18.637 ms  21.841 ms  26.061 ms
13  ae-2-2.ebr2.Miami1.Level3.net (4.69.140.141)  29.121 ms  32.777 ms  36.370 ms
14  ae-2-52.edge2.Miami1.Level3.net (4.69.138.102)  28.909 ms  28.445 ms  28.545 ms
15  4.59.85.46 (4.59.85.46)  29.504 ms  29.760 ms  29.013 ms
16  pos-0-13-0-0-ar03.northdade.fl.pompano.comcast.net (68.86.90.230)  30.111 ms  31.494 ms  32.045 ms
17  te-8-7-ar01.stuart.fl.pompano.comcast.net (68.85.127.194)  33.002 ms  32.879 ms  33.023 ms
18  te-9-1-ur01.bocaraton.fl.pompano.comcast.net (68.86.165.110)  35.068 ms  34.887 ms  34.901 ms
19  te-9-4-ur02.delrayeast.fl.pompano.comcast.net (68.86.165.193)  36.183 ms  35.679 ms  35.730 ms
20  te-17-10-cdn04.delrayeast.fl.pompano.comcast.net (68.85.125.146)  48.517 ms  56.562 ms  55.199 ms
21  c-76-109-130-xx.hsd1.fl.comcast.net (76.109.130.xx)  43.331 ms  49.565 ms  45.136 ms

I have also tested traceroute against the other EC2 box that I am having trouble connecting to and I see the same results. Traceroute does seem to work on my macbook for servers that I can access via ssh. Again, I have no trouble getting to those boxes from my browser.

Solution

Ok, I actually got this resolved. Thanks everyone who responded, especially Eric Hammond. If I hadn't done the traceroute, I wouldn't have googled the 'no route to host' issue and would not have come up with the solution. What I found was two things, I'm not sure which one did the trick, so I will include both here. First, I found some people complaining that the PeerGuardian app had caused these sorts of issues. I deleted the app and the library directory for PeerGuardian.

The other thing solution mentioned was Lion Cache Cleaner, which I downloaded and ran. I did the deep clean on everything, along with making sure that the trash was completely emptied (after deleting PeerGuardian). After the cache cleaner ran, I rebooted and was able to successfully connect to my server, and my client's box.

Thanks again for the helpful suggestions, I would not have gotten this resolved without all this help.

Comments

[EEAA]

BTW - you should never be generating your ssh keypairs anywhere but on your own workstation. Your private key should never go on the server, only your public key.

[Handyman5]

I'd be curious to know what happens if you were to set up Linux under VirtualBox and try to connect there. Perhaps that will help you distinguish between a configuration issue with Mac OS and an underlying network issue somewhere.

[Eric Van Joshnon]

You seem to imply it but just for clarification, your external IP address (the one the EC2 instance would be seeing) is the same when SSH from your Linux box as it is from the Mac? Are there any IP restrictions to SSH from the security group? You may want to try and open up port 22 to the world to test and see if that resolves it. Also try running SSH on another port and see if the Mac can connect on that port

[Eric Van Joshnon]

Sorry I missed where you said the security group is wide open...

[Eric Hammond]

If you can telnet to port 22 using the IP address with one box, but can't with a different box, then it has nothing to do with ssh configuration or keys. This is a networking problem and should be phrased as such to get the best chance of a helpful response. The issue could be on your local box, somewhere in the network route between you and EC2, or in iptables type configuration on the server. My money is on the idea that you didn't use the same IP address in both telnet's, but perhaps you've verified that three times. It might be helpful to see a traceroute from both boxes.

[ejf]

@EricHammond I updated my question with the results of the traceroute. Seems like you are correct that it is some sort of networking issue.

[ejf]

I actually tried this already. I double checked in my current known_hosts file and there is no entry for my server.

[gWaldo]

That sucks. It was worth a shot...

[gWaldo]

Don't forget to mark the question as solved. (Click the checkbox by your answer.)

[ejf]

As soon as it lets me, I will do that. It says that I need to wait another 9 hours.