SSH: Troubleshooting "Remote port forwarding failed for listen port" errors

Question

Question: Why does ssh -N -R 2222:localhost:22 <bluehost_user>@<bluehost_ip> result in a "Remote port forwarding failed for listen port" error? The objective is to establish a reverse tunnel with port forwarding in order to consistently ssh into a host behind a NAT router that has a dynamic private IP. See image for details.

Already Tried:

1. Researched existing literature on Google, Stackoverflow, etc. There are topics concerning this error message, however the resolutions given resolve root causes different than that of this particular instance because those resolutions do not resolve the error in this case.
2. I've performed several diagnostics to validate the required ports are open. Some of those results are shown in the image below.

Reverse SSH Tunnel

Update

I was trying the following command for Step 2: reduser@redhost:~ ssh greenuser@greenhost -p 2222

It should be: reduser@redhost:~ ssh greenuser@bluehost -p 2222 You want to use the greenuser credentials on the bluehost IP because the host you are loging into when you use port 2222 is really the greenhost.

Solution 1

Why does ssh -N -R 2222:localhost:22 <bluehost_user>@<bluehost_ip> result in a "Remote port forwarding failed for listen port" error?

I get this exact warning when I attempt to use a port that is already taken on the remote side.

The output of netstat from bluehost indicates that something is already listening on port 2222 there. It doesn't show what it is though.

Solutions:

1. Change 2222 in your ssh invocation to some other port which is not in use on bluehost. Just make it greater than 1023 because regular users can't bind to well-known ports; otherwise you will get the same warning regardless of whether the port is in use or not.
2. Or identify the listening process (on bluehost) with sudo lsof -i TCP:2222; terminate or reconfigure it to make the port 2222 available.

---

Edit:

In your case this part of man ssh seems important:

-R [bind_address:]port:host:hostport
-R [bind_address:]port:local_socket
-R remote_socket:host:hostport
-R remote_socket:local_socket

[…] By default, TCP listening sockets on the server will be bound to the loopback interface only. This may be overridden by specifying a bind_address. An empty bind_address, or the address ‘*’, indicates that the remote socket should listen on all interfaces. Specifying a remote bind_address will only succeed if the server's GatewayPorts option is enabled (see sshd_config(5)).

It means you should have GatewayPorts yes in the sshd_config on bluehost. Read man 5 sshd_config to learn more. Don't forget to reload the service afterwards.

Solution 2

Make sure there is no hanging connection on port 2222 at bluehost. Test at bluehost lsof -t -i:2222 whether any process id is using port 2222. Additionally, kill this process (for example with kill $(lsof -t -i:2222)).

This resolved the issue for me. Hopefully this information is useful for someone else. :)

Comments

[balwa]

Have you checked out serverfault.com/questions/595323/… ?

[ngm_code]

@balwa Checked but that is about connections expiring after a period of time. The connection never succeeds in my case. I also make use of the KeepAliveInterval parameters so there is explicit control over some of the time factors.

[ngm_code]

@ModeratorImpersonator tried ssh -N -R <bluehost_ip>:2222:localhost:22 and ssh seemed to take that as invalid syntax. ssh simply returned usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec] [-D [bind_address:]port] [-E log_file] [-e escape_char] [-F configfile] [-I pkcs11] [-i identity_file] [-L [bind_address:]port:host:hostport] [-l login_name] [-m mac_spec] ...

[ngm_code]

@Ramhound I reworded the question to be more direct. Please take post off hold or offer more specific comments as to why this was put on hold.

[MariusMatutiae]

@DavidPostill I disagree with this post being put on hold. The question, what causes the error message so and so is in my view a perfectly acceptable one.

[barlop]

SSH aside, what if you go to the remote computer and try to run a server on port 2222? e.g. if you have cygwin on there and do nc -l 2222 or nc -l -p 2222 (whichever notation the nc with cygwin uses)

[MariusMatutiae]

The above command works just fine on my system (just add the -f flag, so that you can get a prompt back). You do not seem to have the -R flag, are you sure? Are you sure there is nothing already using port 2222?

[DavidPostill]

@MariusMatutiae The modified question is better. I have reopened.

[barlop]

@MariusMatutiae he does have -R, look carefully, just before the 2222:... and if he didn't have a -R or -L there then the command would get less far than it has. He just put that -R port:ip:port near the end of his command line, so perhaps that's why you couldn't see it.

[Tom]

I know it's not the cause of this specific question, but since googling the message leads here I'd note that using a port < 1024 also results in this same message.

[ngm_code]

Interestingly, I'm seeing a connection from China (note I was using this host as a VPN from China when I was there a few weeks ago). While this host is a sandbox, it's still concerning <bluehost_user>@<bluehost_ip>:~$ netstat | grep 2222 tcp 0 0 <bluehost_ip>:2222 htuidc.bgp.ip:2599 SYN_RECV Port 2222 open or not, how would one connect without a username and password (note this is tcp not ssh)? I do have sshd listening on 2222 as configured in /etc/ssh/sshd_config.

[ngm_code]

Additional info from lsof: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sshd 1034 root 3u IPv4 13344 0t0 TCP *:2222 (LISTEN) sshd 1034 root 4u IPv6 13353 0t0 TCP *:2222 (LISTEN)

[Kamil Maciorowski]

@ngm_code So it's your sshd that needs to be reconfigured. Or just use another port when you ssh -R from elsewhere.

[ngm_code]

Ok, I will take a look. So you're saying it's incompatible to open a port via /etc/ssh/sshd_config prior to opening it up at runtime via a call to ssh in the CLI?

[Kamil Maciorowski]

@ngm_code Yes. Ports (or one port, usually 22) configured in sshd_config are for SSH clients to connect to. In your case you connect to one of these ports and order sshd to open additional port and tunnel it to your machine. If it could be the same port then sshd wouldn't know whether incoming packets are destined to it or to the tunnel.

[ngm_code]

Ok, removed Port 2222 from /etc/ssh/sshd_config and now I can successfully execute the ssh -N -R 2222:localhost:22 <bluehost_user>@<bluehost_ip> on the greenhost without error. That part is now working, however I'm now trying to accomplish the last step; when I ssh <bluehost_ip> -p 2222 I am expecting to be forwarded to <greenhost_ip>:22 but instead I am getting a ssh: connect to host <bluehost_ip> port 2222: Connection refused error.

[Kamil Maciorowski]

@ngm_code I have expanded my answer to address this issue.

[ngm_code]

Made it further, but the authentication process is convoluted and failing. Added ECDSA to known hosts, it finds the keys I do have, fails to find the ones I don't and then prompts for <bluehost_user>@<bluehost_ip> password as an alternative to the keys. I enter it but it is rejected.

[ngm_code]

unix.stackexchange.com/questions/131886/… This seems to be a related issue.

[ngm_code]

Solved it! Red host should initiate ssh <greenhost_user>@<bluehost_ip> -p 2222. Easy gotcha, you want to use the greenhost username with the bluehost ip.

[mehov]

@ Googlers: To free up a TCP port, use fuser -k 2222/tcp (source)

[Hugo Maxwell]

In my case it was a previous sshd process, because I have a loop on the client side restarting the tunnel in case it fails...