Why am I unable to telnet to a local port that has a listening service?

Question

I suspect this is either a very simple question, or a very complex one.

I have a headless server running ubuntu 10.04 that I can ssh into. I have full root access to the system. I am trying to set up an ssh tunnel to allow me to vnc to the system (but that isn't my question.

I have vnc running on port 5903, here is the netstat output for that:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5903            0.0.0.0:*               LISTEN      7173/Xtightvnc
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      
465/sshd

But when I try to telnet to that port, from within the same system and login, I get unable to connect errors

# telnet localhost 5903
Trying ::1...
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection timed out

I am able to telnet to port 22 (as a verification)

~# telnet localhost 22
Trying ::1...
Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH_5.3p1 Debian-3ubuntu7

I have tried to open up any possible ports using ufw (probably clumsy fashion)

# ufw status numbered
Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 5903                       ALLOW IN    Anywhere
[ 2] 22                         ALLOW IN    Anywhere

What else might be blocking this connection locally?

Thank you,

Edit:

The only reference to port 5903 in iptable -L -n is this:

Chain ufw-user-input (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:5903
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:5903
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:22
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:8080
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:8080

I can post the whole output if that will be useful.

hosts.allow and hosts.deny both contain only comments.

Re-Edit: Some other questions pointed me to nmap, so I ran a portscan through that utility:

# nmap -v -sT localhost -p1-65535

Starting Nmap 5.00 ( http://nmap.org ) at 2011-11-09 09:58 PST
NSE: Loaded 0 scripts for scanning.
Warning: Hostname localhost resolves to 2 IPs. Using 127.0.0.1.
Initiating Connect Scan at 09:58
Scanning localhost (127.0.0.1) [65535 ports]
Discovered open port 22/tcp on 127.0.0.1
Connect Scan Timing: About 18.56% done; ETC: 10:01 (0:02:16 remaining)
Connect Scan Timing: About 44.35% done; ETC: 10:00 (0:01:17 remaining)
Completed Connect Scan at 10:00, 112.36s elapsed (65535 total ports)
Host localhost (127.0.0.1) is up (0.00s latency).
Interesting ports on localhost (127.0.0.1):
Not shown: 65533 filtered ports
PORT   STATE  SERVICE
22/tcp open   ssh
80/tcp closed http

Read data files from: /usr/share/nmap
Nmap done: 1 IP address (1 host up) scanned in 112.43 seconds
           Raw packets sent: 0 (0B) | Rcvd: 0 (0B)

I think this shows that 5903 is blocked somehow. Which I pretty much knew. The question remains what is blocking it and how to modify.

Re-re-edit:

To check Paul Lathrop's suggested answer, I first verified my ip address with ifconfig:

eth0      Link encap:Ethernet  HWaddr 02:16:3e:42:28:8f
          inet addr:10.0.10.3  Bcast:10.0.10.255  Mask:255.255.255.0

Then tried to telnet to 5903 from that address:

# telnet 10.0.10.3 5903
Trying 10.0.10.3...
telnet: Unable to connect to remote host: Connection timed out

No luck.

Re-re-re-re-edit:

Ok, I think I have isolated it a bit to vncserver, not the firewall, darn it. I shut off vncserver and had netcat listen on port 5903. My vnc client then was able to establish a connnection and sit and wait for a response. Looks like I should be chasing a vnc problem. At least that is progress Thanks for the help

Solution 1

Ok, I think I have isolated it a bit to vncserver, not the firewall, darn it. I shut off vncserver and had netcat listen on port 5903. My vnc client then was able to establish a connnection and sit and wait for a response. Looks like I should be chasing a vnc problem. At least that is progress Thanks for the help

Solution 2

Most likely your VNC server isn't binding to the loopback interface. I've run into this with some Java daemons as well, they claim to be listening on 0.0.0.0 which intuitively should include loopback, but in fact they are only bound to the "external" network interfaces. Try: telnet <LOCAL_IP> 5904; if it connects then your VNC server is not binding to the loopback interface.

Comments

[Deb]

The 'connection timed out' message is there because the connection SYN packet didn't receive either an ACK or a RST. That could be because the firewall is still blocking the port, the service isn't actually listening there (the FW still ate the RST packet), or the service refused to talk for some reason (the FW still ate the RST packet). So, hard to say.

[David]

What does iptables -L -n show? Who know what extra's ufw added in. Anything in hosts.allow or hosts.deny?

[iainlbc]

Is this in production? If not you can verify its a firewall/iptables issue by stopping the iptables service entirely and retesting

[Skip Huffman]

Nope, not in production.

[Skip Huffman]

I think I turned off iptables (what command/set do you recommend) and still see the same "unable to connect" result.

[MastaJeet]

(Stab in the dark:) It may have something to do with IPv6. See the ubuntu section of cyberciti.biz/tips/linux-how-to-disable-the-ipv6-protocol.ht‌​ml. I thought localhost should only be 127.0.0.1, not ::1 as well. Can you change your /etc/hosts file?

[Skip Huffman]

No luck there. See main post for formatted response.