Debugging a connection refused response on port 21

Question

My colleagues and I are having trouble accessing Heart Internet's FTP servers.

We are running a mixture of Macs and PCs and we have a Linux box in the office too. None of our machines can connect.

When running nmap from my IP address, port 21 does not appear.

Using software like FileZilla just returns "Connection timed out", as does ftp on a Linux box.

Using Wireshark I can see "ICMP Destination unreachable (Port unreachable)" responses to the TCP SYN packets.

I can access the same servers on other ports, I can ping the servers and I can traceroute to the servers.

From my IP:

$ telnet ftp20.extendcp.co.uk 21
Trying 79.170.44.20...
telnet: Unable to connect to remote host: Connection refused

From remote server:

$ telnet ftp20.extendcp.co.uk 21
Trying 79.170.44.20...
Connected to ftp20.extendcp.co.uk.
Escape character is '^]'.
220 FTP server ready

Telneting to a different server works fine:

$ telnet ftp.mirrorservice.org 21
Trying 212.219.56.184...
Connected to ftp.mirrorservice.org.
Escape character is '^]'.
220-----------------------------------------------------------------------------
220-Welcome to the University of Kent's UK Mirror Service.
220-
220-More information can be found at our web site: http://www.mirrorservice.org/
220-Please send comments or questions to [email protected].
220-----------------------------------------------------------------------------
220

Traceroute:

$ traceroute ftp20.extendcp.co.uk
traceroute to ftp20.extendcp.co.uk (79.170.44.20), 30 hops max, 60 byte packets
 1  home.gateway.home.gateway (192.168.2.254)  0.608 ms  0.904 ms  2.152 ms
 2  88.215.57.252 (88.215.57.252)  20.283 ms  21.010 ms  21.254 ms
 3  88.215.62.78 (88.215.62.78)  20.983 ms  20.973 ms  21.014 ms
 4  88.215.62.230 (88.215.62.230)  27.175 ms  27.348 ms  27.509 ms
 5  88.215.62.218 (88.215.62.218)  50.552 ms  49.958 ms  50.975 ms
 6  * * *
 7  mx02-xe0.0.1-lon.gs.nodefour.net (83.166.164.85)  24.098 ms  26.726 ms  22.204 ms
 8  mx01-xe2.0.0-lon.gs.nodefour.net (83.166.164.34)  22.147 ms  22.165 ms  24.208 ms
 9  mx02-xe1.2.0-dry.dc2.nodefour.net (83.166.164.38)  35.475 ms  35.532 ms  35.879 ms
10  83.166.164.54 (83.166.164.54)  25.316 ms  34.015 ms  34.401 ms
11  ftp20.extendcp.co.uk (79.170.44.20)  25.529 ms  28.205 ms  26.298 ms

Ping:

$ ping ftp20.extendcp.co.uk
PING ftp20.extendcp.co.uk (79.170.44.20) 56(84) bytes of data.
64 bytes from ftp20.extendcp.co.uk (79.170.44.20): icmp_req=1 ttl=53 time=24.3 ms
64 bytes from ftp20.extendcp.co.uk (79.170.44.20): icmp_req=2 ttl=53 time=21.3 ms
64 bytes from ftp20.extendcp.co.uk (79.170.44.20): icmp_req=3 ttl=53 time=24.7 ms
64 bytes from ftp20.extendcp.co.uk (79.170.44.20): icmp_req=4 ttl=53 time=21.5 ms
64 bytes from ftp20.extendcp.co.uk (79.170.44.20): icmp_req=5 ttl=53 time=25.1 ms

Telnet to different port:

$ telnet ftp20.extendcp.co.uk 22
Trying 79.170.44.20...
Connected to ftp20.extendcp.co.uk.
Escape character is '^]'.
SSH-2.0-OpenSSH_5.3

TCPdump of telnet 79.170.44.20 21:

$ sudo tcpdump -n -n -v -i eth0 host 79.170.44.20
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:30:27.194966 IP (tos 0x10, ttl 64, id 1237, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.2.10.58366 > 79.170.44.20.21: Flags [S], cksum 0x3e9f (incorrect -> 0x00e5), seq 530375445, win 14600, options [mss 1460,sackOK,TS val 18737088 ecr 0,nop,wscale 6], length 0
14:30:27.216103 IP (tos 0xc0, ttl 53, id 12906, offset 0, flags [none], proto ICMP (1), length 88)
    79.170.44.20 > 192.168.2.10: ICMP 79.170.44.20 tcp port 21 unreachable, length 68
        IP (tos 0x0, ttl 54, id 1237, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.2.10.58366 > 79.170.44.20.21: Flags [S], cksum 0x0e94 (incorrect -> 0x00ed), seq 530375445, win 14600, options [mss 1452,sackOK,TS val 18737088 ecr 0,nop,wscale 6], length 0

TCPdump of telnet 79.170.44.20 23 continued from above

14:31:23.250970 IP (tos 0x10, ttl 64, id 15043, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.2.10.58533 > 79.170.44.20.23: Flags [S], cksum 0x3e9f (incorrect -> 0x4da7), seq 1506485437, win 14600, options [mss 1460,sackOK,TS val 18751102 ecr 0,nop,wscale 6], length 0
14:31:23.273111 IP (tos 0x0, ttl 53, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    79.170.44.20.23 > 192.168.2.10.58533: Flags [R.], cksum 0x0e1a (correct), seq 0, ack 1506485438, win 0, length 0

My ISP and Heart Internet are reporting no visible problems.


What tools can I use to determine where the problem is?

Is it possible to discover where along the network route to the servers the connection is being refused? (i.e. if it's my router, or their firewalls, is it possible to discover that's what's refusing the connection?)

Solution

We've ruled out a general block on outgoing connections to ftp servers from your network, which is good. Next step is probably traceroute; your network may be having problems accessing the netblock where the destination server lives. Fortunately, they allow traceroute, so your next step is probably to traceroute to the destination, and see where things fall down. I've pasted my traceroute output in for comparison.

[me@risby]$ traceroute ftp20.extendcp.co.uk 
traceroute to ftp20.extendcp.co.uk (79.170.44.20), 30 hops max, 60 byte packets
 1  192.168.3.1 (192.168.3.1)  0.200 ms  0.113 ms  0.102 ms
 2  lns18.inx.dsl.enta.net (188.39.1.30)  23.466 ms  23.318 ms  24.988 ms
 3  gi1-8.inx.dist.dsl.enta.net (188.39.1.29)  23.782 ms  24.622 ms  25.546 ms
 4  te2-2.interxion.dsl.enta.net (78.33.141.89)  26.186 ms  26.963 ms  26.802 ms
 5  te2-3.interxion.core.enta.net (87.127.236.209)  27.554 ms  28.360 ms  29.085 ms
 6  te4-2.telehouse-east.core.enta.net (87.127.236.137)  28.818 ms  28.512 ms  28.349 ms
 7  * * *
 8  mx01-xe2.3.0-lon.gs.nodefour.net (83.166.164.85)  28.397 ms  29.967 ms  30.681 ms
 9  mx01-xe2.2.0-lon.gs.nodefour.net (83.166.164.34)  31.404 ms  31.112 ms  32.733 ms
10  mx02-xe1.2.0-dry.dc2.nodefour.net (83.166.164.38)  32.465 ms  33.060 ms  33.776 ms
11  83.166.164.54 (83.166.164.54)  33.529 ms  34.628 ms  34.356 ms
12  ftp20.extendcp.co.uk (79.170.44.20)  34.043 ms  30.541 ms  30.543 ms

It would also be useful to see whether you can access anything else on the destination; could you paste the output of ping ftp20.extendcp.co.uk?

Edit: now we've established you're using linux on the desktop, you could productively use tcpdump to see how far away the refusal is coming from. Here's my output from tcpdump -n -n -v -i p1p1 host 79.170.44.20, when I do a telnet ftp20.extendcp.co.uk 22 (which connects), then a telnet ftp20.extendcp.co.uk 23 (which gets a Connection refused, as it should):

14:20:40.047720 IP (tos 0x10, ttl 64, id 57773, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.3.11.57105 > 79.170.44.20.22: Flags [S], cksum 0x5a67 (correct), seq 2606771394, win 14600, options [mss 1460,sackOK,TS val 6671727 ecr 0,nop,wscale 7], length 0
14:20:40.078615 IP (tos 0x0, ttl 47, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    79.170.44.20.22 > 192.168.3.11.57105: Flags [S.], cksum 0xc8ec (correct), seq 28398605, ack 2606771395, win 14480, options [mss 1412,sackOK,TS val 609884153 ecr 6671727,nop,wscale 7], length 0
[packets deleted]

14:20:48.193195 IP (tos 0x10, ttl 64, id 34283, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.3.11.57462 > 79.170.44.20.23: Flags [S], cksum 0x1fa0 (correct), seq 2030528683, win 14600, options [mss 1460,sackOK,TS val 6679872 ecr 0,nop,wscale 7], length 0
14:20:48.222609 IP (tos 0x0, ttl 47, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    79.170.44.20.23 > 192.168.3.11.57462: Flags [R.], cksum 0xae1d (correct), seq 0, ack 2030528684, win 0, length 0

Note how the ttl field in the very first packet back from the server in the first case is 47. Note how the ttl field in the reset packet (flags [R.]) in the second case is also 47, which is right and proper for a reset that comes from the destination server. If you see a much higher TTL, it strongly suggests the refusal is coming from somewhere much closer.

Edit 2: given what you've said about the TTLs in your case, it really does look as if that server has decided not to accept your connection. It's possible that something en route is faking that TCP-port-unreachable, but getting that right is difficult, and most firewall tools don't bother (iirc, even the great firewall of China famously failed to set TTLs on its refusals correctly).

As for why the remote server has decided to do this (automatic, maybe via fail2ban? manual, because of excessive downloads?), who can say? Unless you can contact the server admins, you probably won't find out why. If you have a business relationship with ftp20.extendcp.co.uk, escalate via those routes. Otherwise, I'd shrug, and use a proxy if there were a couple of files I desperately needed to get to or from that server.

Comments

[Jon]

I've updated the question with additional telnet info and also to say it's not just my machine which can't connect.

[MadHatter]

Do you control your local firewall?

[Jon]

iptables has no rules on the Linux box and our office router has its firewall disabled afaik. Turning off antiviruses and Windows Firewall makes no difference.

[Jon]

Thanks for the tcpdump info, I've added the tcpdump from my Linux machine. The TTLs match (suggesting the actual server is rejecting), but is there any additional info you can glean from the output?

[Jon]

Thanks for all your help, I've escalated the issue (again!) with Heart Internet using this additional network debugging info.