Cannot telnet to private IP or to port

Question

What specific changes need to be made so that a CentOS 7 installation at local IP 192.168.1.6 can telnet to another CentOS 7 installation at another local IP 192.168.1.5?

As you can see, 192.168.1.6 IS ABLE TO PING 192.168.1.5 as follows:

[root@localhost /]# ping 192.168.1.5
PING 192.168.1.5 (192.168.1.5) 56(84) bytes of data.
64 bytes from 192.168.1.5: icmp_seq=1 ttl=64 time=0.515 ms
64 bytes from 192.168.1.5: icmp_seq=2 ttl=64 time=0.565 ms
^C
--- 192.168.1.5 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.515/0.540/0.565/0.025 ms

But telnet FROM 192.168.1.6 TO 192.168.1.5 fails as follows:

[root@localhost /]# telnet 192.168.1.5
Trying 192.168.1.5...
telnet: connect to address 192.168.1.5: No route to host

And telnet FROM 192.168.1.6 TO port 5432 at 192.168.1.5 also fails as follows:

[root@localhost /]# telnet 192.168.1.5:5432
telnet: 192.168.1.5:5432: Name or service not known
192.168.1.5:5432: Unknown host
[root@localhost /]#

PostgreSQL is running at 192.168.1.5, and should be receiving telnet 192.168.1.5:5432. Therefore, I added the following line to pg_hba.conf BEFORE running the above:

host    all    all    192.168.1.6/24    trust

And I restarted PostgreSQL before running the above ping and telnet by typing systemctl restart postgresql.

Similarly, before running the above ping and telnet commands, I also created the following firewall rules on 192.168.1.5:

[root@localhost ~]# firewall-cmd --zone=public --add-port=5432/tcp
[root@localhost ~]# firewall-cmd --permanent --zone=trusted --add-source=192.168.1.6/32
[root@localhost ~]# firewall-cmd --reload  

Also, I confirmed that PostgreSQL is running at port 5432 with the following command typed in the terminal of 192.168.1.5:

[root@localhost ~]# ss -l -n | grep 5432
u_str  LISTEN     0      128    /var/run/postgresql/.s.PGSQL.5432 71466                 * 0
u_str  LISTEN     0      128    /tmp/.s.PGSQL.5432 71468                 * 0
tcp    LISTEN     0      128    127.0.0.1:5432                  *:*
tcp    LISTEN     0      128     ::1:5432                 :::*
[root@localhost ~]#

---

@roaima's suggestions:

---

Per @roaima's suggestion, I tried the following, but am still not able to connect:

From 192.168.1.6, I sent:

[root@localhost ~]# telnet 192.168.1.5 5432
Trying 192.168.1.5...
telnet: connect to address 192.168.1.5: No route to host

And on 192.168.1.5, the tcpdump of the receiving end of the telnet request was:

[root@localhost ~]# tcpdump -i eth0 port 5432 or arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:52:49.309526 IP 192.168.1.6.53328 > localhost.localdomain.postgres: Flags [S], seq 3210933916, win 29200, options [mss 1460,sackOK,TS val 629624820 ecr 0,nop,wscale 7], length 0
16:52:54.312716 ARP, Request who-has localhost.localdomain tell 192.168.1.6, length 28
16:52:54.312750 ARP, Reply localhost.localdomain is-at 52:54:00:ef:35:18 (oui Unknown), length 28
^C
3 packets captured
4 packets received by filter
0 packets dropped by kernel  

Similarly, from 192.168.1.6 I sent the following telnet to the IP level only:

[root@localhost ~]# telnet 192.168.1.5
Trying 192.168.1.5...
telnet: connect to address 192.168.1.5: No route to host
[root@localhost ~]#

And on 192.168.1.5, the tcpdump of the receiving end of the telnet request was:

[root@localhost ~]# tcpdump -i eth0 port 5432 or arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
//THESE 2 LINES PRINTED BEFORE 2ND TELNET WAS RUN: 16:58:11.619638 ARP, Request who-has gateway tell localhost.localdomain, length 28
//THESE 2 LINES PRINTED BEFORE 2ND TELNET WAS RUN: 16:58:11.619940 ARP, Reply gateway is-at b8:ec:a3:11:74:6e (oui Unknown), length 46
16:58:35.555570 ARP, Request who-has 192.168.1.6 tell localhost.localdomain, length 28
16:58:35.555753 ARP, Reply 192.168.1.6 is-at 52:54:00:ab:31:40 (oui Unknown), length 28
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
[root@localhost ~]#  

---

@cutrightjm's suggestions:

---

On 192.168.1.5, I typed the following in one Putty session:

[root@localhost ~]# telnet localhost 5432
Trying ::1...
Connected to localhost.
Escape character is '^]'.

Simultaneously, in a separate Putty session to 192.168.1.5, I saw no results from the tcpdump, as follows:

[root@localhost ~]# tcpdump -i eth0 port 5432 or arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
[root@localhost ~]#

---

@JeffSchaller's Suggestions:

---

Per @JeffSchaller's suggestions, I ran the following commands on 192.168.1.6. Note that this is CentOS 7, which has replaced netstat with ss, and which has replaced iptables with firewalld:

ss -rn produced 90 lines of output. Can you suggest a meaningful grep or other filter to reduce the output to the allowable amount to add to a posting?

[root@localhost ~]# iptables -Ln
iptables: No chain/target/match by that name.


[root@localhost ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources:
  services: dhcpv6-client ssh
  ports: 8080/tcp
  protocols:
  masquerade: no
  forward-ports:
  sourceports:
  icmp-blocks:
  rich rules:

[root@localhost ~]#

I also ran the following commands on 192.168.1.6:

[root@localhost ~]# ip route
default via 192.168.1.1 dev eth0  proto static  metric 100
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.6  metric 100

[root@localhost ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:ab:31:40 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.6/24 brd 192.168.1.255 scope global dynamic eth0
       valid_lft 133013sec preferred_lft 133013sec
    inet6 fe80::5054:ff:feab:3140/64 scope link
       valid_lft forever preferred_lft forever
[root@localhost ~]#

---

Removing Firewalls On Both Machines

---

As an extreme test, I removed the firewalls on both 192.168.1.5 and 192.168.1.6 by typing yum remove firewalld and yum remove iptables on both machines. I then validated both removals as follows:

On 192.168.1.5 :

[root@localhost ~]# systemctl status firewalld
Unit firewalld.service could not be found.
[root@localhost ~]# iptables -L -n
-bash: /sbin/iptables: No such file or directory

On 192.168.1.6 :

[root@localhost ~]# systemctl status firewalld
Unit firewalld.service could not be found.
[root@localhost ~]# iptables -L -n
-bash: /sbin/iptables: No such file or directory

Next, I typed tcpdump -i eth0 port 5432 or arp on 192.168.1.5, followed by typing telnet 192.168.1.5 5432 on 192.168.1.6.

The result of the telnet was the following refusal message printed on 192.168.1.6:

[root@localhost ~]# telnet 192.168.1.5 5432
Trying 192.168.1.5...
telnet: connect to address 192.168.1.5: Connection refused
[root@localhost ~]#

Simultaneously, the tcpdump printout on 192.168.1.5 from the telnet call from 1.6 was:

[root@localhost ~]# tcpdump -i eth0 port 5432 or arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:25:11.349238 ARP, Request who-has localhost.localdomain tell gateway, length 46
10:25:11.349261 ARP, Reply localhost.localdomain is-at 52:54:00:ef:35:18 (oui Unknown), length 28
10:25:14.391222 IP 192.168.1.6.53344 > localhost.localdomain.postgres: Flags [S], seq 3043089625, win 29200, options [mss 1460,sackOK,TS val 692769902 ecr 0,nop,wscale 7], length 0
10:25:14.391265 IP localhost.localdomain.postgres > 192.168.1.6.53344: Flags [R.], seq 0, ack 3043089626, win 0, length 0
10:25:19.395578 ARP, Request who-has 192.168.1.6 tell localhost.localdomain, length 28
10:25:19.396039 ARP, Reply 192.168.1.6 is-at 52:54:00:ab:31:40 (oui Unknown), length 28
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel
[root@localhost ~]#

To determine whether or not PostgreSQL is listening on port 5432, I typed the following two commands on 192.168.1.5:

Note that firewalld and iptables are both still removed while the following commands are run:

First, I viewed the pg_hba.conf file in 192.168.1.5 to see that there is a rule to trust 192.168.1.6:

[root@localhost ~]# vi /var/lib/pgsql/data/pg_hba.conf
# LOTS OF # COMMENTED LINES OMITTED HERE FOR BREVITY
# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
host    all             all             192.168.1.6/24          trust
# IPv6 local connections:
host    all             all             ::1/128                 trust

Next, I typed the following netstat command on 192.168.1.5 to see if there is a rule for port 5432:

[root@localhost ~]# netstat -anpt | grep LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      943/sshd
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      25166/postgres
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1483/master
tcp6       0      0 127.0.0.1:45228         :::*                    LISTEN      19089/java
tcp6       0      0 127.0.0.1:8020          :::*                    LISTEN      14338/java
tcp6       0      0 :::7990                 :::*                    LISTEN      19089/java
tcp6       0      0 :::22                   :::*                    LISTEN      943/sshd
tcp6       0      0 ::1:5432                :::*                    LISTEN      25166/postgres
tcp6       0      0 127.0.0.1:7992          :::*                    LISTEN      19066/java
tcp6       0      0 ::1:7992                :::*                    LISTEN      19066/java
tcp6       0      0 ::1:25                  :::*                    LISTEN      1483/master
tcp6       0      0 127.0.0.1:36122         :::*                    LISTEN      19089/java
tcp6       0      0 :::8095                 :::*                    LISTEN      14338/java
tcp6       0      0 :::5701                 :::*                    LISTEN      19089/java
[root@localhost ~]#

Solution

The first problem was that you were using the wrong syntax for the telnet command. Running man telnet will show you that the syntax is something like this:

telnet <host> [<port>]

So in your case, you should run this:

telnet 192.168.1.5 5432

---

A second problem is that you have a firewall rule on each host preventing outbound traffic to 5432/tcp. (And probably other ports too.) The error message, "No route to host", is generated by an iptables --j REJECT rule in the OUTPUT chain that has --reject-with icmp-host-prohibited. Here is an example of creating such a rule:

iptables -I OUTPUT -p tcp --dport 5432 -j REJECT --reject-with icmp-host-prohibited

This satisfies the situation where the route clearly exists because ping succeeds, but where your telnet session fails. You yourself can check this with the command iptables --line-numbers -nvL (not iptables -Ln, which tries to list the rules for the chain n).

Two temporary fixes to confirm that traffic really can be established are

• Disable the firewall on both systems entirely

• Run these two commands on both systems (you can delete them afterwards by replacing -I with -D)

  iptables -I INPUT -p tcp --src 192.168.1.5/30 -j ACCEPT
  iptables -I OUTPUT -p tcp --dst 192.168.1.5/30 -j ACCEPT
  

I'm not (yet) sufficiently familiar with the CentOS 7 firewall tool to give you a full solution for that. I can go digging or maybe someone else might like to edit this answer to provide that information.

Comments

[cutrightjm]

Can you telnet localhost on 5432?

[RabT]

@cutrightjm Thank you. Please see the results of your suggestion, which I just posted at the end of the OP.

[Jeff Schaller]

The output of netstat -rn would be useful -- from the .6 host. Odd that you can ping it but have no tcp route.

[Jeff Schaller]

You don't have .5 as a local IP on .6, do you?

[RabT]

@JeffSchaller .5 and .6 are each virtual machines whose IP addresses are assigned by a hardware firewall based on their MAC addresses. I can answer any questions now using my phone. I will be able to post results of any diagnostic commands such as your netstat -rn when I am back at my computer later.

[Jeff Schaller]

I wonder also if the .6 host has iptables rules that aren't allowing those outbound ports. Check iptables -Ln and any firewall logs

[Jeff Schaller]

Sorry, looks like that'd be "ip route "

[Jeff Schaller]

And ip addr show -- to confirm that .5 isn't living on .6

[RabT]

@JeffSchaller I just added results of ip route and ip addr show to the end of my OP. Any other suggestions?

[dave_thompson_085]

(@JeffSchaller) That should be iptables -L -n or iptables -nL because iptables -Ln treats the n as a (bogus) chain. But I believe iptables and/or firewall-cmd on the target (1.5) is more useful than source (1.6) because the first tcpdump on 1.5 shows SYN arriving but no TCP response not even RST, and 1 packet filtered which could be ICMP. But even if that is due to firewall, ss on 1.5 shows postgresql listening on 127.1 and ::1 not the external address, which will also prevent the desired connection.

[RabT]

@dave_thompson_085 iptables -L -n produced 166 lines of output. Can you suggest some sort of filter to narrow the results to only those lines which will help in this case?

[RabT]

@JeffSchaller I just posted results at the end of the OP of completely uninstalling firewalls on both machines. Does this help you narrow the problem? Or see any possible solutions?

[Jeff Schaller]

192.168.1.5: Connection refused leads directly to dave_thompson_085's suggestion that postgres isn't listening on an external address

[RabT]

@JeffSchaller I just posted the pg_hba.conf and netstat results at the end of my OP to test whether postgres is listening. Can you please take a look? Does this confirm or deny your assessment? Can you please suggest a next step?

[Jeff Schaller]

short of completely removing the firewalld package, you could have just systemctl stop firewalld

[Jeff Schaller]

I don't know much about postgresql, but according to this random internet page you also have to edit postgresql.conf to change listen_addresses = '*'

[RabT]

@JeffSchaller You are right. Adding listen_addresses = '192.168.1.5, localhost' to postgresql.conf got the database connection from 192.168.1.6 to work. Then reinstalling and restarting firewalld and iptables on 192.168.1.5 still enabled it to work. The problem has now been isolated to the 192.168.1.6 installation of firewalld because reinstalling/restarting firewalld on 192.168.1.6 causes the connection to fail, while stopping firewalld on 192.168.1.6 makes the connection work again. I need to translate @roaima's suggestion into firewalld to finish this.

[RabT]

I just posted results at the end of the OP of completely uninstalling firewalls on both machines. Does this help you narrow the problem? Or see any possible solutions?

[roaima]

@CodeMed removing iptables doesn't remove the firewall from your systems, it just removes the user interface. Please re-install that and run iptables -nvL to demonstrate (at least to yourself) that the POLICY is ACCEPT for the INPUT and OUTPUT chains, and that these chains are empty.

[RabT]

Thank you and +1. I am able to telnet to .1.5 now after adding listen_addresses = '192.168.1.5, localhost' to postgresql.conf. The problem was not the firewall, but your syntax suggestion helped move us towards the solution. I am still not able to make a connection from a web app on 1.6 to the database at 1.5:5432, but that is out of scope for the present question.

[roaima]

@CodeMed based on the error messages you showed in your question, it was related to the firewall. The last part of the puzzle was to have PostgreSQL listening, but the other parts were also necessary.

[roaima]

@CodeMed You would have received "connection refused" if it had simply been a daemon not listening. You got "no route to host", which in your scenario was indicative of a firewall rule blocking outbound traffic.

[RabT]

I had not included the --permanent argument to the firewall-cmd statement, so the rule did not survive restarting firewalld, and needed to be reentered. But still, the database config was the true problem. .... Also, iptables is legacy code from obsolete historical artefacts and does not apply to modern operating systems like RHEL7.

[roaima]

@CodeMed the firewall layer is implemented in the kernel and in user space as iptables. Tools like firewalld are an abstraction layer on top of that. See access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Lin‌​ux/…