Ping only working in one direction

Question

I have a small cluster running centos 7 composed of machines A, B and C. They are all separate physical machines connected on the same LAN.

Machine A can ping B and C. Machine B can ping A and C. However, machine C can only ping machine A.

My problem is that machine C cannot ping machine B. It says Destination Host Unreachable.

I have tried changing the network switch, changing the Ethernet, disabling firewalls and SELinux on all the machines and still the problem persists. I have also checked the IP address and the /etc/hosts file to ensure they are all consistent.

I'm wondering what I could try to help diagnose the cause of this problem.

Thank you !

Solution 1

When you say "It says Destination Host Unreachable", what is the ip address of the device that reports this? What I mean is, in the ping output you get something like:

From x.x.x.x icmp_seq=2 Destination Host Unreachable

So what is the x.x.x.x in that output? It is probably C's own ip address, but worth to check.

If it is C's own address then this typically means it does not have B in its ARP table and is not getting a reply to an ARP request. To confirm, check the output of "arp -a" (while the ping is running, or immediately after stopping the ping), does it show a line with the IP address of B and its MAC address, a line with B's IP and "incomplete" or no line with B's IP at all?

What may also be of help is to run a packet capture using e.g. tcpdump or wireshark to see what is actually occurring on the network when you run the ping. It may be even more useful to get 2 captures on B and C at the same time.

Solution 2

Are you pinging by machine name, or by IP address, or is it the same both ways?

Are any of these machines performing routing (multiple NICs)?

I'm assuming that when you say they are on the same LAN, you mean that each machine is using a one and only one network card attached to the same switch, and that there are no routers between them. I am also assuming they are on the same logical IP subnet. If this is incorrect, please give more details in the question. I have reason to believe that these assumptions are not 100% correct.

Things to try:

• Ping by IP address first. This will rule out name resolution problems. If this is successful, then you probably have a name resolution issue (like a DNS server disagreeing with the hosts file).
• Check your interface's subnet mask on each machine. If they don't match, this can happen in some circumstances.
• Check for multiple gateways. Machine C may have more than one defined. This shouldn't be an issue if they are on the same IP subnet. The message Destination host unreachable implies that it knows the address of what it is trying to ping, but can't seem to get there with the routes that are defined.

It would help if you could provide your routing table for each machine, which is typically done with route -n on Linux.

Comments

[Paul]

This could be a subnet mask issue. Can you add the interface config to the question?