Multiple NIC arp issue

Question

I'm troubleshooting an issue on Centos 6.4. I have a few systems setting up for greenplum. Each one has two "interconnect" interfaces connecting to an isolated subnet. It is like a cluster interconnect. So each has eth2 and eth3 both in 10.1.1.0/24 network. Each interface has its own IP address.

when I ping the two IPs of host2 from host1, the traffic always come from eth2 with the same MAC address. This can be seen from the ARP table and tcpdump. Looks like the arp table on host1 has both IP on the Mac address of eth2 on host2

I read something about ARP FLux. I tried all of the followings but nothing seem to work. Any suggestion?

echo 1 > /proc/sys/net/ipv4/conf/eth2/arp_filter
echo 1 > /proc/sys/net/ipv4/conf/eth3/arp_filter
echo 1 > /proc/sys/net/ipv4/conf/eth2/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/eth3/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/eth2/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/eth3/arp_announce

Solution

sysctl -w net.ipv4.conf.all.arp_ignore=1

1 - reply only if the target IP address is local address configured on the incoming interface

sysctl -w net.ipv4.conf.all.arp_announce=2

2 - Always use the best local address for this target. In this mode we ignore the source address in the IP packet and try to select local address that we prefer for talks with the target host. Such local address is selected by looking for primary IP addresses on all our subnets on the outgoing interface that include the target IP address. If no suitable local address is found we select the first local address we have on the outgoing interface or on all other interfaces, with the hope we will receive reply for our request and even sometimes no matter the source IP address we announce.

http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.arp_problem.html#2_6_arp_announce

Comments

[chuck]

I am not familiar with greenplum so would you mind me asking why you have two interfaces in the same subnet range? If you are pushing for more throughput I would think you would want to look at channel bonding. If you have different services(storage, interconnect, etc.) I would consider having two subnets where you could use one for storage and the other for interconnect.