Specifying a preferred route when there are multiple links to same network

Question

I have a server with multiple interfaces and IP addresses on the same network. I would like for non-iSCSI traffic to exclusively use one of the interfaces, and for iSCSI traffic to exclusively use the rest of the interfaces. Limiting the iSCSI traffic to a subset of my interfaces is easy; I simply don't create entries in /var/lib/iscsi/ifaces/ for the interfaces I don't want it to use. However, I'm not sure what is a good approach to limiting the non-ISCSI traffic to one interface. As far as linux is concerned, the iSCSI and non-iSCSI interfaces are equally good routes to the network.

Here's an example configuration

The iSCSI storage has the IP addresses 172.16.50.70-78.

The server has the following interfaces, addresses, and routes.

$ ip route list
149.76.12.0/24 dev eth0  proto kernel  scope link  src 149.76.12.4
172.16.0.0/16 dev eth1  proto kernel  scope link  src 172.16.50.80 
172.16.0.0/16 dev eth2  proto kernel  scope link  src 172.16.50.81 
172.16.0.0/16 dev eth3  proto kernel  scope link  src 172.16.50.82
default via 149.76.12.1 dev eth0

The desired configuration is for eth3 to be used for non-ISCSI traffic and for eth1 and eth2 to be used for iSCSI traffic. However, non-iSCSI traffic currently goes out eth1.

$ ip route get to 172.16.50.90
172.16.50.90 dev eth1  src 172.16.50.80

(some edits since original posting below)

With my current configuration, if eth1 and eth2 are fully saturated sending iSCSI traffic, my non-iSCSI traffic will compete with the iSCSI traffic on eth1 while eth3 sits idle.

How could I configure linux to prefer to send traffic to the local network using eth3 rather than eth1 or eth2?

I have already set net.ipv4.conf.all.arp_ignore to 1 and net.ipv4.conf.all.arp_announce to 2. This should prevent my IP addresses from floating between the interfaces, e.g. arp flux. I think I just need help with the routing.

(more edits)

Thanks to pfo I started looking at metrics. If I delete the routes and recreate them with the iSCSI interfaces having a higher metric than the non-iSCSI interface, things seem to work the way I want. iSCSI traffic still uses the dedicated interfaces without me setting up static routes to the iSCSI IP addresses. All other local traffic goes out eth3. Now I need to figure out the proper way to set the metrics automatically when the interfaces are brought up. This is on RHEL 5.5.

ip route delete to 172.16.0.0/16 dev eth1
ip route delete to 172.16.0.0/16 dev eth2
ip route delete to 172.16.0.0/16 dev eth3
ip route add to 172.16.0.0/16 dev eth1 src 172.16.50.80 metric 1
ip route add to 172.16.0.0/16 dev eth2 src 172.16.50.81 metric 1
ip route add to 172.16.0.0/16 dev eth3 src 172.16.50.82 metric 0

(final update)

Assigning a different metric using the existing RHEL network scripts seems impossible, https://bugzilla.redhat.com/show_bug.cgi?id=498472

Solution 1

To solve my problem I wrote nethook, a daemon that runs scripts when network interfaces change state on RHEL-based distributions. I have it run this script for interfaces whose route's metric I want to increase.

EDIT: I wrote nethook before I was aware of ifup-local and ifdown-local. You can probably use them instead.

Solution 2

There are some things to take care about when you're multi-homing hosts. First thing is that you need to be aware of the way the Linux TCP/IP stack will handle multiple interfaces being in the same subnet regarding ARP queries and answers - this setting is the interface's arp_filter value that you can query via sysctl(1) or the /proc file system.

0 - (default) The TCP/IP stack will respond to ARP requests with addresses from other interfaces. This may seem wrong but it usually makes sense, because it increases the chance of successful communication. IP addresses are owned by the complete host on Linux, not by particular interfaces.

1 - Allows you to have multiple network interfaces on the same subnet, and have the ARPs for each interface be answered based on whether or not the kernel would route a packet from the ARP'd IP out that interface. In other words it allows control of which NICs will respond to an ARP request and will finally let your TCP/IP flows run.

You should first enable arp_filter on all the interfaces that are in the same subnet and you could easily add an entry to your routing table for your iSCSI portal to use a specific iface and adjust the other interfaces metric to so that one is prefere one over the other.

Another option is to setup source based routing as the default destination based routing will act exactly as you described wrt all ifaces being in the same subnet.

The reason why eth1 is taken out is that it's IP has the lowest numerical value and is such chosen for communication with that network.

Comments

[Bittrance]

Are you fine with the trafic only using "its" interface each or do you require fallback if an interface fails?

[Bittrance]

Come to think of it, if all the interfaces are on the same net, what you really want is for traffic to be equally spread over all of them, no?

[ravi yarlagadda]

If the goal is to segment iSCSI traffic... go for it. Put it on its own unrouted network segment, be it a vlan or dedicated hardware. Enable jumbo frames, dedicate interfaces, and don't have to worry about accidentally disconnecting the wrong interface of the three that, on their face, appear to be identical.

[sciurus]

@Bittrance: I'm fine for the traffic only using its interface. If I could just bond the interfaces and have roughly 3Gb of bandwidth available for all my local traffic I would, but bonding the interfaces doesn't increase my iscsi throughput from 1Gb. I have to keep the interfaces separate and use multipathing for that.

[sciurus]

@Shane: If I had control of the networking equipment, I would do that. I'm trying to do the best I can without that access.

[ravi yarlagadda]

@sciurus Fair enough!

[sciurus]

Based on my understanding of kernel.org/doc/Documentation/networking/ip-sysctl.txt, I think I've handled ARP properly by setting arp_ignore to 1 and arp_announce to 2. I'm not familiar with adjusting the metric for an interface or route, can you go more into that? Also, if you can explain how source routing applies in this situation that would be helpful. I've been reading LARTC but it hasn't clicked yet.