Secondary IP (eth0:0) acts like main server IP

Question

I have a CentOS server, configured with 4 consecutive IPs:
eth0 5.x.x.251
eth0:0 5.x.x.252
eth0:1 5.x.x.253
eth0:2 5.x.x.254

The problem is that all traffic goes out to the internet with eth0:0 (5.x.x.252) as the source IP, instead of eth0.
# curl ifconfig.me
5.x.x.252

How can I fix this, so that all traffic goes out via eth0, ie my main IP?

PS: My server is VPS running on a Xen dom0, the latter being configured in routed mode networking.

Thanks in advance!

Server configuration

# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:x:x:x:x:AE  
          inet addr:5.x.x.251  Bcast:5.x.x.255  Mask:255.255.255.255
          inet6 addr: fe80::x:x:x:x/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:14675569 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9463227 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:4122016502 (3.8 GiB)  TX bytes:25959110751 (24.1 GiB)
          Interrupt:23 

eth0:0    Link encap:Ethernet  HWaddr 00:x:x:x:x:AE   
          inet addr:5.x.x.252  Bcast:5.x.x.255  Mask:255.255.255.224
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:23 

eth0:1    Link encap:Ethernet  HWaddr 00:x:x:x:x:AE  
          inet addr:5.x.x.253  Bcast:5.x.x.255  Mask:255.255.255.224
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:23 

eth0:2    Link encap:Ethernet  HWaddr 00:x:x:x:x:AE  
          inet addr:5.x.x.254  Bcast:5.x.x.255  Mask:255.255.255.224
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:23  

---

# cat /etc/hosts
127.0.0.1       localhost.localdomain localhost
5.x.x.251       [fqdn] [hostname]

---

# cat ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
IPADDR=5.x.x.251
NETMASK=255.255.255.224
SCOPE="peer 5.x.y.82"

---

# cat ifcfg-eth0:0
DEVICE=eth0:0
BOOTPROTO=static
ONBOOT=yes
IPADDR=5.x.x.252
NETMASK=255.255.255.224

---

# cat route-eth0 
ADDRESS0=0.0.0.0
NETMASK0=0.0.0.0
GATEWAY0=5.x.y.82

---

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
5.x.y.82        0.0.0.0         255.255.255.255 UH        0 0          0 eth0
5.x.x.224       0.0.0.0         255.255.255.224 U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
0.0.0.0         5.x.y.82        0.0.0.0         UG        0 0          0 eth0

Update 29 August 2012
When I run /etc/init.d/network restart I get an RTNETLINK answers: File exists error, which leads me to believe that there's another route to another network using the default gateway, as seen here.

# /etc/init.d/network restart
Shutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0:  RTNETLINK answers: File exists
                                                           [  OK  ]

The sure thing is that when I first delete ip route del default via 5.x.y.82 dev eth0 and then add it again with the .251 IP, everything works as expected.
Is there any way to add a route del command either to the ifcfg-eth0 or the route-eth0 files? I can see that this can be done with pre-up ip addr del ... on Debian (more info), but what about CentOS?

Solution 1

From this document it appears that at least on CentOS 5 and I suspect above you can specify your routes by simply giving the correct iproute2 command arguments. (see the section on the IP Command Arguments Format)

So instead of writing something like:

# route-eth0 
ADDRESS0=0.0.0.0
NETMASK0=0.0.0.0
GATEWAY0=5.x.y.82

You could just have a file like this:

default via 5.x.y.82 dev eth0  src 5.x.x.251

I don't have a Redhat/Redhat derived box handy to test though.

Solution 2

First clear your 0.0.0.0 default gateway

route del -net default

Then declare default eth0 as default gateway device

route add -net default gw 5.x.y.82 dev eth0

It should work. Verify with

ip ro li

To save that as a rule try this messy thing:

Open /etc/sysconfig/network-scripts/network-functions, locate function add_default_route () and in that function, after the call to find_gateway_dev add a line with GATEWAYDEV="eth0" .

What happens is that the network-functions script decides which device gets to be the default gateway. It examines the route to the gateway IP with ip get route to GATEWAY and, using sed, matches the a device. It seems that eth0:0 gets to be the one each time, so by hardcoding GATEWAYDEV="eth0" you make sure that eth0 is always selected as the gateway device.

In older versions of RH you could just edit /etc/sysconfig/static-routes and add default via 5.x.y.82 dev eth0

As a last resort /etc/rc.local is always your friend!

Solution 3

If the aliased IP addresses are not to be used as source addresses to non-local destinations, they should not be in the same subnet as the target of your default route. So change their netmasks to 255.255.255.255 and remove their broadcast addresses.

Solution 4

Sounds like you might be a victim of a setting in newer kernels. Enabling alias promotion will get past that.

Set this in your /etc/sysctl.conf

net.ipv4.conf.all.promote_secondaries = 1

Incidentally, do you have any funky SNAT/MASQUERADE iptables rules that are altering the outbound source IP?

Solution 5

Since they are all on the same subnet the kernel will pick one to use.. You can verify this by running

 ip addr list dev eth0

You have to do some iptables magic to get it to work correctly. See the following

When IP aliasing how does the OS determine which IP address will be used as source for outbound TCP/IP connections?

View more solutions

Comments

[Andrew Smith]

Try ip ro li and see what is your src

[George Tasioulis]

Here's the output of ip ro li: pastebin.com/A1KRiWTz I think the last line is what I'm looking for... how can I change src 5.x.x.252 to scr 5.x.x.251?

[joechip]

Are you using an official CentOS kernel? Try "uname -a". I've seen similarly weird behavior (e.g., arp flux) with OpenVZ kernels.

[George Tasioulis]

I'm giving the correct answer & bounty to Zoredache who was the one to find the source of the problem in the first place. With his help I managed to at least fix it on the live system, and upon restart (via /etc/rc.local). Only thing left to fix is when I run /etc/init.d/network restart but ok I can live without it :)

[George Tasioulis]

Here's the output of ip addr list dev eth0: pastebin.com/mFv4XCb9 I understand that it can be fixed with iptables, but my main concern first of all is WHY this is happening? I just noticed that the eth0 subnet mask from ifconfig compared to ifcfg-eth0 is different...

[George Tasioulis]

I just tried it, unfortunately even when defining the .251 IP and restarting networking, ip ro li still reports the .252 as source.

[Zoredache]

How annoying. If you remove the default route and add it back manually do things work properly? ip route del default via 5.x.y.82 dev eth0; ip route add default via 5.x.y.82 dev eth0 src 5.x.x.251?

[George Tasioulis]

Let me login via console from the dom0 to try this out and i'll get back to you in a second.

[George Tasioulis]

Yay! deleting the ip default route and adding it back works, BUT if I restart the VM or do a /etc/init.d/network restart I'm back to .252 :( I guess could add the above command to /etc/rc.local to cover the VM restart part...

[George Tasioulis]

Kalispera :) Zoredache already proposed this as a solution and it partially works. What I mean is that it will work fine until I either reboot the VM or run /etc/init.d/network restart. Then I'm back to the .252 IP

[Panagiotis Moustafellos]

On older RH versions you could add at /etc/syconfig/static-routes a default gw 5.x.y.82 dev eth0 On newer versions edit /etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network-scripts/ifcfg-eth0:0, remove from eth0:0 the DEFROUTE=yes and add it to eth0, I am also editing this on my reply above.

[Zoredache]

Hrm, well it sure seems from the documentation that the route-interface file should support any valid iproute command. You may need to either go with the rc.local fix, or hope someone else can give you an answer.

[George Tasioulis]

I tried the DEFROUTE=yes thing on ifcfg-eth0 (also tried adding DEFROUTE=no to the other ifcfg-eth0:? files, but it didn't work. PS: I have updated my question with some extra information

[Panagiotis Moustafellos]

What I am about to suggest is a bit messy but does the trick. Open /etc/sysconfig/network-scripts/network-functions, locate function add_default_route () and in that function, after the call to find_gateway_dev add a line with GATEWAYDEV="eth0" Placing that in my answer.

[George Tasioulis]

Unfortunatelly this didn't do the trick either... :( Here's a screenshot. I got the RTNETLINK answers: File exists error again. Until now the only thing that worked is manually deleting the default route and adding it again. This File exists error occurs because of two default routes. Does my route -n help?

[Panagiotis Moustafellos]

I am assuming that for this "hack" to take place, first you 'll have to route del -net default and then /etc/init.d/network reload. Reload once more to see if the error pops up. If not, you are good to go.

[George Tasioulis]

Tried that too. I connected to the VPS via console, did the ip route del default ... first, then restarted /etc/init.d/network two times to check. Same error both times (RTNETLINK answers: File exists).

[GioMac]

+1 for the answer. Please read official docs, route-eth0 shouldn't contain these fields.