Change the order of IP addresses returned by ifconfig?

Question

I have an Ubuntu server with several IP addresses attached to it. 127.0.0.1 is listed as venet0 by ifconfig. I'm using Chef to configure the server. The problem is that chef is listing 127.0.0.1 as the IP address for the server instead of one of the server's "real" IPs. (apparent "ohai ipaddress" uses the first IP listed by ifconfig to determine the server's IP).

How can I change the order so the servers main IP is listed first instead of the 127.0.0.1?

Can venet0 be deleted and venet0:0 be "promoted" to take its place since 127.0.0.1 is already listed in the "lo" interface?

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:334 errors:0 dropped:0 overruns:0 frame:0
          TX packets:334 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:16700 (16.7 KB)  TX bytes:16700 (16.7 KB)

venet0    Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:127.0.0.1  P-t-P:127.0.0.1  Bcast:0.0.0.0  Mask:255.255.255.255
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1
          RX packets:7622207 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8183436 errors:0 dropped:1 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2102750761 (2.1 GB)  TX bytes:2795213667 (2.7 GB)

venet0:0  Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:XXX.XXX.XXX.XX1  P-t-P:XXX.XXX.XXX.XX1  Bcast:0.0.0.0  Mask:255.255.255.255
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1

venet0:1  Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:XXX.XXX.XXX.XX2  P-t-P:XXX.XXX.XXX.XX2  Bcast:0.0.0.0  Mask:255.255.255.255
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1

route -n

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.0.2.1       0.0.0.0         255.255.255.255 UH    0      0        0 venet0
0.0.0.0         192.0.2.1       0.0.0.0         UG    0      0        0 venet0

Solution

ohai uses route to find the "default interface" by finding your default gateway (destination 0.0.0.0):

network[:default_interface] = from("route -n \| grep -m 1 ^0.0.0.0 \| awk \'{print \$8\}\'")

(From line 21 as of right now)

The problem is that route doesn't care about the aliases on interfaces (venet0:0 and venet0:1 are aliases of the venet0 interface), it assumes that anything sent through the aliases all goes out the same device, so it lists venet0 as the outgoing interface. This is logical when the interface is a physical interface, but when it's a virtual interface it could be wrong (and is wrong in this case). Because of this behavior, it's also technically wrong for ohai to rely on it to determine the "correct" IP address to use, even if the base IP address wasn't 127.0.0.1.

The ideal fix is going to be for you to reconfigure your network settings so that the un-aliased venet0 interface is your "primary IP". If you REALLY need venet0 to be 127.0.0.1 for some reason (I'm not familiar with the venet* interfaces so I don't know why you have it this way or what would happen if venet0 was the primary IP address instead of 127.0.0.1), then you might try finding that network.rb file (/usr/lib/ruby/1.8/ohai/plugins/linux/network.rb in Lucid) and editing it to read

network[:default_interface] = "venet0:0"

(or whichever interface you want it to report as the correct address). I don't know if from() strips the newline that the command would print at the end so it might need to be "venet0:0\n" for ohai to work right. Note that this will be replaced when you upgrade the package it was in.

I am personally curious as to whether ip route list (from the iproute package) shows your "default" route using the venet0 interface or venet0:0 interface. Likewise, netstat -r. If either shows venet0:0 then you might suggest in a bug report to ohai that they try those commands first and if they don't work, try again with the "normal" route command.

Comments

[Andy]

The answer is probably not in the ordering of interfaces but the configuration of chef to use a specific interface rather than pick the first it finds.

[Andy]

At a first glance at the configuration I can't find anything mentioning the interface. Perhaps your server's hostname is set to localhost and so resolves to the loopback adapter? If so, a fix could be to change the hostname to a domain pointing to the external ip.

[DerfK]

Almost certainly the issue isn't the order of interfaces but the order of routes. What does route -n say? Really, though, trying to make the loopback address be something other than localhost screams of "doing the wrong thing". Why are you not using private IPs for your virtual network? (eg, 192.168.55.x to pick a random network)

[erikcw]

@DerfK - I've added the results of route -n to my question.

[erikcw]

Because Chef uses the first one it finds and stores it as the servers main IP. This attribute is used in all kinds of configuration recipes...

[Dennis Williamson]

@erikcw: See my edited answer. The commented line would contain the PCI ID for the device (if it's PCI) and a short description. I believe the comment is optional.

[Tom O'Connor]

Don't! Don't ever remove the loopback device.. This can break some seriously dark magic.

[erikcw]

Tom: I already have a seperate device labeled lo for 127.0.0.1. Is it necessary to also have the IP listed on the venet0 interface? Seems redundant...