Linux: Bind UDP listening socket to specific interface (or find out the interface a datagram came in from)?

Question

I have a daemon I'm working on that listens for UDP broadcast packets and responds also by UDP. When a packet comes in, I'd like to know which IP address (or NIC) the packet came TO so that I can respond with that IP address as the source. (For reasons involving a lot of pain, some users of our system want to connect two NICs on the same machine to the same subnet. We tell them not to, but they insist. I don't need to be reminded how ugly that is.)

There seems to be no way to examine a datagram and find out directly either its destination address or the interface it came in on. Based on a lot of googling, I find that the only way to find out the target of a datagram is to have one listening socket per interface and bind the sockets to their respective interfaces.

First of all, my listening socket is created this way:

s=socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)

To bind the socket, the first thing I tried was this, where nic is a char* to the name of an interface:

// Bind to a single interface
rc=setsockopt(s, SOL_SOCKET, SO_BINDTODEVICE, nic, strlen(nic));
if (rc != 0) { ... }

This has no effect at all and fails silently. Is the ASCII name (e.g. eth0) the correct type of name to pass to this call? Why would it fail silently? According to man 7 socket, "Note that this only works for some socket types, particularly AF_INET sockets. It is not supported for packet sockets (use normal bind(8) there)." I'm not sure what it means by 'packet sockets', but this is an AF_INET socket.

So the next thing I tried was this (based on bind vs SO_BINDTODEVICE socket):

struct sockaddr_ll sock_address;
memset(&sock_address, 0, sizeof(sock_address));
sock_address.sll_family = PF_PACKET;
sock_address.sll_protocol = htons(ETH_P_ALL);
sock_address.sll_ifindex = if_nametoindex(nic);
rc=bind(s, (struct sockaddr*) &sock_address, sizeof(sock_address));
if (rc < 0) { ... }

That fails too, but this time with the error Cannot assign requested address. I also tried changing the family to AF_INET, but it fails with the same error.

One option remains, which is to bind the sockets to specific IP addresses. I can look up interface addresses and bind to those. Unfortunately, is a bad option, because due to DHCP and hot-plugging ethernet cables, the addresses can change on the fly.

This option may also be bad when it comes to broadcasts and multicasts. I'm concerned that binding to a specific address will mean that I cannot receive broadcasts (which are to an address other than what I bound to). I'm actually going to test this later this evening and update this question.

Questions:

• Is it possible to bind a UDP listening socket specifically to an interface?
• Or alternatively, is there a mechanism I can employ that will inform my program that an interface's address has changed, at the moment that change occurs (as opposed to polling)?
• Is there another kind of listening socket I can create (I do have root privileges) that I can bind to a specific interface, which behaves otherwise identically to UDP (i.e other than raw sockets, where I would basically have to implement UDP myself)? For instance, can I use AF_PACKET with SOCK_DGRAM? I don't understand all the options.

Can anyone help me solve this problem? Thanks!

UPDATE:

Binding to specific IP addresses does not work properly. Specifically, I cannot then receive broadcast packets, which is specifically what I am trying to receive.

UPDATE:

I tried using IP_PKTINFO and recvmsg to get more information on packets being received. I can get the receiving interface, the receiving interface address, the target address of the sender, and the address of the sender. Here's an example of a report I get on receipt of one broadcast packet:

Got message from eth0
Peer address 192.168.115.11
Received from interface eth0
Receiving interface address 10.1.2.47
Desination address 10.1.2.47

What's really odd about this is that the address of eth0 is 10.1.2.9, and the address of ech1 is 10.1.2.47. So why in the world is eth0 receiving packets that should be received by eth1? This is definitely a problem.

Note that I enabled net.ipv4.conf.all.arp_filter, although I think that applies only to out-going packets.

Solution 1

The solution that I found to work is as follows. First of all, we have to change ARP and RP settings. To /etc/sysctl.conf, add the following and reboot (there's also a command to set this dynamically):

net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.all.rp_filter = 2

The arp filter was necessary to allow responses from eth0 to route over a WAN. The rp filter option was necessary to strictly associate in-coming packets with the NIC they came in on (as opposed to the weak model that associates them with any NIC that matches the subnet). A comment from EJP led me to this critical step.

After that, SO_BINDTODEVICE started working. Each of two sockets was bound to its own NIC, and I could therefore tell which NIC a message came from based on the socket it came from.

s=socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
rc=setsockopt(s, SOL_SOCKET, SO_BINDTODEVICE, nic, IF_NAMESIZE);
memset((char *) &si_me, 0, sizeof(si_me));
si_me.sin_family = AF_INET;
si_me.sin_port = htons(LISTEN_PORT);
si_me.sin_addr.s_addr = htonl(INADDR_ANY);
rc=bind(s, (struct sockaddr *)&si_me, sizeof(si_me))

Next, I wanted to respond to in-coming datagrams with datagrams whose source address is that of the NIC the original request came from. The answer there is to just look up that NIC's address and bind the out-going socket to that address (using bind).

s=socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)
get_nic_addr(nics, (struct sockaddr *)&sa)
sa.sin_port = 0;
rc = bind(s, (struct sockaddr *)&sa, sizeof(struct sockaddr));
sendto(s, ...);

int get_nic_addr(const char *nic, struct sockaddr *sa)
{
    struct ifreq ifr;
    int fd, r;
    fd = socket(AF_INET, SOCK_DGRAM, 0);
    if (fd < 0) return -1;
    ifr.ifr_addr.sa_family = AF_INET;
    strncpy(ifr.ifr_name, nic, IFNAMSIZ);
    r = ioctl(fd, SIOCGIFADDR, &ifr);
    if (r < 0) { ... }
    close(fd);
    *sa = *(struct sockaddr *)&ifr.ifr_addr;
    return 0;
}

(Maybe looking up the NIC's address every time seems like a waste, but it's way more code to get informed when an address changes, and these transactions occur only once every few seconds on a system that doesn't run on battery.)

Solution 2

You're passing an illegal value to setsockopt.

rc=setsockopt(s, SOL_SOCKET, SO_BINDTODEVICE, nic, strlen(nic));

The man page says of SO_BIND_TO_DEVICE:

The passed option is a variable-length null-terminated interface name string with the maximum size of IFNAMSIZ

strlen doesn't include the terminating null. You can try:

rc=setsockopt(s, SOL_SOCKET, SO_BINDTODEVICE, nic, 1 + strlen(nic));

dnsmasq has this working correctly, and uses

setsockopt(fd, SOL_SOCKET, SO_BINDTODEVICE, intname, IF_NAMESIZE)

Solution 3

You can get the destination address used by the sender via the IP_RECVDSTADDR option if your platform supports it, by using recvmsg(). It's rather complicated, described in Unix Network Programming, volume I, 3rd edition, #22.2, and in the man page.

Re your edit, you are up against what is known as the 'weak end system model' of TCP/IP. Basically once a packet arrives the system can choose to deliver it via any appropriate interface listening to the correct port. It's discussed in the TCP/IP RFCs somewhere.

Comments

[Ben Voigt]

dnsmasq has an example of this method, too

[Timothy Miller]

It turns out that binding to specific addresses is the wrong answer. When I do that, I cannot receive broadcast packets. While I agree that in general, one interface may have more than one address, that will not be the case here since this is a special-purpose system.

[Timothy Miller]

I've tried both strlen+1 and IF_NAMESIZE, and neither one works. I can send specifically to the address of eth1, and the socket I'm attempting to bind to eth0 is the one that receives it. Therefore, either I'm still doing something wrong here, or this approach doesn't work.

[Timothy Miller]

I investigated this and found the rp_filter option (net.ipv4.conf.all.rp_filter). Setting it to 1 doesn't do anything I could identify. But I set it to 2, and all of a sudden, I started receiving broadcasts on both eth0 and eth1. That's progress! NEVERTHELESS, packets sent specifically to 10.1.2.47 still arrive through eth0, so I can't make sense of that.

[Ben Voigt]

@TimothyMiller: IP_RECVDSTADDR doesn't filter, it just tells you where the packet was received. For filtering you need SO_BINDTODEVICE. It works for dnsmasq, perhaps you should look around the code and find what other socket options are being set.

[Timothy Miller]

@EJP: Your comment about the weak end system model was perhaps the most helpful of all. It indirectly lead me to find rp_filter (which I had to set to 2), which caused Linux to mostly associate datagrams with the NIC they came in on. As a result, SO_BINDTODEVICE actually started working (it had no effect before), and IP_PKTINFO started reporting datagrams as coming from eth1 (previously it only reported eth0). Besides upvoting, I wish there were a way to indicate to stackoverflow, "this response was critical in finding the answer" without implying that it is the whole answer.

[Timothy Miller]

Most implementations that claim to work only pass a 4 for the length when something like "eth0" is passed in.

[Riccardo Manfrin]

I'm just about to cry to see there's another person in this small world, that is cursing kernel for such poor documentation over such a huge gigantic piece of information! This behaviour is just insane! Gonna try and report outcome, but up to now, my symptoms are just about the same as yours.

[init_js]

sysctl can be dynamically changed with /sbin/sysct. sysctl net.ipv4.conf.default.arp_filter to read. sysctl -w net.ipv4.conf.default.arp_filter=x to write