Setting the source IP for a UDP socket

Question

I have a UDP socket that is bound to INADDR_ANY to listen to packets on all the IPs my server has. I'm sending out replies through the same socket.

Right now the server chooses automatically which IP is used as the source IP when packets are sent out, but I would like to be able to set the outgoing source IP myself.

Is there any way to do that without having to create a separate socket for each IP ?

Solution 1

Nikolai, using a separate socket and bind(2) for each address or messing with routing tables is often not a feasible option e.g. with dynamic addresses. A single IP_ADDRANY-bound UDP server should be able to appear to respond on the same dynamically-assigned IP address a packet is received on.

Luckily, there is another way. Depending on your system's support you can make use of the IP_PKTINFO socket options to set or receive ancillary data about a message. Ancillary data (via cmsg(3)) is covered in many places online though comp.os.linux.development.system had a full code sample specific to IP_PKTINFO.

The code in the link uses IP_PKTINFO (or IP_RECVDSTADDR depending on the platform) to get the destination address of a UDP message from the ancillary cmsg(3) data. Paraphrased here:

struct msghdr msg;
struct cmsghdr *cmsg;
struct in_addr addr;
// after recvmsg(sd, &msg, flags);
for(cmsg = CMSG_FIRSTHDR(&msg);
    cmsg != NULL;
    cmsg = CMSG_NXTHDR(&msg, cmsg)) {
  if (cmsg->cmsg_level == IPPROTO_IP && cmsg->cmsg_type == IP_PKTINFO) {
    addr = ((struct in_pktinfo*)CMSG_DATA(cmsg))->ipi_addr;
    printf("message received on address %s\n", inet_ntoa(addr));
  }
}

Gene, your question asked how to set the source address on outgoing packets. With IP_PKTINFO it is possible to set the ipi_spec_dst field of the struct in_pktinfo in the ancillary data passed to sendmsg(2). See the post referenced above, cmsg(3), and sendmsg(2) for guidelines on how to create and manipulate the ancillary data in a struct msghdr. An example (no guarantee here) might be:

struct msghdr msg;
struct cmsghdr *cmsg;
struct in_pktinfo *pktinfo;
// after initializing msghdr & control data to CMSG_SPACE(sizeof(struct in_pktinfo))
cmsg = CMSG_FIRSTHDR(&msg);
cmsg->cmsg_level = IPPROTO_IP;
cmsg->cmsg_type = IP_PKTINFO;
cmsg->cmsg_len = CMSG_LEN(sizeof(struct in_pktinfo));
pktinfo = (struct in_pktinfo*) CMSG_DATA(cmsg);
pktinfo->ipi_ifindex = src_interface_index;
pktinfo->ipi_spec_dst = src_addr;
// bytes_sent = sendmsg(sd, &msg, flags);

Note this is different in IPv6: use struct in6_pktinfo::ipi6_addr in both the recvmsg and sendmsg cases.

Note also that Windows does not support an equivalent to ipi_spec_dst in the in_pktinfo struct, so you cannot use this method to set the source address on an outgoing winsock2 packet.

(man pages referenced - getting around 1 hyperlink limit)

http:// linux.die.net/man/2/sendmsg
http:// linux.die.net/man/3/cmsg

Solution 2

I thought I'd expand Jeremy's on how to do this for IPv6. Jeremy leaves out a lot of detail, and some documentation (like Linux's man page for ipv6) is just plain wrong. First on some distributions you have to define _GNU_SOURCE, otherwise some of the IPv6 stuff isn't defined:

#define _GNU_SOURCE
#include <netinet/in.h>
#include <sys/types.h>
#include <sys/socket.h>

Next set up the socket in a fairly standard way that listens for all IP packets (ie, both IPv4 and IPv6) on a particular UDP port:

const int on=1, off=0;
int result;
struct sockaddr_in6 sin6;
int soc;

soc = socket(AF_INET6, SOCK_DGRAM, 0);
setsockopt(soc, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on));
setsockopt(soc, IPPROTO_IP, IP_PKTINFO, &on, sizeof(on));
setsockopt(soc, IPPROTO_IPV6, IPV6_RECVPKTINFO, &on, sizeof(on));
setsockopt(soc, IPPROTO_IPV6, IPV6_V6ONLY, &off, sizeof(off));
memset(&sin6, '\0', sizeof(sin6));
sin6.sin6_family = htons(AF_INET6);
sin6.sin6_port = htons(MY_UDP_PORT);
result = bind(soc, (struct sockaddr*)&sin6, sizeof(sin6));

Notice the code above sets both IP and IPv6 options for an IPv6 socket. Turns out if the packet arrives on an IPv4 address, you will get IP_PKTINFO (ie IPv4) cmsg's even though it is an IPv6 socket, and if you don't enable them they won't be sent. Also notice the IPV6_RECPKTINFO option is set (which isn't mentioned in man 7 ipv6), not IPV6_PKTINFO (which is described wrongly in man 7 ipv6). Now receive a udp packet:

int bytes_received;
struct sockaddr_in6 from;
struct iovec iovec[1];
struct msghdr msg;
char msg_control[1024];
char udp_packet[1500];

iovec[0].iov_base = udp_packet;
iovec[0].iov_len = sizeof(udp_packet);
msg.msg_name = &from;
msg.msg_namelen = sizeof(from);
msg.msg_iov = iovec;
msg.msg_iovlen = sizeof(iovec) / sizeof(*iovec);
msg.msg_control = msg_control;
msg.msg_controllen = sizeof(msg_control);
msg.msg_flags = 0;
bytes_received = recvmsg(soc, &msg, 0);

The next step is to extract the interface and address the UDP packet was received on out of the cmsg:

struct in_pktinfo in_pktinfo;
struct in6_pktinfo in6_pktinfo;
int have_in_pktinfo = 0;
int have_in6_pktinfo = 0;
struct cmsghdr* cmsg;

for (cmsg = CMSG_FIRSTHDR(&msg); cmsg != 0; cmsg = CMSG_NXTHDR(&msg, cmsg))
{
  if (cmsg->cmsg_level == IPPROTO_IP && cmsg->cmsg_type == IP_PKTINFO)
  {
    in_pktinfo = *(struct in_pktinfo*)CMSG_DATA(cmsg);
    have_in_pktinfo = 1;
  }
  if (cmsg->cmsg_level == IPPROTO_IPV6 && cmsg->cmsg_type == IPV6_PKTINFO)
  {
    in6_pktinfo = *(struct in6_pktinfo*)CMSG_DATA(cmsg);
    have_in6_pktinfo = 1;
  }
}

Finally we get to send the response back, using the same destination.

int cmsg_space;

iovec[0].iov_base = udp_response;
iovec[0].iov_len = udp_response_length;
msg.msg_name = &from;
msg.msg_namelen = sizeof(from);
msg.msg_iov = iovec;
msg.msg_iovlen = sizeof(iovec) / sizeof(*iovec);
msg.msg_control = msg_control;
msg.msg_controllen = sizeof(msg_control);
msg.msg_flags = 0;
cmsg_space = 0;
cmsg = CMSG_FIRSTHDR(&msg);
if (have_in6_pktinfo)
{
  cmsg->cmsg_level = IPPROTO_IPV6;
  cmsg->cmsg_type = IPV6_PKTINFO;
  cmsg->cmsg_len = CMSG_LEN(sizeof(in6_pktinfo));
  *(struct in6_pktinfo*)CMSG_DATA(cmsg) = in6_pktinfo;
  cmsg_space += CMSG_SPACE(sizeof(in6_pktinfo));
}
if (have_in_pktinfo)
{
  cmsg->cmsg_level = IPPROTO_IP;
  cmsg->cmsg_type = IP_PKTINFO;
  cmsg->cmsg_len = CMSG_LEN(sizeof(in_pktinfo));
  *(struct in_pktinfo*)CMSG_DATA(cmsg) = in_pktinfo;
  cmsg_space += CMSG_SPACE(sizeof(in_pktinfo));
}
msg.msg_controllen = cmsg_space;
ret = sendmsg(soc, &msg, 0);

Again notice how, if the packet came in via IPv4 we have to put an IPv4 option into the cmsg's even though it is an AF_INET6 socket. At least, that is what you have to do for Linux.

It is a surprising amount of work, but AFAICT it is the minimum you have to do to make a robust UDP server that works in all conceivable Linux environments. Most of it is not required for TCP because it handles multihoming transparently.

Solution 3

You either bind(2) to each interface address and manage multiple sockets, or let the kernel do the implicit source IP assignment with INADDR_ANY. There is no other way.

My question would be - why do you need this? Is normal IP routing not working for you?