How sendmsg works?

Question

As you know sendmsg has this declaration:

int sendmsg(int s, const struct msghdr *msg, int flags);

and msghdr structure has this form:

struct msghdr {
    void         * msg_name;     /* optional address */
    socklen_t    msg_namelen;    /* size of address */
    struct iovec * msg_iov;      /* scatter/gather array */
    size_t       msg_iovlen;     /* # elements in msg_iov */
    void         * msg_control;  /* ancillary data, see below */
    socklen_t    msg_controllen; /* ancillary data buffer len */
    int          msg_flags;      /* flags on received message */
};

As you see msghdr has an array of buffer, iovec and has buffer count msg_iovlen. What I wonder is that how sendmsg sends these buffers. Does it concatenate all buffers and send or does it send in a for loop?

Solution 1

The manpage speaks of a message (singular) and multiple elements (plural):

For send() and sendto(), the message is found in buf and has length len. For sendmsg(), the message is pointed to by the elements of the array msg.msg_iov. The sendmsg() call also allows sending ancillary data (also known as control information).

For a stream socket, it wouldn't matter either way. Any data you send will just end up as one long stream of data on the other side.

For datagram or message sockets, I can see why a bit more clarity would be helpful. But it appears that you send just one datagram or message with a single sndmsg call; not one per buffer element.

I actually went digging in the Linux source code out of curiosity and to get a better feeling about this answer. It looks like send, and sendto are just wrappers for sendmsg in Linux, that build the struct msghdr for you. And in fact, the UDP sendmsg implementation makes room for one UDP header per sendmsg call.

If performance is what you're worried about, it doesn't look like you'll benefit from sendmsg if you pass in just a single iovec. If you're concatenating buffers in user-space, though, this could potentially win you some.

It's a bit similar to writev, with the added benefit that you can specify a destination address for use with connectionless sockets like UDP. You can also add ancillary data, if you're into that sort of thing. (Commonly used to send file descriptors across UNIX domain sockets.)

Solution 2

According to http://opengroup.org/onlinepubs/007908799/xns/sendmsg.html ...

The data from each storage area indicated by msg_iov is sent in turn.

My interpretation is that sendmsg() will not concatenate the message data stored in the iovec's; each will be sent as a separate message.

[Edit: My interpretation was not correct; see the other answers for a better explanation.]

Solution 3

It depends on your TCP/IP stack. Embedded TCP/IP stacks could potentially send the different iovecs directly to the NIC. But on usual TCP/IP stacks there must already be a copy from userspace memory to kernelspace memory, so there is no gain there, and iovecs get conceptually copied to a single big chunk of memory (it can be separate pages of memory, if the driver supports scather/gather I/O, but the important part here is that iovec boundaries don't get preserved).