Blocking sockets: when, exactly, does "send()" return?

Question

When, exactly, does the BSD socket send() function return to the caller?

In non-blocking mode, it should return immediately, correct?

As for blocking mode, the man page says:

When the message does not fit into the send buffer of the socket, send() normally blocks, unless the socket has been placed in non-blocking I/O mode.

Questions:

1. Does this mean that the send() call will always return immediately if there is room in the kernel send buffer?
2. Is the behavior and performance of the send() call identical for TCP and UDP? If not, why not?

Solution 1

Does this mean that the send() call will always return immediately if there is room in the kernel send buffer?

Yes. As long as immediately means after the memory you provided it has been copied to the kernel's buffer. Which, in some edge cases, may not be so immediate. For instance if the pointer you pass in triggers a page fault that needs to pull the buffer in from either a memory mapped file or the swap, that would add significant delay to the call returning.

Is the behavior and performance of the send() call identical for TCP and UDP? If not, why not?

Not quite. Possible performance differences depends on the OS' implementation of the TCP/IP stack. In theory the UDP socket could be slightly cheaper, since the OS needs to do fewer things with it.

EDIT: On the other hand, since you can send much more data per system call with TCP, typically the cost per byte can be a lot lower with TCP. This can be mitigated with sendmmsg() in recent linux kernels.

As for the behavior, it's nearly identical.

For blocking sockets, both TCP and UDP will block until there's space in the kernel buffer. The distinction however is that the UDP socket will wait until your entire buffer can be stored in the kernel buffer, whereas the TCP socket may decide to only copy a single byte into the kernel buffer (typically it's more than one byte though).

If you try to send packets that are larger than 64kiB, a UDP socket will likely consistently fail with EMSGSIZE. This is because UDP, being a datagram socket, guarantees to send your entire buffer as a single IP packet (or train of IP packet fragments) or not send it at all.

Non blocking sockets behave identical to the blocking versions with the single exception that instead of blocking (in case there's not enough space in the kernel buffer), the calls fail with EAGAIN (or EWOULDBLOCK). When this happens, it's time to put the socket back into epoll/kqueue/select (or whatever you're using) to wait for it to become writable again.

As usual when working on POSIX, keep in mind that your call may fail with EINTR (if the call was interrupted by a signal). In this case you most likely want to call send() again.

Solution 2

If there is room in the kernel buffer, then send() copies as many bytes as it can into the buffer and exits immediately, returning how many bytes were actually copied (which can be fewer than how many you requested). If there is no room in the kernel buffer, then send() blocks until either room becomes available or a timeout occurs (if one is configured).

Solution 3

The send() will return as soon as the data has been accepted by the kernel. In case of blocking socket: The send() will block if the kernel buffer is not free enough to intake the data provided to send() call.

Non blocking sockets: send() will not block, but would fail and returns -1 or it may return number of bytes copied partially(depending on the buffer space available). It sets the errno EWOULDBLOCK or EAGAIN. This means at that time of send(), the buffer was not able to intake all the bytes and you should try again with select() call to send() the data again. Or you could put a loop with a sleep() and call send(), but you have to take care of number of bytes actually sent and the remaining number of bytes that are to be sent.

Comments

[David Citron]

Er, I'm trying to ascertain if the function call will block on any kind of I/O and, if so, for how long. It wasn't an existential question :-)

[K. M. Rajib Faysal]

How does that work with the TCP socket only copying part of your buffer? Does it just return a written value < your message length? If it returns EAGAIN then you have no way of knowing how many bytes were written...

[Arvid]

EAGAIN is not the return value, it's the value of errno. send() on a TCP socket either tells you how many bytes it took or -1 if something went wrong. In the case of error you're expected to look at errno for details. EAGAIN means it can't take anymore since its send buffer is already full (or above the high watermark)

[Remember Monica]

jumbo frames are ethernet frames larger than the standard size - udp itself is always limited to 64KiB, jumbo frames or not.

[Kr0e]

[...] The distinction however is that the UDP socket will wait until your entire buffer can be stored in the kernel buffer [...] - Not true in BSD systems. Packets simply get dropped unfortunately, if the sending buffer is full. A real shame! I've noticed this a couple of months ago. It's hard to write correct UDP apps on BSD when writing at full speed.

[Arvid]

interesting. do you get any indication of error? EAGAIN for instance?

[Guillaume Paris]

What happens if the bytes can be delivered to client? how originator will be notified?

[Remy Lebeau]

Once the kernel has accepted bytes into the buffer, it is out of your hands. All you can do is accept that those bytes will be transmitted in the background and move on. There is no notification that they have actually been transmitted, or if/when they have been received (unless the receiver sends a reply). If buffer space runs out, send() will block (if in blocking mode) until space becomes available, or will fail immediately with EWOULDBLOCK or EAGAIN (if in non-blocking mode). You can use select(), poll(), etc to detect if space is available before calling send(), if needed.

[Mecki]

@Arvid On BSD and macOS/iOS, if you try so send more data over an UDP socket and that data does not fit into the socket buffer anymore, send() fails at once with error ENOBUFS. This happens regardless if the socket is blocking or non-blocking. So a UDP socket will never block on send and also never return EAGAIN on these systems. And this is legit behavior as UDP gives no guarantees on data delivery to begin with. If the send buffer runs full, there is backlog and somewhere along the way backlog will always drop, preferable early so that the app notices it and can react accordingly.