what happens when I write data to a blocking socket, faster than the other side reads?

Question

suppose I write data really fast [I have all the data in memory] to a blocking socket. further suppose the other side will read data very slow [like sleep 1 second between each read].

what is the expected behavior on the writing side in this case? would the write operation block until the other side reads enough data, or will the write return an error like connection reset?

Solution 1

For a blocking socket, the send() call will block until all the data has been copied into the networking stack's buffer for that connection. It does not have to be received by the other side. The size of this buffer is implementation dependent.

Data is cleared from the buffer when the remote side acknowledges it. This is an OS thing and is not dependent upon the remote application actually reading the data. The size of this buffer is also implementation dependent.

When the remote buffer is full, it tells your local stack to stop sending. When data is cleared from the remote buffer (by being read by the remote application) then the remote system will inform the local system to send more data.

In both cases, small systems (like embedded systems) may have buffers of a few KB or smaller and modern servers may have buffers of a few MB or larger.

Once space is available in the local buffer, more data from your send() call will be copied. Once all of that data has been copied, your call will return.

You won't get a "connection reset" error (from the OS -- libraries may do anything) unless the connection actually does get reset.

So... It really doesn't matter how quickly the remote application is reading data until you've sent as much data as both local & remote buffer sizes combined. After that, you'll only be able to send() as quickly as the remote side will recv().

Solution 2

Output (send) buffer gets filled until it gets full and send() block until the buffer get freed enough to enqueue the packet.

As send manual page says:

When the message does not fit into the send buffer of the socket, send() normally blocks, unless the socket has been placed in non- blocking I/O mode.

Look at this: http://manpages.ubuntu.com/manpages/lucid/man2/send.2.html

Comments

[wildplasser]

It will block once your side's buffers are full enough.

[Aviad Rozenhek]

@wildplasser this is what I thought will happen too, however I'm using the poco c++ libs to do just that, and the write operation throws an exception of connection reset by peer ...

[wildplasser]

The library can do whatever it wishes. And that may or may not be related to the state of the socket at that moment.

[user207421]

@wildplasser 'Connection reset by peer' is a TCP condition originating in the OS. Any library that reports that when it isn't true should be turfed immediately. I've never heard of one.

[user207421]

If what you are really asking is why you are getting connection resets, the usual cause is that you have written to a connection that had already been closed by the peer.

[wildplasser]

@EJP : I know what it means. What I wanted to say is that the "connection reset" errno is totally unrelated to the question in the OP (can a blocking read block?) The library is -at best- irrelevant, and -at worst- faulty.

[Aviad Rozenhek]

hmm, what happens when the send buffer is big[say 1 mb], but the receive buffer is very small [say 10 bytes] and the writer fills the send buffer, and the reader does not read anything ?

[Davide Berra]

The OS remove a packet from the send buffer only if it got confirmation it's delivered. If the receiver never communicate the packet was correctly sent (reply with an ACK after receiving a PUSH DATA), the sender buffer never get freed.

[WorM]

nice answer. with remote (client), local(server) in mind, if remote buffer is full, can it still be able to send data to local ? is read and write buffer is shared or separate ?

[Brian White]

The incoming and outgoing buffers ("windows") are completely independent. A stall in one direction does not impede traffic in the other direction.

[cslotty]

Which also means on a lack of processing power on the sender side, processes might get "blocked" and you don't see any of it in your TCP trace, because no packets are outgoing.

[Brian White]

@cslotty, in that case you should see an occasional "zero window probe" where the client sends single-byte packets to the server to check if space has freed up but the announcement of such was lost.