Why is my TCP throughput much greater than UDP throughput?

Question

I haven't done anything unusual to my hardware or kernel configurations (all default settings, fresh OS install, Linux kernel 3.11 TCP/IP stack) and I'm averaging about 3.83 million messages per second through TCP while I'm only averaging 0.75 million messages per second through UDP. This seems to completely defy what I expect of the two protocols.

What's the most likely cause for the drastic difference and how can I diagnose it on Ubuntu 13.10?

#TCP RESULTS
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

87380  65536     64    10.00      1963.43   32.96    17.09    5.500   2.852

#UDP RESULTS
Socket  Message  Elapsed      Messages                   CPU      Service
Size    Size     Time         Okay Errors   Throughput   Util     Demand
bytes   bytes    secs            #      #   10^6bits/sec % SS     us/KB

4194304      64   10.00     7491010      0      383.5     28.97    24.751
212992            10.00     1404941              71.9     25.03    21.381

For this test I have two test servers that are identical and directly connected via a 10G crossover cable. The NICs used in this case are Intel X520's with out-of-box configurations and connected onto a PCIe 3.0 x8 slot on the motherboard, which communicates with the CPU via a NUMA controller.

Solution

Apart from not getting detailed information about your test setup the main problem seems to be, that you use a message size of 64 byte. This is far away from the usual MTU of 1500 bytes and makes UDP highly inefficient: while TCP merges multiple sends into a single packet on the wire (except if TCP_NODELAY is set) to make efficient use of the link, each UDP message will result in a separate packet. In numbers: about 23 messages of size 64 byte will be combined into a single TCP packet of MTU size, while it will need 23 single packets for UDP for the same amount of data. Each of these packets means overhead with sending from the host, transmitting on the wire and receiving by the peer. And as seen in your case about 80% of the UDP packets get lost because your hardware is not fast enough to transmit and receive all these packets.

So what you can learn from this benchmark is:

• UDP is unreliable (80% packet loss)
• UDP is inefficient if used with packet sizes far below MTU
• TCP is highly optimized to make best use of the link

As for your expectation, that UDP should be better: did you ever wonder why all the major file transfers (ftp, http,...) are done with TCP based protocols? The benchmark shows you the reason.

So why do people use UDP at all?

• With real-time data (e.g. voice over IP) you don't care about older messages, so you don't want the sender to combine messages into larger packets to make effective use of the link. And you rather accept that a packet gets lost than to have it arrive too late.
• With high-latency links (like with satellites) the default behavior of TCP is not optimal to make effective use of the link. So some people switch to UDP in this case and re-implement the reliability layer of TCP and optimize it for high-latency links, while others tune the existing TCP stack to make better use of the link.
• "throw away" data: sometimes it is more important to send the data away and don't care about packet loss, like with log messages (syslog)
• Short interactions: with TCP you need to establish a connection an maintain a state, which costs time and resources at client and server. For short interactions (like short request and reply) this might be too much overhead. Because of this DNS is usually done with UDP but has built retries on top of UDP.

Comments

[elleciel]

Interesting! Upvoted. I was under the impression that the extra error-checking, ack and recovery overhead on TCP would incur a lower throughput, but it seems that UDP is only faster but does not produce higher throughput. (stackoverflow.com/questions/47903/…) I will try lower the MTU setting on the interface and see if it makes a difference. Will accept this as answer if no one has a competing theory.

[Steffen Ullrich]

You should also have a look at your 80% packet loss with UDP. It looks like your hardware is not fast enough to process the packets in the same speed they get send. While TCP adapts to this kind of packet loss with slowing down, UDP will just send at the same speed and continue to loose packets. But at the end it is not relevant how fast you can send, but what you receive.

[cpugeniusmv]

Something else that might be a factor is TCP acceleration/offloading to the network card (if it supports it).

[elleciel]

Great follow-up answer, it seems that I am either CPU-bound or memory-bound, which is shocking - I am using a 4.1 GHz Ivy Bridge test machine and the socket library should be single-threaded.

[Steffen Ullrich]

Packet sending might be more efficient than receiving, especially if the last one is interrupt driven.

[ratchet freak]

people also use UDP for an embedded device to broadcast the data it is collecting over a wire and not bother with the connection setup

[Roney Michael]

Here is given one answer to TCP faster than UDP. But there are other situation where TCP is faster than UDP too - stackoverflow.com/questions/47903/… (@Robert S. Barnes)

[dev_ter]

You are most likely IO bound by the PCI express bus. The network cards will have TCP segment offloading enabled, most likely. This means that TCP transfers will be sent to the card asone big block, then the card slices and dices them into packets and puts them on the wire. There is no equivalent for UDP, so the result is one PCIe transaction (and all associated overheads) for each packet.

[yegle]

I think its worth emphasis on the fact that reliability can be archived in some situation with simple retry, like you mentioned in the DNS case.

[nhed]

sometimes, if the nature of the traffic is bursty it may make sense to allocate more mem to UDP - If netstat -su show drops it may help