Kernel socket structure and TCP_DIAG

Question

I'm working on a software which connects to a Real Time data server (using TCP) and I have some connections dropping. My guess is that the clients do not read the data coming from the server fast enough. Therefore I would like to monitor my TCP sockets. For this I found the "ss" tool.

This tool allows to see the state of every socket - here's an example line of the output of the command ss -inm 'src *:50000'

ESTAB      0      0             184.7.60.2:50000       184.92.35.104:1105
  mem:(r0,w0,f0,t0) sack rto:204 rtt:1.875/0.75 ato:40

My question is: what does the memory part mean? Looking at the source code of the tool I found that the data is coming from a kernel structure (sock in sock.h). More precisely, it comes from the fields :

r = sk->sk_rmem_alloc
w = sk->sk_wmem_queued;
f = sk->sk_forward_alloc;
t = sk->sk_wmem_alloc;

Does somebody know what they mean? My guesses are:

• rmem_alloc : size of the inbound buffer
• wmem_alloc : size of the outbound buffer
• sk_forward_alloc : ???
• sk->sk_wmem_queued : ???

Here are my buffers sizes :

net.ipv4.tcp_rmem = 4096        87380   174760
net.ipv4.tcp_wmem = 4096        16384   131072
net.ipv4.tcp_mem = 786432       1048576 1572864
net.core.rmem_default = 110592
net.core.wmem_default = 110592
net.core.rmem_max = 1048576
net.core.wmem_max = 131071

Solution 1

sk_forward_alloc is the forward allocated memory which is the total memory currently available in the socket's quota.

sk_wmem_queued is the amount of memory used by the socket send buffer queued in the transmit queue and are either not yet sent out or not yet acknowledged.

You can learn more about TCP Memory Management in chapter 9 of TCP/IP Architecture, Design and Implementation in Linux By Sameer Seth, M. Ajaykumar Venkatesulu

Solution 2

See the man page of ss.

<fwd_alloc>
   The  memory allocated by the socket as cache, but not used for receiving/sending packet yet. If need memory to send/receive packet, the memory in this cache will be used before allocate additional memory.

<wmem_queued>
   The memory allocated for sending packet (which has not been sent to layer 3)

Solution 3

Regarding sk_wmem_queued and sk_wmem_alloc, I asked the same question so I'll copy the answer here:

I emailed Eric Dumazet, who contributes to the Linux network stack, and here is the answer:

sk_wmem_alloc tracks the number of bytes for skb queued after transport stack : qdisc layer and NIC TX ring buffers.

If you have 1 MB of data sitting in TCP write queue, not yet sent (cwnd limit), sk_wmem_queue will be about 1MB, but sk_wmem_alloc will be about 0

A very good document for understanding what these three types of queues (socket buffer, qdisc queue and device queue) are is this article (rather long) article. In a nutshell, the socket starts by pushing the packets directly onto the qdisc queue, which forwards them to the device queue. When the qdisc queue is full, the socket starts buffering the data in its own write queue.

the network stack places packets directly into the queueing discipline or else pushes back on the upper layers (eg socket buffer) if the queue is full

So basically: sk_wmem_queues is the memory used by the socket buffer (sock.sk_write_queue) while sk_wmem_alloc is the memory used by the packets in the qdisc and device queues.

Comments

[little-dude]

I don't understand how this definition of sk_wmem_queued differs from sk_wmem_alloc, could you expand a little on this? (If you know the answer, feel free to add an answer to this question: unix.stackexchange.com/questions/551444/…)

[milosgajdos]

It this bytes or pages?