What is the effect of setting a linux socket - high priority?

Question

From the linux socket manpage:

SO_PRIORITY
Set the protocol-defined priority for all packets to be sent on this socket. Linux uses this value to order the networking queues: packets with a higher priority may be processed first depending on the selected device queueing discipline.

And this is set using:

int optval=7 // valid values are in the range [1,7]  
             // 1- low priority, 7 - high priority  
setsockopt(socket, SOL_SOCKET, SO_PRIORITY, &optval, optlen)   

And say, the process has:
a. 10 low priority sockets (priority=4) from socket_1 - socket_10,
b. 1 high priority socket (priority=7) - socket_11

What will happen in the following scenarios:

1. send(): process sends multiple msgs on socket_1-socket_10 & on socket_11, IMO msgs on the socket_11 will be given preference over the msgs being sent over socket_1-socket_10.

2. recv(): if there are mutilple msgs being received at all the sockets described above, does socket_11 gets higher priority in reading the messages over socket_1-socket_10 ?

3. Is there a way to measure the socket priority from command line using tools such as lsof, netstat etc?

Solution 1

Every Linux network interface has a so called qdisc (queuing discipline) attached to it. And the answer to your questions depends on the qdisc in use. Some queuing disciplines like pfifo and bfifo, have no concept of priority. So if they're used, the answer is simple - there will be no prioritization

However, with a prioritizing qdisc such as pfifo_fast (which typically the default qdisc on Linux), the socket priority can have an effect.

This image describes what's going on in a pfifo_fast qdisc:

We see that packets are placed in queues depending on their priority. When the time comes for the interface to send the next packet (frame actually, but let's not get into that), it will always choose to send the packet with the highest priority. That means that if multiple packets are waiting, those with the highest priority will be sent first. Note that this requires the interface to be congested - if the interface isn't congested and packets are sent as soon as they arrive from the OS, then there is no queuing and therefore no prioritization.

Other qdiscs have different structures and policies. For example an SFQ qdisc:

With that in mind, let's get back to your questions:

1. Depending on the qdisc, yes, packets from socket_11 may be sent ahead of packets from other sockets. If pfifo_fast is used, and if socket_11 sends enough traffic to saturate the outbound network interface, then the packets from the other sockets might even not be sent at all. This is unlikely in practice since it's usually hard to saturate a network interface before saturating some other resource, unless it's a wireless interface.

2. The path that packets take from the machine's network interface to the socket is much faster than the network itself. And, as you recall, for prioritization to have any effect, there has to be congestion. In a typical scenario, packets that reached as far as your network interface have already passed the bottleneck of their journey across the network, so congestion is unlikely.

   You can of course use an ingress qdisc or other mechanisms to artificially create a bottleneck, and prioritize incoming traffic. But why would you? That only makes sense if you're building a traffic shaper or a similar network device. Plus, since this qdiscs are a low level mechanism that happens well below the higher level sockets (even before bridging or routing), I doubt that the socket's priority could have any effect on in.

3. Not that I'm aware of, but I'd be happy to learn. This kernel module comes close, but it doesn't seem to be able to show priority flags, just regular socket options.

Solution 2

Answers on your questions:

1. Yes by default, explanation is below
2. No priorities work only on send, explanation is also below
3. I don't think so because this options is not seen even via /proc interface

Details about 1

Some words about priorities, network queues and device disciplines. All this stuff is related to quality of service and particularly to Differentiated Services (DiffServ).

When packet is sent then it put to interface "queue" that processed by network device. By default queue is not real queue but three real fifos that are strongly prioritised. If there is packet in fifo0 than packets in fifo1 waiting. Socket priority is mapped to this fifo by following mapping:

• 0 (Best Effort) is fifo1
• 1-3 (Filler, Bulk, ...) is fifo2
• 4 is fifo1
• 5 is fifo2
• 6-7 (Interactive, Control) is fifo0
• 8-15 is fifo1

So priority 1 will be dispatched after priority 0.

For changing default behaviour "traffic control" (tc) utility is used. With it you can configure priority queues on the network interface. This is so called "device queueing discipline". You can define how priorities are served by the particular network device (answer of Malt has good explanation of this).

Details about 2

Socket has states "ready for read"/"not ready for read" and this is bool. If any data is arrived to not ready socket it changes state from "not ready" to "ready" and this is seen by functions like select/poll or by returning from blocked recv call. What thread will be woken depends not by socket priority but by thread priority.

So if you want to prioritise sockets you need

• either put them on prioritized threads
• or prioritise by code after select/poll

Comments

[brokenfoot]

Thanks for the detailed answer.. it helps a lot..how can i check the qdisc being used in my linux distribution?

[Malt]

tc qdisc show. But it's going to be pfifo_fast :)

[brokenfoot]

Thanks for the detailed response.. for #1, if my queuing discipline is pfifo_fast, then the ans will be yes? as per @Malt's answer

[brokenfoot]

mm.. tc isn't installed on my system. Is there an alternative way to check? or I can safely assume my kernel has pfifo_fast ? My linux kernel version is 3.10.19

[Malt]

try ip addr show. You should see something like eth0: <BLAH BLAH BLAH> mtu 1500 qdisc pfifo_fast

[Dmitry Poroh]

@brokenfoot I've looked at linux kernel sources. SO_PRIORITY comes to socket priority field and after copied to all packets (skb->priority) outgoing from this socket. skb->priority is really used for selection of fifo. But with strange mapping: priority 0 mapped to fifo1, 1-3 to fifo2, 4 to fifo1, 5 to fifo2 and 6 to fifo0.

[Dmitry Poroh]

@brokenfoot I've updated answer in agree with my new knowledge.

[brokenfoot]

can you please mention the source information about socket-priority mapping to fifos. Thanks!

[EML]

@Dmitry: are you sure about this? Your mapping shows the mapping from the 4 TOS bits to queue/FIFO - isn't this different from the mapping from SO_PRIORITY to queue?

[joar]

You can install tc by installing iproute apt-get install iproute