How do I increase the ring parameters for a NIC on a Linux server

Question

I used the ethtool utility to increase the rx and tx values for the NIC on one of our servers. I ran the following command:

ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:             2040
RX Mini:        0
RX Jumbo:       8160
TX:             255
Current hardware settings:
RX:             2040
RX Mini:        0
RX Jumbo:       0
TX:             255

Can I change the preset maximums on the card in some way? Or are they a hardware limitation. the NIC we have on the server is : Broadcom NetXtreme II BCM5709 1000Base-T

Solution

Well, there's the example of the pre-set maximum ring buffer figures on Broadcom bnx2 devices being modified in the kernel from 1020 to 2040 a few years ago, so it is possible.

diff --git a/drivers/net/bnx2.h b/drivers/net/bnx2.h
index efdfbc2..62ac83e 100644
--- a/drivers/net/bnx2.h
+++ b/drivers/net/bnx2.h
@@ -6502,8 +6502,8 @@ struct l2_fhdr {
 #define TX_DESC_CNT  (BCM_PAGE_SIZE / sizeof(struct tx_bd))
 #define MAX_TX_DESC_CNT (TX_DESC_CNT - 1)

-#define MAX_RX_RINGS        4
-#define MAX_RX_PG_RINGS        16
+#define MAX_RX_RINGS        8
+#define MAX_RX_PG_RINGS        32
 #define RX_DESC_CNT  (BCM_PAGE_SIZE / sizeof(struct rx_bd))
 #define MAX_RX_DESC_CNT (RX_DESC_CNT - 1)
 #define MAX_TOTAL_RX_DESC_CNT (MAX_RX_DESC_CNT * MAX_RX_RINGS)

You can attempt some of this; I've seen those MAX_RX_RINGS and MAX_RX_PG_RINGS values pushed to 16 and 64 before in certain kernel/driver builds. These are routinely the onboard NICs for Dell PowerEdge and HP ProLiant servers, and a few people in my industry would hack these drivers to make the NICs a bit more usable. But know that it may make sense to understand where the performance issues are. Also know that other NICs models/drivers have bigger ring buffers than the Broadcom.

Intel:

# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:     4096
RX Mini:    0
RX Jumbo:   0
TX:     4096
Current hardware settings:
RX:     4096
RX Mini:    0
RX Jumbo:   0
TX:     2048

Try profiling your application and seeing where the drops are. You didn't specify OS distribution or version, so I can't give too much distro-specific info. A handy portable tool is dropwatch. You can use it to see if drops are happening at the IP, link or application layers.

# dropwatch -l kas

1 drops at tcp_rcv_established+916 (0xffffffff814ae5c6)
2 drops at tcp_v4_rcv+aa (0xffffffff814b78aa)
2 drops at tcp_rcv_established+916 (0xffffffff814ae5c6)
1 drops at skb_copy_datagram_from_iovec+2fe (0xffffffff81455dde)
1 drops at skb_copy_datagram_from_iovec+2fe (0xffffffff81455dde)
2 drops at tcp_v4_rcv+aa (0xffffffff814b78aa)
2 drops at skb_copy_datagram_from_iovec+2fe (0xffffffff81455dde)
1 drops at tcp_v4_rcv+aa (0xffffffff814b78aa)
1 drops at tcp_v4_rcv+aa (0xffffffff814b78aa)
18 drops at unix_stream_connect+1dc (0xffffffff814f4cdc)
2 drops at tcp_v4_rcv+aa (0xffffffff814b78aa)

Comments

[ewwhite]

People don't usually tune these parameters in general applications, but are you seeing performance issues or drops at the moment? Which specific OS/distribution/kernel are you using?

[user3405780]

We did a pcap by mirroring the port on which the server is connected. But there were no packet drops that we observed.There are performance issues on the application which is running on the server. We are using Linux/Kernel 2.6/. However if these parameters are a limitation of the hardware, we will go back and tune the application parameters.

[ewwhite]

Which specific Linux OS/distribution and kernel are you using? It matters.

[user3405780]

We are using CentOS 6.5

[ewwhite]

Good. You have some options. See my answer below.