ixgbe: setting the number of RX/TX queues

Question

I want to set the number of RX/TX queues used by an Intel 10G NIC. Let me explain why:

I am using an Intel 10G NIC of type X520, on a Dell R720 system. I am using ixgbe version 3.6.7-k. The kernel in Ubuntu 3.2.0-59.

I am running my network application on 4 out of the 24 cores on the machine. Currently the NIC is using flow-director so I've got 24 TX and RX queues, while most of the IRQs finally run on the 4 cores running the application.

However, I see that some IRQs are running on the other 20 queues (this is probably happening as flow-director samples about 20% of the traffic so some traffic goes through regular RSS). Now I don't want any IRQ to be run on the other 20 cores as they are doing a different task which is damaged by the IRQs running.

I tried setting the affinity of the interrupts only to the 4 cores I use, but this does not work well with flow-director. I guess a better approach will be using only 4 RX/TX queues and assigning them to the dedicated cores. But I couldn't find a way to set the number of RX/TX queue in the ixgbe driver (though this is quite simple with other 10G drivers I am familiar with, such as Broadcom's bnx2x).

Any idea?

Solution 1

This is not possible with the version of ixgbe (currently 3.19.1-k) in the latest Linux kernel source (as of 3.18.0-rc1).

You need to grab the latest ixgbe driver (currently 3.22.3) from e1000.sf.net, which supports the RSS parameter. From modinfo ixgbe:

parm: RSS:Number of Receive-Side Scaling Descriptor Queues, default 0=number of cpus (array of int)

So if you have one ixgbe NIC and want 4 queues, you'll need to add a line like this to modprobe.conf (or equivalent in your distro):

options ixgbe RSS=4

Then you'll want to set /proc/irq/*/smp_affinity cpu mask for whatever the irqs are in /proc/interrupts that match your NIC.

Solution 2

Some versions of ixgbe driver included into linux kernel (since 2013, 3.9 kernel, "3.11.33-k" version of ixgbe) can change RSS (queue) count in runtime even without RSS module option. There is ethtool tool to change parameters of network cards, and there are options to change channels:

   ethtool -l|--show-channels devname

   ethtool -L|--set-channels devname [rx N] [tx N] [other N]
          [combined N]


   -l --show-channels
          Queries the specified network device for the numbers of
          channels it has.  A channel is an IRQ and the set of queues
          that can trigger that IRQ.

   -L --set-channels
          Changes the numbers of channels of the specified network
          device.

       rx N   Changes the number of channels with only receive queues.

       tx N   Changes the number of channels with only transmit queues.

       other N
              Changes the number of channels used only for other
              purposes e.g. link interrupts or SR-IOV co-ordination.

       combined N
              Changes the number of multi-purpose channels.

Test current channel (RSS, queue) count of ixgbe eth1 with ethtool -l eth1 and change with ethtool -L eth1 combined 4 or ethtool -L eth1 rx 2 tx 2.

Implemented in net/ethernet/intel/ixgbe/ixgbe_ethtool.c: http://elixir.free-electrons.com/linux/v4.12/source/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c#L3442 static const struct ethtool_ops ixgbe_ethtool_ops = { ... .get_channels = ixgbe_get_channels, .set_channels = ixgbe_set_channels, ... }

ixgbe_get_channels: http://elixir.free-electrons.com/linux/v4.12/source/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c#L3127

ixgbe_set_channels to change adapter->ring_feature[RING_F_RSS].limit: http://elixir.free-electrons.com/linux/v4.12/source/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c#L3164

Implemented since 3.9 version of Linux kernel (around 2013): * http://elixir.free-electrons.com/linux/v3.9/ident/ixgbe_get_channels * https://patchwork.ozlabs.org/patch/211119/ "[RFC,v2,09/10] ixgbe: Add support for displaying the number of Tx/Rx channels" * https://patchwork.ozlabs.org/patch/211120/ "[RFC,v2,10/10] ixgbe: Add support for set_channels ethtool operation diffmbox"