ixgbe RSS not working as expected with 8021q and bridging

ixgbe RSS not working as expected with 8021q and bridging

[Eldon Koyle]

We have built a firewall with two 10 Gbit interfaces (intel 82598EB) and
are doing some testing.  A simple bridge between the two interfaces acts
as expected with packets being distributed fairly evenly across all of
the rx/tx queues.

We then switched to tagged vlans on both interfaces (10 vlans each, 8
source and 8 dest addresses per vlan) and bridged eth0.N to eth1.N, and
many of our queues (and CPUs) remained idle, and all of our VLAN traffic
went out on the same tx queue.  Are multiple transmit queues supported
with 802.1q?  How do we figure out what is causing some of our receive
queues to be unused?

We are using 2.6.31 (from Debian) and ixgbe-2.0.44.14 .

eth0:
     tx_queue_0_packets: 0
     tx_queue_1_packets: 0
     tx_queue_2_packets: 138
     tx_queue_3_packets: 123960995
     tx_queue_4_packets: 0
     tx_queue_5_packets: 0
     tx_queue_6_packets: 36
     tx_queue_7_packets: 36833
     rx_queue_0_packets: 24841529
     rx_queue_1_packets: 24832268
     rx_queue_2_packets: 0
     rx_queue_3_packets: 24835876
     rx_queue_4_packets: 0
     rx_queue_5_packets: 24833101
     rx_queue_6_packets: 0
     rx_queue_7_packets: 24833463

eth1:
     tx_queue_0_packets: 0
     tx_queue_1_packets: 0
     tx_queue_2_packets: 138
     tx_queue_3_packets: 124174228
     tx_queue_4_packets: 0
     tx_queue_5_packets: 0
     tx_queue_6_packets: 82
     tx_queue_7_packets: 78
     rx_queue_0_packets: 24832675
     rx_queue_1_packets: 24789655
     rx_queue_2_packets: 0
     rx_queue_3_packets: 24793220
     rx_queue_4_packets: 0
     rx_queue_5_packets: 24797484
     rx_queue_6_packets: 0
     rx_queue_7_packets: 24786851


-- 
Eldon Koyle
-- 
BOFH excuse #46:
waste water tank overflowed onto computer

Re: ixgbe RSS not working as expected with 8021q and bridging

[Eric Dumazet]

Le 11/12/2009 00:11, Eldon Koyle a écrit :
> We have built a firewall with two 10 Gbit interfaces (intel 82598EB) and
> are doing some testing.  A simple bridge between the two interfaces acts
> as expected with packets being distributed fairly evenly across all of
> the rx/tx queues.
> 
> We then switched to tagged vlans on both interfaces (10 vlans each, 8
> source and 8 dest addresses per vlan) and bridged eth0.N to eth1.N, and
> many of our queues (and CPUs) remained idle, and all of our VLAN traffic
> went out on the same tx queue.  Are multiple transmit queues supported
> with 802.1q?  How do we figure out what is causing some of our receive
> queues to be unused?
> 
> We are using 2.6.31 (from Debian) and ixgbe-2.0.44.14 .
> 

You need more recent kernel (2.6.32) to get multi queue support on vlans, sorry.

commit 2f8bc32b7a08502a79e0ccec8697000f2977f2fd
Author: Eric Dumazet <eric.dumazet@gmail.com>
Date:   Thu Sep 3 02:19:58 2009 -0700

    vlan: enable multiqueue xmits

    vlan_dev_hard_start_xmit() & vlan_dev_hwaccel_hard_start_xmit()
    select txqueue number 0, instead of using index provided by
    skb_get_queue_mapping().

    This is not correct after commit 2e59af3dcbdf11635c03f
    [vlan: multiqueue vlan device] because
    txq->tx_packets  & txq->tx_bytes changes are performed on
    a single location, and not the right locking.

    Fix is to take the appropriate struct netdev_queue pointer

    Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>

Re: ixgbe RSS not working as expected with 8021q and bridging

[Eldon Koyle]

On  Dec 11  1:11+0100, Eric Dumazet wrote:
> Le 11/12/2009 00:11, Eldon Koyle a écrit :
> > We have built a firewall with two 10 Gbit interfaces (intel 82598EB) and
> > are doing some testing.  A simple bridge between the two interfaces acts
> > as expected with packets being distributed fairly evenly across all of
> > the rx/tx queues.
> > 
> > We then switched to tagged vlans on both interfaces (10 vlans each, 8
> > source and 8 dest addresses per vlan) and bridged eth0.N to eth1.N, and
> > many of our queues (and CPUs) remained idle, and all of our VLAN traffic
> > went out on the same tx queue.  Are multiple transmit queues supported
> > with 802.1q?  How do we figure out what is causing some of our receive
> > queues to be unused?
> > 
> > We are using 2.6.31 (from Debian) and ixgbe-2.0.44.14 .
> > 
> 
> You need more recent kernel (2.6.32) to get multi queue support on vlans, sorry.

Excellent.  2.6.32 solved half of the problem.  Now, we are using the
same number of rx and tx queues.  We are still seeing no packets on 3 of
our 8 rx queues on each interface, though (no packets on 2, 4 or 6).
Does the card assign queues in hardware, or is that handled by the
kernel?

-- 
Eldon Koyle

Re: ixgbe RSS not working as expected with 8021q and bridging

[Eric Dumazet]

Le 15/12/2009 18:21, Eldon Koyle a écrit :
> On  Dec 11  1:11+0100, Eric Dumazet wrote:
>> Le 11/12/2009 00:11, Eldon Koyle a écrit :
>>> We have built a firewall with two 10 Gbit interfaces (intel 82598EB) and
>>> are doing some testing.  A simple bridge between the two interfaces acts
>>> as expected with packets being distributed fairly evenly across all of
>>> the rx/tx queues.
>>>
>>> We then switched to tagged vlans on both interfaces (10 vlans each, 8
>>> source and 8 dest addresses per vlan) and bridged eth0.N to eth1.N, and
>>> many of our queues (and CPUs) remained idle, and all of our VLAN traffic
>>> went out on the same tx queue.  Are multiple transmit queues supported
>>> with 802.1q?  How do we figure out what is causing some of our receive
>>> queues to be unused?
>>>
>>> We are using 2.6.31 (from Debian) and ixgbe-2.0.44.14 .
>>>
>>
>> You need more recent kernel (2.6.32) to get multi queue support on vlans, sorry.
> 
> Excellent.  2.6.32 solved half of the problem.  Now, we are using the
> same number of rx and tx queues.  We are still seeing no packets on 3 of
> our 8 rx queues on each interface, though (no packets on 2, 4 or 6).
> Does the card assign queues in hardware, or is that handled by the
> kernel?
> 

When a packet is received (ethernet -> Card), the hardware chooses a RX queue using
hash function.

Then, in your forwarding setup, we (the kernel) automaticaly use same queue to transmit packet.

So if only 5 queues out of 8 receive trafic, it might be because of the flows all map to only 5 queues,
but you should ask Intel people for details :)

(They usualy read netdev)

Re: ixgbe RSS not working as expected with 8021q and bridging

[Eldon Koyle]

On  Dec 15 20:33+0100, Eric Dumazet wrote:
> Le 15/12/2009 18:21, Eldon Koyle a écrit :
> > On  Dec 11  1:11+0100, Eric Dumazet wrote:
> >> Le 11/12/2009 00:11, Eldon Koyle a écrit :
> >>> We have built a firewall with two 10 Gbit interfaces (intel 82598EB) and
> >>> are doing some testing.  A simple bridge between the two interfaces acts
> >>> as expected with packets being distributed fairly evenly across all of
> >>> the rx/tx queues.
> >>>
> >>> We then switched to tagged vlans on both interfaces (10 vlans each, 8
> >>> source and 8 dest addresses per vlan) and bridged eth0.N to eth1.N, and
> >>> many of our queues (and CPUs) remained idle, and all of our VLAN traffic
> >>> went out on the same tx queue.  Are multiple transmit queues supported
> >>> with 802.1q?  How do we figure out what is causing some of our receive
> >>> queues to be unused?
> >>>
> >>> We are using 2.6.31 (from Debian) and ixgbe-2.0.44.14 .
> >>>
> >>
> >> You need more recent kernel (2.6.32) to get multi queue support on vlans, sorry.
> > 
> > Excellent.  2.6.32 solved half of the problem.  Now, we are using the
> > same number of rx and tx queues.  We are still seeing no packets on 3 of
> > our 8 rx queues on each interface, though (no packets on 2, 4 or 6).
> > Does the card assign queues in hardware, or is that handled by the
> > kernel?
> > 
> 
> When a packet is received (ethernet -> Card), the hardware chooses a RX queue using
> hash function.
> 
> Then, in your forwarding setup, we (the kernel) automaticaly use same queue to transmit packet.
> 
> So if only 5 queues out of 8 receive trafic, it might be because of the flows all map to only 5 queues,
> but you should ask Intel people for details :)

I am now trying 2.6.32.7 with the in-tree ixgbe driver.  I am still
seeing some unusual behavior when bridging VLAN interfaces.  It looks
like there is an off-by-one error in the mapping from rx queue to tx
queue (ie. packets are sent on <rx queue number>-1 instead of using the
same rx and tx queue number).

Any idea what might cause this?

The following shows queues vs packets on the physical interfaces (the
counts aren't exact, since not all traffic was bridged):

with vlans:
br3990 (eth0.3990 and eth1.3990)
 queue|      eth0-rx|      eth1-tx|      eth1-rx|      eth0-tx
--------------------------------------------------------------
     0|         6344|       150003|          222|       261103
     1|       150488|       118250|       261103|        80550
     2|       118909|        97282|        80545|       133299
     3|        97588|       104442|       133299|       170111
     4|       105242|       231104|       170001|        68381
     5|       231551|       100548|        68381|       262220
     6|       101202|       262220|       262220|        88248
     7|       262309|            0|        88253|            0

without vlans (expected behavior):
br0 (eth0 and eth1)
 queue|      eth0-rx|      eth1-tx|      eth1-rx|      eth0-tx
--------------------------------------------------------------
     0|         6001|         5977|          159|          159
     1|          541|          541|      1419199|      1419199
     2|      1420084|      1419051|         6270|         6270
     3|        74354|        74312|       305867|       305867
     4|       232547|       232421|       163397|       163397
     5|          367|          365|            2|            2
     6|       170854|       170174|            5|            5
     7|           99|           97|            9|            9

-- 
Eldon Koyle
System Administration Operations
Information Technology
Utah State University
-- 
The rule is, jam to-morrow and jam yesterday, but never jam today.
		-- Lewis Carroll

Re: ixgbe RSS not working as expected with 8021q and bridging

[Eldon Koyle]

On  Feb 10 13:50-0700, Eldon Koyle wrote:
<snip>
> I am now trying 2.6.32.7 with the in-tree ixgbe driver.  I am still
> seeing some unusual behavior when bridging VLAN interfaces.  It looks
> like there is an off-by-one error in the mapping from rx queue to tx
> queue (ie. packets are sent on <rx queue number>-1 instead of using the
> same rx and tx queue number).
> 
> Any idea what might cause this?

Looking at the code, we are having difficulty finding the problem.  I
think we did find an accounting error in the vlan code, though.

In net/8021q/vlan_dev.c:

static netdev_tx_t vlan_dev_hard_start_xmit(struct sk_buff *skb,
                                            struct net_device *dev)
{
        int i = skb_get_queue_mapping(skb);
        struct netdev_queue *txq = netdev_get_tx_queue(dev, i);      
...
        if (likely(ret == NET_XMIT_SUCCESS)) {
                txq->tx_packets++;
                txq->tx_bytes += len;
        } else
                txq->tx_dropped++;
...
}

skb_get_queue_mapping returns skb->queue_mapping, which is rx_queue + 1
or 0 for unset.

Does dev->_tx[0] have some special meaning, or is this wrong?

-- 
Eldon Koyle
-- 
I have a theory that it's impossible to prove anything, but I can't prove it.

Re: ixgbe RSS not working as expected with 8021q and bridging

[Eric Dumazet]

Le vendredi 12 février 2010 à 12:03 -0700, Eldon Koyle a écrit :
> On  Feb 10 13:50-0700, Eldon Koyle wrote:
> <snip>
> > I am now trying 2.6.32.7 with the in-tree ixgbe driver.  I am still
> > seeing some unusual behavior when bridging VLAN interfaces.  It looks
> > like there is an off-by-one error in the mapping from rx queue to tx
> > queue (ie. packets are sent on <rx queue number>-1 instead of using the
> > same rx and tx queue number).
> > 
> > Any idea what might cause this?
> 
> Looking at the code, we are having difficulty finding the problem.  I
> think we did find an accounting error in the vlan code, though.
> 
> In net/8021q/vlan_dev.c:
> 
> static netdev_tx_t vlan_dev_hard_start_xmit(struct sk_buff *skb,
>                                             struct net_device *dev)
> {
>         int i = skb_get_queue_mapping(skb);
>         struct netdev_queue *txq = netdev_get_tx_queue(dev, i);      
> ...
>         if (likely(ret == NET_XMIT_SUCCESS)) {
>                 txq->tx_packets++;
>                 txq->tx_bytes += len;
>         } else
>                 txq->tx_dropped++;
> ...
> }
> 
> skb_get_queue_mapping returns skb->queue_mapping, which is rx_queue + 1
> or 0 for unset.
> 
> Does dev->_tx[0] have some special meaning, or is this wrong?
> 

All multi queue drivers do same thing than vlan_dev_hard_start_xmit()

TX path is supposed to go through dev_pick_tx() & skb_tx_hash(), maybe
bridging has a shortcut and need a fix, like we also do in pktgen (sorry
I wont be able to check this before few days)

Re: ixgbe RSS not working as expected with 8021q and bridging

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Fri, 12 Feb 2010 20:48:17 +0100

> TX path is supposed to go through dev_pick_tx() & skb_tx_hash(), maybe
> bridging has a shortcut and need a fix, like we also do in pktgen (sorry
> I wont be able to check this before few days)

I just made a quick check and bridging seems lack any such shortcuts
and is a good citizen. :-)

It transmits packets using dev_queue_xmit().