Re: VLAN egress performance

Eric Dumazet
2010-02-08T13:24:36Z

Le lundi 08 février 2010 à 14:13 +0100, Patrick McHardy a écrit :
> Mattias Rönnblom wrote:
> > Hi.
> >
> > I'm running Linux on PC w/ a Core i7 CPU and two Intel 82598 NICs, and
> > I see some anomalies when it comes to egress VLAN performance. I
> > thought maybe someone on this list was interested in my results.
> >
> > I'm running the stock Ubuntu 2.6.31 kernel, but with a newer ixgbe
> > driver (2.0.44.14).
> >
> > The benchmark is IP forwarding with unidirectional UDP flows @ 64 byte
> > packets, and I get:
> >
> > Ingress VLAN Egress VLAN Packet Rate CPU utilization (all cores)
> > No No 5.0 Mpacket/s ~70%
> > Yes No 5.0 Mpacket/s ~75%
> > No Yes 1.4 Mpacket/s ~26%
> > Yes Yes 1.3 Mpacket/s ~26%
> >
> > "VLAN" here mean I've put a VLAN device on top of the real ixgbe
> > device.
> >
> > As you can see, if the egress i/f is a VLAN i/f, the performance is
> > reduced to less than a third. And in the case of egress VLAN, the
> > systems basically only uses one HW thread (with a softirqd process
> > taking up all the time).
> >
> > Enabling lockdep, it looks like execution is serialized to a large
> > extent by contention around the "vlan_netdev_xmit_lock_key" lock.
> >
> > I call this anomaly because I was surprised to see it (as oppose to
> > other performance degradations/scalability issues in the area of
> > multicore and IP traffic handling performance).
>
> 2.6.32 contains VLAN multiqueue support and should scale better.

Yes, patches were added in September 2009, 2.6.32 should be OK.

commit 2f8bc32b7a08502a79e0ccec8697000f2977f2fd
Author: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu Sep 3 02:19:58 2009 -0700

vlan: enable multiqueue xmits

vlan_dev_hard_start_xmit() & vlan_dev_hwaccel_hard_start_xmit()
select txqueue number 0, instead of using index provided by
skb_get_queue_mapping().

This is not correct after commit 2e59af3dcbdf11635c03f
[vlan: multiqueue vlan device] because
txq->tx_packets & txq->tx_bytes changes are performed on
a single location, and not the right locking.

Fix is to take the appropriate struct netdev_queue pointer

Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

commit 2e59af3dcbdf11635c03f22bfc9706744465d589
Author: Eric Dumazet <eric.dumazet@gmail.com>
Date: Wed Sep 2 18:03:00 2009 -0700

vlan: multiqueue vlan device

vlan devices are currently not multi-queue capable.

We can do that with a new rtnl_link_ops method,
get_tx_queues(), called from rtnl_create_link()

This new method gets num_tx_queues/real_num_tx_queues
from real device.

register_vlan_device() is also handled.

Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>