Re: VLAN egress performance

[Eric Dumazet <eric.dumazet@gmail.com>]

Le lundi 08 février 2010 à 14:13 +0100, Patrick McHardy a écrit :
> Mattias Rönnblom wrote:
> > Hi.
> > 
> > I'm running Linux on PC w/ a Core i7 CPU and two Intel 82598 NICs, and
> > I see some anomalies when it comes to egress VLAN performance. I
> > thought maybe someone on this list was interested in my results.
> > 
> > I'm running the stock Ubuntu 2.6.31 kernel, but with a newer ixgbe
> > driver (2.0.44.14).
> > 
> > The benchmark is IP forwarding with unidirectional UDP flows @ 64 byte
> > packets, and I get:
> > 
> > Ingress VLAN  Egress VLAN  Packet Rate    CPU utilization (all cores)
> > No            No           5.0 Mpacket/s  ~70%
> > Yes           No           5.0 Mpacket/s  ~75%
> > No            Yes          1.4 Mpacket/s  ~26%
> > Yes           Yes          1.3 Mpacket/s  ~26%
> > 
> > "VLAN" here mean I've put a VLAN device on top of the real ixgbe
> > device.
> > 
> > As you can see, if the egress i/f is a VLAN i/f, the performance is
> > reduced to less than a third. And in the case of egress VLAN, the
> > systems basically only uses one HW thread (with a softirqd process
> > taking up all the time).
> > 
> > Enabling lockdep, it looks like execution is serialized to a large
> > extent by contention around the "vlan_netdev_xmit_lock_key" lock.
> > 
> > I call this anomaly because I was surprised to see it (as oppose to
> > other performance degradations/scalability issues in the area of
> > multicore and IP traffic handling performance).
> 
> 2.6.32 contains VLAN multiqueue support and should scale better.

Yes, patches were added in September 2009, 2.6.32 should be OK.

commit 2f8bc32b7a08502a79e0ccec8697000f2977f2fd
Author: Eric Dumazet <eric.dumazet@gmail.com>
Date:   Thu Sep 3 02:19:58 2009 -0700

    vlan: enable multiqueue xmits
    
    vlan_dev_hard_start_xmit() & vlan_dev_hwaccel_hard_start_xmit()
    select txqueue number 0, instead of using index provided by
    skb_get_queue_mapping().
    
    This is not correct after commit 2e59af3dcbdf11635c03f
    [vlan: multiqueue vlan device] because
    txq->tx_packets  & txq->tx_bytes changes are performed on
    a single location, and not the right locking.
    
    Fix is to take the appropriate struct netdev_queue pointer
    
    Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>

commit 2e59af3dcbdf11635c03f22bfc9706744465d589
Author: Eric Dumazet <eric.dumazet@gmail.com>
Date:   Wed Sep 2 18:03:00 2009 -0700

    vlan: multiqueue vlan device
    
    vlan devices are currently not multi-queue capable.
    
    We can do that with a new rtnl_link_ops method,
    get_tx_queues(), called from rtnl_create_link()
    
    This new method gets num_tx_queues/real_num_tx_queues
    from real device.
    
    register_vlan_device() is also handled.
    
    Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>