[RFC PATCH v2 00/10] running qdiscs without qdisc_lock

[John Fastabend <john.fastabend@gmail.com>]

Hi,

I thought I should go ahead and send this series out for comments.
Here I allow qdiscs to be run without taking the qdisc lock. As a
result statistics, gso skb, tx bad skb and a few other things need
to be "safe" to run without locks. It _should_ all be covered here.
Although I just noticed I must be missing a dec on the backlog
counter somewhere as one of my tests just ended with 0packets but
a nonzero bytes counter.

Also of note in this series I used the skb_array implementation
already in net-next for the tun/tap devices. With this implementation
for cases where lots of threads are hitting the same qdisc I see
a modest improvement but for cases like mq with pktgen where
everything is lined up nicely I see a fairly unpleasant regression.

I have a few thoughts on how to resolve this. First if we support
bulk_dequeue as an operation on the skb_array this should help
vs getting the consumer lock repeatedly. Also we really don't need
the HARD_TX_LOCK if we have a core per queue and XPS setup like many
multiqueue nics default to. And I need to go back and look at the
original alf ring implementation as well to see how it compares I
don't recall seeing the mq regression there.

Also after the above it might be nice to make all qdiscs support
the per cpu statistics and drop non per cpu cases just to simplify
the code and all the if/else branching where its not needed.

As usual any thoughts, comments, etc are welcome.

And I wasn't going to add these numbers just because they come from
an untuned system but why not.

Here are some initial numbers from pktgen on my development which
is a reasonable system (E5-2695) but I didn't do any work to tweak
the config so there is still a bunch of debug/hacking options still
running.

The pktgen command is

./samples/pktgen/pktgen_bench_xmit_mode_queue_xmit.sh -i eth3  -t X -s 64

pfifo_fast

        original pps        lockless        diff
1       1418168             1269450         -148718
2       1587390             1553408         -33982
4       1084961             1683639         +598678
8       989636              1522723         +533087
12      1014018             1348172         +334154

mq
       original pps        lockless        diff
1      1442018             1205180        -236838 
2      2646069             2266095        -379974
4      5136200             4269470        -866730
8      
12     13275671            10810909       -2464762

---

John Fastabend (10):
      net: sched: allow qdiscs to handle locking
      net: sched: qdisc_qlen for per cpu logic
      net: sched: provide per cpu qstat helpers
      net: sched: a dflt qdisc may be used with per cpu stats
      net: sched: per cpu gso handlers
      net: sched: support qdisc_reset on NOLOCK qdisc
      net: sched: support skb_bad_tx with lockless qdisc
      net: sched: pfifo_fast use alf_queue
      net: sched: helper to sum qlen
      net: sched: add support for TCQ_F_NOLOCK subqueues to sch_mq


 include/net/gen_stats.h   |    3 
 include/net/sch_generic.h |  105 ++++++++++++
 net/core/dev.c            |   32 +++-
 net/core/gen_stats.c      |    9 +
 net/sched/sch_api.c       |   12 +
 net/sched/sch_generic.c   |  385 +++++++++++++++++++++++++++++++++++----------
 net/sched/sch_mq.c        |   25 ++-
 7 files changed, 467 insertions(+), 104 deletions(-)