From: Jesper Dangaard Brouer <brouer@redhat.com>
To: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Alexander Duyck <alexander.duyck@gmail.com>,
netdev <netdev@vger.kernel.org>,
Alexander Duyck <aduyck@mirantis.com>,
John Fastabend <john.r.fastabend@intel.com>,
Jamal Hadi Salim <jhs@mojatatu.com>,
brouer@redhat.com
Subject: Re: [RFC] net: remove busylock
Date: Mon, 23 May 2016 11:50:00 +0200 [thread overview]
Message-ID: <20160523115000.40e25fab@redhat.com> (raw)
In-Reply-To: <1463779960.5939.7.camel@edumazet-glaptop3.roam.corp.google.com>
On Fri, 20 May 2016 14:32:40 -0700
Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Fri, 2016-05-20 at 19:49 +0200, Jesper Dangaard Brouer wrote:
> > On Fri, 20 May 2016 07:16:55 -0700
> > Eric Dumazet <eric.dumazet@gmail.com> wrote:
> >
> > > Since bonding pretends to be multiqueue, TCQ_F_ONETXQUEUE is not set
> > > on sch->flags when HTB is installed at the bonding device root.
> >
> > If would be cool if you could run a test with removed busylock and
> > allow HTB to bulk dequeue.
>
> I added a /sys/class/net/eth0/tx_bulk_limit to tune the number of
> packets that we could bulk dequeue from a virtual device
> (no BQL on them)
>
> 200 TCP_RR through HTB on eth0 (bonding device)
>
> 1) busylock enabled
> --------------------
>
> With tx_bulk_limit set to 8, we get 12.7 % increase.
That is actually a quite good performance increase for this workload.
> lpaa23:~# for f in `seq 1 16`; do echo $f|tee /sys/class/net/eth0/tx_bulk_limit; sar -n DEV 3 3|grep eth0|grep Average; done
> 1
> Average: eth0 868625.67 868487.44 57055.69 56826.37 0.00 0.00 0.56
> 2
> Average: eth0 927081.67 926920.78 60923.83 60649.90 0.00 0.00 0.44
> 3
> Average: eth0 957678.11 957554.00 62877.04 62653.89 0.00 0.00 0.56
> 4
> Average: eth0 966912.44 966747.33 63532.72 63255.51 0.00 0.00 0.56
> 5
> Average: eth0 973137.56 972950.44 63958.31 63661.39 0.00 0.00 0.44
> 6
> Average: eth0 958589.22 958449.44 62961.79 62712.56 0.00 0.00 0.67
> 7
> Average: eth0 960737.67 960672.22 62968.34 62857.97 0.00 0.00 0.44
> 8
> Average: eth0 979271.78 979201.67 64199.47 64070.84 0.00 0.00 0.56
> 9
> Average: eth0 982464.33 982390.33 64418.42 64278.93 0.00 0.00 0.56
> 10
> Average: eth0 982698.00 982578.22 64495.25 64291.28 0.00 0.00 0.44
> 11
> Average: eth0 981862.22 981746.00 64438.16 64236.31 0.00 0.00 0.56
> 12
> Average: eth0 983277.44 983096.33 64632.79 64327.79 0.00 0.00 0.44
> 13
> Average: eth0 981221.11 981018.00 64521.82 64189.26 0.00 0.00 0.67
> 14
> Average: eth0 981754.11 981555.89 64553.19 64224.39 0.00 0.00 0.44
> 15
> Average: eth0 982484.33 982301.67 64572.00 64273.38 0.00 0.00 0.44
> 16
> Average: eth0 978529.56 978326.67 64350.89 64013.39 0.00 0.00 0.67
>
>
> 2) busylock disabled
> --------------------
>
> Well, bulk dequeue helps, but does not close the gap.
> (busylock is needed)
Interesting observation, that busylock trick is still needed. I would
have liked to see this getting cleaned up, but I guess it is not that
trivial to "fix"/remove.
> lpaa23:~# for f in `seq 1 16`; do echo $f|tee /sys/class/net/eth0/tx_bulk_limit; sar -n DEV 3 3|grep eth0|grep Average; done
> 1
> Average: eth0 795408.44 795407.67 52044.66 52045.93 0.00 0.00 0.56
> 2
> Average: eth0 843411.78 843415.11 55185.23 55184.51 0.00 0.00 0.56
> 3
> Average: eth0 876175.89 876175.00 57329.50 57327.98 0.00 0.00 0.44
> 4
> Average: eth0 890631.22 890629.44 58274.58 58274.25 0.00 0.00 0.67
> 5
> Average: eth0 900672.00 900668.89 58931.29 58930.54 0.00 0.00 0.44
> 6
> Average: eth0 908325.78 908328.22 59432.97 59431.76 0.00 0.00 0.56
> 7
> Average: eth0 913895.33 913885.11 59796.89 59795.46 0.00 0.00 0.56
> 8
> Average: eth0 914429.11 914433.56 59832.26 59831.23 0.00 0.00 0.67
> 9
> Average: eth0 918701.11 918699.67 60110.68 60110.36 0.00 0.00 0.55
> 10
> Average: eth0 920382.33 920376.56 60223.31 60220.54 0.00 0.00 0.67
> 11
> Average: eth0 914341.67 914344.67 59826.25 59825.90 0.00 0.00 0.67
> 12
> Average: eth0 912697.00 912693.78 59718.77 59717.44 0.00 0.00 0.44
> 13
> Average: eth0 917392.56 917385.00 60025.79 60024.34 0.00 0.00 0.44
> 14
> Average: eth0 918232.89 918233.78 60081.04 60079.94 0.00 0.00 0.67
> 15
> Average: eth0 918377.11 918381.00 60091.14 60089.79 0.00 0.00 0.44
> 16
> Average: eth0 913817.56 913812.33 59792.09 59790.66 0.00 0.00 0.56
>
Thanks for doing the experiment.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer
next prev parent reply other threads:[~2016-05-23 9:50 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-05-19 17:08 [RFC] net: remove busylock Eric Dumazet
2016-05-19 18:03 ` Alexander Duyck
2016-05-19 18:41 ` Rick Jones
2016-05-19 18:56 ` Eric Dumazet
2016-05-19 19:35 ` Eric Dumazet
2016-05-19 20:39 ` Alexander Duyck
2016-05-20 4:49 ` John Fastabend
2016-05-20 4:56 ` Eric Dumazet
2016-05-20 7:29 ` Jesper Dangaard Brouer
2016-05-20 13:11 ` Eric Dumazet
2016-05-20 13:47 ` Eric Dumazet
2016-05-20 14:16 ` Eric Dumazet
2016-05-20 17:49 ` Jesper Dangaard Brouer
2016-05-20 21:32 ` Eric Dumazet
2016-05-23 9:50 ` Jesper Dangaard Brouer [this message]
2016-05-23 21:24 ` [PATCH net] net_sched: avoid too many hrtimer_start() calls Eric Dumazet
2016-05-24 21:49 ` David Miller
2016-05-24 13:50 ` [RFC] net: remove busylock David Laight
2016-05-24 14:37 ` Eric Dumazet
2016-05-20 16:01 ` John Fastabend
2016-05-19 18:12 ` David Miller
2016-05-19 18:44 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160523115000.40e25fab@redhat.com \
--to=brouer@redhat.com \
--cc=aduyck@mirantis.com \
--cc=alexander.duyck@gmail.com \
--cc=eric.dumazet@gmail.com \
--cc=jhs@mojatatu.com \
--cc=john.r.fastabend@intel.com \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).