From: Eric Dumazet <eric.dumazet@gmail.com>
To: Dave Taht <dave.taht@gmail.com>
Cc: jg@freedesktop.org, Stephen Hemminger <shemminger@vyatta.com>,
Juliusz Chroboczek <jch@pps.jussieu.fr>,
Kathleen Nichols <nichols@pollere.com>,
netdev <netdev@vger.kernel.org>
Subject: Re: BQL + Basic Latency under load results - 100Mbit, GSO/TSO off, pfifo_fast vs SFQ vs QFQ
Date: Mon, 02 Jan 2012 05:55:25 +0100 [thread overview]
Message-ID: <1325480125.2526.13.camel@edumazet-laptop> (raw)
In-Reply-To: <1325478811.2526.10.camel@edumazet-laptop>
Le lundi 02 janvier 2012 à 05:33 +0100, Eric Dumazet a écrit :
> Le lundi 02 janvier 2012 à 00:17 +0100, Dave Taht a écrit :
> > QFQ wins even bigger vs SFQ at 50 iperfs
> >
> > http://www.teklibre.com/~d/bloat/pfifo_sfq_vs_qfq_linear50.png
> >
> > And I think it's going to win even bigger at 10 Mbit.
> >
>
> Happy new year !
>
> This makes no sense to me for such a low amount of flows, SFQ should
> perform the same than QFQ :)
>
> You dont find out why it is so.
>
>
>
> Please try following patch :
>
> [PATCH net-next] sch_sfq: dont put new flow at the end of flows
>
> SFQ enqueue algo puts a new flow _behind_ all pre-existing flows in the
> circular list. In fact this is probably an old SFQ implementation bug.
>
> 100 Mbits = ~8333 full frames per second, or ~8 frames per ms.
>
> With 50 flows, it means your "new flow" will have to wait 50 packets
> being sent before its own packet. Thats the ~6ms.
>
> We certainly can change SFQ to give a priority advantage to new flows,
> so that next dequeued packet is taken from a new flow, not an old one.
>
> Reported-by: Dave Taht <dave.taht@gmail.com>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> ---
> diff --git a/net/sched/sch_sfq.c b/net/sched/sch_sfq.c
> index c23b957..f7f62a5 100644
> --- a/net/sched/sch_sfq.c
> +++ b/net/sched/sch_sfq.c
> @@ -366,11 +366,11 @@ sfq_enqueue(struct sk_buff *skb, struct Qdisc *sch)
> if (slot->qlen == 1) { /* The flow is new */
> if (q->tail == NULL) { /* It is the first flow */
> slot->next = x;
> + q->tail = slot;
> } else {
> slot->next = q->tail->next;
> q->tail->next = x;
> }
> - q->tail = slot;
> slot->allot = q->scaled_quantum;
> }
> if (++sch->q.qlen <= q->limit)
>
I tested this patch with a 50 concurrent netperf workload, and indeed
this fixes the problem for me.
# ping 192.168.20.108
PING 192.168.20.108 (192.168.20.108) 56(84) bytes of data.
64 bytes from 192.168.20.108: icmp_req=1 ttl=64 time=0.021 ms
64 bytes from 192.168.20.108: icmp_req=2 ttl=64 time=0.011 ms
64 bytes from 192.168.20.108: icmp_req=3 ttl=64 time=0.011 ms
64 bytes from 192.168.20.108: icmp_req=4 ttl=64 time=0.010 ms
64 bytes from 192.168.20.108: icmp_req=5 ttl=64 time=0.010 ms
64 bytes from 192.168.20.108: icmp_req=6 ttl=64 time=0.010 ms
# tc -s -d qdisc show dev eth3
qdisc htb 1: root refcnt 18 r2q 10 default 1 direct_packets_stat 0 ver 3.17
Sent 1178661043 bytes 806848 pkt (dropped 9068, overlimits 382834 requeues 3)
rate 97748Kbit 8355pps backlog 0b 122p requeues 3
qdisc sfq 10: parent 1:1 limit 127p quantum 1514b flows 127/1024 divisor 1024
Sent 1178661043 bytes 806848 pkt (dropped 17568, overlimits 0 requeues 0)
rate 97748Kbit 8355pps backlog 962708b 122p requeues 0
# tc -s -d cl show dev eth3
class htb 1:1 root leaf 10: prio 0 quantum 80000 rate 100000Kbit ceil 100000Kbit
burst 40000b/256 mpu 0b overhead 0b cburst 40000b/256 mpu 0b overhead 0b level 0
Sent 1367302331 bytes 935622 pkt (dropped 10494, overlimits 0 requeues 0)
rate 100156Kbit 8560pps backlog 0b 125p requeues 0
lended: 207922 borrowed: 0 giants: 0
tokens: -14605 ctokens: -14605
class sfq 10:15 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 14678b 3p requeues 0
allot -3264
class sfq 10:17 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 20Kb 3p requeues 0
allot -6912
class sfq 10:22 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 17508b 2p requeues 0
allot -808
class sfq 10:3c parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 17508b 2p requeues 0
allot -4208
class sfq 10:5e parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 27Kb 2p requeues 0
allot -6024
class sfq 10:66 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 11716b 2p requeues 0
allot -4856
class sfq 10:8d parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 14612b 2p requeues 0
allot 1296
class sfq 10:9b parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 7504b 4p requeues 0
allot 1184
class sfq 10:9c parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 14612b 2p requeues 0
allot -104
class sfq 10:a9 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 18956b 2p requeues 0
allot -5456
class sfq 10:ab parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 14678b 3p requeues 0
allot -88
class sfq 10:ba parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 4542b 3p requeues 0
allot -624
class sfq 10:c3 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 27710b 3p requeues 0
allot -8440
class sfq 10:ce parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 18956b 2p requeues 0
allot -3160
class sfq 10:f2 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 20Kb 3p requeues 0
allot -672
class sfq 10:11c parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 5990b 3p requeues 0
allot -712
class sfq 10:17c parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 17508b 2p requeues 0
allot -4000
class sfq 10:188 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 16060b 2p requeues 0
allot -6640
class sfq 10:1b9 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 10202b 1p requeues 0
allot 120
class sfq 10:1c2 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 5924b 2p requeues 0
allot 264
class sfq 10:1cb parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 19022b 3p requeues 0
allot -2904
class sfq 10:1e5 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 23366b 3p requeues 0
allot -7800
class sfq 10:1f9 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 8984b 4p requeues 0
allot 560
class sfq 10:210 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 17574b 3p requeues 0
allot -384
class sfq 10:231 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 19022b 3p requeues 0
allot 1416
class sfq 10:27c parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 20404b 2p requeues 0
allot 752
class sfq 10:289 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 19022b 3p requeues 0
allot 1208
class sfq 10:2a3 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 14612b 2p requeues 0
allot -2592
class sfq 10:2a6 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 19022b 3p requeues 0
allot -5008
class sfq 10:2c7 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 7570b 5p requeues 0
allot -1168
class sfq 10:2d0 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 8820b 2p requeues 0
allot -1088
class sfq 10:2e5 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 20404b 2p requeues 0
allot -10648
class sfq 10:2ec parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 13164b 2p requeues 0
allot -1800
class sfq 10:2ee parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 20Kb 3p requeues 0
allot -9016
class sfq 10:305 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 10400b 4p requeues 0
allot -872
class sfq 10:318 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 4542b 3p requeues 0
allot 504
class sfq 10:31f parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 10400b 4p requeues 0
allot -4024
class sfq 10:328 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 16060b 2p requeues 0
allot -16
class sfq 10:32d parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 21918b 3p requeues 0
allot -840
class sfq 10:37d parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 14612b 2p requeues 0
allot 432
class sfq 10:3ac parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 4542b 3p requeues 0
allot 1224
class sfq 10:3d0 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 20Kb 3p requeues 0
allot -11232
class sfq 10:3d1 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 26262b 3p requeues 0
allot -1832
class sfq 10:3d8 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 5924b 2p requeues 0
allot -512
class sfq 10:3e7 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 29158b 3p requeues 0
allot -10976
class sfq 10:3f0 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 32054b 3p requeues 0
allot -2080
class sfq 10:3f9 parent 10:
(dropped 0, overlimits 0 requeues 0)
backlog 18956b 2p requeues 0
allot -1704
next prev parent reply other threads:[~2012-01-02 4:55 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CAA93jw6r45zBn4p76R9-2UNdp=OM+uiYqPKcKnjqBReR_oh1BA@mail.gmail.com>
[not found] ` <CAA93jw5JjVBQOFL=td55rOpXbUYc=uT_rjeY04O137g_-tBJHw@mail.gmail.com>
[not found] ` <CAA93jw7bLnHtVnZGN5wVPQZc46XT_B--cY1ocUbnm1AS=J_YHw@mail.gmail.com>
2012-01-02 4:33 ` BQL + Basic Latency under load results - 100Mbit, GSO/TSO off, pfifo_fast vs SFQ vs QFQ Eric Dumazet
2012-01-02 4:55 ` Eric Dumazet [this message]
2012-01-02 5:07 ` Eric Dumazet
2012-01-02 5:27 ` Eric Dumazet
2012-01-03 17:52 ` David Miller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1325480125.2526.13.camel@edumazet-laptop \
--to=eric.dumazet@gmail.com \
--cc=dave.taht@gmail.com \
--cc=jch@pps.jussieu.fr \
--cc=jg@freedesktop.org \
--cc=netdev@vger.kernel.org \
--cc=nichols@pollere.com \
--cc=shemminger@vyatta.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox