Improved performance on up/download even though only sip was prioritized - please explain

Improved performance on up/download even though only sip was prioritized - please explain

[Stefan Bauer]

Hi,

my wan connection is 7000kbit down and 800kbit up.

If i setup a download and upload at the same time, speed drops to 60KB/s down and 70KB/s up. This is an expected behavior without any shaping.

As i only wanted to give my PBX (192.168.0.101) a fixed bandwidth for sip, i came up with the following:

tc qdisc add dev pppoe-wan root       handle 1:    htb default 30
tc class add dev pppoe-wan parent 1:  classid 1:1  htb rate 800kbit
tc class add dev pppoe-wan parent 1:1 classid 1:10 htb rate 120kbit # voip
tc class add dev pppoe-wan parent 1:1 classid 1:30 htb rate 680kbit # default
 
iptables -t mangle -A FORWARD -s 192.168.0.101 -j CLASSIFY --set-class 1:10

It does what i want, reserves 120kbit for traffic from 192.168.0.101. 

However the nice side effect is, that this also boosts my download/uploads and i have no idea why. 

Download is at 650KB/s and upload is around 65KB/s.

Can anyone explain this behavior?

Thank you in advance.

Stefan

root@OpenWrt:~# tc -s class show dev pppoe-wan
class htb 1:10 parent 1:1 prio 0 rate 120000bit ceil 120000bit burst 1599b cburst 1599b 
 Sent 140214 bytes 295 pkt (dropped 486, overlimits 0 requeues 0) 
 rate 7584bit 2pps backlog 0b 0p requeues 0 
 lended: 295 borrowed: 0 giants: 0
 tokens: -583208 ctokens: -583208

class htb 1:1 root rate 800000bit ceil 800000bit burst 1600b cburst 1600b 
 Sent 12418167 bytes 25721 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 535272bit 235pps backlog 0b 0p requeues 0 
 lended: 0 borrowed: 0 giants: 0
 tokens: -26351 ctokens: -26351

class htb 1:30 parent 1:1 prio 0 rate 680000bit ceil 680000bit burst 1599b cburst 1599b 
 Sent 12280937 bytes 25428 pkt (dropped 6548, overlimits 0 requeues 0) 
 rate 527480bit 233pps backlog 0b 2p requeues 0 
 lended: 25426 borrowed: 0 giants: 0
 tokens: -274731 ctokens: -274731

Re: Improved performance on up/download even though only sip was prioritized - please explain

[Martin A. Brown]

[-- Attachment #1: Type: text/plain, Size: 4439 bytes --]


Greetings Stefan,

>my wan connection is 7000kbit down and 800kbit up.
>
>If i setup a download and upload at the same time, speed drops to 
>60KB/s down and 70KB/s up. This is an expected behavior without any 
>shaping.
>
>As i only wanted to give my PBX (192.168.0.101) a fixed bandwidth 
>for sip, i came up with the following:
>
>tc qdisc add dev pppoe-wan root       handle 1:    htb default 30
>tc class add dev pppoe-wan parent 1:  classid 1:1  htb rate 800kbit
>tc class add dev pppoe-wan parent 1:1 classid 1:10 htb rate 120kbit # voip
>tc class add dev pppoe-wan parent 1:1 classid 1:30 htb rate 680kbit # default
> 
>iptables -t mangle -A FORWARD -s 192.168.0.101 -j CLASSIFY --set-class 1:10
>
>It does what i want, reserves 120kbit for traffic from 192.168.0.101. 
>
>However the nice side effect is, that this also boosts my download/uploads and i have no idea why. 
>
>Download is at 650KB/s and upload is around 65KB/s.
>
>Can anyone explain this behavior?

Without more information, I cannot identify the exact explanation, 
however, I can give you a partial answer.

Your HTB script does this:

   tc qdisc add dev pppoe-wan root       handle 1:    htb default 30

Point #1: (observation)
-----------------------
So, you say that you are classifying only traffic from your 
192.168.0.101, however, defining "default 30" on your HTB root qdisc 
means that packets otherwise not classified are going to show up in 
class 1:30.

You can see that this class is handling bytes/packets in your 'tc -s 
class show dev pppoe-wan' output.  Take a look at the dropped 
statistics that you posted in class 1:30.  In addition to limiting 
the dequeue rate, that class has even dropped 6548 packets.  
[Dropping isn't always bad.]

In short, you are shaping all traffic dequeued on this interface.


Point #2:
---------
Since you are shaping all traffic dequeued on this interface, you 
have reduced the number of packets in-flight passing through your 
modem (or lower-layer device).  The most congested link here is very 
probably the ATM (or ADSL? or something?) link between the modem and 
your provider's head end gear (a DSLAM or some other concentrator).

It is this link that is usually saturated.  Before you employed any 
shaping, the packets you were sending were getting queued in great 
big traffic jams (Stau!) in the modem headed up to the head end or 
(buffer bloat) queued in the head end gear to be sent to you.

Every queue has a size (or depth).  If the packet cannot be 
delivered, then it is dropped.  This goes for upload packets as well 
as download packets.  A small amount of packet dropping is routine 
for the Internet.  But, as congestion increases and packet-dropping, 
correspondingly, increases, then communication between endpoints can 
be significantly impaired.  You were experiencing this significant 
impairment when you were saturating your link without any HTB 
shaping.  I suspect that this was an example of buffer bloat (a term 
of art, widely publicized in 2009 and following).

Your success here is employing a shaper on your OpenWrt box.  You 
have effectively made your box the smallest link (setting the 
throughput to 800kbit/680kbit), you have effectively reduced the 
congestion at the upstream device, the modem or provider head end.


To get a more precise or exact explanation, you would need to be 
able to observe the queuing states of the devices immediately 
upstream of your OpenWrt box.

I hope this was a helpful explanation,

-Martin

>root@OpenWrt:~# tc -s class show dev pppoe-wan
>class htb 1:10 parent 1:1 prio 0 rate 120000bit ceil 120000bit burst 1599b cburst 1599b 
> Sent 140214 bytes 295 pkt (dropped 486, overlimits 0 requeues 0) 
> rate 7584bit 2pps backlog 0b 0p requeues 0 
> lended: 295 borrowed: 0 giants: 0
> tokens: -583208 ctokens: -583208
>
>class htb 1:1 root rate 800000bit ceil 800000bit burst 1600b cburst 1600b 
> Sent 12418167 bytes 25721 pkt (dropped 0, overlimits 0 requeues 0) 
> rate 535272bit 235pps backlog 0b 0p requeues 0 
> lended: 0 borrowed: 0 giants: 0
> tokens: -26351 ctokens: -26351
>
>class htb 1:30 parent 1:1 prio 0 rate 680000bit ceil 680000bit burst 1599b cburst 1599b 
> Sent 12280937 bytes 25428 pkt (dropped 6548, overlimits 0 requeues 0) 
> rate 527480bit 233pps backlog 0b 2p requeues 0 
> lended: 25426 borrowed: 0 giants: 0
> tokens: -274731 ctokens: -274731

-- 
Martin A. Brown
http://linux-ip.net/

AW: Improved performance on up/download even though only sip was prioritized - please explain

[Stefan Bauer]

Hi Martin,

thank you for your time and help. You are right, i queued all traffic excect traffic from 192.168.0.101 to class 1:30 but with no special threatment for specific packages.

So because i limited the link throughput, the upstream device was able to better deal with the packages it has to process and that improved the upload/download throughput.

It seems to be better in terms of performance to limit the link on my "site" and avoid congestion at the upstream device. Even though both devices have a queue i still do not see, why it's better to not let the modem do the "job".

Stefan

-----Ursprüngliche Nachricht-----
> Von:Martin A. Brown <martin@linux-ip.net>
> Gesendet: Son 7 Februar 2016 17:33
> An: Stefan Bauer <sb@plzk.de>
> CC: lartc@vger.kernel.org
> Betreff: Re: Improved performance on up/download even though only sip was prioritized - please explain
> 
> 
> Greetings Stefan,
> 
> >my wan connection is 7000kbit down and 800kbit up.
> >
> >If i setup a download and upload at the same time, speed drops to 
> >60KB/s down and 70KB/s up. This is an expected behavior without any 
> >shaping.
> >
> >As i only wanted to give my PBX (192.168.0.101) a fixed bandwidth 
> >for sip, i came up with the following:
> >
> >tc qdisc add dev pppoe-wan root       handle 1:    htb default 30
> >tc class add dev pppoe-wan parent 1:  classid 1:1  htb rate 800kbit
> >tc class add dev pppoe-wan parent 1:1 classid 1:10 htb rate 120kbit # voip
> >tc class add dev pppoe-wan parent 1:1 classid 1:30 htb rate 680kbit # default
> > 
> >iptables -t mangle -A FORWARD -s 192.168.0.101 -j CLASSIFY --set-class 1:10
> >
> >It does what i want, reserves 120kbit for traffic from 192.168.0.101. 
> >
> >However the nice side effect is, that this also boosts my download/uploads and i have no idea why. 
> >
> >Download is at 650KB/s and upload is around 65KB/s.
> >
> >Can anyone explain this behavior?
> 
> Without more information, I cannot identify the exact explanation, 
> however, I can give you a partial answer.
> 
> Your HTB script does this:
> 
>    tc qdisc add dev pppoe-wan root       handle 1:    htb default 30
> 
> Point #1: (observation)
> -----------------------
> So, you say that you are classifying only traffic from your 
> 192.168.0.101, however, defining "default 30" on your HTB root qdisc 
> means that packets otherwise not classified are going to show up in 
> class 1:30.
> 
> You can see that this class is handling bytes/packets in your 'tc -s 
> class show dev pppoe-wan' output.  Take a look at the dropped 
> statistics that you posted in class 1:30.  In addition to limiting 
> the dequeue rate, that class has even dropped 6548 packets.  
> [Dropping isn't always bad.]
> 
> In short, you are shaping all traffic dequeued on this interface.
> 
> 
> Point #2:
> ---------
> Since you are shaping all traffic dequeued on this interface, you 
> have reduced the number of packets in-flight passing through your 
> modem (or lower-layer device).  The most congested link here is very 
> probably the ATM (or ADSL? or something?) link between the modem and 
> your provider's head end gear (a DSLAM or some other concentrator).
> 
> It is this link that is usually saturated.  Before you employed any 
> shaping, the packets you were sending were getting queued in great 
> big traffic jams (Stau!) in the modem headed up to the head end or 
> (buffer bloat) queued in the head end gear to be sent to you.
> 
> Every queue has a size (or depth).  If the packet cannot be 
> delivered, then it is dropped.  This goes for upload packets as well 
> as download packets.  A small amount of packet dropping is routine 
> for the Internet.  But, as congestion increases and packet-dropping, 
> correspondingly, increases, then communication between endpoints can 
> be significantly impaired.  You were experiencing this significant 
> impairment when you were saturating your link without any HTB 
> shaping.  I suspect that this was an example of buffer bloat (a term 
> of art, widely publicized in 2009 and following).
> 
> Your success here is employing a shaper on your OpenWrt box.  You 
> have effectively made your box the smallest link (setting the 
> throughput to 800kbit/680kbit), you have effectively reduced the 
> congestion at the upstream device, the modem or provider head end.
> 
> 
> To get a more precise or exact explanation, you would need to be 
> able to observe the queuing states of the devices immediately 
> upstream of your OpenWrt box.
> 
> I hope this was a helpful explanation,

Re: AW: Improved performance on up/download even though only sip was prioritized - please explain

[Martin A. Brown]

Hello again Stefan,

>thank you for your time and help. You are right, i queued all 
>traffic excect traffic from 192.168.0.101 to class 1:30 but with no 
>special threatment for specific packages.

But, the important factor here is that your OpenWrt device is 
managing the shaping and scheduling.  That goes for both default 
traffic and your VoIP traffic.

>So because i limited the link throughput, the upstream device was 
>able to better deal with the packages it has to process and that 
>improved the upload/download throughput.

Correct.  But, more importantly the packets are no longer piling up 
in congestion on the modem.  So, the modem can pass packets through 
right away and be ready to deal with the next.

>It seems to be better in terms of performance to limit the link on 
>my "site" and avoid congestion at the upstream device.

Absolutely.

>Even though both devices have a queue i still do not see, why it's 
>better to not let the modem do the "job".

The modem software/hardware was created to queue packets a certain 
way and you didn't like how it did the job.

Without knowing your specific modem (it is unimportant, anyway), I 
will assert that the problem with your modem is the problem that has 
been described by Jim Gettys as 'buffer bloat' [0].

The gear and provider configurations that lead to 'buffer bloat' 
result in two problems:

  * deep queues trade throughput for latency (leading you to want to 
    prioritize your VoIP in the first place)

  * deep queues cause strange and unpleasant interactions with 
    packet drops [which packet drops, in turn, cause unpredictable 
    throughput numbers]

You like better how your OpenWrt box is handling the queueing, 
because it is making better tradeoffs between throughput and latency 
(and packet dropping during saturation situations).  All of this 
results in a better experience for your users.

Since you added the HTB qdisc, you have taught your OpenWrt device 
to "slow down" its dequeue rate so that the packet congestion occurs 
on the OpenWrt device, not on the modem.

In short, you have just implemented the bottleneck rule [0].  Your 
box has become the bottleneck (which is a good thing)!

Best regards,

-Martin

 [0] https://gettys.wordpress.com/
     http://www.bufferbloat.net/

 [0] http://tldp.org/HOWTO/Traffic-Control-HOWTO/rules.html

-- 
Martin A. Brown
http://linux-ip.net/

AW: AW: Improved performance on up/download even though only sip was prioritized - please explain

[Stefan Bauer]

Hi Martin,

thank you again. That was very well and clear explained!

Stefan
 
-----Ursprüngliche Nachricht-----
> Von:Martin A. Brown <martin@linux-ip.net>
> Gesendet: Son 7 Februar 2016 19:14
> An: Stefan Bauer <sb@plzk.de>
> CC: lartc@vger.kernel.org
> Betreff: Re: AW: Improved performance on up/download even though only sip was prioritized - please explain
> 
> 
> Hello again Stefan,
> 
> >thank you for your time and help. You are right, i queued all 
> >traffic excect traffic from 192.168.0.101 to class 1:30 but with no 
> >special threatment for specific packages.
> 
> But, the important factor here is that your OpenWrt device is 
> managing the shaping and scheduling.  That goes for both default 
> traffic and your VoIP traffic.
> 
> >So because i limited the link throughput, the upstream device was 
> >able to better deal with the packages it has to process and that 
> >improved the upload/download throughput.
> 
> Correct.  But, more importantly the packets are no longer piling up 
> in congestion on the modem.  So, the modem can pass packets through 
> right away and be ready to deal with the next.
> 
> >It seems to be better in terms of performance to limit the link on 
> >my "site" and avoid congestion at the upstream device.
> 
> Absolutely.
> 
> >Even though both devices have a queue i still do not see, why it's 
> >better to not let the modem do the "job".
> 
> The modem software/hardware was created to queue packets a certain 
> way and you didn't like how it did the job.
> 
> Without knowing your specific modem (it is unimportant, anyway), I 
> will assert that the problem with your modem is the problem that has 
> been described by Jim Gettys as 'buffer bloat' [0].
> 
> The gear and provider configurations that lead to 'buffer bloat' 
> result in two problems:
> 
>   * deep queues trade throughput for latency (leading you to want to 
>     prioritize your VoIP in the first place)
> 
>   * deep queues cause strange and unpleasant interactions with 
>     packet drops [which packet drops, in turn, cause unpredictable 
>     throughput numbers]
> 
> You like better how your OpenWrt box is handling the queueing, 
> because it is making better tradeoffs between throughput and latency 
> (and packet dropping during saturation situations).  All of this 
> results in a better experience for your users.
> 
> Since you added the HTB qdisc, you have taught your OpenWrt device 
> to "slow down" its dequeue rate so that the packet congestion occurs 
> on the OpenWrt device, not on the modem.
> 
> In short, you have just implemented the bottleneck rule [0].  Your 
> box has become the bottleneck (which is a good thing)!
> 
> Best regards,
> 
> -Martin
> 
>  [0] https://gettys.wordpress.com/
>      http://www.bufferbloat.net/
> 
>  [0] http://tldp.org/HOWTO/Traffic-Control-HOWTO/rules.html
> 
> -- 
> Martin A. Brown
> http://linux-ip.net/
> 
> 

Re: Improved performance on up/download even though only sip was prioritized - please explain

[Andy Furniss]

Stefan Bauer wrote:
> Hi,
>
> my wan connection is 7000kbit down and 800kbit up.
>
> If i setup a download and upload at the same time, speed drops to 60KB/s down and 70KB/s up. This is an expected behavior without any shaping.
>
> As i only wanted to give my PBX (192.168.0.101) a fixed bandwidth for sip, i came up with the following:
>
> tc qdisc add dev pppoe-wan root       handle 1:    htb default 30
> tc class add dev pppoe-wan parent 1:  classid 1:1  htb rate 800kbit
> tc class add dev pppoe-wan parent 1:1 classid 1:10 htb rate 120kbit # voip
> tc class add dev pppoe-wan parent 1:1 classid 1:30 htb rate 680kbit # default
>
> iptables -t mangle -A FORWARD -s 192.168.0.101 -j CLASSIFY --set-class 1:10
>
> It does what i want, reserves 120kbit for traffic from 192.168.0.101.
>
> However the nice side effect is, that this also boosts my download/uploads and i have no idea why.
>
> Download is at 650KB/s and upload is around 65KB/s.
>
> Can anyone explain this behavior?

Martin has given good advice, I think there may also be a bit of luck 
involved as well.

Historically at least (I am out of date WRT current behavior)
a ppp connection has a a short qlen and if you don't make a queue for your
htb classes then the default fifos used will inherit this. It used to be 
possible for this to hurt throughput.

This means that had you happened to put the same rules on an eth 
(default qlen 1000)
then you would have seen the same buffer bloat as you so without.

The correct thing to do really is to use something like sfq/fq_codel on
the classes. Or even fifo/bfifo - but you then get to choose the length 
yourself.

One more comment below -

> Thank you in advance.
>
> Stefan
>
> root@OpenWrt:~# tc -s class show dev pppoe-wan
> class htb 1:10 parent 1:1 prio 0 rate 120000bit ceil 120000bit burst 1599b cburst 1599b
>   Sent 140214 bytes 295 pkt (dropped 486, overlimits 0 requeues 0)

You shouldn't really get drops here - it implies either not enough 
bandwidth reserved
or that the default fifo is too short.

In addition with htb it's possible to give more priority to "valuable" 
classes.