QoS weirdness : HTB accuracy

QoS weirdness : HTB accuracy

[Julien Vehent]

[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]

Hello folks,

I observe unused bandwidth on my QoS policy that I cannot explain.
Conditions are: I have a HTB tree with 8 classes and a total rate of
768kbits. I use the ATM option so I assume the real rate to be something
close to 675kbits (about 88% of the ethernet rate).

The sum of my 8 rates is exactly 768kbits. Some have ceil values up to
768kbits.

When class 20 "tcp_acks" starts borrowing, TC reduces the total bandwidth
down to 595kbits/S (minus 79kbits/s). And I can't explain why....

The attached graph "tc_htb_weirdness.png" shows the result: there are
'holes' in the sending rate. 

I tried to play with burst sizes, r2q value and hysteresis mode, but the
results are the same.

System is debian squeeze, kernel version is 2.6.26, iproute2 version is
2.6.26 - 07/25/2008.

I have attached two files:
- "tcrules.txt" : the traffic control rules
- "tc_htb_weirdness.png" : the rrdtool graph, resolution is 1 second.

And here: http://jve.linuxwall.info/ressources/code/tc_hole_analysis.html
a sheet with some of the measures values. I used it to calculate the size
of one of the hole. The last table (with green and red cells) shows that,
when class 20 "tcp_acks" starts sending at unixtime 1265496813, there is a
lot of bandwidth left over (last column is all green). During the 95
seconds while class 20 is sending, 3880776 bits could be sent but are not.
That's about 40kbits/s on average. 

Does anybody observess the same behavior? Any logical explanation to this
or is it a bug ?


Julien

[-- Attachment #2: tcrules.txt --]
[-- Type: text/plain, Size: 2619 bytes --]

#root
tc qdisc add dev eth0 root handle 1: htb default 99 r2q 100

#main branch
tc class add dev eth0 parent 1:0 classid 1:1 htb rate 768kbit ceil 768kbit linklayer atm overhead 5

# class 10 "interactive"
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 128kbit ceil 200kbit burst 5k prio 1 linklayer atm overhead 5
tc qdisc add dev eth0 parent 1:10 handle 110: pfifo limit 1000
tc filter add dev eth0 parent 1:0 protocol ip prio 1 handle 10 fw flowid 1:10

# class 20 "tcp_acks"
tc class add dev eth0 parent 1:1 classid 1:20 htb rate 64kbit ceil 768kbit burst 3k prio 2 linklayer atm overhead 5
tc qdisc add dev eth0 parent 1:20 handle 120: pfifo limit 1000
tc filter add dev eth0 parent 1:0 protocol ip prio 2 handle 20 fw flowid 1:20

# class 30 "ssh"
tc class add dev eth0 parent 1:1 classid 1:30 htb rate 64kbit ceil 300kbit burst 2k prio 3 linklayer atm overhead 5
tc qdisc add dev eth0 parent 1:30 handle 130: sfq perturb 10
tc filter add dev eth0 parent 1:0 protocol ip prio 3 handle 30 fw flowid 1:30

# class 40 "main web"
tc class add dev eth0 parent 1:1 classid 1:40 htb rate 256kbit ceil 768kbit burst 20k prio 4 linklayer atm overhead 5

# class 41 "sub web 1"
tc class add dev eth0 parent 1:40 classid 1:41 htb rate 100kbit ceil 768kbit burst 20k prio 4 linklayer atm overhead 5
tc qdisc add dev eth0 parent 1:41 handle 141: sfq perturb 10
tc filter add dev eth0 parent 1:0 protocol ip prio 4 handle 41 fw flowid 1:41

# class 42 "sub web 2"
tc class add dev eth0 parent 1:40 classid 1:42 htb rate 100kbit ceil 768kbit burst 20k prio 4 linklayer atm overhead 5
tc qdisc add dev eth0 parent 1:42 handle 142: sfq perturb 10
tc filter add dev eth0 parent 1:0 protocol ip prio 5 handle 42 fw flowid 1:42

# class 49 "sub web gen"
tc class add dev eth0 parent 1:40 classid 1:49 htb rate 56kbit ceil 768kbit burst 20k prio 4 linklayer atm overhead 5
tc qdisc add dev eth0 parent 1:49 handle 149: sfq perturb 10
tc filter add dev eth0 parent 1:0 protocol ip prio 6 handle 49 fw flowid 1:49

# class 50 "torrent"
tc class add dev eth0 parent 1:1 classid 1:50 htb rate 128kbit ceil 768kbit burst 2k prio 5 linklayer atm overhead 5
tc qdisc add dev eth0 parent 1:50 handle 150: sfq perturb 10
tc filter add dev eth0 parent 1:0 protocol ip prio 7 handle 50 fw flowid 1:50

# class 99 "default"
tc class add dev eth0 parent 1:1 classid 1:99 htb rate 128kbit ceil 768kbit burst 2k prio 5 linklayer atm overhead 5 mtu 1500
tc qdisc add dev eth0 parent 1:99 handle 199: sfq perturb 10
tc filter add dev eth0 parent 1:0 protocol ip prio 99 handle 99 fw flowid 1:99

[-- Attachment #3: tc_htb_weirdness.png --]
[-- Type: image/png, Size: 29681 bytes --]

Re: QoS weirdness : HTB accuracy

[Philip A. Prindeville]

On 03/25/2010 12:06 PM, Julien Vehent wrote:
> Hello folks,
>
> I observe unused bandwidth on my QoS policy that I cannot explain.
> Conditions are: I have a HTB tree with 8 classes and a total rate of
> 768kbits. I use the ATM option so I assume the real rate to be something
> close to 675kbits (about 88% of the ethernet rate).
>
> The sum of my 8 rates is exactly 768kbits. Some have ceil values up to
> 768kbits.
>
> When class 20 "tcp_acks" starts borrowing, TC reduces the total bandwidth
> down to 595kbits/S (minus 79kbits/s). And I can't explain why....
>
> The attached graph "tc_htb_weirdness.png" shows the result: there are
> 'holes' in the sending rate. 
>
> I tried to play with burst sizes, r2q value and hysteresis mode, but the
> results are the same.
>
> System is debian squeeze, kernel version is 2.6.26, iproute2 version is
> 2.6.26 - 07/25/2008.
>
> I have attached two files:
> - "tcrules.txt" : the traffic control rules
> - "tc_htb_weirdness.png" : the rrdtool graph, resolution is 1 second.
>
> And here: http://jve.linuxwall.info/ressources/code/tc_hole_analysis.html
> a sheet with some of the measures values. I used it to calculate the size
> of one of the hole. The last table (with green and red cells) shows that,
> when class 20 "tcp_acks" starts sending at unixtime 1265496813, there is a
> lot of bandwidth left over (last column is all green). During the 95
> seconds while class 20 is sending, 3880776 bits could be sent but are not.
> That's about 40kbits/s on average. 
>
> Does anybody observess the same behavior? Any logical explanation to this
> or is it a bug ?
>
>
> Julien

Sorry for the late response:  could this be an "aliasing" issue caused
by sampling intervals (granularity)?

-Philip

Re: QoS weirdness : HTB accuracy

[Julien Vehent]

On Tue, 18 May 2010 18:07:12 -0600, "Philip A. Prindeville"
<philipp_subx@redfish-solutions.com> wrote:
> 
> Sorry for the late response:  could this be an "aliasing" issue caused
> by sampling intervals (granularity)?
> 
> -Philip

I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at
both the branch and leaf levels, so the overhead was computed twice,
creating those holes in the bandwidth.

Julien

Re: QoS weirdness : HTB accuracy

[Andrew Beverley]

> > Sorry for the late response:  could this be an "aliasing" issue caused
> > by sampling intervals (granularity)?
> > 
> 
> I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at
> both the branch and leaf levels, so the overhead was computed twice,
> creating those holes in the bandwidth.

I am seeing similar behaviour with my setup. Am I making the same
mistake? A subset of my rules is as follows:


tc qdisc add dev ppp0 root handle 1: htb r2q 1

tc class add dev ppp0 parent 1: classid 1:1 htb \
    rate ${DOWNLINK}kbit ceil ${DOWNLINK}kbit \
    overhead $overhead linklayer atm                   <------- Here

tc class add dev ppp0 parent 1:1 classid 1:10 htb \
    rate 612kbit ceil 612kbit prio 0 \
    overhead $overhead linklayer atm                   <------- And here

tc qdisc add dev ppp0 parent 1:10 handle 4210: \
    sfq perturb 10 limit 50

tc filter add dev ppp0 parent 1:0 protocol ip \
    prio 10 handle 10 fw flowid 1:10



Thanks,

Andy

Re: QoS weirdness : HTB accuracy

[Andrew Beverley]

> > I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at
> > both the branch and leaf levels, so the overhead was computed twice,
> > creating those holes in the bandwidth.
> 
> I am seeing similar behaviour with my setup. Am I making the same
> mistake? A subset of my rules is as follows:
> 
> 
> tc qdisc add dev ppp0 root handle 1: htb r2q 1
> 
> tc class add dev ppp0 parent 1: classid 1:1 htb \
>     rate ${DOWNLINK}kbit ceil ${DOWNLINK}kbit \
>     overhead $overhead linklayer atm                   <------- Here
> 
> tc class add dev ppp0 parent 1:1 classid 1:10 htb \
>     rate 612kbit ceil 612kbit prio 0 \
>     overhead $overhead linklayer atm                   <------- And here
> 
> tc qdisc add dev ppp0 parent 1:10 handle 4210: \
>     sfq perturb 10 limit 50
> 
> tc filter add dev ppp0 parent 1:0 protocol ip \
>     prio 10 handle 10 fw flowid 1:10

I removed the overhead option on the first leaf, and the speeds change
to what I expect. However, the rules above are taken straight from the
ADSL Optimizer project, which was the source of the original overhead
patch for tc. So is the ADSL Optimizer project wrong?

Andy

Re: QoS weirdness : HTB accuracy

[Jesper Dangaard Brouer]

On Sun, 4 Jul 2010, Andrew Beverley wrote:

>>> I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at
>>> both the branch and leaf levels, so the overhead was computed twice,
>>> creating those holes in the bandwidth.
>>
>> I am seeing similar behaviour with my setup. Am I making the same
>> mistake? A subset of my rules is as follows:
>>
>> tc qdisc add dev ppp0 root handle 1: htb r2q 1
>>
>> tc class add dev ppp0 parent 1: classid 1:1 htb \
>>     rate ${DOWNLINK}kbit ceil ${DOWNLINK}kbit \
>>     overhead $overhead linklayer atm                   <------- Here
>>
>> tc class add dev ppp0 parent 1:1 classid 1:10 htb \
>>     rate 612kbit ceil 612kbit prio 0 \
>>     overhead $overhead linklayer atm                   <------- And here
>>
>> tc qdisc add dev ppp0 parent 1:10 handle 4210: \
>>     sfq perturb 10 limit 50
>>
>> tc filter add dev ppp0 parent 1:0 protocol ip \
>>     prio 10 handle 10 fw flowid 1:10
>
> I removed the overhead option on the first leaf, and the speeds change
> to what I expect. However, the rules above are taken straight from the
> ADSL Optimizer project, which was the source of the original overhead
> patch for tc. So is the ADSL Optimizer project wrong?

After looking at the HTB kernel code I believe that the ADSL Optimizer 
project is NOT wrong.  You should/must set the linklayer option on both 
the root class and leaf (else you would be charging the root/parent node 
too little).

It is the expected behavior that small packets cause a significant
reduction in the available bandwidth on the ATM link.  Small packets will 
(almost) always cause 2 ATM packets (being send) using 106 bytes, thus
eg. sending a 40 bytes TCP ACK packet result in approx 62% overhead.

Cheers,
   Jesper Brouer

--
-------------------------------------------------------------------
MSc. Master of Computer Science
Dept. of Computer Science, University of Copenhagen
Author of http://www.adsl-optimizer.dk
-------------------------------------------------------------------

Re: QoS weirdness : HTB accuracy

[Jussi Kivilinna]

Quoting "Jesper Dangaard Brouer" <hawk@diku.dk>:

>
> On Sun, 4 Jul 2010, Andrew Beverley wrote:
>
>>>> I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at
>>>> both the branch and leaf levels, so the overhead was computed twice,
>>>> creating those holes in the bandwidth.
>>>
>>> I am seeing similar behaviour with my setup. Am I making the same
>>> mistake? A subset of my rules is as follows:
>>>
>>> tc qdisc add dev ppp0 root handle 1: htb r2q 1
>>>
>>> tc class add dev ppp0 parent 1: classid 1:1 htb \
>>>    rate ${DOWNLINK}kbit ceil ${DOWNLINK}kbit \
>>>    overhead $overhead linklayer atm                   <------- Here
>>>
>>> tc class add dev ppp0 parent 1:1 classid 1:10 htb \
>>>    rate 612kbit ceil 612kbit prio 0 \
>>>    overhead $overhead linklayer atm                   <------- And here
>>>
>>> tc qdisc add dev ppp0 parent 1:10 handle 4210: \
>>>    sfq perturb 10 limit 50
>>>
>>> tc filter add dev ppp0 parent 1:0 protocol ip \
>>>    prio 10 handle 10 fw flowid 1:10
>>
>> I removed the overhead option on the first leaf, and the speeds change
>> to what I expect. However, the rules above are taken straight from the
>> ADSL Optimizer project, which was the source of the original overhead
>> patch for tc. So is the ADSL Optimizer project wrong?
>
> After looking at the HTB kernel code I believe that the ADSL  
> Optimizer project is NOT wrong.  You should/must set the linklayer  
> option on both the root class and leaf (else you would be charging  
> the root/parent node too little).
>

It's been while I looked at the linklayer/size-table code, but if I  
remember right overhead is calculated with first linklayer packet sees  
in qdisc. So when packet goes to leaf with 'linklayer atm', packet get  
packet size with overhead for ATM and root linklayer is not used for  
that packet. Because of this you can have leafs with different  
overheads (pppoe, ipv6-tunnel, etc, with ) and all get right  
overhead... ..

..

...Uh oh...

HTB still has linklayer/overhead of its own, I was talking about the  
generic linklayer code that can be used with all qdiscs. Never mind,  
sorry.

-Jussi

Re: QoS weirdness : HTB accuracy

[Andrew Beverley]

On Wed, 2010-07-07 at 18:07 +0300, Jussi Kivilinna wrote:
> >>>> I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at
> >>>> both the branch and leaf levels, so the overhead was computed twice,
> >>>> creating those holes in the bandwidth.
> >>>
> >>> I am seeing similar behaviour with my setup. Am I making the same
> >>> mistake? A subset of my rules is as follows:
> >>>
> >>> tc qdisc add dev ppp0 root handle 1: htb r2q 1
> >>>
> >>> tc class add dev ppp0 parent 1: classid 1:1 htb \
> >>>    rate ${DOWNLINK}kbit ceil ${DOWNLINK}kbit \
> >>>    overhead $overhead linklayer atm                   <------- Here
> >>>
> >>> tc class add dev ppp0 parent 1:1 classid 1:10 htb \
> >>>    rate 612kbit ceil 612kbit prio 0 \
> >>>    overhead $overhead linklayer atm                   <------- And here
> >>>
> >>> tc qdisc add dev ppp0 parent 1:10 handle 4210: \
> >>>    sfq perturb 10 limit 50
> >>>
> >>> tc filter add dev ppp0 parent 1:0 protocol ip \
> >>>    prio 10 handle 10 fw flowid 1:10
> >>
> >> I removed the overhead option on the first leaf, and the speeds change
> >> to what I expect. However, the rules above are taken straight from the
> >> ADSL Optimizer project, which was the source of the original overhead
> >> patch for tc. So is the ADSL Optimizer project wrong?
> >
> > After looking at the HTB kernel code I believe that the ADSL  
> > Optimizer project is NOT wrong.  You should/must set the linklayer  
> > option on both the root class and leaf (else you would be charging  
> > the root/parent node too little).
> >
> 
> It's been while I looked at the linklayer/size-table code, but if I  
> remember right overhead is calculated with first linklayer packet sees  
> in qdisc. So when packet goes to leaf with 'linklayer atm', packet get  
> packet size with overhead for ATM and root linklayer is not used for  
> that packet. Because of this you can have leafs with different  
> overheads (pppoe, ipv6-tunnel, etc, with ) and all get right  
> overhead... ..
> 
> ..
> 
> ...Uh oh...
> 
> HTB still has linklayer/overhead of its own, I was talking about the  
> generic linklayer code that can be used with all qdiscs. Never mind,  
> sorry.

Ah, I hadn't come across the tc-stab functionality until now. So that
does exactly the same for all qdiscs as the original ATM patch does for
HTB? But they work slightly differently as you have alluded to above?

Andy

Re: QoS weirdness : HTB accuracy

[Jussi Kivilinna]

Quoting "Andrew Beverley" <andy@andybev.com>:

> On Wed, 2010-07-07 at 18:07 +0300, Jussi Kivilinna wrote:
>> >>>> I was, in fact, an error in my ruleset. I had put the  
>> 'linklayer atm' at
>> >>>> both the branch and leaf levels, so the overhead was computed twice,
>> >>>> creating those holes in the bandwidth.
>> >>>
>> >>> I am seeing similar behaviour with my setup. Am I making the same
>> >>> mistake? A subset of my rules is as follows:
>> >>>
>> >>> tc qdisc add dev ppp0 root handle 1: htb r2q 1
>> >>>
>> >>> tc class add dev ppp0 parent 1: classid 1:1 htb \
>> >>>    rate ${DOWNLINK}kbit ceil ${DOWNLINK}kbit \
>> >>>    overhead $overhead linklayer atm                   <------- Here
>> >>>
>> >>> tc class add dev ppp0 parent 1:1 classid 1:10 htb \
>> >>>    rate 612kbit ceil 612kbit prio 0 \
>> >>>    overhead $overhead linklayer atm                   <------- And here
>> >>>
>> >>> tc qdisc add dev ppp0 parent 1:10 handle 4210: \
>> >>>    sfq perturb 10 limit 50
>> >>>
>> >>> tc filter add dev ppp0 parent 1:0 protocol ip \
>> >>>    prio 10 handle 10 fw flowid 1:10
>> >>
>> >> I removed the overhead option on the first leaf, and the speeds change
>> >> to what I expect. However, the rules above are taken straight from the
>> >> ADSL Optimizer project, which was the source of the original overhead
>> >> patch for tc. So is the ADSL Optimizer project wrong?
>> >
>> > After looking at the HTB kernel code I believe that the ADSL
>> > Optimizer project is NOT wrong.  You should/must set the linklayer
>> > option on both the root class and leaf (else you would be charging
>> > the root/parent node too little).
>> >
>>
>> It's been while I looked at the linklayer/size-table code, but if I
>> remember right overhead is calculated with first linklayer packet sees
>> in qdisc. So when packet goes to leaf with 'linklayer atm', packet get
>> packet size with overhead for ATM and root linklayer is not used for
>> that packet. Because of this you can have leafs with different
>> overheads (pppoe, ipv6-tunnel, etc, with ) and all get right
>> overhead... ..
>>
>> ..
>>
>> ...Uh oh...
>>
>> HTB still has linklayer/overhead of its own, I was talking about the
>> generic linklayer code that can be used with all qdiscs. Never mind,
>> sorry.
>
> Ah, I hadn't come across the tc-stab functionality until now. So that
> does exactly the same for all qdiscs as the original ATM patch does for
> HTB? But they work slightly differently as you have alluded to above?
>

Yes, tc-stab works with all qdiscs. Packet size is calculated  
everytime when qdisc_enqueue() is called and qdisc has size-table set.  
When none of qdiscs has size-table, packet size ends up being same as  
skb->len. So if leaf has size-table, that takes preference over  
earlier parent size-tables.

-Jussi

Re: QoS weirdness : HTB accuracy

[Andrew Beverley]

On Wed, 2010-07-07 at 15:07 +0200, Jesper Dangaard Brouer wrote:
> On Sun, 4 Jul 2010, Andrew Beverley wrote:
> 
> >>> I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at
> >>> both the branch and leaf levels, so the overhead was computed twice,
> >>> creating those holes in the bandwidth.
> >>
> >> I am seeing similar behaviour with my setup. Am I making the same
> >> mistake? A subset of my rules is as follows:
> >>
> >> tc qdisc add dev ppp0 root handle 1: htb r2q 1
> >>
> >> tc class add dev ppp0 parent 1: classid 1:1 htb \
> >>     rate ${DOWNLINK}kbit ceil ${DOWNLINK}kbit \
> >>     overhead $overhead linklayer atm                   <------- Here
> >>
> >> tc class add dev ppp0 parent 1:1 classid 1:10 htb \
> >>     rate 612kbit ceil 612kbit prio 0 \
> >>     overhead $overhead linklayer atm                   <------- And here
> >>
> >> tc qdisc add dev ppp0 parent 1:10 handle 4210: \
> >>     sfq perturb 10 limit 50
> >>
> >> tc filter add dev ppp0 parent 1:0 protocol ip \
> >>     prio 10 handle 10 fw flowid 1:10
> >
> > I removed the overhead option on the first leaf, and the speeds change
> > to what I expect. However, the rules above are taken straight from the
> > ADSL Optimizer project, which was the source of the original overhead
> > patch for tc. So is the ADSL Optimizer project wrong?
> 
> After looking at the HTB kernel code I believe that the ADSL Optimizer 
> project is NOT wrong.  You should/must set the linklayer option on both 
> the root class and leaf (else you would be charging the root/parent node 
> too little).
> 
> It is the expected behavior that small packets cause a significant
> reduction in the available bandwidth on the ATM link.  Small packets will 
> (almost) always cause 2 ATM packets (being send) using 106 bytes, thus
> eg. sending a 40 bytes TCP ACK packet result in approx 62% overhead.

[sorry for the late reply]

Maybe I am using the linklayer atm option wrong:

1. I am using it with a PCI ADSL modem. As the kernel "knows" that it's
dealing with an ATM device, is there any need to even specify the option
in the first place?

2. My methodology for calculating the overall speed limit for the device
is to download several large files and see what throughput I get. I then
use this figure in the rules above. However, as soon as I add a
linklayer option, the maximum throughout of the device obviously drops,
but means that I never get the throughput that I know the line is
capable of. Assuming that I should even be using the linklayer atm
option, should I be specifying the line speed higher than that which I
physically observe during line speed tests?

For example, if I observe an ADSL line speed of 4000kbit/s, I then
specify 3600kbit/s for the root of the HTB class, but with the linklayer
atm option I only get about 3200kbit/s. Should I be specifying a higher
value in order to get the most out of the connection?

Andy

Re: QoS weirdness : HTB accuracy

[Jesper Dangaard Brouer]

Julien Vehent <julien <at> linuxwall.info> writes:

> I observe unused bandwidth on my QoS policy that I cannot explain.
> Conditions are: I have a HTB tree with 8 classes and a total rate of
> 768kbits. I use the ATM option so I assume the real rate to be something
> close to 675kbits (about 88% of the ethernet rate).
> 
> The sum of my 8 rates is exactly 768kbits. Some have ceil values up to
> 768kbits.
> 
> When class 20 "tcp_acks" starts borrowing, TC reduces the total bandwidth
> down to 595kbits/S (minus 79kbits/s). And I can't explain why....

Fortunately, I can explain why.  This is actually the expected/correct
behavior.  The "tcp_acks" class only receives small packets, and small
packets have a significantly higher overhead than larger packets.

The simple explanation is that small packets will (almost) always
result in two ATM frames being transmittet, thus resulting in 106
bytes (2x 53 bytes) being used on the link.

 
> The attached graph "tc_htb_weirdness.png" shows the result: there are
> 'holes' in the sending rate. 

This is as explained above the behavior I expected to see.  As the
small packets eats/consumes more bandwidth on the physical link, and
your observations are based upon what happens on the Ethernet layer.

-- 
Best regards
  Jesper Brouer
  ComX Networks A/S
  Linux Network Kernel Developer
  Cand. Scient Datalog / MSc.CS
  Author of http://adsl-optimizer.dk
  LinkedIn: http://www.linkedin.com/in/brouer