* QoS weirdness : HTB accuracy
@ 2010-03-25 18:06 Julien Vehent
2010-03-25 18:31 ` Marek Kierdelewicz
` (2 more replies)
0 siblings, 3 replies; 17+ messages in thread
From: Julien Vehent @ 2010-03-25 18:06 UTC (permalink / raw)
To: Netdev, netfilter
[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]
Hello folks,
I observe unused bandwidth on my QoS policy that I cannot explain.
Conditions are: I have a HTB tree with 8 classes and a total rate of
768kbits. I use the ATM option so I assume the real rate to be something
close to 675kbits (about 88% of the ethernet rate).
The sum of my 8 rates is exactly 768kbits. Some have ceil values up to
768kbits.
When class 20 "tcp_acks" starts borrowing, TC reduces the total bandwidth
down to 595kbits/S (minus 79kbits/s). And I can't explain why....
The attached graph "tc_htb_weirdness.png" shows the result: there are
'holes' in the sending rate.
I tried to play with burst sizes, r2q value and hysteresis mode, but the
results are the same.
System is debian squeeze, kernel version is 2.6.26, iproute2 version is
2.6.26 - 07/25/2008.
I have attached two files:
- "tcrules.txt" : the traffic control rules
- "tc_htb_weirdness.png" : the rrdtool graph, resolution is 1 second.
And here: http://jve.linuxwall.info/ressources/code/tc_hole_analysis.html
a sheet with some of the measures values. I used it to calculate the size
of one of the hole. The last table (with green and red cells) shows that,
when class 20 "tcp_acks" starts sending at unixtime 1265496813, there is a
lot of bandwidth left over (last column is all green). During the 95
seconds while class 20 is sending, 3880776 bits could be sent but are not.
That's about 40kbits/s on average.
Does anybody observess the same behavior? Any logical explanation to this
or is it a bug ?
Julien
[-- Attachment #2: tcrules.txt --]
[-- Type: text/plain, Size: 2619 bytes --]
#root
tc qdisc add dev eth0 root handle 1: htb default 99 r2q 100
#main branch
tc class add dev eth0 parent 1:0 classid 1:1 htb rate 768kbit ceil 768kbit linklayer atm overhead 5
# class 10 "interactive"
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 128kbit ceil 200kbit burst 5k prio 1 linklayer atm overhead 5
tc qdisc add dev eth0 parent 1:10 handle 110: pfifo limit 1000
tc filter add dev eth0 parent 1:0 protocol ip prio 1 handle 10 fw flowid 1:10
# class 20 "tcp_acks"
tc class add dev eth0 parent 1:1 classid 1:20 htb rate 64kbit ceil 768kbit burst 3k prio 2 linklayer atm overhead 5
tc qdisc add dev eth0 parent 1:20 handle 120: pfifo limit 1000
tc filter add dev eth0 parent 1:0 protocol ip prio 2 handle 20 fw flowid 1:20
# class 30 "ssh"
tc class add dev eth0 parent 1:1 classid 1:30 htb rate 64kbit ceil 300kbit burst 2k prio 3 linklayer atm overhead 5
tc qdisc add dev eth0 parent 1:30 handle 130: sfq perturb 10
tc filter add dev eth0 parent 1:0 protocol ip prio 3 handle 30 fw flowid 1:30
# class 40 "main web"
tc class add dev eth0 parent 1:1 classid 1:40 htb rate 256kbit ceil 768kbit burst 20k prio 4 linklayer atm overhead 5
# class 41 "sub web 1"
tc class add dev eth0 parent 1:40 classid 1:41 htb rate 100kbit ceil 768kbit burst 20k prio 4 linklayer atm overhead 5
tc qdisc add dev eth0 parent 1:41 handle 141: sfq perturb 10
tc filter add dev eth0 parent 1:0 protocol ip prio 4 handle 41 fw flowid 1:41
# class 42 "sub web 2"
tc class add dev eth0 parent 1:40 classid 1:42 htb rate 100kbit ceil 768kbit burst 20k prio 4 linklayer atm overhead 5
tc qdisc add dev eth0 parent 1:42 handle 142: sfq perturb 10
tc filter add dev eth0 parent 1:0 protocol ip prio 5 handle 42 fw flowid 1:42
# class 49 "sub web gen"
tc class add dev eth0 parent 1:40 classid 1:49 htb rate 56kbit ceil 768kbit burst 20k prio 4 linklayer atm overhead 5
tc qdisc add dev eth0 parent 1:49 handle 149: sfq perturb 10
tc filter add dev eth0 parent 1:0 protocol ip prio 6 handle 49 fw flowid 1:49
# class 50 "torrent"
tc class add dev eth0 parent 1:1 classid 1:50 htb rate 128kbit ceil 768kbit burst 2k prio 5 linklayer atm overhead 5
tc qdisc add dev eth0 parent 1:50 handle 150: sfq perturb 10
tc filter add dev eth0 parent 1:0 protocol ip prio 7 handle 50 fw flowid 1:50
# class 99 "default"
tc class add dev eth0 parent 1:1 classid 1:99 htb rate 128kbit ceil 768kbit burst 2k prio 5 linklayer atm overhead 5 mtu 1500
tc qdisc add dev eth0 parent 1:99 handle 199: sfq perturb 10
tc filter add dev eth0 parent 1:0 protocol ip prio 99 handle 99 fw flowid 1:99
[-- Attachment #3: tc_htb_weirdness.png --]
[-- Type: image/png, Size: 29681 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: QoS weirdness : HTB accuracy 2010-03-25 18:06 QoS weirdness : HTB accuracy Julien Vehent @ 2010-03-25 18:31 ` Marek Kierdelewicz 2010-03-25 19:17 ` Julien Vehent [not found] ` <A846BBAEAA674FAA90FA2008DDD1ED12@hakkenden> 2010-05-19 0:07 ` Philip A. Prindeville 2 siblings, 1 reply; 17+ messages in thread From: Marek Kierdelewicz @ 2010-03-25 18:31 UTC (permalink / raw) To: Julien Vehent; +Cc: netfilter > Hello folks, Hi, >I observe unused bandwidth on my QoS policy that I cannot explain. >Conditions are: I have a HTB tree with 8 classes and a total rate of >768kbits. I use the ATM option so I assume the real rate to be >something close to 675kbits (about 88% of the ethernet rate). Try adding p/bfifo or sfq qdisc under the classes. See if it chances things. Don't ask me why. It just solved some problems in my case. W/o qdisc I could sometimes see stalls like yours. Best regards, Marek ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: QoS weirdness : HTB accuracy 2010-03-25 18:31 ` Marek Kierdelewicz @ 2010-03-25 19:17 ` Julien Vehent 2010-03-25 19:36 ` Marek Kierdelewicz 0 siblings, 1 reply; 17+ messages in thread From: Julien Vehent @ 2010-03-25 19:17 UTC (permalink / raw) To: Marek Kierdelewicz; +Cc: netfilter On Thu, 25 Mar 2010 19:31:40 +0100, Marek Kierdelewicz <marek@piasta.pl> wrote: >> Hello folks, > > Hi, > >>I observe unused bandwidth on my QoS policy that I cannot explain. >>Conditions are: I have a HTB tree with 8 classes and a total rate of >>768kbits. I use the ATM option so I assume the real rate to be >>something close to 675kbits (about 88% of the ethernet rate). > > Try adding p/bfifo or sfq qdisc under the classes. See if it chances > things. Don't ask me why. It just solved some problems in my case. W/o > qdisc I could sometimes see stalls like yours. I have pfifo under class 10 and 20 and SFQ under the others. From my point of view, it looks like class 20 is borrowing, so the other classes are not authorized to borrow whatever is left. > > Best regards, > Marek > -- > To unsubscribe from this list: send the line "unsubscribe netfilter" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: QoS weirdness : HTB accuracy 2010-03-25 19:17 ` Julien Vehent @ 2010-03-25 19:36 ` Marek Kierdelewicz 2010-03-25 21:50 ` Julien Vehent 0 siblings, 1 reply; 17+ messages in thread From: Marek Kierdelewicz @ 2010-03-25 19:36 UTC (permalink / raw) To: Julien Vehent, netfilter Hi, From my point of view, it looks like class 20 is borrowing, so the >other classes are not authorized to borrow whatever is left. How do you measure the bandwidth consumed by each fraction of traffic that you visualise on image? Are values gathered from iptables rule counters? Best regards, Marek ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: QoS weirdness : HTB accuracy 2010-03-25 19:36 ` Marek Kierdelewicz @ 2010-03-25 21:50 ` Julien Vehent 0 siblings, 0 replies; 17+ messages in thread From: Julien Vehent @ 2010-03-25 21:50 UTC (permalink / raw) To: Marek Kierdelewicz; +Cc: netfilter On Thu, 25 Mar 2010 20:36:30 +0100, Marek Kierdelewicz <marek@piasta.pl> wrote: > Hi, > >>From my point of view, it looks like class 20 is borrowing, so the >>other classes are not authorized to borrow whatever is left. > > How do you measure the bandwidth consumed by each fraction of traffic > that you visualise on image? Are values gathered from iptables rule > counters? Nope, I parse tc command line output with a perl script, get the 'rate' value for each classe, multiply it by 8 to have it in bits and not bytes, and insert it into a RRD database. script is somewhat similar to this http://jve.linuxwall.info/ressources/code/tcparsestat.pl Julien ^ permalink raw reply [flat|nested] 17+ messages in thread
[parent not found: <A846BBAEAA674FAA90FA2008DDD1ED12@hakkenden>]
* Re: QoS weirdness : HTB accuracy [not found] ` <A846BBAEAA674FAA90FA2008DDD1ED12@hakkenden> @ 2010-03-25 21:49 ` Julien Vehent [not found] ` <F44DB77B8BA04D108C884E966BD65F9E@hakkenden> 0 siblings, 1 reply; 17+ messages in thread From: Julien Vehent @ 2010-03-25 21:49 UTC (permalink / raw) To: Nikolay Rybalov; +Cc: netfilter This seems to be correct, my initial calculation didn't take into account the 8 bytes AAL5 tail (which I didn't know about). So, if this reduction effect is due to the AAL5 tail, how come it only shows on tcp acks packets ? The rest of the time, I have an average rate that's around 680kbits/s, which is the expected rate if I only take into account the 5 bytes overhead from ATM without the 8 bytes overhead from the AAL5 packets. Julien On Thu, 25 Mar 2010 22:39:11 +0300, "Nikolay Rybalov" <nowhere@hakkenden.ath.cx> wrote: > ATM option accounts for ATM overhead. Single TCP ACK is a 64 byte long, > which is 2 ATM cells (5 byte header each)+ 2 AAL5 tails (8 octets each). > So > single ack is has 16 bytes overhead, or 40%. With 200k ack peaks, HTB > accounts for additional 80k of overhead > > -------------------------------------------------- > From: "Julien Vehent" <julien@linuxwall.info> > Sent: Thursday, March 25, 2010 9:06 PM > To: "Netdev" <netdev@vger.kernel.org>; "netfilter" > <netfilter@vger.kernel.org> > Subject: QoS weirdness : HTB accuracy > >> Hello folks, >> >> I observe unused bandwidth on my QoS policy that I cannot explain. >> Conditions are: I have a HTB tree with 8 classes and a total rate of >> 768kbits. I use the ATM option so I assume the real rate to be something >> close to 675kbits (about 88% of the ethernet rate). >> >> The sum of my 8 rates is exactly 768kbits. Some have ceil values up to >> 768kbits. >> >> When class 20 "tcp_acks" starts borrowing, TC reduces the total bandwidth >> down to 595kbits/S (minus 79kbits/s). And I can't explain why.... >> >> The attached graph "tc_htb_weirdness.png" shows the result: there are >> 'holes' in the sending rate. >> >> I tried to play with burst sizes, r2q value and hysteresis mode, but the >> results are the same. >> >> System is debian squeeze, kernel version is 2.6.26, iproute2 version is >> 2.6.26 - 07/25/2008. >> >> I have attached two files: >> - "tcrules.txt" : the traffic control rules >> - "tc_htb_weirdness.png" : the rrdtool graph, resolution is 1 second. >> >> And here: http://jve.linuxwall.info/ressources/code/tc_hole_analysis.html >> a sheet with some of the measures values. I used it to calculate the size >> of one of the hole. The last table (with green and red cells) shows that, >> when class 20 "tcp_acks" starts sending at unixtime 1265496813, there is >> a >> lot of bandwidth left over (last column is all green). During the 95 >> seconds while class 20 is sending, 3880776 bits could be sent but are >> not. >> That's about 40kbits/s on average. >> >> Does anybody observess the same behavior? Any logical explanation to this >> or is it a bug ? >> >> >> Julien ^ permalink raw reply [flat|nested] 17+ messages in thread
[parent not found: <F44DB77B8BA04D108C884E966BD65F9E@hakkenden>]
* Re: QoS weirdness : HTB accuracy [not found] ` <F44DB77B8BA04D108C884E966BD65F9E@hakkenden> @ 2010-03-25 22:15 ` Julien Vehent 0 siblings, 0 replies; 17+ messages in thread From: Julien Vehent @ 2010-03-25 22:15 UTC (permalink / raw) To: Nikolay Rybalov; +Cc: netfilter Well, I'm currently testing this setup on a vmware player installation, locally, on my ethernet network. So there is no real modem involved... yet.. I intend to put this configuration on the server that's connected to the modem as soon as I figure out this little details :) On Fri, 26 Mar 2010 01:11:24 +0300, "Nikolay Rybalov" <nowhere@hakkenden.ath.cx> wrote: > Nah, sorry. I am wrong. ATM opt accounts only for ATM overhead, 5 bytes > per > 40 byte tcp ack. You also add 5 bytes overhead, so total overhead per ack > is > 10 bytes, or 25%. I think this saturates physical ADSL link, which causes > drops on modem's hwsar and affects all outgoing traffic. > PPPoE (I presume) needs 44 bytes of overhead if you ratelimit ppp > interface, and modem uses AAL5/SNAP encap: > 8 bytes SNAP +18 bytes Ethernet + 6 bytes PPPoE + 4 bytes PPP + 8 bytes > AAL5 > > Can you check your modem's HWSAR counters? Are there any drops? > > > -------------------------------------------------- > From: "Julien Vehent" <julien@linuxwall.info> > Sent: Friday, March 26, 2010 12:49 AM > To: "Nikolay Rybalov" <nowhere@hakkenden.ath.cx> > Cc: "netfilter" <netfilter@vger.kernel.org> > Subject: Re: QoS weirdness : HTB accuracy > >> This seems to be correct, my initial calculation didn't take into account >> the 8 bytes AAL5 tail (which I didn't know about). >> >> So, if this reduction effect is due to the AAL5 tail, how come it only >> shows on tcp acks packets ? >> >> The rest of the time, I have an average rate that's around 680kbits/s, >> which is the expected rate if I only take into account the 5 bytes >> overhead >> from ATM without the 8 bytes overhead from the AAL5 packets. >> >> >> Julien >> >> On Thu, 25 Mar 2010 22:39:11 +0300, "Nikolay Rybalov" >> <nowhere@hakkenden.ath.cx> wrote: >>> ATM option accounts for ATM overhead. Single TCP ACK is a 64 byte long, >>> which is 2 ATM cells (5 byte header each)+ 2 AAL5 tails (8 octets each). >>> So >>> single ack is has 16 bytes overhead, or 40%. With 200k ack peaks, HTB >>> accounts for additional 80k of overhead >>> >>> -------------------------------------------------- >>> From: "Julien Vehent" <julien@linuxwall.info> >>> Sent: Thursday, March 25, 2010 9:06 PM >>> To: "Netdev" <netdev@vger.kernel.org>; "netfilter" >>> <netfilter@vger.kernel.org> >>> Subject: QoS weirdness : HTB accuracy >>> >>>> Hello folks, >>>> >>>> I observe unused bandwidth on my QoS policy that I cannot explain. >>>> Conditions are: I have a HTB tree with 8 classes and a total rate of >>>> 768kbits. I use the ATM option so I assume the real rate to be >> something >>>> close to 675kbits (about 88% of the ethernet rate). >>>> >>>> The sum of my 8 rates is exactly 768kbits. Some have ceil values up to >>>> 768kbits. >>>> >>>> When class 20 "tcp_acks" starts borrowing, TC reduces the total >> bandwidth >>>> down to 595kbits/S (minus 79kbits/s). And I can't explain why.... >>>> >>>> The attached graph "tc_htb_weirdness.png" shows the result: there are >>>> 'holes' in the sending rate. >>>> >>>> I tried to play with burst sizes, r2q value and hysteresis mode, but >> the >>>> results are the same. >>>> >>>> System is debian squeeze, kernel version is 2.6.26, iproute2 version is >>>> 2.6.26 - 07/25/2008. >>>> >>>> I have attached two files: >>>> - "tcrules.txt" : the traffic control rules >>>> - "tc_htb_weirdness.png" : the rrdtool graph, resolution is 1 second. >>>> >>>> And here: >> http://jve.linuxwall.info/ressources/code/tc_hole_analysis.html >>>> a sheet with some of the measures values. I used it to calculate the >> size >>>> of one of the hole. The last table (with green and red cells) shows >> that, >>>> when class 20 "tcp_acks" starts sending at unixtime 1265496813, there >> is >>>> a >>>> lot of bandwidth left over (last column is all green). During the 95 >>>> seconds while class 20 is sending, 3880776 bits could be sent but are >>>> not. >>>> That's about 40kbits/s on average. >>>> >>>> Does anybody observess the same behavior? Any logical explanation to >> this >>>> or is it a bug ? >>>> >>>> >>>> Julien >> ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: QoS weirdness : HTB accuracy 2010-03-25 18:06 QoS weirdness : HTB accuracy Julien Vehent 2010-03-25 18:31 ` Marek Kierdelewicz [not found] ` <A846BBAEAA674FAA90FA2008DDD1ED12@hakkenden> @ 2010-05-19 0:07 ` Philip A. Prindeville 2010-05-22 14:29 ` Julien Vehent 2 siblings, 1 reply; 17+ messages in thread From: Philip A. Prindeville @ 2010-05-19 0:07 UTC (permalink / raw) To: Julien Vehent; +Cc: Netdev, netfilter On 03/25/2010 12:06 PM, Julien Vehent wrote: > Hello folks, > > I observe unused bandwidth on my QoS policy that I cannot explain. > Conditions are: I have a HTB tree with 8 classes and a total rate of > 768kbits. I use the ATM option so I assume the real rate to be something > close to 675kbits (about 88% of the ethernet rate). > > The sum of my 8 rates is exactly 768kbits. Some have ceil values up to > 768kbits. > > When class 20 "tcp_acks" starts borrowing, TC reduces the total bandwidth > down to 595kbits/S (minus 79kbits/s). And I can't explain why.... > > The attached graph "tc_htb_weirdness.png" shows the result: there are > 'holes' in the sending rate. > > I tried to play with burst sizes, r2q value and hysteresis mode, but the > results are the same. > > System is debian squeeze, kernel version is 2.6.26, iproute2 version is > 2.6.26 - 07/25/2008. > > I have attached two files: > - "tcrules.txt" : the traffic control rules > - "tc_htb_weirdness.png" : the rrdtool graph, resolution is 1 second. > > And here: http://jve.linuxwall.info/ressources/code/tc_hole_analysis.html > a sheet with some of the measures values. I used it to calculate the size > of one of the hole. The last table (with green and red cells) shows that, > when class 20 "tcp_acks" starts sending at unixtime 1265496813, there is a > lot of bandwidth left over (last column is all green). During the 95 > seconds while class 20 is sending, 3880776 bits could be sent but are not. > That's about 40kbits/s on average. > > Does anybody observess the same behavior? Any logical explanation to this > or is it a bug ? > > > Julien Sorry for the late response: could this be an "aliasing" issue caused by sampling intervals (granularity)? -Philip ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: QoS weirdness : HTB accuracy 2010-05-19 0:07 ` Philip A. Prindeville @ 2010-05-22 14:29 ` Julien Vehent 2010-06-10 21:22 ` Andrew Beverley 2010-07-07 12:50 ` Jesper Dangaard Brouer 0 siblings, 2 replies; 17+ messages in thread From: Julien Vehent @ 2010-05-22 14:29 UTC (permalink / raw) To: Philip A. Prindeville; +Cc: Netdev, netfilter On Tue, 18 May 2010 18:07:12 -0600, "Philip A. Prindeville" <philipp_subx@redfish-solutions.com> wrote: > > Sorry for the late response: could this be an "aliasing" issue caused > by sampling intervals (granularity)? > > -Philip I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at both the branch and leaf levels, so the overhead was computed twice, creating those holes in the bandwidth. Julien ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: QoS weirdness : HTB accuracy 2010-05-22 14:29 ` Julien Vehent @ 2010-06-10 21:22 ` Andrew Beverley 2010-07-04 17:50 ` Andrew Beverley 2010-07-07 12:50 ` Jesper Dangaard Brouer 1 sibling, 1 reply; 17+ messages in thread From: Andrew Beverley @ 2010-06-10 21:22 UTC (permalink / raw) To: Julien Vehent; +Cc: Philip A. Prindeville, Netdev, netfilter > > Sorry for the late response: could this be an "aliasing" issue caused > > by sampling intervals (granularity)? > > > > I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at > both the branch and leaf levels, so the overhead was computed twice, > creating those holes in the bandwidth. I am seeing similar behaviour with my setup. Am I making the same mistake? A subset of my rules is as follows: tc qdisc add dev ppp0 root handle 1: htb r2q 1 tc class add dev ppp0 parent 1: classid 1:1 htb \ rate ${DOWNLINK}kbit ceil ${DOWNLINK}kbit \ overhead $overhead linklayer atm <------- Here tc class add dev ppp0 parent 1:1 classid 1:10 htb \ rate 612kbit ceil 612kbit prio 0 \ overhead $overhead linklayer atm <------- And here tc qdisc add dev ppp0 parent 1:10 handle 4210: \ sfq perturb 10 limit 50 tc filter add dev ppp0 parent 1:0 protocol ip \ prio 10 handle 10 fw flowid 1:10 Thanks, Andy ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: QoS weirdness : HTB accuracy 2010-06-10 21:22 ` Andrew Beverley @ 2010-07-04 17:50 ` Andrew Beverley 2010-07-07 13:07 ` Jesper Dangaard Brouer 0 siblings, 1 reply; 17+ messages in thread From: Andrew Beverley @ 2010-07-04 17:50 UTC (permalink / raw) To: Julien Vehent; +Cc: Philip A. Prindeville, Netdev, netfilter > > I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at > > both the branch and leaf levels, so the overhead was computed twice, > > creating those holes in the bandwidth. > > I am seeing similar behaviour with my setup. Am I making the same > mistake? A subset of my rules is as follows: > > > tc qdisc add dev ppp0 root handle 1: htb r2q 1 > > tc class add dev ppp0 parent 1: classid 1:1 htb \ > rate ${DOWNLINK}kbit ceil ${DOWNLINK}kbit \ > overhead $overhead linklayer atm <------- Here > > tc class add dev ppp0 parent 1:1 classid 1:10 htb \ > rate 612kbit ceil 612kbit prio 0 \ > overhead $overhead linklayer atm <------- And here > > tc qdisc add dev ppp0 parent 1:10 handle 4210: \ > sfq perturb 10 limit 50 > > tc filter add dev ppp0 parent 1:0 protocol ip \ > prio 10 handle 10 fw flowid 1:10 I removed the overhead option on the first leaf, and the speeds change to what I expect. However, the rules above are taken straight from the ADSL Optimizer project, which was the source of the original overhead patch for tc. So is the ADSL Optimizer project wrong? Andy ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: QoS weirdness : HTB accuracy 2010-07-04 17:50 ` Andrew Beverley @ 2010-07-07 13:07 ` Jesper Dangaard Brouer 2010-07-07 15:07 ` Jussi Kivilinna 2010-08-11 14:27 ` Andrew Beverley 0 siblings, 2 replies; 17+ messages in thread From: Jesper Dangaard Brouer @ 2010-07-07 13:07 UTC (permalink / raw) To: Andrew Beverley Cc: Julien Vehent, Philip A. Prindeville, Netdev, netfilter, hawk On Sun, 4 Jul 2010, Andrew Beverley wrote: >>> I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at >>> both the branch and leaf levels, so the overhead was computed twice, >>> creating those holes in the bandwidth. >> >> I am seeing similar behaviour with my setup. Am I making the same >> mistake? A subset of my rules is as follows: >> >> tc qdisc add dev ppp0 root handle 1: htb r2q 1 >> >> tc class add dev ppp0 parent 1: classid 1:1 htb \ >> rate ${DOWNLINK}kbit ceil ${DOWNLINK}kbit \ >> overhead $overhead linklayer atm <------- Here >> >> tc class add dev ppp0 parent 1:1 classid 1:10 htb \ >> rate 612kbit ceil 612kbit prio 0 \ >> overhead $overhead linklayer atm <------- And here >> >> tc qdisc add dev ppp0 parent 1:10 handle 4210: \ >> sfq perturb 10 limit 50 >> >> tc filter add dev ppp0 parent 1:0 protocol ip \ >> prio 10 handle 10 fw flowid 1:10 > > I removed the overhead option on the first leaf, and the speeds change > to what I expect. However, the rules above are taken straight from the > ADSL Optimizer project, which was the source of the original overhead > patch for tc. So is the ADSL Optimizer project wrong? After looking at the HTB kernel code I believe that the ADSL Optimizer project is NOT wrong. You should/must set the linklayer option on both the root class and leaf (else you would be charging the root/parent node too little). It is the expected behavior that small packets cause a significant reduction in the available bandwidth on the ATM link. Small packets will (almost) always cause 2 ATM packets (being send) using 106 bytes, thus eg. sending a 40 bytes TCP ACK packet result in approx 62% overhead. Cheers, Jesper Brouer -- ------------------------------------------------------------------- MSc. Master of Computer Science Dept. of Computer Science, University of Copenhagen Author of http://www.adsl-optimizer.dk ------------------------------------------------------------------- ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: QoS weirdness : HTB accuracy 2010-07-07 13:07 ` Jesper Dangaard Brouer @ 2010-07-07 15:07 ` Jussi Kivilinna 2010-08-11 17:59 ` Andrew Beverley 2010-08-11 14:27 ` Andrew Beverley 1 sibling, 1 reply; 17+ messages in thread From: Jussi Kivilinna @ 2010-07-07 15:07 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: Andrew Beverley, Julien Vehent, Philip A. Prindeville, Netdev, netfilter, hawk Quoting "Jesper Dangaard Brouer" <hawk@diku.dk>: > > On Sun, 4 Jul 2010, Andrew Beverley wrote: > >>>> I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at >>>> both the branch and leaf levels, so the overhead was computed twice, >>>> creating those holes in the bandwidth. >>> >>> I am seeing similar behaviour with my setup. Am I making the same >>> mistake? A subset of my rules is as follows: >>> >>> tc qdisc add dev ppp0 root handle 1: htb r2q 1 >>> >>> tc class add dev ppp0 parent 1: classid 1:1 htb \ >>> rate ${DOWNLINK}kbit ceil ${DOWNLINK}kbit \ >>> overhead $overhead linklayer atm <------- Here >>> >>> tc class add dev ppp0 parent 1:1 classid 1:10 htb \ >>> rate 612kbit ceil 612kbit prio 0 \ >>> overhead $overhead linklayer atm <------- And here >>> >>> tc qdisc add dev ppp0 parent 1:10 handle 4210: \ >>> sfq perturb 10 limit 50 >>> >>> tc filter add dev ppp0 parent 1:0 protocol ip \ >>> prio 10 handle 10 fw flowid 1:10 >> >> I removed the overhead option on the first leaf, and the speeds change >> to what I expect. However, the rules above are taken straight from the >> ADSL Optimizer project, which was the source of the original overhead >> patch for tc. So is the ADSL Optimizer project wrong? > > After looking at the HTB kernel code I believe that the ADSL > Optimizer project is NOT wrong. You should/must set the linklayer > option on both the root class and leaf (else you would be charging > the root/parent node too little). > It's been while I looked at the linklayer/size-table code, but if I remember right overhead is calculated with first linklayer packet sees in qdisc. So when packet goes to leaf with 'linklayer atm', packet get packet size with overhead for ATM and root linklayer is not used for that packet. Because of this you can have leafs with different overheads (pppoe, ipv6-tunnel, etc, with ) and all get right overhead... .. .. ...Uh oh... HTB still has linklayer/overhead of its own, I was talking about the generic linklayer code that can be used with all qdiscs. Never mind, sorry. -Jussi ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: QoS weirdness : HTB accuracy 2010-07-07 15:07 ` Jussi Kivilinna @ 2010-08-11 17:59 ` Andrew Beverley 2010-08-14 17:27 ` Jussi Kivilinna 0 siblings, 1 reply; 17+ messages in thread From: Andrew Beverley @ 2010-08-11 17:59 UTC (permalink / raw) To: Jussi Kivilinna Cc: Jesper Dangaard Brouer, Julien Vehent, Philip A. Prindeville, Netdev, netfilter, hawk On Wed, 2010-07-07 at 18:07 +0300, Jussi Kivilinna wrote: > >>>> I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at > >>>> both the branch and leaf levels, so the overhead was computed twice, > >>>> creating those holes in the bandwidth. > >>> > >>> I am seeing similar behaviour with my setup. Am I making the same > >>> mistake? A subset of my rules is as follows: > >>> > >>> tc qdisc add dev ppp0 root handle 1: htb r2q 1 > >>> > >>> tc class add dev ppp0 parent 1: classid 1:1 htb \ > >>> rate ${DOWNLINK}kbit ceil ${DOWNLINK}kbit \ > >>> overhead $overhead linklayer atm <------- Here > >>> > >>> tc class add dev ppp0 parent 1:1 classid 1:10 htb \ > >>> rate 612kbit ceil 612kbit prio 0 \ > >>> overhead $overhead linklayer atm <------- And here > >>> > >>> tc qdisc add dev ppp0 parent 1:10 handle 4210: \ > >>> sfq perturb 10 limit 50 > >>> > >>> tc filter add dev ppp0 parent 1:0 protocol ip \ > >>> prio 10 handle 10 fw flowid 1:10 > >> > >> I removed the overhead option on the first leaf, and the speeds change > >> to what I expect. However, the rules above are taken straight from the > >> ADSL Optimizer project, which was the source of the original overhead > >> patch for tc. So is the ADSL Optimizer project wrong? > > > > After looking at the HTB kernel code I believe that the ADSL > > Optimizer project is NOT wrong. You should/must set the linklayer > > option on both the root class and leaf (else you would be charging > > the root/parent node too little). > > > > It's been while I looked at the linklayer/size-table code, but if I > remember right overhead is calculated with first linklayer packet sees > in qdisc. So when packet goes to leaf with 'linklayer atm', packet get > packet size with overhead for ATM and root linklayer is not used for > that packet. Because of this you can have leafs with different > overheads (pppoe, ipv6-tunnel, etc, with ) and all get right > overhead... .. > > .. > > ...Uh oh... > > HTB still has linklayer/overhead of its own, I was talking about the > generic linklayer code that can be used with all qdiscs. Never mind, > sorry. Ah, I hadn't come across the tc-stab functionality until now. So that does exactly the same for all qdiscs as the original ATM patch does for HTB? But they work slightly differently as you have alluded to above? Andy ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: QoS weirdness : HTB accuracy 2010-08-11 17:59 ` Andrew Beverley @ 2010-08-14 17:27 ` Jussi Kivilinna 0 siblings, 0 replies; 17+ messages in thread From: Jussi Kivilinna @ 2010-08-14 17:27 UTC (permalink / raw) To: Andrew Beverley Cc: Jesper Dangaard Brouer, Julien Vehent, Philip A. Prindeville, Netdev, netfilter, hawk Quoting "Andrew Beverley" <andy@andybev.com>: > On Wed, 2010-07-07 at 18:07 +0300, Jussi Kivilinna wrote: >> >>>> I was, in fact, an error in my ruleset. I had put the >> 'linklayer atm' at >> >>>> both the branch and leaf levels, so the overhead was computed twice, >> >>>> creating those holes in the bandwidth. >> >>> >> >>> I am seeing similar behaviour with my setup. Am I making the same >> >>> mistake? A subset of my rules is as follows: >> >>> >> >>> tc qdisc add dev ppp0 root handle 1: htb r2q 1 >> >>> >> >>> tc class add dev ppp0 parent 1: classid 1:1 htb \ >> >>> rate ${DOWNLINK}kbit ceil ${DOWNLINK}kbit \ >> >>> overhead $overhead linklayer atm <------- Here >> >>> >> >>> tc class add dev ppp0 parent 1:1 classid 1:10 htb \ >> >>> rate 612kbit ceil 612kbit prio 0 \ >> >>> overhead $overhead linklayer atm <------- And here >> >>> >> >>> tc qdisc add dev ppp0 parent 1:10 handle 4210: \ >> >>> sfq perturb 10 limit 50 >> >>> >> >>> tc filter add dev ppp0 parent 1:0 protocol ip \ >> >>> prio 10 handle 10 fw flowid 1:10 >> >> >> >> I removed the overhead option on the first leaf, and the speeds change >> >> to what I expect. However, the rules above are taken straight from the >> >> ADSL Optimizer project, which was the source of the original overhead >> >> patch for tc. So is the ADSL Optimizer project wrong? >> > >> > After looking at the HTB kernel code I believe that the ADSL >> > Optimizer project is NOT wrong. You should/must set the linklayer >> > option on both the root class and leaf (else you would be charging >> > the root/parent node too little). >> > >> >> It's been while I looked at the linklayer/size-table code, but if I >> remember right overhead is calculated with first linklayer packet sees >> in qdisc. So when packet goes to leaf with 'linklayer atm', packet get >> packet size with overhead for ATM and root linklayer is not used for >> that packet. Because of this you can have leafs with different >> overheads (pppoe, ipv6-tunnel, etc, with ) and all get right >> overhead... .. >> >> .. >> >> ...Uh oh... >> >> HTB still has linklayer/overhead of its own, I was talking about the >> generic linklayer code that can be used with all qdiscs. Never mind, >> sorry. > > Ah, I hadn't come across the tc-stab functionality until now. So that > does exactly the same for all qdiscs as the original ATM patch does for > HTB? But they work slightly differently as you have alluded to above? > Yes, tc-stab works with all qdiscs. Packet size is calculated everytime when qdisc_enqueue() is called and qdisc has size-table set. When none of qdiscs has size-table, packet size ends up being same as skb->len. So if leaf has size-table, that takes preference over earlier parent size-tables. -Jussi ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: QoS weirdness : HTB accuracy 2010-07-07 13:07 ` Jesper Dangaard Brouer 2010-07-07 15:07 ` Jussi Kivilinna @ 2010-08-11 14:27 ` Andrew Beverley 1 sibling, 0 replies; 17+ messages in thread From: Andrew Beverley @ 2010-08-11 14:27 UTC (permalink / raw) To: hawk; +Cc: Julien Vehent, Philip A. Prindeville, Netdev, netfilter On Wed, 2010-07-07 at 15:07 +0200, Jesper Dangaard Brouer wrote: > On Sun, 4 Jul 2010, Andrew Beverley wrote: > > >>> I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at > >>> both the branch and leaf levels, so the overhead was computed twice, > >>> creating those holes in the bandwidth. > >> > >> I am seeing similar behaviour with my setup. Am I making the same > >> mistake? A subset of my rules is as follows: > >> > >> tc qdisc add dev ppp0 root handle 1: htb r2q 1 > >> > >> tc class add dev ppp0 parent 1: classid 1:1 htb \ > >> rate ${DOWNLINK}kbit ceil ${DOWNLINK}kbit \ > >> overhead $overhead linklayer atm <------- Here > >> > >> tc class add dev ppp0 parent 1:1 classid 1:10 htb \ > >> rate 612kbit ceil 612kbit prio 0 \ > >> overhead $overhead linklayer atm <------- And here > >> > >> tc qdisc add dev ppp0 parent 1:10 handle 4210: \ > >> sfq perturb 10 limit 50 > >> > >> tc filter add dev ppp0 parent 1:0 protocol ip \ > >> prio 10 handle 10 fw flowid 1:10 > > > > I removed the overhead option on the first leaf, and the speeds change > > to what I expect. However, the rules above are taken straight from the > > ADSL Optimizer project, which was the source of the original overhead > > patch for tc. So is the ADSL Optimizer project wrong? > > After looking at the HTB kernel code I believe that the ADSL Optimizer > project is NOT wrong. You should/must set the linklayer option on both > the root class and leaf (else you would be charging the root/parent node > too little). > > It is the expected behavior that small packets cause a significant > reduction in the available bandwidth on the ATM link. Small packets will > (almost) always cause 2 ATM packets (being send) using 106 bytes, thus > eg. sending a 40 bytes TCP ACK packet result in approx 62% overhead. [sorry for the late reply] Maybe I am using the linklayer atm option wrong: 1. I am using it with a PCI ADSL modem. As the kernel "knows" that it's dealing with an ATM device, is there any need to even specify the option in the first place? 2. My methodology for calculating the overall speed limit for the device is to download several large files and see what throughput I get. I then use this figure in the rules above. However, as soon as I add a linklayer option, the maximum throughout of the device obviously drops, but means that I never get the throughput that I know the line is capable of. Assuming that I should even be using the linklayer atm option, should I be specifying the line speed higher than that which I physically observe during line speed tests? For example, if I observe an ADSL line speed of 4000kbit/s, I then specify 3600kbit/s for the root of the HTB class, but with the linklayer atm option I only get about 3200kbit/s. Should I be specifying a higher value in order to get the most out of the connection? Andy ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: QoS weirdness : HTB accuracy 2010-05-22 14:29 ` Julien Vehent 2010-06-10 21:22 ` Andrew Beverley @ 2010-07-07 12:50 ` Jesper Dangaard Brouer 1 sibling, 0 replies; 17+ messages in thread From: Jesper Dangaard Brouer @ 2010-07-07 12:50 UTC (permalink / raw) To: netfilter Julien Vehent <julien <at> linuxwall.info> writes: > On Tue, 18 May 2010 18:07:12 -0600, "Philip A. Prindeville" > <philipp_subx <at> redfish-solutions.com> wrote: > > > > Sorry for the late response: could this be an "aliasing" issue caused > > by sampling intervals (granularity)? > > > > -Philip > > I was, in fact, an error in my ruleset. I had put the 'linklayer atm' at > both the branch and leaf levels, so the overhead was computed twice, > creating those holes in the bandwidth. I have checked the HTB kernel code, and my understanding is that, you must set it on both the leaf and the root, for the shaping to be correct. It is the expected behavior that small packets cause a significant reduction in the available bandwidth on the ATM link. See my answer to your original post. -- Med venlig hilsen / Best regards Jesper Brouer ComX Networks A/S Linux Network Kernel Developer Cand. Scient Datalog / MSc.CS Author of http://adsl-optimizer.dk LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2010-08-14 17:27 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-03-25 18:06 QoS weirdness : HTB accuracy Julien Vehent
2010-03-25 18:31 ` Marek Kierdelewicz
2010-03-25 19:17 ` Julien Vehent
2010-03-25 19:36 ` Marek Kierdelewicz
2010-03-25 21:50 ` Julien Vehent
[not found] ` <A846BBAEAA674FAA90FA2008DDD1ED12@hakkenden>
2010-03-25 21:49 ` Julien Vehent
[not found] ` <F44DB77B8BA04D108C884E966BD65F9E@hakkenden>
2010-03-25 22:15 ` Julien Vehent
2010-05-19 0:07 ` Philip A. Prindeville
2010-05-22 14:29 ` Julien Vehent
2010-06-10 21:22 ` Andrew Beverley
2010-07-04 17:50 ` Andrew Beverley
2010-07-07 13:07 ` Jesper Dangaard Brouer
2010-07-07 15:07 ` Jussi Kivilinna
2010-08-11 17:59 ` Andrew Beverley
2010-08-14 17:27 ` Jussi Kivilinna
2010-08-11 14:27 ` Andrew Beverley
2010-07-07 12:50 ` Jesper Dangaard Brouer
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).