[LARTC] HTB: far unequal behaivor at a slight conf rate change

[LARTC] HTB: far unequal behaivor at a slight conf rate change

[Luciano Ruete]

[-- Attachment #1: Type: text/plain, Size: 1920 bytes --]

Hi all!
I'm working in a " _really_ end user simple", yet powerfull, flexible, 
scalable, GPL htb/htb-init front end, but now, in the "test fase" im facing 
an utlimate problem... 

Absoult minimalistic scenario: #Use a FIXEDFONT to read this
#inet-ethx-|FIREWALL|-lan-ethx <-> |client_host_1|
#
#FIREWALL class(rate,ceil) at lan-ethx
root->parent_all_host(256,256)->client_host_1(X,X)->host_1_prio(X*0.9,X)
                                                  ->host_1_dfl(X*0.1,X)

then i genrate equal traffic for the to leafs and:
 if X<=200 then "all works ok"
 the 'prio' class takes 90% of the rate
 the "dfl" class takes 10% of the rate 

 if X>230 then "all works like no htb present at all"
 the 'prio' and 'dfl' class figth equaly for the rate
 
Things that i have tried that don't help to solve the problem:
 -Play with several r2q/quantum convinations
 -Play with prio parameter for the leafs
 -Tray thiferent leaf's algo(sfq with quantum 1500,pfifo with limit 10)
 -change parent_all_hosts rate to a high value (ie:512,1024)


I've attached simplified ad-hoc scripts that reproduce the scenarios:
tc_at_200 (full tc/iptables commands to recreate the X<200 scenario)
tc_at_230 (full tc/iptables commands to recreate the X>200 scenario)

To test i run in the client_host_1 2 lftp instances ie:
lftp -c "pget -n ftp://somehost/very.long.file" #4 tcp conns "stress" ;-) 
lftp -c "get http://samehost/very.long.file"

note that first generates ftp traffic while second one is http traffic, the 
firewall marks will make that http goes in 'prio' class and ftp goes in 'dfl' 
class. I leave this traffic running...

if i run ./tc_at_200 everithing goes to its place
if i run ./tc_at_230 everithing goes wrong, the 4 ftp's 'dfl' traffic 
connection monopolice at lest 80% of the total rate.
if i run ./tc_at_200 everithing goes to its place again
and so on...

Whell hope someone give me a hint on this!

-- 
Luciano

[-- Attachment #2: tc_at_200 --]
[-- Type: application/x-shellscript, Size: 1398 bytes --]

[-- Attachment #3: tc_at_230 --]
[-- Type: application/x-shellscript, Size: 1398 bytes --]

[-- Attachment #4: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HTB: far unequal behaivor at a slight conf rate change

[Andreas Klauer]

On Thu, Feb 23, 2006 at 07:08:27PM -0300, Luciano Ruete wrote:
> root->parent_all_host(256,256)->client_host_1(X,X)->host_1_prio(X*0.9,X)
>                                                   ->host_1_dfl(X*0.1,X)

What's the purpose of the 256kbit class? In the setup you posted, 
the 200/230kbit child class does not seem to have any siblings. 
Except for the root class, classes without siblings don't make sense. 
At least, I haven't seen any useful purpose for them so far.

> I've attached simplified ad-hoc scripts that reproduce the scenarios:
> tc_at_200 (full tc/iptables commands to recreate the X<200 scenario)
> tc_at_230 (full tc/iptables commands to recreate the X>200 scenario)

I haven't tested them, but they seem to be all right (except for the 
question above). I don't know if it will help at all, but could you 
post tc statistics for both 200 and 230 cases? You can get the statistics 
using 'tc -s -d qdisc/class show dev $iface' or similar command.
Also, did you check wether HTB is complaining about anything in dmesg 
when setting up the 230 class tree? 

Which kernel version and iproute/tc version are you running? Just in case 
you're still suffering from old HTB bugs...

Regards
Andreas Klauer
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HTB: far unequal behaivor at a slight conf rate change

[Luciano Ruete]

[-- Attachment #1: Type: text/plain, Size: 3421 bytes --]

El Thursday 23 February 2006 19:38, Andreas Klauer escribió:
> On Thu, Feb 23, 2006 at 07:08:27PM -0300, Luciano Ruete wrote:
> > root->parent_all_host(256,256)->client_host_1(X,X)->host_1_prio(X*0.9,X)
> >                                                   ->host_1_dfl(X*0.1,X)
>
> What's the purpose of the 256kbit class? In the setup you posted,
> the 200/230kbit child class does not seem to have any siblings.
> Except for the root class, classes without siblings don't make sense.
> At least, I haven't seen any useful purpose for them so far.

That's becouse the _real_ scenario will look like this:

root->parent_all_hosts->client_host_1->prio 
                                     ->dfl
                      ->client_host_2->prio
                                     ->dfl
                      ->client_host_3->prio
                                     ->dfl
                      ->client_host_N->prio
                                     ->dfl

I'will use this parent_all_host class to set the total_real iface rate.
I've posted the reduced example for simplicity and to isolate the real 
problem.

Anyway i've tested a new version of the example attaching the client direct to 
the root class witouth any luck. Script that i use for testing attached. Also 
full tc stadistics, taked from begining and after aprox 3~4 minutes. 
Here some highlights

This is 'dfl' class 
class htb 1:7005 parent 1:7000[...]rate 23000bit ceil 230000bit
 Sent 4521301 bytes 3095 pkt (dropped 0, overlimits 0 requeues 0)
 lended: 467 borrowed: 2628 giants: 0
 tokens: -2193687 ctokens: 8085

This is 'prio' class
class htb 1:7004 parent 1:7000[...]rate 207000bit ceil 230000bit
 Sent 1741028 bytes 1168 pkt (dropped 0, overlimits 0 requeues 0)
 lended: 1168 borrowed: 0 giants: 0
 tokens: 397670 ctokens: 8085


As you see, after 3 minutes the lower rate class has sent 3000 packets vs 1000 
packets from the high rate one. Don't know what to think... 

> > I've attached simplified ad-hoc scripts that reproduce the scenarios:
> > tc_at_200 (full tc/iptables commands to recreate the X<200 scenario)
> > tc_at_230 (full tc/iptables commands to recreate the X>200 scenario)
>
> I haven't tested them, but they seem to be all right (except for the
> question above). I don't know if it will help at all, but could you
> post tc statistics for both 200 and 230 cases? You can get the statistics
> using 'tc -s -d qdisc/class show dev $iface' or similar command.

> Also, did you check wether HTB is complaining about anything in dmesg
> when setting up the 230 class tree?

no complaints

> Which kernel version and iproute/tc version are you running? Just in case
> you're still suffering from old HTB bugs...

My test bed at this moment is a gentoo-kernel-2.6.14 in a x86 gentoo, in the 
server side(iproute2-2.6.11.20050310-r1), and a user-mode-linux-skas3-2.6.14. 
in the client side, with vde_switch daemon listening in a tuntap device. I 
suppose that htb is device independet, i hope it does not matter. 
I have production enviroments waiting for my results (a couple of small local 
ISPs, an a similar WISP), but the problem is that im coding this far from 
home, and beside the servers, i have no access to any of the clients to force 
the rates. 
If you think this enviroment is somewath "buggy" plz tell.

Txs for your response!

-- 
Luciano

[-- Attachment #2: tc_at_230_no_onlychild --]
[-- Type: application/x-shellscript, Size: 1292 bytes --]

[-- Attachment #3: tc_debug_begining --]
[-- Type: text/x-java, Size: 956 bytes --]

class htb 1:7005 parent 1:7000 leaf 7005: prio 3 quantum 1500 rate 23000bit ceil 230000bit burst 12Kb/8 mpu 0b overhead 0b cburst 1714b/8 mpu 0b overhead 0b level 0 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 lended: 0 borrowed: 0 giants: 0
 tokens: 4376664 ctokens: 61083

class htb 1:7004 parent 1:7000 leaf 7004: prio 1 quantum 1500 rate 207000bit ceil 230000bit burst 12Kb/8 mpu 0b overhead 0b cburst 1714b/8 mpu 0b overhead 0b level 0 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 lended: 0 borrowed: 0 giants: 0
 tokens: 486295 ctokens: 61083

class htb 1:7000 root rate 256000bit ceil 256000bit burst 12Kb/8 mpu 0b overhead 0b cburst 1728b/8 mpu 0b overhead 0b level 7 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 lended: 0 borrowed: 0 giants: 0
 tokens: 393216 ctokens: 55296


[-- Attachment #4: tc_debug_final --]
[-- Type: text/x-java, Size: 1008 bytes --]

class htb 1:7005 parent 1:7000 leaf 7005: prio 3 quantum 1500 rate 23000bit ceil 230000bit burst 12Kb/8 mpu 0b overhead 0b cburst 1714b/8 mpu 0b overhead 0b level 0 
 Sent 4521301 bytes 3095 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 173552bit 14pps backlog 0b 0p requeues 0 
 lended: 467 borrowed: 2628 giants: 0
 tokens: -2193687 ctokens: 8085

class htb 1:7004 parent 1:7000 leaf 7004: prio 1 quantum 1500 rate 207000bit ceil 230000bit burst 12Kb/8 mpu 0b overhead 0b cburst 1714b/8 mpu 0b overhead 0b level 0 
 Sent 1741028 bytes 1168 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 48040bit 4pps backlog 0b 0p requeues 0 
 lended: 1168 borrowed: 0 giants: 0
 tokens: 397670 ctokens: 8085

class htb 1:7000 root rate 256000bit ceil 256000bit burst 12Kb/8 mpu 0b overhead 0b cburst 1728b/8 mpu 0b overhead 0b level 7 
 Sent 6262329 bytes 4263 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 222040bit 18pps backlog 0b 0p requeues 0 
 lended: 2628 borrowed: 0 giants: 0
 tokens: 345600 ctokens: 7680


[-- Attachment #5: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HTB: far unequal behaivor at a slight conf rate change

[Andreas Klauer]

On Thu, Feb 23, 2006 at 11:42:16PM -0300, Luciano Ruete wrote:
> That's becouse the _real_ scenario will look like this:
> 
> root->parent_all_hosts->client_host_1->prio 
>                                      ->dfl
[...]
>                       ->client_host_N->prio
>                                      ->dfl

Oh, okay, so you simplified it that way. All right.

> As you see, after 3 minutes the lower rate class has sent 3000 packets vs 1000 
> packets from the high rate one. Don't know what to think... 

There may be a misunderstanding between us, the way you modified your class 
tree now, it seems to have errors. I'll explain below.

> My test bed at this moment is a gentoo

Right. No complaints here. ;-)

> with vde_switch daemon listening in a tuntap device. 
> I suppose that htb is device independet, i hope it does not matter.

I don't have any experience with vde_switch and tuntap's (I don't even know 
what those are, so much for ignorance). The only device-dependent factor I 
came across with HTB so far is the overhead problem - not all devices have 
the same overhead (PPP over Ethernet or whatever). So HTB calculating the 
rate incorrectly is a possibility. It can be tuned using overhead/mpu 
parameters, however in order to do that, you'd need to know correct values 
first, and they can be a little hard to come by. I also doubt it's the 
cause of your problem in this case.
 
> class htb 1:7005 parent 1:7000 leaf 7005: prio 3 quantum 1500 rate 23000bit ceil 230000bit burst 12Kb/8 mpu 0b overhead 0b cburst 1714b/8 mpu 0b overhead 0b level 0 
> class htb 1:7004 parent 1:7000 leaf 7004: prio 1 quantum 1500 rate 207000bit ceil 230000bit burst 12Kb/8 mpu 0b overhead 0b cburst 1714b/8 mpu 0b overhead 0b level 0 
> class htb 1:7000 root rate 256000bit ceil 256000bit burst 12Kb/8 mpu 0b overhead 0b cburst 1728b/8 mpu 0b overhead 0b level 7 

The problem with this tree is that you took out the client class (the one 
with rate and ceil 230000bit). When I said that child classes without 
siblings don't make sense, I didn't mean to actually take out the child 
class, but rather take out the parent of this child class; in your example 
that would mean making the 230000bit class the root class.

In your setup now, by just looking at the class tree, not the statistics, 
my guess would be that while each leaf class has a ceil of 230000bit, 
they won't share the same 230000kbit, but rather utilize the full 256kbit 
of their parent. That does not seem to be what you want. It still does 
not explain the rates in this setup, too.

Especially the rate of the parent class seems low - if this is a testing 
environment where you are filling out the classes to their maximum, it's 
really odd that the parent class does not use it's full bandwidth. On the 
other hand, I don't know how accurate the rate statistics of HTB are.
I don't have access to a properly working shaping setup right now to 
verify wether it's the same on my setup.

If it isn't, I'd probably check first how much rate HTB can actually 
use, because it's a very bad situation for HTB when it thinks it can 
use more bandwidth than the link actually can guarantee.

Regards
Andreas Klauer
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HTB: far unequal behaivor at a slight conf rate change

[Luciano Ruete]

El Friday 24 February 2006 06:36, Andreas Klauer escribió:
> On Thu, Feb 23, 2006 at 11:42:16PM -0300, Luciano Ruete wrote:
> > with vde_switch daemon listening in a tuntap device.
> > I suppose that htb is device independet, i hope it does not matter.
>
> I don't have any experience with vde_switch and tuntap's (I don't even know
> what those are, so much for ignorance). The only device-dependent factor I
> came across with HTB so far is the overhead problem - not all devices have
> the same overhead (PPP over Ethernet or whatever). So HTB calculating the
> rate incorrectly is a possibility. It can be tuned using overhead/mpu
> parameters, however in order to do that, you'd need to know correct values
> first, and they can be a little hard to come by. I also doubt it's the
> cause of your problem in this case.

you're rigth, and will not be the only time in this mail :-) 

> > class htb 1:7005 parent 1:7000 leaf 7005: prio 3 quantum 1500 rate
> > 23000bit ceil 230000bit burst 12Kb/8 mpu 0b overhead 0b cburst 1714b/8
> > mpu 0b overhead 0b level 0 class htb 1:7004 parent 1:7000 leaf 7004: prio
> > 1 quantum 1500 rate 207000bit ceil 230000bit burst 12Kb/8 mpu 0b overhead
> > 0b cburst 1714b/8 mpu 0b overhead 0b level 0 class htb 1:7000 root rate
> > 256000bit ceil 256000bit burst 12Kb/8 mpu 0b overhead 0b cburst 1728b/8
> > mpu 0b overhead 0b level 7
>
> The problem with this tree is that you took out the client class (the one
> with rate and ceil 230000bit). When I said that child classes without
> siblings don't make sense, I didn't mean to actually take out the child
> class, but rather take out the parent of this child class; in your example
> that would mean making the 230000bit class the root class.
>
> In your setup now, by just looking at the class tree, not the statistics,
> my guess would be that while each leaf class has a ceil of 230000bit,
> they won't share the same 230000kbit, but rather utilize the full 256kbit
> of their parent. That does not seem to be what you want. It still does
> not explain the rates in this setup, too.

yes, once again you're rigth, but i consider it irrelevant for the test that i 
was doing. Anyway, as you say, it still does not explain the situation... 

> Especially the rate of the parent class seems low - if this is a testing
> environment where you are filling out the classes to their maximum, it's
> really odd that the parent class does not use it's full bandwidth. On the
> other hand, I don't know how accurate the rate statistics of HTB are.
> I don't have access to a properly working shaping setup right now to
> verify wether it's the same on my setup.
>
> If it isn't, I'd probably check first how much rate HTB can actually
> use, because it's a very bad situation for HTB when it thinks it can
> use more bandwidth than the link actually can guarantee.

For the third time you are so rigth, i boot a second virtual machine, and test 
the same htb setup betwen the two virtual machins and using my gentoo as 
firewall. Obviously the bandwhidth between them can be considered infinite, 
and the setup works properly. So i made severals test to the link whit i was 
testing before and at any moment i get real 256kbit/s. This expains all the 
situation. Shame on me!!! 

I've added a FAQ about this to my project after this.
When i have a first public release(wich will be son), i will post here(if does 
not bother) for one time only the project url, and some words of what it 
does. For now i leave a stable snaphost of my git tree at 
git-clone http://www.lugmen.org.ar/~luciano/git-repo/htb-gen/.git
or simply point a browser to
http://www.lugmen.org.ar/~luciano/git-repo/htb-gen/

Many tanks for your help!

-- 
Luciano
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc