[LARTC] HTB, strange capacity distribution

[LARTC] HTB, strange capacity distribution

[Boris Gereg]

Hello,

after spending several hours reading archives, I decided to write new
post.

I successfully set up packet classification, made some basic HTB
setup, made some simple graphical representation from HTB statistics
data...

BUT, I cannot figure out how to refine HTB to get this behaviour:
I need that class "p2p" should be the last one to get some link
capacity.

If I set both RATE and CEIL for "p2p" to some constant value,
everything is OK. When I set CEIL for "p2p" e.g. to 2048 kbps (total
link capacity), "p2p" starts to get most of the link capacity and does
not want to give capacity to other classes with higher priorities!


I made a screen to help explain my problem. Please, see this picture:
http://elusion.sk/visual_inet_hory.png

Point A - "p2p" class (red line) has both RATE and CEIL set to fixed
value. I started download from web to fill "www" class. You can see,
"www" got remaining link capacity, "p2p" without a change - all as
expected.

Point B - I reset CEIL of "p2p" class to 2048 kbit (to use rest of the
unused capacity by other classes with higher priorites, up to link
capacity)

Point C - I started the same download from web as in Point A and this
is the PROBLEM. You can see, that "p2p" lowered a bit, and "www" class
got some capacity. What I expect is, that "www" class get most of the
link capacity and "p2p" class will fall to it's RATE, because "http"
class has higher priority than "p2p"

Point D - CEIL of "p2p" reset to fixed value

Point E - I tried the same download from web just to be sure it works.


This is my HTB config (using latest htb-init script):

root:
RATE\x100Mbit

( local traffic:
RATEPMbit
CEIL\x100Mbit
PRIO=0 )

"ctrl" class:
RATEdkbit
CEIL%6kbit
PRIO=1

"other" class:
RATE\x128kbit
CEILv8kbit
PRIO=2

"www" class:
RATEQ2kbit
CEIL 48kbit
PRIO=3

"p2p" class:
RATE86kbit
CEIL86kbit (or 2048, for Points B and C in figure above)
PRIO=4

All want to set up is: If class "www" with prio 3 gets fully utilized
(I start 10 http downloads at the same time), I expect "p2p" class
with prio 4 to fall down to it's RATE 386 kbit - to free capacity for
"www" class

In another words: I expect "www" to overrule "p2p" ("p2p" still
getting it's RATE)


Can someone help to explain this? Maybe I did not get purpose of PRIO
setting, not sure. Help, please


Best Regards,
B. Gereg


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HTB, strange capacity distribution

[Andreas Klauer]

On Mon, Feb 20, 2006 at 10:59:33PM +0100, Boris Gereg wrote:
> I made a screen to help explain my problem. Please, see this picture:
> http://elusion.sk/visual_inet_hory.png

Nice graph. I assume this is on downstream, and you rely on HTB to drop 
packets for you. You may have read this in the archives already - it's 
much harder to shape downstream than upstream, because you can't really 
influence what the other side is sending you. So no matter what you do 
it's probably hard to get near-optimal results.

> This is my HTB config (using latest htb-init script):

I must admit I'm not familiar with htb-init. What are the parent-child 
relationships here? I'm missing the "internet" parent class that groups 
all the other traffics (except local) together. Does htb-init generate 
that on it's own somehow?

If not, chances are your HTB tree is just exceeding your line capacity 
in general, as all classes are allowed to borrow without limit, rendering 
the prio setting uneffective, leading to random results.

Could you post the output of 'tc -d qdisc/class show dev $DEVICE'?

Regards,
Andreas Klauer
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HTB, strange capacity distribution

[Andreas Klauer]

On Tue, Feb 21, 2006 at 12:49:59AM +0100, Boris Gereg wrote:
> (first of all, please, how to reply to some article in LARTC via mail
> to post it into right thread?)

Using 'reply all', or 'reply list' if your mail software offers it. 
If all else fails, just hit 'reply' and add the mailing list to CC.

> So, I am definitely shaping outgoing traffic (upstream)

Yes, outgoing traffic from router to your network, which actually 
contains the downstream traffic from the internet. Right?

> tc -d class show dev eth0
> 
> class htb 1:2 root rate 100000Kbit ceil 100000Kbit burst 51587b/8 mpu
> 0b overhead 0b cburst 51587b/8 mpu 0b overhead 0b level 7
> 
> class htb 1:2000 parent 1:2 leaf 2000: prio 0 quantum 200000 rate
> 50000Kbit ceil 100000Kbit burst 26593b/8 mpu 0b overhead 0b cburst
> 51587b/8 mpu 0b overhead 0b level 0
> 
> class htb 1:3010 parent 1:2 leaf 3010: prio 1 quantum 1000 rate
> 64000bit ceil 256000bit burst 1631b/8 mpu 0b overhead 0b cburst
> 1727b/8 mpu 0b overhead 0b level 0
> 
> class htb 1:3020 parent 1:2 leaf 3020: prio 2 quantum 1600 rate
> 128000bit ceil 768000bit burst 1663b/8 mpu 0b overhead 0b cburst
> 1983b/8 mpu 0b overhead 0b level 0
> 
> class htb 1:3030 parent 1:2 leaf 3030: prio 3 quantum 6400 rate
> 512000bit ceil 2048Kbit burst 1855b/8 mpu 0b overhead 0b cburst
> 2623b/8 mpu 0b overhead 0b level 0
> 
> class htb 1:5040 parent 1:2 leaf 5040: prio 4 quantum 4825 rate
> 386000bit ceil 386000bit burst 1792b/8 mpu 0b overhead 0b cburst
> 1792b/8 mpu 0b overhead 0b level 0

It's as I suspected, your current HTB tree looks like this:

1: HTB Qdisc
|
\--- 1:2 HTB root class (100000Kbit:100000kbit)
     |
     \--- 1:2000 HTB leaf class (50000Kbit:100000Kbit)
     \--- 1:3010 HTB leaf class (64000bit:256000bit)
     \--- 1:3020 HTB leaf class (128000bit:768000bit)
     \--- 1:3030 HTB leaf class (512000bit:2048Kbit)
     \--- 1:5040 HTB leaf class (386000bit:386000bit)

HTB classes borrow from their parent; in this setup, the parent class 
offers a whopping 100000Kbit for that purpose. Unless the 1:2000 class 
has got a very high priority and is maxing out the line all the time, 
there is no limit to borrowing at all, because the other classes will 
never reach the 100000Kbit from their parent.

So the classes above are actually not limited by their rate, but by 
their ceil; the only class that will respect its rate in this setup 
is 1:5040, because it's got the same rate and ceil.

Assuming that 1:5040 was your P2P class, if you set the ceil of this 
class to 2048Kbit, it will (try to) use 2048Kbit at all times, because 
the parent (thinks it) is able to offer it.

You need a class that knows of your total internet bandwidth somewhere. 
Assuming that it is 2048Kbit, your tree should maybe look more like this:

1: HTB Qdisc
|
\--- 1:2 HTB root class (100000Kbit:100000kbit)
     |
     \--- 1:2000 HTB leaf class (50000Kbit:100000Kbit)
     |
     \--- 1:3000 HTB parent class (2048Kbit:2048Kbit)
          |
          \--- 1:3010 HTB leaf class (64000bit:256000bit)
          \--- 1:3020 HTB leaf class (128000bit:768000bit)
          \--- 1:3030 HTB leaf class (512000bit:2048Kbit)
          \--- 1:5040 HTB leaf class (386000bit:386000bit)

In this setup, the 2048Kbit class is the limiting factor for the leaf 
classes, except for the 1:2000 class, which should be used for local 
LAN traffic only.

HTH
Andreas Klauer
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HTB, strange capacity distribution

[Boris Gereg]

Hello,

thanks Andreas, I reconfigured HTB to get your suggested hierarhy:

AK> 1: HTB Qdisc
AK> |
AK> \--- 1:2 HTB root class (100000Kbit:100000kbit)
AK>      |
AK>      \--- 1:2000 HTB leaf class (50000Kbit:100000Kbit)     #local
AK>      |
AK>      \--- 1:3000 HTB parent class (2048Kbit:2048Kbit)
AK>           |
AK>           \--- 1:3010 HTB leaf class (64000bit:256000bit)  #ctrl
AK>           \--- 1:3020 HTB leaf class (128000bit:768000bit) #other
AK>           \--- 1:3030 HTB leaf class (512000bit:2048Kbit)  #www
AK>           \--- 1:5040 HTB leaf class (386000bit:386000bit) #p2p


tc -d class show dev eth0

class htb 1:2 root rate 100000Kbit ceil 100000Kbit burst 51587b/8 mpu
0b overhead 0b cburst 51587b/8 mpu 0b overhead 0b level 7

# local:
class htb 1:2000 parent 1:2 leaf 2000: prio 0 quantum 200000 rate
50000Kbit ceil 100000Kbit burst 26593b/8 mpu 0b overhead 0b cburst
51587b/8 mpu 0b overhead 0b level 0

# root for internet trafic
class htb 1:3000 parent 1:2 rate 2048Kbit ceil 2048Kbit burst 2623b/8
mpu 0b overhead 0b cburst 2623b/8 mpu 0b overhead 0b level 6

# "ctrl" class
class htb 1:3010 parent 1:3000 leaf 3010: prio 1 quantum 1000 rate
64000bit ceil 256000bit burst 1631b/8 mpu 0b overhead 0b cburst
1727b/8 mpu 0b overhead 0b level 0

# "other" class
class htb 1:3020 parent 1:3000 leaf 3020: prio 2 quantum 1600 rate
128000bit ceil 768000bit burst 1663b/8 mpu 0b overhead 0b cburst
1983b/8 mpu 0b overhead 0b level 0

# "www" class
class htb 1:3030 parent 1:3000 leaf 3030: prio 3 quantum 12800 rate
1024Kbit ceil 2048Kbit burst 2111b/8 mpu 0b overhead 0b cburst 2623b/8
mpu 0b overhead 0b level 0

# "p2p" class
class htb 1:5040 parent 1:3000 leaf 5040: prio 4 quantum 1600 rate
128000bit ceil 256000bit burst 1663b/8 mpu 0b overhead 0b cburst
1727b/8 mpu 0b overhead 0b level 0


Sorry to say, results are not as expected. I made new measurements,
please see following picture (based on HTB config above):
http://elusion.sk/visual_inet_6.png

Point A - until now, "p2" clas had CEIL 386 kbit. Now, I reset "p2p"
class CEIL to 2048 kbit. As you can see, "p2p" rised as expected

Point B - "www" class RATE-CEIL is 512-2048 kbit, "p2p" class
RATE-CEIL is 128-2048 kbit. PROBLEM: why did not "www" get more
capacity?

Point C - "www" class RATE-CEIL is 1536-2048 kbit, "p2p" class
RATE-CEIL is 128-2048 kbit. PROBLEM: RATE of "www" is 1536, but class
did not get this capacity; "p2p" should fall to it's RATE 256 kbit

Point D - "www" class RATE-CEIL is 1024-2048 kbit, "p2p" class
RATE-CEIL is 128-2048 kbit - no change

Point E - "www" class RATE-CEIL is 1024-2048 kbit, "p2p" class
RATE-CEIL is 128-256 kbit - as expected: "p2p" is bounded to 256 kbit
limit, so the rest of the capacity was used by "www" class


I have expected that in points B, C and D, "p2p" class falls down
to it's specified rate and "www" gets most of the capacity. But this
is not happening - "p2p" will not fall down


Any other things to test, please?

Best regards,
B. Gereg

mailto:qwerty@elusion.sk


-----Originalna sprava-----
Od: Andreas Klauer [mailto:Andreas.Klauer@metamorpher.de]
Poslane: Tuesday, February 21, 2006, 8:52:49 AM
Komu: qwerty@elusion.sk
Predmet: [LARTC] HTB, strange capacity distribution

AK> On Tue, Feb 21, 2006 at 12:49:59AM +0100, Boris Gereg wrote:
>> (first of all, please, how to reply to some article in LARTC via mail
>> to post it into right thread?)

AK> Using 'reply all', or 'reply list' if your mail software offers it.
AK> If all else fails, just hit 'reply' and add the mailing list to CC.

>> So, I am definitely shaping outgoing traffic (upstream)

AK> Yes, outgoing traffic from router to your network, which actually 
AK> contains the downstream traffic from the internet. Right?

>> tc -d class show dev eth0
>> 
>> class htb 1:2 root rate 100000Kbit ceil 100000Kbit burst 51587b/8 mpu
>> 0b overhead 0b cburst 51587b/8 mpu 0b overhead 0b level 7
>> 
>> class htb 1:2000 parent 1:2 leaf 2000: prio 0 quantum 200000 rate
>> 50000Kbit ceil 100000Kbit burst 26593b/8 mpu 0b overhead 0b cburst
>> 51587b/8 mpu 0b overhead 0b level 0
>> 
>> class htb 1:3010 parent 1:2 leaf 3010: prio 1 quantum 1000 rate
>> 64000bit ceil 256000bit burst 1631b/8 mpu 0b overhead 0b cburst
>> 1727b/8 mpu 0b overhead 0b level 0
>> 
>> class htb 1:3020 parent 1:2 leaf 3020: prio 2 quantum 1600 rate
>> 128000bit ceil 768000bit burst 1663b/8 mpu 0b overhead 0b cburst
>> 1983b/8 mpu 0b overhead 0b level 0
>> 
>> class htb 1:3030 parent 1:2 leaf 3030: prio 3 quantum 6400 rate
>> 512000bit ceil 2048Kbit burst 1855b/8 mpu 0b overhead 0b cburst
>> 2623b/8 mpu 0b overhead 0b level 0
>> 
>> class htb 1:5040 parent 1:2 leaf 5040: prio 4 quantum 4825 rate
>> 386000bit ceil 386000bit burst 1792b/8 mpu 0b overhead 0b cburst
>> 1792b/8 mpu 0b overhead 0b level 0

AK> It's as I suspected, your current HTB tree looks like this:

AK> 1: HTB Qdisc
AK> |
AK> \--- 1:2 HTB root class (100000Kbit:100000kbit)
AK>      |
AK>      \--- 1:2000 HTB leaf class (50000Kbit:100000Kbit)
AK>      \--- 1:3010 HTB leaf class (64000bit:256000bit)
AK>      \--- 1:3020 HTB leaf class (128000bit:768000bit)
AK>      \--- 1:3030 HTB leaf class (512000bit:2048Kbit)
AK>      \--- 1:5040 HTB leaf class (386000bit:386000bit)

AK> HTB classes borrow from their parent; in this setup, the parent class
AK> offers a whopping 100000Kbit for that purpose. Unless the 1:2000 class
AK> has got a very high priority and is maxing out the line all the time,
AK> there is no limit to borrowing at all, because the other classes will
AK> never reach the 100000Kbit from their parent.

AK> So the classes above are actually not limited by their rate, but by
AK> their ceil; the only class that will respect its rate in this setup
AK> is 1:5040, because it's got the same rate and ceil.

AK> Assuming that 1:5040 was your P2P class, if you set the ceil of this
AK> class to 2048Kbit, it will (try to) use 2048Kbit at all times, because
AK> the parent (thinks it) is able to offer it.

AK> You need a class that knows of your total internet bandwidth somewhere.
AK> Assuming that it is 2048Kbit, your tree should maybe look more like this:

AK> 1: HTB Qdisc
AK> |
AK> \--- 1:2 HTB root class (100000Kbit:100000kbit)
AK>      |
AK>      \--- 1:2000 HTB leaf class (50000Kbit:100000Kbit)
AK>      |
AK>      \--- 1:3000 HTB parent class (2048Kbit:2048Kbit)
AK>           |
AK>           \--- 1:3010 HTB leaf class (64000bit:256000bit)
AK>           \--- 1:3020 HTB leaf class (128000bit:768000bit)
AK>           \--- 1:3030 HTB leaf class (512000bit:2048Kbit)
AK>           \--- 1:5040 HTB leaf class (386000bit:386000bit)

AK> In this setup, the 2048Kbit class is the limiting factor for the leaf
AK> classes, except for the 1:2000 class, which should be used for local
AK> LAN traffic only.

AK> HTH
AK> Andreas Klauer

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HTB, strange capacity distribution

[Andreas Klauer]

On Tue, Feb 21, 2006 at 02:21:36PM +0100, Boris Gereg wrote:
> thanks Andreas, I reconfigured HTB to get your suggested hierarhy:

One thing I forgot in my graph: Make sure that the rates always add up, 
i.e. the sum of the child class rates should equal the parent class rate. 
It's unlikely to be the cause of your problem, but it's important to get 
this right nevertheless.

> Any other things to test, please?

Just to see wether we are going into the right direction at all, could 
you run the following experiment:

- Lower the rate and ceil of class 1:2 to 8096kbit.
- Lower the rate and ceil of class 1:2000 to 7072kbit.
- Lower the rate and ceil of class 1:3000 to 1024kbit.
- For class 1:3010, set rate to 64kbit, ceil to 256kbit.
- For class 1:3020, set rate to 128kbit, ceil to 768kbit.
- For class 1:3040, set rate to 704kbit, ceil to 1024kbit.
- For class 1:5040, set rate to 128kbit, ceil to 1024kbit.

(You can adjust the rates for these classes as you like, just make sure 
 that the sum is 1024kbit)

If in this setup the shaping works as expected - WWW should get 704kbit 
at all times, P2P only slightly more than 128kbit while WWW downloads 
are active - then the limiting and distribution of HTB most likely works, 
and it's just too high rates or r2q/quantum that make it go bad. In this 
case, you'd have to measure realistic throughput rates of your network 
(even a 100mbit LAN may not be able to guarantee 100000kbit at all times) 
and of your internet connection (may not be able to serve 2048kbit at 
all times). For downstream shaping to work, you have to be the bottleneck.

If you get the same problem in this setup (P2P taking all the bandwidth 
away from WWW), then the problem is most likely something different, 
and we have to look at it from a different angle.

Regards
Andreas Klauer
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HTB, strange capacity distribution

[Andreas Klauer]

On Thu, Feb 23, 2006 at 05:00:12AM +0100, Boris Gereg wrote:
> I did what you suggested and the results are as expected!
> You can see this picture to verify: http://elusion.sk/visual_inet_7.png
> 
> At 4:25 I started HTTP download. P2P class immediately droped down to
> it's RATE, WWW class got it's RATE. At 4:33 I stopped HTTP download,
> P2P class got rest of capacity.

Alright. I suggest you do some measuring, to find out your real rates, 
and set HTB rates to be slightly lower so that you are the bottleneck. 
Most likely you'll have to experiment a little until you find the best 
setting for your setup.

> Please, are there some hints for setting r2q or quantum parameters?

Actually, I specify the quantum directly, with 'quantum $MTU'
for every class. I don't know wether that's a good thing or a 
bad thing, but it worked very well for me, and seems to work 
well for others... at least nobody reported a problem to me so 
far that could be traced to be caused by this quantum setting.

It should not be smaller than your MTU, and not too big.

With a huge difference in rates (100Mbit vs 64kbit) there is no r2q 
that will fit all classes. So there is no other way as to set quantum 
directly at least for some classes (and I set it for all...).

Regards
Andreas Klauer
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc