[LARTC] HTB not working quite the way I'd expect?

[LARTC] HTB not working quite the way I'd expect?

[Jason Tackaberry]

I have a 2Mbit connection, and I would like to distribute that 250k/s
(upstream bandwidth, of course, because I am adding the qdisc on our
internet interface) as follows:
        * From our residence students (199.212.55.128/25), min 50k/s,
          ceil 200k/s
        * From our DMZ network (where our webserver is located,
          199.212.53.0/24), min 200k/s, ceil 250k/s

This is the configuration I _think_ should work:

    tc qdisc add dev eth8 root handle 1: htb default 11
    tc class add dev eth8 parent 1: classid 1:1 htb rate 250kbps
    tc class add dev eth8 parent 1:1 classid 1:10 htb rate 200kbps\
     ceil 250kbps burst 10k
    tc qdisc add dev eth8 parent 1:10 sfq perturb 5
    tc filter add dev eth8 protocol ip parent 1: prio 1 u32 match ip\
     src 199.212.53.0/24 flowid 1:10
    
    tc class add dev eth8 parent 1:1 classid 1:11 htb rate 50kbps \
     ceil 200kbps burst 2k
    tc qdisc add dev eth8 parent 1:11 sfq perturb 10
    
    tc filter add dev eth8 protocol ip parent 1: prio 4 u32 match \
     ip src 199.212.55.128/25 flowid 1:11
    
So, our residence students are assigned to classid 1:11.  Notice that
1:10, our DMZ, is given prio 1, because I want it to get any extra
bandwidth before our residence.  Indeed, residence users are happily
throttled at 200k/s (sustaining it, thanks to the proliferation of p2p
file sharing programs).  However, when I hit our webserver (from 3
outside sites with lots of bandwidth), I am only able to transfer at
about 65k/s (peak 85k/s), while the residence students only drop down to
about 160k-180k/s.  What I would expect to happen is residence dropping
to 50k/s, and our webserver serving up at 200k/s.  (Incidentally I am
monitoring the input/output on the interfaces using iptraf to measure
bandwidth usage.)  I am sustaining the transfer for 20-30 seconds before
measuring bandwidth usage.

If I drop the ceil on 1:11 to 150kbps, I am able to get about 100-120k/s
from the web server.  And if I again drop the ceil to 100kbps, I get
around 175k/s from the webserver.  So clearly my remote sites are able
to pull down the full 200k/s, which is my goal.

So knowing what I want, and my current configuration, can anyone see
what I'm doing wrong?

Thanks,
Jason.

-- 
Academic Computing Support Specialist
Algoma University College
Sault Ste. Marie, Ontario
705-949-2301 x330                                   Personal Home Page
http://www.auc.ca                                     http://sault.org

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/

Re: [LARTC] HTB not working quite the way I'd expect?

[Martin Devera]

>     tc class add dev eth8 parent 1:1 classid 1:10 htb rate 200kbps\
>      ceil 250kbps burst 10k
>     tc filter add dev eth8 protocol ip parent 1: prio 1 u32 match ip\
>      src 199.212.53.0/24 flowid 1:10
>     
>     tc class add dev eth8 parent 1:1 classid 1:11 htb rate 50kbps \
>      ceil 200kbps burst 2k

> So, our residence students are assigned to classid 1:11.  Notice that
> 1:10, our DMZ, is given prio 1, because I want it to get any extra
> bandwidth before our residence.  Indeed, residence users are happily

You did it wrong ! prio 1 attached to filter has to do nothing with 
prio 1 attached to the class. I have a BIG objectives to the overal
filter design in Linux QoS but I can't do anything with it.
Simply add prio 1 to the class definition (last tc line above).
And let me know ! ;)

devik

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/

Re: [LARTC] HTB not working quite the way I'd expect?

[Jason Tackaberry]

Hi Martin.  Thanks for your reply.

On Sat, 2002-02-02 at 08:14, Martin Devera wrote:
> filter design in Linux QoS but I can't do anything with it.
> Simply add prio 1 to the class definition (last tc line above).
> And let me know ! ;)

In fact, there is no change.  Here is my current configuration, with the
changes you suggested:

    tc class add dev eth8 parent 1: classid 1:1 htb rate 250kbps
    tc class add dev eth8 parent 1:1 classid 1:10 htb rate 200kbps \
      ceil 250kbps burst 10k prio 1
    tc qdisc add dev eth8 parent 1:10 sfq perturb 5
    tc filter add dev eth8 protocol ip parent 1: u32 match ip \
      src 199.212.53.0/24 flowid 1:10
    
    tc class add dev eth8 parent 1:1 classid 1:11 htb rate 50kbps \
      ceil 200kbps burst 2k prio 4
    tc qdisc add dev eth8 parent 1:11 sfq perturb 5
    
    tc filter add dev eth8 protocol ip parent 1: u32 match ip \
       src 199.212.55.128/25 flowid 1:11
    
Monitoring eth8, I see it sending out at 200k/s, all traffic generated
from 199.212.55.128/25.  This is what I'd expect.  When I hit the
webserver (on 199.212.53.0/24) hard, eth8 begins to send at 250k/s (it
is a 2Mbit uplink), but the portion of that from the webserver is only
about 50-90/s, not 200k/s like I'd expect based on id 1:10.

Any other thoughts?

Regards,
Jason.

-- 
Academic Computing Support Specialist
Algoma University College
Sault Ste. Marie, Ontario
705-949-2301 x330                                   Personal Home Page
http://www.auc.ca                                     http://sault.org

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/

Re: [LARTC] HTB not working quite the way I'd expect?

[Martin Devera]

> > Simply add prio 1 to the class definition (last tc line above).
> > And let me know ! ;)
> 
> In fact, there is no change.  Here is my current configuration, with the
> changes you suggested:
> 
>       ceil 200kbps burst 2k prio 4

default max prio is 3 by the way but it is not problem here ..

> Monitoring eth8, I see it sending out at 200k/s, all traffic generated
> from 199.212.55.128/25.  This is what I'd expect.  When I hit the
> webserver (on 199.212.53.0/24) hard, eth8 begins to send at 250k/s (it
> is a 2Mbit uplink), but the portion of that from the webserver is only
> about 50-90/s, not 200k/s like I'd expect based on id 1:10.

very interesting ! I'm interested in "tc -s -d class show dev eth8" output
in middle of your run. Please take several snapshots, copy into file along
with time (I need to know the time difference between them).
I'll look at them and try to deduce the problem. I use configuration like
your regulary with no probs so that I'm really interested.

devik

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/

Re: [LARTC] HTB not working quite the way I'd expect?

[Jason Tackaberry]

[-- Attachment #1: Type: text/plain, Size: 1177 bytes --]

On Sun, 2002-02-03 at 15:20, Martin Devera wrote:
> very interesting ! I'm interested in "tc -s -d class show dev eth8" output
> in middle of your run. Please take several snapshots, copy into file along
> with time (I need to know the time difference between them).

Okay.  Attached is a commented typescript taken.  The first invocation
of 'tc show' is before I initiated the remote wgets, the next 2 are
during, and the last is after I stopped them.  I don't quite understand
all the data, but strictly looking at the rate field, I think it pretty
nicely shows what's happening.  (Except on the last reading, which shows
a rate of ~40k/s even though the web server's interface was idle -- I
assumed it was just an averaging window issue.)

> I'll look at them and try to deduce the problem. I use configuration like
> your regulary with no probs so that I'm really interested.

Thanks very much for your help on this, Martin. :)

Jason.

-- 
Academic Computing Support Specialist
Algoma University College
Sault Ste. Marie, Ontario
705-949-2301 x330                                   Personal Home Page
http://www.auc.ca                                     http://sault.org

[-- Attachment #2: Type: text/plain, Size: 5689 bytes --]

###
# Before hitting the webserver.  This is normal traffic generated by
# our residence.  iptraf showed ~200k/s being sent on eth8.
########

[root@hawk tack]# date && tc -s -d class show dev eth8
Sun Feb  3 15:47:22 EST 2002
class htb 1:11 parent 1:1 leaf 8066: prio 3 rate 400Kbit ceil 1600Kbit burst 2Kb/8 mpu 0b cburst 3647b/8 mpu 0b quantum 5120 level 0 
 Sent 75213499 bytes 106516 pkts (dropped 34, overlimits 70905) 
 rate 192901bps 236pps backlog 36p 
 lended: 38257 borrowed: 68223 giants: 0 injects: 0
 tokens: -118 ctokens: -3836

class htb 1:1 root prio 0 rate 2000Kbit ceil 2000Kbit burst 4159b/8 mpu 0b cburst 4159b/8 mpu 0b quantum 25600 level 3 
 Sent 87895315 bytes 121163 pkts (dropped 0, overlimits 0) 
 rate 191724bps 239pps 
 lended: 68223 borrowed: 0 giants: 0 injects: 0
 tokens: 3968 ctokens: 3968

class htb 1:10 parent 1:1 leaf 8065: prio 1 rate 1600Kbit ceil 2000Kbit burst 10Kb/8 mpu 0b cburst 4159b/8 mpu 0b quantum 20480 level 0 
 Sent 12717097 bytes 14683 pkts (dropped 0, overlimits 2668) 
 rate 1380bps 9pps 
 lended: 14683 borrowed: 0 giants: 0 injects: 0
 tokens: 40577 ctokens: 13004






###
# Immediately following the previous command, I began the run by initiating
# wget on 3 separate sites downloading from our webserver (a large file).
########

[root@hawk tack]# date && tc -s -d class show dev eth8
Sun Feb  3 15:47:44 EST 2002
class htb 1:11 parent 1:1 leaf 8066: prio 3 rate 400Kbit ceil 1600Kbit burst 2Kb/8 mpu 0b cburst 3647b/8 mpu 0b quantum 5120 level 0 
 Sent 78931896 bytes 110921 pkts (dropped 34, overlimits 76204) 
 rate 182403bps 219pps backlog 18p 
 lended: 39606 borrowed: 71297 giants: 0 injects: 0
 tokens: -7489 ctokens: -4151

class htb 1:1 root prio 0 rate 2000Kbit ceil 2000Kbit burst 4159b/8 mpu 0b cburst 4159b/8 mpu 0b quantum 25600 level 3 
 Sent 93010578 bytes 126696 pkts (dropped 0, overlimits 0) 
 rate 1663Kbit 247pps 
 lended: 71297 borrowed: 0 giants: 0 injects: 0
 tokens: -2 ctokens: -2

class htb 1:10 parent 1:1 leaf 8065: prio 1 rate 1600Kbit ceil 2000Kbit burst 10Kb/8 mpu 0b cburst 4159b/8 mpu 0b quantum 20480 level 0 
 Sent 14099891 bytes 15793 pkts (dropped 0, overlimits 2988) 
 rate 32109bps 28pps 
 lended: 15793 borrowed: 0 giants: 0 injects: 0
 tokens: 19305 ctokens: 2537




###
# The wgets are still going. 
########

[root@hawk tack]# date && tc -s -d class show dev eth8
Sun Feb  3 15:48:00 EST 2002
class htb 1:11 parent 1:1 leaf 8066: prio 3 rate 400Kbit ceil 1600Kbit burst 2Kb/8 mpu 0b cburst 3647b/8 mpu 0b quantum 5120 level 0 
 Sent 81834500 bytes 114311 pkts (dropped 34, overlimits 80140) 
 rate 182926bps 217pps backlog 16p 
 lended: 40580 borrowed: 73715 giants: 0 injects: 0
 tokens: -596 ctokens: -1591

class htb 1:1 root prio 0 rate 2000Kbit ceil 2000Kbit burst 4159b/8 mpu 0b cburst 4159b/8 mpu 0b quantum 25600 level 3 
 Sent 96964624 bytes 130954 pkts (dropped 0, overlimits 0) 
 rate 230521bps 258pps 
 lended: 73715 borrowed: 0 giants: 0 injects: 0
 tokens: 7272 ctokens: 7272

class htb 1:10 parent 1:1 leaf 8065: prio 1 rate 1600Kbit ceil 2000Kbit burst 10Kb/8 mpu 0b cburst 4159b/8 mpu 0b quantum 20480 level 0 
 Sent 15150851 bytes 16659 pkts (dropped 0, overlimits 3082) 
 rate 47974bps 41pps 
 lended: 16659 borrowed: 0 giants: 0 injects: 0
 tokens: 21062 ctokens: 667





###
# Immediately following the last command, I stopped the wgets.  Our 
# webserver was idle.  For whatever reason, it shows the rate of 1:10 at
# 62790bps but the web server's interface was clearly idle.  I assume it
# averages rate over a wide window, like 20-30 seconds?
########

[root@hawk tack]# date && tc -s -d class show dev eth8
Sun Feb  3 15:48:14 EST 2002
class htb 1:11 parent 1:1 leaf 8066: prio 3 rate 400Kbit ceil 1600Kbit burst 2Kb/8 mpu 0b cburst 3647b/8 mpu 0b quantum 5120 level 0 
 Sent 84370895 bytes 117247 pkts (dropped 34, overlimits 83715) 
 rate 177121bps 208pps backlog 19p 
 lended: 41432 borrowed: 75796 giants: 0 injects: 0
 tokens: -13591 ctokens: -216

class htb 1:1 root prio 0 rate 2000Kbit ceil 2000Kbit burst 4159b/8 mpu 0b cburst 4159b/8 mpu 0b quantum 25600 level 3 
 Sent 100567608 bytes 134827 pkts (dropped 0, overlimits 0) 
 rate 240215bps 261pps 
 lended: 75796 borrowed: 0 giants: 0 injects: 0
 tokens: -3658 ctokens: -3658

class htb 1:10 parent 1:1 leaf 8065: prio 1 rate 1600Kbit ceil 2000Kbit burst 10Kb/8 mpu 0b cburst 4159b/8 mpu 0b quantum 20480 level 0 
 Sent 16215919 bytes 17599 pkts (dropped 0, overlimits 3251) 
 rate 62790bps 52pps 
 lended: 17599 borrowed: 0 giants: 0 injects: 0
 tokens: 22819 ctokens: -1203

[root@hawk tack]# date && tc -s -d class show dev eth8
Sun Feb  3 15:48:37 EST 2002
class htb 1:11 parent 1:1 leaf 8066: prio 3 rate 400Kbit ceil 1600Kbit burst 2Kb/8 mpu 0b cburst 3647b/8 mpu 0b quantum 5120 level 0 
 Sent 88862765 bytes 122423 pkts (dropped 34, overlimits 89207) 
 rate 187439bps 221pps backlog 25p 
 lended: 42738 borrowed: 79660 giants: 0 injects: 0
 tokens: -17782 ctokens: -2329

class htb 1:1 root prio 0 rate 2000Kbit ceil 2000Kbit burst 4159b/8 mpu 0b cburst 4159b/8 mpu 0b quantum 25600 level 3 
 Sent 105201282 bytes 140534 pkts (dropped 0, overlimits 0) 
 rate 231031bps 267pps 
 lended: 79660 borrowed: 0 giants: 0 injects: 0
 tokens: 7044 ctokens: 7044

class htb 1:10 parent 1:1 leaf 8065: prio 1 rate 1600Kbit ceil 2000Kbit burst 10Kb/8 mpu 0b cburst 4159b/8 mpu 0b quantum 20480 level 0 
 Sent 16367152 bytes 18136 pkts (dropped 0, overlimits 3262) 
 rate 42641bps 45pps 
 lended: 18136 borrowed: 0 giants: 0 injects: 0
 tokens: 40609 ctokens: 13031

Re: [LARTC] HTB not working quite the way I'd expect?

[Jason Tackaberry]

(tc output below cropped)
> class htb 1:11 parent 1:1 leaf 8066: prio 3 rate 400Kbit ceil
>   1600Kbit burst 2Kb/8 mpu 0b cburst 3647b/8 mpu 0b quantum 5120 ]
>   level 0
>  rate 192901bps 236pps backlog 36p 
> 
> class htb 1:1 root prio 0 rate 2000Kbit ceil 2000Kbit burst 4159b/8
>    mpu 0b cburst 4159b/8 mpu 0b quantum 25600 level 3 
>  rate 191724bps 239pps 
> 
> class htb 1:10 parent 1:1 leaf 8065: prio 1 rate 1600Kbit ceil
>   2000Kbit burst 10Kb/8 mpu 0b cburst 4159b/8 mpu 0b quantum 20480 
>   level 0 
>  rate 1380bps 9pps 

Now that I study this a bit more, how can it be that the parent class,
1:1, has a rate that is LESS than the sum of its children's rates? 
Either I've uncovered a bug, or I'm greatly misunderstanding something.
;)

Jason.

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/

Re: [LARTC] HTB not working quite the way I'd expect?

[Martin Devera]

> >  rate 192901bps 236pps backlog 36p 
> >  rate 191724bps 239pps 
> >  rate 1380bps 9pps 
> 
> Now that I study this a bit more, how can it be that the parent class,
> 1:1, has a rate that is LESS than the sum of its children's rates? 
> Either I've uncovered a bug, or I'm greatly misunderstanding something.
> ;)

The rate discrepancy is ok. Each class is averaged on its own and
there is really possible to overshot parent's rate for a while. It
is accounted to the child later.
But is seems to me that problem is at your www server. First, the
HTB reports no backlog for the www connection - VERY unlikely ! It
means that all data has been sent and no new arrived.
The same is shown by tokens. They are positive which means that
class is not throtling your data.
I'm not sure what is going here but absence of backlog is alarming.
SFQ can hold up to 128 packets and server which is trying to send
more data would create some backlog, wouldn't it ?
devik

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/

Re: [LARTC] HTB not working quite the way I'd expect?

[Jason Tackaberry]

On Sun, 2002-02-03 at 17:25, Martin Devera wrote:
> But is seems to me that problem is at your www server. First, the
> HTB reports no backlog for the www connection - VERY unlikely ! It
> means that all data has been sent and no new arrived.
> The same is shown by tokens. They are positive which means that
> class is not throtling your data.

Okay, except if I drop the ceil of 1:11 to 50kbps, I can get about
150-200k/s from the webserver, which tells me that there is no
bottleneck on the webserver or on the remote sites pulling the data
down.

What else could account for the fact that there is no backlog?

Jason.

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/

Re: [LARTC] HTB not working quite the way I'd expect?

[Martin Devera]

> Okay, except if I drop the ceil of 1:11 to 50kbps, I can get about
> 150-200k/s from the webserver, which tells me that there is no
> bottleneck on the webserver or on the remote sites pulling the data
> down.
> 
> What else could account for the fact that there is no backlog?

Only bug. :-| Just to eliminate potential bugs please try to add
burst 20k to all your classes (1:1,10,11) and send me statistics
again - the one during download is enough. It would be nice to
do it like:
while true; do tc -s -d class show dev eth8; sleep 1;done > file

to get idea what is done during whole time. There is still "brute
force" way: run htb with debug 3333333 and do tcpdump of packets.
But it would be time consuming way :(
By the way, what speed is the machine ?

devik

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/

Re: [LARTC] HTB not working quite the way I'd expect?

[Jason Tackaberry]

On Mon, 2002-02-04 at 02:01, Martin Devera wrote:
> burst 20k to all your classes (1:1,10,11) and send me statistics
> again - the one during download is enough. It would be nice to
> do it like:
> while true; do tc -s -d class show dev eth8; sleep 1;done > file

Okay.  I've accumulated about 30k worth of data, which I will attach to
a separate email just to you (so as not to spam the list).

> By the way, what speed is the machine ?

It is a Celeron 366, with a load average of about 0.10.

Jason.

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/