From mboxrd@z Thu Jan 1 00:00:00 1970 From: "matt" Date: Tue, 09 Dec 2003 09:31:14 +0000 Subject: [LARTC] Strange HTB rate and ceil setting situation with adsl MIME-Version: 1 Content-Type: multipart/mixed; boundary="----=_NextPart_000_001B_01C3BE7A.3EBA7340" Message-Id: List-Id: To: lartc@vger.kernel.org This is a multi-part message in MIME format. ------=_NextPart_000_001B_01C3BE7A.3EBA7340 Content-Type: text/plain; charset="big5" Content-Transfer-Encoding: quoted-printable Hi, all: I have been used HTB Queue within netfilter bridge environment over=20 two year and It's working fine. Very thank for htb writer Martin Devera = and=20 experience share with Stef Coene... :) Last week I helped my friend to setup box that use nf-bridge + htb = with 512/64Kbit ADSL. When I use two polciy to limit the bandwith and use = ceil to=20 share bandwith. The rate and ceil parameter can not work according to = setting. I only test the download rule. Rule 1: Http download have 384Kbit rate and 512Kbit ceil. Rule 2: Ftp download have 64Kbit rate and 512Kbit ceil. Default rule: download have 64Kbit rate and 512Kbit ceil. =20 When I start ftp session to download one file, bandwith can be about = 500kbit. But when I start another http session to download file(ftp seeion still = on). The http session only get 266Kbps bandwith and ftp session rate down to = 167Kbps or sometime ftp will get more bandwidth. Why http session can't get the 384Kbit rate? In my enviroment (nat box = +=20 bridge qos), the ceil setting can work. The only different between me = and my=20 friend is the nat gateway. I was using linux be nat gateway and my = firend using normally IP Sharing router.=20 PS: We also use same ISP. To found out why, I have done some experiment that remove my friend's = nat=20 router and only use nf-bridge + htb to test those tc rule. It the same. The orign kernel is 2.4.18 + htb2 + nf-bridge-0.0.8. Even though I = change=20 to linux-2.4.21 + htb3 + nf-bridge-0.0.9. It also the same result.=20 If I change test enviroment to local ethernet network,it can work again. In my guess, may be it is dsl problem. But if have linux nat router = before=20 bridge box, htb queue will be work. It's a little strange... :( Of course, I can use nat + htb in same box to solve the problem. But may be to understand to reason will be good. Have someone talk me. Does other some reason will cause this situation = or have some method to solve this problem? Thank's for your advise. =20 The tc rule and qdisc show are list below: Wan nic is eth0, Lan nic is eth1. The test client ip address is = 192.168.1.10. /sbin/tc qdisc del dev eth0 root /sbin/tc qdisc add dev eth0 root handle 10:0 htb r2q 10 default ffff /sbin/tc class add dev eth0 parent 10:0 classid 10:1 htb rate 64kbit = ceil 64kbit=20 /sbin/tc class add dev eth0 parent 10:1 classid 10:ffff htb rate 12kbit = ceil 64kbit prio 3 /sbin/tc qdisc add dev eth0 parent 10:ffff handle ffff:0 sfq perturb 10=20 /sbin/tc class add dev eth0 parent 10:1 classid 10:2 htb rate 36kbit = ceil 64kbit prio 1 /sbin/tc qdisc add dev eth0 parent 10:2 handle 22:0 sfq perturb 10=20 /sbin/tc filter add dev eth0 parent 10:0 protocol ip prio 1 handle 2 fw = classid 10:2 /sbin/tc class add dev eth0 parent 10:1 classid 10:4 htb rate 16kbit = ceil 64kbit prio 2 /sbin/tc qdisc add dev eth0 parent 10:4 handle 24:0 sfq perturb 10=20 /sbin/tc filter add dev eth0 parent 10:0 protocol ip prio 2 handle 4 fw = classid 10:4 /sbin/tc qdisc del dev eth1 root /sbin/tc qdisc add dev eth1 root handle 10:0 htb r2q 10 default ffff /sbin/tc class add dev eth1 parent 10:0 classid 10:1 htb rate 512kbit = ceil 512kbit=20 /sbin/tc class add dev eth1 parent 10:1 classid 10:ffff htb rate 64kbit = ceil 512kbit prio 3 /sbin/tc qdisc add dev eth1 parent 10:ffff handle ffff:0 sfq perturb 10=20 /sbin/tc class add dev eth1 parent 10:1 classid 10:3 htb rate 384kbit = ceil 512kbit prio 1 /sbin/tc qdisc add dev eth1 parent 10:3 handle 23:0 sfq perturb 10=20 /sbin/tc filter add dev eth1 parent 10:0 protocol ip prio 2 handle 3 fw = classid 10:3 /sbin/tc class add dev eth1 parent 10:1 classid 10:5 htb rate 64kbit = ceil 512kbit prio 2 /sbin/tc qdisc add dev eth1 parent 10:5 handle 25:0 sfq perturb 10=20 /sbin/tc filter add dev eth1 parent 10:0 protocol ip prio 2 handle 5 fw = classid 10:5 /sbin/iptables -I PREROUTING -t mangle -p tcp -s 192.168.1.10 -m mport = --ports 80 -j MARK --set-mark 2 /sbin/iptables -I PREROUTING -t mangle -p tcp -d 192.168.1.10 -m mport = --ports 80 -j MARK --set-mark 3 /sbin/iptables -I PREROUTING -t mangle -p all -s 192.168.1.10 -m helper = --helper ftp -j MARK --set-mark 4 /sbin/iptables -I PREROUTING -t mangle -p all -d 192.168.1.10 -m helper = --helper ftp -j MARK --set-mark 5 ###### iptables mark rule list ###### Chain PREROUTING (policy ACCEPT 9102 packets, 259K bytes) pkts bytes target prot opt in out source = destination 10117 15M MARK all -- * * 0.0.0.0/0 = 192.168.1.10 helper match "ftp" MARK set 0x5 7172 392K MARK all -- * * 192.168.1.10 = 0.0.0.0/0 helper match "ftp" MARK set 0x4 26520 40M MARK tcp -- * * 0.0.0.0/0 = 192.168.1.10 mport ports 80 MARK set 0x3 18545 796K MARK tcp -- * * 192.168.1.10 = 0.0.0.0/0 mport ports 80 MARK set 0x2 ###### /sbin/tc -s -d qdisc show dev eth0 ###### class htb 10:ffff parent 10:1 leaf ffff: prio 3 rate 12Kbit ceil 64Kbit = burst 1614b/8 mpu 0b cburst 1680b/8 mpu 0b quantum 153 level 0=20 Sent 7372729 bytes 61380 pkts (dropped 158729, overlimits 58778)=20 rate 5bps=20 lended: 12217 borrowed: 49163 giants: 0 injects: 0 tokens: 831467 ctokens: 162500 class htb 10:1 root prio 0 rate 64Kbit ceil 64Kbit burst 1680b/8 mpu 0b = cburst 1680b/8 mpu 0b quantum 819 level 3=20 Sent 9345277 bytes 78138 pkts (dropped 0, overlimits 0)=20 rate 1603bps 25pps=20 lended: 49163 borrowed: 0 giants: 0 injects: 0 tokens: 163300 ctokens: 163300 class htb 10:2 parent 10:1 leaf 22: prio 1 rate 36Kbit ceil 64Kbit burst = 1645b/8 mpu 0b cburst 1680b/8 mpu 0b quantum 460 level 0=20 Sent 403214 bytes 7034 pkts (dropped 0, overlimits 0)=20 rate 736bps 13pps=20 lended: 7034 borrowed: 0 giants: 0 injects: 0 tokens: 284089 =20 ##### /sbin/tc -s -d qdisc show dev eth1 ###### class htb 10:ffff parent 10:1 leaf ffff: prio 3 rate 64Kbit ceil 512Kbit = burst 1680b/8 mpu 0b cburst 2Kb/8 mpu 0b quantum 819 level 0=20 Sent 58166130 bytes 518222 pkts (dropped 1713, overlimits 482227)=20 lended: 72484 borrowed: 445738 giants: 0 injects: 0 tokens: 162500 ctokens: 27488 class htb 10:1 root prio 0 rate 512Kbit ceil 512Kbit burst 2Kb/8 mpu 0b = cburst 2Kb/8 mpu 0b quantum 6553 level 3=20 Sent 98025731 bytes 544613 pkts (dropped 0, overlimits 0)=20 rate 55148bps 36pps=20 lended: 458124 borrowed: 0 giants: 0 injects: 0 tokens: 6773 ctokens: 6773 class htb 10:3 parent 10:1 leaf 23: prio 1 rate 384Kbit ceil 512Kbit = burst 2Kb/8 mpu 0b cburst 2Kb/8 mpu 0b quantum 4915 level 0=20 Sent 16020166 bytes 10591 pkts (dropped 0, overlimits 8155)=20 rate 266Kbit 22pps=20 lended: 10323 borrowed: 268 giants: 0 injects: 0 tokens: 212 ctokens: 6773 class htb 10:5 parent 10:1 leaf 25: prio 2 rate 64Kbit ceil 512Kbit = burst 1680b/8 mpu 0b cburst 2Kb/8 mpu 0b quantum 819 level 0=20 Sent 23876440 bytes 15800 pkts (dropped 0, overlimits 1)=20 rate 167Kbit 14pps=20 lended: 3682 borrowed: 12118 giants: 0 injects: 0 tokens: -140306 ctokens: 6773 =20 ##### /sbin/tc -s -d qdisc show dev eth1 ###### Again ###### class htb 10:ffff parent 10:1 leaf ffff: prio 3 rate 64Kbit ceil 512Kbit = burst 1680b/8 mpu 0b cburst 2Kb/8 mpu 0b quantum 819 level 0 Sent 58166256 bytes 518225 pkts (dropped 1713, overlimits 482227) lended: 72487 borrowed: 445738 giants: 0 injects: 0 tokens: 164100 ctokens: 27688 class htb 10:1 root prio 0 rate 512Kbit ceil 512Kbit burst 2Kb/8 mpu 0b = cburst 2Kb/8 mpu 0b quantum 6553 level 3 Sent 118164481 bytes 557926 pkts (dropped 0, overlimits 0) rate 430Kbit 36pps lended: 461746 borrowed: 0 giants: 0 injects: 0 tokens: 6773 ctokens: 6773 class htb 10:3 parent 10:1 leaf 23: prio 1 rate 384Kbit ceil 512Kbit = burst 2Kb/8 mpu 0b cburst 2Kb/8 mpu 0b quantum 4915 level 0 Sent 28120752 bytes 18590 pkts (dropped 0, overlimits 8155) rate 28117bps 18pps lended: 18183 borrowed: 407 giants: 0 injects: 0 tokens: 212 ctokens: 6773 class htb 10:5 parent 10:1 leaf 25: prio 2 rate 64Kbit ceil 512Kbit = burst 1680b/8 mpu 0b cburst 2Kb/8 mpu 0b quantum 819 level 0 Sent 31914478 bytes 21111 pkts (dropped 0, overlimits 1) rate 26828bps 17pps lended: 5510 borrowed: 15601 giants: 0 injects: 0 tokens: -115091 ctokens: 9288 Regards, Matt. ------=_NextPart_000_001B_01C3BE7A.3EBA7340 Content-Type: text/html; charset="big5" Content-Transfer-Encoding: quoted-printable
Hi, all:
 
   I have been used HTB = Queue within=20 netfilter bridge environment over 
two year and It's working = fine. Very=20 thank for htb writer Martin Devera and
experience share with Stef = Coene...=20 :)
 
   Last week I helped my = friend to=20 setup box that use nf-bridge + htb with
512/64Kbit ADSL. When I use = two=20 polciy to limit the bandwith and use ceil to
share bandwith. The = rate and=20 ceil parameter can not work according to setting.
 
I only test the download = rule.
 
Rule 1: Http download have 384Kbit = rate and=20 512Kbit ceil.
Rule 2: Ftp download have 64Kbit rate and 512Kbit=20 ceil.
Default rule: download have 64Kbit rate and 512Kbit=20 ceil.
 
When I start ftp session to download one file, = bandwith can=20 be about 500kbit.
But when I start another http session to download = file(ftp=20 seeion still on).
The http session only get 266Kbps bandwith and ftp = session=20 rate down to 167Kbps
or sometime ftp will get more=20 bandwidth.

Why http session can't get the = 384Kbit=20 rate?  In my enviroment (nat = box +=20
bridge qos), the ceil = setting can work. The only different between me = and my=20
friend is the nat gateway. I was = using linux be=20 nat gateway and my firend using
normally IP Sharing router. =
PS: We also use same = ISP.
 
To found out why,  I have = done some=20 experiment that remove my friend's nat
router and only use nf-bridge = + htb=20 to test those tc rule. It the same.
The orign kernel is 2.4.18 + htb2 +=20 nf-bridge-0.0.8. Even though I change
to linux-2.4.21 + htb3 + = nf-bridge-0.0.9. =20 It also the same result.
If I change test enviroment to local = ethernet=20 network,it can work again.
 
In my guess, may be it is dsl = problem. But if=20 have linux nat router before
bridge box, htb queue will be work. = It's a little=20 strange... :(
 
Of course, I can use nat + htb in = same box to=20 solve the problem. But may
be to understand to reason = will be good.
 
Have someone talk me. Does other some = reason will=20 cause this situation or
have some method to solve this=20 problem?
 
Thank's for your = advise.
 
The tc rule=20 and qdisc show are list below:
Wan nic is eth0, Lan nic is eth1. The = test=20 client ip address is 192.168.1.10.
 
/sbin/tc qdisc del dev eth0 = root
/sbin/tc=20 qdisc add dev eth0 root handle 10:0 htb r2q 10 default ffff
/sbin/tc = class=20 add dev eth0 parent 10:0 classid 10:1 htb rate 64kbit ceil 64kbit =
/sbin/tc=20 class add dev eth0 parent 10:1 classid 10:ffff htb rate 12kbit ceil = 64kbit prio=20 3
/sbin/tc qdisc add dev eth0 parent 10:ffff handle ffff:0 sfq = perturb 10=20
/sbin/tc class add dev eth0 parent 10:1  classid 10:2 htb rate = 36kbit=20 ceil 64kbit prio 1
/sbin/tc qdisc add dev eth0 parent 10:2 handle = 22:0 sfq=20 perturb 10
/sbin/tc filter add dev eth0 parent 10:0 protocol ip prio = 1=20 handle 2 fw classid 10:2
/sbin/tc class add dev eth0 parent = 10:1 =20 classid 10:4 htb rate 16kbit ceil 64kbit prio 2
/sbin/tc qdisc add = dev eth0=20 parent 10:4 handle 24:0 sfq perturb 10
/sbin/tc filter add dev eth0 = parent=20 10:0 protocol ip prio 2 handle 4 fw classid 10:4
 
/sbin/tc qdisc del dev eth1 = root
/sbin/tc=20 qdisc add dev eth1 root handle 10:0 htb r2q 10 default ffff
/sbin/tc = class=20 add dev eth1 parent 10:0 classid 10:1 htb rate 512kbit ceil 512kbit =
/sbin/tc=20 class add dev eth1 parent 10:1 classid 10:ffff htb rate 64kbit ceil = 512kbit prio=20 3
/sbin/tc qdisc add dev eth1 parent 10:ffff handle ffff:0 sfq = perturb 10=20
/sbin/tc class add dev eth1 parent 10:1  classid 10:3 htb rate = 384kbit=20 ceil 512kbit prio 1
/sbin/tc qdisc add dev eth1 parent 10:3 handle = 23:0 sfq=20 perturb 10
/sbin/tc filter add dev eth1 parent 10:0 protocol ip prio = 2=20 handle 3 fw classid 10:3
/sbin/tc class add dev eth1 parent = 10:1 =20 classid 10:5 htb rate 64kbit ceil 512kbit prio 2
/sbin/tc qdisc add = dev eth1=20 parent 10:5 handle 25:0 sfq perturb 10
/sbin/tc filter add dev eth1 = parent=20 10:0 protocol ip prio 2 handle 5 fw classid 10:5
/sbin/iptables -I = PREROUTING=20 -t mangle -p tcp -s 192.168.1.10 -m mport --ports 80 -j MARK --set-mark=20 2
/sbin/iptables -I PREROUTING -t mangle -p tcp -d 192.168.1.10 -m = mport=20 --ports 80 -j MARK --set-mark 3
/sbin/iptables -I PREROUTING -t = mangle -p=20 all  -s 192.168.1.10 -m helper --helper ftp -j MARK --set-mark=20 4
/sbin/iptables -I PREROUTING -t mangle -p all  -d 192.168.1.10 = -m=20 helper --helper ftp -j MARK --set-mark 5
 
###### iptables mark rule list=20 ######
Chain PREROUTING (policy ACCEPT 9102 = packets,=20 259K bytes)
 pkts bytes target     prot opt=20 in     out    =20 source           &= nbsp;  =20 destination
10117   15M = MARK      =20 all  --  *     =20 *      =20 0.0.0.0/0          &nbs= p;=20 192.168.1.10       helper match "ftp" MARK = set=20 0x5
 7172  392K MARK       = all =20 --  *      = *      =20 192.168.1.10       =20 0.0.0.0/0           = helper=20 match "ftp" MARK set 0x4
26520   40M=20 MARK       tcp  -- =20 *      *      =20 0.0.0.0/0          &nbs= p;=20 192.168.1.10       mport ports 80 MARK set = 0x3
18545  796K MARK       = tcp =20 --  *      = *      =20 192.168.1.10       =20 0.0.0.0/0           = mport=20 ports 80 MARK set 0x2
 
###### /sbin/tc -s -d qdisc show dev = eth0=20 ######
class htb 10:ffff parent 10:1 leaf ffff: prio 3 rate 12Kbit = ceil=20 64Kbit burst 1614b/8 mpu 0b cburst 1680b/8 mpu 0b quantum 153 level 0=20
 Sent 7372729 bytes 61380 pkts (dropped 158729, overlimits = 58778)=20
 rate 5bps
 lended: 12217 borrowed: 49163 giants: 0 = injects:=20 0
 tokens: 831467 ctokens: 162500
 
class htb 10:1 root prio 0 rate = 64Kbit ceil=20 64Kbit burst 1680b/8 mpu 0b cburst 1680b/8 mpu 0b quantum 819 level 3=20
 Sent 9345277 bytes 78138 pkts (dropped 0, overlimits 0) =
 rate=20 1603bps 25pps
 lended: 49163 borrowed: 0 giants: 0 injects:=20 0
 tokens: 163300 ctokens: 163300
 
class htb 10:2 parent 10:1 leaf 22: = prio 1 rate=20 36Kbit ceil 64Kbit burst 1645b/8 mpu 0b cburst 1680b/8 mpu 0b quantum = 460 level=20 0
 Sent 403214 bytes 7034 pkts (dropped 0, overlimits 0) =
 rate=20 736bps 13pps
 lended: 7034 borrowed: 0 giants: 0 injects:=20 0
 tokens: 284089
 
##### /sbin/tc -s -d qdisc show = dev eth1=20 ######
class htb 10:ffff parent 10:1 leaf ffff: prio 3 rate 64Kbit = ceil=20 512Kbit burst 1680b/8 mpu 0b cburst 2Kb/8 mpu 0b quantum 819 level 0=20
 Sent 58166130 bytes 518222 pkts (dropped 1713, overlimits = 482227)=20
 lended: 72484 borrowed: 445738 giants: 0 injects: = 0
 tokens:=20 162500 ctokens: 27488
 
class htb 10:1 root prio 0 rate = 512Kbit ceil=20 512Kbit burst 2Kb/8 mpu 0b cburst 2Kb/8 mpu 0b quantum 6553 level 3=20
 Sent 98025731 bytes 544613 pkts (dropped 0, overlimits 0)=20
 rate 55148bps 36pps
 lended: 458124 borrowed: 0 = giants: 0=20 injects: 0
 tokens: 6773 ctokens: 6773
 
class htb 10:3 parent 10:1 leaf 23: = prio 1 rate=20 384Kbit ceil 512Kbit burst 2Kb/8 mpu 0b cburst 2Kb/8 mpu 0b quantum 4915 = level 0=20
 Sent 16020166 bytes 10591 pkts (dropped 0, overlimits 8155)=20
 rate 266Kbit 22pps
 lended: 10323 borrowed: 268 = giants: 0=20 injects: 0
 tokens: 212 ctokens: 6773
 
class htb 10:5 parent 10:1 leaf 25: = prio 2 rate=20 64Kbit ceil 512Kbit burst 1680b/8 mpu 0b cburst 2Kb/8 mpu 0b quantum 819 = level 0=20
 Sent 23876440 bytes 15800 pkts (dropped 0, overlimits 1)=20
 rate 167Kbit 14pps
 lended: 3682 borrowed: 12118 = giants: 0=20 injects: 0
 tokens: -140306 ctokens: 6773
 
##### = /sbin/tc -s=20 -d qdisc show dev eth1 ###### Again ######
class htb 10:ffff parent = 10:1 leaf=20 ffff: prio 3 rate 64Kbit ceil 512Kbit burst 1680b/8 mpu 0b cburst 2Kb/8 = mpu 0b=20 quantum 819 level 0
 Sent 58166256 bytes 518225 pkts (dropped = 1713,=20 overlimits 482227)
 lended: 72487 borrowed: 445738 giants: 0 = injects:=20 0
 tokens: 164100 ctokens: 27688
 
class htb 10:1 root prio 0 rate = 512Kbit ceil=20 512Kbit burst 2Kb/8 mpu 0b cburst 2Kb/8 mpu 0b quantum 6553 level=20 3
 Sent 118164481 bytes 557926 pkts (dropped 0, overlimits=20 0)
 rate 430Kbit 36pps
 lended: 461746 borrowed: 0 = giants: 0=20 injects: 0
 tokens: 6773 ctokens: 6773
 
class htb 10:3 parent 10:1 leaf 23: = prio 1 rate=20 384Kbit ceil 512Kbit burst 2Kb/8 mpu 0b cburst 2Kb/8 mpu 0b quantum 4915 = level=20 0
 Sent 28120752 bytes 18590 pkts (dropped 0, overlimits=20 8155)
 rate 28117bps 18pps
 lended: 18183 borrowed: 407 = giants:=20 0 injects: 0
 tokens: 212 ctokens: 6773
 
class htb 10:5 parent 10:1 leaf 25: = prio 2 rate=20 64Kbit ceil 512Kbit burst 1680b/8 mpu 0b cburst 2Kb/8 mpu 0b quantum 819 = level=20 0
 Sent 31914478 bytes 21111 pkts (dropped 0, overlimits=20 1)
 rate 26828bps 17pps
 lended: 5510 borrowed: 15601 = giants: 0=20 injects: 0
 tokens: -115091 ctokens: 9288
 
Regards,
Matt.
------=_NextPart_000_001B_01C3BE7A.3EBA7340-- _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/