[LARTC] Fw: HTB - Really Big problem

[LARTC] Fw: HTB - Really Big problem

[mjoachimiak]

[-- Attachment #1: Type: text/plain, Size: 10153 bytes --]


----- Original Message ----- 
From: Michał Joachimiak 
To: lartc@mailman.ds9a.nl 
Sent: Tuesday, July 20, 2004 3:45 PM
Subject: HTB - Really Big problem


Hello everybody!
Since week i dig lists and www and can't find solution for my problem.
I'm using HTB 3.13 kernel 2-4-25 smp iptables 1.2.9.
I've got situation like this:

LAN------Linux Box(routing only)------- Linux Box (HTB)--------------Hardware Router(say:HD)------Internet
When I start HTB it takes about 5 min. to start working and it works...
within this 5min starting i can't ping HD and after about 5min I start pinging. It works like this for cuple of hours, then something strange is happening. Ping stops, www doesn't work but radio (36kbps) works. There is no ping at all for about 1min and it starts pinging for about 2-3min and it stops for 1-2min and so on.....
When I stop HTB ping starts. It's look like HTB is filled to much(sorry for my english :/)
All my children classes rate doesn't exceeds root classes. I have 50 classes on 900kbit-10kbit(for default class) - downstream and 800-10kbit(for default)  up.
I shape bandwidth matching ip. r2q is set to 1 . no erros during doing shape script. 
I'm attaching this script
in ip1 ip2 and so on are  files where are written ip's for C classes.

I'm including my script.
---------------------------------------------------cut here -----------------------------------------------------------------------------------
#!/bin/bash
#x=$[100/20]
#echo "$1" > /skrytpy/status
rxmax=900 #WAN max transfer -down (physically it is 960kbit/960kbit)
kbit=kbit
rxmaxluser=250
txmax=800 #WAN max transfer - up 
txmaxluser=100
#counting users
# ip1 file is like this:
# 11 #Somebody
# 23 #Somebody II
#EOF
licznik=0
for   x in $(awk '{ print $1 }' /skrytpy/ip1); do
licznik=$[$licznik+1]
done
for   x in $(awk '{ print $1 }' /skrytpy/ip2); do
licznik=$[$licznik+1]
done
for   x in $(awk '{ print $1 }' /skrytpy/ip3); do
licznik=$[$licznik+1]
done
for   x in $(awk '{ print $1 }' /skrytpy/ip128); do
licznik=$[$licznik+1]
done
for   x in $(awk '{ print $1 }' /skrytpy/ip4); do
licznik=$[$licznik+1]
done
for   x in $(awk '{ print $1 }' /skrytpy/ipzew); do
licznik=$[$licznik+1]
done
for   x in $(awk '{ print $1 }' /skrytpy/ip6); do
licznik=$[$licznik+1]
done

#Server
licznik=$[$licznik+1]
#plus router
licznik=$[$licznik+1]

echo number of users to $licznik
#counting rate
rx1=$[$rxmax-10]
tx1=$[$txmax-10] # dla klasy domyslnej
rxmin=$[$rx1/$licznik]
txmin=$[$tx1/$licznik]
echo rx $rxmin tx $txmin
#echo $rxmin
#root classes
#rx
tc qdisc del root dev eth1
tc qdisc add dev eth1 root handle 1:0 htb r2q 1 default 2
tc class add dev eth1 parent 1:0 classid 1:1 htb rate $rxmax$kbit ceil $rxmax$kbit
#tx
tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 2:0 htb r2q 1 default 4
tc class add dev eth0 parent 2:0 classid 2:1 htb rate $txmax$kbit ceil $txmax$kbit
#r="$rxmax$kbit"
#default classes
#rx
tc class add dev eth1 parent 1:1 classid 1:2 htb rate 10kbit ceil 10kbit 
#tx
tc class add dev eth0 parent 2:1 classid 2:4 htb rate 10kbit ceil 10kbit
#siec 1.0
siec=1
for ip in $(awk '{ print $1 }' /skrytpy/ip1); do
echo -n "$ip "
#rx
tc class add dev eth1 parent 1:1 classid 1:$ip htb rate $rxmin$kbit ceil $rxmaxluser$kbit
tc filter add dev eth1 protocol ip parent 1:0 u32 match ip dst 192.168.$siec.$ip flowid 1:$ip
tc qdisc add dev eth1 parent 1:$ip handle $ip:0 sfq perturb 10 
#tx
#marking packets
iptables -t mangle -A MYSHAPER-OUT -p tcp -s 192.168.$siec.$ip -j MARK --set-mark $ip
#tc
tc class add dev eth0 parent 2:1 classid 2:$ip htb rate $txmin$kbit ceil $txmaxluser$kbit
#filtering by mark
tc filter add dev eth0 protocol ip preference 1 parent 2:0 handle $ip fw flowid 2:$ip
tc qdisc add dev eth0 parent 2:$ip handle $ip:1 sfq perturb 10
done
echo ""
#siec 2.0
siec=2
for ip in $(awk '{ print $1 }' /skrytpy/ip2); do
echo -n "$ip " 
tc class add dev eth1 parent 1:1 classid 1:$ip htb rate $rxmin$kbit ceil $rxmaxluser$kbit
tc filter add dev eth1 protocol ip parent 1:0 u32 match ip dst 192.168.$siec.$ip flowid 1:$ip
tc qdisc add dev eth1 parent 1:$ip handle $ip:0 sfq perturb 10 
#mark
iptables -t mangle -A MYSHAPER-OUT -p tcp -s 192.168.$siec.$ip -j MARK --set-mark $ip
#tc
tc class add dev eth0 parent 2:1 classid 2:$ip htb rate $txmin$kbit ceil $txmaxluser$kbit
tc qdisc add dev eth0 parent 2:$ip handle $ip:1 sfq perturb 10
#filtrowanie po marku
tc filter add dev eth0 protocol ip preference 1 parent 2:0 handle $ip fw flowid 2:$ip
done
#siec 3.0
siec=3
echo " " 
for ip in $(awk '{ print $1 }' /skrytpy/ip3); do
echo -n "$ip "
tc class add dev eth1 parent 1:1 classid 1:$ip htb rate $rxmin$kbit ceil 128kbit
tc filter add dev eth1 protocol ip parent 1:0 u32 match ip dst 192.168.$siec.$ip flowid 1:$ip
tc qdisc add dev eth1 parent 1:$ip handle $ip:0 sfq perturb 10 
#mark
iptables -t mangle -A MYSHAPER-OUT -p tcp -s 192.168.$siec.$ip -j MARK --set-mark $ip
#tc
tc class add dev eth0 parent 2:1 classid 2:$ip htb rate $txmin$kbit ceil 64kbit
tc qdisc add dev eth0 parent 2:$ip handle $ip:1 sfq perturb 10
#filtrowanie po marku
tc filter add dev eth0 protocol ip preference 1 parent 2:0 handle $ip fw flowid 2:$ip
done
echo " "
#Siec 4
siec=4
for ip in $(awk '{ print $1 }' /skrytpy/ip4); do
echo -n "$ip "
tc class add dev eth1 parent 1:1 classid 1:$ip htb rate $rxmin$kbit ceil 128kbit
tc filter add dev eth1 protocol ip parent 1:0 u32 match ip dst 192.168.$siec.$ip flowid 1:$ip
tc qdisc add dev eth1 parent 1:$ip handle $ip:0 sfq perturb 10 
#mark
iptables -t mangle -A MYSHAPER-OUT -p tcp -s 192.168.$siec.$ip -j MARK --set-mark $ip
#tc
tc class add dev eth0 parent 2:1 classid 2:$ip htb rate $txmin$kbit ceil 64kbit
tc qdisc add dev eth0 parent 2:$ip handle $ip:1 sfq perturb 10
#filtrowanie po marku
tc filter add dev eth0 protocol ip preference 1 parent 2:0 handle $ip fw flowid 2:$ip
done
echo " "
#walas
siec=6
for ip in $(awk '{ print $1 }' /skrytpy/ip6); do
echo -n "$ip "
tc class add dev eth1 parent 1:1 classid 1:$ip htb rate $rxmin$kbit ceil 128kbit
tc filter add dev eth1 protocol ip parent 1:0 u32 match ip dst 192.168.$siec.$ip flowid 1:$ip
tc qdisc add dev eth1 parent 1:$ip handle $ip:0 sfq perturb 10 
#mark
iptables -t mangle -A MYSHAPER-OUT -p tcp -s 192.168.$siec.$ip -j MARK --set-mark $ip
#tc
tc class add dev eth0 parent 2:1 classid 2:$ip htb rate $txmin$kbit ceil 64kbit
tc qdisc add dev eth0 parent 2:$ip handle $ip:1 sfq perturb 10
#filtrowanie po marku
tc filter add dev eth0 protocol ip preference 1 parent 2:0 handle $ip fw flowid 2:$ip
done
echo " "

#Siec ip zewnetrznych
siec=62.87.193
for ip in $(awk '{ print $1 }' /skrytpy/ipzew); do
echo -n "$ip "
tc class add dev eth1 parent 1:1 classid 1:$ip htb rate $rxmin$kbit ceil $rxmaxluser$kbit
tc filter add dev eth1 protocol ip parent 1:0 u32 match ip dst $siec.$ip flowid 1:$ip
tc qdisc add dev eth1 parent 1:$ip handle $ip:0 sfq perturb 10 
#mark
iptables -t mangle -A MYSHAPER-OUT -p tcp -s $siec.$ip -j MARK --set-mark $ip
#tc
tc class add dev eth0 parent 2:1 classid 2:$ip htb rate $txmin$kbit ceil $txmaxluser$kbit
tc qdisc add dev eth0 parent 2:$ip handle $ip:1 sfq perturb 10
#filtrowanie po marku
tc filter add dev eth0 protocol ip preference 1 parent 2:0 handle $ip fw flowid 2:$ip
done
echo " "

#Wyjatki z ip128
siec=1
ip=54
echo $ip
tc class add dev eth1 parent 1:1 classid 1:$ip htb rate $rxmin$kbit ceil 64kbit
tc filter add dev eth1 protocol ip parent 1:0 u32 match ip dst 192.168.$siec.$ip flowid 1:$ip
tc qdisc add dev eth1 parent 1:$ip handle $ip:0 sfq perturb 10 
iptables -t mangle -A MYSHAPER-OUT -p tcp -s 192.168.$siec.$ip -j MARK --set-mark $ip
#tc
tc class add dev eth0 parent 2:1 classid 2:$ip htb rate $txmin$kbit ceil 20kbit
tc qdisc add dev eth0 parent 2:$ip handle $ip:1 sfq perturb 10
#filtrowanie po marku
tc filter add dev eth0 protocol ip preference 1 parent 2:0 handle $ip fw flowid 2:$ip

siec=2
ip=53
echo $ip
tc class add dev eth1 parent 1:1 classid 1:$ip htb rate $rxmin$kbit ceil 128kbit
tc filter add dev eth1 protocol ip parent 1:0 u32 match ip dst 192.168.$siec.$ip flowid 1:$ip
tc qdisc add dev eth1 parent 1:$ip handle $ip:0 sfq perturb 10 
#tx
iptables -t mangle -A MYSHAPER-OUT -p tcp -s 192.168.$siec.$ip -j MARK --set-mark $ip
#tc
tc class add dev eth0 parent 2:1 classid 2:$ip htb rate $txmin$kbit ceil 64kbit
tc qdisc add dev eth0 parent 2:$ip handle $ip:1 sfq perturb 10
#filtrowanie po marku
tc filter add dev eth0 protocol ip preference 1 parent 2:0 handle $ip fw flowid 2:$ip

siec=4
ip=138
echo $ip
tc class add dev eth1 parent 1:1 classid 1:$ip htb rate $rxmin$kbit ceil 400kbit
tc filter add dev eth1 protocol ip parent 1:0 u32 match ip dst 62.87.193.$ip flowid 1:$ip
tc qdisc add dev eth1 parent 1:$ip handle $ip:0 sfq perturb 10 
#tx
iptables -t mangle -A MYSHAPER-OUT -p tcp -s 62.87.193.$ip -j MARK --set-mark $ip
#tc
tc class add dev eth0 parent 2:1 classid 2:$ip htb rate $txmin$kbit ceil 256kbit
tc qdisc add dev eth0 parent 2:$ip handle $ip:1 sfq perturb 10
#filtrowanie po marku
tc filter add dev eth0 protocol ip preference 1 parent 2:0 handle $ip fw flowid 2:$ip

#Serwer
siec=5
ip=3
echo "5.2/3" #nadle x:3 because x:3 are saved for default classess
tc class add dev eth1 parent 1:1 classid 1:$ip htb rate $rxmin$kbit ceil 200kbit
tc filter add dev eth1 protocol ip parent 1:0 u32 match ip dst 192.168.5.2 flowid 1:$ip
tc qdisc add dev eth1 parent 1:$ip handle $ip:0 sfq perturb 10 
#tx
iptables -t mangle -A MYSHAPER-OUT -p tcp -s 192.168.5.2 -j MARK --set-mark $ip
tc class add dev eth0 parent 2:1 classid 2:$ip htb rate $txmin$kbit ceil 256kbit
tc qdisc add dev eth0 parent 2:$ip handle $ip:1 sfq perturb 10
#filtrowanie po marku
tc filter add dev eth0 protocol ip preference 1 parent 2:0 handle $ip fw flowid 2:$ip
--------------------------------------------------------------------cut here ------------------------------------------------------------------

Please help - I have no idea why it's not works.

[-- Attachment #2: Type: text/html, Size: 12790 bytes --]

Re: [LARTC] Fw: HTB - Really Big problem

[Ed Wildgoose]

> Please help - I have no idea why it's not works.


What do the tc stats look like?
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Fw: HTB - Really Big problem

[mjoachimiak]

Here are the stats. If you want full stats please write then i'll send as
attachment but it is 0,5MB.
tc qdisc show
qdisc sfq 3: dev eth0 quantum 1514b perturb 10sec
qdisc sfq 138: dev eth0 quantum 1514b perturb 10sec
qdisc sfq 53: dev eth0 quantum 1514b perturb 10sec
qdisc sfq 54: dev eth0 quantum 1514b perturb 10sec
qdisc sfq 140: dev eth0 quantum 1514b perturb 10sec
qdisc sfq 137: dev eth0 quantum 1514b perturb 10sec
......
qdisc sfq 11: dev eth0 quantum 1514b perturb 10sec
qdisc sfq 10: dev eth0 quantum 1514b perturb 10sec
qdisc sfq 9: dev eth0 quantum 1514b perturb 10sec
qdisc htb 2: dev eth0 r2q 1 default 4 direct_packets_stat 0
qdisc sfq 3: dev eth1 quantum 1514b perturb 10sec
qdisc sfq 138: dev eth1 quantum 1514b perturb 10sec
.....
qdisc sfq 10: dev eth1 quantum 1514b perturb 10sec
qdisc sfq 9: dev eth1 quantum 1514b perturb 10sec
qdisc htb 1: dev eth1 r2q 1 default 2 direct_packets_stat 2
EOF

tc filter show dev eth1:

filter parent 1: protocol ip pref 49151 u32
filter parent 1: protocol ip pref 49151 u32 fh 831: ht divisor 1
filter parent 1: protocol ip pref 49151 u32 fh 831::800 order 2048 key ht
831 bkt 0 flowid 1:3
  match c0a80502/ffffffff at 16
filter parent 1: protocol ip pref 49151 u32 fh 830: ht divisor 1
filter parent 1: protocol ip pref 49151 u32 fh 830::800 order 2048 key ht
830 bkt 0 flowid 1:138
  match 3e57c18a/ffffffff at 16
filter parent 1: protocol ip pref 49151 u32 fh 82f: ht divisor 1
filter parent 1: protocol ip pref 49151 u32 fh 82f::800 order 2048 key ht
82f bkt 0 flowid 1:53
  match c0a80235/ffffffff at 16
filter parent 1: protocol ip pref 49151 u32 fh 82e: ht divisor 1
filter parent 1: protocol ip pref 49151 u32 fh 82e::800 order 2048 key
..... #many lines
filter parent 1: protocol ip pref 49152 u32 fh 802: ht divisor 1
filter parent 1: protocol ip pref 49152 u32 fh 802::800 order 2048 key ht
802 bkt 0 flowid 1:11
  match c0a8010b/ffffffff at 16
filter parent 1: protocol ip pref 49152 u32 fh 801: ht divisor 1
filter parent 1: protocol ip pref 49152 u32 fh 801::800 order 2048 key ht
801 bkt 0 flowid 1:10
  match c0a8010a/ffffffff at 16
filter parent 1: protocol ip pref 49152 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 49152 u32 fh 800::800 order 2048 key ht
800 bkt 0 flowid 1:9
  match c0a80109/ffffffff at 16

tc filter show dev eth0:

tc filter show dev eth0
filter parent 2: protocol ip pref 1 fw
filter parent 2: protocol ip pref 1 fw handle 0x3 classid 2:3
filter parent 2: protocol ip pref 1 fw handle 0x8 classid 2:8
filter parent 2: protocol ip pref 1 fw handle 0x9 classid 2:9
filter parent 2: protocol ip pref 1 fw handle 0xa classid 2:10
filter parent 2: protocol ip pref 1 fw handle 0xb classid 2:11
filter parent 2: protocol ip pref 1 fw handle 0xc classid 2:12
filter parent 2: protocol ip pref 1 fw handle 0xf classid 2:15
filter parent 2: protocol ip pref 1 fw handle 0x11 classid 2:17
...#many lines
filter parent 2: protocol ip pref 1 fw handle 0x97 classid 2:151
filter parent 2: protocol ip pref 1 fw handle 0x98 classid 2:152
filter parent 2: protocol ip pref 1 fw handle 0xbd classid 2:189


tc class show dev eth1:
class htb 1:11 parent 1:1 leaf 11: prio 0 rate 17Kbit ceil 250Kbit burst
1620b cburst 1919b
class htb 1:22 parent 1:1 leaf 22: prio 0 rate 17Kbit ceil 250Kbit burst
1620b
...
class htb 1:189 parent 1:1 leaf 189: prio 0 rate 17Kbit ceil 128Kbit burst
1620b cburst 1762b
class htb 1:1 root rate 900Kbit ceil 900Kbit burst 2751b cburst 2751b
class htb 1:10 parent 1:1 leaf 10: prio 0 rate 17Kbit ceil 250Kbit burst
1620b cburst 1919b
class htb 1:23 parent 1:1 leaf 23: prio 0 rate 17Kbit ceil 250Kbit burst
1620b cburst 1919b
class htb 1:32 parent 1:1 leaf 32: prio 0 rate 17Kbit ceil 250Kbit burst
1620b cburst 1919b
class htb 1:54 parent 1:1 leaf 54: prio 0 rate 17Kbit ceil 64Kbit burst
1620b cburst 1680b
class htb 1:2 parent 1:1 prio 0 rate 10Kbit ceil 10Kbit burst 1611b cburst
1611b
class htb 1:20 parent 1:1 leaf 20: prio 0 rate 17Kbit ceil 250Kbit burst
1620b cburst 1919b
class htb 1:64 parent 1:1 leaf 64: prio 0 rate 17Kbit ceil 250Kbit burst
1620b
....
class htb 1:59 parent 1:1 leaf 59: prio 0 rate 17Kbit ceil 250Kbit burst
1620b cburst 1919b
class htb 1:58 parent 1:1 leaf 58: prio 0 rate 17Kbit ceil 250Kbit burst
1620b cburst 1919b


tc class show dev eth0:
class htb 2:11 parent 2:1 leaf 11: prio 0 rate 15Kbit ceil 100Kbit burst
1618b cburst 1727b
class htb 2:22 parent 2:1 leaf 22: prio 0 rate 15Kbit ceil 100Kbit burst
1618b cburst 1727b
....
class htb 2:1 root rate 800Kbit ceil 800Kbit burst 2623b cburst 2623b
class htb 2:10 parent 2:1 leaf 10: prio 0 rate 15Kbit ceil 100Kbit burst
1618b cburst 1727b
.....
class htb 2:3 parent 2:1 leaf 3: prio 0 rate 15Kbit ceil 256Kbit burst 1618b
cburst 1926b
class htb 2:4 parent 2:1 prio 0 rate 10Kbit ceil 10Kbit burst 1611b cburst
1611b


> What do the tc stats look like?
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/