Re: [LARTC] big problem with HTB/CBQ and CPU for more than 1.700

[VladSun <vladsun@relef.net>]

Pablo Fernandes Yahoo написа:
>
> Hello,
>
> have HTB „rules“ in 4 different ISPs and i control for each customer 
> this way:
>
> Flush and 1:0 class
>
> tc qdisc del dev eth0 root
>
> tc qdisc add dev eth0 root handle 1:0 htb
>
> tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100mbit
>
> tc qdisc del dev eth1 root
>
> tc qdisc add dev eth1 root handle 1:0 htb
>
> tc class add dev eth1 parent 1:0 classid 1:1 htb rate 100mbit
>
> Upload and Download: user1
>
> tc class add dev eth0 parent 1:1 classid 1:5 htb rate 150kbit ceil 150kbit
>
> tc qdisc add dev eth0 parent 1:5 handle 5: sfq perturb 10
>
> tc class add dev eth1 parent 1:1 classid 1:5 htb rate 50kbit ceil 50kbit
>
> tc qdisc add dev eth1 parent 1:5 handle 5: sfq perturb 10
>
> iptables -t mangle -A POSTROUTING --dest x.x.x.x -o eth0 -j CLASSIFY 
> --set-class 1:5
>
> iptables -t mangle -A FORWARD --src x.x.x.x -o eth1 -j CLASSIFY 
> --set-class 1:5
>
> Upload and Download: user2
>
> tc class add dev eth0 parent 1:1 classid 1:8 htb rate 150kbit ceil 150kbit
>
> tc qdisc add dev eth0 parent 1:8 handle 8: sfq perturb 10
>
> tc class add dev eth1 parent 1:1 classid 1:8 htb rate 50kbit ceil 50kbit
>
> tc qdisc add dev eth1 parent 1:8 handle 8: sfq perturb 10
>
> iptables -t mangle -A POSTROUTING --dest y.y.y.y -o eth0 -j CLASSIFY 
> --set-class 1:8
>
> iptables -t mangle -A FORWARD --src y.y.y.y -o eth1 -j CLASSIFY 
> --set-class 1:8
>
> (…)
>
> This rules works fine, but just for less than 1.700 customers. More 
> than 1.700 customers, i have my load avarage in the sky and Ksoftirqd 
> process (top information) in 100% fulltime. I don’t know why. I used 
> to use CBQ instead HTB because i had the same problem and Ron (a guy 
> in this list) gave this rules and told me that he uses this for more 
> than 3.000 customers. I tested it in more than 7 different computers 
> (but the same hadware specifications) and i had the same problem with 
> either CBQ or HTB rules. The computers that i have are all of them 
> DELL PowerEdge 1850. I will put some hardware iformations here:
>
> top
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 3 root 39 19 0 0 0 R 100 0.0 5316:20 ksoftirqd/0
>
> [root@fw ~]# uptime
>
> 10:38:11 up 161 days, 17:21, 3 users, load average: 1.58, 1.65, 1.51 
> (unfortunately when i took this, the load average was „pretty good“, 
> but minutes ago, it was more than 11.0
>
> [root@fw ~]# lspci
>
> 00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub 
> (rev 09)
>
> 00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express 
> Port A (rev 09)
>
> 00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port B 
> (rev 09)
>
> 00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1 (rev 09)
>
> 00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev 09)
>
> 00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB 
> UHCI Controller #1 (rev 02)
>
> 00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB 
> UHCI Controller #2 (rev 02)
>
> 00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB 
> UHCI Controller #3 (rev 02)
>
> 00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 
> EHCI Controller (rev 02)
>
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)
>
> 00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC 
> Interface Bridge (rev 02)
>
> 00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE 
> Controller (rev 02)
>
> 01:00.0 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor 
> (A-Segment Bridge) (rev 06)
>
> 01:00.2 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor 
> (B-Segment Bridge) (rev 06)
>
> 02:0c.0 Ethernet controller: Intel Corporation 82545GM Gigabit 
> Ethernet Controller (rev 04)
>
> 02:0e.0 RAID bus controller: Dell PowerEdge Expandable RAID controller 
> 4 (rev 06)
>
> 03:0b.0 Ethernet controller: Intel Corporation 82545GM Gigabit 
> Ethernet Controller (rev 04)
>
> 05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI 
> Bridge A (rev 09)
>
> 05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI 
> Bridge B (rev 09)
>
> 06:07.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit 
> Ethernet Controller (rev 05)
>
> 07:08.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit 
> Ethernet Controller (rev 05)
>
> 09:0d.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 
> QY [Radeon 7000/VE]
>
> [root@fw ~]# free -m
>
> total used free shared buffers cached
>
> Mem: 2021 1479 542 0 400 654
>
> -/+ buffers/cache: 424 1597
>
> Swap: 1027 0 1027
>
> [root@fw ~]# cat /proc/cpuinfo
>
> processor : 0
>
> vendor_id : GenuineIntel
>
> cpu family : 15
>
> model : 4
>
> model name : Intel(R) Xeon(TM) CPU 3.00GHz
>
> stepping : 3
>
> cpu MHz : 2992.674
>
> cache size : 2048 KB
>
> physical id : 0
>
> siblings : 2
>
> core id : 0
>
> cpu cores : 1
>
> fdiv_bug : no
>
> hlt_bug : no
>
> f00f_bug : no
>
> coma_bug : no
>
> fpu : yes
>
> fpu_exception : yes
>
> cpuid level : 5
>
> wp : yes
>
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
> pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm 
> constant_tsc pni monitor ds_cpl cid cx16 xtpr
>
> bogomips : 5990.78
>
> processor : 1
>
> vendor_id : GenuineIntel
>
> cpu family : 15
>
> model : 4
>
> model name : Intel(R) Xeon(TM) CPU 3.00GHz
>
> stepping : 3
>
> cpu MHz : 2992.674
>
> cache size : 2048 KB
>
> physical id : 0
>
> siblings : 2
>
> core id : 0
>
> cpu cores : 1
>
> fdiv_bug : no
>
> hlt_bug : no
>
> f00f_bug : no
>
> coma_bug : no
>
> fpu : yes
>
> fpu_exception : yes
>
> cpuid level : 5
>
> wp : yes
>
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
> pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm 
> constant_tsc pni monitor ds_cpl cid cx16 xtpr
>
> bogomips : 5985.13
>
> Any help/Tipp/hint will be very welcome.
>
> Thanks in Advance!
>
> Pablo Fernandes
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>   
You may find this:
http://openfmi.net/frs/download.php/410/IPCLASSIFY.zip
useful
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc