From mboxrd@z Thu Jan 1 00:00:00 1970 From: Stoimen Gerenski Date: Sun, 27 May 2007 22:46:41 +0000 Subject: [LARTC] big problem with HTB/CBQ and CPU for more than 1.700 Message-Id: <1180306001.465a0a518d9ce@mail.bulsattv.com> List-Id: References: <20070526135435.C96F540DB@outpost.ds9a.nl> In-Reply-To: <20070526135435.C96F540DB@outpost.ds9a.nl> MIME-Version: 1.0 Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: quoted-printable To: lartc@vger.kernel.org Hello everybody, I have a problem which is similar: 450 customers with 450 HTB classes and their corresponding 450 filters in a subclass (1:3) of the root qdisc (PRIO) of the interface. The problem is manifesting at times, when I try to ping a host behind the r= outer from a host before the router, the latency becomes 1-1,5 msec. On the machine is running also iptables firewall with a bunch of rules for dropping/accepting/natting specific traffic, plus routing about 30 Mbits/se= c. When I remove the HTB qdisc, the latency is normal, 0,3-0,4 msec. Anyone has an idea what could cause this? Any input much appreciated! Regards, Stoimen -------------- Pablo, Here we have HTB being used for more than 10.000 customers. The =20 difference, is that we use tc and u32 filters to classify the packets.. I use the same Dell PE 1850, but I have two Quad-Core Xeon (1.86GHz) =20 on it :) # uptime 13:18:08 up 16 days, 12:32, 1 user, load average: 0.02, 0.02, 0.00 mpstat says: 01:19:11 PM CPU %user %nice %sys %iowait %irq %soft %=20 steal %idle intr/s 01:19:13 PM all 0.00 0.00 0.00 0.00 0.57 13.81 =20 0.00 85.61 10568.88 And as you can see.. the use of cpu is not that big.. On May 26, 2007, at 6:54 AM, Pablo Fernandes Yahoo wrote: > Hello, > > > > have HTB =84rules=93 in 4 different ISPs and i control for each =20 > customer this way: > > > > Flush and 1:0 class > > tc qdisc del dev eth0 root > > tc qdisc add dev eth0 root handle 1:0 htb > > tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100mbit > > tc qdisc del dev eth1 root > > tc qdisc add dev eth1 root handle 1:0 htb > > tc class add dev eth1 parent 1:0 classid 1:1 htb rate 100mbit > > > > Upload and Download: user1 > > tc class add dev eth0 parent 1:1 classid 1:5 htb rate 150kbit ceil =20 > 150kbit > > tc qdisc add dev eth0 parent 1:5 handle 5: sfq perturb 10 > > tc class add dev eth1 parent 1:1 classid 1:5 htb rate 50kbit ceil =20 > 50kbit > > tc qdisc add dev eth1 parent 1:5 handle 5: sfq perturb 10 > > iptables -t mangle -A POSTROUTING --dest x.x.x.x -o eth0 -j =20 > CLASSIFY --set-class 1:5 > > iptables -t mangle -A FORWARD --src x.x.x.x -o eth1 -j CLASSIFY --=20 > set-class 1:5 > > > > Upload and Download: user2 > > tc class add dev eth0 parent 1:1 classid 1:8 htb rate 150kbit ceil =20 > 150kbit > > tc qdisc add dev eth0 parent 1:8 handle 8: sfq perturb 10 > > tc class add dev eth1 parent 1:1 classid 1:8 htb rate 50kbit ceil =20 > 50kbit > > tc qdisc add dev eth1 parent 1:8 handle 8: sfq perturb 10 > > iptables -t mangle -A POSTROUTING --dest y.y.y.y -o eth0 -j =20 > CLASSIFY --set-class 1:8 > > iptables -t mangle -A FORWARD --src y.y.y.y -o eth1 -j CLASSIFY --=20 > set-class 1:8 > > > > (=85) > > > > This rules works fine, but just for less than 1.700 customers. More =20 > than 1.700 customers, i have my load avarage in the sky and =20 > Ksoftirqd process (top information) in 100% fulltime. I don=92t know =20 > why. I used to use CBQ instead HTB because i had the same problem =20 > and Ron (a guy in this list) gave this rules and told me that he =20 > uses this for more than 3.000 customers. I tested it in more than 7 =20 > different computers (but the same hadware specifications) and i had =20 > the same problem with either CBQ or HTB rules. The computers that i =20 > have are all of them DELL PowerEdge 1850. I will put some hardware =20 > iformations here: > > > > top > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 3 root 39 19 0 0 0 R 100 0.0 5316:20 =20 > ksoftirqd/0 > > > > [root at fw ~]# uptime > > 10:38:11 up 161 days, 17:21, 3 users, load average: 1.58, 1.65, =20 > 1.51 (unfortunately when i took this, the load average was =20 > =84pretty good=93, but minutes ago, it was more than 11.0 > > > > [root at fw ~]# lspci > > 00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub =20 > (rev 09) > > 00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express =20 > Port A (rev 09) > > 00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port =20 > B (rev 09) > > 00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1 =20 > (rev 09) > > 00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev =20 > 09) > > 00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) =20 > USB UHCI Controller #1 (rev 02) > > 00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) =20 > USB UHCI Controller #2 (rev 02) > > 00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) =20 > USB UHCI Controller #3 (rev 02) > > 00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) =20 > USB2 EHCI Controller (rev 02) > > 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2) > > 00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC =20 > Interface Bridge (rev 02) > > 00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) =20 > IDE Controller (rev 02) > > 01:00.0 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor =20 > (A-Segment Bridge) (rev 06) > > 01:00.2 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor =20 > (B-Segment Bridge) (rev 06) > > 02:0c.0 Ethernet controller: Intel Corporation 82545GM Gigabit =20 > Ethernet Controller (rev 04) > > 02:0e.0 RAID bus controller: Dell PowerEdge Expandable RAID =20 > controller 4 (rev 06) > > 03:0b.0 Ethernet controller: Intel Corporation 82545GM Gigabit =20 > Ethernet Controller (rev 04) > > 05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI =20 > Bridge A (rev 09) > > 05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI =20 > Bridge B (rev 09) > > 06:07.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit =20 > Ethernet Controller (rev 05) > > 07:08.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit =20 > Ethernet Controller (rev 05) > > 09:0d.0 VGA compatible controller: ATI Technologies Inc Radeon =20 > RV100 QY [Radeon 7000/VE] > > > > [root at fw ~]# free -m > > total used free shared buffers =20 > cached > > Mem: 2021 1479 542 0 =20 > 400 654 > > -/+ buffers/cache: 424 1597 > > Swap: 1027 0 1027 > > > > [root at fw ~]# cat /proc/cpuinfo > > processor : 0 > > vendor_id : GenuineIntel > > cpu family : 15 > > model : 4 > > model name : Intel(R) Xeon(TM) CPU 3.00GHz > > stepping : 3 > > cpu MHz : 2992.674 > > cache size : 2048 KB > > physical id : 0 > > siblings : 2 > > core id : 0 > > cpu cores : 1 > > fdiv_bug : no > > hlt_bug : no > > f00f_bug : no > > coma_bug : no > > fpu : yes > > fpu_exception : yes > > cpuid level : 5 > > wp : yes > > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr =20 > pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm =20 > pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr > > bogomips : 5990.78 > > > > processor : 1 > > vendor_id : GenuineIntel > > cpu family : 15 > > model : 4 > > model name : Intel(R) Xeon(TM) CPU 3.00GHz > > stepping : 3 > > cpu MHz : 2992.674 > > cache size : 2048 KB > > physical id : 0 > > siblings : 2 > > core id : 0 > > cpu cores : 1 > > fdiv_bug : no > > hlt_bug : no > > f00f_bug : no > > coma_bug : no > > fpu : yes > > fpu_exception : yes > > cpuid level : 5 > > wp : yes > > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr =20 > pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm =20 > pbe nx lm constant_tsc pni monitor ds_cpl cid cx16 xtpr > > bogomips : 5985.13 > > > > > > Any help/Tipp/hint will be very welcome. > > > > Thanks in Advance! > > > > Pablo Fernandes --------------------------------------------------- Webmail of Bulsat Ltd. at http://mail.bulsattv.com/ _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc