TC-HTB issue : low throughput

TC-HTB issue : low throughput

[Jean Tourrilhes]

	Hi,

	I'm playing with a TC-HTB. I'm noticing that the throughput is
low. As the LARTC mailing list seems to be dead, I would welcome
suggestions...
	The example below may seem contrived, but I reduced it to
a simple testcase to make it easier to reproduce.

System :
------
	Debian Lenny with 2.6.30.9
	Quad Core 2.5 GHz Q9300
	Intel 1Gb/s NIC, e1000e driver

Setup :
-----
# brctl addbr br0
# brctl addif br0 eth5.33
# brctl addif br0 eth6.34
# brctl addif br0 eth7.38

#iptables -A FORWARD -t mangle -s 10.10.32.38 -j CLASSIFY --set-class 1:2

#tc qdisc add dev eth6.34 root handle 1: htb default 3
#tc class add dev eth6.34 parent 1: classid 1:1 htb rate 1000mbit ceil 1000mbit
#tc class add dev eth6.34 parent 1:1 classid 1:2 htb rate 3kbit ceil 1000mbit prio 1
#tc class add dev eth6.34 parent 1:1 classid 1:3 htb rate 3kbit ceil 1000mbit prio 2

Topology :
--------
Four PCs with Linux :
10.10.10.32 on br0
10.10.10.33 connected to eth5.33 (sender)
10.10.10.38 connected to eth7.38 (sender)
10.10.10.34 connected to eth6.34 (receiver)

Behaviour :
---------
	If I set only qdisc 1: and class 1:1, but *NOT* 1:2 and 1:3 :
	Each host independantly : ~935 Mb/s
	Both host together, for 10.10.10.38 : ~527 Mb/s ;
	Both host together, for 10.10.10.33 : ~443 Mb/s

	If I add classes 1:2 and 1:3 :
	Each host independantly : ~170 Mb/s.
	Both host together, for 10.10.10.38 : ~106 Mb/s ;
	Both host together, for 10.10.10.33 : ~135 Mb/s

	So, not only performance did drop significantely, but
prioritisation did not happen as expected.

Weird detail :
------------
	I've noticed that /sbin/tc calculates a very low burst
value. This is due to the content of /proc/net/psched. I'm wondering
if the burst calculation is what causes the issue here.
	However, I tried with "burst 50kb" and saw no difference...

Dump of TC
----------
# tc -s class show dev eth6.34
class htb 1:1 root rate 1000Mbit ceil 1000Mbit burst 1375b cburst 1375b 
 Sent 770653596 bytes 509070 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 82776Kbit 6835pps backlog 0b 0p requeues 0 
 lended: 509026 borrowed: 0 giants: 0
 tokens: 11 ctokens: 11

class htb 1:2 parent 1:1 prio 1 rate 3000bit ceil 1000Mbit burst 1599b cburst 13
75b 
 Sent 369190389 bytes 243876 pkt (dropped 3239, overlimits 0 requeues 0) 
 rate 44781Kbit 3698pps backlog 0b 0p requeues 0 
 lended: 20 borrowed: 243856 giants: 0
 tokens: -1384355 ctokens: 11

class htb 1:3 parent 1:1 prio 2 rate 3000bit ceil 1000Mbit burst 1599b cburst 13
75b 
 Sent 401463207 bytes 265194 pkt (dropped 3005, overlimits 0 requeues 0) 
 rate 37996Kbit 3137pps backlog 0b 0p requeues 0 
 lended: 24 borrowed: 265170 giants: 0
 tokens: -3719389 ctokens: 11

# tc -s qdisc show dev eth6.34
qdisc htb 1: root r2q 10 default 3 direct_packets_stat 0
 Sent 770653852 bytes 509071 pkt (dropped 6244, overlimits 25973 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 


	Thanks in advance for any help...

	Jean

Re: TC-HTB issue : low throughput

[Jarek Poplawski]

Jean Tourrilhes wrote, On 11/07/2009 03:43 AM:

> 	Hi,
> 
> 	I'm playing with a TC-HTB. I'm noticing that the throughput is
> low. As the LARTC mailing list seems to be dead, I would welcome
> suggestions...
> 	The example below may seem contrived, but I reduced it to
> a simple testcase to make it easier to reproduce.
> 
> System :
> ------
> 	Debian Lenny with 2.6.30.9
> 	Quad Core 2.5 GHz Q9300
> 	Intel 1Gb/s NIC, e1000e drive

...

> Topology :
> --------
> Four PCs with Linux :
> 10.10.10.32 on br0
> 10.10.10.33 connected to eth5.33 (sender)
> 10.10.10.38 connected to eth7.38 (sender)
> 10.10.10.34 connected to eth6.34 (receiver)
> 
> Behaviour :
> ---------
> 	If I set only qdisc 1: and class 1:1, but *NOT* 1:2 and 1:3 :
> 	Each host independantly : ~935 Mb/s
> 	Both host together, for 10.10.10.38 : ~527 Mb/s ;
> 	Both host together, for 10.10.10.33 : ~443 Mb/s
> 
> 	If I add classes 1:2 and 1:3 :
> 	Each host independantly : ~170 Mb/s.
> 	Both host together, for 10.10.10.38 : ~106 Mb/s ;
> 	Both host together, for 10.10.10.33 : ~135 Mb/s
> 
> 	So, not only performance did drop significantely, but
> prioritisation did not happen as expected.

If these eths are vlans (or other virtuals) something like this
often happens if you forget to set dev's txqueuelen before
adding classes (or a subqdisc with some 'limit').

> Weird detail :
> ------------
> 	I've noticed that /sbin/tc calculates a very low burst
> value. This is due to the content of /proc/net/psched. I'm wondering
> if the burst calculation is what causes the issue here.
> 	However, I tried with "burst 50kb" and saw no difference...

There is (probably still) unfixed overflow in tc.

BTW, v2.6.31 should be more exact (but so much...) for above 100mbit
scheduling, especially with this patch to iproute2:
http://marc.info/?l=linux-netdev&m=124453482324409&w=2

Jarek P.

Re: TC-HTB issue : low throughput

[Jarek Poplawski]

Jarek Poplawski wrote, On 11/08/2009 12:31 AM:
> BTW, v2.6.31 should be more exact (but so much...) for above 100mbit
should be:
BTW, v2.6.31 should be more exact (but _not_ so much...) for above 100mbit

Jarek P.

Re: TC-HTB issue : low throughput

[Jean Tourrilhes]

On Sun, Nov 08, 2009 at 12:31:21AM +0100, Jarek Poplawski wrote:
> Jean Tourrilhes wrote, On 11/07/2009 03:43 AM:
> 
> > Behaviour :
> > ---------
> > 
> > 	If I add classes 1:2 and 1:3 :
> > 	Each host independantly : ~170 Mb/s.
> > 	Both host together, for 10.10.10.38 : ~106 Mb/s ;
> > 	Both host together, for 10.10.10.33 : ~135 Mb/s
> > 
> > 	So, not only performance did drop significantely, but
> > prioritisation did not happen as expected.
> 
> If these eths are vlans (or other virtuals) something like this
> often happens if you forget to set dev's txqueuelen before
> adding classes (or a subqdisc with some 'limit').

	Thanks a lot ! I've just added :
ifconfig eth6.34 txqueuelen 5
	And now I get 450 Mb/s + 133 Mb/s. Not perfect, but much much
better, and probably good enough for now ;-)


> > 	I've noticed that /sbin/tc calculates a very low burst
> > value. This is due to the content of /proc/net/psched. I'm wondering
> > if the burst calculation is what causes the issue here.
> > 	However, I tried with "burst 50kb" and saw no difference...
> 
> There is (probably still) unfixed overflow in tc.
> 
> BTW, v2.6.31 should be more exact (but so much...) for above 100mbit
> scheduling, especially with this patch to iproute2:
> http://marc.info/?l=linux-netdev&m=124453482324409&w=2
> 
> Jarek P.

	Thanks. I'll look into upgrading.
	Regards,

	Jean