Re: tbf/htb qdisc limitations

[Bill Fink <billfink@mindspring.com>]

On Fri, 15 Oct 2010, Eric Dumazet wrote:

> Le vendredi 15 octobre 2010 à 02:37 -0400, Bill Fink a écrit :
> > On Thu, 14 Oct 2010, Jarek Poplawski wrote:
> > 
> > > On Thu, Oct 14, 2010 at 08:09:39AM +0000, Jarek Poplawski wrote:
> > > > On Thu, Oct 14, 2010 at 03:13:54AM -0400, Bill Fink wrote:
> > > > > TSO/GSO was disabled and was using 9000-byte jumbo frames
> > > > > (and specified mtu 9000 to tc command).
> > > > > 
> > > > > Here was one attempt I made using tbf:
> > > > > 
> > > > > tc qdisc add dev eth2 root handle 1: prio
> > > > > tc qdisc add dev eth2 parent 1:1 handle 10: tbf rate 8900mbit buffer 1112500 limit 10000 mtu 9000
> > > > > tc filter add dev eth2 protocol ip parent 1: prio 1 u32 match ip dst 192.168.1.23 flowid 10:1
> > > > > 
> > > > > I tried many variations of the above, all without success.
> > > > 
> > > > The main problem are smaller packets. If you had (almost) only 9000b
> > > > frames this probably could work. [...]
> > > 
> > > On the other hand, e.g. the limit above seems too low wrt mtu & rate.
> > 
> > Actually, I discovered my commands above work just fine on
> > a 2.6.35 box:
> > 
> > i7test7% nuttcp -T10 -i1 192.168.1.17
> >  1045.3125 MB /   1.00 sec = 8768.3573 Mbps     0 retrans
> >  1045.6875 MB /   1.00 sec = 8772.0292 Mbps     0 retrans
> >  1049.5625 MB /   1.00 sec = 8804.2627 Mbps     0 retrans
> >  1043.1875 MB /   1.00 sec = 8750.9960 Mbps     0 retrans
> >  1048.6875 MB /   1.00 sec = 8796.3246 Mbps     0 retrans
> >  1033.4375 MB /   1.00 sec = 8669.3188 Mbps     0 retrans
> >  1040.7500 MB /   1.00 sec = 8730.7057 Mbps     0 retrans
> >  1047.0000 MB /   1.00 sec = 8783.2063 Mbps     0 retrans
> >  1040.0000 MB /   1.00 sec = 8724.0564 Mbps     0 retrans
> >  1037.4375 MB /   1.00 sec = 8702.5434 Mbps     0 retrans
> > 
> > 10431.5608 MB /  10.00 sec = 8749.7542 Mbps 25 %TX 35 %RX 0 retrans 0.11 msRTT
> > 
> > The problems I encountered were on a field system running
> > 2.6.30.10.  I will investigate upgrading the field system
> > to 2.6.35.
> > 
> 
> Yes, I noticed same thing for me on net-next-2.6 
> 
> Please report :
> 
> tc -s -d qdisc

Not sure why you want this on the older 2.6.30.10 kernel,
but here it is:

i7test6% nuttcp -T10 -i1 192.168.1.14
 1169.1875 MB /   1.00 sec = 9807.2868 Mbps     0 retrans
 1181.1875 MB /   1.00 sec = 9908.9054 Mbps     0 retrans
 1181.1250 MB /   1.00 sec = 9907.9253 Mbps     0 retrans
 1181.1875 MB /   1.00 sec = 9908.4991 Mbps     0 retrans
 1180.6875 MB /   1.00 sec = 9904.3345 Mbps     0 retrans
 1181.1250 MB /   1.00 sec = 9908.0838 Mbps     0 retrans
 1181.1875 MB /   1.00 sec = 9908.4099 Mbps     0 retrans
 1181.0625 MB /   1.00 sec = 9907.3911 Mbps     0 retrans
 1181.3750 MB /   1.00 sec = 9910.2801 Mbps     0 retrans
 1181.1875 MB /   1.00 sec = 9908.2118 Mbps     0 retrans

11801.1382 MB /  10.04 sec = 9858.7159 Mbps 24 %TX 40 %RX 0 retrans 0.11 msRTT

i7test6% tc -s -d qdisc show dev eth2
qdisc prio 1: root refcnt 32 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 12448974085 bytes 1381173 pkt (dropped 266, overlimits 0 requeues 12) 
 rate 0bit 0pps backlog 0b 0p requeues 12 
qdisc tbf 10: parent 1:1 rate 8900Mbit burst 1111387b/64 mpu 0b lat 4295.0s 
 Sent 12448974043 bytes 1381172 pkt (dropped 266, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 

I'm guessing this is probably related to the schedulers
time resolution issue that Jarek mentioned.

And for completeness, here's the info for the working
2.6.35 case:

i7test7% nuttcp -T10 -i1 192.168.1.17
 1045.5625 MB /   1.00 sec = 8770.6210 Mbps     0 retrans
 1032.1875 MB /   1.00 sec = 8658.3825 Mbps     0 retrans
 1039.8125 MB /   1.00 sec = 8722.7801 Mbps     0 retrans
 1050.2500 MB /   1.00 sec = 8810.0739 Mbps     0 retrans
 1050.6875 MB /   1.00 sec = 8813.9378 Mbps     0 retrans
 1048.8125 MB /   1.00 sec = 8798.0857 Mbps     0 retrans
 1046.1875 MB /   1.00 sec = 8775.9954 Mbps     0 retrans
 1045.7500 MB /   1.00 sec = 8771.9307 Mbps     0 retrans
 1051.1250 MB /   1.00 sec = 8817.8900 Mbps     0 retrans
 1044.0625 MB /   1.00 sec = 8757.8019 Mbps     0 retrans

10454.7500 MB /  10.00 sec = 8769.2206 Mbps 26 %TX 35 %RX 0 retrans 0.11 msRTT

i7test7% tc -s -d qdisc show dev eth2
qdisc prio 1: root refcnt 33 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 11028687119 bytes 1223828 pkt (dropped 293, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc tbf 10: parent 1:1 rate 8900Mbit burst 1112500b/64 mpu 0b lat 4295.0s 
 Sent 11028687077 bytes 1223827 pkt (dropped 293, overlimits 593 requeues 0) 
 backlog 0b 0p requeues 0 

I'm not sure how you can have so many dropped but not have
any TCP retransmissions (or not show up as requeues).  But
there's probably something basic I just don't understand
about how all this stuff works.

					-Bill