Re: Possible regression in HTB

[Simon Horman <horms@verge.net.au>]

On Wed, Oct 08, 2008 at 08:03:40AM +0000, Jarek Poplawski wrote:
> On Wed, Oct 08, 2008 at 06:22:04PM +1100, Simon Horman wrote:
> > On Wed, Oct 08, 2008 at 06:55:51AM +0000, Jarek Poplawski wrote:
> > > On Wed, Oct 08, 2008 at 02:31:26AM +0200, Patrick McHardy wrote:
> > > ...
> > > > I'm pretty sure that the differences are caused by HTB not being
> > > > in control of the queue since the device is the real bottleneck
> > > > in this configuration.
> > > 
> > > Yes, otherwise there would be no requeuing. And, btw. the golden rule
> > > of scheduling/shaping is limiting below "hardware" limits.
> > > 
> > > > Its quite possible that there simply might
> > > > a subtle timing change that causes feedback through HTBs borrowing
> > > > and ceiling.
> > > 
> > > I'd add my previous suspicion there could be not enough enqeuing on
> > > time for the fastest class (could be also very bursty), so other
> > > classes can borrow more.
> > > 
> > > >
> > > > So what would really be useful to understand this is to make HTB
> > > > control the queue and see if it behaves as expected.
> > > >
> > > 
> > > Right, trying with lower rates/ceils should explain this.
> > 
> > As I mentioned earlier things seem to work quite well with lower
> > rates/ceilings. When I set up the classes with 10x lower values
> > for rate and celing, as follows:
> > 
> > 
> >                            [ rate=100Mbit/s ]
> >                            [ ceil=100Mbit/s ]
> >                                   |
> >              +--------------------+--------------------+
> >              |                    |                    |
> >       [ rate= 50Mbit/s ]   [ rate= 10Mbit/s ]   [ rate= 10Mbit/s ]
> >       [ ceil=100Mbit/s ]   [ ceil=100Mbit/s ]   [ ceil= 100Mbit/s ]
> > 
> > Then I get results that are fairly close to the ideal values.
> > 
> > net-next-2.6 - d877984
> > ----------------------
> > 10194: 68075482bits/s 68Mbits/s
> > 10197: 14464848bits/s 14Mbits/s
> > 10196: 14465632bits/s 14Mbits/s
> > -----------------------------------
> > total: 97005962bits/s 97Mbits/s
> > 
> > And I get those kind of results consistently for various
> > different kernel versions.
> 
> OK. But as Patrick mentioned it would be interesting to try a little
> below hardware limits: 950, or maybe lower, until HTB starts getting
> accuracy.

Hi,

it seems that for my particular setup the magic number is 935Mbit/s.


Kernel is net-next-2.6 071d7ab6649eb34a873a53e71635186e9117101d
("ipvs: Remove stray file left over from ipvs move"),
which is after Jarek's "pkt_sched: Update qdisc requeue stats in
dev_requeue_skb()" patch.

ideal (based on 950Mbit/s aggregate)
500mbit class (10194): 500mbit + 250mbit/7*5 == 678.57mbit
100mbit class (10196): 100mbit + 250mbit/7*1 == 135.71mbit
100mbit class (10197): 100mbit + 250mbit/7*1 == 135.71mbit
                                                ==========
						950.00mbit
n=900
10194: 677727637bits/s 677Mbits/s
10197: 136662048bits/s 136Mbits/s
10196: 136725637bits/s 136Mbits/s
-----------------------------------
total: 951115322bits/s 951Mbits/s

n=920
10194: 676271338bits/s 676Mbits/s
10197: 137301090bits/s 137Mbits/s
10196: 137301877bits/s 137Mbits/s
-----------------------------------
total: 950874306bits/s 950Mbits/s

n=930
10194: 674681581bits/s 674Mbits/s
10197: 137538965bits/s 137Mbits/s
10196: 137541320bits/s 137Mbits/s
-----------------------------------
total: 949761866bits/s 949Mbits/s

n=933
10194: 675568704bits/s 675Mbits/s
10197: 137661437bits/s 137Mbits/s
10196: 137662221bits/s 137Mbits/s
-----------------------------------
total: 950892362bits/s 950Mbits/s

n=934
10194: 675399128bits/s 675Mbits/s
10197: 137653586bits/s 137Mbits/s
10196: 137704613bits/s 137Mbits/s
-----------------------------------
total: 950757328bits/s 950Mbits/s

n=935
10194: 675169893bits/s 675Mbits/s
10197: 137667714bits/s 137Mbits/s
10196: 137670858bits/s 137Mbits/s
-----------------------------------
total: 950508466bits/s 950Mbits/s

n=936
10194: 385295016bits/s 385Mbits/s
10197: 285078114bits/s 285Mbits/s
10196: 286588581bits/s 286Mbits/s
-----------------------------------
total: 956961712bits/s 956Mbits/s

n=937
10194: 384569616bits/s 384Mbits/s
10197: 285480072bits/s 285Mbits/s
10196: 286627050bits/s 286Mbits/s
-----------------------------------
total: 956676738bits/s 956Mbits/s

n=940
10194: 384577466bits/s 384Mbits/s
10197: 285655138bits/s 285Mbits/s
10196: 286846872bits/s 286Mbits/s
-----------------------------------
total: 957079477bits/s 957Mbits/s

n=950
10194: 384097789bits/s 384Mbits/s
10197: 285950325bits/s 285Mbits/s
10196: 286894760bits/s 286Mbits/s
-----------------------------------
total: 956942874bits/s 956Mbits/s

n=1000
10194: 384639482bits/s 384Mbits/s
10197: 285133069bits/s 285Mbits/s
10196: 287172674bits/s 287Mbits/s
-----------------------------------
total: 956945226bits/s 956Mbits/s


# HTB
###########################################################################
n=933

tc qdisc del dev eth0 root

tc qdisc add dev eth0 root handle 1: htb default 10 r2q $((n * 10))

tc class add dev eth0 parent 1:  classid 1:1   htb \
        rate ${n}Mbit ceil ${n}Mbit

tc class add dev eth0 parent 1:1 classid 1:10 htb \
        rate ${n}Mbit ceil ${n}Mbit
tc class add dev eth0 parent 1:1 classid 1:11 htb \
        rate 500Mbit ceil ${n}Mbit
tc class add dev eth0 parent 1:1 classid 1:12 htb \
        rate 100Mbit ceil ${n}Mbit
tc class add dev eth0 parent 1:1 classid 1:13 htb \
        rate 100Mbit ceil ${n}Mbit

#tc filter add dev eth0 protocol ip parent 1: \
#       u32 match ip src 172.17.60.194 flowid 1:11
#tc filter add dev eth0 protocol ip parent 1: \
#       u32 match ip src 172.17.60.196 flowid 1:12
#tc filter add dev eth0 protocol ip parent 1: \
#       u32 match ip src 172.17.60.197 flowid 1:13

tc filter add dev eth0 protocol ip parent 1: \
        u32 match ip dport 10194 0xffff flowid 1:11
tc filter add dev eth0 protocol ip parent 1: \
        u32 match ip dport 10196 0xffff flowid 1:12
tc filter add dev eth0 protocol ip parent 1: \
        u32 match ip dport 10197 0xffff flowid 1:13

tc -s -d class show dev eth0