HTB - very bad precision? HFSC works fine! 2.6.28

HTB - very bad precision? HFSC works fine! 2.6.28

[Denys Fedoryschenko]

I notice that our uplink ISP complaining, that we are exceeding our UP rate.

I have configured at HTB 60Mbps, but he is actually receiving bursts up to 
64Mbps.

I tried to lower down HTB values to 57Mbps, and even on Cisco switch interface 
counter(it is averaging 30 seconds of data i guess, to calculate bandwidth) - 
it shows 60.1 - 60.4.

Funny thing, root class in HTB shows 57.1 - 57.3 Mbps. So maybe byte counting 
is wrong in HTB?

I have also "default filter" 
tc filter add dev ${DEV} parent 1:0 protocol ip prio 800 u32 match ip src 
0.0.0.0/0 flowid 1:50

and for HFSC i did also filter for arp.

I tried just to replace HTB plainly by HFSC, and it works perfectly and 
precise. Just what i did - removed prio in class, PARAM variable, and changed 
names by replace 
"htb rate" to "est 1sec 8sec hfsc sc rate"
and
"ceil" to "ul rate"

And 57 and bandwidth on switch went to 56.7 - 56.9, and stays very stable.

P.S. overhead 16 - i tried first without it, then with - no difference.

Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Denys Fedoryschenko]

Well, but byte couting on HFSC works fine.

On Thursday 08 January 2009 00:12:30 Jorge Bastos wrote:
> I have a similar problem with CBQ that I've been reporting, that may be
> related to byte counting also.
> It just happens when I use a 1000Mbit NIC.
> I've reported it to netfilter@ and created a but report here.
> When I insert the old 100Mbit card, it works OK.
>
> http://bugzilla.kernel.org/show_bug.cgi?id=12364
>

Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Denys Fedoryschenko]

No, it is completely different disciplines

Here is my rules
http://www.nuclearcat.com/badhtb.txt


On Thursday 08 January 2009 00:40:03 Jorge Bastos wrote:
> Sorry my ignorance, HFSC = CBQ ?
> If not, on CBQ it's not working when I apply the rules that I have in the
> bug report.
> One interesting thing, this when I'm using the 1000Mbit card, if I remove
> the TBF line, it works +/- good on the 1000Mbit card.
>
> ---
> /sbin/tc qdisc add dev eth0 parent 1:1280 handle 1280 tbf rate 200Kbit
> buffer 10Kb/8 limit 15Kb mtu 1500
> ---
>
> > -----Original Message-----
> > From: Denys Fedoryschenko [mailto:denys@visp.net.lb]
> > Sent: quarta-feira, 7 de Janeiro de 2009 22:37
> > To: Jorge Bastos
> > Cc: netdev@vger.kernel.org
> > Subject: Re: HTB - very bad precision? HFSC works fine! 2.6.28
> >
> > Well, but byte couting on HFSC works fine.
> >
> > On Thursday 08 January 2009 00:12:30 Jorge Bastos wrote:
> > > I have a similar problem with CBQ that I've been reporting, that may
> >
> > be
> >
> > > related to byte counting also.
> > > It just happens when I use a 1000Mbit NIC.
> > > I've reported it to netfilter@ and created a but report here.
> > > When I insert the old 100Mbit card, it works OK.
> > >
> > > http://bugzilla.kernel.org/show_bug.cgi?id=12364

Re: HTB - very bad precision? HFSC works fine! 2.6.28

[mysql.jorge]

Is there anyway to debug this?




> No, it is completely different disciplines
>
> Here is my rules
> http://www.nuclearcat.com/badhtb.txt
>
>
> On Thursday 08 January 2009 00:40:03 Jorge Bastos wrote:
>> Sorry my ignorance, HFSC = CBQ ?
>> If not, on CBQ it's not working when I apply the rules that I have in
>> the
>> bug report.
>> One interesting thing, this when I'm using the 1000Mbit card, if I
>> remove
>> the TBF line, it works +/- good on the 1000Mbit card.
>>
>> ---
>> /sbin/tc qdisc add dev eth0 parent 1:1280 handle 1280 tbf rate 200Kbit
>> buffer 10Kb/8 limit 15Kb mtu 1500
>> ---
>>
>> > -----Original Message-----
>> > From: Denys Fedoryschenko [mailto:denys@visp.net.lb]
>> > Sent: quarta-feira, 7 de Janeiro de 2009 22:37
>> > To: Jorge Bastos
>> > Cc: netdev@vger.kernel.org
>> > Subject: Re: HTB - very bad precision? HFSC works fine! 2.6.28
>> >
>> > Well, but byte couting on HFSC works fine.
>> >
>> > On Thursday 08 January 2009 00:12:30 Jorge Bastos wrote:
>> > > I have a similar problem with CBQ that I've been reporting, that may
>> >
>> > be
>> >
>> > > related to byte counting also.
>> > > It just happens when I use a 1000Mbit NIC.
>> > > I've reported it to netfilter@ and created a but report here.
>> > > When I insert the old 100Mbit card, it works OK.
>> > >
>> > > http://bugzilla.kernel.org/show_bug.cgi?id=12364
>
>
>

Re: [BUG 12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On 08-01-2009 00:42, mysql.jorge@decimal.pt wrote:
> Is there anyway to debug this?

Could you try: ethtool -k eth0
with quite fresh ethtool, and to turn off GSO etc., if it's on?

Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Thu, Jan 08, 2009 at 09:42:31AM +0000, Jarek Poplawski wrote:
> On 08-01-2009 00:42, mysql.jorge@decimal.pt wrote:
> > Is there anyway to debug this?
> 
> Could you try: ethtool -k eth0
> with quite fresh ethtool, and to turn off GSO etc., if it's on?

Hmm... Actually, it's probably about config: I'm not cbq user or
expert, but AFAIK it depends on precise info about the real link
rate, so it probably needs an update to 1000Mbit.

Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Thu, Jan 08, 2009 at 09:54:03AM +0000, Jarek Poplawski wrote:
> On Thu, Jan 08, 2009 at 09:42:31AM +0000, Jarek Poplawski wrote:
> > On 08-01-2009 00:42, mysql.jorge@decimal.pt wrote:
> > > Is there anyway to debug this?
> > 
> > Could you try: ethtool -k eth0
> > with quite fresh ethtool, and to turn off GSO etc., if it's on?
> 
> Hmm... Actually, it's probably about config: I'm not cbq user or
> expert, but AFAIK it depends on precise info about the real link
> rate, so it probably needs an update to 1000Mbit.

Hmm#2... I missed this tbf thing: this could point to GSO again, but
generally it's considered as a bug to use more than one non-work-
conserving qdisc in a tree (so e.g. cbq with any of tbf/htb/hfsc
together).

Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[mysql.jorge]

Well, even with that, why it works perfectly when i use the 100Mbit NIC?
There must be something here that needs an ajustment, no?




>>
>> Hmm... Actually, it's probably about config: I'm not cbq user or
>> expert, but AFAIK it depends on precise info about the real link
>> rate, so it probably needs an update to 1000Mbit.
>
> Hmm#2... I missed this tbf thing: this could point to GSO again, but
> generally it's considered as a bug to use more than one non-work-
> conserving qdisc in a tree (so e.g. cbq with any of tbf/htb/hfsc
> together).
>
> Jarek P.
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Thu, Jan 08, 2009 at 10:35:39AM -0000, mysql.jorge@decimal.pt wrote:
> Well, even with that, why it works perfectly when i use the 100Mbit NIC?
> There must be something here that needs an ajustment, no?

It's hard to tell because these two things (cbq and tbf) use different
methods, which could interact strange way (and are not expected to work
together). I think you should test them alone first (especially tbf),
and send here the tc comannd plus "tc -s qdisc show dev eth0" and
"tc -s class show dev eth0" output while traffic is run. You could also
add this "ethtool -k eth0" and "ifconfig eth0".

Jarek P. 


> >>
> >> Hmm... Actually, it's probably about config: I'm not cbq user or
> >> expert, but AFAIK it depends on precise info about the real link
> >> rate, so it probably needs an update to 1000Mbit.
> >
> > Hmm#2... I missed this tbf thing: this could point to GSO again, but
> > generally it's considered as a bug to use more than one non-work-
> > conserving qdisc in a tree (so e.g. cbq with any of tbf/htb/hfsc
> > together).
> >
> > Jarek P.
> >
> 
> 

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[mysql.jorge]

here are them:

i see in the TBF class, 25000bps, where did it get that value?
the rules i used was:
---
/sbin/tc qdisc del dev eth0 root
/sbin/tc qdisc add dev eth0 root handle 1 cbq bandwidth 100Mbit avpkt 1000
cell 8
/sbin/tc class change dev eth0 root cbq weight 10Mbit allot 1514

/sbin/tc class add dev eth0 parent 1: classid 1:1280 cbq bandwidth 100Mbit
rate 200Kbit weight 6kbit prio 5 allot 1514 cell 8
maxburst 20 avpkt 1000 bounded isolated
/sbin/tc qdisc add dev eth0 parent 1:1280 handle 1280 tbf rate 200Kbit
buffer 10Kb/8 limit 15Kb mtu 1500
/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 100 u32 match ip
src 195.23.114.76 match ip sport 80 0xffff classid 1
:1280

---



lira:~# tc -s class show dev eth0
class cbq 1: root rate 125000000bps (bounded,isolated) prio no-transmit
 Sent 3165780 bytes 4451 pkts (dropped 0, overlimits 0)
  borrowed 0 overactions 0 avgidle 7 undertime 0
class cbq 1:1280 parent 1: leaf 1280: rate 25000bps (bounded,isolated) prio 5
 Sent 644302 bytes 455 pkts (dropped 218, overlimits 0)
  borrowed 0 overactions 0 avgidle 1.10849e+06 undertime 0
class tbf 1280:1 parent 1280: [UNKNOWN]
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lira:~# tc -s qdisc show dev eth0
qdisc cbq 1: root rate 125000000bps (bounded,isolated) prio no-transmit
 Sent 4214629 bytes 5893 pkts (dropped 251, overlimits 0)
  borrowed 0 overactions 0 avgidle 7 undertime 0

 qdisc tbf 1280: parent 1:1280 rate 25000bps burst 10Kb lat 209.7ms
 Sent 747550 bytes 527 pkts (dropped 251, overlimits 0)

lira:~# ethtool -k eth0
Offload parameters for eth0:
Cannot get device flags: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off


(the shaping is done in the eth0:1 IP ADDR)

lira:~# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:13:d4:68:c6:33
          inet addr:195.23.114.76  Bcast:195.23.114.79  Mask:255.255.255.248
          inet6 addr: fe80::213:d4ff:fe68:c633/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4544600 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4047072 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2148440536 (2.0 GiB)  TX bytes:2942330389 (2.7 GiB)
          Interrupt:17

eth0:0    Link encap:Ethernet  HWaddr 00:13:d4:68:c6:33
          inet addr:192.168.1.222  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:17



> On Thu, Jan 08, 2009 at 10:35:39AM -0000, mysql.jorge@decimal.pt wrote:
>> Well, even with that, why it works perfectly when i use the 100Mbit NIC?
>> There must be something here that needs an ajustment, no?
>
> It's hard to tell because these two things (cbq and tbf) use different
> methods, which could interact strange way (and are not expected to work
> together). I think you should test them alone first (especially tbf),
> and send here the tc comannd plus "tc -s qdisc show dev eth0" and
> "tc -s class show dev eth0" output while traffic is run. You could also
> add this "ethtool -k eth0" and "ifconfig eth0".
>
> Jarek P.
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Thu, Jan 08, 2009 at 11:22:01AM -0000, mysql.jorge@decimal.pt wrote:
...
> lira:~# ethtool -k eth0
> Offload parameters for eth0:
> Cannot get device flags: Operation not supported
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp segmentation offload: on
> udp fragmentation offload: off
> generic segmentation offload: off
> large receive offload: off
> 

Could you try to check this with scatter-gather and tcp segmentation
offload off too? If there is no difference please send the stats
again both for working and non-working case.

Thanks,
Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[mysql.jorge]

An active connection when running this commands:
(now it seams to work, the 200Kbit i defined, are gaving me 22.5/23Kbyte
/sec)

Did you catch it?


lira:~# ethtool -k eth0
Offload parameters for eth0:
Cannot get device flags: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off
lira:~# tc -s qdisc show dev eth0
qdisc cbq 1: root rate 12500000bps (bounded,isolated) prio no-transmit
 Sent 5336353 bytes 5737 pkts (dropped 16, overlimits 1382)
 backlog 8p
  borrowed 0 overactions 0 avgidle 78 undertime 0

 qdisc tbf 1280: parent 1:1280 rate 25000bps burst 10Kb lat 209.7ms
 Sent 441812 bytes 310 pkts (dropped 16, overlimits 2215)
 backlog 8p

lira:~# tc -s class show dev eth0
class cbq 1: root rate 12500000bps (bounded,isolated) prio no-transmit
 Sent 5128779 bytes 5827 pkts (dropped 0, overlimits 0)
  borrowed 0 overactions 0 avgidle 78 undertime 0
class cbq 1:1280 parent 1: leaf 1280: rate 25000bps (bounded,isolated) prio 5
 Sent 811784 bytes 568 pkts (dropped 22, overlimits 0)
 backlog 9p
  borrowed 0 overactions 0 avgidle 4507 undertime 0
class tbf 1280:1 parent 1280: [UNKNOWN]
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)





> Could you try to check this with scatter-gather and tcp segmentation
> offload off too? If there is no difference please send the stats
> again both for working and non-working case.
>
> Thanks,
> Jarek P.
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[mysql.jorge]

I asume there's some issue with the NIC Driver os the kernel modules with
the 1000Mbit NIC's? am i correct?


> An active connection when running this commands:
> (now it seams to work, the 200Kbit i defined, are gaving me 22.5/23Kbyte
> /sec)
>
> Did you catch it?
>
>
> lira:~# ethtool -k eth0
> Offload parameters for eth0:
> Cannot get device flags: Operation not supported
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: off
> tcp segmentation offload: off
> udp fragmentation offload: off
> generic segmentation offload: off
> large receive offload: off
> lira:~# tc -s qdisc show dev eth0
> qdisc cbq 1: root rate 12500000bps (bounded,isolated) prio no-transmit
>  Sent 5336353 bytes 5737 pkts (dropped 16, overlimits 1382)
>  backlog 8p
>   borrowed 0 overactions 0 avgidle 78 undertime 0
>
>  qdisc tbf 1280: parent 1:1280 rate 25000bps burst 10Kb lat 209.7ms
>  Sent 441812 bytes 310 pkts (dropped 16, overlimits 2215)
>  backlog 8p
>
> lira:~# tc -s class show dev eth0
> class cbq 1: root rate 12500000bps (bounded,isolated) prio no-transmit
>  Sent 5128779 bytes 5827 pkts (dropped 0, overlimits 0)
>   borrowed 0 overactions 0 avgidle 78 undertime 0
> class cbq 1:1280 parent 1: leaf 1280: rate 25000bps (bounded,isolated)
> prio 5
>  Sent 811784 bytes 568 pkts (dropped 22, overlimits 0)
>  backlog 9p
>   borrowed 0 overactions 0 avgidle 4507 undertime 0
> class tbf 1280:1 parent 1280: [UNKNOWN]
>  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
>
>
>
>
>
>> Could you try to check this with scatter-gather and tcp segmentation
>> offload off too? If there is no difference please send the stats
>> again both for working and non-working case.
>>
>> Thanks,
>> Jarek P.
>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Denys Fedoryschenko]

I disable tso/gso and others. No difference at all. Here is actual output, 
after running it for 1-2 minutes.

Cisco
  30 second output rate 59600000 bits/sec, 32978 packets/sec
Linux

class htb 1:33 parent 1:10 leaf 33: prio 0 rate 1000Kbit ceil 57000Kbit burst 
2849b cburst 72838b
 Sent 63746261 bytes 492929 pkt (dropped 0, overlimits 0 requeues 0)
 rate 61200bit 48pps backlog 0b 0p requeues 0
 lended: 491208 borrowed: 1721 giants: 0
 tokens: 21453 ctokens: 9971

class htb 1:32 parent 1:10 leaf 32: prio 0 rate 4000Kbit ceil 57000Kbit burst 
6599b cburst 72838b
 Sent 7976915758 bytes 45773720 pkt (dropped 0, overlimits 0 requeues 0)
 rate 7242Kbit 5300pps backlog 0b 0p requeues 0
 lended: 28141768 borrowed: 17631952 giants: 0
 tokens: -1245 ctokens: 9972

class htb 1:10 root rate 57000Kbit ceil 57000Kbit burst 72838b cburst 72838b
 Sent 75203300117 bytes 328680653 pkt (dropped 0, overlimits 0 requeues 0)
 rate 58008Kbit 32891pps backlog 0b 0p requeues 0
 lended: 223536594 borrowed: 0 giants: 0
 tokens: 165 ctokens: 165

class htb 1:31 parent 1:10 leaf 31: prio 1 rate 400000bit ceil 57000Kbit burst 
2099b cburst 72838b
 Sent 9434582792 bytes 34978695 pkt (dropped 15721, overlimits 0 requeues 0)
 rate 7667Kbit 3561pps backlog 0b 0p requeues 0
 lended: 1739830 borrowed: 33238865 giants: 0
 tokens: -279 ctokens: 9973

class htb 1:30 parent 1:10 leaf 30: prio 0 rate 500000bit ceil 57000Kbit burst 
2224b cburst 72838b
 Sent 2197649866 bytes 13793068 pkt (dropped 0, overlimits 0 requeues 0)
 rate 2217Kbit 1270pps backlog 0b 0p requeues 0
 lended: 4124325 borrowed: 9668743 giants: 0
 tokens: -1671 ctokens: 9973

class htb 1:15 parent 1:10 leaf 15: prio 0 rate 1000Kbit ceil 2000Kbit burst 
2849b cburst 4Kb
 Sent 813164294 bytes 10048814 pkt (dropped 0, overlimits 0 requeues 0)
 rate 798896bit 1100pps backlog 0b 0p requeues 0
 lended: 10029689 borrowed: 19125 giants: 0
 tokens: 21703 ctokens: 15734

class htb 1:51 parent 1:10 leaf 51: prio 7 rate 50000bit ceil 57000Kbit burst 
1661b cburst 72838b
 Sent 706714762 bytes 3749931 pkt (dropped 11334, overlimits 0 requeues 0)
 rate 496816bit 379pps backlog 0b 75p requeues 0
 lended: 333427 borrowed: 3416429 giants: 0
 tokens: -11240 ctokens: 9975

class htb 1:14 parent 1:10 leaf 14: prio 0 rate 7000Kbit ceil 9000Kbit burst 
10348b cburst 12848b
 Sent 8537716658 bytes 48711461 pkt (dropped 24071, overlimits 0 requeues 0)
 rate 6797Kbit 4708pps backlog 0b 0p requeues 0
 lended: 43171840 borrowed: 5539621 giants: 0
 tokens: 1524 ctokens: 9829

class htb 1:50 parent 1:10 leaf 50: prio 6 rate 450000bit ceil 57000Kbit burst 
2161b cburst 72838b
 Sent 17728543906 bytes 59672028 pkt (dropped 2279236, overlimits 0 requeues 
0)
 rate 8312Kbit 4094pps backlog 0b 127p requeues 0
 lended: 1870954 borrowed: 57800947 giants: 0
 tokens: -26247 ctokens: 9777

class htb 1:34 parent 1:10 leaf 34: prio 0 rate 100000bit ceil 57000Kbit burst 
1724b cburst 72838b
 Sent 766 bytes 10 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
 lended: 10 borrowed: 0 giants: 0
 tokens: 93732 ctokens: 9972

class htb 1:20 parent 1:10 leaf 20: prio 0 rate 7500Kbit ceil 57000Kbit burst 
10973b cburst 72838b
 Sent 13347788718 bytes 9570719 pkt (dropped 47, overlimits 0 requeues 0)
 rate 11340Kbit 998pps backlog 0b 0p requeues 0
 lended: 6344649 borrowed: 3226070 giants: 0
 tokens: -853 ctokens: 9614

class htb 1:40 parent 1:10 leaf 40: prio 1 rate 1000Kbit ceil 57000Kbit burst 
2849b cburst 72838b
 Sent 14456405280 bytes 101216398 pkt (dropped 468764, overlimits 0 requeues 
0)
 rate 13221Kbit 11365pps backlog 0b 33p requeues 0
 lended: 8223510 borrowed: 92992855 giants: 0
 tokens: -761 ctokens: 9964

class htb 1:60 parent 1:10 leaf 60: prio 0 rate 500000bit ceil 57000Kbit burst 
2224b cburst 72838b
 Sent 51868158 bytes 673116 pkt (dropped 0, overlimits 0 requeues 0)
 rate 47024bit 81pps backlog 0b 0p requeues 0
 lended: 672849 borrowed: 267 giants: 0
 tokens: 33265 ctokens: 9972

class sfq 40:3 parent 40:
 (dropped 0, overlimits 0 requeues 0)
 backlog 0b 1p requeues 0
 allot -20340

class sfq 40:72 parent 40:
 (dropped 0, overlimits 0 requeues 0)
 backlog 0b 1p requeues 0
 allot 12112

class sfq 40:98 parent 40:
 (dropped 0, overlimits 0 requeues 0)
 backlog 0b 1p requeues 0
 allot -4228

..... others is SFQ





On Thursday 08 January 2009 19:27:31 mysql.jorge@decimal.pt wrote:
> I asume there's some issue with the NIC Driver os the kernel modules with
> the 1000Mbit NIC's? am i correct?
>
> > An active connection when running this commands:
> > (now it seams to work, the 200Kbit i defined, are gaving me 22.5/23Kbyte
> > /sec)
> >

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jorge Bastos]

What's your NIC?
Speed, and driver that you are using.


> I disable tso/gso and others. No difference at all. Here is actual output,
> after running it for 1-2 minutes.
>
> Cisco
>   30 second output rate 59600000 bits/sec, 32978 packets/sec
> Linux
>
> class htb 1:33 parent 1:10 leaf 33: prio 0 rate 1000Kbit ceil 57000Kbit
> burst
> 2849b cburst 72838b
>  Sent 63746261 bytes 492929 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 61200bit 48pps backlog 0b 0p requeues 0
>  lended: 491208 borrowed: 1721 giants: 0
>  tokens: 21453 ctokens: 9971
>
> class htb 1:32 parent 1:10 leaf 32: prio 0 rate 4000Kbit ceil 57000Kbit
> burst
> 6599b cburst 72838b
>  Sent 7976915758 bytes 45773720 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 7242Kbit 5300pps backlog 0b 0p requeues 0
>  lended: 28141768 borrowed: 17631952 giants: 0
>  tokens: -1245 ctokens: 9972
>
> class htb 1:10 root rate 57000Kbit ceil 57000Kbit burst 72838b cburst
> 72838b
>  Sent 75203300117 bytes 328680653 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 58008Kbit 32891pps backlog 0b 0p requeues 0
>  lended: 223536594 borrowed: 0 giants: 0
>  tokens: 165 ctokens: 165
>
> class htb 1:31 parent 1:10 leaf 31: prio 1 rate 400000bit ceil 57000Kbit
> burst
> 2099b cburst 72838b
>  Sent 9434582792 bytes 34978695 pkt (dropped 15721, overlimits 0 requeues
> 0)
>  rate 7667Kbit 3561pps backlog 0b 0p requeues 0
>  lended: 1739830 borrowed: 33238865 giants: 0
>  tokens: -279 ctokens: 9973
>
> class htb 1:30 parent 1:10 leaf 30: prio 0 rate 500000bit ceil 57000Kbit
> burst
> 2224b cburst 72838b
>  Sent 2197649866 bytes 13793068 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 2217Kbit 1270pps backlog 0b 0p requeues 0
>  lended: 4124325 borrowed: 9668743 giants: 0
>  tokens: -1671 ctokens: 9973
>
> class htb 1:15 parent 1:10 leaf 15: prio 0 rate 1000Kbit ceil 2000Kbit
> burst
> 2849b cburst 4Kb
>  Sent 813164294 bytes 10048814 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 798896bit 1100pps backlog 0b 0p requeues 0
>  lended: 10029689 borrowed: 19125 giants: 0
>  tokens: 21703 ctokens: 15734
>
> class htb 1:51 parent 1:10 leaf 51: prio 7 rate 50000bit ceil 57000Kbit
> burst
> 1661b cburst 72838b
>  Sent 706714762 bytes 3749931 pkt (dropped 11334, overlimits 0 requeues 0)
>  rate 496816bit 379pps backlog 0b 75p requeues 0
>  lended: 333427 borrowed: 3416429 giants: 0
>  tokens: -11240 ctokens: 9975
>
> class htb 1:14 parent 1:10 leaf 14: prio 0 rate 7000Kbit ceil 9000Kbit
> burst
> 10348b cburst 12848b
>  Sent 8537716658 bytes 48711461 pkt (dropped 24071, overlimits 0 requeues
> 0)
>  rate 6797Kbit 4708pps backlog 0b 0p requeues 0
>  lended: 43171840 borrowed: 5539621 giants: 0
>  tokens: 1524 ctokens: 9829
>
> class htb 1:50 parent 1:10 leaf 50: prio 6 rate 450000bit ceil 57000Kbit
> burst
> 2161b cburst 72838b
>  Sent 17728543906 bytes 59672028 pkt (dropped 2279236, overlimits 0
> requeues
> 0)
>  rate 8312Kbit 4094pps backlog 0b 127p requeues 0
>  lended: 1870954 borrowed: 57800947 giants: 0
>  tokens: -26247 ctokens: 9777
>
> class htb 1:34 parent 1:10 leaf 34: prio 0 rate 100000bit ceil 57000Kbit
> burst
> 1724b cburst 72838b
>  Sent 766 bytes 10 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0
>  lended: 10 borrowed: 0 giants: 0
>  tokens: 93732 ctokens: 9972
>
> class htb 1:20 parent 1:10 leaf 20: prio 0 rate 7500Kbit ceil 57000Kbit
> burst
> 10973b cburst 72838b
>  Sent 13347788718 bytes 9570719 pkt (dropped 47, overlimits 0 requeues 0)
>  rate 11340Kbit 998pps backlog 0b 0p requeues 0
>  lended: 6344649 borrowed: 3226070 giants: 0
>  tokens: -853 ctokens: 9614
>
> class htb 1:40 parent 1:10 leaf 40: prio 1 rate 1000Kbit ceil 57000Kbit
> burst
> 2849b cburst 72838b
>  Sent 14456405280 bytes 101216398 pkt (dropped 468764, overlimits 0
> requeues
> 0)
>  rate 13221Kbit 11365pps backlog 0b 33p requeues 0
>  lended: 8223510 borrowed: 92992855 giants: 0
>  tokens: -761 ctokens: 9964
>
> class htb 1:60 parent 1:10 leaf 60: prio 0 rate 500000bit ceil 57000Kbit
> burst
> 2224b cburst 72838b
>  Sent 51868158 bytes 673116 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 47024bit 81pps backlog 0b 0p requeues 0
>  lended: 672849 borrowed: 267 giants: 0
>  tokens: 33265 ctokens: 9972
>
> class sfq 40:3 parent 40:
>  (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 1p requeues 0
>  allot -20340
>
> class sfq 40:72 parent 40:
>  (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 1p requeues 0
>  allot 12112
>
> class sfq 40:98 parent 40:
>  (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 1p requeues 0
>  allot -4228
>
> ..... others is SFQ
>
>
>
>
>
> On Thursday 08 January 2009 19:27:31 mysql.jorge@decimal.pt wrote:
>> I asume there's some issue with the NIC Driver os the kernel modules
>> with
>> the 1000Mbit NIC's? am i correct?
>>
>> > An active connection when running this commands:
>> > (now it seams to work, the 200Kbit i defined, are gaving me
>> 22.5/23Kbyte
>> > /sec)
>> >
>
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Denys Fedoryschenko]

on this interface 

Router-Dora /config # ethtool -i eth0
driver: e1000e
version: 0.3.3.3-k6
firmware-version: 2.1-11
bus-info: 0000:04:00.0

Router-Dora /config # ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: pumbag
        Wake-on: g
        Current message level: 0x00000001 (1)
        Link detected: yes


On Thursday 08 January 2009 20:37:16 Jorge Bastos wrote:
> What's your NIC?
> Speed, and driver that you are using.
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Denys Fedoryschenko]

I check, also HFSC not keeping bandwidth accurate.
Seems i am totally lost, what to do.
Any ideas what i can try?

On Thursday 08 January 2009 20:32:16 Denys Fedoryschenko wrote:
> I disable tso/gso and others. No difference at all. Here is actual output,
> after running it for 1-2 minutes.
>
> Cisco
>   30 second output rate 59600000 bits/sec, 32978 packets/sec
> Linux
>
> class htb 1:33 parent 1:10 leaf 33: prio 0 rate 1000Kbit ceil 57000Kbit
> burst 2849b cburst 72838b
>  Sent 63746261 bytes 492929 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 61200bit 48pps backlog 0b 0p requeues 0
>  lended: 491208 borrowed: 1721 giants: 0
>  tokens: 21453 ctokens: 9971
>
> class htb 1:32 parent 1:10 leaf 32: prio 0 rate 4000Kbit ceil 57000Kbit

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jorge Bastos]

Well... i'm out of ideias.
CBQ seems to be working OK without the parameters disables that Jarek told
me to disable.

What can you say about this Jarek?


> I check, also HFSC not keeping bandwidth accurate.
> Seems i am totally lost, what to do.
> Any ideas what i can try?
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Denys Fedoryschenko]

Can 100HZ be issue?
I have hrtimers, tsc now. Tried to change to HPET, nothing changes.


On Thursday 08 January 2009 23:46:04 Jorge Bastos wrote:
> Well... i'm out of ideias.
> CBQ seems to be working OK without the parameters disables that Jarek told
> me to disable.
>
> What can you say about this Jarek?
>
> > I check, also HFSC not keeping bandwidth accurate.
> > Seems i am totally lost, what to do.
> > Any ideas what i can try?

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jorge Bastos]

I've tried before changing this NIC values with ethtool, with 100, 250,
300HZ, with Preempble = server or desktop, and nothing.
Didn't worked in any of those ways.


> Can 100HZ be issue?
> I have hrtimers, tsc now. Tried to change to HPET, nothing changes.
>
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Thu, Jan 08, 2009 at 11:57:13PM +0200, Denys Fedoryschenko wrote:
> Can 100HZ be issue?

Generally, if you need precise scheduling HZ=1000 should be used if
higher cpu usage isn't a problem.

Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Denys Fedoryschenko]

HZ still matter? With tickless kernel and hrtimers?

On Friday 09 January 2009 12:30:30 Jarek Poplawski wrote:
> On Thu, Jan 08, 2009 at 11:57:13PM +0200, Denys Fedoryschenko wrote:
> > Can 100HZ be issue?
>
> Generally, if you need precise scheduling HZ=1000 should be used if
> higher cpu usage isn't a problem.
>
> Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Fri, Jan 09, 2009 at 12:32:09PM +0200, Denys Fedoryschenko wrote:
> HZ still matter? With tickless kernel and hrtimers?

Sure, "jiffies" value which depends on CONFIG_HZ is added/checked for
in many places in networking and elsewhere.

Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jorge Bastos]

For my case this has any matter?


> On Fri, Jan 09, 2009 at 12:32:09PM +0200, Denys Fedoryschenko wrote:
>> HZ still matter? With tickless kernel and hrtimers?
>
> Sure, "jiffies" value which depends on CONFIG_HZ is added/checked for
> in many places in networking and elsewhere.
>
> Jarek P.
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Fri, Jan 09, 2009 at 10:47:11AM -0000, Jorge Bastos wrote:
> For my case this has any matter?
> 

I guess not: it's about better resolution, so e.g. when 2% matters.
And it matters for a desktop box, if you care about responsiveness.

Jarek P.

> 
> > On Fri, Jan 09, 2009 at 12:32:09PM +0200, Denys Fedoryschenko wrote:
> >> HZ still matter? With tickless kernel and hrtimers?
> >
> > Sure, "jiffies" value which depends on CONFIG_HZ is added/checked for
> > in many places in networking and elsewhere.
> >
> > Jarek P.
> >
> 
> 

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Patrick McHardy]

Jarek Poplawski wrote:
> On Fri, Jan 09, 2009 at 10:47:11AM -0000, Jorge Bastos wrote:
>> For my case this has any matter?
>>
> 
> I guess not: it's about better resolution, so e.g. when 2% matters.
> And it matters for a desktop box, if you care about responsiveness.

Just wondering since this thread is very hard to follow with all
the top postings, incorrect timestamps on mails etc. - has there
been a resolution to this problem?

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Mon, Jan 12, 2009 at 08:27:10AM +0100, Patrick McHardy wrote:
> Jarek Poplawski wrote:
>> On Fri, Jan 09, 2009 at 10:47:11AM -0000, Jorge Bastos wrote:
>>> For my case this has any matter?
>>>
>>
>> I guess not: it's about better resolution, so e.g. when 2% matters.
>> And it matters for a desktop box, if you care about responsiveness.
>
> Just wondering since this thread is very hard to follow with all
> the top postings, incorrect timestamps on mails etc. - has there
> been a resolution to this problem?
>

AFAIK, Jorge's CBQ+TBF problem was resolved with turning off TSO, but
it looks like the resolution will be some warning added somewhere...

Denys didn't yet confirm his HTB problem was resolved.

Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jorge Bastos]

>> Just wondering since this thread is very hard to follow with all
>> the top postings, incorrect timestamps on mails etc. - has there
>> been a resolution to this problem?
>>
>
> AFAIK, Jorge's CBQ+TBF problem was resolved with turning off TSO, but
> it looks like the resolution will be some warning added somewhere...
>
> Denys didn't yet confirm his HTB problem was resolved.
>
> Jarek P.
>

Reading this, this warnings must/will be added to TC correcto?

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Mon, Jan 12, 2009 at 09:49:16AM -0000, Jorge Bastos wrote:
...
> Reading this, this warnings must/will be added to TC correcto?
> 

I think it'll be in the kernel.

Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Denys Fedoryschenko]

On Monday 12 January 2009 09:27:10 Patrick McHardy wrote:
> Jarek Poplawski wrote:
> > On Fri, Jan 09, 2009 at 10:47:11AM -0000, Jorge Bastos wrote:
> >> For my case this has any matter?
> >
> > I guess not: it's about better resolution, so e.g. when 2% matters.
> > And it matters for a desktop box, if you care about responsiveness.
>
> Just wondering since this thread is very hard to follow with all
> the top postings, incorrect timestamps on mails etc. - has there
> been a resolution to this problem?

1000HZ made situation much better. But i'm still debugging the case.
Difficult part, there is many small packets, and i didn't found any reference 
how Cisco count packets. I know that HTB counts with Ethernet header, but 
maybe Cisco catching something else.
One thing i can say, HFSC more precise than HTB for now.
HFSC started to be more precise after changing to 1000HZ, but i had system 
crashed yesterday, while applying new rules. Probably it is old bug with 
timers, which we debugged before. It is rare case now, but seems happened 
yesterday.
But anyway, i will try to test today again, and compare with old results.

In numbers:
Before HFSC was reaching 60-61Mbps, when set 57. Now set 60, and now rarely 
reach 61. Why i'm not happy about that, because it is mrtg results, which is 
averaged by 5 minutes. I understand if there can be short bursts for 61 megs, 
but why average(for long period of time) results become higher?
I think misprecision must vary to both side, when it becomes higher for few 
(milli?)seconds, it must become lower also for same time.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Denys Fedoryschenko]

End of my story:
1000HZ solved issue. HTB and HFSC keeps rate MUCH better.
Both looks like perform similar, but because it is real load scenario, i 
cannot measure which one is better.

On Monday 12 January 2009 10:30:15 Denys Fedoryschenko wrote:
> On Monday 12 January 2009 09:27:10 Patrick McHardy wrote:
> > Jarek Poplawski wrote:
> > > On Fri, Jan 09, 2009 at 10:47:11AM -0000, Jorge Bastos wrote:
> > >> For my case this has any matter?
> > >
> > > I guess not: it's about better resolution, so e.g. when 2% matters.
> > > And it matters for a desktop box, if you care about responsiveness.
> >
> > Just wondering since this thread is very hard to follow with all
> > the top postings, incorrect timestamps on mails etc. - has there
> > been a resolution to this problem?
>
> 1000HZ made situation much better. But i'm still debugging the case.
> Difficult part, there is many small packets, and i didn't found any
> reference how Cisco count packets. I know that HTB counts with Ethernet
> header, but maybe Cisco catching something else.
> One thing i can say, HFSC more precise than HTB for now.
> HFSC started to be more precise after changing to 1000HZ, but i had system
> crashed yesterday, while applying new rules. Probably it is old bug with
> timers, which we debugged before. It is rare case now, but seems happened
> yesterday.
> But anyway, i will try to test today again, and compare with old results.
>
> In numbers:
> Before HFSC was reaching 60-61Mbps, when set 57. Now set 60, and now rarely
> reach 61. Why i'm not happy about that, because it is mrtg results, which
> is averaged by 5 minutes. I understand if there can be short bursts for 61
> megs, but why average(for long period of time) results become higher?
> I think misprecision must vary to both side, when it becomes higher for few
> (milli?)seconds, it must become lower also for same time.
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jesper Dangaard Brouer]

On Mon, 12 Jan 2009, Denys Fedoryschenko wrote:

> End of my story:
> 1000HZ solved issue. HTB and HFSC keeps rate MUCH better.
> Both looks like perform similar, but because it is real load scenario, i
> cannot measure which one is better.

If that was you problem, you should consider enabling 
CONFIG_HIGH_RES_TIMERS (and possibly CONFIG_NO_HZ).

Patrick McHardy has at some point made the sched code independent of the 
HZ value, but only when the HIGH_RES_TIMERS are enabled.

Cheers,
   Jesper Brouer

--
-------------------------------------------------------------------
MSc. Master of Computer Science
Dept. of Computer Science, University of Copenhagen
Author of http://www.adsl-optimizer.dk
-------------------------------------------------------------------

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Denys Fedoryschenko]

They are enabled

Router-Dora /config # zcat /proc/config.gz |grep HIGH_RES
CONFIG_HIGH_RES_TIMERS=y
Router-Dora /config # zcat /proc/config.gz |grep HZ
CONFIG_NO_HZ=y


On Monday 12 January 2009 21:58:22 Jesper Dangaard Brouer wrote:
> On Mon, 12 Jan 2009, Denys Fedoryschenko wrote:
> > End of my story:
> > 1000HZ solved issue. HTB and HFSC keeps rate MUCH better.
> > Both looks like perform similar, but because it is real load scenario, i
> > cannot measure which one is better.
>
> If that was you problem, you should consider enabling
> CONFIG_HIGH_RES_TIMERS (and possibly CONFIG_NO_HZ).
>
> Patrick McHardy has at some point made the sched code independent of the
> HZ value, but only when the HIGH_RES_TIMERS are enabled.
>
> Cheers,
>    Jesper Brouer
>
> --
> -------------------------------------------------------------------
> MSc. Master of Computer Science
> Dept. of Computer Science, University of Copenhagen
> Author of http://www.adsl-optimizer.dk
> -------------------------------------------------------------------
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Patrick McHardy]

Denys Fedoryschenko wrote:
> They are enabled
> 
> Router-Dora /config # zcat /proc/config.gz |grep HIGH_RES
> CONFIG_HIGH_RES_TIMERS=y
> Router-Dora /config # zcat /proc/config.gz |grep HZ
> CONFIG_NO_HZ=y

Its odd that HZ=1000 solved the problem despite HR timers being active.
Are you sure they're actually enabled? You should see something like
this in the ringbuffer:

[    0.241972] Switched to high resolution mode on CPU 0
[    0.242708] Switched to high resolution mode on CPU 1

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Stephen Hemminger]

On Tue, 13 Jan 2009 05:33:14 +0100
Patrick McHardy <kaber@trash.net> wrote:

> Denys Fedoryschenko wrote:
> > They are enabled
> > 
> > Router-Dora /config # zcat /proc/config.gz |grep HIGH_RES
> > CONFIG_HIGH_RES_TIMERS=y
> > Router-Dora /config # zcat /proc/config.gz |grep HZ
> > CONFIG_NO_HZ=y
> 
> Its odd that HZ=1000 solved the problem despite HR timers being active.
> Are you sure they're actually enabled? You should see something like
> this in the ringbuffer:
> 
> [    0.241972] Switched to high resolution mode on CPU 0
> [    0.242708] Switched to high resolution mode on CPU 1
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Even with NO_HZ the regular kernel won't schedule timers sooner
than HZ. I believe it caused regressions so it was disabled.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Eric Dumazet]

Stephen Hemminger a écrit :
> On Tue, 13 Jan 2009 05:33:14 +0100
> Patrick McHardy <kaber@trash.net> wrote:
> 
>> Denys Fedoryschenko wrote:
>>> They are enabled
>>>
>>> Router-Dora /config # zcat /proc/config.gz |grep HIGH_RES
>>> CONFIG_HIGH_RES_TIMERS=y
>>> Router-Dora /config # zcat /proc/config.gz |grep HZ
>>> CONFIG_NO_HZ=y
>> Its odd that HZ=1000 solved the problem despite HR timers being active.
>> Are you sure they're actually enabled? You should see something like
>> this in the ringbuffer:
>>
>> [    0.241972] Switched to high resolution mode on CPU 0
>> [    0.242708] Switched to high resolution mode on CPU 1
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> Even with NO_HZ the regular kernel won't schedule timers sooner
> than HZ. I believe it caused regressions so it was disabled.

Are you sure of this Stephen ?

This is not what I see with a CBQ setup, shaping a Gigabit interface with 60 Mb constraint

tc qdisc del dev eth2 root

tc qdisc add dev eth2 root handle 1: cbq avpkt 1000 rate 1000Mbit bandwidth 1000Mbit

tc class add dev eth2 parent 1:  classid 1:1   cbq allot 1500 rate 1000Mbit prio 1 avpkt 1500 bounded

# VOIP : essayons d'espacer un peu les paquets plutot que de les envoyer en burst
tc class add dev eth2 parent 1:1 classid 1:11 cbq allot 1500 rate 60Mbit avpkt 200 bounded

tc filter add dev eth2 protocol ip parent 1: u32 \
                match ip dst 10.170.72.0/24 \
        flowid 1:11

All following RTP frames are sent at once by the producer, yet we can see them leaving machine at the right rate

08:47:59.492059 IP 10.170.74.20.40026 > 10.170.72.109.8270: UDP, length 172
08:47:59.492066 IP 10.170.74.20.40002 > 10.170.72.109.12540: UDP, length 172
08:47:59.492104 IP 10.170.74.20.40016 > 10.170.72.110.8230: UDP, length 172
08:47:59.492109 IP 10.170.74.20.40044 > 10.170.72.109.4040: UDP, length 172
08:47:59.492136 IP 10.170.74.20.40038 > 10.170.72.109.13780: UDP, length 172
08:47:59.492163 IP 10.170.74.20.40060 > 10.170.72.110.7670: UDP, length 172
08:47:59.492191 IP 10.170.74.20.40066 > 10.170.72.109.8860: UDP, length 172
08:47:59.492219 IP 10.170.74.20.40020 > 10.170.72.110.12160: UDP, length 172
08:47:59.492246 IP 10.170.74.20.40078 > 10.170.72.110.12010: UDP, length 172
08:47:59.492273 IP 10.170.74.20.40010 > 10.170.72.109.12670: UDP, length 172
08:47:59.492306 IP 10.170.74.20.40018 > 10.170.72.109.4340: UDP, length 172
08:47:59.492333 IP 10.170.74.20.40008 > 10.170.72.109.7450: UDP, length 172
08:47:59.492361 IP 10.170.74.20.40024 > 10.170.72.110.10300: UDP, length 172
08:47:59.492388 IP 10.170.74.20.40006 > 10.170.72.109.13390: UDP, length 172
08:47:59.492389 IP 10.170.74.20.40036 > 10.170.72.109.9040: UDP, length 172
08:47:59.492416 IP 10.170.74.20.40042 > 10.170.72.110.5420: UDP, length 172
08:47:59.492444 IP 10.170.74.20.40040 > 10.170.72.109.8080: UDP, length 172
08:47:59.492470 IP 10.170.74.20.40048 > 10.170.72.110.17010: UDP, length 172
08:47:59.492502 IP 10.170.74.20.40032 > 10.170.72.109.8340: UDP, length 172
08:47:59.492529 IP 10.170.74.20.40004 > 10.170.72.109.8420: UDP, length 172

2.6.27.10 kernel, HZ=1000, HighRES timers on

Switched to high resolution mode on CPU 0
Switched to high resolution mode on CPU 5
Switched to high resolution mode on CPU 4
Switched to high resolution mode on CPU 7
Switched to high resolution mode on CPU 1
Switched to high resolution mode on CPU 3
Switched to high resolution mode on CPU 6
Switched to high resolution mode on CPU 2

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Michal Soltys]

Stephen Hemminger wrote:
> On Tue, 13 Jan 2009 05:33:14 +0100
> 
> Even with NO_HZ the regular kernel won't schedule timers sooner
> than HZ. I believe it caused regressions so it was disabled.

While making manpage(s) for hfsc (I'll post it soon), I've done some 
tests - setting trivial hfsc class, essentially as a tbf emulator, 
produces impressive amount of timer interrupts.

tc qdisc add dev eth0 root handle 1:0 hfsc default 1
tc class add dev eth0 parent 1:0 classid 1:1 hfsc rt m2 300mbit

nc -u dst.host.com 54321 </dev/zero
nc -l -p 54321 >/dev/null

319: 42124229   0  HPET_MSI-edge  hpet2 (before)
319: 42436214   0  HPET_MSI-edge  hpet2 (after ~10s.)

That's over 30,000 per second. CPU load is another thing in this case 
(still easily tolerable by cheap dual core amd cpu), but interrupt rate 
goes easily far beyond HZ mark. Of course both 'tickless system' and 'hr 
timers' must be enabled.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On 13-01-2009 05:33, Patrick McHardy wrote:
...
> Its odd that HZ=1000 solved the problem despite HR timers being active.
> Are you sure they're actually enabled? You should see something like
> this in the ringbuffer:

If packets are received in netif_rx_action() for ~2 jiffies why is it
odd they will be xmitted differently after changing HZ?

Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Denys Fedoryschenko]

On Tuesday 13 January 2009 06:33:14 Patrick McHardy wrote:
>
> Its odd that HZ=1000 solved the problem despite HR timers being active.
> Are you sure they're actually enabled? You should see something like
> this in the ringbuffer:
>
> [    0.241972] Switched to high resolution mode on CPU 0
> [    0.242708] Switched to high resolution mode on CPU 1
> --
I had nmi_watchdog enabled, and kernel log was too short to check that 
message. But i think you are right, because of nmi_watchdog it was not in 
high resolution.

Plus my kernel is spammed by messages 
[  105.417236] UDP: bad checksum. From 58.137.24.102:33333 to 
213.254.233.11:42741 ulen 106
[  105.527209] UDP: bad checksum. From 200.7.99.22:33132 to 
213.254.233.11:61520 ulen 111
[  130.498537] UDP: bad checksum. From 210.213.100.122:33890 to 
213.254.233.10:64060 ulen 111
[  132.974111] UDP: bad checksum. From 189.144.209.72:2028 to 
213.254.233.11:1466 ulen 8259
[  133.499172] UDP: bad checksum. From 66.41.236.245:16496 to 
213.254.233.11:15055 ulen 71
[  140.826429] UDP: bad checksum. From 220.165.176.236:16001 to 
213.254.233.10:9422 ulen 297

Is net.core.warnings = 0 will remove them?
Their amount really annoying on router.

I will remove nmi_watchdog, apply Jarek patches to avoid crashes which i had 
before, and will check how much is precise shaper at peak time (after 9-10 
hours).  For now it will be 1000 HZ (not 100), 

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jorge Bastos]

Hi Patrick,

>>
>> I guess not: it's about better resolution, so e.g. when 2% matters.
>> And it matters for a desktop box, if you care about responsiveness.
>
> Just wondering since this thread is very hard to follow with all
> the top postings, incorrect timestamps on mails etc. - has there
> been a resolution to this problem?
>

Top posting are my fault, sorry!
Yes for me got solved with disable GSO, TSO and SG.
With them disabled, CBQ+TCF works as expected.
My 100Mbit NIC always worked because this values were off by default,
that's Why i start posting that it worked with a 100Mbit NIC and not with
a 1000Mbit NIC, the reason were the values were off dby default.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Denys Fedoryschenko]

Jorge - i guess your traffic is locally generated (on same PC where is shaper, 
this traffic generated), thats why shaping was not precise, with enabled 
GSO/TSO. I am right?

My conclusion based on:
eth0      Link encap:Ethernet  HWaddr 00:13:d4:68:c6:33
          inet addr:195.23.114.76  Bcast:195.23.114.79  Mask:255.255.255.248

and
/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 100 u32 match ip
src 195.23.114.76 match ip sport 80 0xffff classid 1
:1280



My traffic is "transit", means just passing router. I guess if it is correct, 
there must be statement about difference in locally generated traffic and 
transit traffic. TSO/GSO offloading i guess not done for "passing thru" 
traffic.

On Monday 12 January 2009 11:48:08 Jorge Bastos wrote:
> Hi Patrick,
>
> >> I guess not: it's about better resolution, so e.g. when 2% matters.
> >> And it matters for a desktop box, if you care about responsiveness.
> >
> > Just wondering since this thread is very hard to follow with all
> > the top postings, incorrect timestamps on mails etc. - has there
> > been a resolution to this problem?
>
> Top posting are my fault, sorry!
> Yes for me got solved with disable GSO, TSO and SG.
> With them disabled, CBQ+TCF works as expected.
> My 100Mbit NIC always worked because this values were off by default,
> that's Why i start posting that it worked with a 100Mbit NIC and not with
> a 1000Mbit NIC, the reason were the values were off dby default.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jorge Bastos]

> Jorge - i guess your traffic is locally generated (on same PC where is
> shaper,
> this traffic generated), thats why shaping was not precise, with enabled
> GSO/TSO. I am right?
>
> My conclusion based on:
> eth0      Link encap:Ethernet  HWaddr 00:13:d4:68:c6:33
>           inet addr:195.23.114.76  Bcast:195.23.114.79
>  Mask:255.255.255.248
>
> and
> /sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 100 u32 match ip
> src 195.23.114.76 match ip sport 80 0xffff classid 1
> :1280
>
>
>
> My traffic is "transit", means just passing router. I guess if it is
> correct,
> there must be statement about difference in locally generated traffic and
> transit traffic. TSO/GSO offloading i guess not done for "passing thru"
> traffic.
>

Denys,
Not correct, my traffic is transit as you say.
That is a public IP and that traffic is for public access only.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Wed, Jan 14, 2009 at 07:37:00PM -0000, Jorge Bastos wrote:
...
> > My traffic is "transit", means just passing router. I guess if it is
> > correct,
> > there must be statement about difference in locally generated traffic and
> > transit traffic. TSO/GSO offloading i guess not done for "passing thru"
> > traffic.
> >
> 
> Denys,
> Not correct, my traffic is transit as you say.
> That is a public IP and that traffic is for public access only.
> 

Yes, TSO/GSO affect shaping of any traffic going through (usually out)
a device with these options enabled. The main issue here was TBF which
treats it different (drops too big packets) than e.g. HTB.

Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Thu, Jan 08, 2009 at 08:32:16PM +0200, Denys Fedoryschenko wrote:
> I disable tso/gso and others. No difference at all. Here is actual output, 
> after running it for 1-2 minutes.
> 
> Cisco
>   30 second output rate 59600000 bits/sec, 32978 packets/sec
> Linux
> 
...
> class htb 1:10 root rate 57000Kbit ceil 57000Kbit burst 72838b cburst 72838b
>  Sent 75203300117 bytes 328680653 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 58008Kbit 32891pps backlog 0b 0p requeues 0

Alas you forgot to add "tc -s qdisc" output, so there could be
something more, but what I can see here doesn't look terribly wrong.
The rate is exceeded less than 2%, so it seems some tweaking is
needed. My proposal is to start with setting lower cburst params for
all classes e.g. half of the current values (if no difference go
lower). If it doesn't help send new stats.

Thanks,
Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jorge Bastos]

The below info is for the other user correct?

I've closed my bug report:

http://bugzilla.kernel.org/show_bug.cgi?id=12364

Is there's something you want me to change just say.
Thanks for all Jarek.
Jorge,

> On Thu, Jan 08, 2009 at 08:32:16PM +0200, Denys Fedoryschenko wrote:
>> I disable tso/gso and others. No difference at all. Here is actual
>> output,
>> after running it for 1-2 minutes.
>>
>> Cisco
>>   30 second output rate 59600000 bits/sec, 32978 packets/sec
>> Linux
>>
> ....
>> class htb 1:10 root rate 57000Kbit ceil 57000Kbit burst 72838b cburst
>> 72838b
>>  Sent 75203300117 bytes 328680653 pkt (dropped 0, overlimits 0 requeues
>> 0)
>>  rate 58008Kbit 32891pps backlog 0b 0p requeues 0
>
> Alas you forgot to add "tc -s qdisc" output, so there could be
> something more, but what I can see here doesn't look terribly wrong.
> The rate is exceeded less than 2%, so it seems some tweaking is
> needed. My proposal is to start with setting lower cburst params for
> all classes e.g. half of the current values (if no difference go
> lower). If it doesn't help send new stats.
>
> Thanks,
> Jarek P.
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Fri, Jan 09, 2009 at 09:51:59AM -0000, Jorge Bastos wrote:
> The below info is for the other user correct?
> 
> I've closed my bug report:
> 
> http://bugzilla.kernel.org/show_bug.cgi?id=12364
> 

I think it's OK. I hope there will be soon added some warning to
qdiscs to avoid such problems.

Thanks,
Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jorge Bastos]

I'll be monotoring it to see when something is done about it.
Thanks again Jarek.

PS: Now i know why my 100Mbit card ever worked, they have by default this
values set to off (of they don't suport it).

Jorge,


> On Fri, Jan 09, 2009 at 09:51:59AM -0000, Jorge Bastos wrote:
>> The below info is for the other user correct?
>>
>> I've closed my bug report:
>>
>> http://bugzilla.kernel.org/show_bug.cgi?id=12364
>>
>
> I think it's OK. I hope there will be soon added some warning to
> qdiscs to avoid such problems.
>
> Thanks,
> Jarek P.
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Thu, Jan 08, 2009 at 05:27:31PM -0000, mysql.jorge@decimal.pt wrote:
> I asume there's some issue with the NIC Driver os the kernel modules with
> the 1000Mbit NIC's? am i correct?

In this case the issue is packet schedulers usually don't work with
gso/tso enabled, at least without some tweaking with mtu. Some qdisc
get inacurrate, others (like tbf) can drop too big packets (have a
look at "dropped" stats).

BTW, Jorge, since my tries to Cc bugzilla failed, please add some
comment to your bug report (I guess it could be closed too).

Thanks
Jarek P.

> 
> 
> > An active connection when running this commands:
> > (now it seams to work, the 200Kbit i defined, are gaving me 22.5/23Kbyte
> > /sec)
> >
> > Did you catch it?
> >
> >
> > lira:~# ethtool -k eth0
> > Offload parameters for eth0:
> > Cannot get device flags: Operation not supported
> > rx-checksumming: on
> > tx-checksumming: on
> > scatter-gather: off
> > tcp segmentation offload: off
> > udp fragmentation offload: off
> > generic segmentation offload: off
> > large receive offload: off
> > lira:~# tc -s qdisc show dev eth0
> > qdisc cbq 1: root rate 12500000bps (bounded,isolated) prio no-transmit
> >  Sent 5336353 bytes 5737 pkts (dropped 16, overlimits 1382)
> >  backlog 8p
> >   borrowed 0 overactions 0 avgidle 78 undertime 0
> >
> >  qdisc tbf 1280: parent 1:1280 rate 25000bps burst 10Kb lat 209.7ms
> >  Sent 441812 bytes 310 pkts (dropped 16, overlimits 2215)
> >  backlog 8p
> >
> > lira:~# tc -s class show dev eth0
> > class cbq 1: root rate 12500000bps (bounded,isolated) prio no-transmit
> >  Sent 5128779 bytes 5827 pkts (dropped 0, overlimits 0)
> >   borrowed 0 overactions 0 avgidle 78 undertime 0
> > class cbq 1:1280 parent 1: leaf 1280: rate 25000bps (bounded,isolated)
> > prio 5
> >  Sent 811784 bytes 568 pkts (dropped 22, overlimits 0)
> >  backlog 9p
> >   borrowed 0 overactions 0 avgidle 4507 undertime 0
> > class tbf 1280:1 parent 1280: [UNKNOWN]
> >  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> >
> >
> >
> >
> >
> >> Could you try to check this with scatter-gather and tcp segmentation
> >> offload off too? If there is no difference please send the stats
> >> again both for working and non-working case.
> >>
> >> Thanks,
> >> Jarek P.
> >>
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> 

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jorge Bastos]

Going to.
But tell me, for this, all 3 params should be disabled, correct?

> On Thu, Jan 08, 2009 at 05:27:31PM -0000, mysql.jorge@decimal.pt wrote:
>> I asume there's some issue with the NIC Driver os the kernel modules
>> with
>> the 1000Mbit NIC's? am i correct?
>
> In this case the issue is packet schedulers usually don't work with
> gso/tso enabled, at least without some tweaking with mtu. Some qdisc
> get inacurrate, others (like tbf) can drop too big packets (have a
> look at "dropped" stats).
>
> BTW, Jorge, since my tries to Cc bugzilla failed, please add some
> comment to your bug report (I guess it could be closed too).
>
> Thanks
> Jarek P.
>
>>
>>
>> > An active connection when running this commands:
>> > (now it seams to work, the 200Kbit i defined, are gaving me
>> 22.5/23Kbyte
>> > /sec)
>> >
>> > Did you catch it?
>> >
>> >
>> > lira:~# ethtool -k eth0
>> > Offload parameters for eth0:
>> > Cannot get device flags: Operation not supported
>> > rx-checksumming: on
>> > tx-checksumming: on
>> > scatter-gather: off
>> > tcp segmentation offload: off
>> > udp fragmentation offload: off
>> > generic segmentation offload: off
>> > large receive offload: off
>> > lira:~# tc -s qdisc show dev eth0
>> > qdisc cbq 1: root rate 12500000bps (bounded,isolated) prio no-transmit
>> >  Sent 5336353 bytes 5737 pkts (dropped 16, overlimits 1382)
>> >  backlog 8p
>> >   borrowed 0 overactions 0 avgidle 78 undertime 0
>> >
>> >  qdisc tbf 1280: parent 1:1280 rate 25000bps burst 10Kb lat 209.7ms
>> >  Sent 441812 bytes 310 pkts (dropped 16, overlimits 2215)
>> >  backlog 8p
>> >
>> > lira:~# tc -s class show dev eth0
>> > class cbq 1: root rate 12500000bps (bounded,isolated) prio no-transmit
>> >  Sent 5128779 bytes 5827 pkts (dropped 0, overlimits 0)
>> >   borrowed 0 overactions 0 avgidle 78 undertime 0
>> > class cbq 1:1280 parent 1: leaf 1280: rate 25000bps (bounded,isolated)
>> > prio 5
>> >  Sent 811784 bytes 568 pkts (dropped 22, overlimits 0)
>> >  backlog 9p
>> >   borrowed 0 overactions 0 avgidle 4507 undertime 0
>> > class tbf 1280:1 parent 1280: [UNKNOWN]
>> >  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
>> >
>> >
>> >
>> >
>> >
>> >> Could you try to check this with scatter-gather and tcp segmentation
>> >> offload off too? If there is no difference please send the stats
>> >> again both for working and non-working case.
>> >>
>> >> Thanks,
>> >> Jarek P.
>> >>
>> >
>> >
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe netdev" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >
>>
>>
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Fri, Jan 09, 2009 at 09:39:44AM -0000, Jorge Bastos wrote:
> Going to.
> But tell me, for this, all 3 params should be disabled, correct?

Yes, but only if you are doing packet scheduling on an interface.
Otherwise they could be needed to get full performance from new
network cards.

Jarek P. 

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Thu, Jan 08, 2009 at 05:27:31PM -0000, mysql.jorge@decimal.pt wrote:
...
> > lira:~# tc -s qdisc show dev eth0
> > qdisc cbq 1: root rate 12500000bps (bounded,isolated) prio no-transmit
> >  Sent 5336353 bytes 5737 pkts (dropped 16, overlimits 1382)
-------------------------------------------------------------> ^^^^^^^^^

BTW, I wonder why you get this output (very) old way - without "requeues"
stats. Could you tell your iproute2 and kernel (or distro) version?

Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jorge Bastos]

Well, Debian using SID.
iproute (that's the last for SID):
ii  iproute                         20080725-2                 networking
and traffic control tools

lira:~# uname -a
Linux lira 2.6.28 #1 SMP Thu Dec 25 12:31:23 WET 2008 i686 GNU/Linux

maybe i should ask the iproute package mainteiner to update it no?



> On Thu, Jan 08, 2009 at 05:27:31PM -0000, mysql.jorge@decimal.pt wrote:
> ....
>> > lira:~# tc -s qdisc show dev eth0
>> > qdisc cbq 1: root rate 12500000bps (bounded,isolated) prio no-transmit
>> >  Sent 5336353 bytes 5737 pkts (dropped 16, overlimits 1382)
> -------------------------------------------------------------> ^^^^^^^^^
>
> BTW, I wonder why you get this output (very) old way - without "requeues"
> stats. Could you tell your iproute2 and kernel (or distro) version?
>
> Jarek P.
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Fri, Jan 09, 2009 at 09:42:20AM -0000, Jorge Bastos wrote:
> Well, Debian using SID.
> iproute (that's the last for SID):
> ii  iproute                         20080725-2                 networking
> and traffic control tools
> 
> lira:~# uname -a
> Linux lira 2.6.28 #1 SMP Thu Dec 25 12:31:23 WET 2008 i686 GNU/Linux
> 
> maybe i should ask the iproute package mainteiner to update it no?
> 

Very strange. The versions look OK. Is it all distro's or
recompiled? If the latter, could you try distro versions?

Jarek P.

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jorge Bastos]

Iproute is from distro, the kernel i compiled it by hand as always.
GCC 4.3.1 (could this interfer since it's very eary gcc version?)


> On Fri, Jan 09, 2009 at 09:42:20AM -0000, Jorge Bastos wrote:
>> Well, Debian using SID.
>> iproute (that's the last for SID):
>> ii  iproute                         20080725-2
>> networking
>> and traffic control tools
>>
>> lira:~# uname -a
>> Linux lira 2.6.28 #1 SMP Thu Dec 25 12:31:23 WET 2008 i686 GNU/Linux
>>
>> maybe i should ask the iproute package mainteiner to update it no?
>>
>
> Very strange. The versions look OK. Is it all distro's or
> recompiled? If the latter, could you try distro versions?
>
> Jarek P.
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Fri, Jan 09, 2009 at 10:15:32AM -0000, Jorge Bastos wrote:
> Iproute is from distro, the kernel i compiled it by hand as always.
> GCC 4.3.1 (could this interfer since it's very eary gcc version?)

I don't now, could be also something about kernel headers. Distro
versions are more supposed to work, I hope.

Jarek P.

> 
> 
> > On Fri, Jan 09, 2009 at 09:42:20AM -0000, Jorge Bastos wrote:
> >> Well, Debian using SID.
> >> iproute (that's the last for SID):
> >> ii  iproute                         20080725-2
> >> networking
> >> and traffic control tools
> >>
> >> lira:~# uname -a
> >> Linux lira 2.6.28 #1 SMP Thu Dec 25 12:31:23 WET 2008 i686 GNU/Linux
> >>
> >> maybe i should ask the iproute package mainteiner to update it no?
> >>
> >
> > Very strange. The versions look OK. Is it all distro's or
> > recompiled? If the latter, could you try distro versions?
> >
> > Jarek P.
> >
> 
> 

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jorge Bastos]

Hum... i have on that machine:

ii  linux-libc-dev                  2.6.26-12                  Linux
support headers for userspace developm




> On Fri, Jan 09, 2009 at 10:15:32AM -0000, Jorge Bastos wrote:
>> Iproute is from distro, the kernel i compiled it by hand as always.
>> GCC 4.3.1 (could this interfer since it's very eary gcc version?)
>
> I don't now, could be also something about kernel headers. Distro
> versions are more supposed to work, I hope.
>
> Jarek P.
>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jorge Bastos]

Ops i missread your post.
I always prefer to compile my own kernels.


> On Fri, Jan 09, 2009 at 10:15:32AM -0000, Jorge Bastos wrote:
>> Iproute is from distro, the kernel i compiled it by hand as always.
>> GCC 4.3.1 (could this interfer since it's very eary gcc version?)
>
> I don't now, could be also something about kernel headers. Distro
> versions are more supposed to work, I hope.
>
> Jarek P.
>
>>

Re: [BUG #12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[mysql.jorge]

My rules, iproute, or the kernel QoS modules?


> Hmm... Actually, it's probably about config: I'm not cbq user or
> expert, but AFAIK it depends on precise info about the real link
> rate, so it probably needs an update to 1000Mbit.
>
> Jarek P.
>

Re: [BUG 12364] Re: HTB - very bad precision? HFSC works fine! 2.6.28

[mysql.jorge]

GSO Was on.
So i changed it to off.
---
ethtool -K eth0 gso off
---

lira:~# ethtool -h|grep -i version
ethtool version 6

but still not shaping OK, now i'm trying with 10Kbit, the value goes down
to 1/2, sometimes 3Kbit/sec





> On 08-01-2009 00:42, mysql.jorge@decimal.pt wrote:
>> Is there anyway to debug this?
>
> Could you try: ethtool -k eth0
> with quite fresh ethtool, and to turn off GSO etc., if it's on?
>
> Jarek P.
>

Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On 07-01-2009 23:02, Denys Fedoryschenko wrote:
> I notice that our uplink ISP complaining, that we are exceeding our UP rate.
> 
> I have configured at HTB 60Mbps, but he is actually receiving bursts up to 
> 64Mbps.
> 
> I tried to lower down HTB values to 57Mbps, and even on Cisco switch interface 
> counter(it is averaging 30 seconds of data i guess, to calculate bandwidth) - 
> it shows 60.1 - 60.4.
> 
> Funny thing, root class in HTB shows 57.1 - 57.3 Mbps. So maybe byte counting 
> is wrong in HTB?

Does this "2.6.28" in the subject mean it was better before or not
tested? Could you do a few snapshots with: tc -s qdisc show dev $DEV

> 
> I have also "default filter" 
> tc filter add dev ${DEV} parent 1:0 protocol ip prio 800 u32 match ip src 
> 0.0.0.0/0 flowid 1:50

I can see "tc qdisc add ... default 1000" in your config; is there a
class with this classid? Is there something in "direct_packets_stats"?
Maybe: "ethtool -k $DEV", BTW?

Jarek P.

Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Denys Fedoryschenko]

I will check all this things, as soon as traffic reach peak (after 6 hours i 
think).

I cannot reverse to old kernel, but it looks like previous kernel was not much 
better. But sadly i cannot say for sure. I think previous kernels had issue, 
because i had to move to HFSC on our satellite backbone from HTB in 2.6.25, 
because it is very sensitive to shaping precision. But there i didn't had 
access to switch, to be sure it is shaping precision issue or something else.

On Thursday 08 January 2009 11:27:20 Jarek Poplawski wrote:
> On 07-01-2009 23:02, Denys Fedoryschenko wrote:
> > I notice that our uplink ISP complaining, that we are exceeding our UP
> > rate.
> >
> > I have configured at HTB 60Mbps, but he is actually receiving bursts up
> > to 64Mbps.
> >
> > I tried to lower down HTB values to 57Mbps, and even on Cisco switch
> > interface counter(it is averaging 30 seconds of data i guess, to
> > calculate bandwidth) - it shows 60.1 - 60.4.
> >
> > Funny thing, root class in HTB shows 57.1 - 57.3 Mbps. So maybe byte
> > counting is wrong in HTB?
>
> Does this "2.6.28" in the subject mean it was better before or not
> tested? Could you do a few snapshots with: tc -s qdisc show dev $DEV
>
> > I have also "default filter"
> > tc filter add dev ${DEV} parent 1:0 protocol ip prio 800 u32 match ip src
> > 0.0.0.0/0 flowid 1:50
>
> I can see "tc qdisc add ... default 1000" in your config; is there a
> class with this classid? Is there something in "direct_packets_stats"?
> Maybe: "ethtool -k $DEV", BTW?
>
> Jarek P.

Re: HTB - very bad precision? HFSC works fine! 2.6.28

[Jarek Poplawski]

On Thu, Jan 08, 2009 at 11:34:41AM +0200, Denys Fedoryschenko wrote:
> I will check all this things, as soon as traffic reach peak (after 6 hours i 
> think).
...
> On Thursday 08 January 2009 11:27:20 Jarek Poplawski wrote:
...
> > tested? Could you do a few snapshots with: tc -s qdisc show dev $DEV

I forgot to mention: tc -s class show dev $DEV
as well.

Thanks,
Jarek P.