[LARTC] htb overrate with 2.6.16

[LARTC] htb overrate with 2.6.16

[Yanko Kaneti]

Hi

Here is something that worked with with 2.6.10-1.771_FC2smp and stopped
working when I upgraded to 2.6.16-1.2069_FC4smp.
These are fedora kernels and the network controller is an Intel Gbit
(e1000) running a 100 Mbps Full Duplex.
Don't know how or if this matters but the 2.6.10 kernel has
CONFIG_X86_HZ\x1000 and the 2.6.16 has CONFIG_HZ%0

The idea is to just shape to , say 2Mbit, a certain kind of traffic
everything else should goes unshaped.

# tc qdisc add dev eth0 root handle 1: htb default 20
# tc class add dev eth0 parent 1: classid 1:2 htb rate 2Mbit
# tc qdisc add dev eth0 parent 1:2 sfq perturb 10
# tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 50 fw flowid 1:2

This was working as expected with 2.6.10
I've tried creating a proper 1:1 100Mbit parent to be the default but it
didn't help. And it was working fine without it on 2.6.10

With the 2.6.16 kernel I get results like

# tc -s -d class show dev eth0
class htb 1:2 root leaf 800b: prio 0 quantum 25000 rate 2000Kbit ceil 2000Kbit burst 2600b/8 mpu 0b overhead 0b cburst 2600b/8 mpu 0b overhead 0b level 0
 Sent 189796883 bytes 20626 pkt (dropped 0, overlimits 0 requeues 0)
 rate 3484Kbit 45pps backlog 0b 0p requeues 0
 lended: 20627 borrowed: 0 giants: 30926
 tokens: -9768 ctokens: -9768

As you can see the correctly calculated rate is way above the ceiling
and I've seen it go as fast as this certain type of traffic can go
(something like 8-10Mbits), i.e. the class doesn't seem to shape at all.
No drops or overlimits.

FWIW the cbq in 2.6.10 kernel was working fine as well until the
controller changed from a 100Mbit it a Gbit one and thats why I switched
to htb. The cbq in 2.6.16 is still similarly broken (in a different than
htb way)


Thanks in advance for any insight on this.
Yanko

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] htb overrate with 2.6.16

[Andy Furniss]

Yanko Kaneti wrote:
> Hi
> 
> Here is something that worked with with 2.6.10-1.771_FC2smp and stopped
> working when I upgraded to 2.6.16-1.2069_FC4smp.
> These are fedora kernels and the network controller is an Intel Gbit
> (e1000) running a 100 Mbps Full Duplex.
> Don't know how or if this matters but the 2.6.10 kernel has
> CONFIG_X86_HZ\x1000 and the 2.6.16 has CONFIG_HZ%0
> 
> The idea is to just shape to , say 2Mbit, a certain kind of traffic
> everything else should goes unshaped.
> 
> # tc qdisc add dev eth0 root handle 1: htb default 20

Why default 20 - if you don't have 1:20 it would be better to use 
default 0 which is unshaped and is the default - so ommitting default is 
the same - unclassifed traffic goes through unshaped.

> # tc class add dev eth0 parent 1: classid 1:2 htb rate 2Mbit
> # tc qdisc add dev eth0 parent 1:2 sfq perturb 10
> # tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 50 fw flowid 1:2
> 
> This was working as expected with 2.6.10
> I've tried creating a proper 1:1 100Mbit parent to be the default but it
> didn't help. And it was working fine without it on 2.6.10
> 
> With the 2.6.16 kernel I get results like
> 
> # tc -s -d class show dev eth0
> class htb 1:2 root leaf 800b: prio 0 quantum 25000 rate 2000Kbit ceil 2000Kbit burst 2600b/8 mpu 0b overhead 0b cburst 2600b/8 mpu 0b overhead 0b level 0
>  Sent 189796883 bytes 20626 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 3484Kbit 45pps backlog 0b 0p requeues 0
>  lended: 20627 borrowed: 0 giants: 30926

The giants are the problem - if you specify mtu XXXXX on 1:2 class it 
should work.

Or you could consider setting mtu on nic to 1500 if that is practical 
for you ie. this traffic is headed somewhere that is going to frag it 
down to 1500 anyway.

Andy.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] htb overrate with 2.6.16

[Yanko Kaneti]

On Sun, 2006-04-16 at 00:13 +0100, Andy Furniss wrote:
> Yanko Kaneti wrote:
> > Hi
> > 
> > Here is something that worked with with 2.6.10-1.771_FC2smp and stopped
> > working when I upgraded to 2.6.16-1.2069_FC4smp.
> > These are fedora kernels and the network controller is an Intel Gbit
> > (e1000) running a 100 Mbps Full Duplex.
> > Don't know how or if this matters but the 2.6.10 kernel has
> > CONFIG_X86_HZ\x1000 and the 2.6.16 has CONFIG_HZ%0
> > 
> > The idea is to just shape to , say 2Mbit, a certain kind of traffic
> > everything else should goes unshaped.
> > 
> > # tc qdisc add dev eth0 root handle 1: htb default 20
> 
> Why default 20 - if you don't have 1:20 it would be better to use 
> default 0 which is unshaped and is the default - so ommitting default is 
> the same - unclassifed traffic goes through unshaped.

No reason. I obviously missed the explanation for the 0 class. Will omit
default in the future.

> > # tc class add dev eth0 parent 1: classid 1:2 htb rate 2Mbit
> > # tc qdisc add dev eth0 parent 1:2 sfq perturb 10
> > # tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 50 fw flowid 1:2
> > 
> > This was working as expected with 2.6.10
> > I've tried creating a proper 1:1 100Mbit parent to be the default but it
> > didn't help. And it was working fine without it on 2.6.10
> > 
> > With the 2.6.16 kernel I get results like
> > 
> > # tc -s -d class show dev eth0
> > class htb 1:2 root leaf 800b: prio 0 quantum 25000 rate 2000Kbit ceil 2000Kbit burst 2600b/8 mpu 0b overhead 0b cburst 2600b/8 mpu 0b overhead 0b level 0
> >  Sent 189796883 bytes 20626 pkt (dropped 0, overlimits 0 requeues 0)
> >  rate 3484Kbit 45pps backlog 0b 0p requeues 0
> >  lended: 20627 borrowed: 0 giants: 30926
> 
> The giants are the problem - if you specify mtu XXXXX on 1:2 class it 
> should work.

> Or you could consider setting mtu on nic to 1500 if that is practical 
> for you ie. this traffic is headed somewhere that is going to frag it 
> down to 1500 anyway.

Setting mtu 16500  for the class fixed it. But I wonder where did these
giants come from in the first place? The mtu of the interface is and was
1500. Or so ifconfig and ip link tell me. Or this is some other mtu we
are talking about...


Thanks
Yanko

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] htb overrate with 2.6.16

[Andy Furniss]

Yanko Kaneti wrote:

> Setting mtu 16500  for the class fixed it. But I wonder where did these
> giants come from in the first place? The mtu of the interface is and was
> 1500. Or so ifconfig and ip link tell me. Or this is some other mtu we
> are talking about...

Hmm I didn't expect that - maybe there is some problem with the nic 
drivers not obeying kernel - is there any tso offload etc. at work here ?

Andy.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] htb overrate with 2.6.16

[Yanko Kaneti]

On Sun, 2006-04-16 at 03:03 +0100, Andy Furniss wrote: 
> Yanko Kaneti wrote:
> 
> > Setting mtu 16500  for the class fixed it. But I wonder where did these
> > giants come from in the first place? The mtu of the interface is and was
> > 1500. Or so ifconfig and ip link tell me. Or this is some other mtu we
> > are talking about...
> 
> Hmm I didn't expect that - maybe there is some problem with the nic 
> drivers not obeying kernel - is there any tso offload etc. at work here ?

Yes and its on by default. The interface mtu still says 1500.
I've tried deleting and attaching the qdisc+class (without explicit
large mtu) with both tso on (ethtool -K eth0 tso on) and tso off , it
doesnt seem to matter - giants appear in both cases. 
With large mtu for the class no giants with both tso on and off.

Yanko

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] htb overrate with 2.6.16

[Andy Furniss]

Yanko Kaneti wrote:
> On Sun, 2006-04-16 at 03:03 +0100, Andy Furniss wrote: 
> 
>>Yanko Kaneti wrote:
>>
>>
>>>Setting mtu 16500  for the class fixed it. But I wonder where did these
>>>giants come from in the first place? The mtu of the interface is and was
>>>1500. Or so ifconfig and ip link tell me. Or this is some other mtu we
>>>are talking about...
>>
>>Hmm I didn't expect that - maybe there is some problem with the nic 
>>drivers not obeying kernel - is there any tso offload etc. at work here ?
> 
> 
> Yes and its on by default. The interface mtu still says 1500.
> I've tried deleting and attaching the qdisc+class (without explicit
> large mtu) with both tso on (ethtool -K eth0 tso on) and tso off , it
> doesnt seem to matter - giants appear in both cases. 
> With large mtu for the class no giants with both tso on and off.
> 
> Yanko
> 
> 

I think you need to ask fedora or intel driver maintainer about this. 
AIUI tso is not in vanilla kernels and the patches are quite invasive.

Andy.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] htb overrate with 2.6.16

[Yanko Kaneti]

On Sun, 2006-04-16 at 19:40 +0100, Andy Furniss wrote:
> Yanko Kaneti wrote:
> > On Sun, 2006-04-16 at 03:03 +0100, Andy Furniss wrote: 
> > 
> >>Yanko Kaneti wrote:
> >>
> >>
> >>>Setting mtu 16500  for the class fixed it. But I wonder where did these
> >>>giants come from in the first place? The mtu of the interface is and was
> >>>1500. Or so ifconfig and ip link tell me. Or this is some other mtu we
> >>>are talking about...
> >>
> >>Hmm I didn't expect that - maybe there is some problem with the nic 
> >>drivers not obeying kernel - is there any tso offload etc. at work here ?
> > 
> > 
> > Yes and its on by default. The interface mtu still says 1500.
> > I've tried deleting and attaching the qdisc+class (without explicit
> > large mtu) with both tso on (ethtool -K eth0 tso on) and tso off , it
> > doesnt seem to matter - giants appear in both cases. 
> > With large mtu for the class no giants with both tso on and off.
> > 
> 
> I think you need to ask fedora or intel driver maintainer about this. 
> AIUI tso is not in vanilla kernels and the patches are quite invasive.

Well, as much as google tells me TSO has been in the kernel and enabled
since 2.5.33 and e1000 was the first driver to support it. 
The FC4 2.6.16 kernel doesn't have any tso related patches as can be
seen here http://cvs.fedora.redhat.com/viewcvs/rpms/kernel/FC-4/

Since my immediate problem was solved with the mtu param I plan on
forgetting about htb and traffic control in general for the time
being :) Thanks again.

Cheers
Yanko

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] htb overrate with 2.6.16

[Andy Furniss]

Yanko Kaneti wrote:

>>I think you need to ask fedora or intel driver maintainer about this. 
>>AIUI tso is not in vanilla kernels and the patches are quite invasive.
> 
> 
> Well, as much as google tells me TSO has been in the kernel and enabled
> since 2.5.33 and e1000 was the first driver to support it. 
> The FC4 2.6.16 kernel doesn't have any tso related patches as can be
> seen here http://cvs.fedora.redhat.com/viewcvs/rpms/kernel/FC-4/

Ahh OK I must have been thinking of another card.

Andy.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] htb overrate with 2.6.16

[Andy Furniss]

Yanko Kaneti wrote:
> On Sun, 2006-04-16 at 19:40 +0100, Andy Furniss wrote:
> 
>>Yanko Kaneti wrote:
>>
>>>On Sun, 2006-04-16 at 03:03 +0100, Andy Furniss wrote: 
>>>
>>>
>>>>Yanko Kaneti wrote:
>>>>
>>>>
>>>>
>>>>>Setting mtu 16500  for the class fixed it. But I wonder where did these
>>>>>giants come from in the first place? The mtu of the interface is and was
>>>>>1500. Or so ifconfig and ip link tell me. Or this is some other mtu we
>>>>>are talking about...
>>>>
>>>>Hmm I didn't expect that - maybe there is some problem with the nic 
>>>>drivers not obeying kernel - is there any tso offload etc. at work here ?
>>>
>>>
>>>Yes and its on by default. The interface mtu still says 1500.
>>>I've tried deleting and attaching the qdisc+class (without explicit
>>>large mtu) with both tso on (ethtool -K eth0 tso on) and tso off , it
>>>doesnt seem to matter - giants appear in both cases. 
>>>With large mtu for the class no giants with both tso on and off.
>>>
>>
>>I think you need to ask fedora or intel driver maintainer about this. 
>>AIUI tso is not in vanilla kernels and the patches are quite invasive.
> 
> 
> Well, as much as google tells me TSO has been in the kernel and enabled
> since 2.5.33 and e1000 was the first driver to support it. 
> The FC4 2.6.16 kernel doesn't have any tso related patches as can be
> seen here http://cvs.fedora.redhat.com/viewcvs/rpms/kernel/FC-4/
> 
> Since my immediate problem was solved with the mtu param I plan on
> forgetting about htb and traffic control in general for the time
> being :) Thanks again.

One more thing I just thought - sfq sets its quantum from the dev mtu.

While I always thought that the "must be >=mtu" comment in the source 
was a bit OTT, it still "should" be >= mtu for the drr to be 0(1) for 
cpu work.

You can set it explicitly by adding quantum=X on the sfq line.

For ethernet X is mtu + 14.

Andy.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] htb overrate with 2.6.16

[Patrick McHardy]

Andy Furniss wrote:
>> Well, as much as google tells me TSO has been in the kernel and enabled
>> since 2.5.33 and e1000 was the first driver to support it. The FC4
>> 2.6.16 kernel doesn't have any tso related patches as can be
>> seen here http://cvs.fedora.redhat.com/viewcvs/rpms/kernel/FC-4/
>>
>> Since my immediate problem was solved with the mtu param I plan on
>> forgetting about htb and traffic control in general for the time
>> being :) Thanks again.
> 
> 
> One more thing I just thought - sfq sets its quantum from the dev mtu.

One more possibility: current kernels support UDP fragmentation offload
(UFO), which has similar effects as TSO. The in-tree e1000 driver
doesn't support it, but maybe the fedora one does.

Changes in the fragmentation behaviour of conntrack in 2.6.16 could also
be responsible (if you're using it). Can you please post your NAT and
marking rules, routing rules etc?
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] htb overrate with 2.6.16

[Yanko Kaneti]

On Mon, 2006-04-17 at 21:32 +0200, Patrick McHardy wrote:
> Andy Furniss wrote:
> >> Well, as much as google tells me TSO has been in the kernel and enabled
> >> since 2.5.33 and e1000 was the first driver to support it. The FC4
> >> 2.6.16 kernel doesn't have any tso related patches as can be
> >> seen here http://cvs.fedora.redhat.com/viewcvs/rpms/kernel/FC-4/
> >>
> >> Since my immediate problem was solved with the mtu param I plan on
> >> forgetting about htb and traffic control in general for the time
> >> being :) Thanks again.
> > 
> > 
> > One more thing I just thought - sfq sets its quantum from the dev mtu.

Riiight. I should have tried without the sfq earlier. Without it this
works as expected without explicit mtu setting for the htb class. And no
giants.

# tc qdisc add dev eth0 root handle 1: htb
# tc class add dev eth0 parent 1: classid 1:2 htb rate 2Mbit
# tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 50 fw flowid 1:2


> One more possibility: current kernels support UDP fragmentation offload
> (UFO), which has similar effects as TSO. The in-tree e1000 driver
> doesn't support it, but maybe the fedora one does.

No mention of ufo or e1000 in any of the patches that can be found in
the url above.

> Changes in the fragmentation behaviour of conntrack in 2.6.16 could also
> be responsible (if you're using it). Can you please post your NAT and
> marking rules, routing rules etc?

Here are the "interesting" rules where the packets in question pass.
Have no rules other than the ones in the mangle table 

mangle
  OUTPUT
    -m owner --uid-owner shaped -j userchain
  userchain
    ....
    -m length --length 512:65535 -j MARK --set-mark 0x32   -> the fw filter


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] htb overrate with 2.6.16

[Andy Furniss]

Yanko Kaneti wrote:

>>>One more thing I just thought - sfq sets its quantum from the dev mtu.
> 
> 
> Riiight. I should have tried without the sfq earlier. Without it this
> works as expected without explicit mtu setting for the htb class. And no
> giants.
> 
> # tc qdisc add dev eth0 root handle 1: htb
> # tc class add dev eth0 parent 1: classid 1:2 htb rate 2Mbit
> # tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 50 fw flowid 1:2

I wouldn't have expected that to make any difference to the giants.

Looking again at your stats -

Sent 189796883 bytes 20626 pkt (dropped 0, overlimits 0 requeues 0)
  rate 3484Kbit 45pps backlog 0b 0p requeues 0
  lended: 20627 borrowed: 0 giants: 30926
  tokens: -9768 ctokens: -9768

The giants count is higher than the packet count so now I am really 
confused.

Andy.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] htb overrate with 2.6.16

[Andy Furniss]

Andy Furniss wrote:

> Looking again at your stats -
> 
> Sent 189796883 bytes 20626 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 3484Kbit 45pps backlog 0b 0p requeues 0
>  lended: 20627 borrowed: 0 giants: 30926
>  tokens: -9768 ctokens: -9768
> 
> The giants count is higher than the packet count so now I am really 
> confused.

Doh - I suppose thats just the way HTB counts so you add them together.

Andy.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] htb overrate with 2.6.16

[Andy Furniss]

Andy Furniss wrote:
> Andy Furniss wrote:
> 
>> Looking again at your stats -
>>
>> Sent 189796883 bytes 20626 pkt (dropped 0, overlimits 0 requeues 0)
>>  rate 3484Kbit 45pps backlog 0b 0p requeues 0
>>  lended: 20627 borrowed: 0 giants: 30926
>>  tokens: -9768 ctokens: -9768
>>
>> The giants count is higher than the packet count so now I am really 
>> confused.
> 
> 
> Doh - I suppose thats just the way HTB counts so you add them together.

LOL - Third try, testing on lo which is confusing and maybe misleading 
but it looks like the giants count gets doubled up but the packet count 
doesn't.

Andy.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc