[LARTC] HTB speed

[LARTC] HTB speed

[Jeroen Vriesman]

Dear all,

I'm running htb on 2.4.24, following setup:

running on a bridge, ebtables patch, firewall marks with iptables, and an SFQ for every HTB.

Everything works fine, so I don't go into details, there is just one thing which suprises me:

When I configure HTB to shape to about 900 kbit/s, the resulting speed is about 400 kbit/s.

For higher speeds it's more accurate (but stil too slow), for lower speeds it gets worse.

Does anyone here have any idea why? I've just added a "correction factor" to my scripts, so it's kind of allright, but I would like to know if others experience the same thing.

Jeroen.
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] HTB speed

[Jeroen Vriesman]

Hi,

just putting the answer to my own question here, for those who have the same problem, and read the mailing list archive.

The timing of the P4 based on "jiffies" is hopeless, it's different for every processor, and can be a wrong by a factor 3.

If the tsc (time stamp counter) is used, the htb works fine, the error in speed is now only about 1 %.

It's set by:

in pkt_sched.h:

#define PSCHED_CLOCK_SOURCE     PSCHED_CPU

that's all, I wonder why it's not default to do this, or maybe it's an idea to make the packet scheduler detect the presence of tsc when the module is loaded.

Cheers,
Jeroen.


On Fri, 12 Mar 2004 10:58:40 +0100
Jeroen Vriesman <j.vriesman@prompt.nl> wrote:

> Dear all,
> 
> I'm running htb on 2.4.24, following setup:
> 
> running on a bridge, ebtables patch, firewall marks with iptables, and an SFQ for every HTB.
> 
> Everything works fine, so I don't go into details, there is just one thing which suprises me:
> 
> When I configure HTB to shape to about 900 kbit/s, the resulting speed is about 400 kbit/s.
> 
> For higher speeds it's more accurate (but stil too slow), for lower speeds it gets worse.
> 
> Does anyone here have any idea why? I've just added a "correction factor" to my scripts, so it's kind of allright, but I would like to know if others experience the same thing.
> 
> Jeroen.
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> 
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] HTB speed

[Maria Joana Urbano]

[-- Attachment #1: Type: text/plain, Size: 856 bytes --]


>in pkt_sched.h:
>
>#define PSCHED_CLOCK_SOURCE     PSCHED_CPU
>
>that's all, I wonder why it's not default to do this, or maybe it's an 
>idea to make the packet scheduler detect the presence of tsc when the 
>module is loaded.

Hi,

I think not all processors accept this #define PSCHED_CLOCK_SOURCE 
PSCHED_CPU. At least, I couldn't do it in my router.

I am not using HTB but TBF, but I think the timing problems are similar. As 
to tune TBF with higher precision, I had to manually change Hz value from 
100 to 1000, in param.h, and actually I am getting better results.

My question is, does anybody tried to do this in an production environment? 
For instance, in a router that integrates traffic control, NAT, firewall, 
services gateway, etc., can this change in HZ value harm in any way the 
rest of the system?

Thanks in advance,
Joana Urbano

Re: [LARTC] HTB speed

[Jeroen Vriesman]

Hi,

to use the TSC, the processor has to have a tsc, you can see that in /proc/cpuinfo, for as far as I know, every P4 has it (but I'm not sure), it's in the "flags" of cpuinfo:

On my P4-1800:

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm

The "loop calibrated" jiffies stuff dates back to 386, it's used everywhere in the kernel timing, and it's very inaccurate, I don't know if 2.6 still uses it everwhere, what kind of processor do you have?

I've never tried to change the Hz value, does changing it in param.h really changes the frequency of the clock ticks? If so, why is the default only 100Hz these days? doesn't make sense to me.

I do use the PSCHED_CPU in a production environement at the moment, works fine.

Cheers,
Jeroen.




On Mon, 15 Mar 2004 11:37:16 +0000
Maria Joana Urbano <stmaria@dei.uc.pt> wrote:

> 
> >in pkt_sched.h:
> >
> >#define PSCHED_CLOCK_SOURCE     PSCHED_CPU
> >
> >that's all, I wonder why it's not default to do this, or maybe it's an 
> >idea to make the packet scheduler detect the presence of tsc when the 
> >module is loaded.
> 
> Hi,
> 
> I think not all processors accept this #define PSCHED_CLOCK_SOURCE 
> PSCHED_CPU. At least, I couldn't do it in my router.
> 
> I am not using HTB but TBF, but I think the timing problems are similar. As 
> to tune TBF with higher precision, I had to manually change Hz value from 
> 100 to 1000, in param.h, and actually I am getting better results.
> 
> My question is, does anybody tried to do this in an production environment? 
> For instance, in a router that integrates traffic control, NAT, firewall, 
> services gateway, etc., can this change in HZ value harm in any way the 
> rest of the system?
> 
> Thanks in advance,
> Joana Urbano
> 
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] HTB speed

[Jeroen Vriesman]

Hi,

Indeed, changing the timer to 1000Hz is possible, it turned out that I have a machine here running with a 1000Hz timer ticker. (I've installed a "realtime" kernel on it for audio recording).

About your previous question, I've noticed that the system with 1000Hz ticker (which has been running for a few months) is UNSTABLE.

might be more to it than just the 1000Hz, but I did notice that some "realtime" applications running in userspace where able to crash the system completely, happened 2 times in 3 months.

Cheers,
Jeroen.




On Mon, 15 Mar 2004 11:37:16 +0000
Maria Joana Urbano <stmaria@dei.uc.pt> wrote:

> 
> >in pkt_sched.h:
> >
> >#define PSCHED_CLOCK_SOURCE     PSCHED_CPU
> >
> >that's all, I wonder why it's not default to do this, or maybe it's an 
> >idea to make the packet scheduler detect the presence of tsc when the 
> >module is loaded.
> 
> Hi,
> 
> I think not all processors accept this #define PSCHED_CLOCK_SOURCE 
> PSCHED_CPU. At least, I couldn't do it in my router.
> 
> I am not using HTB but TBF, but I think the timing problems are similar. As 
> to tune TBF with higher precision, I had to manually change Hz value from 
> 100 to 1000, in param.h, and actually I am getting better results.
> 
> My question is, does anybody tried to do this in an production environment? 
> For instance, in a router that integrates traffic control, NAT, firewall, 
> services gateway, etc., can this change in HZ value harm in any way the 
> rest of the system?
> 
> Thanks in advance,
> Joana Urbano
> 
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] HTB speed

[Simon Byrnand]

At 11:14 14/03/2004, Jeroen Vriesman wrote:
>Hi,
>
>just putting the answer to my own question here, for those who have the 
>same problem, and read the mailing list archive.
>
>The timing of the P4 based on "jiffies" is hopeless, it's different for 
>every processor, and can be a wrong by a factor 3.
>
>If the tsc (time stamp counter) is used, the htb works fine, the error in 
>speed is now only about 1 %.
>
>It's set by:
>
>in pkt_sched.h:
>
>#define PSCHED_CLOCK_SOURCE     PSCHED_CPU
>
>that's all, I wonder why it's not default to do this, or maybe it's an 
>idea to make the packet scheduler detect the presence of tsc when the 
>module is loaded.

Hi,

Which pkt_sched.h are you refering to ? 
/usr/src/linux/include/linux/pkt_sched.h or 
/usr/src/linux/include/net/pkt_sched.h ?

And after changing it what did you do ? Recompile the kernel ?

Or recompile tc ?

I too see the same problems with htb (very poor accuracy of speed, 
significantly too slow, also very jerky) while cbq is very accurate and 
smooth. (But lacks some features I need)

Regards,
Simon

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] HTB speed

[Andrew Hall]

You need to recompile the kernel after altering this value in
linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf
qdisc, then the queue length may cause delay problems if it's too long
(default is 128). Changing this to 32 for rates below 100kb/s, I have found
to help things considerably. This change needs to be done in
linux/net/sched/sch_sfq.c. which also needs a kernel recompilation.

----- Original Message -----
From: "Simon Byrnand" <simon@igrin.co.nz>
To: "Jeroen Vriesman" <j.vriesman@prompt.nl>; <lartc@mailman.ds9a.nl>
Sent: Thursday, March 25, 2004 4:36 PM
Subject: Re: [LARTC] HTB speed


> At 11:14 14/03/2004, Jeroen Vriesman wrote:
> >Hi,
> >
> >just putting the answer to my own question here, for those who have the
> >same problem, and read the mailing list archive.
> >
> >The timing of the P4 based on "jiffies" is hopeless, it's different for
> >every processor, and can be a wrong by a factor 3.
> >
> >If the tsc (time stamp counter) is used, the htb works fine, the error in
> >speed is now only about 1 %.
> >
> >It's set by:
> >
> >in pkt_sched.h:
> >
> >#define PSCHED_CLOCK_SOURCE     PSCHED_CPU
> >
> >that's all, I wonder why it's not default to do this, or maybe it's an
> >idea to make the packet scheduler detect the presence of tsc when the
> >module is loaded.
>
> Hi,
>
> Which pkt_sched.h are you refering to ?
> /usr/src/linux/include/linux/pkt_sched.h or
> /usr/src/linux/include/net/pkt_sched.h ?
>
> And after changing it what did you do ? Recompile the kernel ?
>
> Or recompile tc ?
>
> I too see the same problems with htb (very poor accuracy of speed,
> significantly too slow, also very jerky) while cbq is very accurate and
> smooth. (But lacks some features I need)
>
> Regards,
> Simon
>
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] HTB speed

[Jeroen Vriesman]

Would the problem with queue lengths also exist if CONFIG_HZ\x1000 in .config? (it certainly improves latency)

I've configured rates from 10kbit/s...2Mbit/s on one machine, so changing the kernel code "for low rates" would probably effect the high rates too. 

Or, in other words, what is exactly the problem with an SFQ length of 128 for low rates?

Cheers,
Jeroen.



On Thu, 25 Mar 2004 17:17:44 +1100
"Andrew Hall" <temp02@bluereef.com.au> wrote:

> You need to recompile the kernel after altering this value in
> linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf
> qdisc, then the queue length may cause delay problems if it's too long
> (default is 128). Changing this to 32 for rates below 100kb/s, I have found
> to help things considerably. This change needs to be done in
> linux/net/sched/sch_sfq.c. which also needs a kernel recompilation.
> 
> ----- Original Message -----
> From: "Simon Byrnand" <simon@igrin.co.nz>
> To: "Jeroen Vriesman" <j.vriesman@prompt.nl>; <lartc@mailman.ds9a.nl>
> Sent: Thursday, March 25, 2004 4:36 PM
> Subject: Re: [LARTC] HTB speed
> 
> 
> > At 11:14 14/03/2004, Jeroen Vriesman wrote:
> > >Hi,
> > >
> > >just putting the answer to my own question here, for those who have the
> > >same problem, and read the mailing list archive.
> > >
> > >The timing of the P4 based on "jiffies" is hopeless, it's different for
> > >every processor, and can be a wrong by a factor 3.
> > >
> > >If the tsc (time stamp counter) is used, the htb works fine, the error in
> > >speed is now only about 1 %.
> > >
> > >It's set by:
> > >
> > >in pkt_sched.h:
> > >
> > >#define PSCHED_CLOCK_SOURCE     PSCHED_CPU
> > >
> > >that's all, I wonder why it's not default to do this, or maybe it's an
> > >idea to make the packet scheduler detect the presence of tsc when the
> > >module is loaded.
> >
> > Hi,
> >
> > Which pkt_sched.h are you refering to ?
> > /usr/src/linux/include/linux/pkt_sched.h or
> > /usr/src/linux/include/net/pkt_sched.h ?
> >
> > And after changing it what did you do ? Recompile the kernel ?
> >
> > Or recompile tc ?
> >
> > I too see the same problems with htb (very poor accuracy of speed,
> > significantly too slow, also very jerky) while cbq is very accurate and
> > smooth. (But lacks some features I need)
> >
> > Regards,
> > Simon
> >
> > _______________________________________________
> > LARTC mailing list / LARTC@mailman.ds9a.nl
> > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> 
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> 
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] HTB speed

[ThE LinuX_KiD]

Thanks, Andrew, it's intresting...


-> (default is 128). Changing this to 32 for rates below 100kb/s, I
-> have found

you mean that  try change "linux/net/sched/sch_sfq.c" this: ?

#define SFQ_DEPTH               128
#define SFQ_HASH_DIVISOR        1024

for this.

#define SFQ_DEPTH               32
#define SFQ_HASH_DIVISOR        1024


I 've 2 questions about it

1) what is I use various rates (from 8k to 256k)

2) how I must proceed in Esfq  ??
   (in order to work with those htb classes)


	Usage: ... esfq [ perturb SECS ] [ quantum BYTES ] [ depth FLOWS ]
	        [ divisor HASHBITS ] [ limit PKTS ] [ hash HASHTYPE]
	Where:
	HASHTYPE := { classic | src | dst }

Thankyou!
andres



->
-> You need to recompile the kernel after altering this value in
-> linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf
-> qdisc, then the queue length may cause delay problems if it's too long
-> (default is 128). Changing this to 32 for rates below 100kb/s, I
-> have found
-> to help things considerably. This change needs to be done in
-> linux/net/sched/sch_sfq.c. which also needs a kernel recompilation.
->
-> ----- Original Message -----
-> From: "Simon Byrnand" <simon@igrin.co.nz>
-> To: "Jeroen Vriesman" <j.vriesman@prompt.nl>; <lartc@mailman.ds9a.nl>
-> Sent: Thursday, March 25, 2004 4:36 PM
-> Subject: Re: [LARTC] HTB speed
->
->
-> > At 11:14 14/03/2004, Jeroen Vriesman wrote:
-> > >Hi,
-> > >
-> > >just putting the answer to my own question here, for those
-> who have the
-> > >same problem, and read the mailing list archive.
-> > >
-> > >The timing of the P4 based on "jiffies" is hopeless, it's
-> different for
-> > >every processor, and can be a wrong by a factor 3.
-> > >
-> > >If the tsc (time stamp counter) is used, the htb works fine,
-> the error in
-> > >speed is now only about 1 %.
-> > >
-> > >It's set by:
-> > >
-> > >in pkt_sched.h:
-> > >
-> > >#define PSCHED_CLOCK_SOURCE     PSCHED_CPU
-> > >
-> > >that's all, I wonder why it's not default to do this, or maybe it's an
-> > >idea to make the packet scheduler detect the presence of tsc when the
-> > >module is loaded.
-> >
-> > Hi,
-> >
-> > Which pkt_sched.h are you refering to ?
-> > /usr/src/linux/include/linux/pkt_sched.h or
-> > /usr/src/linux/include/net/pkt_sched.h ?
-> >
-> > And after changing it what did you do ? Recompile the kernel ?
-> >
-> > Or recompile tc ?
-> >
-> > I too see the same problems with htb (very poor accuracy of speed,
-> > significantly too slow, also very jerky) while cbq is very accurate and
-> > smooth. (But lacks some features I need)
-> >
-> > Regards,
-> > Simon
-> >
-> > _______________________________________________
-> > LARTC mailing list / LARTC@mailman.ds9a.nl
-> > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
->
-> _______________________________________________
-> LARTC mailing list / LARTC@mailman.ds9a.nl
-> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
->

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] HTB speed

[Simon Byrnand]

At 18:17 25/03/2004, Andrew Hall wrote:
>You need to recompile the kernel after altering this value in
>linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf
>qdisc, then the queue length may cause delay problems if it's too long
>(default is 128). Changing this to 32 for rates below 100kb/s, I have found
>to help things considerably. This change needs to be done in
>linux/net/sched/sch_sfq.c. which also needs a kernel recompilation.

Hmm,

When I use sfq with cbq at speeds like 256Kbit there is no problem at all, 
runs very sweetly, but with HTB and sfq, it is very jerky and poor. I'll 
try the change in pkt_sched.h first and see how I go...

Regards,
Simon

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] HTB speed

[Simon Byrnand]

At 12:46 26/03/2004, Simon Byrnand wrote:
>At 18:17 25/03/2004, Andrew Hall wrote:
>>You need to recompile the kernel after altering this value in
>>linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf
>>qdisc, then the queue length may cause delay problems if it's too long
>>(default is 128). Changing this to 32 for rates below 100kb/s, I have found
>>to help things considerably. This change needs to be done in
>>linux/net/sched/sch_sfq.c. which also needs a kernel recompilation.
>
>Hmm,
>
>When I use sfq with cbq at speeds like 256Kbit there is no problem at all, 
>runs very sweetly, but with HTB and sfq, it is very jerky and poor. I'll 
>try the change in pkt_sched.h first and see how I go...

Ok, I tried the change in pkt_sched.h and didn't notice any difference 
whatsoever. Any other ideas ? cbq is still fine but htb for the same speed 
is very jerky and the speed fluctuates around 60-80% of the wanted speed, 
while cbq gives a steady 99% of the wanted speed...

Regards,
Simon

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] HTB speed

[Andy Furniss]

Simon Byrnand wrote:
> At 12:46 26/03/2004, Simon Byrnand wrote:
> 
>> At 18:17 25/03/2004, Andrew Hall wrote:
>>
>>> You need to recompile the kernel after altering this value in
>>> linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf
>>> qdisc, then the queue length may cause delay problems if it's too long
>>> (default is 128). Changing this to 32 for rates below 100kb/s, I have 
>>> found
>>> to help things considerably. This change needs to be done in
>>> linux/net/sched/sch_sfq.c. which also needs a kernel recompilation.
>>
>>
>> Hmm,
>>
>> When I use sfq with cbq at speeds like 256Kbit there is no problem at 
>> all, runs very sweetly, but with HTB and sfq, it is very jerky and 
>> poor. I'll try the change in pkt_sched.h first and see how I go...

> 
> Ok, I tried the change in pkt_sched.h and didn't notice any difference 
> whatsoever. Any other ideas ? cbq is still fine but htb for the same 
> speed is very jerky and the speed fluctuates around 60-80% of the wanted 
> speed, while cbq gives a steady 99% of the wanted speed...

You can make HTB more accurate by setting HTB_HYSTERESIS to 0 in 
net/sched/sch_htb.c.

To save time - if you built HTB as a module, you can probably (well it 
worked for me) get away with editing htb.c and do

make SUBDIRS=net/sched modules
and replacing /lib/modules/[kversion]/kernel/net/sched/htb.o with the 
new htb.o from your source tree.

If you are doing it live stop shaping and check with lsmod that modprobe 
-r gets rid (do it again if it's still there) of the old htb.o and 
reload shaping scripts.

Make sure the quantum is your mtu, set 0 burst for bulk classes and 
don't set perturb too low on sfq (I use 20 as it causes packet reordering).

Are you shaping egress or ingress and how are you measuring speed?

Andy.







_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] HTB speed

[Jeroen Vriesman]

Strange, must be somthing else going on.

I had the same 40 to 50 percent too slow, which was completely fixed bij using PSCHED_CPU in pkt_sched.h.

Using kernel 2.4.24, I measure the speed with the iptables byte counters.



On Fri, 26 Mar 2004 16:27:42 +1200
Simon Byrnand <simon@igrin.co.nz> wrote:

> At 12:46 26/03/2004, Simon Byrnand wrote:
> >At 18:17 25/03/2004, Andrew Hall wrote:
> >>You need to recompile the kernel after altering this value in
> >>linux/include/net/pkt_sched.h. Also remember that if using SFQ on leaf
> >>qdisc, then the queue length may cause delay problems if it's too long
> >>(default is 128). Changing this to 32 for rates below 100kb/s, I have found
> >>to help things considerably. This change needs to be done in
> >>linux/net/sched/sch_sfq.c. which also needs a kernel recompilation.
> >
> >Hmm,
> >
> >When I use sfq with cbq at speeds like 256Kbit there is no problem at all, 
> >runs very sweetly, but with HTB and sfq, it is very jerky and poor. I'll 
> >try the change in pkt_sched.h first and see how I go...
> 
> Ok, I tried the change in pkt_sched.h and didn't notice any difference 
> whatsoever. Any other ideas ? cbq is still fine but htb for the same speed 
> is very jerky and the speed fluctuates around 60-80% of the wanted speed, 
> while cbq gives a steady 99% of the wanted speed...
> 
> Regards,
> Simon
> 
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> 
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/