[LARTC] HTB not that exact

[LARTC] HTB not that exact

[Stefan Rompf]

Hi,

I'm experimenting using the HTB queueing discipline for traffic shaping.
However, it is not really exact. Currently, I try this setup:

tc qdisc add dev eth0 root handle 1: htb default 3

tc class add dev eth0 parent 1: classid 1:1 htb rate 20Mbit burst 4kB
tc qdisc add dev eth0 parent 1:1 sfq

tc class add dev eth0 parent 1: classid 1:2 htb rate 3Mbit burst 2kB
tc qdisc add dev eth0 parent 1:2 sfq

tc class add dev eth0 parent 1: classid 1:3 htb rate 77Mbit burst 150kB
tc qdisc add dev eth0 parent 1:3 sfq

tc filter add dev eth0 parent 1:0 prio 7 protocol ip handle 1 fw classid
1:1
tc filter add dev eth0 parent 1:0 prio 7 protocol ip handle 2 fw classid
1:2

/usr/local/sbin/iptables -t mangle -F OUTPUT
/usr/local/sbin/iptables -t mangle -A OUTPUT -p tcp --dport 9021 -j MARK
--set-mark 1
/usr/local/sbin/iptables -t mangle -A OUTPUT -p tcp --dport 9022:9023 -j
MARK --set-mark 2

The network adaptor is connected to a 100MBit switch. When testing with
netio, I can send up to 370kB/sec through class 1:2 and up to 2,4MB/sec
via class 1:1, both measured by one/multiple instances of netio and the
rate output of "tc -s class dev eth0". This effect occurs with Linux
2.4.16,  kernel either compiled with HZ set to 100 or 1024, and of
course independant of the filter type used. During the tests, no packets
need to be dropped, htb just delays.

Output when two netio instances are sending to class 1:2 for some time:

root@barkeeper /root/ >tc -s class list dev eth0
[...]

class htb 1:2 root leaf 800e: prio 0 rate 3Mbit ceil 3Mbit burst 2Kb
cburst 2Kb
 Sent 236734498 bytes 156441 pkts (dropped 0, overlimits 301823)
 rate 378286bps 250pps backlog 44p
 lended: 156397 borrowed: 0 giants: 0 injects: 0
 tokens: -3714 ctokens: -3714

[...]


tbf seems not to be able to do exact rate limiting, too. Anything wrong
in my configuration, or am I just taking false statistics?

Stefan

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] HTB not that exact

[bert hubert]

On Thu, Dec 06, 2001 at 08:28:07PM +0100, Stefan Rompf wrote:
> Hi,
> 
> I'm experimenting using the HTB queueing discipline for traffic shaping.
> However, it is not really exact. Currently, I try this setup:
> 
> tc qdisc add dev eth0 root handle 1: htb default 3
> 
> tc class add dev eth0 parent 1: classid 1:1 htb rate 20Mbit burst 4kB
> tc qdisc add dev eth0 parent 1:1 sfq

Raise burst, for 20Mbit it needs to be at least 20kbyte!

> The network adaptor is connected to a 100MBit switch. When testing with
> netio, I can send up to 370kB/sec through class 1:2 and up to 2,4MB/sec
> via class 1:1, both measured by one/multiple instances of netio and the
> rate output of "tc -s class dev eth0". This effect occurs with Linux
> 2.4.16,  kernel either compiled with HZ set to 100 or 1024, and of

The rate output of tc -s class is not that reliable. 

> tbf seems not to be able to do exact rate limiting, too. Anything wrong
> in my configuration, or am I just taking false statistics?

Both I think. There are some indications that raising HZ may not work, as it
may not raise the resolution of the timers in the Linux Kernel - I am
investigating this! So try keeping HZ stable, and raising the size of your
bucket.

Regards,

bert

-- 
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] HTB not that exact

[Daniel Ryczek]

On Thu, Dec 06, 2001 at 08:28:07PM +0100, Stefan Rompf wrote:
> Hi,
> 
> tc qdisc add dev eth0 root handle 1: htb default 3
> 
> tc class add dev eth0 parent 1: classid 1:1 htb rate 20Mbit burst 4kB
> tc qdisc add dev eth0 parent 1:1 sfq
> 
> tc class add dev eth0 parent 1: classid 1:2 htb rate 3Mbit burst 2kB
> tc qdisc add dev eth0 parent 1:2 sfq
> 
> tc class add dev eth0 parent 1: classid 1:3 htb rate 77Mbit burst 150kB
> tc qdisc add dev eth0 parent 1:3 sfq
> 
> tc filter add dev eth0 parent 1:0 prio 7 protocol ip handle 1 fw classid
> 1:1
> tc filter add dev eth0 parent 1:0 prio 7 protocol ip handle 2 fw classid
> 1:2
> 
> /usr/local/sbin/iptables -t mangle -F OUTPUT
> /usr/local/sbin/iptables -t mangle -A OUTPUT -p tcp --dport 9021 -j MARK
> --set-mark 1
> /usr/local/sbin/iptables -t mangle -A OUTPUT -p tcp --dport 9022:9023 -j
> MARK --set-mark 2
> 
> The network adaptor is connected to a 100MBit switch. When testing with
> netio, I can send up to 370kB/sec through class 1:2 and up to 2,4MB/sec
> via class 1:1, both measured by one/multiple instances of netio and the
> rate output of "tc -s class dev eth0". This effect occurs with Linux
> 2.4.16,  kernel either compiled with HZ set to 100 or 1024, and of
> course independant of the filter type used. During the tests, no packets
> need to be dropped, htb just delays.
I think your results are quite exact. The difference is because of
different units. 
370kB = 370 * 8 = 2.96 Mbit
2.4MB = 19.2 Mbit

--
Daniel


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] HTB not that exact

[Stefan Rompf]

Hi,

> I think your results are quite exact. The difference is because of
> different units.
> 370kB = 370 * 8 = 2.96 Mbit
> 2.4MB = 19.2 Mbit

obviously ;-) When calculating, I did remember that one megabit is not
1000000 bits, but forgot about the byte. Thanks for bringing that into
my mind again.

Stefan

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] HTB not that exact

[david]

> obviously ;-) When calculating, I did remember that one megabit is not
> 1000000 bits, but forgot about the byte. Thanks for bringing that into
> my mind again.

One megabit IS actually 10^6 bit - take a look at:

http://physics.nist.gov/cuu/Units/binary.html

What you are thinking about is one mebibit, which is 2^20 bit, take a
look at the boot message of a recent kernel and see those prefix'es
used for ide disk cache sice.

- David

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] HTB not that exact

[Sami Farin]

On Thu, Dec 06, 2001 at 08:48:11PM +0100, bert hubert wrote:
> On Thu, Dec 06, 2001 at 08:28:07PM +0100, Stefan Rompf wrote:
...
> > tbf seems not to be able to do exact rate limiting, too. Anything wrong
> > in my configuration, or am I just taking false statistics?
> 
> Both I think. There are some indications that raising HZ may not work, as it
> may not raise the resolution of the timers in the Linux Kernel - I am
> investigating this! So try keeping HZ stable, and raising the size of your
> bucket.

would it  make any difference if you do...

--- linux/include/net/pkt_sched.h.bak   Sun Nov 19 20:14:10 2000
+++ linux/include/net/pkt_sched.h       Sun Nov 19 20:20:34 2000
@@ -5,7 +5,7 @@
 #define PSCHED_JIFFIES                 2
 #define PSCHED_CPU             3
 
-#define PSCHED_CLOCK_SOURCE    PSCHED_JIFFIES
+#define PSCHED_CLOCK_SOURCE    PSCHED_CPU
 
 #include <linux/config.h>
 #include <linux/pkt_sched.h>

I didn't see PSCHED_CLOCK_SOURCE in the 2.4routing docs...
Can you tell more about PSCHED_CLOCK_SOURCE ?

-- 
Safari - safari@iki.fi - PGP key 0x427E7914 - http://www.iki.fi/safari/
  The UNIX Guru's View of Sex: "unzip ; strip ; touch ; finger ; \
  mount ; fsck ; more ; yes ; umount ; sleep"

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] HTB not that exact

[Martin Devera]

> > Both I think. There are some indications that raising HZ may not work, as it
> > may not raise the resolution of the timers in the Linux Kernel - I am
> > investigating this! So try keeping HZ stable, and raising the size of your
> > bucket.
> 
> would it  make any difference if you do...
> 
> -#define PSCHED_CLOCK_SOURCE    PSCHED_JIFFIES
> +#define PSCHED_CLOCK_SOURCE    PSCHED_CPU

by the way if you have pentium with tsc the patch above
is stronly recommended. It allows for sub nanosecond
precision...
devik


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] HTB not that exact

[bert hubert]

On Fri, Dec 07, 2001 at 06:57:39PM +0200, Sami Farin wrote:
> Ahu wrote:
> > Both I think. There are some indications that raising HZ may not work, as it
> > may not raise the resolution of the timers in the Linux Kernel - I am
> > investigating this! So try keeping HZ stable, and raising the size of your
> > bucket.
> 
> would it  make any difference if you do...

Ok - I've since investigated this. Shapers use timers internally to block
themselves for set periods of time. These timers appear to have HZ
resolution, but a comment in the very good Linux Device Drivers book
                    http://www.xml.com/ldd/chapter/book/
appeared to indicate that raising HZ would not raise their resolution.

I contacted Jonathan Corbet, one of the authors, and he checked and found
that the book was incorrect: raising HZ *does* increase timer resolution. 

> -#define PSCHED_CLOCK_SOURCE    PSCHED_JIFFIES
> +#define PSCHED_CLOCK_SOURCE    PSCHED_CPU
> 
> I didn't see PSCHED_CLOCK_SOURCE in the 2.4routing docs...
> Can you tell more about PSCHED_CLOCK_SOURCE ?

As far as I can see, this only increases the *resolution*, not *precision*.
In other words, you can't specify shorter delays this way but you can
measure a delay that already happened far more precisely.

For example, if you decide to delay for 1 jiffy, 10ms, you may in fact have
waited 14ms. PSCHED_JIFFIES will not see this - 1.4=1 jiffy has passed.
PSCHED_CPU *will* see it, and let the right amount of packets out.

But I'm hoping that Martin Devera can check this for me, devik?

Regards,

bert

-- 
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] HTB not that exact

[bert hubert]

On Sat, Dec 08, 2001 at 09:51:22AM +0100, Martin Devera wrote:
> > > Both I think. There are some indications that raising HZ may not work, as it
> > > may not raise the resolution of the timers in the Linux Kernel - I am
> > > investigating this! So try keeping HZ stable, and raising the size of your
> > > bucket.
> > 
> > would it  make any difference if you do...
> > 
> > -#define PSCHED_CLOCK_SOURCE    PSCHED_JIFFIES
> > +#define PSCHED_CLOCK_SOURCE    PSCHED_CPU
> 
> by the way if you have pentium with tsc the patch above
> is stronly recommended. It allows for sub nanosecond
> precision...
> devik

Is there a reason not to do this? In which case we should probably submit a
patch that makes PSCHED_CPU the default if you have a TSC.

Regards,

bert

-- 
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] HTB not that exact

[devik]

> > > -#define PSCHED_CLOCK_SOURCE    PSCHED_JIFFIES
> > > +#define PSCHED_CLOCK_SOURCE    PSCHED_CPU
> >
> > by the way if you have pentium with tsc the patch above
> > is stronly recommended. It allows for sub nanosecond
> > precision...
> > devik
>
> Is there a reason not to do this? In which case we should probably submit a
> patch that makes PSCHED_CPU the default if you have a TSC.

It would be nice - but when ANK does somethink in some way
there is reason usualy.
I've already been thinking about it as

#ifdef CONFIG_X86_TSC
 #define PSCHED_CLOCK_SOURCE    PSCHED_CPU
....

I can found no reason why don't use TSC. With highly loaded
interface where eth interrupts are faster than timer ones
it should make flow smoother.
Without TSC all packets within single timer tick (10ms) are
treated as havin the same arival time.
CBQ's clock integrator based on HW speed computations overcomes
this and it might be reason why ANK left JIFFIES here.

If you think that you have such power to convince people
about it then go ahead. :)

devik


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/