* Re: [LARTC] Low latency on large uploads - almost done but not quite.
2003-06-14 15:54 [LARTC] Low latency on large uploads - almost done but not quite Thilo Schulz
@ 2003-06-15 9:09 ` Stef Coene
2003-06-15 11:44 ` Thilo Schulz
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Stef Coene @ 2003-06-15 9:09 UTC (permalink / raw)
To: lartc
> Here's still my script, if you are interested to look at it.
I'm interested and I have some remarks.
> #!/bin/bash
>
> DEV=ppp0
>
> # delete any qdiscs or rule sets created so far.
> tc qdisc del dev $DEV root 2> /dev/null > /dev/null
> # tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null
>
> # create the root qdisc
> tc qdisc add dev $DEV root handle 1: htb default 13
>
> # install a root class, so that other clients can borrow from each other.
> tc class add dev $DEV parent 1: classid 1:1 htb rate 15kbps ceil 15kbps
>
> # now install 4 sub classes for different priorities
> # highest priority for low latency games like quake3 and ssh / ftp control.
> tc class add dev $DEV parent 1:1 classid 1:10 htb rate 7kbps ceil 15kbps \
> prio 0 burst 20000b cburst 22000b
> # not as high but still high priority for ACK's - useful for keeping large
> # d/l's alive :)
> tc class add dev $DEV parent 1:1 classid 1:11 htb rate 7kbps ceil 15kbps
> prio 1 burst 200b cburst 200b
> # very few data allowed for HTTP requests, but still higher priority than
> bulk uploads.
> tc class add dev $DEV parent 1:1 classid 1:12 htb rate 2kbps ceil 15kbps
> prio 10 burst 1b cburst 1b
> # bulk uploads have no prio :D
> tc class add dev $DEV parent 1:1 classid 1:13 htb rate 1bps ceil 15kbps
> prio 20 burst 1b cburst 1b
Your burst is too low. I understand you want a minimum burst, but you have to
follow some rules. The best you can do is to remove the burst/cburst option
so htb can calculate the minimum burst/cburst for you.
And don't you get quantum errors in your kernel log? That's because your
quantum is too low for the classes. There is a long explanation for this,
see www.docum.org on the faq page.
You also use different prio's. This can be ok in most cases, except if you
have a low prio class that's sending more data then the configured rate. If
you do so, the latency can go up for that class. I (still) didn't test it
myself, but you can find prove of it on the htb homepage. The solution for
this is to make sure you never put too much traffic in a low prio class.
> # now make all qdiscs simple pfifo
> # small queues for minimum latency
> tc qdisc add dev $DEV parent 1:10 handle 20: pfifo limit 0
> tc qdisc add dev $DEV parent 1:11 handle 30: pfifo limit 0
Are you sure limit 0 is possible ????
> # larger queues for more latency.
> tc qdisc add dev $DEV parent 1:12 handle 40: pfifo limit 5
> tc qdisc add dev $DEV parent 1:13 handle 50: pfifo limit 20
>
> #quake3-style udp games have been marked in iptables
> tc filter add dev $DEV protocol ip parent 1: prio 0 handle 1 fw flowid 1:10
> # icmp to get the response times.
> tc filter add dev $DEV protocol ip parent 1: prio 1 u32 match ip protocol 1
> 0xff flowid 1:10
> # ssh - not scp! scp is seperated by the TOS bits from ssh
> tc filter add dev $DEV protocol ip parent 1: prio 2 u32 match ip dport 22
> 0xffff match ip tos 0x10 0xff flowid 1:10
> # ftp
> tc filter add dev $DEV protocol ip parent 1: prio 3 u32 match ip dport 21
> 0xffff match ip tos 0x10 0xff flowid 1:10
> # ACK packets ..
> tc filter add dev $DEV protocol ip parent 1: prio 4 u32 match ip protocol 6
> 0xff match u8 0x05 0x0f at 0 match u16 0x0000 0xffc0 at 2 match u8 0x10
> 0xff at 33 flowid 1:11
> # HTTP requests
> tc filter add dev $DEV protocol ip parent 1: prio 10 u32 match ip dport 80
> 0xffff flowid 1:12
Stef
--
stef.coene@docum.org
"Using Linux as bandwidth manager"
http://www.docum.org/
#lartc @ irc.oftc.net
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [LARTC] Low latency on large uploads - almost done but not quite.
2003-06-14 15:54 [LARTC] Low latency on large uploads - almost done but not quite Thilo Schulz
2003-06-15 9:09 ` Stef Coene
@ 2003-06-15 11:44 ` Thilo Schulz
2003-06-15 12:00 ` Stef Coene
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Thilo Schulz @ 2003-06-15 11:44 UTC (permalink / raw)
To: lartc
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Sunday 15 June 2003 11:09, you wrote:
> > Here's still my script, if you are interested to look at it.
> I'm interested and I have some remarks.
> Your burst is too low. I understand you want a minimum burst, but you have
> to follow some rules. The best you can do is to remove the burst/cburst
> option so htb can calculate the minimum burst/cburst for you.
yes, sounds reasonable now that I spend a second thought about it.
> And don't you get quantum errors in your kernel log? That's because your
> quantum is too low for the classes. There is a long explanation for this,
> see www.docum.org on the faq page.
hmm .. quantum? I have never set quantum with any parameter, or have I?
> You also use different prio's. This can be ok in most cases, except if you
> have a low prio class that's sending more data then the configured rate.
> If you do so, the latency can go up for that class. I (still) didn't test
> it myself, but you can find prove of it on the htb homepage. The solution
> for this is to make sure you never put too much traffic in a low prio
> class.
I have given plenty of bandwidth to the 1:10 class. Quake3 streams are max.
1500 bytes/s. And ssh does not use that much either.
> > # now make all qdiscs simple pfifo
> > # small queues for minimum latency
> > tc qdisc add dev $DEV parent 1:10 handle 20: pfifo limit 0
> > tc qdisc add dev $DEV parent 1:11 handle 30: pfifo limit 0
>
> Are you sure limit 0 is possible ????
Yes, at least the status command showed me, that the limit was set to 0.
- Thilo Schulz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
iD8DBQE+7FwPZx4hBtWQhl4RAn8XAKDSJR6E7w3Q6I0ki4bVpDGfH//anwCfestd
aj5fVwoC9ANATJ1CA50N5P4=9XOi
-----END PGP SIGNATURE-----
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [LARTC] Low latency on large uploads - almost done but not quite.
2003-06-14 15:54 [LARTC] Low latency on large uploads - almost done but not quite Thilo Schulz
2003-06-15 9:09 ` Stef Coene
2003-06-15 11:44 ` Thilo Schulz
@ 2003-06-15 12:00 ` Stef Coene
2003-06-16 7:54 ` [LARTC] Low latency on large uploads - almost done but not Corey Rogers
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Stef Coene @ 2003-06-15 12:00 UTC (permalink / raw)
To: lartc
On Sunday 15 June 2003 13:44, Thilo Schulz wrote:
> On Sunday 15 June 2003 11:09, you wrote:
> > > Here's still my script, if you are interested to look at it.
> >
> > I'm interested and I have some remarks.
> >
> > Your burst is too low. I understand you want a minimum burst, but you
> > have to follow some rules. The best you can do is to remove the
> > burst/cburst option so htb can calculate the minimum burst/cburst for
> > you.
>
> yes, sounds reasonable now that I spend a second thought about it.
>
> > And don't you get quantum errors in your kernel log? That's because your
> > quantum is too low for the classes. There is a long explanation for
> > this, see www.docum.org on the faq page.
>
> hmm .. quantum? I have never set quantum with any parameter, or have I?
No. Quantum is used for leaf classes to determine the amount of packets they
can send. It's calculates as rate / r2q. And r2q is 10 by default. You can
overrule r2q if you add the htb qdisc and you can overrule quantum if you add
a htb class. Quantum must be > 1500 (the size of 1 packet) and < 60000.
> > You also use different prio's. This can be ok in most cases, except if
> > you have a low prio class that's sending more data then the configured
> > rate. If you do so, the latency can go up for that class. I (still)
> > didn't test it myself, but you can find prove of it on the htb homepage.
> > The solution for this is to make sure you never put too much traffic in a
> > low prio class.
>
> I have given plenty of bandwidth to the 1:10 class. Quake3 streams are max.
> 1500 bytes/s. And ssh does not use that much either.
Ok. As long as you are aware of the problem. You can also try to limit the
amount of packets the filters with a policer. So there are never too much
packets in a class.
> > > # now make all qdiscs simple pfifo
> > > # small queues for minimum latency
> > > tc qdisc add dev $DEV parent 1:10 handle 20: pfifo limit 0
> > > tc qdisc add dev $DEV parent 1:11 handle 30: pfifo limit 0
> >
> > Are you sure limit 0 is possible ????
>
> Yes, at least the status command showed me, that the limit was set to 0.
Ok.
Stef
--
stef.coene@docum.org
"Using Linux as bandwidth manager"
http://www.docum.org/
#lartc @ irc.oftc.net
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [LARTC] Low latency on large uploads - almost done but not
2003-06-14 15:54 [LARTC] Low latency on large uploads - almost done but not quite Thilo Schulz
` (2 preceding siblings ...)
2003-06-15 12:00 ` Stef Coene
@ 2003-06-16 7:54 ` Corey Rogers
2003-06-17 18:16 ` [LARTC] Low latency on large uploads - almost done but not quite sufcrusher
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Corey Rogers @ 2003-06-16 7:54 UTC (permalink / raw)
To: lartc
[-- Attachment #1: Type: text/plain, Size: 4021 bytes --]
I maybe a bit late but incase this helps anyone, I was trying to place
icmp in its own queue and give a high priority. I couldnt mark with
iptables due to the fact it core dumps when I attempt to use the mangle
chain (perhaps because these boxes also run freeswan. So i shaped based
on protocol number. My problem was it wasn't working until I specified a
prio for the icmp filter, also using prio 0 it did not work but it did
with prio 1.
tc filter add dev $DEV protocol ip parent 1: prio 1 u32 \
match ip protocol 1 0xff flowid 1:21
On Sun, 2003-06-15 at 08:00, Stef Coene wrote:
> On Sunday 15 June 2003 13:44, Thilo Schulz wrote:
> > On Sunday 15 June 2003 11:09, you wrote:
> > > > Here's still my script, if you are interested to look at it.
> > >
> > > I'm interested and I have some remarks.
> > >
> > > Your burst is too low. I understand you want a minimum burst, but you
> > > have to follow some rules. The best you can do is to remove the
> > > burst/cburst option so htb can calculate the minimum burst/cburst for
> > > you.
> >
> > yes, sounds reasonable now that I spend a second thought about it.
> >
> > > And don't you get quantum errors in your kernel log? That's because your
> > > quantum is too low for the classes. There is a long explanation for
> > > this, see www.docum.org on the faq page.
> >
> > hmm .. quantum? I have never set quantum with any parameter, or have I?
> No. Quantum is used for leaf classes to determine the amount of packets they
> can send. It's calculates as rate / r2q. And r2q is 10 by default. You can
> overrule r2q if you add the htb qdisc and you can overrule quantum if you add
> a htb class. Quantum must be > 1500 (the size of 1 packet) and < 60000.
>
> > > You also use different prio's. This can be ok in most cases, except if
> > > you have a low prio class that's sending more data then the configured
> > > rate. If you do so, the latency can go up for that class. I (still)
> > > didn't test it myself, but you can find prove of it on the htb homepage.
> > > The solution for this is to make sure you never put too much traffic in a
> > > low prio class.
> >
> > I have given plenty of bandwidth to the 1:10 class. Quake3 streams are max.
> > 1500 bytes/s. And ssh does not use that much either.
> Ok. As long as you are aware of the problem. You can also try to limit the
> amount of packets the filters with a policer. So there are never too much
> packets in a class.
>
> > > > # now make all qdiscs simple pfifo
> > > > # small queues for minimum latency
> > > > tc qdisc add dev $DEV parent 1:10 handle 20: pfifo limit 0
> > > > tc qdisc add dev $DEV parent 1:11 handle 30: pfifo limit 0
> > >
> > > Are you sure limit 0 is possible ????
> >
> > Yes, at least the status command showed me, that the limit was set to 0.
> Ok.
>
> Stef
--
Corey Rogers
Junior System Administrator
Wamco Technology Group Ltd (Barbados)
#3 Mahogany Court, Wildey, St. Michael
Phone: (246)437-3154 FAX: (246)228-4319
[F]or those of you who are constantly belittled by your peers for
believing that Big Brother is out to get you, be assured, it is. In
fact,you are probably not paranoid enough."
- editorial, "Today's Technology Can Easily Track Criminals and
Ex-offenders", _The_ECHO_ newspaper, Jan. 1998
CONFIDENTIALITY NOTICE: This e-mail message including attachments, if
any,is (are) for the intended recipient only (person or entity)and may
contain confidential or proprietary information some or all of which may
be legally privileged. Any unauthorised review, use, copy, print,
disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply e-mail and destroy all
copies of the original message and do not in any way rely on this
e-mail. If you are the intended recipient but do not wish to receive
communications through this medium, please so advise the sender
immediately.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [LARTC] Low latency on large uploads - almost done but not quite.
2003-06-14 15:54 [LARTC] Low latency on large uploads - almost done but not quite Thilo Schulz
` (3 preceding siblings ...)
2003-06-16 7:54 ` [LARTC] Low latency on large uploads - almost done but not Corey Rogers
@ 2003-06-17 18:16 ` sufcrusher
2003-06-18 12:32 ` Thilo Schulz
2003-06-18 19:10 ` sufcrusher
6 siblings, 0 replies; 8+ messages in thread
From: sufcrusher @ 2003-06-17 18:16 UTC (permalink / raw)
To: lartc
Playing CounterStrike myself regurlarly and being on a LAN with a few
professional P2Pers, I had the same problem and also experienced the
non-stable pings. At first I started experimenting with the rates and
ceilings, but in practice that didn't help much.
(one of) the reasons for the unstable ping is that a packet of ~ 1500 bytes
on a 128kbit connection (like yours and mine) takes roughly a 10th of a
second to send (100ms). So at the moment a large packet is being sent and a
quake packet is next in the queue, it still has to wait 100ms (worst case).
This latency of course adds to the normal latency you already have to the
quake server.
What does seem to help a little is lowering the maximum packet size (MTU) in
your routing table:
#!/bin/sh
oldroute=`ip route | grep default | cut -d' ' -f-5`
ip route change $oldroute mtu 500
The cut takes care of removing the 'mtu xxx' from a line like this (type "ip
route": to see it)
default via a.b.c.d dev eth0 mtu 900
It only takes the first 5 words. I presume that number of words can be
different in other situations, so you might have to adapt for that. (or use
something other than 'cut' to do it properly).
Note that the MTU only effects the outgoing packetsize, so downloads are not
affected at all. Uploads do get a little less efficient (the packetheaders
consist of a larger portion of the traffic) but in practise this is still
acceptable. Game packets are pretty small anyway, so won't be affected at
all.
I'm assuming the burst and quantum settings can be optimized for smaller
packet sizes to take full advantage of this. But to be honest I haven't
really done that yet.
I use a cable connection which has a more than 10 times faster download than
upload, so for me shaping the download isn't very effective. If you want to
limit the maximum packetsize for incoming packets as well (at least for TCP)
you can simply do this:
iptables -I PREROUTING -t mangle -i eth0 -j TCPMSS --set-mss 1000 -p
TCP --tcp-flags SYN,RST SYN
iptables -I INPUT -t mangle -i eth0 -j TCPMSS --set-mss
1000 -p TCP --tcp-flags SYN,RST SYN
For 1000 bytes packets. You can also use --clamp-mss-to-mtu option, which
probably makes sense. Note that the MSS thing only works for new
connections.
Note: After you change the MSS value, existing connections will still use
the old size. MTU changes have effect immediately.
Also make sure you patched the kernel to use the high resolution timer (info
at www.docum.org somewhere). That helped a lot in my case (you can put the
ceilingrates closer to the actual 128kbit and therefore reduce latency as
well). I'm not sure it's still necessary on 2.4.20 and/or 2.4.21.
Jannes Faber
----- Original Message -----
From: "Thilo Schulz" <arny@ats.s.bawue.de>
To: <lartc@mailman.ds9a.nl>
Sent: Saturday, June 14, 2003 5:54 PM
Subject: [LARTC] Low latency on large uploads - almost done but not quite.
Does anyone of you have an idea how I can minimize this effect, and let
pings
be stable at 60 ms? stable 80ms delay are okay for me too, no question.
If I let the worst-priority bulkdownload class ceil up only to 10kbyte/s I
have the same effect, only when the max ceil class is put down under 6 i do
not have this changing ping effect.
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [LARTC] Low latency on large uploads - almost done but not quite.
2003-06-14 15:54 [LARTC] Low latency on large uploads - almost done but not quite Thilo Schulz
` (4 preceding siblings ...)
2003-06-17 18:16 ` [LARTC] Low latency on large uploads - almost done but not quite sufcrusher
@ 2003-06-18 12:32 ` Thilo Schulz
2003-06-18 19:10 ` sufcrusher
6 siblings, 0 replies; 8+ messages in thread
From: Thilo Schulz @ 2003-06-18 12:32 UTC (permalink / raw)
To: lartc
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Tuesday 17 June 2003 20:16, sufcrusher wrote:
> (one of) the reasons for the unstable ping is that a packet of ~ 1500 bytes
> on a 128kbit connection (like yours and mine) takes roughly a 10th of a
> second to send (100ms). So at the moment a large packet is being sent and a
> quake packet is next in the queue, it still has to wait 100ms (worst case).
> This latency of course adds to the normal latency you already have to the
> quake server.
Yes, I have thought so too and thought about playing around with the MTU ..
but did not really want to change it yet.
Thank you anyways for these helpful hints, I am going to try it as soon as
possible :)
> I'm assuming the burst and quantum settings can be optimized for smaller
> packet sizes to take full advantage of this. But to be honest I haven't
> really done that yet.
The lower the burst settings, the less delay I have in theory, so high-prio
queues _definitely_ get their turn in time.
> I use a cable connection which has a more than 10 times faster download
> than upload, so for me shaping the download isn't very effective. If you
> want to limit the maximum packetsize for incoming packets as well (at least
> for TCP) you can simply do this:
The same applies to me, I have 768 kbyte/s downlink, I doubt that this is an
issue when gaming.
> For 1000 bytes packets. You can also use --clamp-mss-to-mtu option, which
> probably makes sense. Note that the MSS thing only works for new
> connections.
I have to do this anyways, as my pppoe is limiting the MTU on the virtual ppp0
device to 1492 because ethernet frames can only have a certain size and pppoe
still encapsules ppp in the ethernet.
> Also make sure you patched the kernel to use the high resolution timer
> (info at www.docum.org somewhere). That helped a lot in my case (you can
> put the ceilingrates closer to the actual 128kbit and therefore reduce
> latency as well). I'm not sure it's still necessary on 2.4.20 and/or
> 2.4.21.
I have a 133 Mhz AMD 486 - whether setting the resolution timer up would be
very good for performance I don't know.
- Thilo Schulz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
iD8DBQE+8FviZx4hBtWQhl4RAkyNAKC+3wE3bqMmQEr8qwkxdpPX6cuzdwCff03Z
6YTVuELLr1BNRPl/hym44Fw=4Qwt
-----END PGP SIGNATURE-----
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [LARTC] Low latency on large uploads - almost done but not quite.
2003-06-14 15:54 [LARTC] Low latency on large uploads - almost done but not quite Thilo Schulz
` (5 preceding siblings ...)
2003-06-18 12:32 ` Thilo Schulz
@ 2003-06-18 19:10 ` sufcrusher
6 siblings, 0 replies; 8+ messages in thread
From: sufcrusher @ 2003-06-18 19:10 UTC (permalink / raw)
To: lartc
You couldn't even if you wanted. High res timer requeres at least one of the
faster pentiums (not all pentiums can do it).
If you can't get a stable latency in the end, it might be worthwhile to
upgrade to a pentium, but for now I'd keep trying some finetuning.
Jannes Faber
----- Original Message -----
From: "Thilo Schulz" <arny@ats.s.bawue.de>
To: <lartc@mailman.ds9a.nl>
Sent: Wednesday, June 18, 2003 2:32 PM
Subject: Re: [LARTC] Low latency on large uploads - almost done but not
quite.
On Tuesday 17 June 2003 20:16, sufcrusher wrote:
> Also make sure you patched the kernel to use the high resolution timer
> (info at www.docum.org somewhere). That helped a lot in my case (you can
> put the ceilingrates closer to the actual 128kbit and therefore reduce
> latency as well). I'm not sure it's still necessary on 2.4.20 and/or
> 2.4.21.
I have a 133 Mhz AMD 486 - whether setting the resolution timer up would be
very good for performance I don't know.
- Thilo Schulz
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
^ permalink raw reply [flat|nested] 8+ messages in thread