[LARTC] Latency low enough for gaming

[LARTC] Latency low enough for gaming

[Patrick Petersen]

Welcome to me, and my very first lartc post.
As with most first timers, i made a mistake. Admin, disregard the
earlier message, as i was still waiting for the subscription
confirmation. Should it get through still, i apoligize.

For the last few weeks i have been trying to make it so our 2048/512
adsl line can be used for gaming and for leeching at the same time. The
current result is what can be found at http://www.schmakk.dk/~schmakk
which is what is running at the NAT gateway. This has done a lot for the
latency, but still there is huge problems with eg. massive http
downloads (5+ threads makes ping go up to at least 200).

I have learned a lot from the lartc list archive, but this specifik
leaves me with no clue. I have been able to get real close to normal
latency by capping incoming traffic at around 1200kbits, but its no fun
throwing away almost half your bandwidth.

Can i get any recommendations?

Also, if you have the time, a look through my script is much appreciated.
(Im concerned about the calculations for dividing the bandwidth, the
general setup of everything and the ipp2p+connmark tagging.)
-- 
Patrick Petersen <lartc@schmakk.dk>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Latency low enough for gaming

[Patrick Petersen]

On Mon, 16 Feb 2004 05:34:20 +0200, Nuutti Kotivuori <naked@iki.fi> wrote:

> Let's get the problem statement clear first.

[snip - lots of tcp]

> What can be done to improve the situation then?

[SNIP - more info and advice]

I think this is where it breaks for me. The solution looks to be just to 
replace esfq with eg the fifo qdisc? Or perhaps something that will do a
little fair sharing without queuing, if that exists.

[SNIP - use RED]

I am working on making RED do my magic, but until now im a little unsure
of the result. Ive started from scratch, and it seems that the RED setup
im trying from Jim diGriz is working a little. At least a few packets
are dropped when the link is under use, and transferrate seems to
remain at configured level. A guided tour of RED is much appreciated, as
i find the sparse info online to be difficult.

[SNIP - carefull with ingress]

In the new script, im trying to always let important packets go through,
so only low priority packets gets dropped. Things such as ACK and the
like are prioritized over other things, and should hopefulle be accepted
no matter what, and if bandwidth is full, unimportant traffic is dropped.

> Long rant, phew,

Its fine with me. Sometimes getting things cut in cardboard helps one
way or the other. Im still learning.

I've put the beta of the new shaperscript here: http://schmakk.dk/~schmakk/
Its ugly, and might not even work at the moment.

Can somebody explain to me where IMQ hoks into the packet stream and
where it is put back in? Im thinking the packets should go out through
imq at the iptables rule, but i think im wrong.

-- 
Patrick Petersen <lartc@schmakk.dk>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Latency low enough for gaming

[Andy Furniss]

Patrick Petersen wrote:
> Welcome to me, and my very first lartc post.
> As with most first timers, i made a mistake. Admin, disregard the
> earlier message, as i was still waiting for the subscription
> confirmation. Should it get through still, i apoligize.
> 
> For the last few weeks i have been trying to make it so our 2048/512
> adsl line can be used for gaming and for leeching at the same time. The
> current result is what can be found at http://www.schmakk.dk/~schmakk
> which is what is running at the NAT gateway. This has done a lot for the
> latency, but still there is huge problems with eg. massive http
> downloads (5+ threads makes ping go up to at least 200).
> 
> I have learned a lot from the lartc list archive, but this specifik
> leaves me with no clue. I have been able to get real close to normal
> latency by capping incoming traffic at around 1200kbits, but its no fun
> throwing away almost half your bandwidth.
> 
> Can i get any recommendations?
> 
> Also, if you have the time, a look through my script is much appreciated.
> (Im concerned about the calculations for dividing the bandwidth, the
> general setup of everything and the ipp2p+connmark tagging.)

I see you have a newer version now anyway, but I tried you script last 
night (not connmark/ipp2p as it clashed with connbytes). I have 256/512 
so things in theory should be nicer for you.

I am still testing myself, so can't post a solution but can make some 
observations -

The delete rule for iptabled needs -D not -A.

esfq won't work hashing src on egress if you are doing NAT see the KPTD 
on www.docum.org - egress qos comes after NAT. Ingress with dst should 
be ok if you are NATing as long as you used the IMQ NAT patch.

The trouble with esfq hashing per user (unless you use limit) is that 
each user gets a 128 packet queue, which if they have many connections 
gets full and drops take a long time to be noticed. I have a modified 
esfq which overcomes this somewhat, but I also use classic hash and 
restrict the size of the queue.

I can see why commercial bandwidth controllers use advertised window 
manipulation - often dropping is needed to get the sender to back off a 
bit and set its' congestion window, but if you queue this may result in 
a resync burst later. Being able to reduce adv window on dups/sacks and 
increase slowly/randomly would be handy.

One thing that helps me is to give new bulk connections their own class 
with a short queue for the first 80000 bytes using connbytes (netfilter 
extra patch). This is limited to rate downrate/5 ceil /3 and stops tcp 
slowstart from overshooting. I have also tried connbytes just to drop 
early packets, but with browsers making many simultaneous connections, 
the resyncs cause a latency burst.

I see you are trying RED - in theory this should be nice, but remember 
the settings/docs you read about don't take into account that you are 
trying to shape behind a fifo (at ISP/teleco) that dequeues only 20% 
faster than what you are aiming for.

I am not convinced that just dropping ingress is best - a short queue 
yes, then at least you don't ever drop game packets.

If I had 2048 down I reckon I could keep down below 100 now - apart from 
the odd 1 sec blip.

Andy.

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Latency low enough for gaming

[Patrick Petersen]

On Thu, 19 Feb 2004 09:45:16 +0000, Andy Furniss <andy.furniss@dsl.pipex.com> wrote:

> The delete rule for iptabled needs -D not -A.

Yes, that one was bad. I noticed it when i discovered how to list rules
in chains... I think all my rules was there about 10 times each, since i
never removed anything :)

> esfq won't work hashing src on egress if you are doing NAT see the KPTD 
> on www.docum.org - egress qos comes after NAT. Ingress with dst should 
> be ok if you are NATing as long as you used the IMQ NAT patch.

I thought that with the NAT patch, imq would see incoming packets with
the real ip on the internal net?

> The trouble with esfq hashing per user (unless you use limit) is that 
> each user gets a 128 packet queue, which if they have many connections 
> gets full and drops take a long time to be noticed. I have a modified 
> esfq which overcomes this somewhat, but I also use classic hash and 
> restrict the size of the queue.

I didnt thing a 128 packet queue would do any real difference, but im
testing with other qdiscs at the moment, since it seems that bandwidth
is being divided, but there is still latency problems.

> I can see why commercial bandwidth controllers use advertised window 
> manipulation - often dropping is needed to get the sender to back off a 
> bit and set its' congestion window, but if you queue this may result in 
> a resync burst later. Being able to reduce adv window on dups/sacks and 
> increase slowly/randomly would be handy.

Ah yes, the holy grail it seems. Its a mystery that noone has started an
open source project for this.

> One thing that helps me is to give new bulk connections their own class 
> with a short queue for the first 80000 bytes using connbytes (netfilter 
> extra patch). This is limited to rate downrate/5 ceil /3 and stops tcp 
> slowstart from overshooting. I have also tried connbytes just to drop 
> early packets, but with browsers making many simultaneous connections, 
> the resyncs cause a latency burst.

If im getting this right, you are using iptables to manage bandwidth
directly? Im real bad with iptables still, i dont think ive gotten to
know half of it yet.

> I see you are trying RED - in theory this should be nice, but remember 
> the settings/docs you read about don't take into account that you are 
> trying to shape behind a fifo (at ISP/teleco) that dequeues only 20% 
> faster than what you are aiming for.

Im still kind of blank on RED, what im trying out now is to use the RED
part of Jim diGriz' (i think) script. It seems that a few packets are
actually dropped when the link is getting full, but only about 5-10 in a
couple of minutes.. Seems a bit low?

> I am not convinced that just dropping ingress is best - a short queue 
> yes, then at least you don't ever drop game packets.

This is what im trying to do now, using IMQ for incoming traffic.
However, it seems that my 2 root qdiscs are delaying packets a lot.
According to tc -s qdisc etc etc about 100-500 packets are overlimits,
even when dataflow is no more than around 5-10kb/s. Setting a ceil on
the root classes seems to help it out a little, but not completely. This
i dont understand.
-- 
Patrick Petersen <lartc@schmakk.dk>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Latency low enough for gaming

[Andy Furniss]

Patrick Petersen wrote:

>>esfq won't work hashing src on egress if you are doing NAT see the KPTD 
>>on www.docum.org - egress qos comes after NAT. Ingress with dst should 
>>be ok if you are NATing as long as you used the IMQ NAT patch.
> 
> 
> I thought that with the NAT patch, imq would see incoming packets with
> the real ip on the internal net?

Yes - it is OK for incoming, but not for outbound AFAIK whether you use 
imq or the interface direct. The NAT patch for IMQ only changes the 
ingress hooks, not egress.

> 
> 
>>The trouble with esfq hashing per user (unless you use limit) is that 
>>each user gets a 128 packet queue, which if they have many connections 
>>gets full and drops take a long time to be noticed. I have a modified 
>>esfq which overcomes this somewhat, but I also use classic hash and 
>>restrict the size of the queue.
> 
> 
> I didnt thing a 128 packet queue would do any real difference, but im
> testing with other qdiscs at the moment, since it seems that bandwidth
> is being divided, but there is still latency problems.

Are the problems brief or does it totally loose it. I just tested the 
ingress policer at 80% and with 8 tcps going it looses control.

> 
> 
>>I can see why commercial bandwidth controllers use advertised window 
>>manipulation - often dropping is needed to get the sender to back off a 
>>bit and set its' congestion window, but if you queue this may result in 
>>a resync burst later. Being able to reduce adv window on dups/sacks and 
>>increase slowly/randomly would be handy.
> 
> 
> Ah yes, the holy grail it seems. Its a mystery that noone has started an
> open source project for this.
> 
> 
>>One thing that helps me is to give new bulk connections their own class 
>>with a short queue for the first 80000 bytes using connbytes (netfilter 
>>extra patch). This is limited to rate downrate/5 ceil /3 and stops tcp 
>>slowstart from overshooting. I have also tried connbytes just to drop 
>>early packets, but with browsers making many simultaneous connections, 
>>the resyncs cause a latency burst.
> 
> 
> If im getting this right, you are using iptables to manage bandwidth
> directly? Im real bad with iptables still, i dont think ive gotten to
> know half of it yet.

I did, just experementing try dropping an early packet from new 
connections. It was better in some cases, but not as good as putting new 
connections in their own limited bandwidth short queue. This still 
causes drops, but delays the packets aswell.

I use the netfilter connbytes patch to mark < 80000 bytes then put them 
in their own htb class.


> 
> 
>>I see you are trying RED - in theory this should be nice, but remember 
>>the settings/docs you read about don't take into account that you are 
>>trying to shape behind a fifo (at ISP/teleco) that dequeues only 20% 
>>faster than what you are aiming for.
> 
> 
> Im still kind of blank on RED, what im trying out now is to use the RED
> part of Jim diGriz' (i think) script. It seems that a few packets are
> actually dropped when the link is getting full, but only about 5-10 in a
> couple of minutes.. Seems a bit low?

I guess it depends on how many connections - just queuing will throttle 
a low number without many drops. I tested your RED settings with 8 and 
it handled it OK and drops enough. It temporarily looses it when a new 
connection is started - but it's hard to stop this. It also causes a 
blip when my link is otherwise empty - putting new connections into a 
restricted seperate class should stops this.

I think it will be hard to get RED perfect as it really ought to live at 
your ISP, before the bottleneck, but it's still worth seeing how well it 
can be tweaked.


> 
> 
>>I am not convinced that just dropping ingress is best - a short queue 
>>yes, then at least you don't ever drop game packets.
> 
> 
> This is what im trying to do now, using IMQ for incoming traffic.
> However, it seems that my 2 root qdiscs are delaying packets a lot.
> According to tc -s qdisc etc etc about 100-500 packets are overlimits,
> even when dataflow is no more than around 5-10kb/s. Setting a ceil on
> the root classes seems to help it out a little, but not completely. This
> i dont understand.

This is OK, you want to see overlimits - tcp will send packets in bursts 
and these will come in at full link speed, and so be seen by HTB as 
overlimits.

Andy.

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/