[LARTC] Making tcp start transfers slow

[LARTC] Making tcp start transfers slow

[Patrick Petersen]

Hey list
I have almost gotten my shaping setup up and running as planned. The
last barrier seems to be tcp overshooting availible bandwidth when its
starting a transfer, and thereby bursting the line, so ping rises for a
moment. At least this is my best guess at the problem :)
There is a possibility that its just plain old traffic being bursty for
some reason.. I am using bittorrent to test this, as it seems to be what
stresses the line the most.
Would it be possible to lower the default window size, and thereby
making tcp start up slower, or would this just lower the overall speed?
My efforts so far can be seen here:
http://tc.schmakk.dk/betashaper
-- 
Patrick Petersen <lartc@schmakk.dk>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Making tcp start transfers slow

[Roy]

It is possible to slow down tcp start and it realy helped for me to get
better pings,
but expense is too high, so I think it do not work as it should,

if you start 5 coonections at once you receive 5  1.5kbyte packets, what
fills your queue at isp side
the more connections you start at once the worse delay will be.
this can be partialy fixed if you give some reserve using about 80-90% of
link like everybody usualy do

now I am working on new driver which could help to solve this,
the only way probably is predict new coonecions and reduce speed of
exsisting ones before new ones start.

----- Original Message ----- 
From: "Patrick Petersen" <lartc@schmakk.dk>
To: <lartc@mailman.ds9a.nl>
Sent: Thursday, April 15, 2004 1:26 PM
Subject: [LARTC] Making tcp start transfers slow


> Hey list
> I have almost gotten my shaping setup up and running as planned. The
> last barrier seems to be tcp overshooting availible bandwidth when its
> starting a transfer, and thereby bursting the line, so ping rises for a
> moment. At least this is my best guess at the problem :)
> There is a possibility that its just plain old traffic being bursty for
> some reason.. I am using bittorrent to test this, as it seems to be what
> stresses the line the most.
> Would it be possible to lower the default window size, and thereby
> making tcp start up slower, or would this just lower the overall speed?
> My efforts so far can be seen here:
> http://tc.schmakk.dk/betashaper
> -- 
> Patrick Petersen <lartc@schmakk.dk>
>
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Making tcp start transfers slow

[Andy Furniss]

Patrick Petersen wrote:
> Hey list
> I have almost gotten my shaping setup up and running as planned. The
> last barrier seems to be tcp overshooting availible bandwidth when its
> starting a transfer, and thereby bursting the line, so ping rises for a
> moment. At least this is my best guess at the problem :)

I sortof workaround this using the connbytes netfilter patch to make the 
first 80k of new connections go into a short queue limited to 1/3 - 1/2 
of my downstream bandwidth. It works well in the case where the link is 
empty apart from a gamestream and someone is browsing "heavy .jpg" type 
web paged. It also helps a bit if there is other traffic - but if there 
are enough tcp connections on the go there will be higher latency bursts
caused by new connections as HTB can't throttle until it's a bit too late.


> There is a possibility that its just plain old traffic being bursty for
> some reason.. I am using bittorrent to test this, as it seems to be what
> stresses the line the most.

Ahh bittorrent - this is a special case. It uses full duplex tcp - so 
may break some upstream shapers, you can assume that a fair number of 
your peers have flooded modem buffers - so there could be quite a few 
"unstoppable" packets headed your way. The worse thing about it, though 
is that it runs it's own algorithm on allready open tcps - so existing 
connections may go back into slowstart.

> Would it be possible to lower the default window size, and thereby
> making tcp start up slower, or would this just lower the overall speed?

It could help, but may also give you less of a share of a given 
uploaders bandwidth. Reducing MTU may also help (if you run BT on linux 
it may reduce rwin for you aswell).

> My efforts so far can be seen here:
> http://tc.schmakk.dk/betashaper

Had a quick look - Some thoughts:

I don't think you can catch all BT traffic by marking  the BT ports, I 
see ipp2p - can you do it with this or maybe do per IP fairness for bulk 
traffic?

Be carefull about priorotising acks - don't all TCP packets after syn 
have ack set. Being lazy on a home setup I get away with giving small 
packets priority - saves alot of marking :-)

For ingress shaping - I find it nicer to shape per IP with htb and use 
esfq classic to get per tcp fairness rather than esfq on dst which is 
going to effectively make many bittorrent connections go into a FIFO, 
which could make for more burstiness.

Andy.




_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Making tcp start transfers slow

[Roy]

I just had one idea, about this.

what if  make iptables module which will make something like enlarged copy
of syn packet and send it back to the sender?
(another option would be to kill 1 or 2 ack packets for one syn packet this
whould force server to reduce speed)


that way htb could count upcoming packet and prepare by reducing other
connetions speed?
of course that synthetic packet will have higest possible priority since it
supposed to be appear in future so  cant be shaped anyway


I will try to add this functionality to my imq module next week probably.
------------------------
connbytes solution is not good for this, it slows down small picture loading
in web pages very much, and big downloads get even more "unused" bandwitch.
so effect is not good. expecialy that looks bad on network, when pages
become incredibly slow, but big downloads fast.





----- Original Message ----- 
From: "Andy Furniss" <andy.furniss@dsl.pipex.com>
To: "Patrick Petersen" <lartc@schmakk.dk>
Cc: <lartc@mailman.ds9a.nl>
Sent: Sunday, April 18, 2004 2:13 AM
Subject: Re: [LARTC] Making tcp start transfers slow


> Patrick Petersen wrote:
> > Hey list
> > I have almost gotten my shaping setup up and running as
> planned. The
> > last barrier seems to be tcp overshooting availible bandwidth
> when its
> > starting a transfer, and thereby bursting the line, so ping
> rises for a
> > moment. At least this is my best guess at the problem
> :)
>
> I sortof workaround this using the connbytes netfilter patch to make the
> first 80k of new connections go into a short queue limited to 1/3 - 1/2
> of my downstream bandwidth. It works well in the case where the link is
> empty apart from a gamestream and someone is browsing '"'heavy .jpg'"'
type
> web paged. It also helps a bit if there is other traffic - but if there
> are enough tcp connections on the go there will be higher latency bursts
> caused by new connections as HTB can't throttle until it's a bit too late.
>
>
> > There is a possibility that its just plain old traffic being
> bursty for
> > some reason.. I am using bittorrent to test this, as it seems
> to be what
> > stresses the line the most.
>
> Ahh bittorrent - this is a special case. It uses full duplex tcp - so
> may break some upstream shapers, you can assume that a fair number of
> your peers have flooded modem buffers - so there could be quite a few
> '"'unstoppable'"' packets headed your way. The worse thing about it,
though
> is that it runs it's own algorithm on allready open tcps - so existing
> connections may go back into slowstart.
>
> > Would it be possible to lower the default window size, and
> thereby
> > making tcp start up slower, or would this just lower the
> overall speed?
>
> It could help, but may also give you less of a share of a given
> uploaders bandwidth. Reducing MTU may also help (if you run BT on linux
> it may reduce rwin for you aswell).
>
> > My efforts so far can be seen here:
> > http://tc.schmakk.dk/betashaper
>
> Had a quick look - Some thoughts:
>
> I don't think you can catch all BT traffic by marking  the BT ports, I
> see ipp2p - can you do it with this or maybe do per IP fairness for bulk
> traffic?
>
> Be carefull about priorotising acks - don't all TCP packets after syn
> have ack set. Being lazy on a home setup I get away with giving small
> packets priority - saves alot of marking :-)
>
> For ingress shaping - I find it nicer to shape per IP with htb and use
> esfq classic to get per tcp fairness rather than esfq on dst which is
> going to effectively make many bittorrent connections go into a FIFO,
> which could make for more burstiness.
>
> Andy.
>
>
>
>
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Making tcp start transfers slow

[Andy Furniss]

Roy wrote:
> I just had one idea, about this.
> 
> what if  make iptables module which will make something like enlarged copy
> of syn packet and send it back to the sender?
> (another option would be to kill 1 or 2 ack packets for one syn packet this
> whould force server to reduce speed)
> 
> 
> that way htb could count upcoming packet and prepare by reducing other
> connetions speed?
> of course that synthetic packet will have higest possible priority since it
> supposed to be appear in future so  cant be shaped anyway
> 

I don't really get this :-)


> 
> I will try to add this functionality to my imq module next week probably.
> ------------------------
> connbytes solution is not good for this, it slows down small picture loading
> in web pages very much, and big downloads get even more "unused" bandwitch.
> so effect is not good. expecialy that looks bad on network, when pages
> become incredibly slow, but big downloads fast.

Depends on lots of things I suppose - the way I have it set new 
connections get 256kbit - not that bad for browsing. ISTR seeing one of 
your scripts that did similar, IIRC using sfq with low rates. I don't 
quite do it like that - for a start sfq 128 queue length is too much and 
if you use it on ingress sfq will hash the ~4 simoultaneous connections 
your browser makes into one slot. I guess yours simulated a drop with 
the reordering when they swapped queues rather than really dropping with 
a short queue to get out of slowstart.

SFQ causes instability every time it rehashes on ingress because of this 
- there is a todo in the code somewhere. I like to set perturb high.

This ingress shaping with stuff made for egress is a bit tricky - but it 
can be tweaked a bit.

Andy.



_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Making tcp start transfers slow

[Roy]

> Roy wrote:
> > I just had one idea, about this.
> >
> > what if  make iptables module which will make something like
> enlarged copy
> > of syn packet and send it back to the sender?
> > (another option would be to kill 1 or 2 ack packets for one syn
> packet this
> > whould force server to reduce speed)
> >
> >
> > that way htb could count upcoming packet and prepare by
> reducing other
> > connetions speed?
> > of course that synthetic packet will have higest possible
> priority since it
> > supposed to be appear in future so  cant be shaped
> anyway
> >
>
> I don't really get this :-)
>
-----------------------------------------------
Ok, probably I was not able to explain it quite well,
but basicaly it is how to predict incomming connections and
decrease speed of exsisting one before new connections will start
this prediction method should be simple and easy to implement

(all concept of slow start is wrong, we exactly need  fast start and slow
download later)


> >
> > I will try to add this functionality to my imq module next week
> probably.
> > ------------------------
> > connbytes solution is not good for this, it slows down small
> picture loading
> > in web pages very much, and big downloads get even more
> '"'unused'"' bandwitch.
> > so effect is not good. expecialy that looks bad on network,
> when pages
> > become incredibly slow, but big downloads fast.
>
> Depends on lots of things I suppose - the way I have it set new
> connections get 256kbit - not that bad for browsing. ISTR seeing one of
> your scripts that did similar, IIRC using sfq with low rates. I don't
> quite do it like that - for a start sfq 128 queue length is too much and
> if you use it on ingress sfq will hash the ~4 simoultaneous connections
> your browser makes into one slot. I guess yours simulated a drop with
> the reordering when they swapped queues rather than really dropping with
> a short queue to get out of slowstart.
>
> SFQ causes instability every time it rehashes on ingress because of this
> - there is a todo in the code somewhere. I like to set perturb high.
>
> This ingress shaping with stuff made for egress is a bit tricky - but it
> can be tweaked a bit.
>
> Andy.
>

all this work well, on small number of new coonections at once , but try 20
or more at once and you will see that it is not good at all
I am also using this way now, and that why I say it is not good.

I dont think sfq may create any prolems, because it is basicaly same as few
random fifo's


> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Making tcp start transfers slow

[Patrick Petersen]

Sorry for the long response time.

On Sun, 18 Apr 2004 00:13:49 +0100, Andy Furniss <andy.furniss@dsl.pipex.com> wrote:

> I sortof workaround this using the connbytes netfilter patch to make the 
> first 80k of new connections go into a short queue limited to 1/3 - 1/2 
> of my downstream bandwidth.

I will try this soon.. I recently screwed my kernel over bad with
patch-o-matic, so its fixing time at the moment :)

> Ahh bittorrent - this is a special case. It uses full duplex tcp - so 
> may break some upstream shapers, you can assume that a fair number of 
> your peers have flooded modem buffers - so there could be quite a few 
> "unstoppable" packets headed your way. The worse thing about it, though 
> is that it runs it's own algorithm on allready open tcps - so existing 
> connections may go back into slowstart.

Thats pretty bad.. But if i get this working, even with bittorrent, i
guess that means it will probably work with anything! :)

> > Would it be possible to lower the default window size, and thereby
> > making tcp start up slower, or would this just lower the overall speed?
> 
> It could help, but may also give you less of a share of a given 
> uploaders bandwidth. Reducing MTU may also help (if you run BT on linux 
> it may reduce rwin for you aswell).
> 
> I don't think you can catch all BT traffic by marking  the BT ports, I 
> see ipp2p - can you do it with this or maybe do per IP fairness for bulk 
> traffic?

Im fortunate enough to be able to run a small network with only a few
people, so theres no real bad things with people doing whatever they can
to cheat. Also, ipp2p should catch torrent connections, and seems to be
doing so.

> Be carefull about priorotising acks - don't all TCP packets after syn 
> have ack set. Being lazy on a home setup I get away with giving small 
> packets priority - saves alot of marking :-)
> 
> For ingress shaping - I find it nicer to shape per IP with htb and use 
> esfq classic to get per tcp fairness rather than esfq on dst which is 
> going to effectively make many bittorrent connections go into a FIFO, 
> which could make for more burstiness.

I'll make a note, but shouldnt esfq make things more or less fair no
matter how much traffic one client has over another?

-- 
Patrick Petersen <lartc@schmakk.dk>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Making tcp start transfers slow

[Andy Furniss]

Patrick Petersen wrote:
> Sorry for the long response time.
> 
> On Sun, 18 Apr 2004 00:13:49 +0100, Andy Furniss <andy.furniss@dsl.pipex.com> wrote:
> 
> 
>>I sortof workaround this using the connbytes netfilter patch to make the 
>>first 80k of new connections go into a short queue limited to 1/3 - 1/2 
>>of my downstream bandwidth.
> 
> 
> I will try this soon.. I recently screwed my kernel over bad with
> patch-o-matic, so its fixing time at the moment :)

One possible problem - I don't think connbytes will patch with connmark 
(which IIRC ipp2p uses). I have been told that it is possible to get 
them to work, but patch will fail as they both change the same struct, 
so it's a do it by hand job...

> 
> 
>>Ahh bittorrent - this is a special case. It uses full duplex tcp - so 
>>may break some upstream shapers, you can assume that a fair number of 
>>your peers have flooded modem buffers - so there could be quite a few 
>>"unstoppable" packets headed your way. The worse thing about it, though 
>>is that it runs it's own algorithm on allready open tcps - so existing 
>>connections may go back into slowstart.
> 
> 
> Thats pretty bad.. But if i get this working, even with bittorrent, i
> guess that means it will probably work with anything! :)

LOL - I think your idea about reducing rwin may help - easy on a small 
setup where you have control.

> 
> 
>>>Would it be possible to lower the default window size, and thereby
>>>making tcp start up slower, or would this just lower the overall speed?
>>
>>It could help, but may also give you less of a share of a given 
>>uploaders bandwidth. Reducing MTU may also help (if you run BT on linux 
>>it may reduce rwin for you aswell).
>>
>>I don't think you can catch all BT traffic by marking  the BT ports, I 
>>see ipp2p - can you do it with this or maybe do per IP fairness for bulk 
>>traffic?
> 
> 
> Im fortunate enough to be able to run a small network with only a few
> people, so theres no real bad things with people doing whatever they can
> to cheat. Also, ipp2p should catch torrent connections, and seems to be
> doing so.

Yes - so you don't need to mark all those ports then.


> 
> 
>>Be carefull about priorotising acks - don't all TCP packets after syn 
>>have ack set. Being lazy on a home setup I get away with giving small 
>>packets priority - saves alot of marking :-)
>>
>>For ingress shaping - I find it nicer to shape per IP with htb and use 
>>esfq classic to get per tcp fairness rather than esfq on dst which is 
>>going to effectively make many bittorrent connections go into a FIFO, 
>>which could make for more burstiness.
> 
> 
> I'll make a note, but shouldnt esfq make things more or less fair no
> matter how much traffic one client has over another?

Yes esfq on dst will make things fair per IP address - but in the case 
of ingress with the many tcps that BT uses dropping is important to get 
the sender to back off. Esfq on dst is not really a fairness queue any 
more WRT individual tcp connections, it may not make much difference, 
but if you use htb for per IP fairness and esfq classic, then the drops 
are a bit more likely to get at the most active connections.

I modified esfq to drop from the head of a slot rather than the end - 
which in theory should make things slightly nicer and changed hash to 
use dst port. I really ought to make them into a switches (when I work 
out how) I also intend to patch with the change to sfq that was recently 
talked about on here. If it looks OK I'll put it up on my webspace 
sometime - I don't really know what I am doing though :-)

For us small bandwidth users a R(real)FQ would be nice - (e)sfq is OK 
but it often hashes into the same slot and perturb causes packet 
reordering which hurts more when used for ingress.

Andy.


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Making tcp start transfers slow

[Patrick Petersen]

On Sat, 24 Apr 2004 00:11:32 +0100, Andy Furniss <andy.furniss@dsl.pipex.com> wrote:

> One possible problem - I don't think connbytes will patch with connmark 
> (which IIRC ipp2p uses). I have been told that it is possible to get 
> them to work, but patch will fail as they both change the same struct, 
> so it's a do it by hand job...

Im usually all for the "do it by hand" thing, but im not really up for
screwing with this yet i think :)
Its correct that ipp2p uses connmark... That way it is only necessary to
tag a connection once, then leave it. Great for those huge setups, where
the ipp2p usage should be kept at a minimum.. Afaik its a bit heavy on
ressources.

[SNIP - torrent talk]
> LOL - I think your idea about reducing rwin may help - easy on a small 
> setup where you have control.

That was my idea, but so far ive only found out how to alter the window
size on a per route basis. If it could just be extended a little by
making new connections have a small size, then putting them back to
normal after a while. But if i went all the way out there, i might just
as well start a complete dynamic window chaning thing.

[SNIP - more torrent and ipp2p]
> Yes - so you don't need to mark all those ports then.

Probably not, but i dont think the extra overhead will do any harm as
long as its a puny adsl im dealing with :) I guess im not trusting ipp2p
enough.

> Yes esfq on dst will make things fair per IP address - but in the case 
> of ingress with the many tcps that BT uses dropping is important to get 
> the sender to back off. Esfq on dst is not really a fairness queue any 
> more WRT individual tcp connections, it may not make much difference, 
> but if you use htb for per IP fairness and esfq classic, then the drops 
> are a bit more likely to get at the most active connections.
> 
> I modified esfq to drop from the head of a slot rather than the end - 
> which in theory should make things slightly nicer and changed hash to 
> use dst port. I really ought to make them into a switches (when I work 
> out how) I also intend to patch with the change to sfq that was recently 
> talked about on here. If it looks OK I'll put it up on my webspace 
> sometime - I don't really know what I am doing though :-)

Sounds nice, share when you think its stable. Im all for betatesting too
if you need any.

> For us small bandwidth users a R(real)FQ would be nice - (e)sfq is OK 
> but it often hashes into the same slot and perturb causes packet 
> reordering which hurts more when used for ingress.

I will try fiddling without esfq with fingers crossed - thanks.

-- 
Patrick Petersen <lartc@schmakk.dk>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/