[LARTC] TCP Rate Control

[LARTC] TCP Rate Control

[Varun Varma]

Hi!

Take a setup as follows:

Clients---->[eth0]Proxy/Firewall[eth1]---->Internet

The clients are using the proxy/firewall for various client like 
activities like browsing, email, ftp etc. Most of the traffic is 
"download" traffic, i.e. clients send very small requests and recieve 
very large replies.

I want to do the most common thing...configure download limits/fairness 
of use among the clients.

I understand that I can achieve this by traffic shaping the *outgoing* 
traffic on eth0, but that is a less than optimal solution. The problem 
is that the link to the Internet can still get choked, if too many 
clients start pulling too much data. Traffic shaping on eth0 would limit 
the bandwidth each client sees and TCPs congestion control mechanisms 
*might* indeed slow down the download requests from clients, but this 
does not seem like a elegant way to achieve the result.

I am wondering if there is any implementation [or planned project] for 
TCP based rate control, like that provided by commercial solutions like 
PacketShaper and Sitara.

Regards,
-Varun

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] TCP Rate Control

[Stef Coene]

On Wednesday 14 May 2003 13:24, Varun Varma wrote:
> Hi!
>
> Take a setup as follows:
>
> Clients---->[eth0]Proxy/Firewall[eth1]---->Internet
>
> The clients are using the proxy/firewall for various client like
> activities like browsing, email, ftp etc. Most of the traffic is
> "download" traffic, i.e. clients send very small requests and recieve
> very large replies.
>
> I want to do the most common thing...configure download limits/fairness
> of use among the clients.
>
> I understand that I can achieve this by traffic shaping the *outgoing*
> traffic on eth0, but that is a less than optimal solution. The problem
> is that the link to the Internet can still get choked, if too many
> clients start pulling too much data. Traffic shaping on eth0 would limit
> the bandwidth each client sees and TCPs congestion control mechanisms
> *might* indeed slow down the download requests from clients, but this
> does not seem like a elegant way to achieve the result.
Why not?
The only problem you will have is the Proxy.  If you shape on eth0, you shape 
cached objects and objects fetched from the internet.  But you don't want to 
limit traffic that's cached.  So somehow you have to be able to separate 
cached and non cached traffic.

> I am wondering if there is any implementation [or planned project] for
> TCP based rate control, like that provided by commercial solutions like
> PacketShaper and Sitara.
You can do the same (and even more) with a linux box.

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] TCP Rate Control

[Brandis Jaroslav]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="windows-1253", Size: 383 bytes --]

> >I am wondering if there is any implementation [or planned project] for
> >TCP based rate control, like that provided by commercial solutions like
> >PacketShaper and Sitara.

 
>You can do the same (and even more) with a linux box.

Can you write how to do it ? 
 
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿË\x01\x14™¨¥Šx%ŠË\x7f,\x04S\vùšŠYšŸ÷lõ¯ç–^[m§ÿÿ™¨¥™©ÿvÏZþy\x7f™¨¥™©ÿ–+-ŠwèþV«µÁÎY3ÿ†Ûiÿÿåj»\þŠà

Re: [LARTC] TCP Rate Control

[Stef Coene]

On Wednesday 14 May 2003 21:59, Brandis Jaroslav wrote:
> > >I am wondering if there is any implementation [or planned project] for
> > >TCP based rate control, like that provided by commercial solutions like
> > >PacketShaper and Sitara.
> >
> >You can do the same (and even more) with a linux box.
>
> Can you write how to do it ?
www.lartc.org
But what exactly do you mean with tcp based rate control?

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] TCP Rate Control

[Jose Luis Domingo Lopez]

On Wednesday, 14 May 2003, at 22:09:58 +0200,
Stef Coene wrote:

> But what exactly do you mean with tcp based rate control?
> 
I suppose he means the way products like PacketShaper (from Packeteer)
work: instead of using queues to limit traffic, the use waht they call
"TCP rate control". It consists of modifiying TCP "packets" on the fly,
transparently, adapting TCP window sizes, shortly delaying ACKs, and so
on, so the sender and receiver "agree" to send traffic at a lower
(configured) rate.

Seems to be a patented "idea" in the USA, but I remember someone on this
list talked not long ago about he waaas implementing something like this
for Linux, from outside the USA. Check the archives for the message:
Subject: Re: [LARTC] How far can TC go?
From: Patrick McHardy <kaber@trash.net>
Date: Sat, 29 Mar 2003 11:08:30 +0100

Hope it helps.

-- 
Jose Luis Domingo Lopez
Linux Registered User #189436     Debian Linux Sid (Linux 2.5.69)
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] TCP Rate Control

[Stef Coene]

On Thursday 15 May 2003 01:06, Jose Luis Domingo Lopez wrote:
> On Wednesday, 14 May 2003, at 22:09:58 +0200,
>
> Stef Coene wrote:
> > But what exactly do you mean with tcp based rate control?
>
> I suppose he means the way products like PacketShaper (from Packeteer)
> work: instead of using queues to limit traffic, the use waht they call
> "TCP rate control". It consists of modifiying TCP "packets" on the fly,
> transparently, adapting TCP window sizes, shortly delaying ACKs, and so
> on, so the sender and receiver "agree" to send traffic at a lower
> (configured) rate.
You mean something like the gred or red qdisc.
But I wonder if it will be a big different between "TCP rate contorl" and 
plain old shaping.

> Seems to be a patented "idea" in the USA, but I remember someone on this
> list talked not long ago about he waaas implementing something like this
> for Linux, from outside the USA. Check the archives for the message:
> Subject: Re: [LARTC] How far can TC go?
> From: Patrick McHardy <kaber@trash.net>
> Date: Sat, 29 Mar 2003 11:08:30 +0100
>
> Hope it helps.
ok.

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] TCP Rate Control

[Stef Coene]

On Thursday 15 May 2003 19:41, Varun Varma wrote:
> Stef Coene wrote:
> > You mean something like the gred or red qdisc.
>
> The difference is RED drops the packets after they arrive basing it on
> the understanding that the TCP implementations would take this as a hint
> and slow down the transmission rates.
>
> Rate control on the other hand takes a more "hands on" approach to the
> problem. As Jose mentioned, ack pacing, window sizing etc...it actually
> exploits fundamentals in TCP flow control itself.
>
> > But I wonder if it will be a big different between "TCP rate contorl" and
> > plain old shaping.
>
> There are lot of comparisions on this. I don't want to get into a debate
>   here, since people from each group [rate control vs. queueing] feel
> very stongly about their stand.
But I'm still intested in the comparision :)

> Simply put, there are dozens of queueing disciplines, because some might
> "behave better" than other in some cases. I *feel* that rate control
> would work better in some particular cases, so I was interested in
> knowing if a rate control like implementation is available under Linux.
Not in the default linux kernel.

> >>Seems to be a patented "idea" in the USA, but I remember someone on this
> >>list talked not long ago about he waaas implementing something like this
> >>for Linux, from outside the USA. Check the archives for the message:
> >>Subject: Re: [LARTC] How far can TC go?
> >>From: Patrick McHardy <kaber@trash.net>
> >>Date: Sat, 29 Mar 2003 11:08:30 +0100
>
> Thanks for the reference...still need to check it out.
I already did :

Packeteer has various patents covering tcp rate control and everything else
they do, including the "idea" to look at upper layers to detect the type 
of traffic.
I live in germany so i don't really care that much about their patents 
(they had none in europe last time i checked). last summer i started 
implementing tcp rate control as qdisc for linux. i haven't worked on it for 
a couple of month now,  but if anyone wishes to participate i would be glad 
to dig out my source again. it is basically working, the remaining problems 
are mostly how to detect and handle interactive traffic.

Patrick

> About the patent thing...I know for sure that ack pacing has been
> published in paper before. Note: I am not suggesting how to circumvent
> this patent in any way. That said, I think it should be possible to work
> out other mechanisms, based around TCP flow control, which don't work
> the way Packeteer's patent works.
>
> If you google for "linux rate control", you come up with a link like:
>
> http://www.postel.org/pipermail/end2end-interest/2001-April/000701.html
>
> Which is a master's thesis, proposing another way to implement rate
> control.

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] TCP Rate Control

[Varun Varma]

Stef Coene wrote:
> 
> You mean something like the gred or red qdisc.

The difference is RED drops the packets after they arrive basing it on 
the understanding that the TCP implementations would take this as a hint 
and slow down the transmission rates.

Rate control on the other hand takes a more "hands on" approach to the 
problem. As Jose mentioned, ack pacing, window sizing etc...it actually 
exploits fundamentals in TCP flow control itself.

> But I wonder if it will be a big different between "TCP rate contorl" and 
> plain old shaping.

There are lot of comparisions on this. I don't want to get into a debate 
  here, since people from each group [rate control vs. queueing] feel 
very stongly about their stand.

Simply put, there are dozens of queueing disciplines, because some might 
"behave better" than other in some cases. I *feel* that rate control 
would work better in some particular cases, so I was interested in 
knowing if a rate control like implementation is available under Linux.

> 
>>Seems to be a patented "idea" in the USA, but I remember someone on this
>>list talked not long ago about he waaas implementing something like this
>>for Linux, from outside the USA. Check the archives for the message:
>>Subject: Re: [LARTC] How far can TC go?
>>From: Patrick McHardy <kaber@trash.net>
>>Date: Sat, 29 Mar 2003 11:08:30 +0100

Thanks for the reference...still need to check it out.

About the patent thing...I know for sure that ack pacing has been 
published in paper before. Note: I am not suggesting how to circumvent 
this patent in any way. That said, I think it should be possible to work 
out other mechanisms, based around TCP flow control, which don't work 
the way Packeteer's patent works.

If you google for "linux rate control", you come up with a link like:

http://www.postel.org/pipermail/end2end-interest/2001-April/000701.html

Which is a master's thesis, proposing another way to implement rate control.

Regards,
-Varun

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] TCP Rate Control

[Varun Varma]

Stef,

>>There are lot of comparisons on this. I don't want to get into a debate
>>  here, since people from each group [rate control vs. queuing] feel
>>very strongly about their stand.
> 
> But I'm still intested in the comparision :)

It's you quest for the truth that endears you to all of us. :)

Googling for "tcp rate control" would lead you to a lot of material, 
including documents "proving" queuing is better than rate control.

Here's the test jig I plan to setup...

Windows client running Gozilla, for crude throughput reporting
|
|(1)
|
->"Gateway" m/c with two NICs [RTL8139 based] running nisnet and tbf, 
iptraf and MRTG--
                 |
                 |(2)
                 |
                 ->Linux based FTP/HTTP Server

I will try with sustained downloads, perhaps 1GB. The tbf would be set 
to limit at 100kbps.

 From what I expect, I would only get 100kbps on the windows client m/c, 
i.e. at point (1). But between the gateway and server, i.e. point (2), 
the bandwidth consumption would be higher. How much higher I don't know 
- it might be anything from 100kbps to NIC speed. If it is within a 5% 
tolerance, i.e. from 95kbps to 100kbps, I would say that the world is a 
great place and I am happy to be here...i.e. queuing would have achieved 
what I want.

But, I suspect the difference would be much more...

Would most probably do this on this weekend and publish my results. I 
don't have too much experience with tc, so I might get the values wrong 
for the tbf.

Has anyone done this kind of test? Where can I get some published 
results? All the tests I have seen are done from the client 
perspective...or is it just a case of RTFM?

I am going to go ahead with the test in any case.

This only tests one side of the story. I have tested two commercial 
solutions and can testify that they give very fine-grained control on 
the bandwidth, albeit at the cost of huge latency fluctuations. Don't 
have access to these units right now, so won't be able to include the 
results from there.

I as see it, if the above test goes as I think it would, then that 
itself is a strong case for looking into other mechanisms.

> 
>>Simply put, there are dozens of queuing disciplines, because some might
>>"behave better" than other in some cases. I *feel* that rate control
>>would work better in some particular cases, so I was interested in
>>knowing if a rate control like implementation is available under Linux.
> 
> Not in the default linux kernel.

I have been tracking the tc development for quite some time, and have 
been hoping for someone to develop this.

> 
>>>>Seems to be a patented "idea" in the USA, but I remember someone on this
>>>>list talked not long ago about he waaas implementing something like this
>>>>for Linux, from outside the USA. Check the archives for the message:
>>>>Subject: Re: [LARTC] How far can TC go?
>>>>From: Patrick McHardy <kaber@trash.net>
>>>>Date: Sat, 29 Mar 2003 11:08:30 +0100
>>
>>Thanks for the reference...still need to check it out.
> 
> I already did :
> 
> Packeteer has various patents covering tcp rate control and everything else
> they do, including the "idea" to look at upper layers to detect the type 
> of traffic.
> I live in germany so i don't really care that much about their patents 
> (they had none in europe last time i checked). last summer i started 
> implementing tcp rate control as qdisc for linux. i haven't worked on it for 
> a couple of month now,  but if anyone wishes to participate i would be glad 
> to dig out my source again. it is basically working, the remaining problems 
> are mostly how to detect and handle interactive traffic.
> 
> Patrick

Like I said, it's you quest for the truth that endears you to us!


Regards,
-Varun


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] TCP Rate Control

[Stef Coene]

> > But I'm still intested in the comparision :)
>
> It's you quest for the truth that endears you to all of us. :)
>
> Googling for "tcp rate control" would lead you to a lot of material,
> including documents "proving" queuing is better than rate control.
>
> Here's the test jig I plan to setup...
>
> Windows client running Gozilla, for crude throughput reporting
>
> |(1)
>
> ->"Gateway" m/c with two NICs [RTL8139 based] running nisnet and tbf,
> iptraf and MRTG--
>
>                  |(2)
>
>                  ->Linux based FTP/HTTP Server
>
> I will try with sustained downloads, perhaps 1GB. The tbf would be set
> to limit at 100kbps.
>
>  From what I expect, I would only get 100kbps on the windows client m/c,
> i.e. at point (1). But between the gateway and server, i.e. point (2),
> the bandwidth consumption would be higher. How much higher I don't know
> - it might be anything from 100kbps to NIC speed. If it is within a 5%
> tolerance, i.e. from 95kbps to 100kbps, I would say that the world is a
> great place and I am happy to be here...i.e. queuing would have achieved
> what I want.
>
> But, I suspect the difference would be much more...
Why?  What will happen with the packets that are arrived too much?  Do you 
really think they all will be lost?  
I'm no tcp specialist, but the traffic at 2 will be exactly 100kbps.  I see it 
like this : client starts download and goes faster and faster.  When it 
reaches 100kbps, all tokens in the bucket are used.  After that, the gateway 
starts delaying packets.  From the tbf manpage :

If no tokens are available, packets are queued, up to a configured limit

Packets are not dropped but delayed.  At that time, the server noticed that 
the packets are not arriving at a speed > 100kbps (they are queued in the 
bucket) at the client and throttle downs.  When the server throttles down 
below 100kbps, the queued packets can be send again because there are tokens 
available in the bucket.  So after some time it will be stable at 100kbps.

Again, I don't know how tbf works.  This is just the way I think it works.

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] TCP Rate Control

[Stef Coene]

On Thursday 15 May 2003 21:17, Michael T. Babcock wrote:
> > Packets are not dropped but delayed.  At that time, the server noticed
>
> that
>
> > the packets are not arriving at a speed > 100kbps (they are queued in the
> > bucket) at the client and throttle downs.  When the server throttles down
> > below 100kbps, the queued packets can be send again because there are
>
> tokens
>
> > available in the bucket.  So after some time it will be stable at
> > 100kbps.
> >
> > Again, I don't know how tbf works.  This is just the way I think it
> > works.
>
> Its worth reading up on how RED works while you're at it.  There are many
> good
> whitepapers on RED/GRED w.r.t. how TCP handles exponential backoff.  As I
> understand it, TBF may not work all the time because the TCP server
> (sending the packets) may respond incorrectly to the delayed ACKs on the
> part of the client.  Dropping packets in a RED fashion as the queue builds
> up instead of queuing the packets should get perfectly stable 100kbit/s.
I think we can use some test results to prove this.  Varun ??

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] TCP Rate Control

[Varun Varma]

Stef, Michael,

>>Its worth reading up on how RED works while you're at it.  There are many
>>good
>>whitepapers on RED/GRED w.r.t. how TCP handles exponential backoff.  As I
>>understand it, TBF may not work all the time because the TCP server
>>(sending the packets) may respond incorrectly to the delayed ACKs on the
>>part of the client.  Dropping packets in a RED fashion as the queue builds
>>up instead of queuing the packets should get perfectly stable 100kbit/s.
> 
> I think we can use some test results to prove this.  Varun ??

This mail caught me in the middle of writing a long theoritical 
explanation of why I think rate control might be better, which included 
my technical, philosophical, financial and ..er.. sexual reasons.

But then I looked at this mail and realised that the best way to take 
this forward is tests. I would be game to testing both TBF and RED in 
the jig I have mentioned in my previous mail. Hopefully, should be able 
produce comprehensive documentation, including detailed explanation of 
the test setup, and the results over this weekend.

Regards,
-Varun


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] TCP Rate Control

[Stef Coene]

On Thursday 15 May 2003 23:23, Michael T. Babcock wrote:
> > But then I looked at this mail and realised that the best way to take
> > this forward is tests. I would be game to testing both TBF and RED in
> > the jig I have mentioned in my previous mail. Hopefully, should be able
> > produce comprehensive documentation, including detailed explanation of
> > the test setup, and the results over this weekend.
>
> I would be specifically interested in how an HTB + RED setup compares with
> the same (effective) configuration performs with just HTB.
>
> 1) HTB class on 100mbit line allocating 1mbit queue.
> 2) HTB class on 100mbit line allocating 1mbit RED queue.
> 3) HTB class on 100mbit line allocating 100 1mbit queues.
> 4) HTB class on 100mbit line allocating 100 1mbit RED queues.
>
> At near-saturation points, RED *should* give better performance (closer to
> complete saturation of the link).
And also how it reaches the saturation.  How many packets are dropped?  Does 
it go straight to a flat line or is "jumping" around the line before it 
stabelizes?  And what happens if you change the rate?  How fast does it 
react?  How many packets are dropped?
(I'm planning to create such a setup my self, but I have no spare NIC)

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/