problems achieving decent throughput with latency.

problems achieving decent throughput with latency.

[Ben Greear]

[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]

I am testing my latency-insertion tool, and I notice that tcp will not use
all of the available bandwidth if there is any significant amount of latency
on the wire.

For example, with 25ms latency in both directions, I see about 8Mbps
bi-directional throughput.

If I lower that to 15ms, I see 12Mbps bi-directional throughput.

I see 27Mbps at 5ms.

Here is the /proc/net/tcp output at 5ms latency.

machine demo2
   13: 050302AC:80EB 070302AC:80EB 01 0005900C:0002012E 01:00000016 00000000     0        0 578943 3 c6628a80 22 4 1 45 -1

machine demo1
   11: 070302AC:80EB 050302AC:80EB 01 00010DDB:00000000 01:00000014 00000000     0        0 513094 3 c62c5080 21 4 1 45 -1


Any ideas why it is so slow at the higher latencies?  Any other info
I can gather to help determine the cause?

(UDP does not experience this slowdown, so I believe my latency
insertion tool is working as designed, but it's always possible it is
to blame...)


-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear


[-- Attachment #2: problems achieving decent throughput with latency. --]
[-- Type: message/rfc822, Size: 1707 bytes --]

From: Ben Greear <greearb@candelatech.com>
To: "'netdev@oss.sgi.com'" <netdev@oss.sgi.com>
Subject: problems achieving decent throughput with latency.
Date: Sat, 01 Feb 2003 14:13:01 -0800
Message-ID: <3E3C466D.7030602@candelatech.com>

<div class="moz-text-flowed" style="font-family: -moz-fixed">I am testing my latency-insertion tool, and I notice that tcp will not use
all of the available bandwidth if there is any significant amount of latency
on the wire.

For example, with 25ms latency in both directions, I see about 8Mbps
bi-directional throughput.

If I lower that to 15ms, I see 12Mbps bi-directional throughput.

I see 27Mbps at 5ms.

Here is the /proc/net/tcp output at 5ms latency.

machine demo2
   13: 050302AC:80EB 070302AC:80EB 01 0005900C:0002012E 01:00000016 00000000     0        0 578943 3 c6628a80 22 4 1 45 -1

machine demo1
   11: 070302AC:80EB 050302AC:80EB 01 00010DDB:00000000 01:00000014 00000000     0        0 513094 3 c62c5080 21 4 1 45 -1


Any ideas why it is so slow at the higher latencies?  Any other info
I can gather to help determine the cause?

(UDP does not experience this slowdown, so I believe my latency
insertion tool is working as designed, but it's always possible it is
to blame...)


-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear


</div>

Re: problems achieving decent throughput with latency.

[bert hubert]

On Sat, Feb 01, 2003 at 11:38:02PM -0800, Ben Greear wrote:
> I am testing my latency-insertion tool, and I notice that tcp will not use
> all of the available bandwidth if there is any significant amount of latency
> on the wire.
> 
> For example, with 25ms latency in both directions, I see about 8Mbps
> bi-directional throughput.

Check if large windows are being used, if window scaling is enabled.

Regards,

bert


-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO
http://netherlabs.nl                         Consulting

Re: problems achieving decent throughput with latency.

[David S. Miller]

TCP can only send into a pipe as fast as it can see the
ACKs coming back.  That is how TCP clocks its sending rate,
and latency thus affects that.

Re: problems achieving decent throughput with latency.

[Chris Friesen]

David S. Miller wrote:
> TCP can only send into a pipe as fast as it can see the
> ACKs coming back.  That is how TCP clocks its sending rate,
> and latency thus affects that.

Wouldn't you just need larger windows?  The problem is latency, not 
bandwidth.

Chris



-- 
Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: problems achieving decent throughput with latency.

[John Bradford]

> > TCP can only send into a pipe as fast as it can see the
> > ACKs coming back.  That is how TCP clocks its sending rate,
> > and latency thus affects that.
> 
> Wouldn't you just need larger windows?  The problem is latency, not 
> bandwidth.

Exactly - the original post says that no problems are experienced
using UDP, which backs that up.

John.

Re: problems achieving decent throughput with latency.

[bert hubert]

On Mon, Feb 03, 2003 at 04:11:46PM +0000, John Bradford wrote:
> > > TCP can only send into a pipe as fast as it can see the
> > > ACKs coming back.  That is how TCP clocks its sending rate,
> > > and latency thus affects that.
> > 
> > Wouldn't you just need larger windows?  The problem is latency, not 
> > bandwidth.
> 
> Exactly - the original post says that no problems are experienced
> using UDP, which backs that up.

My TCP may be getting hazy but with latency, TCP may limit the amount of
packets 'in flight'. I think this is called the 'cwnd', congestion window. I
also seem to recall that this window is limited to the actually negotiated
TCP window size.

But here I leave this discussion as I've swapped out the finer details of
congestion windows to the Stevens books which other's have probably read
more recently than I did.

Regards,

bert

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO
http://netherlabs.nl                         Consulting

Re: problems achieving decent throughput with latency.

[Ben Greear]

John Bradford wrote:
>>>TCP can only send into a pipe as fast as it can see the
>>>ACKs coming back.  That is how TCP clocks its sending rate,
>>>and latency thus affects that.
>>
>>Wouldn't you just need larger windows?  The problem is latency, not 
>>bandwidth.
> 
> 
> Exactly - the original post says that no problems are experienced
> using UDP, which backs that up.

I started poking around, and found the tcp_mem, tcp_rmem, and tcp_wmem
tunables in /proc/sys/net/ipv4...

If I change the values, I see up to 25Mbps with 25ms of latency.
It would go higher, but I have uncovered a performance bug in my code that
drops a packet every now and then at those higher rates, so that backs tcp
off quickly.  I should have that fixed this evening and will continue testing.


Here are the values that I used.  The documentation I found is not overly
descriptive, so if anyone has any suggestions for improving my tunings, please
let me know!

Also, if it's as simple as allocating a few more buffers for tcp, maybe we
should consider defaulting to higher in the normal kernel?  (I'm not suggesting
**my** numbers..)

# See the kernel documentation: Documentation/networking/ip-sysctl.txt
my $tcp_rmem_min     = 4096;
my $tcp_rmem_default = 256000;  # TCP specific receive memory pool size.
my $tcp_rmem_max     = 3000000;  # TCP specific receive memory pool size.

my $tcp_wmem_min     = 4096;
my $tcp_wmem_default = 256000;  # TCP specific receive memory pool size.
my $tcp_wmem_max     = 3000000;  # TCP specific receive memory pool size.

my $tcp_mem_lo       = 20000000; # Below here there is no memory pressure.
my $tcp_mem_pressure = 30000000; # Can use up to 30MB for TCP buffers.
my $tcp_mem_high     = 30000000; # Can use up to 30MB for TCP buffers.




> 
> John.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: problems achieving decent throughput with latency.

[Eric Weigle]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

You might want to have a look at
	http://www.psc.edu/networking/perf_tune.html
It's a pretty good reference on TCP tuning.

Also, one gotcha: in earlier versions of 2.4 the window scaling factor was
set using the DEFAULT window size instead of the MAXIMUM window size; that
is, window scaling is set such that neither autotuning nor application level
tuning can actually tune to the maximum size (it can't be represented with
the default window scale factor). I don't think this has been fixed yet,
so you may have to set your default to be the same as your maximum for tests.


-Eric

-- 
------------------------------------------------------------
        Eric H. Weigle -- http://public.lanl.gov/ehw/
"They that can give up essential liberty to obtain a little
temporary safety deserve neither" -- Benjamin Franklin
------------------------------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: problems achieving decent throughput with latency.

[David S. Miller]

   From: Ben Greear <greearb@candelatech.com>
   Date: Mon, 03 Feb 2003 10:03:48 -0800
   
   Also, if it's as simple as allocating a few more buffers for tcp, maybe we
   should consider defaulting to higher in the normal kernel?  (I'm not suggesting
   **my** numbers..)

The current values are the only "safe" defaults.  Here "safe" means
that if you have thousands of web connections, clients cannot force
the serve to queue large amounts of traffic per socket.

The attack goes something like: Open N thousand connections to
server, ask for large static object, do not ACK any of the data
packets.  Server must thus hold onto N thousnad * maximum socket
write buffer bytes amount of memory.

Re: problems achieving decent throughput with latency.

[Ben Greear]

David S. Miller wrote:
>    From: Ben Greear <greearb@candelatech.com>
>    Date: Mon, 03 Feb 2003 10:03:48 -0800
>    
>    Also, if it's as simple as allocating a few more buffers for tcp, maybe we
>    should consider defaulting to higher in the normal kernel?  (I'm not suggesting
>    **my** numbers..)
> 
> The current values are the only "safe" defaults.  Here "safe" means
> that if you have thousands of web connections, clients cannot force
> the serve to queue large amounts of traffic per socket.
> 
> The attack goes something like: Open N thousand connections to
> server, ask for large static object, do not ACK any of the data
> packets.  Server must thus hold onto N thousnad * maximum socket
> write buffer bytes amount of memory.

Why would it use the maximum socket for a connection with low to
no acks, ie low to no throughput?  Seems like the connection would
have to scale up to full speed/sliding-window, which would require the DoS guy
to have large receive bandwidth, and also enough precision to stop acking
as soon as the window gets big (but before the object download has
completed.)  This does not seem like a great DoS to me.

On my system, the default memory seems to be about 80k (docs say it
is based on how much memory I have (128MB)).  How big can N get?
If N is 10k I can be DOS'd for only 800k?

-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: problems achieving decent throughput with latency.

[David S. Miller]

   From: Ben Greear <greearb@candelatech.com>
   Date: Mon, 03 Feb 2003 23:50:05 -0800
   
   Why would it use the maximum socket for a connection with low to
   no acks, ie low to no throughput?

You open up the congestion window by ACK'ing a few windows
worth of data, then you stop ACK'ing.

I'm sorry if that wasn't obvious.

Re: problems achieving decent throughput with latency.

[Ben Greear]

David S. Miller wrote:
>    From: Ben Greear <greearb@candelatech.com>
>    Date: Mon, 03 Feb 2003 23:50:05 -0800
>    
>    Why would it use the maximum socket for a connection with low to
>    no acks, ie low to no throughput?
> 
> You open up the congestion window by ACK'ing a few windows
> worth of data, then you stop ACK'ing.

I think I understand, but on my system it seem to take 5-8 seconds for
the bandwidth to get up to ~20Mbps (with my larger buffer settings mentioned
earlier).  This is with 25ms latency. With the default settings I can run about
8Mbps, so it would appear to me that only 3x the current default buffer settings
should get a window size enough to go ~20Mbps at 25ms latency.

Am I correct that if I have 10k clients doing their worst tricks, and
3 * (80k, my default according to the kernel) == 240k, then I have at most
2.4MB denial of service?  Assuming 60k clients, that is only about 15MB
of DoS?  If true, that is a fairly small time DoS considering the RAM available
on today's machines.

You claim for a very large N that the denial of service can happen.  I
am just trying to understand the upper bound of N, and thus the upperbound
of the memory consumption assuming each connection is using it's maximum
buffer size.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: problems achieving decent throughput with latency.

[David S. Miller]

   From: Ben Greear <greearb@candelatech.com>
   Date: Tue, 04 Feb 2003 00:42:02 -0800
   
   Am I correct that if I have 10k clients doing their worst tricks, and
   3 * (80k, my default according to the kernel) == 240k, then I have at most
   2.4MB denial of service?  Assuming 60k clients, that is only about 15MB
   of DoS?  If true, that is a fairly small time DoS considering the RAM available
   on today's machines.

Add in the struct sk_buff for each packet as well, which is dependant
upon MSS.  Thus you could make the clients use a super-small MSS to
get more per-packet struct sk_buff overhead.  The list goes on and on.
At least Linux, unlike BSD, makes an attempt to account for the
sk_buff overhead in the limits :-)

Re: problems achieving decent throughput with latency.

[Ben Greear]

Ben Greear wrote:

[ a bunch of bogus math]

Sorry...I think my math is off by many zeros.  I should
stick to reading slashdot this time of night!

Ben

-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear