Re: BUG: Slowdown on 3000 socket-machines tracked down

[Christian Schmid <webmaster@rapidforum.com>]

> Initial test setup:  two machines, running connections between them.
> Mostly asymetric (about 50Mbps in one direction,
> GigE in the other).  Each connection is trying some random rate between 
> 128kbps
> and 3Mbps in one direction, and 1kbps in the other direction.
> 
> Sending machine is dual 3.0Ghz xeons, 1MB cache, HT, and emt64 (running 
> 32-bit
> kernel & user space though). 1GB of RAM
> 
> Receiving machine is dual 2.8Ghz xeons, 512 MB cache, HT, 32-bit.  2GB 
> of RAM
> (but only 850Mbps of low memory of course...saw the thing OOM kill me 
> with 1GB of
> free high memory :( )
> 
> 
> Zero latency:
> 
> 2000 TCP connections:  When I first start, I see errors indicating I'm 
> out of low
>         memory..but it quickly recovers.  Probably because my program 
> takes a small
>         bit of time before it starts reading the sockets.
>         986Mbps of ethernet traffic (counting all ethernet headers)
> 
> 3000 TCP connections:  Same memory issue
>         986Mbps of ethernet traffic, about 82kpps
> 
> 4000 TCP connections:  Had to drop max_backlog to 5000 from 10000 to keep
>         the machine from going OOM and killing my traffic generator (on
>         the receiving side).
>     986Mbps of ethernet traffic
> 
> I will work on some numbers with latency tomorrow (had to stop and
> re-write some of my code to better handle managing the 8000 endpoints
> that 4000 connections requires!)
> 
> I think we can assume that the problem is either related to latency,
> or sendfile, since 4000 connections with no latency rocks along just
> fine...

Hmmmm.... can you try to following just to exclude some theories:

Run it with 4000 sockets and then do the following on the server-machine:

dd if=/dev/zero of=file1 bs=1M count=1024
dd if=/dev/zero of=file2 bs=1M count=1024
dd if=/dev/zero of=file3 bs=1M count=1024
cat file1 > /dev/zero & cat file2 > /dev/zero & cat file3 > /dev/zero &

I THINK it might have something to do with caching-pressure or so. See if there is a slow-down on 
the sending if the page-cache gets full and has to be cleared again.

You are running 2.6.11?

Chris