Re: BUG: Slowdown on 3000 socket-machines tracked down

Christian Schmid
2005-03-08T16:41:37Z

> Initial test setup: two machines, running connections between them.
> Mostly asymetric (about 50Mbps in one direction,
> GigE in the other). Each connection is trying some random rate between
> 128kbps
> and 3Mbps in one direction, and 1kbps in the other direction.
>
> Sending machine is dual 3.0Ghz xeons, 1MB cache, HT, and emt64 (running
> 32-bit
> kernel & user space though). 1GB of RAM
>
> Receiving machine is dual 2.8Ghz xeons, 512 MB cache, HT, 32-bit. 2GB
> of RAM
> (but only 850Mbps of low memory of course...saw the thing OOM kill me
> with 1GB of
> free high memory :( )
>
>
> Zero latency:
>
> 2000 TCP connections: When I first start, I see errors indicating I'm
> out of low
> memory..but it quickly recovers. Probably because my program
> takes a small
> bit of time before it starts reading the sockets.
> 986Mbps of ethernet traffic (counting all ethernet headers)
>
> 3000 TCP connections: Same memory issue
> 986Mbps of ethernet traffic, about 82kpps
>
> 4000 TCP connections: Had to drop max_backlog to 5000 from 10000 to keep
> the machine from going OOM and killing my traffic generator (on
> the receiving side).
> 986Mbps of ethernet traffic
>
> I will work on some numbers with latency tomorrow (had to stop and
> re-write some of my code to better handle managing the 8000 endpoints
> that 4000 connections requires!)
>
> I think we can assume that the problem is either related to latency,
> or sendfile, since 4000 connections with no latency rocks along just
> fine...

Hmmmm.... can you try to following just to exclude some theories:

Run it with 4000 sockets and then do the following on the server-machine:

dd if=/dev/zero of=file1 bs=1M count=1024
dd if=/dev/zero of=file2 bs=1M count=1024
dd if=/dev/zero of=file3 bs=1M count=1024
cat file1 > /dev/zero & cat file2 > /dev/zero & cat file3 > /dev/zero &

I THINK it might have something to do with caching-pressure or so. See if there is a slow-down on
the sending if the page-cache gets full and has to be cleared again.

You are running 2.6.11?

Chris