Re: BUG: Slowdown on 3000 socket-machines tracked down

[Christian Schmid <webmaster@rapidforum.com>]

Ben Greear wrote:
> Christian Schmid wrote:
> 
>> Hello.
>>
>> After weeks of work, I can now give a detailed report about the bug 
>> and when it appears:
>>
>> Attached is another traffic-image. This one is with 2.6.10 and a 3/1 
>> split, preemtive kernel, so all defaults.
> 
> 
> What are the units on your graph.  You say "MB" several places, but
> do you mean Mb (ie, Mega-bit) instead?

The unit on this graph is kilobytes. So 80000 there means 80 megabytes per second.

> I have a tool that can also generate TCP traffic on a large number of
> sockets.  If I can understand what you are trying to do, I may be able
> to reproduce the problem.  My biggest machine at present has only
> 2GB of RAM, however...not sure if that matters or not.

It should not matter. Low-memory is both just 1 GB if you have default 32 bit with 3/1 split.

> Are you sending traffic in only one direction, or more of a full-duplex
> configuration?

Its a full-duplex. Its a download-service with 3000 downloaders all over the world.

>  Is each socket running the same bandwidth?

No. It ranges from 3 kb/sec to 100 kb/sec. 100 kb/sec is the limit because of the send-buffer limits.

> What is this bandwidth?

1000 MBit

> Are you setting the send & rcv buffers in the socket creation
> code?  (To what values if so?)

Yes. send-buffer to 64 kbytes and receive buffer to 16 kbytes.

> How many bytes are you sending with each call to write()/sendto() whatever?

I am using sendfile-call every 100 ms per socket with the poll-api. So basically around 40 kb per round.

> Is there any significant latency between your sender and receiver machine?
> If so, how much?

3000 different downloaders, 3000 different locations, 3000 different machines ;)

> What is the physical transport...GigE?  1500 MTU?

Yes.

Chris