Re: BUG: Slowdown on 3000 socket-machines tracked down

[Ben Greear <greearb@candelatech.com>]

Christian Schmid wrote:
> Ben Greear wrote:
> 

>> I have a tool that can also generate TCP traffic on a large number of
>> sockets.  If I can understand what you are trying to do, I may be able
>> to reproduce the problem.  My biggest machine at present has only
>> 2GB of RAM, however...not sure if that matters or not.
> 
> It should not matter. Low-memory is both just 1 GB if you have default 
> 32 bit with 3/1 split.
> 
>> Are you sending traffic in only one direction, or more of a full-duplex
>> configuration?
> 
> Its a full-duplex. Its a download-service with 3000 downloaders all over 
> the world.

So actually it's really mostly one-way traffic, ie in the download direction.
Anything significant at all going upstream, other than ACKs, etc?

>>  Is each socket running the same bandwidth?
> 
> No. It ranges from 3 kb/sec to 100 kb/sec. 100 kb/sec is the limit 
> because of the send-buffer limits.
> 
>> What is this bandwidth?
> 
> 1000 MBit
> 
>> Are you setting the send & rcv buffers in the socket creation
>> code?  (To what values if so?)
> 
> Yes. send-buffer to 64 kbytes and receive buffer to 16 kbytes.

With regard to this note in the 'man 7 socket' man page:

NOTES
        Linux assumes that half of the send/receive buffer is used for internal kernel struc-
        tures; thus the sysctls are twice what can be observed on the wire.

What value are you using for the sockopt call?

>> How many bytes are you sending with each call to write()/sendto() 
>> whatever?
>  
> I am using sendfile-call every 100 ms per socket with the poll-api. So 
> basically around 40 kb per round.

My application is single-threaded, uses non-blocking IO, and sends/rcvs from/to memory.
It will be a good test of the TCP stack, but will not use the sendfile logic,
nor will it touch the HD.

>> Is there any significant latency between your sender and receiver 
>> machine?
>> If so, how much?
> 
> 3000 different downloaders, 3000 different locations, 3000 different 
> machines ;)

I can emulate delay if I need to, but I'd rather just stick with one
delay setting and not have to set up a separate delay for each connection.

Maybe 30ms is average for round-trip time?

Have you tried benchmarking your app in a controlled manner, or are you just
letting a random 3000 machines hit it and start downloading?  If the latter,
then I'd suggest getting more controll over your testing environment, otherwise
it may be impossible to really figure out where the problem lies.

I'll set up a configuration similar to the values discussed above and see
what I can see.  Will probably be late tomorrow before I can do the
test though...

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com