Re: many outgoing tcp sockets are slower than a few

[Christian Schmid <webmaster@rapidforum.com>]

bert hubert wrote:
> On Mon, Feb 21, 2005 at 06:17:16PM +0100, Christian Schmid wrote:
> 
> 
>>Because thats the first I excluded. I changed the program by replacing 
>>sendfile with a caching-routine plus syswrite. And it was really 
>>interesting that the syswrite was the one which needs most of the 
>>real-time, not the caching-routine. syswrite blocked 100 ms per socket.
> 
> 
> Do you poll/select for socket readiness before doing write() with EAGAIN?
> (shouldn't matter all that much, but could narrow down debugging)

Yes. Using the IO::Poll in Perl.

> 100ms is a bit of an odd number - how big is your write (in bytes)?

Its an estimation. Its a non-blocking write with sendfile() so it writes around 90 KB if the buffer 
is set to 128 KB. It writes not the full because POLLOUT comes active before all is empty on the queue.

> Do you have 3500 threads or processes, or one big one?

Tried several types. 2 processes with 2000 connections each and 6 processes with 500 connections 
each. Both versions slow down if the system-wide socket-count goes over 3500.

> What does 'id' say in vmstat 1 typically, both below and above 3500 sockets?

(s02) [19:51:33] root:~# vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
  r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
  2  2      0   8196  27192 7994020    0    0  2383   275 1540   650 16 19 41 24
  0  3      0  12888  27172 7989144    0    0 36872     0 6542  2501 14 20 37 28
  1  2      0   8004  27248 7993828    0    0 34988   448 6876  2270 13 19 38 31
  0  2      0  17928  27280 7984208    0    0 30660     0 6571  2999 15 20 36 30
  3  2      0   9480  27316 7991108    0    0 37168     0 6349  2476 16 22 33 29
  2  1      0   9244  27360 7992492    0    0 37616     0 6397  2934 16 23 33 28
  0  2      0   8988  27332 7991840    0    0 36628     0 6187  3010 17 21 34 28
  3  1      0   9400  27484 7989648    0    0 35556  5708 6072  2696 18 28 27 27