Annoying bug with many sockets.

Annoying bug with many sockets.

[Christian Schmid]

Hi.

This is really annoying. With 3500 sockets onwards, linux 2.6.10 completely lags. This is a bug and 
I am not willing to buy new servers just because linux has a BUG. tcp_mem _rmem and _wmem have been 
set to 1024000 for testing but this doesnt help as well. so whats WRONG there... please?

Best regards,
Chris

Re: Annoying bug with many sockets.

[Nivedita Singhvi]

Christian Schmid wrote:

> Hi.
> 
> This is really annoying. With 3500 sockets onwards, linux 2.6.10 
> completely lags. This is a bug and I am not willing to buy new servers 
> just because linux has a BUG. tcp_mem _rmem and _wmem have been set to 
> 1024000 for testing but this doesnt help as well. so whats WRONG 
> there... please?
> 
> Best regards,
> Chris

You have not actually said what the problem is - do
new connections not get made? Or existing connections
slow down?

You are trying to run many simultaneous connections, so
bumping up the individual socket buffer allocation will
not necessarily help - you need to bump up the global
TCP limit (tcp_mem[]) - it's a 3-tuple - if you have
the memory in your system, bump it way up. netstat -tan
will tell you if there is unread data in the queues..

Are you running into memory pressure? Or aborts?
netstat -s might give you some info on what is happening.

Bump up the port space (/proc/sys/net/ipv4/ip_local_port_range)
available - typical default is 32K - 61000 (can lower min to 4K)

Are they all receiving data or sending? Are they talking to
different hosts?

You can increase tcp_max_syn_backlog, core/netdev_max_backlog,
for a start.

But it would help if you looked at the stats and ifconfig
to see who's dropping packets, how many retransmissions there
are, memory failures, or the bottleneck is some other issue altogether...


thanks,
Nivedita

Re: Annoying bug with many sockets.

[Christian Schmid]

Nivedita Singhvi wrote:
> Christian Schmid wrote:
> 
>> Hi.
>>
>> This is really annoying. With 3500 sockets onwards, linux 2.6.10 
>> completely lags. This is a bug and I am not willing to buy new servers 
>> just because linux has a BUG. tcp_mem _rmem and _wmem have been set to 
>> 1024000 for testing but this doesnt help as well. so whats WRONG 
>> there... please?
>>
>> Best regards,
>> Chris
> 
> 
> You have not actually said what the problem is - do
> new connections not get made? Or existing connections
> slow down?

New connections get made without any problems. Just existing connections slow down painfully.

> You are trying to run many simultaneous connections, so
> bumping up the individual socket buffer allocation will
> not necessarily help - you need to bump up the global
> TCP limit (tcp_mem[]) - it's a 3-tuple - if you have
> the memory in your system, bump it way up. netstat -tan
> will tell you if there is unread data in the queues..

I already set it to 1024000 1025000 1026000 (just to be sure). Its a 8 GB system with 2/2 split, so 
2 GB of low memory.

> Are you running into memory pressure? Or aborts?
> netstat -s might give you some info on what is happening.
> 
> Bump up the port space (/proc/sys/net/ipv4/ip_local_port_range)
> available - typical default is 32K - 61000 (can lower min to 4K)
> 
> Are they all receiving data or sending? Are they talking to
> different hosts?
> 
> You can increase tcp_max_syn_backlog, core/netdev_max_backlog,
> for a start.

netdev_max_backlog has been raised from 300 to 3000 without any result. syn_backlog is normal but 
its no problem to create new connections. Just existing connections slow down suddenly. Like this:

3000 sockets = no slowdown at all (500 MBit in use)
3300 sockets = 10% slowdown
3600 sockets = 30% slowdown
4000 sockets = 60% slowdown (i aborted here, as it only uses 200 MBit for sending... catastrophy!)

They are all receiving data. Its a download-service. receive-buffer is set to 24 KB and send-buffer 
set to 224 KB. I don't see a problem with port-space. I only have 3500 sockets when the problem 
appears but it appears suddenly.

> But it would help if you looked at the stats and ifconfig
> to see who's dropping packets, how many retransmissions there
> are, memory failures, or the bottleneck is some other issue altogether...

No way. Doing 30000 packets per second and your stats are 32 bit integers ;)

Chris

many outgoing tcp sockets are slower than a few

[bert hubert]

On Mon, Feb 21, 2005 at 01:35:33AM +0100, Christian Schmid wrote:

> New connections get made without any problems. Just existing connections 
> slow down painfully.

Incoming our outgoing data mostly?

> 3000 sockets = no slowdown at all (500 MBit in use)
> 3300 sockets = 10% slowdown
> 3600 sockets = 30% slowdown
> 4000 sockets = 60% slowdown (i aborted here, as it only uses 200 MBit for 
> sending... catastrophy!)
> 
> They are all receiving data. Its a download-service. receive-buffer is set 
> to 24 KB and send-buffer set to 224 KB. I don't see a problem with 
> port-space. I only have 3500 sockets when the problem appears but it 
> appears suddenly.

I'm a bit confused, it is a download service so you are probably *sending*
data? 

> >But it would help if you looked at the stats and ifconfig
> >to see who's dropping packets, how many retransmissions there
> >are, memory failures, or the bottleneck is some other issue altogether...
> 
> No way. Doing 30000 packets per second and your stats are 32 bit integers ;)

So? You could be a bit more helpful. Sample over 5 seconds, you won't
overflow that.

Also, can you tcpdump a bit? Are you using iptables? The conntrack table
might be slowing you down. 

I have a hunch this problem has to do with high-mem issues though.

Good luck!

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services

Re: many outgoing tcp sockets are slower than a few

[Christian Schmid]

bert hubert wrote:
> On Mon, Feb 21, 2005 at 01:35:33AM +0100, Christian Schmid wrote:
> 
> 
>>New connections get made without any problems. Just existing connections 
>>slow down painfully.
> 
> 
> Incoming our outgoing data mostly?

Outgoing data. I am using sendfile() to send the file on a non-blocking socket but the call blocks 
for 100 ms per socket if I get over 3000 sockets. Thats causing the massive slowdown in sum. I first 
thought its a disk-issue but I tried with pure-cache data as well and it still blocks.

>>3000 sockets = no slowdown at all (500 MBit in use)
>>3300 sockets = 10% slowdown
>>3600 sockets = 30% slowdown
>>4000 sockets = 60% slowdown (i aborted here, as it only uses 200 MBit for 
>>sending... catastrophy!)
>>
>>They are all receiving data. Its a download-service. receive-buffer is set 
>>to 24 KB and send-buffer set to 224 KB. I don't see a problem with 
>>port-space. I only have 3500 sockets when the problem appears but it 
>>appears suddenly.
> 
> 
> I'm a bit confused, it is a download service so you are probably *sending*
> data? 

Only sending. Receiving ACKs of course.

>>>But it would help if you looked at the stats and ifconfig
>>>to see who's dropping packets, how many retransmissions there
>>>are, memory failures, or the bottleneck is some other issue altogether...
>>
>>No way. Doing 30000 packets per second and your stats are 32 bit integers ;)
> So? You could be a bit more helpful. Sample over 5 seconds, you won't
> overflow that.

5 seconds, here you go:

eth0      Link encap:Ethernet  HWaddr 00:30:48:27:84:28
           inet addr:80.237.244.12  Bcast:80.237.244.63  Mask:255.255.255.192
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:2861390424 errors:5991038 dropped:5991056 overruns:38681 frame:4294967287
           TX packets:2906616171 errors:4294967279 dropped:0 overruns:0 carrier:4294967278
           collisions:4294967289 txqueuelen:1000
           RX bytes:1613084024 (1538.3 Mb)  TX bytes:2528808392 (2411.6 Mb)
           Base address:0x4000 Memory:fc000000-fc020000

eth0      Link encap:Ethernet  HWaddr 00:30:48:27:84:28
           inet addr:80.237.244.12  Bcast:80.237.244.63  Mask:255.255.255.192
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:2861511619 errors:5991038 dropped:5991056 overruns:38681 frame:4294967287
           TX packets:2906804392 errors:4294967279 dropped:0 overruns:0 carrier:4294967278
           collisions:4294967289 txqueuelen:1000
           RX bytes:1625289307 (1549.9 Mb)  TX bytes:2802520805 (2672.6 Mb)
           Base address:0x4000 Memory:fc000000-fc020000

> Also, can you tcpdump a bit? Are you using iptables? The conntrack table
> might be slowing you down. 

Hmmm tcpdump what exactly? I am using iptables and the conntrack-problem has been solved in the past 
already by disabling conn-tracking. The table overran and it dropped packets massively. I disabled 
conn-tracking and the problem was gone. This is a different problem though.

> I have a hunch this problem has to do with high-mem issues though.

Well, 6 GB of high-mem, 2 GB of low-mem... at least there is not too less memory.

MemTotal:      8314392 kB
MemFree:         13936 kB
Buffers:         28732 kB
Cached:        7926792 kB
SwapCached:          0 kB
Active:        3129028 kB
Inactive:      5031240 kB
HighTotal:     6421952 kB
HighFree:          640 kB
LowTotal:      1892440 kB
LowFree:         13296 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:           54160 kB
Writeback:           0 kB
Mapped:         210108 kB
Slab:           128176 kB
CommitLimit:   4157196 kB
Committed_AS:   674072 kB
PageTables:       1604 kB
VmallocTotal:   114680 kB
VmallocUsed:      1160 kB
VmallocChunk:   113416 kB

Best regards,
Chris

Re: many outgoing tcp sockets are slower than a few

[Lennert Buytenhek]

On Mon, Feb 21, 2005 at 11:36:14AM +0100, Christian Schmid wrote:

> Outgoing data. I am using sendfile() to send the file on a non-blocking 
> socket but the call blocks for 100 ms per socket if I get over 3000 
> sockets. Thats causing the massive slowdown in sum. I first thought its a 
> disk-issue but I tried with pure-cache data as well and it still blocks.

O_NONBLOCK send() is really nonblocking, but O_NONBLOCK sendfile()
really isn't, as it still does the disk read (if any) synchronously.

How are you making sure that you're sending "pure-cache data"?


--L

Re: many outgoing tcp sockets are slower than a few

[bert hubert]

On Mon, Feb 21, 2005 at 01:02:23PM +0100, Lennert Buytenhek wrote:
> O_NONBLOCK send() is really nonblocking, but O_NONBLOCK sendfile()

It is? So it returns EAGAIN if the data to be sent is not in the page cache?

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services

Re: many outgoing tcp sockets are slower than a few

[Lennert Buytenhek]

On Mon, Feb 21, 2005 at 01:25:41PM +0100, bert hubert wrote:

> > O_NONBLOCK send() is really nonblocking, but O_NONBLOCK sendfile()
> 
> It is? So it returns EAGAIN if the data to be sent is not in the
> page cache?

No, it just blocks.  It only returns EAGAIN if the data is in the
page cache but the socket TX queue doesn't have enough space.  (At
least, when I last tried it.)


--L

Re: many outgoing tcp sockets are slower than a few

[Christian Schmid]

Lennert Buytenhek wrote:
> On Mon, Feb 21, 2005 at 11:36:14AM +0100, Christian Schmid wrote:
> 
> 
>>Outgoing data. I am using sendfile() to send the file on a non-blocking 
>>socket but the call blocks for 100 ms per socket if I get over 3000 
>>sockets. Thats causing the massive slowdown in sum. I first thought its a 
>>disk-issue but I tried with pure-cache data as well and it still blocks.
> 
> 
> O_NONBLOCK send() is really nonblocking, but O_NONBLOCK sendfile()
> really isn't, as it still does the disk read (if any) synchronously.
> 
> How are you making sure that you're sending "pure-cache data"?

Because thats the first I excluded. I changed the program by replacing sendfile with a 
caching-routine plus syswrite. And it was really interesting that the syswrite was the one which 
needs most of the real-time, not the caching-routine. syswrite blocked 100 ms per socket.

> 
> 
> --L
> 
> 

Re: many outgoing tcp sockets are slower than a few

[bert hubert]

On Mon, Feb 21, 2005 at 06:17:16PM +0100, Christian Schmid wrote:

> Because thats the first I excluded. I changed the program by replacing 
> sendfile with a caching-routine plus syswrite. And it was really 
> interesting that the syswrite was the one which needs most of the 
> real-time, not the caching-routine. syswrite blocked 100 ms per socket.

Do you poll/select for socket readiness before doing write() with EAGAIN?
(shouldn't matter all that much, but could narrow down debugging)

100ms is a bit of an odd number - how big is your write (in bytes)?

Do you have 3500 threads or processes, or one big one?

What does 'id' say in vmstat 1 typically, both below and above 3500 sockets?

Thanks.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services

Re: many outgoing tcp sockets are slower than a few

[Christian Schmid]

bert hubert wrote:
> On Mon, Feb 21, 2005 at 06:17:16PM +0100, Christian Schmid wrote:
> 
> 
>>Because thats the first I excluded. I changed the program by replacing 
>>sendfile with a caching-routine plus syswrite. And it was really 
>>interesting that the syswrite was the one which needs most of the 
>>real-time, not the caching-routine. syswrite blocked 100 ms per socket.
> 
> 
> Do you poll/select for socket readiness before doing write() with EAGAIN?
> (shouldn't matter all that much, but could narrow down debugging)

Yes. Using the IO::Poll in Perl.

> 100ms is a bit of an odd number - how big is your write (in bytes)?

Its an estimation. Its a non-blocking write with sendfile() so it writes around 90 KB if the buffer 
is set to 128 KB. It writes not the full because POLLOUT comes active before all is empty on the queue.

> Do you have 3500 threads or processes, or one big one?

Tried several types. 2 processes with 2000 connections each and 6 processes with 500 connections 
each. Both versions slow down if the system-wide socket-count goes over 3500.

> What does 'id' say in vmstat 1 typically, both below and above 3500 sockets?

(s02) [19:51:33] root:~# vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
  r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
  2  2      0   8196  27192 7994020    0    0  2383   275 1540   650 16 19 41 24
  0  3      0  12888  27172 7989144    0    0 36872     0 6542  2501 14 20 37 28
  1  2      0   8004  27248 7993828    0    0 34988   448 6876  2270 13 19 38 31
  0  2      0  17928  27280 7984208    0    0 30660     0 6571  2999 15 20 36 30
  3  2      0   9480  27316 7991108    0    0 37168     0 6349  2476 16 22 33 29
  2  1      0   9244  27360 7992492    0    0 37616     0 6397  2934 16 23 33 28
  0  2      0   8988  27332 7991840    0    0 36628     0 6187  3010 17 21 34 28
  3  1      0   9400  27484 7989648    0    0 35556  5708 6072  2696 18 28 27 27

Re: many outgoing tcp sockets are slower than a few

[Lennert Buytenhek]

On Mon, Feb 21, 2005 at 06:17:16PM +0100, Christian Schmid wrote:

> >>Outgoing data. I am using sendfile() to send the file on a non-blocking 
> >>socket but the call blocks for 100 ms per socket if I get over 3000 
> >>sockets. Thats causing the massive slowdown in sum. I first thought its a 
> >>disk-issue but I tried with pure-cache data as well and it still blocks.
> >
> >O_NONBLOCK send() is really nonblocking, but O_NONBLOCK sendfile()
> >really isn't, as it still does the disk read (if any) synchronously.
> >
> >How are you making sure that you're sending "pure-cache data"?
> 
> Because thats the first I excluded. I changed the program by replacing 
> sendfile with a caching-routine plus syswrite. And it was really 
> interesting that the syswrite was the one which needs most of the 
> real-time, not the caching-routine. syswrite blocked 100 ms per socket.

'syswrite'.

Is your application written in C or perl?


--L

Re: many outgoing tcp sockets are slower than a few

[Christian Schmid]

Lennert Buytenhek wrote:
> On Mon, Feb 21, 2005 at 06:17:16PM +0100, Christian Schmid wrote:
> 
> 
>>>>Outgoing data. I am using sendfile() to send the file on a non-blocking 
>>>>socket but the call blocks for 100 ms per socket if I get over 3000 
>>>>sockets. Thats causing the massive slowdown in sum. I first thought its a 
>>>>disk-issue but I tried with pure-cache data as well and it still blocks.
>>>
>>>O_NONBLOCK send() is really nonblocking, but O_NONBLOCK sendfile()
>>>really isn't, as it still does the disk read (if any) synchronously.
>>>
>>>How are you making sure that you're sending "pure-cache data"?
>>
>>Because thats the first I excluded. I changed the program by replacing 
>>sendfile with a caching-routine plus syswrite. And it was really 
>>interesting that the syswrite was the one which needs most of the 
>>real-time, not the caching-routine. syswrite blocked 100 ms per socket.
> 
> 
> 'syswrite'.
> 
> Is your application written in C or perl?


Perl. But this is not the source because as stated in another mail, I tried with 6 processes when it 
slows down at 500 connections each and with 2 processes where it only starts to slowdown at 2000 
processes each.

Chris

Re: many outgoing tcp sockets are slower than a few

[Baruch Even]

Christian Schmid wrote:
> bert hubert wrote:
> 
>> On Mon, Feb 21, 2005 at 01:35:33AM +0100, Christian Schmid wrote:
> Outgoing data. I am using sendfile() to send the file on a non-blocking 
> socket but the call blocks for 100 ms per socket if I get over 3000 
> sockets. Thats causing the massive slowdown in sum. I first thought its 
> a disk-issue but I tried with pure-cache data as well and it still blocks.
> 
>>> 3000 sockets = no slowdown at all (500 MBit in use)
>>> 3300 sockets = 10% slowdown
>>> 3600 sockets = 30% slowdown
>>> 4000 sockets = 60% slowdown (i aborted here, as it only uses 200 MBit 
>>> for sending... catastrophy!)
>>>
[snip]
>> I'm a bit confused, it is a download service so you are probably 
>> *sending*
>> data? 
> 
> Only sending. Receiving ACKs of course.

I've been doing some work to improve ACK performance, you can find some 
patches for Linux 2.6.6 at http://hamilton.ie/net/

I've only tested these patches for a single (or few) very fast 
connections, but I'd expect the problem might manifest itself for a very 
large number of connections as well. Though then you might hit other 
bottlenecks (memory access for different structures).

Baruch