Leak in network memory?

["William M. Shubert" <wms@igoweb.org>]

Hi. I have an application that does a lot of nonblocking networking I/O 
and is fairly sensitive to how much data can be held in the output 
buffers of sockets. All sockets are set to have 64KB (the default) of 
output buffering. This application had been running well with very long 
uptimes for over a year in the 2.2 kernels.

A couple months ago I upgraded my server to RH 7.1 (with the 2.4.2-2 red 
hat kernel). At first it ran fine, but now after an uptime of 67 days 
I'm starting to see strange problems. It seems as if only a very small 
amount of memory can be held in the output buffer of each socket, even 
though they are still set to 64KB! There isn't a tremendous amount of 
network traffic going on (about 30-100 sockets open at a time, but 
rather low total bandwidth). The fact that each write to a socket only 
writes a few (<8) kbytes is really messing with my performance. I did 
not see this problem until the past week. I tried to trace through the 
kernel code to see why the kernel would be refusing to give me the 
buffering that I ask for, and it looks like if the network code thinks 
that it is using too much memory, then it will behave this way. I'm not 
100% sure of this, though...which is why I'm posting this message.

Does anybody have any hints on how I can track down exactly why my 
output buffers aren't working? I see lots of /proc info related to 
network parameters, but there is little documentation on them. Is there 
a known bug like this in the RH 2.4.2-2 kernel? Would a newer kernel 
help me? (I know, I could just try upgrading and waiting another 60 
days, but 24x7 reliability is very important to my users so I'd rather 
not reboot unless I know that it will help). I searched the archives of 
this mailing list, and found a few interesting references network memory 
consumption in the changelog of the Alan Cox series, but nothing that 
explicitly described a problem like this. Thanks to anybody who can help 
me out here.
-- 

Bill Shubert (wms@igoweb.org) <mailto:wms@igoweb.org>
http://www.igoweb.org/~wms/ <http://igoweb.org/%7Ewms/>