Re: PROBLEM: Linux kernel 2.6.31 IPv4 TCP fails to open huge amount of outgoing connections (unable to bind ... )

["George B." <georgeb@gmail.com>]

On Wed, Apr 21, 2010 at 4:27 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> Here is the patch I use now and my test application is now able to open
> and connect 1000000 sockets (ulimit -n 1000000)

I believe we hit this very yesterday in our test lab.  We had a stress
test running of one of our applications with about a dozen instances
of it running on the box.  Suddenly dns requests began failing with
the complaint that it couldn't make a request out because there were
no sockets.

root@champagne:/proc/sys/net/ipv4> host gh
host: isc_socket_bind: address in use

Netstat showed 61580 total sockets (UDP and TCP) on the address being
used by the above dns request. (local port range 1025 65535).  That
dns request should not have been failing.

I noticed that the number of UDP sockets was close to the maximum
allowed by the port range, but they were across different IP
addresses, no one IP address had too many and there should have been
available ports on all IP addresses.

Further, the number of udp sockets in use seemed to hit the wall at a
little above 64,000 and I never got above that number.

If that is the normal behavior of the kernel, it could be a big
problem for scaling the application.