Ports becoming unbindable

[<Ivan.Novick@emc.com>]

Hello,

I have a problem where a linux machine has gotten into a state where a range
of ports are not bindable and yet it seems no application is using those
ports based on netstat and lsof output.  This has happened multiple times on
different machines but I currently have a single machine in this state that
I can do experiments on.

The port range that I cant use is: 59969-60000
The OS is: CentOS release 5.5 -- 2.6.18-194.3.1.el5

Here is python code I use to do the testing:

#############################################################
import socket
HOST = ''
for i in range(59900, 60010):
   try:
      s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
      print i
      s.bind((HOST,i))

   except Exception, e:
      print str(e)
#############################################################

The error message on ports 59969-60000 is (98, 'Address already in use')

#############################################################

Using systemtap to debug the issue I got the following output for 1 call to
bind on a bad port and 1 call to bind on a good port:

     0 python(23637): -> sys_socket
    13 python(23637): <- sys_socket (3)
     0 python(23637): -> sys_bind
     6 python(23637):  -> move_addr_to_kernel
    10 python(23637):  <- move_addr_to_kernel (0)
    15 python(23637): <- sys_bind (-98)
     0 python(23637): -> sys_socket
     8 python(23637): <- sys_socket (4)
     0 python(23637): -> sys_bind
     4 python(23637):  -> move_addr_to_kernel
     7 python(23637):  <- move_addr_to_kernel (0)
    13 python(23637): <- sys_bind (0)


It shows the return code for bind is 98 in the first call (failure) and 0 in
the second call (success)

The call to move_addr_to_kernel returns 0 in both cases and from looking at
kernel sources it does not seem the system call does anything after calling
move_addr_to_kernl and before returning from bind.

Any ideas what could be the issue and or how to debug it?

Cheers,
Ivan