connect returns EADDRNOTAVAIL on ~600k+ sockets host

[Denys Fedoryshchenko <nuclearcat@nuclearcat.com>]

Hi

I noticed strange behaviour on loaded server (haproxy), while it is 
handling load for https traffic well, tcp monitoring started to give 
false alerts. And here what i found:

HTTPS-BALANCER ~ # telnet 127.0.0.1 23
telnet: can't connect to remote host (127.0.0.1): Cannot assign 
requested address
HTTPS-BALANCER ~ # nc -v 127.0.0.1 23
127.0.0.1 (127.0.0.1:23) open
��\x01��\x1f��\x01��\x03
HTTPS-BALANCER login: ^Cpunt!

The difference is (relevant lines from strace):

socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
bind(3, {sa_family=AF_INET, sin_port=htons(0), 
sin_addr=inet_addr("0.0.0.0")}, 16) = 0
rt_sigaction(SIGALRM, {0x41ecfb, [ALRM], SA_RESTORER|SA_RESTART, 
0x7f80725bf5b0}, {SIG_DFL, [], 0}, 8) = 0
alarm(0)                                = 0
connect(3, {sa_family=AF_INET, sin_port=htons(23), 
sin_addr=inet_addr("127.0.0.1")}, 16) = 0
rt_sigaction(SIGALRM, {SIG_IGN, [ALRM], SA_RESTORER|SA_RESTART, 
0x7f80725bf5b0}, {0x41ecfb, [ALRM], SA_RESTORER|SA_RESTART, 0x7


socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(23), 
sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EADDRNOTAVAIL (Cannot assign 
requested address)
write(2, "telnet: can't connect to remote host (127.0.0.1): Cannot 
assign requested address\n", 82telnet: can't connect to remote host 
(127.0.0.1): Cannot assign requested address

So,
bind(3, {sa_family=AF_INET, sin_port=htons(0), 
sin_addr=inet_addr("0.0.0.0")}, 16) = 0
is fixing issue, but is it really supposed to be like this?

Just for information host load and some network stats:
from /proc/net/protocols, i removed irrelevant zero values entries

protocol  size sockets  memory press maxhdr  slab module     cl co di ac 
io in de sh ss gs se re sp bi br ha uh gp em
UDP        816      1       0   NI       0   yes  kernel      y  y  y  n 
  y  n  y  n  y  y  y  y  y  n  y  y  y  y  n
TCP       1640 728255  751777   no     208   yes  kernel      y  y  y  y 
  y  y  y  y  y  y  y  y  y  n  y  y  y  y  y

sockets: used 674375
TCP: inuse 727092 orphan 55119 tw 138533 alloc 729419 mem 745187
UDP: inuse 1 mem 0
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

0.0.0.0 sockets:
tcp        0      0 0.0.0.0:65530           0.0.0.0:*               
LISTEN
tcp        0      0 0.0.0.0:443             0.0.0.0:*               
LISTEN
tcp        0      0 0.0.0.0:8880            0.0.0.0:*               
LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               
LISTEN
tcp        0      0 0.0.0.0:23              0.0.0.0:*               
LISTEN
udp        0      0 0.0.0.0:53073           0.0.0.0:*

127.0.0.1 sockets (this is watchdog, but if i disable, it wont change 
anything)
tcp        0      0 127.0.0.1:65530         127.0.0.1:55649         
TIME_WAIT
tcp        0      0 127.0.0.1:65530         127.0.0.1:41762         
FIN_WAIT2
tcp        0      0 127.0.0.1:65530         127.0.0.1:36863         
TIME_WAIT
tcp        0      0 127.0.0.1:65530         127.0.0.1:34533         
TIME_WAIT
tcp        0      0 127.0.0.1:65530         127.0.0.1:35262         
TIME_WAIT
tcp        0      0 127.0.0.1:65530         127.0.0.1:45872         
TIME_WAIT