Re: connect returns EADDRNOTAVAIL on ~600k+ sockets host

[Denys Fedoryshchenko <nuclearcat@nuclearcat.com>]

After some "printk" debugging, also found additional information,
error is returned in __inet_hash_connect, as i understand because it 
can't find
free local port.
But i dont think correct, that on such loaded server basic 
socket/connect sequence will not work, while there is for sure free 
ports on ips used for outgoing connection.

On 2014-09-12 21:18, Denys Fedoryshchenko wrote:
> Hi
> 
> I noticed strange behaviour on loaded server (haproxy), while it is
> handling load for https traffic well, tcp monitoring started to give
> false alerts. And here what i found:
> 
> HTTPS-BALANCER ~ # telnet 127.0.0.1 23
> telnet: can't connect to remote host (127.0.0.1): Cannot assign
> requested address
> HTTPS-BALANCER ~ # nc -v 127.0.0.1 23
> 127.0.0.1 (127.0.0.1:23) open
> ��\x01��\x1f��\x01��\x03
> HTTPS-BALANCER login: ^Cpunt!
> 
> The difference is (relevant lines from strace):
> 
> socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> bind(3, {sa_family=AF_INET, sin_port=htons(0),
> sin_addr=inet_addr("0.0.0.0")}, 16) = 0
> rt_sigaction(SIGALRM, {0x41ecfb, [ALRM], SA_RESTORER|SA_RESTART,
> 0x7f80725bf5b0}, {SIG_DFL, [], 0}, 8) = 0
> alarm(0)                                = 0
> connect(3, {sa_family=AF_INET, sin_port=htons(23),
> sin_addr=inet_addr("127.0.0.1")}, 16) = 0
> rt_sigaction(SIGALRM, {SIG_IGN, [ALRM], SA_RESTORER|SA_RESTART,
> 0x7f80725bf5b0}, {0x41ecfb, [ALRM], SA_RESTORER|SA_RESTART, 0x7
> 
> 
> socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> connect(3, {sa_family=AF_INET, sin_port=htons(23),
> sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EADDRNOTAVAIL (Cannot
> assign requested address)
> write(2, "telnet: can't connect to remote host (127.0.0.1): Cannot
> assign requested address\n", 82telnet: can't connect to remote host
> (127.0.0.1): Cannot assign requested address
> 
> So,
> bind(3, {sa_family=AF_INET, sin_port=htons(0),
> sin_addr=inet_addr("0.0.0.0")}, 16) = 0
> is fixing issue, but is it really supposed to be like this?
> 
> Just for information host load and some network stats:
> from /proc/net/protocols, i removed irrelevant zero values entries
> 
> protocol  size sockets  memory press maxhdr  slab module     cl co di
> ac io in de sh ss gs se re sp bi br ha uh gp em
> UDP        816      1       0   NI       0   yes  kernel      y  y  y
> n  y  n  y  n  y  y  y  y  y  n  y  y  y  y  n
> TCP       1640 728255  751777   no     208   yes  kernel      y  y  y
> y  y  y  y  y  y  y  y  y  y  n  y  y  y  y  y
> 
> sockets: used 674375
> TCP: inuse 727092 orphan 55119 tw 138533 alloc 729419 mem 745187
> UDP: inuse 1 mem 0
> UDPLITE: inuse 0
> RAW: inuse 0
> FRAG: inuse 0 memory 0
> 
> 0.0.0.0 sockets:
> tcp        0      0 0.0.0.0:65530           0.0.0.0:*               
> LISTEN
> tcp        0      0 0.0.0.0:443             0.0.0.0:*               
> LISTEN
> tcp        0      0 0.0.0.0:8880            0.0.0.0:*               
> LISTEN
> tcp        0      0 0.0.0.0:22              0.0.0.0:*               
> LISTEN
> tcp        0      0 0.0.0.0:23              0.0.0.0:*               
> LISTEN
> udp        0      0 0.0.0.0:53073           0.0.0.0:*
> 
> 127.0.0.1 sockets (this is watchdog, but if i disable, it wont change 
> anything)
> tcp        0      0 127.0.0.1:65530         127.0.0.1:55649         
> TIME_WAIT
> tcp        0      0 127.0.0.1:65530         127.0.0.1:41762         
> FIN_WAIT2
> tcp        0      0 127.0.0.1:65530         127.0.0.1:36863         
> TIME_WAIT
> tcp        0      0 127.0.0.1:65530         127.0.0.1:34533         
> TIME_WAIT
> tcp        0      0 127.0.0.1:65530         127.0.0.1:35262         
> TIME_WAIT
> tcp        0      0 127.0.0.1:65530         127.0.0.1:45872         
> TIME_WAIT