PROBLEM: Infinite listens for haproxy in /proc/net/tcp

PROBLEM: Infinite listens for haproxy in /proc/net/tcp

[Stephane Bakhos]

This happens whether the file is read directly, or using netstat.
It looks like haproxy has millions of LISTEN.

We've tried to see if there is a definitive count, but after 20 minutes of 
running cat /proc/net/tcp we gave up. This is an intermittent problem.

According to ss, there is about 5500 connections running on the system.

ss -t -l -p outputs the correct information.


scripts/ver_linux:
Linux 2.6.36 #3 SMP Tue Nov 16 22:59:17 CET 2010 x86_64 GNU/Linux

Gnu C                  4.3.2
Gnu make               3.81
binutils               2.18.0.20080103
util-linux             2.13.1.1
mount                  2.13.1.1
module-init-tools      3.4
e2fsprogs              1.41.3
Linux C Library        2.7
Dynamic linker (ldd)   2.7
Procps                 3.2.7
Net-tools              1.60
Console-tools          0.2.3
Sh-utils               6.10
udev                   125
Modules Loaded         tcp_diag inet_diag ipt_REJECT xt_tcpudp 
iptable_filter ip_tables x_tables binfmt_misc loop usbhid hid option 
usb_wwan usbserial psmouse evdev pcspkr serio_raw tpm_tis tpm tpm_bios 
ehci_hcd ohci_hcd shpchp pci_hotplug i2c_nforce2 i2c_core processor 
thermal button thermal_sys dcdbas

Re: PROBLEM: Infinite listens for haproxy in /proc/net/tcp

[Willy Tarreau]

Hi,

On Thu, Dec 02, 2010 at 01:18:36PM -0500, Stephane Bakhos wrote:
> This happens whether the file is read directly, or using netstat.
> It looks like haproxy has millions of LISTEN.

I think what you see are not LISTEN sockets but mostly TIME_WAIT.

> We've tried to see if there is a definitive count, but after 20 minutes of 
> running cat /proc/net/tcp we gave up. This is an intermittent problem.

I've been used to experience this on systems which process connections
at higher rates than netstat can output them. Thus, during the time it
takes to netstat to output a given number of connections, new ones
manage to complete the list. I've not tested on 2.6.36 yet, but this
has basically always been the case for me for a very long time, and as
such, I only use ss to display connections. I've recently discovered
that some versions of netstat use netlink to dump the socket list, so
on such versions it would not be an issue.

> According to ss, there is about 5500 connections running on the system.

What normally counts is the rate at which they're refreshed.

Regards,
Willy