Re: cat /proc/net/tcp takes 0.5 seconds on x86_64

[Eric Dumazet <dada1@cosmosbay.com>]

Dave Jones a écrit :
> Just had this bug reported against our development tree..
> 
> 	Dave
> 
> On Tue, Aug 26, 2008 at 11:49:31AM -0400, bugzilla@redhat.com wrote:
>  > Please do not reply directly to this email. All additional
>  > comments should be made in the comments box of this bug.
>  > 
>  > https://bugzilla.redhat.com/show_bug.cgi?id=459782
>  > 
>  > Hans de Goede <j.w.r.degoede@hhs.nl> changed:
>  > 
>  >            What    |Removed                     |Added
>  > ----------------------------------------------------------------------------
>  >                  CC|                            |j.w.r.degoede@hhs.nl
>  >           Component|gkrellm                     |kernel
>  >          AssignedTo|j.w.r.degoede@hhs.nl        |kernel-maint@redhat.com
>  >             Summary|gkrellmd consumes about 75% |cat /proc/net/tcp takes 0.5
>  >                    |cpu time                    |seconds on x86_64, 0.5
>  >                    |                            |seconds !!
>  > 
>  > --- Comment #2 from Hans de Goede <j.w.r.degoede@hhs.nl>  2008-08-26 11:49:30 EDT ---
>  > Thanks for reporting this some stracing of gkrellmd has found that reading from
>  > /proc/net/tcp and reading from /proc/net/tcp6 is the culprit, try this on your
>  > x86_64 machine to confirm:
>  > 
>  > "time cat /proc/net/tcp"
>  > 
>  > To give you an idea on my rawhide x86_64 machine:
>  > [hans@localhost devel]$ time cat /proc/net/tcp
>  > <snip>
>  > real    0m0.520s
>  > user    0m0.000s
>  > sys     0m0.446s
>  > 
>  > Thats amazingly slow, esp as I only have 8 tcp connections open.
>  > 
>  > Some maybe usefull info: top reports a very high load (50%) from soft IRQ's.
>  > 
>  > Anyways changing this to a kernel bug.
> 

I wonder why this qualifies as a "kernel bug". This is a well known problem.

At least, current kernel versions no longer block softirq for long periods while doing this...

cat /proc/net/tcp is slow and deprecated, since it uses a O(N^2) algo.

tcp hash table size might be a litle bit too large for typical setups (few tcp session, even on a 16 GB machine)
Unfortunatly it is fixed at boot time and not dynamic (yet)

You can :

1) Boot your machine with a boot cmd "thash_entries=1024" to reduce size of TCP hash table
(typical size on a 4GB machine is : TCP established hash table entries: 262144 (order: 10, 4194304 bytes))

(max size is 524288 entries for machines with >= 8GB memory if no "thash_entries=..." specified, since October 2007
see http://kerneltrap.org/mailarchive/linux-netdev/2007/10/26/359198 )

2) Switch to netlink interface instead of /proc/net/tcp[6] legacy file.
 Example : netstat -N
   http://www.ducksong.com/misc/netstat-netlink-diag-patch.txt

3) Use both 1) & 2) :)

4) Submit a patch to dynamically grow tcp hash table :)

Links:

http://kerneltrap.org/mailarchive/linux-netdev/2007/11/1/375782
http://kerneltrap.org/mailarchive/linux-netdev/2007/11/1/376907

Time difference between /proc/net/tcp and netlink on a 4GB x86_64 machine :

# dmesg | grep "TCP established hash"
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
# time cat /proc/net/tcp >/dev/null

real    0m0.091s
user    0m0.001s
sys     0m0.090s
# time ss -n >/dev/null # ss uses netlink interface

real    0m0.022s
user    0m0.000s
sys     0m0.022s