Re: cat /proc/net/tcp takes 0.5 seconds on x86_64

[Eric Dumazet <dada1@cosmosbay.com>]

Hans de Goede a écrit :
> Eric Dumazet wrote:
>> Dave Jones a écrit :
>>> Just had this bug reported against our development tree..
> <snip>
>>>  > [hans@localhost devel]$ time cat /proc/net/tcp
>>>  > <snip>
>>>  > real    0m0.520s
>>>  > user    0m0.000s
>>>  > sys     0m0.446s
>>>  >  > Thats amazingly slow, esp as I only have 8 tcp connections open.
>>>  >  > Some maybe usefull info: top reports a very high load (50%) 
>>> from soft IRQ's.
>>>  >  > Anyways changing this to a kernel bug.
>>>
>>
>> I wonder why this qualifies as a "kernel bug". This is a well known 
>> problem.
>>
> 
> No its not, /proc/net/tcp may be slow in general but not *this* slow ...
> 
> <snip>
> 
>>
>> Time difference between /proc/net/tcp and netlink on a 4GB x86_64 
>> machine :
>>
>> # dmesg | grep "TCP established hash"
>> TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
>> # time cat /proc/net/tcp >/dev/null
>>
>> real    0m0.091s
>> user    0m0.001s
>> sys     0m0.090s
> 
> As quoted above my idle x86_64, using the exact same hash table size, 
> running 2.6.27-rc2.git1 uses 0.520 seconds for that same command, thats 
> a difference of more then a factor 50 !!
> 
> This is not about /proc/net/tcp not being fast, this is about it haven 
> gotten slower by a factor of 50!
> 
> Also notice that this slowdown does not happen on i386.

And your .config files on i386 and x86_64 are ? 

Some configuration options can slow down all lock/unlock operations (CONFIG_SMP, CONFIG_PREEMPT, CONFIG_PROVE_LOCKING, CONFIG_DEBUG_SPINLOCK, CONFIG_NR_CPUS ...)

If you TCP hash table has 512.000 slots (I am just guessing, you didnt provide this information), it can make a huge difference. 

> 
> Anyways I'll try 2.6.27-rc4 and report back with its results.
> 

Yes, please, but nothing really changed in this area in the recent times...

We added some checks so that softirqs can preempt us.
Latencies used to be very high, and are now bonded, at the price of potential slowdown for the /proc/net/tcp reader.