Re: cat /proc/net/tcp takes 0.5 seconds on x86_64

[Eric Dumazet <dada1@cosmosbay.com>]

Hans de Goede a écrit :
> Eric Dumazet wrote:
>> Hans de Goede a écrit :
>>> Eric Dumazet wrote:
>>>> Dave Jones a écrit :
>>>>> Just had this bug reported against our development tree..
>>> <snip>
>>>>>  > [hans@localhost devel]$ time cat /proc/net/tcp
>>>>>  > <snip>
>>>>>  > real    0m0.520s
>>>>>  > user    0m0.000s
>>>>>  > sys     0m0.446s
>>>>>  >  > Thats amazingly slow, esp as I only have 8 tcp connections open.
>>>>>  >  > Some maybe usefull info: top reports a very high load (50%) 
>>>>> from soft IRQ's.
>>>>>  >  > Anyways changing this to a kernel bug.
>>>>>
>>>>
>>>> I wonder why this qualifies as a "kernel bug". This is a well known 
>>>> problem.
>>>>
>>>
>>> No its not, /proc/net/tcp may be slow in general but not *this* slow ...
>>>
>>> <snip>
>>>
>>>>
>>>> Time difference between /proc/net/tcp and netlink on a 4GB x86_64 
>>>> machine :
>>>>
>>>> # dmesg | grep "TCP established hash"
>>>> TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
>>>> # time cat /proc/net/tcp >/dev/null
>>>>
>>>> real    0m0.091s
>>>> user    0m0.001s
>>>> sys     0m0.090s
>>>
>>> As quoted above my idle x86_64, using the exact same hash table size, 
>>> running 2.6.27-rc2.git1 uses 0.520 seconds for that same command, 
>>> thats a difference of more then a factor 50 !!
>>>
>>> This is not about /proc/net/tcp not being fast, this is about it 
>>> haven gotten slower by a factor of 50!
>>>
>>> Also notice that this slowdown does not happen on i386.
>>
>> And your .config files on i386 and x86_64 are ?
>> Some configuration options can slow down all lock/unlock operations 
>> (CONFIG_SMP, CONFIG_PREEMPT, CONFIG_PROVE_LOCKING, 
>> CONFIG_DEBUG_SPINLOCK, CONFIG_NR_CPUS ...)
>>
> 
> Attached
> 
>> If you TCP hash table has 512.000 slots (I am just guessing, you didnt 
>> provide this information), it can make a huge difference.
> 
> I did provide that information: "using the exact same hash table size" 
> and then quoting your first mail in this thread:
> "TCP established hash table entries: 262144 (order: 10, 4194304 bytes)"
> 
>>>
>>> Anyways I'll try 2.6.27-rc4 and report back with its results.
>>>
>>
>> Yes, please, but nothing really changed in this area in the recent 
>> times...
>>
> 
> I'm afraid that atleast the Fedora rc4 build won't boot on my machine ...
> 
>> We added some checks so that softirqs can preempt us.
>> Latencies used to be very high, and are now bonded, at the price of 
>> potential slowdown for the /proc/net/tcp reader.
> 
> Slowdown as in 2x or 4x as slow I presume, not 50x ?

I dont know, you tell us 50x, but nowhere I saw your numbers on i386,
nor the amount of memory of your test machine.

One important thing to remember is that on i386, LOWMEM is less than 1GB, 
so a 4GB server will give different hash sizes depending on being 32 or 64 bits.

With a 32 bits kernel:

# dmesg | grep "TCP established"
TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
# time cat /proc/net/tcp >/dev/null

real    0m0.025s
user    0m0.000s
sys     0m0.017s


While on a 64 bits kernel :

# dmesg | grep "TCP established hash"
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
# time cat /proc/net/tcp >/dev/null

real    0m0.091s
user    0m0.001s
sys     0m0.090s 

So I see a 3x on my machine, not a 50x as you ?