Re: Network latency regressions from 2.6.22 to 2.6.29 (results with IRQ affinity)

[Eric Dumazet <dada1@cosmosbay.com>]

Christoph Lameter a écrit :
> On Mon, 20 Apr 2009, Eric Dumazet wrote:
> 
>>> Sounds very good. If I just knew what you are measuring.
>> Rephrasing my email, I was measuring latencies on the receiving machine,
>> by using tcpdump and doing substraction of 'answer_time' and 'request_time'
>> Thing is that timestamps dont care about hardware delays.
>> (we note time when RX interrupt delivers the packet, and time right before giving frame to hardware)
>>
>>
>> 21:04:23.780421 IP 192.168.20.112.9001 > 192.168.20.110.9000: UDP, length 300   (request)
>> 21:04:23.780428 IP 192.168.20.110.9000 > 192.168.20.112.9001: UDP, length 300   (answer)
>>
>> Here, [21:04:23.780428 - 21:04:23.780421] = 7 us
>>
>> So my results are extensively published :)
> 
> But they are not comparable with my results. There could be other effects
> in the system call API etc that have caused this regression. Plus tcpdump
> causes additional copies of the packet to be delivered to user space.

Yep, this was all mentioned in my mail.
I wanted to compare latencies on receiver only, ruling out hardware, and ruling out sender (no need to reboot it)
These latencies are higher than ones without tcpdump, since more copies are involved with tcpdump.

About system call API effects, they are included in my tests.
Since :

t0 : time we receive packet from NIC
  -> wakeup user process, scheduler...
     User process returns from the recvfrom()  copy from system to user space
     User process does the sendto()    copy from user to system space

t1:  -> calling dev_start_xmit()
     packet given to NIC driver (idle during the tests, so should really send the packet asap)

     User call again recvfrom() and block (this one is not accounted in latency, as in your test)
t2: NIC driver acknowledge the TX 

delta = t1 - t0

One thing that could hurt is the TX done interrupt, but this is done a few us after "t1" so it
doesnt hurt your workload, since next frame is received at least 100us after the last answer...

(cpu is idle 99%)

Point is that even with tcpdump running, latencies are very good on 2.6.30-rc2, and were very good
with 2.6.22. I see no significant increase/decrease...


> 
>>> CONFIG_HPET_TIMER=y
>>> CONFIG_HPET_EMULATE_RTC=y
>>> CONFIG_NR_CPUS=32
>>> CONFIG_SCHED_SMT=y
>> OK, I had "# CONFIG_SCHED_SMT is not set"
>> I'll try with this option set
> 
> Should not be relevant since the processor has no hyperthreading.
> 
>> Are you running a 32 or 64 bits kernel ?
> 
> Test was done using a 64 bit kernel.

Ok, I'll try 64bit too :)

1 us is time to access about 10 false shared cache lines.
64 bit arches store less pointers/long per cache line.
So a 64 bit kernel could be slower on this kind of workload in the general case (if several cpus play the game)