Re: Network latency regressions from 2.6.22 to 2.6.29 (results with IRQ affinity)

[Eric Dumazet <dada1@cosmosbay.com>]

Christoph Lameter a écrit :
> On Sat, 18 Apr 2009, Eric Dumazet wrote:
> 
>> I did my tests with 3 machines, trying to reproduce your results and failed.
> 
> Ow.... Where are the results? Seems that you only included the results of
> traces? The numbers would be much higher with udpping since it measures
> the full round trip time from user space of one machine to the other and
> back. The code segment that you are tracing may not be relevant.
> 
>> I only took traces on receivers to rule out different coalescing delays
>> of NIC that could depend on firmware or version
> 
> Hmmm... Okay there could be a firmware influence....
> 
>> bnx2 driver statically included in vmlinux.
>> (on newkernel, its "disable_msi" set to 1 (because this is the only way my BCM5708S can
>>  have interrupt affinities set by the admin used by the hardware on 2.6.28+)
> 
> Here bnx2 is a module and MSI is enabled after it became available
> (2.6.27+). Affinities seem to be effective after resetting the driver.

Hmm, I am not sure I can do that, I would loose my ssh connection...

> 
>> If I force udpping to same cpu than IRQS (CPU0) I get : 7us, 8us, 7us, 7us
>> And yes, this is very good and expected IMHO :)
> 
> Sounds very good. If I just knew what you are measuring.

Rephrasing my email, I was measuring latencies on the receiving machine,
by using tcpdump and doing substraction of 'answer_time' and 'request_time'
Thing is that timestamps dont care about hardware delays.
(we note time when RX interrupt delivers the packet, and time right before giving frame to hardware)


21:04:23.780421 IP 192.168.20.112.9001 > 192.168.20.110.9000: UDP, length 300   (request)
21:04:23.780428 IP 192.168.20.110.9000 > 192.168.20.112.9001: UDP, length 300   (answer)

Here, [21:04:23.780428 - 21:04:23.780421] = 7 us

So my results are extensively published :)


> 
>> In conclusion, something might be wrong in your methodology, or your hardware, or...
> 
> Cannot really say since the results are not published? We can retry this
> by making sure that udpping runs on the same processor as the IRQ.
> 
>> Please Christoph, would you mind posting more information on your config ?
> 
> Sure.
> 
> Dell PowerEdge 1950
> http://www1.ap.dell.com/content/products/productdetails.aspx/pedge_1950?c=au&cs=aubsd1&l=en&s=bsd
> 
>> Any chance you have CONFIG_PREEMT enabled or any expensive kernel debug thing ?
> 
> Nope. Even if so: Would that not be offset by running the same kernel
> version?

I dont know, it was a random thought of mine, since I had in the past bad results
if this one was enabled. This might had changed.

> 
> CONFIG_HPET_TIMER=y
> CONFIG_HPET_EMULATE_RTC=y
> CONFIG_NR_CPUS=32
> CONFIG_SCHED_SMT=y

OK, I had "# CONFIG_SCHED_SMT is not set"
I'll try with this option set

> CONFIG_SCHED_MC=y
> CONFIG_PREEMPT_NONE=y
> # CONFIG_PREEMPT_VOLUNTARY is not set
> # CONFIG_PREEMPT is not set
> # CONFIG_PREEMPT_BKL is not set
> CONFIG_X86_LOCAL_APIC=y
> 
> #
> # Kernel hacking
> #
> CONFIG_TRACE_IRQFLAGS_SUPPORT=y
> # CONFIG_PRINTK_TIME is not set
> CONFIG_ENABLE_MUST_CHECK=y
> CONFIG_MAGIC_SYSRQ=y
> CONFIG_UNUSED_SYMBOLS=y
> # CONFIG_DEBUG_FS is not set
> # CONFIG_HEADERS_CHECK is not set
> CONFIG_DEBUG_KERNEL=y
> # CONFIG_DEBUG_SHIRQ is not set
> CONFIG_DETECT_SOFTLOCKUP=y
> # CONFIG_SCHEDSTATS is not set
> # CONFIG_TIMER_STATS is not set
> # CONFIG_DEBUG_SLAB is not set
> # CONFIG_DEBUG_RT_MUTEXES is not set
> # CONFIG_RT_MUTEX_TESTER is not set
> # CONFIG_DEBUG_SPINLOCK is not set
> # CONFIG_DEBUG_MUTEXES is not set
> # CONFIG_DEBUG_LOCK_ALLOC is not set
> # CONFIG_PROVE_LOCKING is not set
> # CONFIG_DEBUG_SPINLOCK_SLEEP is not set
> # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
> # CONFIG_DEBUG_KOBJECT is not set
> # CONFIG_DEBUG_HIGHMEM is not set
> CONFIG_DEBUG_BUGVERBOSE=y
> # CONFIG_DEBUG_INFO is not set
> # CONFIG_DEBUG_VM is not set
> # CONFIG_DEBUG_LIST is not set
> # CONFIG_FRAME_POINTER is not set
> # CONFIG_FORCED_INLINING is not set
> # CONFIG_RCU_TORTURE_TEST is not set
> # CONFIG_LKDTM is not set
> # CONFIG_FAULT_INJECTION is not set
> CONFIG_EARLY_PRINTK=y
> CONFIG_DEBUG_STACKOVERFLOW=y
> # CONFIG_DEBUG_STACK_USAGE is not set
> # CONFIG_DEBUG_RODATA is not set
> # CONFIG_4KSTACKS is not set
> CONFIG_X86_FIND_SMP_CONFIG=y
> CONFIG_X86_MPPARSE=y
> CONFIG_DOUBLEFAULT=y
> 

Are you running a 32 or 64 bits kernel ?

Thanks