preempt-rt, NUMA and strange latency traces

Sébastien Dugué
2006-02-07T11:25:13Z

Hi,

I've been experimenting with 2.6.15-rt16 on a dual 2.8GHz Xeon box
with quite good results and decided to make a run on a NUMA dual node
IBM x440 (8 1.4GHz Xeon, 28GB ram).

However, the kernel crashes early when creating the slabs. Does the
current preempt-rt patchset supports NUMA machines or has support
been disabled until things settle down?

Going on, I compiled a non NUMA RT kernel which booted just fine,
but when examining the latency traces, I came upon strange jumps
in the latencies such as:

<...>-6459 2D.h1 42us : rcu_pending (update_process_times)
<...>-6459 2D.h1 42us : scheduler_tick (update_process_times)
<...>-6459 2D.h1 43us : sched_clock (scheduler_tick)
<...>-6459 2D.h1 44us!: _raw_spin_lock (scheduler_tick)
<...>-6459 2D.h2 28806us : _raw_spin_unlock (scheduler_tick)
<...>-6459 2D.h1 28806us : rebalance_tick (scheduler_tick)
<...>-6459 2D.h1 28807us : irq_exit (smp_apic_timer_interrupt)
<...>-6459 2D..1 28808us < (608)
<...>-6459 2D..1 28809us : smp_apic_timer_interrupt (c03e2a02 0 0)
<...>-6459 2D.h1 28810us : handle_nextevent_update (smp_apic_timer_interrupt)
<...>-6459 2D.h1 28810us : hrtimer_interrupt (handle_nextevent_update)

or

<idle>-0 0Dn.. 11us : __schedule (cpu_idle)
<idle>-0 0Dn.. 12us : profile_hit (__schedule)
<idle>-0 0Dn.1 12us : sched_clock (__schedule)
<idle>-0 0Dn.1 13us : _raw_spin_lock_irq (__schedule)
<...>-6459 0D..2 14us : __switch_to (__schedule)
<...>-6459 0D..2 15us : __schedule <<idle>-0> (8c 0)
<...>-6459 0D..2 16us : _raw_spin_unlock_irq (__schedule)
<...>-6459 0...1 16us!: trace_stop_sched_switched (__schedule)
<...>-6459 0D..1 28585us : smp_apic_timer_interrupt (c013babb 0 0)
<...>-6459 0D.h1 28585us : handle_nextevent_update (smp_apic_timer_interrupt)
<...>-6459 0D.h1 28586us : hrtimer_interrupt (handle_nextevent_update)

or even

<idle>-0 3D.h4 0us!: __trace_start_sched_wakeup (try_to_wake_up)
<idle>-0 3D.h4 28899us : __trace_start_sched_wakeup <<...>-6459> (0 3)
<idle>-0 3D.h4 28900us : _raw_spin_unlock (__trace_start_sched_wakeup)
<idle>-0 3D.h3 28900us : resched_task (try_to_wake_up)

There does not seem to be a precise code path leading to those jumps, it seems
they can appear anywhere. Furthermore the jump seems to always be of ~ 27 ms

I tried running on only 1 CPU, tried using the TSC instead of the cyclone
timer but to no avail, the phenomenon is still there.

My test program only consists in a thread running at max RT priority doing
a nanosleep().

What could be going on here?

Thanks,

Sébastien.