Re: Rescheduling interrupts

["S.Çağlar Onur" <caglar@pardus.org.tr>]

Hi;

22 Oca 2008 Sal tarihinde, Ingo Molnar şunları yazmıştı: 
> * S.Çağlar Onur <caglar@pardus.org.tr> wrote:
> 
> > Top causes for wakeups:
> >   59,9% (238,4)      <kernel IPI> : Rescheduling interrupts
> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >   14,7% ( 58,6)         amarokapp : schedule_timeout (process_timeout)
> 
> hm, would be nice to figure out what causes these IPIs. Could you stick 
> something like this into arch/x86/kernel/smp_32.c's 
> smp_send_reschedule() function [this is the function that generates the 
> IPI]:
> 
> static void native_smp_send_reschedule(int cpu)
> {
>         WARN_ON(cpu_is_offline(cpu));
>         send_IPI_mask(cpumask_of_cpu(cpu), RESCHEDULE_VECTOR);
> 	if (panic_timeout > 0) {
> 		panic_timeout--;
> 		printk("IPI from task %s:%d on CPU#%d:\n", 
> 			current->comm, current->pid, cpu);
> 		dump_stack();
> 	}
> }
> 
> NOTE: if you run an SMP kernel then first remove these two lines from 
> kernel/printk.c:
> 
>         if (!oops_in_progress && waitqueue_active(&log_wait))
>                 wake_up_interruptible(&log_wait);
> 
> otherwise you'll get lockups. (the IPI is sent while holding the 
> runqueue lock, so the printks will lock up)
> 
> then wait for the bad condition to occur on your system and generate a 
> stream of ~10 backtraces, via:
> 
> 	echo 10 > /proc/sys/kernel/panic
> 
> you should be getting 10 immediate backtraces - please send them to us. 
> The backtraces should show the place that generates the wakeups. [turn 
> on CONFIG_FRAME_POINTERS=y to get high quality backtraces.]
>  
> If you do _not_ get 10 immediate backtraces, then something in the 
> system is generating such IPIs outside of the scheduler's control. That 
> would suggest some other sort of borkage.
> 
> 	Ingo

I grabbed the logs two times to make sure to catch needed info. 1st [1] one is generated while "Rescheduling interrupts" wakeups ~200 times and 2nd one generated for ~350 wakeups.

[1] http://cekirdek.pardus.org.tr/~caglar/dmesg.1st 
[2] http://cekirdek.pardus.org.tr/~caglar/dmesg.2nd

Cheers
-- 
S.Çağlar Onur <caglar@pardus.org.tr>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!