Re: 2.6.21-mm2 - 100% CPU on ksoftirqd/1

[Andrew Morton <akpm@linux-foundation.org>]

On Wed, 09 May 2007 12:08:43 -0400 Valdis.Kletnieks@vt.edu wrote:

> On Wed, 09 May 2007 01:23:22 PDT, Andrew Morton said:
> > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21/2.6.21-mm2/
> 
> Boots up to multiuser mostly OK.  However...
> 
> It comes up with a screaming ksoftirqd - usually /1 but one boot had /0.

erp.

> Just sitting there, 100% CPU according to 'top'.  Tried 'echo t > /proc/sysrq-trigger' to get
> a trace, but it was always running on the other CPU - even after I reniced
> it down to 19 and launched 2 'for(;;)' C programs to suck the cycles.  It would
> be failing to get any CPU - until I did the 'echo t' and then it would be
> "running" again.  Anybody got any good debugging ideas here?
> 
> Oddly enough, the kernel panic I was seeing at X server shutdown, related to
> -x86_64-mm-reloc64-__pa-and-__pa_symbol-address-space-separation.patch,
> seems to be gone now (not extensively tested - but it did survive one shutdown
> without crashing at a point that *had* been a 100% fatal from -rc5-mm3 to 21-mm1.
> I wish I understood why.
> 
> Replicated with a boot to single-user and an untainted kernel - by that point,
> it was spinning at 100% already.
> 
> Any ideas how to debug this one?  
> 

Sure, a kernel profile will tell us.

readprofile -r
sleep 5
readprofile -n -v -m /boot/System.map | sort -n -k 3 | tail -40

or

#!/bin/sh
opcontrol --stop
opcontrol --shutdown
rm -rf /var/lib/oprofile
opcontrol --vmlinux=/boot/vmlinux-$(uname -r)
opcontrol --start-daemon
opcontrol --start
sleep 5
opcontrol --stop
opcontrol --shutdown
opreport -l /boot/vmlinux-$(uname -r) | head -50