Re: BUG: soft lockup in 2.6.25.5

[Andrew Morton <akpm@linux-foundation.org>]

On Fri, 19 Sep 2008 13:49:16 +1000 Brett Pemberton <brett@vpac.org> wrote:

> I'm getting about 3-5 machines in a cluster of 95 hanging with 
> 
> BUG: soft lockup - CPU#7 stuck for 61s! [pdflush:321]
> 
> per week.  Nothing in common each time, different users running
> different jobs on different nodes.
> 
> The most recent is at the end of this email, .config is attached.
> 
> Googling is scary.  Many people reporting these, but never any response.
> It's happening on enough separate nodes that I can't believe it's
> hardware, although they are identical machines:
> 
> - 2x Quad-Core AMD Opteron(tm) Processor 2356
> - 32gb ram
> - 4 x sata drives
> 
> Running CentOS 5.2 with a kernel.org kernel
> Has been happening with a variety of kernels from 2.6.25 - present.

Yes, it's a false positive.  With a lot of memory and a random-access
or lot-of-files writing behaviour, it can take tremendous amounts of
time to get everything stored on the disk.

Not sure what to do about it, really.  Perhaps touch the softlockup
detector somewhere in the writeback code.

> I'd love any advice on where to turn to next and what avenues to pursue.

Set /proc/sys/kernel/softlockup_thresh to zero to shut it up :(

hm, there's a bug...

--- a/Documentation/sysctl/kernel.txt~a
+++ a/Documentation/sysctl/kernel.txt
@@ -351,9 +351,10 @@ kernel.  This value defaults to SHMMAX.
 
 softlockup_thresh:
 
-This value can be used to lower the softlockup tolerance
-threshold. The default threshold is 10s.  If a cpu is locked up
-for 10s, the kernel complains.  Valid values are 1-60s.
+This value can be used to lower the softlockup tolerance threshold.  The
+default threshold is 60 seconds.  If a cpu is locked up for 60 seconds, the
+kernel complains.  Valid values are 1-60 seconds.  Setting this tunable to
+zero will disable the softlockup detection altogether.
 
 ==============================================================
 
_