Re: Weird problem: mdadm blocks

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 1869 bytes --]

On Sun, 22 Dec 2013 10:01:26 +0100 Hans Kraus <hans@hanswkraus.com> wrote:

> Hi,
> 
> my backup system (running backuppc) has developed a weird problem:
> calls from command line "mdadm --detail /dev/mdX" block, for every
> existing raid on the system, and can be only terminated with ^C. This
> is true even for the newest mdadm built from git.
> 
> "cat /proc/mdstat" blocks too. All mounted raids are working (at least
> ls <mountpoint> is), exept for one, md127 (the storage of backuppc).
> There ls is blocking and is not terminable by ^C.
> 
> The raid structure is the following:
> md2, md3, m4	raid1 for swap, /boot, /
> md30		raid0 for short term storage
> md10, md11, md12, md13	raid0, built from 2x 2TB or 1TB + 3TB drives
> md127		raid5 built from md10, md11, md12, md13
> 
> I recently (some 12 hours ago) added md13 again and the system was
> rebuilding from a degraded state. The file system on md127 is xfs. All
> the physical rives are OK, at least according to smartmontools.
> Webmin 1.660 reports:
> CPU load averages 	16.96 (1 min) 15.04 (5 mins) 12.67 (15 mins)
> CPU usage 		0% user, 1% kernel, 99% IO, 0% idle
> 
> Is there any way to diagnose the problem further? I'm reluctant to
> do a reboot.

Either some process has crashed leaving an 'oops' or 'bug' message in the
kernel logs, or some process is stuck in 'D' state in 'ps'.

So:
 1/ look through kernel logs since boot (e.g. output of 'dmesg', though that
   might not be complete) for anything  unusual - there should be a stack
   trace.
 2/ if there is a process in 'D' state, find how which and get a stack trace
    of it.  Possibly by
     echo w > /proc/sysrq-trigger
   or
     cat /proc/$PID/stack
   or event
     echo t > /proc/sysrq-trigger
   (though that might create lots of output that might be hard to capture).

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]