messages after server crash

messages after server crash

[Bernd Schubert]

[-- Attachment #1: Type: text/plain, Size: 2778 bytes --]

Hi,

after our server crashed on Friday and Sunday, we can see some strange raid 
related messages in the logs (please see the messages below).

Well, the 'delaying resync ...' messages are easy to understand, but are they 
related to the 'resync aborted' messages ? At least the /proc/mdstat looks 
fine.

The bzip2-output of dmesg (that might give more information about what is 
going on) is attached.


So do we have do worry about something going wrong with the software raid ? 

Of course, we are a bit worried about the the crashes at all, since this never 
happend before.


Thanks in advance,

Bernd


PS: The kernel is a vanilla 2.4.19 with some of Neils NFS-patches that went 
into 2.4.20pre1 (all except the tcp-patches).


Nov 10 11:38:31 coulomb kernel: raid5: resync aborted!
Nov 10 11:38:31 coulomb kernel: md: delaying resync of md1 until md0 has
finished resync (they share one or more physical units)
Nov 10 11:38:31 coulomb kernel: md: delaying resync of md2 until md0 has
finished resync (they share one or more physical units)
Nov 10 11:38:31 coulomb kernel: md: delaying resync of md3 until md0 has
finished resync (they share one or more physical units)
Nov 10 11:38:31 coulomb kernel: raid5: resync aborted!
Nov 10 11:38:31 coulomb kernel: md: delaying resync of md1 until md0 has
finished resync (they share one or more physical units)
Nov 10 11:38:31 coulomb kernel: md: delaying resync of md2 until md0 has
finished resync (they share one or more physical units)
Nov 10 11:38:31 coulomb kernel: raid5: resync aborted!
Nov 10 11:38:31 coulomb kernel: md: delaying resync of md1 until md0 has
finished resync (they share one or more physical units)
Nov 10 11:38:31 coulomb kernel: raid5: resync aborted!
Nov 10 11:38:31 coulomb kernel: md: recovery thread got woken up ...
Nov 10 11:38:31 coulomb kernel: md: recovery thread finished ...
Nov 10 11:38:31 coulomb kernel: md: md_do_sync() got signal ... exiting
Nov 10 11:38:31 coulomb kernel: raid5: resync aborted!

coulomb:~ # cat /proc/mdstat
Personalities : [raid5]
read_ahead 1024 sectors
md4 : active raid5 sde8[4] sdd8[3] sdc8[2] sdb8[1] sda8[0]
      33960960 blocks level 5, 32k chunk, algorithm 2 [5/5] [UUUUU]

md3 : active raid5 sde7[4] sdd7[3] sdc7[2] sdb7[1] sda7[0]
      31261952 blocks level 5, 32k chunk, algorithm 2 [5/5] [UUUUU]

md2 : active raid5 sde6[4] sdd6[3] sdc6[2] sdb6[1] sda6[0]
      31261952 blocks level 5, 32k chunk, algorithm 2 [5/5] [UUUUU]

md1 : active raid5 sde5[4] sdd5[3] sdc5[2] sdb5[1] sda5[0]
      31261952 blocks level 5, 32k chunk, algorithm 2 [5/5] [UUUUU]

md0 : active raid5 sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
      15614720 blocks level 5, 32k chunk, algorithm 2 [5/5] [UUUUU]



[-- Attachment #2: dmesg.out.bz2 --]
[-- Type: application/x-bzip2, Size: 2988 bytes --]

Re: messages after server crash

[Neil Brown]

On Monday November 11, bernd-schubert@web.de wrote:
> Hi,
> 
> after our server crashed on Friday and Sunday, we can see some strange raid 
> related messages in the logs (please see the messages below).
> 
> Well, the 'delaying resync ...' messages are easy to understand, but are they 
> related to the 'resync aborted' messages ? At least the /proc/mdstat looks 
> fine.

It looks fine ... but the arrays probably aren't in sync.
The 'resync aborted' is caused either by IO errors, which should show
up in dmesg, or the resync threads being signaled, or by the array
being switched into readonly mode.

I'm curious about this raidautorun which said:
md: raidautorun(pid 89) used obsolete MD ioctl, upgrade your software to use new ictls

That appears to be part of mkinitrd.  What distribution are you
running? What version?  What version of mkinitrd?

NeilBrown

Re: messages after server crash

[Bernd Schubert]

Hello Neil,

thanks for your answer.

> > Well, the 'delaying resync ...' messages are easy to understand, but are
> > they related to the 'resync aborted' messages ? At least the /proc/mdstat
> > looks fine.
>
> It looks fine ... but the arrays probably aren't in sync.

How can we force the syncing ?

> The 'resync aborted' is caused either by IO errors, which should show
> up in dmesg, or the resync threads being signaled, or by the array
> being switched into readonly mode.

Hmm, I think the dmesg-output doesn't show IO-errors and since we write to the 
disks (our home-partition is on it), it shouldn't be readonly, either.

>
> I'm curious about this raidautorun which said:
> md: raidautorun(pid 89) used obsolete MD ioctl, upgrade your software to
> use new ictls
>
> That appears to be part of mkinitrd.  What distribution are you
> running? What version?  What version of mkinitrd?
>

This is a Suse-7.3 system, but with vanilla kernel and without initrd-support. 
I guess it comes from the Suse-start-up-scripts, that try to enable the raid, 
though it was already enabled by the kernel.
The raidautorun-binary comes from the Suse-raidtools package (version 0.9), is 
there an upgraded version available (I found only patched debian packages, 
but no general tgz-files) ?

So thanks again,

Bernd

Re: messages after server crash

[Neil Brown]

On Tuesday November 12, bernd-schubert@web.de wrote:
> Hello Neil,
> 
> thanks for your answer.
> 
> > > Well, the 'delaying resync ...' messages are easy to understand, but are
> > > they related to the 'resync aborted' messages ? At least the /proc/mdstat
> > > looks fine.
> >
> > It looks fine ... but the arrays probably aren't in sync.
> 
> How can we force the syncing ?

You could try
		mdadm --readwrite /dev/md?

that should probably do it.

> 
> > The 'resync aborted' is caused either by IO errors, which should show
> > up in dmesg, or the resync threads being signaled, or by the array
> > being switched into readonly mode.
> 
> Hmm, I think the dmesg-output doesn't show IO-errors and since we write to the 
> disks (our home-partition is on it), it shouldn't be readonly, either.
> 

Which only leaves some process sending a SIGKILL to the raid5syncd
thread... is that at all possible?

> >
> > I'm curious about this raidautorun which said:
> > md: raidautorun(pid 89) used obsolete MD ioctl, upgrade your software to
> > use new ictls
> >
> > That appears to be part of mkinitrd.  What distribution are you
> > running? What version?  What version of mkinitrd?
> >
> 
> This is a Suse-7.3 system, but with vanilla kernel and without initrd-support. 
> I guess it comes from the Suse-start-up-scripts, that try to enable the raid, 
> though it was already enabled by the kernel.
> The raidautorun-binary comes from the Suse-raidtools package (version 0.9), is 
> there an upgraded version available (I found only patched debian packages, 
> but no general tgz-files) ?

Ok, I managed to find it. It just does ioctl(, RAID_AUTORUN) on
/dev/md0 which just prints out a bad-ioctl error message.  It isn't
implicated at all.

NeilBrown