invalid superblock - why?

invalid superblock - why?

[Dexter Filmore]

raid5, 4 sata disks, slackware with 2.6.14.6.
Yesterday the machine hung, so I used MagicKey to sync, remount read only and 
reboot.
After that the third disk was not assembled into the array.
dmesg came up with:

[   34.652268] md: md0 stopped. 
[   34.742673] md: bind<sdb1> 
[   34.742900] md: invalid superblock checksum on sdc1 
[   34.742920] md: sdc1 has invalid sb, not importing! 
[   34.742941] md: md_import_device returned -22 
[   34.743257] md: bind<sdd1> 
[   34.743464] md: bind<sda1> 
[   34.743506] raid5: device sda1 operational as raid disk 0 
[   34.743524] raid5: device sdd1 operational as raid disk 3 
[   34.743542] raid5: device sdb1 operational as raid disk 1 
[   34.744128] raid5: allocated 4195kB for md0 
[   34.744146] raid5: raid level 5 set md0 active with 3 out of 4 devices, alg 
orithm 2 
[   34.744173] RAID5 conf printout: 
[   34.744187]  --- rd:4 wd:3 fd:1 
[   34.744202]  disk 0, o:1, dev:sda1 
[   34.744216]  disk 1, o:1, dev:sdb1 
[   34.744231]  disk 3, o:1, dev:sdd1

So after throwing a minor fit (my first degraded array) I started 
investigating options. Whe I was sure the disk was alright I re-added it and 
resyncing started. 
It's currently still at it.

Question: what happened to that superblock? just because it didn't shut down 
properly? Disk seems to be ok.

-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C++++ UL++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS+ PE Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b++(+++) DI+++ D- G++ e* h>++ r* y?
------END GEEK CODE BLOCK------

http://www.stop1984.com
http://www.againsttcpa.com

Re: invalid superblock - why?

[Neil Brown]

On Monday August 21, Dexter.Filmore@gmx.de wrote:
> raid5, 4 sata disks, slackware with 2.6.14.6.
> Yesterday the machine hung, so I used MagicKey to sync, remount read only and 
> reboot.
> After that the third disk was not assembled into the array.
> dmesg came up with:
> 
> [   34.652268] md: md0 stopped. 
> [   34.742673] md: bind<sdb1> 
> [   34.742900] md: invalid superblock checksum on sdc1 
> [   34.742920] md: sdc1 has invalid sb, not importing! 
> [   34.742941] md: md_import_device returned -22 
> [   34.743257] md: bind<sdd1> 
> [   34.743464] md: bind<sda1> 
> [   34.743506] raid5: device sda1 operational as raid disk 0 
> [   34.743524] raid5: device sdd1 operational as raid disk 3 
> [   34.743542] raid5: device sdb1 operational as raid disk 1 
> [   34.744128] raid5: allocated 4195kB for md0 
> [   34.744146] raid5: raid level 5 set md0 active with 3 out of 4 devices, alg 
> orithm 2 
> [   34.744173] RAID5 conf printout: 
> [   34.744187]  --- rd:4 wd:3 fd:1 
> [   34.744202]  disk 0, o:1, dev:sda1 
> [   34.744216]  disk 1, o:1, dev:sdb1 
> [   34.744231]  disk 3, o:1, dev:sdd1
> 
> So after throwing a minor fit (my first degraded array) I started 
> investigating options. Whe I was sure the disk was alright I re-added it and 
> resyncing started. 
> It's currently still at it.
> 
> Question: what happened to that superblock? just because it didn't shut down 
> properly? Disk seems to be ok.

The output of 
  mdadm -E /dev/sdc1
might have been interesting, but I guess it is too late for that now.

Apart from hardware errors, an incorrect checksum should only happen
if some but not all of the superblock is written (it is 8 sectors
long).
I suspect that is extremely unlikely to happen, and given the symptoms
you describe, I think it is extremely unlike to have happened in your
case.

So I'm afraid I don't know what happened, and unless it happens again,
it would be very hard to trace the problem down.

NeilBrown

Re: invalid superblock - why?

[Dexter Filmore]

Am Montag, 21. August 2006 03:26 schrieb Neil Brown:
>
> The output of
>   mdadm -E /dev/sdc1
> might have been interesting, but I guess it is too late for that now.

True. Bummer. As you can imagine I wanted the array degraded for as short as 
possible and was eager to take measures to bring it back on track.
I'll remember for next time - which I hope is not too soon...


-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C++++ UL++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS+ PE Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b++(+++) DI+++ D- G++ e* h>++ r* y?
------END GEEK CODE BLOCK------

http://www.stop1984.com
http://www.againsttcpa.com

Re: invalid superblock - *again*

[Dexter Filmore]

Am Montag, 21. August 2006 13:04 schrieb Dexter Filmore:
I seriously don't know what's going on here.
I upgraded packages and rebooted the machine to find that now disk 4 of 4 is 
not assembled.

Here's dmesg and mdadm -E 

***************** dmesg ******************
[   38.439644] md: md0 stopped.
[   38.536089] md: bind<sdb1>
[   38.536301] md: bind<sdc1>
[   38.536501] md: bind<sdd1>
[   38.536702] md: bind<sda1>
[   38.536733] md: kicking non-fresh sdd1 from array!
[   38.536751] md: unbind<sdd1>
[   38.536765] md: export_rdev(sdd1)
[   38.536794] raid5: device sda1 operational as raid disk 0
[   38.536812] raid5: device sdc1 operational as raid disk 2
[   38.536831] raid5: device sdb1 operational as raid disk 1
[   38.537453] raid5: allocated 4195kB for md0
[   38.537471] raid5: raid level 5 set md0 active with 3 out of 4 devices, 
algor
ithm 2
[   38.537499] RAID5 conf printout:
[   38.537513]  --- rd:4 wd:3 fd:1
[   38.537528]  disk 0, o:1, dev:sda1
[   38.537543]  disk 1, o:1, dev:sdb1
[   38.537558]  disk 2, o:1, dev:sdc1
*********************************************

Most notable: [   38.536733] md: kicking non-fresh sdd1 from array!
What does this mean?

********* mdadm -E /dev/sdd1 ****************
/dev/sdd1:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : 7f103422:7be2c2ce:e67a70be:112a2914
  Creation Time : Tue May  9 01:11:41 2006
     Raid Level : raid5
    Device Size : 244187904 (232.88 GiB 250.05 GB)
     Array Size : 732563712 (698.63 GiB 750.15 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Tue Aug 22 01:42:36 2006
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 33b2d59b - correct
         Events : 0.765488

         Layout : left-symmetric
     Chunk Size : 32K

      Number   Major   Minor   RaidDevice State
this     3       8       49        3      active sync   /dev/sdd1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8       49        3      active sync   /dev/sdd1

*************************************************

What's happening here? What can I do? Do I have to readd sdd and resync? Or is 
there an easier way out? What causes these issues?


-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C++++ UL++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS+ PE Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b++(+++) DI+++ D- G++ e* h>++ r* y?
------END GEEK CODE BLOCK------

http://www.stop1984.com
http://www.againsttcpa.com

Re: invalid superblock - *again*

[Neil Brown]

On Tuesday August 22, Dexter.Filmore@gmx.de wrote:
> Am Montag, 21. August 2006 13:04 schrieb Dexter Filmore:
> I seriously don't know what's going on here.
> I upgraded packages and rebooted the machine to find that now disk 4 of 4 is 
> not assembled.
> 
> Here's dmesg and mdadm -E 
> 
> ***************** dmesg ******************
> [   38.439644] md: md0 stopped.
> [   38.536089] md: bind<sdb1>
> [   38.536301] md: bind<sdc1>
> [   38.536501] md: bind<sdd1>
> [   38.536702] md: bind<sda1>
> [   38.536733] md: kicking non-fresh sdd1 from array!
> [   38.536751] md: unbind<sdd1>
> [   38.536765] md: export_rdev(sdd1)
> [   38.536794] raid5: device sda1 operational as raid disk 0
> [   38.536812] raid5: device sdc1 operational as raid disk 2
> [   38.536831] raid5: device sdb1 operational as raid disk 1
> [   38.537453] raid5: allocated 4195kB for md0
> [   38.537471] raid5: raid level 5 set md0 active with 3 out of 4 devices, 
> algor
> ithm 2
> [   38.537499] RAID5 conf printout:
> [   38.537513]  --- rd:4 wd:3 fd:1
> [   38.537528]  disk 0, o:1, dev:sda1
> [   38.537543]  disk 1, o:1, dev:sdb1
> [   38.537558]  disk 2, o:1, dev:sdc1
> *********************************************
> 
> Most notable: [   38.536733] md: kicking non-fresh sdd1 from array!
> What does this mean?

It means that the 'event' count on sdd1 is old compared to that on
the other partitions.  The most likely explanation is that when the
array was last running, sdd1 was not part of it.

> 
> What's happening here? What can I do? Do I have to readd sdd and resync? Or is 
> there an easier way out? What causes these issues?
> 

Yes, you need to add sdd1 back to the array and it will resync.

I would need some precise recent history of the array to know why this
happened.  That might not be easy to come by.

NeilBrown

Re: invalid superblock - *again*

[Dexter Filmore]

Am Dienstag, 22. August 2006 03:18 schrieb Neil Brown:
> >
> > Most notable: [   38.536733] md: kicking non-fresh sdd1 from array!
> > What does this mean?
>
> It means that the 'event' count on sdd1 is old compared to that on
> the other partitions.  The most likely explanation is that when the
> array was last running, sdd1 was not part of it.

Event count - so: a certain command or set of instructions was sent to all 
disks, but one didn't get it, hence the raid module can't ensure that the 
data on that disk is consistent with the rest of the array?

> > What's happening here? What can I do? Do I have to readd sdd and resync?
> > Or is there an easier way out? What causes these issues?
>
> Yes, you need to add sdd1 back to the array and it will resync.

Ok, if that's what it takes.

> I would need some precise recent history of the array to know why this
> happened.  That might not be easy to come by.

Depends on what exactly you mean. Disk age? smart data? Hardware types? Logs? 
OS?

I don't have more than a few vague guesses about what might have happened. 
First of all it might be possible that the file systems on the array were not 
unmounted properly during shutdown because a remote NFS mount was hogging 
them. If that be the case, LVM couldn't have shut down properly, then the md 
device wouldn't have stopped and the machine just powered down.
That would explain it.
Slackware has no raid runlevel scripts out of the box, I wrote them myself.
Maybe such conditions are not handled properly.

What speaks against that theory is that nfsd is stopped before lvm and raid is 
handled.

-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C++++ UL++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS+ PE Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b++(+++) DI+++ D- G++ e* h>++ r* y?
------END GEEK CODE BLOCK------

http://www.stop1984.com
http://www.againsttcpa.com

Re: invalid superblock - *again*

[Neil Brown]

On Tuesday August 22, Dexter.Filmore@gmx.de wrote:
> Am Dienstag, 22. August 2006 03:18 schrieb Neil Brown:
> > >
> > > Most notable: [   38.536733] md: kicking non-fresh sdd1 from array!
> > > What does this mean?
> >
> > It means that the 'event' count on sdd1 is old compared to that on
> > the other partitions.  The most likely explanation is that when the
> > array was last running, sdd1 was not part of it.
> 
> Event count - so: a certain command or set of instructions was sent to all 
> disks, but one didn't get it, hence the raid module can't ensure that the 
> data on that disk is consistent with the rest of the array?
> 

Not exactly.  Events are thing like starting and stopping the array,
adding or removing drives, drive failure and clean <-> dirty
transitions.  If the event counts are not consistent, then when the
array was last stopped, one drive (at least) was missing from the
array. 

> > > What's happening here? What can I do? Do I have to readd sdd and resync?
> > > Or is there an easier way out? What causes these issues?
> >
> > Yes, you need to add sdd1 back to the array and it will resync.
> 
> Ok, if that's what it takes.
> 
> > I would need some precise recent history of the array to know why this
> > happened.  That might not be easy to come by.
> 
> Depends on what exactly you mean. Disk age? smart data? Hardware types? Logs? 
> OS?
> 

Complete kernel logs since a time when it was known to be good might
be enough - so I could track all the 'event's and see where it went
out of sync.

> I don't have more than a few vague guesses about what might have happened. 
> First of all it might be possible that the file systems on the array were not 
> unmounted properly during shutdown because a remote NFS mount was hogging 
> them. If that be the case, LVM couldn't have shut down properly, then the md 
> device wouldn't have stopped and the machine just powered down.
> That would explain it.

Maybe but even shutting down with the array still active shouldn't
cause the event counts to go out of sync.  It should just trigger a
resync.

NeilBrown