Missing md superblock on added devices after grow ...

Missing md superblock on added devices after grow ...

[rdc]

Yesterday i grew my 4*500GB raid5 array to 7*500GB. The reshape operation 
ended succesfully according to the logs : 

May 22 22:24:56 [kernel] md: md2: reshape done.
May 22 22:24:56 [kernel] RAID5 conf printout:
May 22 22:24:56 [kernel]  --- rd:7 wd:7
May 22 22:24:56 [kernel]  disk 0, o:1, dev:dm-3
May 22 22:24:56 [kernel]  disk 1, o:1, dev:dm-0
May 22 22:24:56 [kernel]  disk 2, o:1, dev:dm-2
May 22 22:24:56 [kernel]  disk 3, o:1, dev:dm-1
May 22 22:24:56 [kernel]  disk 4, o:1, dev:dm-8
May 22 22:24:56 [kernel]  disk 5, o:1, dev:dm-6
May 22 22:24:56 [kernel]  disk 6, o:1, dev:dm-5

However, upon restarting this morning the grown array was no longer detected, 
and my investigation revealed that the 3 device that were added do not contain 
an md superblock at all !

One of the 3 new devices :
mdadm --examine -v /dev/sdb1
mdadm: No md superblock detected on /dev/sdb1.

The 4 original devices do contain a superblock :
mdadm --examine -v /dev/sdc1
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : 9ee9252c:7eeb7160:fd021026:b68a2612
           Name : md/md2
  Creation Time : Thu Jan  1 01:00:00 1970
     Raid Level : raid5
   Raid Devices : 7

 Avail Dev Size : 976751872 (465.75 GiB 500.10 GB)
     Array Size : 5860511232 (2794.51 GiB 3000.58 GB)
   Super Offset : 976751920 sectors
          State : clean
    Device UUID : e2e95a27:c1d3d21d:2c7f1870:97156b44

    Update Time : Sat May 23 03:31:39 2009
       Checksum : 2b125ad9 - correct
         Events : 1790846

         Layout : left-symmetric
     Chunk Size : 32K

    Array Slot : 1 (failed, 1, failed, 3, 6, 2, 0, 5, 4)
   Array State : uUuuuuu 2 failed

Kernel version is 2.6.29-gentoo-r1, mdadm is v2.6.9
Any ideas how i could fix this, or even start the array to get my data off ?

Re: Missing md superblock on added devices after grow ...

[NeilBrown]

On Sun, May 24, 2009 1:03 am, rdc wrote:
> Yesterday i grew my 4*500GB raid5 array to 7*500GB. The reshape operation
> ended succesfully according to the logs :
>
> May 22 22:24:56 [kernel] md: md2: reshape done.
> May 22 22:24:56 [kernel] RAID5 conf printout:
> May 22 22:24:56 [kernel]  --- rd:7 wd:7
> May 22 22:24:56 [kernel]  disk 0, o:1, dev:dm-3
> May 22 22:24:56 [kernel]  disk 1, o:1, dev:dm-0
> May 22 22:24:56 [kernel]  disk 2, o:1, dev:dm-2
> May 22 22:24:56 [kernel]  disk 3, o:1, dev:dm-1
> May 22 22:24:56 [kernel]  disk 4, o:1, dev:dm-8
> May 22 22:24:56 [kernel]  disk 5, o:1, dev:dm-6
> May 22 22:24:56 [kernel]  disk 6, o:1, dev:dm-5

The devices in your array are dm-xx devices, but....

>
> However, upon restarting this morning the grown array was no longer
> detected,
> and my investigation revealed that the 3 device that were added do not
> contain
> an md superblock at all !
>
> One of the 3 new devices :
> mdadm --examine -v /dev/sdb1
> mdadm: No md superblock detected on /dev/sdb1.

... you are looking on sdxx devices for the superblock.  This
seem to me to be a strange thing to do.
What does
   mdadm --example /dev/mapper/whatever
show?

What exactly are the dm-xx devices?

NeilBrown

Re: Missing md superblock on added devices after grow ...

[rdc]

On Saturday 23 May 2009 23:05:35 NeilBrown wrote:
> On Sun, May 24, 2009 1:03 am, rdc wrote:
> > Yesterday i grew my 4*500GB raid5 array to 7*500GB. The reshape operation
> > ended succesfully according to the logs :
> >
> > May 22 22:24:56 [kernel] md: md2: reshape done.
> > May 22 22:24:56 [kernel] RAID5 conf printout:
> > May 22 22:24:56 [kernel]  --- rd:7 wd:7
> > May 22 22:24:56 [kernel]  disk 0, o:1, dev:dm-3
> > May 22 22:24:56 [kernel]  disk 1, o:1, dev:dm-0
> > May 22 22:24:56 [kernel]  disk 2, o:1, dev:dm-2
> > May 22 22:24:56 [kernel]  disk 3, o:1, dev:dm-1
> > May 22 22:24:56 [kernel]  disk 4, o:1, dev:dm-8
> > May 22 22:24:56 [kernel]  disk 5, o:1, dev:dm-6
> > May 22 22:24:56 [kernel]  disk 6, o:1, dev:dm-5
>
> The devices in your array are dm-xx devices, but....
>
> > However, upon restarting this morning the grown array was no longer
> > detected,
> > and my investigation revealed that the 3 device that were added do not
> > contain
> > an md superblock at all !
> >
> > One of the 3 new devices :
> > mdadm --examine -v /dev/sdb1
> > mdadm: No md superblock detected on /dev/sdb1.
>
> ... you are looking on sdxx devices for the superblock.  This
> seem to me to be a strange thing to do.
> What does
>    mdadm --example /dev/mapper/whatever
> show?

mdadm --examine -v /dev/mapper/sdb1
mdadm: No md superblock detected on /dev/mapper/sdb1

>
> What exactly are the dm-xx devices?

Well the dm-xx devices are just the same partitions mapped into dm by the 
mapper. It just creates /dev/mapper/sdXX  devices which are the same as the 
underlying /dev/sdxx devices. I removed the mappings for clarity. The names 
always show up as dm-XX from a kernel perspective.

>
> NeilBrown

Re: Missing md superblock on added devices after grow ...

[Rob Vandermeulen]

Well, I found my error thanks to your comments about dm. While it's true that 
the /dev/mapper/sdxx devices are the same as /dev/sdxx , in my case this 
wasn't entirely true, as the disks i added had a primary partition on them, 
but i deleted them and made that primary partition a little smaller. However, 
dm mappings are only loaded on boot normally, so these were not reloaded . 
Result : the reshape took place on the older "larger" partitions instead of 
the new, slightly smaller ones. Ofcourse, the superblock was in that area too 
ofcourse. I restored the partitions to their original larger size and was able 
to get the superblocks read, and have succesfully recreated a degraded 7 disk 
array and all data is still intact :)

I guess i really should be more carefull using mapper and real devices when 
changing partition layout.

Thanks for the assistance.

On Saturday 23 May 2009 23:05:35 you wrote:
> On Sun, May 24, 2009 1:03 am, rdc wrote:
> > Yesterday i grew my 4*500GB raid5 array to 7*500GB. The reshape operation
> > ended succesfully according to the logs :
> >
> > May 22 22:24:56 [kernel] md: md2: reshape done.
> > May 22 22:24:56 [kernel] RAID5 conf printout:
> > May 22 22:24:56 [kernel]  --- rd:7 wd:7
> > May 22 22:24:56 [kernel]  disk 0, o:1, dev:dm-3
> > May 22 22:24:56 [kernel]  disk 1, o:1, dev:dm-0
> > May 22 22:24:56 [kernel]  disk 2, o:1, dev:dm-2
> > May 22 22:24:56 [kernel]  disk 3, o:1, dev:dm-1
> > May 22 22:24:56 [kernel]  disk 4, o:1, dev:dm-8
> > May 22 22:24:56 [kernel]  disk 5, o:1, dev:dm-6
> > May 22 22:24:56 [kernel]  disk 6, o:1, dev:dm-5
>
> The devices in your array are dm-xx devices, but....
>
> > However, upon restarting this morning the grown array was no longer
> > detected,
> > and my investigation revealed that the 3 device that were added do not
> > contain
> > an md superblock at all !
> >
> > One of the 3 new devices :
> > mdadm --examine -v /dev/sdb1
> > mdadm: No md superblock detected on /dev/sdb1.
>
> ... you are looking on sdxx devices for the superblock.  This
> seem to me to be a strange thing to do.
> What does
>    mdadm --example /dev/mapper/whatever
> show?
>
> What exactly are the dm-xx devices?
>
> NeilBrown

Re: Missing md superblock on added devices after grow ...

[Billy Crook]

On Sun, May 24, 2009 at 11:15, Rob Vandermeulen
<rob.vandermeulen@telenet.be> wrote:
> I guess i really should be more carefull using mapper and real devices when
> changing partition layout.

If your partitioning tool doesn't automatically re-read the
partitioning layout (and it sounds like it didn't in this case), use
hdparm -z /dev/sda to re-read the partitions on sda.  parted usually
re-reads on its own.  And of course after doing the partitioning, and
before using the partitions as raid components, enumerate the drive
partitioning, and verify it is exactly as you expect:

parted /dev/sda print free