RAID-6 disk superblock issue

RAID-6 disk superblock issue

[Troy Telford]

I have a RAID-6 array I'm having trouble with:

md2 : active raid6 sdl1[6] sdh1[0] sdm1[5] sdk1[3] sdj1[2] sdi1[1]
      1953535488 blocks level 6, 128k chunk, algorithm 2 [6/5] [UUUU_U]
      [>....................]  recovery =  0.0% (27164/488383872) 
finish=599.1min speed=13582K/sec

- When the system boots (or when I attempt to build the array), I get a 
message that /dev/sdl and /dev/sdl1 have the same superblock, and that 
I should zero one.
- So, I zero the superblock using 'mdadm --zero-superblock /dev/sdl (or 
/dev/sdl1; doesn't matter which I use).
- I then re-add the device to the array (and the RAID-6 array recovers, 
as above.)
- The next time I boot, I get the exact same error message.


- It doesn't seem to matter which (between the device or the partition) 
that I zero the superblock on.  I get the same error message at boot 
(when the drive is being assembled).
- Since it's redundant data anyway, I've tried dd'ing from /dev/zero 
over the drive, repartitioning the drive, and then adding the drive to 
the array. Same behavior.

- My mdadm.conf just has the array itself, and the array's UUID. It 
doesn't explicitly list devices.

So what am I missing? There's got to be a way I can get the array to 
assemble at boot, instead of having to manually --zero-superblock every 
time.
-- 
Troy Telford

Re: RAID-6 disk superblock issue

[Phil Turmel]

Hi Troy,

On 12/13/2011 05:42 PM, Troy Telford wrote:
> I have a RAID-6 array I'm having trouble with:
> 
> md2 : active raid6 sdl1[6] sdh1[0] sdm1[5] sdk1[3] sdj1[2] sdi1[1]
>      1953535488 blocks level 6, 128k chunk, algorithm 2 [6/5] [UUUU_U]
>      [>....................]  recovery =  0.0% (27164/488383872) finish=599.1min speed=13582K/sec
> 
> - When the system boots (or when I attempt to build the array), I get a message that /dev/sdl and /dev/sdl1 have the same superblock, and that I should zero one.
> - So, I zero the superblock using 'mdadm --zero-superblock /dev/sdl (or /dev/sdl1; doesn't matter which I use).
> - I then re-add the device to the array (and the RAID-6 array recovers, as above.)
> - The next time I boot, I get the exact same error message.
> 
> 
> - It doesn't seem to matter which (between the device or the partition) that I zero the superblock on.  I get the same error message at boot (when the drive is being assembled).
> - Since it's redundant data anyway, I've tried dd'ing from /dev/zero over the drive, repartitioning the drive, and then adding the drive to the array. Same behavior.
> 
> - My mdadm.conf just has the array itself, and the array's UUID. It doesn't explicitly list devices.
> 
> So what am I missing? There's got to be a way I can get the array to assemble at boot, instead of having to manually --zero-superblock every time.

Let me guess:  You have version 0.90 superblock, and sdl1 covers the whole device?

Short term, change your mdadm.conf to only accept device names that end with a digit.  Like so:

DEVICE /dev/sd[a-z][1-9]

Then rebuild your initramfs to include the new mdadm.conf.

Long term, rebuild your array with v1.x metadata.

HTH,

Phil

Re: RAID-6 disk superblock issue

[Troy Telford]

On 2011-12-14 19:01:01 +0000, Phil Turmel said:

> Hi Troy,
> 
> On 12/13/2011 05:42 PM, Troy Telford wrote:
> 
> Let me guess:  You have version 0.90 superblock, and sdl1 covers the 
> whole device?

sdl1 does cover the entire device, but I'm fairly certain I do not have 
a 0.90 superblock (Unless 0.90 was the standard version for Linux 2.6 
about two years ago.)

Unfortunately, I'm not sure how to get my current superblock version:
$ sudo mdadm --examine /dev/md2
mdadm: No md superblock detected on /dev/md2.

I find it curious that I can't detect the superblock for the MD device, 
even though the device is up, active, and working.

> Short term, change your mdadm.conf to only accept device names that end 
> with a digit.  Like so:
> 
> DEVICE /dev/sd[a-z][1-9]

OK, I'll give that a whirl.  For the record, it was DEVICE partitions 
previously.

> Then rebuild your initramfs to include the new mdadm.conf.
> 
> Long term, rebuild your array with v1.x metadata.

You know, I was hoping to be able to wait until btrfs handles RAID-6 
(or "raid-z") arrays by the time I had to rebuild the array.  I guess 
I'm not that lucky.
-- 
Troy Telford

Re: RAID-6 disk superblock issue

[Mark Knecht]

On Wed, Dec 14, 2011 at 1:41 PM, Troy Telford <ttelford.groups@gmail.com> wrote:
<SNIP>
>
> Unfortunately, I'm not sure how to get my current superblock version:
> $ sudo mdadm --examine /dev/md2
> mdadm: No md superblock detected on /dev/md2.
>

use --detail instead:

c2stable ~ # mdadm --detail /dev/md126 | grep Version
        Version : 0.90
c2stable ~ #

HTH,
Mark

Re: RAID-6 disk superblock issue

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1854 bytes --]

On Wed, 14 Dec 2011 14:41:52 -0700 Troy Telford <ttelford.groups@gmail.com>
wrote:

> On 2011-12-14 19:01:01 +0000, Phil Turmel said:
> 
> > Hi Troy,
> > 
> > On 12/13/2011 05:42 PM, Troy Telford wrote:
> > 
> > Let me guess:  You have version 0.90 superblock, and sdl1 covers the 
> > whole device?
> 
> sdl1 does cover the entire device, but I'm fairly certain I do not have 
> a 0.90 superblock (Unless 0.90 was the standard version for Linux 2.6 
> about two years ago.)

It is mdadm that determines the superblock rather than the kernel, but you
definitely have 0.90 superblocks - I can tell from the /proc/mdstat output
(it doesn't list a version, so it must be 0.90).

> 
> Unfortunately, I'm not sure how to get my current superblock version:
> $ sudo mdadm --examine /dev/md2
> mdadm: No md superblock detected on /dev/md2.
> 
> I find it curious that I can't detect the superblock for the MD device, 
> even though the device is up, active, and working.

The array doesn't have a superblock.  Each member device does.
  mdadm --examine /dev/sdl1

> 
> > Short term, change your mdadm.conf to only accept device names that end 
> > with a digit.  Like so:
> > 
> > DEVICE /dev/sd[a-z][1-9]
> 
> OK, I'll give that a whirl.  For the record, it was DEVICE partitions 
> previously.
> 
> > Then rebuild your initramfs to include the new mdadm.conf.
> > 
> > Long term, rebuild your array with v1.x metadata.
> 
> You know, I was hoping to be able to wait until btrfs handles RAID-6 
> (or "raid-z") arrays by the time I had to rebuild the array.  I guess 
> I'm not that lucky.

You don't really need to rebuild the array.  Just change the DEVICE line and
all will be happy.

(but I advise you never to plan on using software that hasn't be released yet
- that way lies madness).

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID-6 disk superblock issue

[Troy Telford]

On 2011-12-14 22:04:12 +0000, Mark Knecht said:

> On Wed, Dec 14, 2011 at 1:41 PM, Troy Telford 
> <ttelford.groups@gmail.com> wrote:
> <SNIP>
>> 
>> Unfortunately, I'm not sure how to get my current superblock version:
>> $ sudo mdadm --examine /dev/md2
>> mdadm: No md superblock detected on /dev/md2.
>> 
> 
> use --detail instead:
> 
> c2stable ~ # mdadm --detail /dev/md126 | grep Version
>         Version : 0.90
> c2stable ~ #
> 
> HTH,
> Mark

Yup. 0.90
$ sudo mdadm --detail /dev/md2
/dev/md2:
        Version : 0.90
  Creation Time : Tue Oct 27 09:45:39 2009

Wow.  It's younger than I thought.

Thanks!
-- 
Troy Telford

Re: RAID-6 disk superblock issue

[Troy Telford]

On 2011-12-14 22:07:51 +0000, NeilBrown said:
> (but I advise you never to plan on using software that hasn't be released yet
> - that way lies madness).

It was more a comment on the pace of btrfs development than a desire to 
switch right away.

I've only tried btrfs on removable drives so far, with unimportant data.

But I won't rule out madness.  It did no end of good for Ford Prefect. 
It sounds like a good deal.
-- 
Troy Telford