MD or MDADM bug?

MD or MDADM bug?

[David M. Strang]

This is somewhat of a crosspost from my thread yesterday; but I think it 
deserves it's own thread atm. Some time ago, I had a device fail -- with the 
help of Neil, Tyler & others on the mailing list; a few patches to mdadm --  
I was able to recover. Using mdadm --remove & mdadm --add, I was able to 
rebuild the bad disc in my array. Everything seemed fine; however -- when I 
rebooted and re-assembled the raid; it wouldn't take the disk that was 
re-added. I had to add it again; and let it rebuild. About 3 weeks ago, I 
lost power -- the outage lasted longer than the UPS, and my system shutdown. 
Upon startup, once again -- I had to re-add 'the disk' back to the array. 
For some reason, if I remove a device and add it back -- when I stop and 
re-assemble the array - it won't 'start' that disk.

Last night, I had a drive fail. With help from Michael & Forrest; I was able 
to attempt to rebuild the array by hot replacing the failed drive without 
rebooting to re-enable disk I/O to that position -- I only had one spare 
available -- it was suspect; and it turns out it was bad. During the 
rebuild, the disk started to have errors -- and the array puked:

Aug 31 21:45:40 abyss kernel: raid5: Disk failure on sda, disabling device. 
Operation continuing on 26 devices
Aug 31 21:45:40 abyss kernel: raid5: Disk failure on sdb, disabling device. 
Operation continuing on 25 devices
Aug 31 21:45:40 abyss kernel: raid5: Disk failure on sdi, disabling device. 
Operation continuing on 18 devices
Aug 31 21:45:40 abyss kernel: raid5: Disk failure on sdj, disabling device. 
Operation continuing on 17 devices
Aug 31 21:45:40 abyss kernel: raid5: Disk failure on sdk, disabling device. 
Operation continuing on 16 devices
Aug 31 21:45:40 abyss kernel: raid5: Disk failure on sdl, disabling device. 
Operation continuing on 15 devices
Aug 31 21:45:40 abyss kernel: raid5: Disk failure on sdn, disabling device. 
Operation continuing on 14 devices

All of this disks tested fine; this happened once before -- simply forcing 
the raid to re-assemble fixes the issue; then replace the bad disk and 
re-sync it.

The problem is; my array is now 26 of 28 disks -- /dev/sdm *IS* bad; it was 
removed and re-added but the new drive is faulty -- however, disk /dev/sdaa 
is not bad -- but, since it was the 'original' disk that was hot removed / 
added so long ago -- it doesn't assemble into the raid. I'm really stuck, I 
can't start the array -- and obviously I can't rebuild the two 'bad'  disks. 
I asked this once before; and was told -- No, you shouldn't have to hotadd 
and resync each time, after hot-adding a "new" device and the initial 
rebuild finishes, unless there's another failure after that, or an unclean 
shutdown.

What can I do? I don't believe this is working as intended.

I'm using mdadm 2.0-devel-3 on a Linux 2.6.11.12 kernel, with version-1 
superblocks.

-- David

Re: MD or MDADM bug?

[Claas Hilbrecht]

--Am Donnerstag, 1. September 2005 17:26 -0400 "David M. Strang" 
<dstrang@shellpower.net> schrieb:

> The problem is; my array is now 26 of 28 disks -- /dev/sdm *IS* bad; it
[...]
> What can I do? I don't believe this is working as intended.

I think the posts:

08.08.2005: How to recover a multiple raid5 disk failure with mdadm?
30.08.2005: 2 partition kicked from 6 raid5

describe the same problem. And it seems that no one was able to help. I 
hope you can rebuild your drive but I think you should use the backup if 
you need a quick solution. And indeed I think a "howto resolve a multiple 
raid5 disk failure" would be a good thing. Sometimes the problems are a 
faulty bus/cable and one knows that most (or even all) data is good.

-- 
 Claas Hilbrecht
 http://www.jucs-kramkiste.de

Re: MD or MDADM bug?

[Claas Hilbrecht]

--Am Donnerstag, 1. September 2005 17:26 -0400 "David M. Strang" 
<dstrang@shellpower.net> schrieb:

> The problem is; my array is now 26 of 28 disks -- /dev/sdm *IS* bad; it
[...]
> What can I do? I don't believe this is working as intended.

Maybe you didn't notice that but there are two recent threads that reports 
nearly the same problem. Look at the posts:

08.08.2005: How to recover a multiple raid5 disk failure with mdadm?
30.08.2005: 2 partition kicked from 6 raid5

And as far as I can see there is no solution yet. Maybe it's faster to 
restore the data from the backup instead of hoping someone can help you. 
But I really think a howto with "how to replace a multiple raid5 disk 
failure" is badly needed. Sometimes it happens that there is a bad cable or 
a problem on a resync. And the users knows that the data is good or at 
least most of the files are.

-- 
 Claas Hilbrecht
 http://www.jucs-kramkiste.de

Re: MD or MDADM bug?

[Neil Brown]

On Thursday September 1, dstrang@shellpower.net wrote:
> This is somewhat of a crosspost from my thread yesterday; but I think it 
> deserves it's own thread atm. Some time ago, I had a device fail -- with the 
> help of Neil, Tyler & others on the mailing list; a few patches to mdadm --  
> I was able to recover. Using mdadm --remove & mdadm --add, I was able to 
> rebuild the bad disc in my array. Everything seemed fine; however -- when I 
> rebooted and re-assembled the raid; it wouldn't take the disk that was 
> re-added. I had to add it again; and let it rebuild. About 3 weeks ago, I 
> lost power -- the outage lasted longer than the UPS, and my system shutdown. 
> Upon startup, once again -- I had to re-add 'the disk' back to the array. 
> For some reason, if I remove a device and add it back -- when I stop and 
> re-assemble the array - it won't 'start' that disk.
....
> 
> I'm using mdadm 2.0-devel-3 on a Linux 2.6.11.12 kernel, with version-1 
> superblocks.

mdadm 2.0 had a fix for assembling version-1 arrays that would
particularly affect raid5.  Try using that instead of -devel-3.

NeilBrown

Re: MD or MDADM bug?

[David M. Strang]

Neil Brown wrote:
> On Thursday September 1, dstrang@shellpower.net wrote:
> > This is somewhat of a crosspost from my thread yesterday; but I think it
> > deserves it's own thread atm. Some time ago, I had a device fail -- with 
> > the
> > help of Neil, Tyler & others on the mailing list; a few patches to 
> > mdadm --
> > I was able to recover. Using mdadm --remove & mdadm --add, I was able to
> > rebuild the bad disc in my array. Everything seemed fine; however --  
> > when I
> > rebooted and re-assembled the raid; it wouldn't take the disk that was
> > re-added. I had to add it again; and let it rebuild. About 3 weeks ago, 
> > I
> > lost power -- the outage lasted longer than the UPS, and my system 
> > shutdown.
> > Upon startup, once again -- I had to re-add 'the disk' back to the 
> > array.
> > For some reason, if I remove a device and add it back -- when I stop and
> > re-assemble the array - it won't 'start' that disk.
> ....
> >
> > I'm using mdadm 2.0-devel-3 on a Linux 2.6.11.12 kernel, with version-1
> > superblocks.

> mdadm 2.0 had a fix for assembling version-1 arrays that would
> particularly affect raid5.  Try using that instead of -devel-3.

No luck -- 

-(root@abyss)-(~)- # mdadm -A /dev/md0 /dev/sda /dev/sdb /dev/sdc /dev/sdd 
/dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl 
/dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt 
/dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa /dev/sdab -f
mdadm: /dev/md0 assembled from 26 drives - not enough to start the array.

Any other suggestions?

-- David 

Re: MD or MDADM bug?

[Neil Brown]

On Friday September 2, dstrang@shellpower.net wrote:
> 
> > mdadm 2.0 had a fix for assembling version-1 arrays that would
> > particularly affect raid5.  Try using that instead of -devel-3.
> 
> No luck -- 
> 
> -(root@abyss)-(~)- # mdadm -A /dev/md0 /dev/sda /dev/sdb /dev/sdc /dev/sdd 
> /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl 
> /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt 
> /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa /dev/sdab -f
> mdadm: /dev/md0 assembled from 26 drives - not enough to start the array.
> 
> Any other suggestions?
> 

Can you run that with '-v' for me?

NeilBrown

Re: MD or MDADM bug?

[David M. Strang]

Neil Brown wrote:
> On Friday September 2, dstrang@shellpower.net wrote:
> >
> > > mdadm 2.0 had a fix for assembling version-1 arrays that would
> > > particularly affect raid5.  Try using that instead of -devel-3.
> >
> > No luck -- 
> >
> > -(root@abyss)-(~)- # mdadm -A /dev/md0 /dev/sda /dev/sdb /dev/sdc 
> > /dev/sdd
> > /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl
> > /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt
> > /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa 
> > /dev/sdab -f
> > mdadm: /dev/md0 assembled from 26 drives - not enough to start the 
> > array.
> >
> > Any other suggestions?
> >
>
> Can you run that with '-v' for me?

mdadm: looking for devices for /dev/md0
mdadm: /dev/sda is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdb is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdf is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sdg is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdh is identified as a member of /dev/md0, slot 7.
mdadm: /dev/sdi is identified as a member of /dev/md0, slot 8.
mdadm: /dev/sdj is identified as a member of /dev/md0, slot 9.
mdadm: /dev/sdk is identified as a member of /dev/md0, slot 10.
mdadm: /dev/sdl is identified as a member of /dev/md0, slot 11.
mdadm: /dev/sdm is identified as a member of /dev/md0, slot -1.
mdadm: /dev/sdn is identified as a member of /dev/md0, slot 13.
mdadm: /dev/sdo is identified as a member of /dev/md0, slot 14.
mdadm: /dev/sdp is identified as a member of /dev/md0, slot 15.
mdadm: /dev/sdq is identified as a member of /dev/md0, slot 16.
mdadm: /dev/sdr is identified as a member of /dev/md0, slot 17.
mdadm: /dev/sds is identified as a member of /dev/md0, slot 18.
mdadm: /dev/sdt is identified as a member of /dev/md0, slot 19.
mdadm: /dev/sdu is identified as a member of /dev/md0, slot 20.
mdadm: /dev/sdv is identified as a member of /dev/md0, slot 21.
mdadm: /dev/sdw is identified as a member of /dev/md0, slot 22.
mdadm: /dev/sdx is identified as a member of /dev/md0, slot 23.
mdadm: /dev/sdy is identified as a member of /dev/md0, slot 24.
mdadm: /dev/sdz is identified as a member of /dev/md0, slot 25.
mdadm: /dev/sdaa is identified as a member of /dev/md0, slot -1.
mdadm: /dev/sdab is identified as a member of /dev/md0, slot 27.
mdadm: added /dev/sdb to /dev/md0 as 1
mdadm: added /dev/sdc to /dev/md0 as 2
mdadm: added /dev/sdd to /dev/md0 as 3
mdadm: added /dev/sde to /dev/md0 as 4
mdadm: added /dev/sdf to /dev/md0 as 5
mdadm: added /dev/sdg to /dev/md0 as 6
mdadm: added /dev/sdh to /dev/md0 as 7
mdadm: added /dev/sdi to /dev/md0 as 8
mdadm: added /dev/sdj to /dev/md0 as 9
mdadm: added /dev/sdk to /dev/md0 as 10
mdadm: added /dev/sdl to /dev/md0 as 11
mdadm: no uptodate device for slot 12 of /dev/md0
mdadm: added /dev/sdn to /dev/md0 as 13
mdadm: added /dev/sdo to /dev/md0 as 14
mdadm: added /dev/sdp to /dev/md0 as 15
mdadm: added /dev/sdq to /dev/md0 as 16
mdadm: added /dev/sdr to /dev/md0 as 17
mdadm: added /dev/sds to /dev/md0 as 18
mdadm: added /dev/sdt to /dev/md0 as 19
mdadm: added /dev/sdu to /dev/md0 as 20
mdadm: added /dev/sdv to /dev/md0 as 21
mdadm: added /dev/sdw to /dev/md0 as 22
mdadm: added /dev/sdx to /dev/md0 as 23
mdadm: added /dev/sdy to /dev/md0 as 24
mdadm: added /dev/sdz to /dev/md0 as 25
mdadm: no uptodate device for slot 26 of /dev/md0
mdadm: added /dev/sdab to /dev/md0 as 27
mdadm: added /dev/sda to /dev/md0 as 0
mdadm: /dev/md0 assembled from 26 drives - not enough to start the array.

Re: MD or MDADM bug?

[Neil Brown]

On Friday September 2, dstrang@shellpower.net wrote:
> >
> > Can you run that with '-v' for me?
> 
> mdadm: looking for devices for /dev/md0
> mdadm: /dev/sda is identified as a member of /dev/md0, slot 0.
> mdadm: /dev/sdb is identified as a member of /dev/md0, slot 1.
> mdadm: /dev/sdc is identified as a member of /dev/md0, slot 2.
...

and just for completeness:

 mdadm -E /dev/sdm /dev/sdaa 

Thanks.

NeilBrown

Re: MD or MDADM bug?

[David M. Strang]

Neil Brown wrote:
> On Friday September 2, dstrang@shellpower.net wrote:
> > >
> > > Can you run that with '-v' for me?
> > 
> > mdadm: looking for devices for /dev/md0
> > mdadm: /dev/sda is identified as a member of /dev/md0, slot 0.
> > mdadm: /dev/sdb is identified as a member of /dev/md0, slot 1.
> > mdadm: /dev/sdc is identified as a member of /dev/md0, slot 2.
> ...
> 
> and just for completeness:
> 
>  mdadm -E /dev/sdm /dev/sdaa 
> 

-(root@abyss)-(~)- # mdadm -E /dev/sdm /dev/sdaa
/dev/sdm:
          Magic : a92b4efc
        Version : 01.00
     Array UUID : 4e2b6b0a:8e92e91c:0c018a4b:f09bb74d
           Name : md/md0
  Creation Time : Wed Dec 31 19:00:00 1969
     Raid Level : raid5
   Raid Devices : 28

    Device Size : 143374632 (68.37 GiB 73.41 GB)
   Super Offset : 143374632 sectors
          State : clean
    Device UUID : 145bcf19:e4b3134a:3f312e60:06e44931
    Update Time : Wed Aug 31 22:23:26 2005
       Checksum : 2920539f - correct
         Events : 2645518

         Layout : left-asymmetric
     Chunk Size : 128K

   Array State : ____________________________ 28 failed
/dev/sdaa:
          Magic : a92b4efc
        Version : 01.00
     Array UUID : 4e2b6b0a:8e92e91c:0c018a4b:f09bb74d
           Name : md/md0
  Creation Time : Wed Dec 31 19:00:00 1969
     Raid Level : raid5
   Raid Devices : 28

    Device Size : 143374632 (68.37 GiB 73.41 GB)
   Super Offset : 143374632 sectors
          State : clean
    Device UUID : 1f4ab1e7:f8529f04:6c273e0a:510614ea
    Update Time : Wed Aug 31 18:57:23 2005
       Checksum : 15179289 - correct
         Events : 337384

         Layout : left-asymmetric
     Chunk Size : 128K

   Array State : uuuuuuuuuuuu_uuuuuuuuuuuuuuu 2 failed

Re: MD or MDADM bug?

[Neil Brown]

On Friday September 2, dstrang@shellpower.net wrote:
> Neil Brown wrote:
> > On Friday September 2, dstrang@shellpower.net wrote:
> > > >
> > > > Can you run that with '-v' for me?
> > > 
> > > mdadm: looking for devices for /dev/md0
> > > mdadm: /dev/sda is identified as a member of /dev/md0, slot 0.
> > > mdadm: /dev/sdb is identified as a member of /dev/md0, slot 1.
> > > mdadm: /dev/sdc is identified as a member of /dev/md0, slot 2.
> > ...
> > 
> > and just for completeness:
> > 
> >  mdadm -E /dev/sdm /dev/sdaa 
> > 
> 
> -(root@abyss)-(~)- # mdadm -E /dev/sdm /dev/sdaa

Thanks.

Looks like --assemble is not going to work any more for you.
However you should be able to recreate the array:

 mdadm -C /dev/md0 -l5 -n28 -c 128 --name=md/md0 -p la  /dev/sd[a-l] missing /dev/sd[n-z] /dev/sda[ab]

should get it right. (You did deliberately choose left-asymmetric I assume).

I've put a note on my todo list to test out these failure modes
and make sure it does the right thing next time.

NeilBrown

Re: MD or MDADM bug?

[David M. Strang]

Neil Brown wrote:
> On Friday September 2, dstrang@shellpower.net wrote:
> > Neil Brown wrote:
> > > On Friday September 2, dstrang@shellpower.net wrote:
> > > > >
> > > > > Can you run that with '-v' for me?
> > > > >
> > > > > mdadm: looking for devices for /dev/md0
> > > > mdadm: /dev/sda is identified as a member of /dev/md0, slot 0.
> > > > mdadm: /dev/sdb is identified as a member of /dev/md0, slot 1.
> > > > mdadm: /dev/sdc is identified as a member of /dev/md0, slot 2.
> > > ...
> > >
> > > and just for completeness:
> > >
> > >  mdadm -E /dev/sdm /dev/sdaa
> > >
> >
> > -(root@abyss)-(~)- # mdadm -E /dev/sdm /dev/sdaa
>
> Thanks.
>
> Looks like --assemble is not going to work any more for you.
> However you should be able to recreate the array:
>
>  mdadm -C /dev/md0 -l5 -n28 -c 128 --name=md/md0 -p la  /dev/sd[a-l] 
> missing /dev/sd[n-z] /dev/sda[ab]
>
> should get it right. (You did deliberately choose left-asymmetric I 
> assume).
>
> I've put a note on my todo list to test out these failure modes
> and make sure it does the right thing next time.

Does this mean I'm going to loose all my data?

-- David 

Re: MD or MDADM bug?

[Neil Brown]

On Friday September 2, dstrang@shellpower.net wrote:
> 
> Does this mean I'm going to loose all my data?
> 

No.
At least, you shouldn't, and doing the --create won't make anything
worse.

So do the --create with the 'missing', and don't add any spares.
Do a 'fsck' or whatever to check that everything is OK.

If it isn't, we'll have to look again at exactly what happened and
figure out which disks we should have created into the array.

NeilBrown

Re: MD or MDADM bug?

[David M. Strang]

Neil Brown wrote:
> On Friday September 2, dstrang@shellpower.net wrote:
> >
> > Does this mean I'm going to loose all my data?
> >
>
> No.
> At least, you shouldn't, and doing the --create won't make anything
> worse.
>
> So do the --create with the 'missing', and don't add any spares.
> Do a 'fsck' or whatever to check that everything is OK.
>
> If it isn't, we'll have to look again at exactly what happened and
> figure out which disks we should have created into the array.

-(root@abyss)-(~)- # mdadm -C /dev/md0 -l5 -n28 -c 128 --name=md/md0 -p la 
/dev/sd[a-l] missing /dev/sd[n-z] /dev/sda[ab]
mdadm: invalid number of raid devices: 28

I'm using mdadm 2.0 now --- should I try 2.0-devel-3 ?

-- David 

Re: MD or MDADM bug?

[Neil Brown]

On Friday September 2, dstrang@shellpower.net wrote:
> Neil Brown wrote:
> > On Friday September 2, dstrang@shellpower.net wrote:
> > >
> > > Does this mean I'm going to loose all my data?
> > >
> >
> > No.
> > At least, you shouldn't, and doing the --create won't make anything
> > worse.
> >
> > So do the --create with the 'missing', and don't add any spares.
> > Do a 'fsck' or whatever to check that everything is OK.
> >
> > If it isn't, we'll have to look again at exactly what happened and
> > figure out which disks we should have created into the array.
> 
> -(root@abyss)-(~)- # mdadm -C /dev/md0 -l5 -n28 -c 128 --name=md/md0 -p la 
> /dev/sd[a-l] missing /dev/sd[n-z] /dev/sda[ab]
> mdadm: invalid number of raid devices: 28

Sorry.  Add
   -e 1

> 
> I'm using mdadm 2.0 now --- should I try 2.0-devel-3 ?

No.  -devel-3 should not be used anymore.

NeilBrown

Re: MD or MDADM bug?

[David M. Strang]

Neil Brown wrote:
> On Friday September 2, dstrang@shellpower.net wrote:
> > Neil Brown wrote:
> > > On Friday September 2, dstrang@shellpower.net wrote:
> > > >
> > > > Does this mean I'm going to loose all my data?
> > > >
> > >
> > > No.
> > > At least, you shouldn't, and doing the --create won't make anything
> > > worse.
> > >
> > > So do the --create with the 'missing', and don't add any spares.
> > > Do a 'fsck' or whatever to check that everything is OK.
> > >
> > > If it isn't, we'll have to look again at exactly what happened and
> > > figure out which disks we should have created into the array.
> >
> > -(root@abyss)-(~)- # mdadm -C /dev/md0 -l5 -n28 -c 128 --name=md/md0 -p 
> > la
> > /dev/sd[a-l] missing /dev/sd[n-z] /dev/sda[ab]
> > mdadm: invalid number of raid devices: 28
>
> Sorry.  Add
>    -e 1

Well, I'm quite happy to report --- that worked!

###########
reiserfsck --check started at Fri Sep  2 12:29:18 2005
###########
Replaying journal..
Trans replayed: mountid 23, transid 172404, desc 3638, len 14, commit 3653, 
next trans offset 3636
Trans replayed: mountid 23, transid 172405, desc 3654, len 1, commit 3656, 
next trans offset 3639
Trans replayed: mountid 23, transid 172406, desc 3657, len 1, commit 3659, 
next trans offset 3642
Trans replayed: mountid 23, transid 172407, desc 3660, len 1, commit 3662, 
next trans offset 3645
Trans replayed: mountid 23, transid 172408, desc 3663, len 1, commit 3665, 
next trans offset 3648
Trans replayed: mountid 23, transid 172409, desc 3666, len 1, commit 3668, 
next trans offset 3651
Trans replayed: mountid 23, transid 172410, desc 3669, len 1, commit 3671, 
next trans offset 3654
Reiserfs journal '/dev/md0' in blocks [18..8211]: 7 transactions replayed
Checking internal tree..finished
Comparing bitmaps..finished
Checking Semantic tree:
finished
No corruptions found
There are on the filesystem:
        Leaves 279225
        Internal nodes 1696
        Directories 1962
        Other files 15922
        Data block pointers 280976644 (0 of them are zero)
        Safe links 0
###########
reiserfsck finished at Fri Sep  2 13:13:53 2005
###########

And, no data corruption!

So, once I get the bad drive replaced; and the array re-synced -- will I 
want to stop the array, and execute:

mdadm -C /dev/md0 -e1 -l5 -n28 -c 128 --name=md/md0 -p la /dev/sd[a-z] 
/dev/sda[ab]

Just so I don't have a problem with a disk 'not really' being part of the 
array? IE; mdadm: /dev/sdm is identified as a member of /dev/md0, slot -1.

Also, most of the drives -- have no partitions on them (ie; cfdisk 
/dev/sda) -- Can I add them and set the type to FD so it will autodetect the 
raid? Or must I do that prior to raid creation?

Thanks again for all the help Neil, once again -- I'm able to recover what 
seemed hopeless with zero dataloss.

-- David

Re: MD or MDADM bug?

[Neil Brown]

On Friday September 2, dstrang@shellpower.net wrote:
> >
> > Sorry.  Add
> >    -e 1
> 
> Well, I'm quite happy to report --- that worked!

Excellent!

> 
> So, once I get the bad drive replaced; and the array re-synced -- will I 
> want to stop the array, and execute:
> 
> mdadm -C /dev/md0 -e1 -l5 -n28 -c 128 --name=md/md0 -p la /dev/sd[a-z] 
> /dev/sda[ab]
> 
> Just so I don't have a problem with a disk 'not really' being part of the 
> array? IE; mdadm: /dev/sdm is identified as a member of /dev/md0,
> slot -1.

That shouldn't be necessary.  Providing you are using mdadm-2.0, you
should just be able to --add the drive and everything should work
fine.


> 
> Also, most of the drives -- have no partitions on them (ie; cfdisk 
> /dev/sda) -- Can I add them and set the type to FD so it will autodetect the 
> raid? Or must I do that prior to raid creation?

Type FD doesn't work with version-1 superblocks.  The kernel will not
auto-assemble them at all.  Just use mdadm to assemble them (in an
rc.d script).

NeilBrown

Re: MD or MDADM bug?

[Tyler]

Neil Brown wrote:

>On Friday September 2, dstrang@shellpower.net wrote:
>  
>
>>>Sorry.  Add
>>>   -e 1
>>>      
>>>
>>Well, I'm quite happy to report --- that worked!
>>    
>>
>
>Excellent!
>
>  
>
>>So, once I get the bad drive replaced; and the array re-synced -- will I 
>>want to stop the array, and execute:
>>
>>mdadm -C /dev/md0 -e1 -l5 -n28 -c 128 --name=md/md0 -p la /dev/sd[a-z] 
>>/dev/sda[ab]
>>
>>Just so I don't have a problem with a disk 'not really' being part of the 
>>array? IE; mdadm: /dev/sdm is identified as a member of /dev/md0,
>>slot -1.
>>    
>>
>
>That shouldn't be necessary.  Providing you are using mdadm-2.0, you
>should just be able to --add the drive and everything should work
>fine.
>
>  
>
>>Also, most of the drives -- have no partitions on them (ie; cfdisk 
>>/dev/sda) -- Can I add them and set the type to FD so it will autodetect the 
>>raid? Or must I do that prior to raid creation?
>>    
>>
>Type FD doesn't work with version-1 superblocks.  The kernel will not
>auto-assemble them at all.  Just use mdadm to assemble them (in an
>rc.d script).
>
>NeilBrown
>
Neil, is this something that will be changed in the future, where FD 
partition types will work with version 1 superblocks again at some point?

Tyler.

Re: MD or MDADM bug?

[Neil Brown]

On Saturday September 3, pml@dtbb.net wrote:
> >Type FD doesn't work with version-1 superblocks.  The kernel will not
> >auto-assemble them at all.  Just use mdadm to assemble them (in an
> >rc.d script).
> >
> >NeilBrown
> >
> Neil, is this something that will be changed in the future, where FD 
> partition types will work with version 1 superblocks again at some point?
> 
> Tyler.

No.
I've never liked kernel autodetect, and while I won't break it, I
would like to migrate people away from it.

What will, very likely, be implemented is something in 'mdadm' which
automatically detects and assembles md arrays.

The version-1 superblocks contains a 32byte 'name' field.
I imagine something like (but probably different from) putting
   host:array
in here, e.g.
   vger:mirror
   notabene:md2
   shellpower:scratch

You run
  mdadm -As --config=partitions --hostid=notabene

then it scans all partitions for md arrays, looks at those with
'notabene' as the hostname in the 'name' field, and assembles them,
creating /dev/$array (e.g. /dev/mirror, /dev/md2, /dev/scratch).
If the name looks like mdX, then the obvious minor is chosen.
If something else, then some unused minor is chosen.

Such a command would reliably assemble all arrays with any need for a
config file, and without any risk of getting confused if arrays are
imported from another machine (which is one of my issues with
autodetect).

Then putting such a command in an appropriate 'rc' script, in an
initrd image would make everything 'just work', just like in-kernel
auto-detect, only better.

The reason why it is 'probably different from' is that there are uses
like stacking of devices (raid0 on raid1) and creation of partitions
that need to be properly understood first.

I hope you find that acceptable.

NeilBrown

Re: MD or MDADM bug?

[danci]

On Sun, 4 Sep 2005, Neil Brown wrote:

> No.
> I've never liked kernel autodetect, and while I won't break it, I
> would like to migrate people away from it.
> 
> What will, very likely, be implemented is something in 'mdadm' which
> automatically detects and assembles md arrays.

How will we use MD for root filesystem? Will we *always* need a tricky 
initrd with mdadm added?

 D.

Re: MD or MDADM bug?

[Mario 'BitKoenig' Holbe]

danci@agenda.si <danci@agenda.si> wrote:
> On Sun, 4 Sep 2005, Neil Brown wrote:
>> I've never liked kernel autodetect, and while I won't break it, I
>> would like to migrate people away from it.
> How will we use MD for root filesystem? Will we *always* need a tricky 
> initrd with mdadm added?

If you don't (like to) have kernel autodetection, you'll always need
initrd with mdadm or something like that to boot up a MD for the root
fs, yes.


regards
   Mario
-- 
> As Luke Leighton said once on samba-ntdom, "now, what was that about
> rebooting?   that was so long ago, i had to look it up with man -k."

Re: MD or MDADM bug?

[Molle Bestefich]

Neil Brown wrote:
> No.
> I've never liked kernel autodetect, and while I won't break it, I
> would like to migrate people away from it.
[snippet]
> I hope you find that acceptable.

I won't go as far as to insinuate that I know what is acceptable and
what is not.
But I definitely like autodetection a lot.

I'm having troubling seeing why you'd like every MD user out there to
start fiddling with mdadm commands in their initrd when it can all be
done automatically behind the scenes.
I'm a big fan of letting computers do manual labour (especially MY
manual labour), and preferably doing it intelligently.

I hope you have the time to enlighten me! :-).

> What will, very likely, be implemented is something in 'mdadm' which
> automatically detects and assembles md arrays.

If that implies that all that distros have to do is put 'mdadm
--auto-assemble' into initrd, and it will do the same as kernel
autodetection does, I for one would be happy happy.

> You run
>   mdadm -As --config=partitions --hostid=notabene
> 
> then it scans all partitions for md arrays, looks at those with
> 'notabene' as the hostname in the 'name' field, and assembles them,

So if you change your hostname, you have to modify your initrd?
I fail to see what the added complexity might bring of good things to the table.

> Such a command would reliably assemble all arrays with any need for a
> config file, and without any risk of getting confused if arrays are
> imported from another machine (which is one of my issues with
> autodetect).

Who's confused?
MD or the user?

If the user is confused with the messages stating that h(is/er) array
cannot be assembled, the easy solution would be to stop swapping RAID
disks between boxes :-).

If MD is confused, then fix MD?
Could you elaborate?  (I do apologize if I'm the only one who's not
getting it..)

I heed the trouble if you take a 4-disk RAID6 and put two disks in one
box and 2 in another and start working on them, and thereafter put
them back together again.  If there's only a sequential counter to
tell which disks are fresh and which has to be rebuilt, the counters
could end up the same and cause MD to barf.  If that's the problem,
how about stuffing a 32-bit random number down there each time the
sequential counter is written, thus if the counters match but the
random number doesn't, the disks are from different machines and MD
should do a complete rebuild from whichever pair it chooses (or
perhaps better yet deny assembling them and let the user choose.  Or
whatever.).

(The counter could also be completely replaced by a random number, but
then you wouldn't know which disks is newer..)


> Then putting such a command in an appropriate 'rc' script, in an
> initrd image would make everything 'just work', just like in-kernel
> auto-detect, only better.

What's better about having to type more commands to get things done?
Still confused..


> The reason why it is 'probably different from' is that there are uses
> like stacking of devices (raid0 on raid1) and creation of partitions
> that need to be properly understood first.

The current implementation in MD needs a brush up in those areas too,
doesn't it?
Would something like

[pseudo]

// nFATLOAT: newlyFoundArraysThatLookOkAndThusShouldBeAssembled
MdDevice[] nFATLOAT = null;
do {
    if (nFATLOAT != null) assembleArrays(nFATLOAT);
    nFATLOAT = findUnassembledButOkArrays();
} while (nFATLOAT.length > 0);

[/pseudo]

work?

I'm curious about what the current status of layered MD device detection is!

Re: MD or MDADM bug?

[Luca Berra]

On Mon, Sep 05, 2005 at 06:45:30PM +0200, Molle Bestefich wrote:
>Neil Brown wrote:
>> No.
>> I've never liked kernel autodetect, and while I won't break it, I
>> would like to migrate people away from it.
>[snippet]
>> I hope you find that acceptable.
>
>I won't go as far as to insinuate that I know what is acceptable and
>what is not.
>But I definitely like autodetection a lot.

because it never bit you :-)

>I'm having troubling seeing why you'd like every MD user out there to
>start fiddling with mdadm commands in their initrd when it can all be
>done automatically behind the scenes.
>I'm a big fan of letting computers do manual labour (especially MY
>manual labour), and preferably doing it intelligently.
there is no reason for user to fiddle with md commands.
just have the command used to create initrd handle this, i have been
doing this for ages

>I hope you have the time to enlighten me! :-).
>
>> What will, very likely, be implemented is something in 'mdadm' which
>> automatically detects and assembles md arrays.
>
>If that implies that all that distros have to do is put 'mdadm
>--auto-assemble' into initrd, and it will do the same as kernel
>autodetection does, I for one would be happy happy.
it would actually work, versus autodetection which does not always.

>> You run
>>   mdadm -As --config=partitions --hostid=notabene
>> 
>> then it scans all partitions for md arrays, looks at those with
>> 'notabene' as the hostname in the 'name' field, and assembles them,
>
>So if you change your hostname, you have to modify your initrd?
no, it is just a tag, it is unrelated to the gethostname() syscall.

>I fail to see what the added complexity might bring of good things to the table.
you probably never had to deal with shared storage

>> Such a command would reliably assemble all arrays with any need for a
>> config file, and without any risk of getting confused if arrays are
>> imported from another machine (which is one of my issues with
>> autodetect).
>
>Who's confused?
>MD or the user?
autodetect is confused.

< big snip>
>
>I'm curious about what the current status of layered MD device detection is!
recent mdadm in the 1.xx series work flawlessly.
in kernel auto-detection, well....

in any case autodetection does not belong in the kernel.
it is far easier to implement and maintain in userspace.
the only reason why it has not been completely removed is backwards
compatibility.
if you don't want to use an initrd use the kernel command line.

Regards,
L.

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

Re: MD or MDADM bug?

[Neil Brown]

On Monday September 5, molle.bestefich@gmail.com wrote:
> Neil Brown wrote:
> > No.
> > I've never liked kernel autodetect, and while I won't break it, I
> > would like to migrate people away from it.
> [snippet]
> > I hope you find that acceptable.
> 
> I won't go as far as to insinuate that I know what is acceptable and
> what is not.
> But I definitely like autodetection a lot.

Have you head the lines by Henry Wadsworth Longfellow:	 
   There was a little girl
   Who had a little curl
   Right in the middle of her forehead;
   And when she was good
   She was very, very good,
   But when she was bad she was horrid.

That's how I feel about in-kernel-autodetection.  When it works, it's
fine.  When it doesn't, you lose.  Probably it hasn't happened to you.
The problem occurs if you move drives from one machine to another.  It
might transparently assemble the wrong array.

That's why I want to included the hostname in the info in the
superblock.  But you cannot be sure that the hostname will be
available in the kernel at any given time.  So the decision on exactly
how to assemble the array's has to be left to the system-integrator.
Just as the decision about which filesystem is the root filesystem.

> 
> I'm having troubling seeing why you'd like every MD user out there to
> start fiddling with mdadm commands in their initrd when it can all be
> done automatically behind the scenes.
> I'm a big fan of letting computers do manual labour (especially MY
> manual labour), and preferably doing it intelligently.
> 
> I hope you have the time to enlighten me! :-).
> 

It cannot be done with 100% reliability behind the scenes.  People
have reported problems with autodetect when moving drives from one
machine to another.


> > What will, very likely, be implemented is something in 'mdadm' which
> > automatically detects and assembles md arrays.
> 
> If that implies that all that distros have to do is put 'mdadm
> --auto-assemble' into initrd, and it will do the same as kernel
> autodetection does, I for one would be happy happy.
> 

That's generally the goal, yes.

> > You run
> >   mdadm -As --config=partitions --hostid=notabene
> > 
> > then it scans all partitions for md arrays, looks at those with
> > 'notabene' as the hostname in the 'name' field, and assembles them,
> 
> So if you change your hostname, you have to modify your initrd?

No.  hostid can be a kernel parameter.  Or it can be extracted from
the hardware somehow (MAC address).


> I fail to see what the added complexity might bring of good things to the table.
> 
> > Such a command would reliably assemble all arrays with any need for a
> > config file, and without any risk of getting confused if arrays are
> > imported from another machine (which is one of my issues with
> > autodetect).
> 
> Who's confused?
> MD or the user?

First MD - which of these two arrays should I assemble as md0?
and then the user - why didn't it assemble the right array just
   because I plugged a couple of extra drives in?

> 
> If the user is confused with the messages stating that h(is/er) array
> cannot be assembled, the easy solution would be to stop swapping RAID
> disks between boxes :-).

I have two boxed with raid arrays.  One dies.  I plug the drives
into the other and reboot it.  It comes up with some bits from one
machines and some bits from the other.

 
> If MD is confused, then fix MD?

I'm working on it, but removing providing a reliable alternative to
in-kernel autodetection.

> The current implementation in MD needs a brush up in those areas too,
> doesn't it?
> Would something like
> 
> [pseudo]
> 
> // nFATLOAT: newlyFoundArraysThatLookOkAndThusShouldBeAssembled
> MdDevice[] nFATLOAT = null;
> do {
>     if (nFATLOAT != null) assembleArrays(nFATLOAT);
>     nFATLOAT = findUnassembledButOkArrays();
> } while (nFATLOAT.length > 0);
> 
> [/pseudo]
> 
> work?
> 

Should a raid5 be assembled when you have found n-1 devices, or do you
wait for all n?  If you assemble as soon as you find n-1, what do you
do when the n'th  turns up?

I've had thoughts about allowing read-only assembly to which drives
can be added if they are found, but nothing concrete yet.

> I'm curious about what the current status of layered MD device detection is!

You put the entries in mdadm.conf in the right order and it should
work.

Why is it that people never complain about having to put information
in /etc/fstab about what to mount, but they cannot cope with having to
put similar information in /etc/mdadm.conf about what to assemble??

NeilBrown

Re: MD or MDADM bug?

[bart]

Neil,

Thanks for your explanation about the kernel autodetect pitfalls. IMHO having
a form of kernel space autodetect is an important feature in Linux. It allows
you to use a real mirror for the root filessystem. When assambling is done
based on root fs files this feature will be lost and the only alternative would
be hardware RAID again.

> 
> On Monday September 5, molle.bestefich@gmail.com wrote:
> > Neil Brown wrote:
> > > No.
> > > I've never liked kernel autodetect, and while I won't break it, I

<big snip>

> 
> Why is it that people never complain about having to put information
> in /etc/fstab about what to mount, but they cannot cope with having to
> put similar information in /etc/mdadm.conf about what to assemble??
> 
They would when mounting the root would depend on /etc/fstab :)

	Bart

Re: MD or MDADM bug?

[Molle Bestefich]

Neil Brown wrote:
> On Monday September 5, molle.bestefich@gmail.com wrote:
> > But I definitely like autodetection a lot.
> 
> Have you head the lines by Henry Wadsworth Longfellow:
>    There was a little girl
>    Who had a little curl
>    Right in the middle of her forehead;
>    And when she was good
>    She was very, very good,
>    But when she was bad she was horrid.

:-D.  Point taken.

> > > You run
> > >   mdadm -As --config=partitions --hostid=notabene
> > >
> > > then it scans all partitions for md arrays, looks at those with
> > > 'notabene' as the hostname in the 'name' field, and assembles them,
> >
> > So if you change your hostname, you have to modify your initrd?
> 
> No.  hostid can be a kernel parameter.  Or it can be extracted from
> the hardware somehow (MAC address).

Ah, ok!
So the concept is that only the arrays "belonging" to the box will be
automatically assembled.
That's a relief, thanks for the explanation.


> I have two boxed with raid arrays.  One dies.  I plug the drives
> into the other and reboot it.  It comes up with some bits from one
> machines and some bits from the other.

Hopefully it doesn't mix real bits from two arrays onto the same device.


> > The current implementation in MD needs a brush up in those areas too,
> > doesn't it?
> > Would something like
> >
> > [pseudo]
> >
> > // nFATLOAT: newlyFoundArraysThatLookOkAndThusShouldBeAssembled
> > MdDevice[] nFATLOAT = null;
> > do {
> >     if (nFATLOAT != null) assembleArrays(nFATLOAT);
> >     nFATLOAT = findUnassembledButOkArrays();
> > } while (nFATLOAT.length > 0);
> >
> > [/pseudo]
> >
> > work?
> >
> 
> Should a raid5 be assembled when you have found n-1 devices,
> or do you wait for all n?

Hm, that wasn't the point of the algorithm, the point was to
correctly (and non-infinitely-looping-ly) assemble layered MD
devices automatically.

To answer your question, I'd wait till the relevant bus(s)
has been scanned before assembling n-1 arrays.

(In the context of the above pseudo code, that wait would occur
 in findUnassembledButOkArrays()..)

> If you assemble as soon as you find n-1, what do you
> do when the n'th  turns up?

I'd consider that a hot-add event: if the other disks have not
been modified, just add the new disk as-is, otherwise kick off
the reconstruction logic.


> I've had thoughts about allowing read-only assembly to which drives
> can be added if they are found, but nothing concrete yet.

Sounded real good when I first read it, but now I'm having
difficulties deciding whether I like the idea or not :-).

(If the context of the feature is 'automatic assembly at boot-time',
 then all it would give is a small speed gain when booting.  At the
 expense of modifying a lot of software to be able to cope.. hmm)


> Why is it that people never complain about having to put information
> in /etc/fstab about what to mount, but they cannot cope with having to
> put similar information in /etc/mdadm.conf about what to assemble??

They've gotten a taste of the promised land.
You should've never given them autodetection in the first place :-).


Luca Berra wrote:
> in any case autodetection does not belong in the kernel.
> it is far easier to implement and maintain in userspace.

Yes, you're absolutely right.  Hadn't thought of that.

Re: MD or MDADM bug?

[berk walker]

Neil Brown wrote:
>Why is it that people never complain about having to put information
>in /etc/fstab about what to mount, but they cannot cope with having to
>put similar information in /etc/mdadm.conf about what to assemble??
>  
Maybe fstab already exists and is pretty readable, and from which a bit
of 'man' is easy.  Sorry, Neil, but to the "avg. user", mdadm.conf is a
different bird.  Personally, I'd like a GUI to do mdadm.conf, LILO,
GRUB, and rc*.

And, BTW, NLD9 needs tools.  Best to you-

b-