SW Raid doesn't play nice with devfs?

SW Raid doesn't play nice with devfs?

[Marc MERLIN]

Short version:
I have 3 disk arrays and I created several SW Raid 5 mds under 2.4.14 with
devfs, and remade them under 2.4.19 to see whether it helped.

All is well unless one of my disk arrays isn't powered when the system boots

Thanks to devfs, my drives still show up where they're supposed to be but md
tries to access the drives with the /dev/sd?1 names instead of 
/dev/scsi/host3/bus0/target?/lun0/part1


Longer version:
Just to make  sure, I remade the  array on top of itself  with the following
lines in /etc/raidtab.
raiddev /dev/md3             
        raid-level              5
        nr-raid-disks           7
        nr-spare-disks          0
        chunk-size              32
        persistent-superblock   1
        device                  /dev/scsi/host3/bus0/target8/lun0/part1
        raid-disk               0
        device                  /dev/scsi/host3/bus0/target9/lun0/part1
        raid-disk               1
        device                  /dev/scsi/host3/bus0/target10/lun0/part1
        raid-disk               2
        device                  /dev/scsi/host3/bus0/target11/lun0/part1
        raid-disk               3
        device                  /dev/scsi/host3/bus0/target12/lun0/part1
        raid-disk               4
        device                  /dev/scsi/host3/bus0/target13/lun0/part1
        raid-disk               5
        device                  /dev/scsi/host3/bus0/target14/lun0/part1
        raid-disk               6

Yet, if my other arrays are down  at boot time, /dev/md3 doesn't get created
at boot time, and a manual raidstart /dev/md3 yields:

kernel: md: could not lock sdm1, zero-size? Marking faulty.
kernel: md: could not import sdm1, trying to run array nevertheless.
kernel: md: could not lock sdn1, zero-size? Marking faulty.
kernel: md: could not import sdn1, trying to run array nevertheless.
kernel: md: could not lock sdo1, zero-size? Marking faulty.
kernel: md: could not import sdo1, trying to run array nevertheless.
kernel: md: could not lock sdp1, zero-size? Marking faulty.
kernel: md: could not import sdp1, trying to run array nevertheless.
kernel: md: could not lock sdq1, zero-size? Marking faulty.
kernel: md: could not import sdq1, trying to run array nevertheless.
kernel: md: could not lock sdr1, zero-size? Marking faulty.
kernel: md: could not import sdr1, trying to run array nevertheless.
kernel: md: could not lock sds1, zero-size? Marking faulty.
kernel: md: could not import sds1, trying to run array nevertheless.
kernel: md: autorun ...
kernel: md: considering scsi/host3/bus0/target8/lun0/part1 ...
kernel: md:  adding scsi/host3/bus0/target8/lun0/part1 ...
kernel: md: created md3
kernel: md: bind<scsi/host3/bus0/target8/lun0/part1,1>
kernel: md: running: <scsi/host3/bus0/target8/lun0/part1>
kernel: md: scsi/host3/bus0/target8/lun0/part1's event counter: 00000002
kernel: md: device name has changed from sdm1 to scsi/host3/bus0/target8/lun0/part1 since last import!
kernel: md3: former device sdn1 is unavailable, removing from array!
kernel: md3: former device sdo1 is unavailable, removing from array!
kernel: md3: former device sdp1 is unavailable, removing from array!
kernel: md3: former device sdq1 is unavailable, removing from array!
kernel: md3: former device sdr1 is unavailable, removing from array!
kernel: md3: former device sds1 is unavailable, removing from array!
kernel: md: md3: raid array is not clean -- starting background reconstruction
kernel: md3: max total readahead window set to 1488k
kernel: md3: 6 data-disks, max readahead per data-disk: 248k
kernel: raid5: device scsi/host3/bus0/target8/lun0/part1 operational as raid disk 0
kernel: raid5: not enough operational devices for md3 (6/7 failed)
kernel: RAID5 conf printout:
kernel:  --- rd:7 wd:1 fd:6
kernel:  disk 0, s:0, o:1, n:0 rd:0 us:1 dev:scsi/host3/bus0/target8/lun0/part1
kernel:  disk 1, s:0, o:0, n:1 rd:1 us:1 dev:[dev 00:00]
kernel:  disk 2, s:0, o:0, n:2 rd:2 us:1 dev:[dev 00:00]
kernel:  disk 3, s:0, o:0, n:3 rd:3 us:1 dev:[dev 00:00]
kernel:  disk 4, s:0, o:0, n:4 rd:4 us:1 dev:[dev 00:00]
kernel:  disk 5, s:0, o:0, n:5 rd:5 us:1 dev:[dev 00:00]
kernel:  disk 6, s:0, o:0, n:6 rd:6 us:1 dev:[dev 00:00]
kernel: raid5: failed to run raid set md3
kernel: md: pers->run() failed ...
kernel: md :do_md_run() returned -22
kernel: md: md3 stopped.
kernel: md: unbind<scsi/host3/bus0/target8/lun0/part1,0>
kernel: md: export_rdev(scsi/host3/bus0/target8/lun0/part1)

I've even tried making the symlinks:
gargamel:/dev# cat /tmp/script
ln -s scsi/host3/bus0/target8/lun0/part1 sdm1
ln -s scsi/host3/bus0/target9/lun0/part1 sdn1
ln -s scsi/host3/bus0/target10/lun0/part1 sdo1
ln -s scsi/host3/bus0/target11/lun0/part1 sdp1
ln -s scsi/host3/bus0/target12/lun0/part1 sdq1
ln -s scsi/host3/bus0/target13/lun0/part1 sdr1
ln -s scsi/host3/bus0/target14/lun0/part1 sds1
gargamel:/dev# bash /tmp/script 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking 
Home page: http://marc.merlins.org/   |   Finger marc_f@merlins.org for PGP key

Re: SW Raid doesn't play nice with devfs?

[Neil Brown]

On Sunday September 8, marc_news@merlins.org wrote:
...
> All is well unless one of my disk arrays isn't powered when the system boots
...
> Yet, if my other arrays are down  at boot time, /dev/md3 doesn't get created
> at boot time, and a manual raidstart /dev/md3 yields:

Yep, raidstart is broken by design.  It depends on device number being
stable, and SCSI device number aren't, especially if they are added
after boot time.

That is part of the reason that I wrote mdadm.  It can do the right
thing for you.

  http://www.cse.unsw.edu.au/~neilb/source/mdadm/

NeilBrown

Re: SW Raid doesn't play nice with devfs?

[Marc MERLIN]

On Mon, Sep 09, 2002 at 04:20:14PM +1000, Neil Brown wrote:
> Yep, raidstart is broken by design.  It depends on device number being
> stable, and SCSI device number aren't, especially if they are added
> after boot time.
> 
> That is part of the reason that I wrote mdadm.  It can do the right
> thing for you.
> 
>   http://www.cse.unsw.edu.au/~neilb/source/mdadm/

I will try this, thanks (I can't reboot the system this exact minute)

That said, the problem occurs both with the kernel autorun and raidstart:
when I load the qla1280 module, the drives get detected, md wakes up and
tries to create md3 since the partitions have type 0xfd, and fails in the
same way:
(...)
kernel:  /dev/scsi/host3/bus0/target13/lun0: p1
kernel: SCSI device sdl: 354600001 512-byte hdwr sectors
kernel:  /dev/scsi/host3/bus0/target14/lun0: p1
kernel:  [events: 00000002]
kernel: md: could not lock sdm1, zero-size? Marking faulty.
(...)

Is the kernel autorun code going to be fixed too or will I need to use mdadm
for the forseable future?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking 
Home page: http://marc.merlins.org/   |   Finger marc_f@merlins.org for PGP key

Re: mdadm -A /dev/md3 fails

[Marc MERLIN]

On Mon, Sep 09, 2002 at 04:20:14PM +1000, Neil Brown wrote:
> That is part of the reason that I wrote mdadm.  It can do the right
> thing for you.

Apparently it can, but I'm having issues with it:

My conf file says:
DEVICE /dev/scsi/host*/bus*/target*/lun*/part*
ARRAY /dev/md3 level=raid5 num-devices=7 UUID=7f69d69c:2d523219:4f3d1ea7:9b7f4057 devices=/dev/scsi/host3/bus0/target8/lun0/part1,/dev/scsi/host3/bus0/target9/lun0/part1,/dev/scsi/host3/bus0/target10/lun0/part1,/dev/scsi/host3/bus0/target11/lun0/part1,/dev/scsi/host3/bus0/target12/lun0/part1,/dev/scsi/host3/bus0/target13/lun0/part1,/dev/scsi/host3/bus0/target14/lun0/part1

gargamel:/etc/mdadm# mdadm -A  -v /dev/md3
mdadm: No identity information available for /dev/md3 - cannot assemble.

Any idea why 
mdadm -A -v /dev/md3 doesn't work?

Am I supposed to actually start the device with
mdadm -A  -v /dev/md3 /dev/scsi/host3/bus0/target*/lun0/part1 ?
or
mdadm -A  -v -u 7f69d69c:2d523219:4f3d1ea7:9b7f4057 /dev/md3 ?

(I thought mdadm would get that data from the config file)

Thanks
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking 
Home page: http://marc.merlins.org/   |   Finger marc_f@merlins.org for PGP key

Re: mdadm -A /dev/md3 fails

[Derek Vadala]

On Mon, 9 Sep 2002, Marc MERLIN wrote:

> gargamel:/etc/mdadm# mdadm -A  -v /dev/md3
> mdadm: No identity information available for /dev/md3 - cannot assemble.

Try mdadm -Avs /dev/md3

-s or --scan tells mdadm to use the config file.

or

mdadm -Avs to start all the arrays found in /etc/mdadm.conf

---
Derek Vadala, derek@cynicism.com, http://www.cynicism.com/~derek

Re: mdadm -A /dev/md3 fails

[Neil Brown]

On Monday September 9, marc_news@merlins.org wrote:
> On Mon, Sep 09, 2002 at 04:20:14PM +1000, Neil Brown wrote:
> > That is part of the reason that I wrote mdadm.  It can do the right
> > thing for you.
> 
> Apparently it can, but I'm having issues with it:
> 
> My conf file says:
> DEVICE /dev/scsi/host*/bus*/target*/lun*/part*
> ARRAY /dev/md3 level=raid5 num-devices=7 UUID=7f69d69c:2d523219:4f3d1ea7:9b7f4057 devices=/dev/scsi/host3/bus0/target8/lun0/part1,/dev/scsi/host3/bus0/target9/lun0/part1,/dev/scsi/host3/bus0/target10/lun0/part1,/dev/scsi/host3/bus0/target11/lun0/part1,/dev/scsi/host3/bus0/target12/lun0/part1,/dev/scsi/host3/bus0/target13/lun0/part1,/dev/scsi/host3/bus0/target14/lun0/part1
> 
> gargamel:/etc/mdadm# mdadm -A  -v /dev/md3
> mdadm: No identity information available for /dev/md3 - cannot assemble.
> 
> Any idea why 
> mdadm -A -v /dev/md3 doesn't work?

As Derek says, add "--scan" to tell it that it is allowed to look in
the config file.
Also, get rid of the "devices=...." part of the ARRAY line or it will
fail if any of the devices change address.

> 
> Am I supposed to actually start the device with
> mdadm -A  -v /dev/md3 /dev/scsi/host3/bus0/target*/lun0/part1 ?

That would probably work..

> or
> mdadm -A  -v -u 7f69d69c:2d523219:4f3d1ea7:9b7f4057 /dev/md3 ?

That wouldn't.  Would would need to to mdadm where the devices are:

mdadm -A  -v -u 7f69d69c:2d523219:4f3d1ea7:9b7f4057 /dev/md3 /dev/disk/*/part*

would work.

> 
> (I thought mdadm would get that data from the config file)

Only when you tell it to.  If that isn't clear from a second reading
of the man page, I would appreciate suggestions on how to make it more
clear.


NeilBrown

Re: SW Raid doesn't play nice with devfs?

[Neil Brown]

On Sunday September 8, marc_news@merlins.org wrote:
> 
> That said, the problem occurs both with the kernel autorun and raidstart:
> when I load the qla1280 module, the drives get detected, md wakes up and
> tries to create md3 since the partitions have type 0xfd, and fails in the
> same way:
> (...)
> kernel:  /dev/scsi/host3/bus0/target13/lun0: p1
> kernel: SCSI device sdl: 354600001 512-byte hdwr sectors
> kernel:  /dev/scsi/host3/bus0/target14/lun0: p1
> kernel:  [events: 00000002]
> kernel: md: could not lock sdm1, zero-size? Marking faulty.
> (...)
> 

I'm not sure what exactly is going on here, but it isn't regular
auto-detect. That *only* happens at boot time.  Never after module
load.

> Is the kernel autorun code going to be fixed too or will I need to use mdadm
> for the forseable future?

I use and recommend mdadm to all non-root devices.  For a raid
containing the root filesystem I use kernel parameters:
   md=0,/dev/xxxx,/dev/yyyy

though in 2.6, I suspect that initramfs will mean that mdadm is the
best choice for that too.  Don't do in kernel-space that which can
effectively and efficiently be done in user-space.

NeilBrown

Re: mdadm -A /dev/md3 fails

[Marc MERLIN]

I apologize for the really late answer, it looks like this message never
left my laptop.

On Mon, Sep 09, 2002 at 01:18:52AM -0700, Derek Vadala wrote:
> On Mon, 9 Sep 2002, Marc MERLIN wrote:
> 
> > gargamel:/etc/mdadm# mdadm -A  -v /dev/md3
> > mdadm: No identity information available for /dev/md3 - cannot assemble.
> 
> Try mdadm -Avs /dev/md3
> 
> -s or --scan tells mdadm to use the config file.

Yep, that worked fine, thanks.

On Mon, Sep 09, 2002 at 09:07:03PM +1000, Neil Brown wrote:
> On Monday September 9, marc_news@merlins.org wrote:
> > On Mon, Sep 09, 2002 at 04:20:14PM +1000, Neil Brown wrote:
> > > That is part of the reason that I wrote mdadm.  It can do the right
> > > thing for you.
> > 
> > Apparently it can, but I'm having issues with it:
> > 
> > My conf file says:
> > DEVICE /dev/scsi/host*/bus*/target*/lun*/part*
> > ARRAY /dev/md3 level=raid5 num-devices=7 UUID=7f69d69c:2d523219:4f3d1ea7:9b7f4057 devices=/dev/scsi/host3/bus0/target8/lun0/part1,/dev/scsi/host3/bus0/target9/lun0/part1,/dev/scsi/host3/bus0/target10/lun0/part1,/dev/scsi/host3/bus0/target11/lun0/part1,/dev/scsi/host3/bus0/target12/lun0/part1,/dev/scsi/host3/bus0/target13/lun0/part1,/dev/scsi/host3/bus0/target14/lun0/part1
> > 
> > gargamel:/etc/mdadm# mdadm -A  -v /dev/md3
> > mdadm: No identity information available for /dev/md3 - cannot assemble.
> > 
> > Any idea why 
> > mdadm -A -v /dev/md3 doesn't work?
> 
> As Derek says, add "--scan" to tell it that it is allowed to look in
> the config file.
> Also, get rid of the "devices=...." part of the ARRAY line or it will
> fail if any of the devices change address.
> 
> > 
> > Am I supposed to actually start the device with
> > mdadm -A  -v /dev/md3 /dev/scsi/host3/bus0/target*/lun0/part1 ?
> 
> That would probably work..
 
Yep, it works.
 
> > or
> > mdadm -A  -v -u 7f69d69c:2d523219:4f3d1ea7:9b7f4057 /dev/md3 ?
> 
> That wouldn't.  Would would need to to mdadm where the devices are:

Actuallly, it does work, that's why I listed it.
gargamel:/etc/mdadm# mdadm -A  -v -u 7f69d69c:2d523219:4f3d1ea7:9b7f4057 /dev/md3
mdadm: looking for devices for /dev/md3
mdadm: /dev/scsi/host3/bus0/target9/lun0/part1 is identified as a member of /dev/md3, slot 1.
mdadm: /dev/scsi/host3/bus0/target8/lun0/part1 is identified as a member of /dev/md3, slot 0.
mdadm: /dev/scsi/host3/bus0/target14/lun0/part1 is identified as a member of /dev/md3, slot 6.
mdadm: /dev/scsi/host3/bus0/target13/lun0/part1 is identified as a member of /dev/md3, slot 5.
mdadm: /dev/scsi/host3/bus0/target12/lun0/part1 is identified as a member of /dev/md3, slot 4.
mdadm: /dev/scsi/host3/bus0/target11/lun0/part1 is identified as a member of /dev/md3, slot 3.
mdadm: /dev/scsi/host3/bus0/target10/lun0/part1 is identified as a member of /dev/md3, slot 2.
mdadm: /dev/scsi/host0/bus0/target4/lun0/part3 has wrong uuid.
mdadm: /dev/scsi/host0/bus0/target4/lun0/part2 has wrong uuid.
mdadm: /dev/scsi/host0/bus0/target4/lun0/part1 has wrong uuid.
mdadm: /dev/scsi/host0/bus0/target3/lun0/part3 has wrong uuid.
mdadm: /dev/scsi/host0/bus0/target3/lun0/part2 has wrong uuid.
mdadm: /dev/scsi/host0/bus0/target3/lun0/part1 has wrong uuid.
mdadm: /dev/scsi/host0/bus0/target10/lun0/part3 has wrong uuid.
mdadm: /dev/scsi/host0/bus0/target10/lun0/part2 has wrong uuid.
mdadm: /dev/scsi/host0/bus0/target10/lun0/part1 has wrong uuid.
mdadm: /dev/scsi/host0/bus0/target1/lun0/part3 has wrong uuid.
mdadm: /dev/scsi/host0/bus0/target1/lun0/part2 has wrong uuid.
mdadm: /dev/scsi/host0/bus0/target1/lun0/part1 has wrong uuid.
mdadm: no RAID superblock on /dev/scsi/host0/bus0/target0/lun0/part4
mdadm: /dev/scsi/host0/bus0/target0/lun0/part4 has wrong uuid.
mdadm: /dev/scsi/host0/bus0/target0/lun0/part3 has wrong uuid.
mdadm: /dev/scsi/host0/bus0/target0/lun0/part2 has wrong uuid.
mdadm: /dev/scsi/host0/bus0/target0/lun0/part1 has wrong uuid.
mdadm: added /dev/scsi/host3/bus0/target9/lun0/part1 to /dev/md3 as 1
mdadm: added /dev/scsi/host3/bus0/target10/lun0/part1 to /dev/md3 as 2
mdadm: added /dev/scsi/host3/bus0/target11/lun0/part1 to /dev/md3 as 3
mdadm: added /dev/scsi/host3/bus0/target12/lun0/part1 to /dev/md3 as 4
mdadm: added /dev/scsi/host3/bus0/target13/lun0/part1 to /dev/md3 as 5
mdadm: added /dev/scsi/host3/bus0/target14/lun0/part1 to /dev/md3 as 6
mdadm: added /dev/scsi/host3/bus0/target8/lun0/part1 to /dev/md3 as 0
mdadm: /dev/md3 has been started with 7 drives.

> mdadm -A  -v -u 7f69d69c:2d523219:4f3d1ea7:9b7f4057 /dev/md3 /dev/disk/*/part*
> would work.
 
I think it worked  without because I already had a  DEVICES line that listed
that in the config file.

> > (I thought mdadm would get that data from the config file)
> 
> Only when you tell it to.  If that isn't clear from a second reading
> of the man page, I would appreciate suggestions on how to make it more
> clear.

It is there, although that didn't sync in my brain at my first readings.
I think  it's just  very counter  intuitive that you  have to  provide a
command line option to use the config file.
If I  put something  in the  config file, I  expect it  to be  picked up
without giving any special command line option.
Would you consider making mdadm read anything in mdadm.conf by default?

On Mon, Sep 09, 2002 at 09:11:18PM +1000, Neil Brown wrote:
> On Sunday September 8, marc_news@merlins.org wrote:
> > 
> > That said, the problem occurs both with the kernel autorun and raidstart:
> > when I load the qla1280 module, the drives get detected, md wakes up and
> > tries to create md3 since the partitions have type 0xfd, and fails in the
> > same way:
> > (...)
> > kernel:  /dev/scsi/host3/bus0/target13/lun0: p1
> > kernel: SCSI device sdl: 354600001 512-byte hdwr sectors
> > kernel:  /dev/scsi/host3/bus0/target14/lun0: p1
> > kernel:  [events: 00000002]
> > kernel: md: could not lock sdm1, zero-size? Marking faulty.
> > (...)
> 
> I'm not sure what exactly is going on here, but it isn't regular
> auto-detect. That *only* happens at boot time.  Never after module
> load.
 
Interesting it would be separate code. Thanks for the info.
 
> > Is the kernel autorun code going to be fixed too or will I need to use mdadm
> > for the forseable future?
> 
> I use and recommend mdadm to all non-root devices.  For a raid
> containing the root filesystem I use kernel parameters:
>    md=0,/dev/xxxx,/dev/yyyy

Good to know, thanks.

> though in 2.6, I suspect that initramfs will mean that mdadm is the
> best choice for that too.  Don't do in kernel-space that which can
> effectively and efficiently be done in user-space.

Right :-)

Thanks for the answers.
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking 
Home page: http://marc.merlins.org/   |   Finger marc_f@merlins.org for PGP key