/dev/.static/dev/md0 works - weird...

/dev/.static/dev/md0 works - weird...

[Mitchell Laks]

Now I try the following. 

I notice that on this sysem (which has that damn udev installed) 
/dev/md0 disappears on reboot.

So I see why mdadm --detail --scan came up with that weird 
/dev/.static/dev/md0, 

/dev/.static/dev/md0 at least this device persists after reboot:
that seems to stay around after reboot...


So I edit /etc/mdadm/mdadm.conf to read as follows

Device /dev/.static/dev/hdg1 /dev/.static/dev/hdb1
ARRAY /dev/md1 level=raid1 num-devices=2 
UUID=86af4f07:91fe306c:d1cb5c86:e87dc7de
   devices=/dev/.static/dev/hdg1
ARRAY /dev/md0 level=raid1 num-devices=2 
UUID=19def19e:3995fe1e:5470d62a:c44b069c
   devices=/dev/.static/dev/hdb1

and  now I reboot the system.

And now if I run 

mdadm -A --run  /dev/.static/dev/md0 /dev/hdb1
it starts it running. and i can mount it.
mount -t ext3 /dev/.static/dev/md0  /big1


So I think that I must create a init.d script to run at the end of the boot to 
do

mdadm -A --run  /dev/.static/dev/md0 /dev/hdb1
mount -t ext3 /dev/.static/dev/md0  /big1

and then I won't have problems....

However I think that raids should boot as long as they are intact, as a matter 
of policy. Otherwise we lose our  ability to rely upon them for remote 
servers...

Mitchell Laks

RAID1 assembly requires manual "mdadm --run"

[Molle Bestefich]

Mitchell Laks wrote:
> However I think that raids should boot as long as they are intact, as a matter
> of policy. Otherwise we lose our  ability to rely upon them for remote
> servers...

It does seem wrong that a RAID 5 starts OK with a disk missing, but a
RAID 1 fails.

Perhaps MD is unable to tell which disk in the RAID 1 is the freshest
and therefore refuses to assemble any RAID 1's with disks missing?

Re: RAID1 assembly requires manual "mdadm --run"

[Neil Brown]

On Thursday July 7, molle.bestefich@gmail.com wrote:
> Mitchell Laks wrote:
> > However I think that raids should boot as long as they are intact, as a matter
> > of policy. Otherwise we lose our  ability to rely upon them for remote
> > servers...
> 
> It does seem wrong that a RAID 5 starts OK with a disk missing, but a
> RAID 1 fails.
> 
> Perhaps MD is unable to tell which disk in the RAID 1 is the freshest
> and therefore refuses to assemble any RAID 1's with disks missing?

This doesn't sound right at all.

"--run" is required to start a degraded array as a way of confirming
to mdadm that you really have listed all the drives available.
The normal way of starting an array at boot time is by describing the
array (usually by UUID) in mdadm.conf and letting mdadm find the
component devices with "mdadm --assemble --scan".

This usage does not require --run.

The only time there is a real reluctance to start a degraded array is
when it is raid5/6 and it suffered an unclean shutdown.
A dirty, degraded raid5/6 can have undetectably data corruption, and I
really want you to be aware of that and not just assume that "because
it started, the data must be OK".

NeilBrown

Re: RAID1 assembly requires manual "mdadm --run"

[Molle Bestefich]

On 7/8/05, Neil Brown <neilb@cse.unsw.edu.au> wrote:
> On Thursday July 7, molle.bestefich@gmail.com wrote:
> > Mitchell Laks wrote:
> > > However I think that raids should boot as long as they are intact, as a matter
> > > of policy. Otherwise we lose our  ability to rely upon them for remote
> > > servers...
> >
> > It does seem wrong that a RAID 5 starts OK with a disk missing, but a
> > RAID 1 fails.
> >
> > Perhaps MD is unable to tell which disk in the RAID 1 is the freshest
> > and therefore refuses to assemble any RAID 1's with disks missing?
> 
> This doesn't sound right at all.
> 
> "--run" is required to start a degraded array as a way of confirming
> to mdadm that you really have listed all the drives available.
> The normal way of starting an array at boot time is by describing the
> array (usually by UUID) in mdadm.conf and letting mdadm find the
> component devices with "mdadm --assemble --scan".
> 
> This usage does not require --run.
> 
> The only time there is a real reluctance to start a degraded array is
> when it is raid5/6 and it suffered an unclean shutdown.
> A dirty, degraded raid5/6 can have undetectably data corruption, and I
> really want you to be aware of that and not just assume that "because
> it started, the data must be OK".

Sounds very sane.

So a clean RAID1 with a disk missing should start without --run, just
like a clean RAID5 with a disk missing?

Nevermind, I'll try to reproduce it instead of asking too many questions.
And I suck a bit at testing MD with loop devices, so if someone could
pitch in and tell me what I'm doing wrong here, I'd appreciate it very
much (-:

# mknod /dev/md0 b 9 0
# dd if=/dev/zero of=test1 bs=1M count=100
# dd if=/dev/zero of=test2 bs=1M count=100
# dd if=/dev/zero of=test3 bs=1M count=100
# losetup /dev/loop1 test1
# losetup /dev/loop2 test2
# losetup /dev/loop3 test3
# mdadm --create /dev/md0 -l 1 -n 3 /dev/loop1 /dev/loop2 /dev/loop3
mdadm: array /dev/md0 started.

# mdadm --detail --scan > /etc/mdadm.conf
# cat /etc/mdadm.conf
ARRAY /dev/md0 level=raid1 num-devices=3
   UUID=1dcc972f:0b856580:05c66483:e14940d8
   devices=/dev/loop/1,/dev/loop/2,/dev/loop/3

# mdadm --stop /dev/md0
# mdadm --assemble --scan
mdadm: no devices found for /dev/md0

// ^^^^^^^^^^^^^^^^^^^^^^^^  ??? Why?

# mdadm --assemble /dev/md0 /dev/loop1 /dev/loop2 /dev/loop3
mdadm: /dev/md0 has been started with 3 drives.

// So far so good..

# mdadm --stop /dev/md0
# losetup -d /dev/loop3
# mdadm --assemble /dev/md0 /dev/loop1 /dev/loop2 /dev/loop3
mdadm: no RAID superblock on /dev/loop7
mdadm: /dev/loop7 has no superblock - assembly aborted

// ^^^^^^^^^^^^^^^^^^^^^^^^  ??? It aborts :-(...
// Doesn't an inactive loop device seem the same as a missing disk to MD?

# rm -f /dev/loop3
# mdadm --assemble /dev/md0 /dev/loop1 /dev/loop2 /dev/loop3
mdadm: cannot open device /dev/loop7: No such file or directory
mdadm: /dev/loop7 has no superblock - assembly aborted

// ^^^^^^^^^^^^^^^^^^^^^^^^  ??? It aborts, just as above...

Hm!

Re: RAID1 assembly requires manual "mdadm --run"

[Tyler]

Molle Bestefich wrote:

>On 7/8/05, Neil Brown <neilb@cse.unsw.edu.au> wrote:
>  
>
>>On Thursday July 7, molle.bestefich@gmail.com wrote:
>>    
>>
>>>Mitchell Laks wrote:
>>>      
>>>
>>>>However I think that raids should boot as long as they are intact, as a matter
>>>>of policy. Otherwise we lose our  ability to rely upon them for remote
>>>>servers...
>>>>        
>>>>
>>>It does seem wrong that a RAID 5 starts OK with a disk missing, but a
>>>RAID 1 fails.
>>>
>>>Perhaps MD is unable to tell which disk in the RAID 1 is the freshest
>>>and therefore refuses to assemble any RAID 1's with disks missing?
>>>      
>>>
>>This doesn't sound right at all.
>>
>>"--run" is required to start a degraded array as a way of confirming
>>to mdadm that you really have listed all the drives available.
>>The normal way of starting an array at boot time is by describing the
>>array (usually by UUID) in mdadm.conf and letting mdadm find the
>>component devices with "mdadm --assemble --scan".
>>
>>This usage does not require --run.
>>
>>The only time there is a real reluctance to start a degraded array is
>>when it is raid5/6 and it suffered an unclean shutdown.
>>A dirty, degraded raid5/6 can have undetectably data corruption, and I
>>really want you to be aware of that and not just assume that "because
>>it started, the data must be OK".
>>    
>>
>
>Sounds very sane.
>
>So a clean RAID1 with a disk missing should start without --run, just
>like a clean RAID5 with a disk missing?
>
>Nevermind, I'll try to reproduce it instead of asking too many questions.
>And I suck a bit at testing MD with loop devices, so if someone could
>pitch in and tell me what I'm doing wrong here, I'd appreciate it very
>much (-:
>
># mknod /dev/md0 b 9 0
># dd if=/dev/zero of=test1 bs=1M count=100
># dd if=/dev/zero of=test2 bs=1M count=100
># dd if=/dev/zero of=test3 bs=1M count=100
># losetup /dev/loop1 test1
># losetup /dev/loop2 test2
># losetup /dev/loop3 test3
># mdadm --create /dev/md0 -l 1 -n 3 /dev/loop1 /dev/loop2 /dev/loop3
>mdadm: array /dev/md0 started.
>
># mdadm --detail --scan > /etc/mdadm.conf
># cat /etc/mdadm.conf
>ARRAY /dev/md0 level=raid1 num-devices=3
>   UUID=1dcc972f:0b856580:05c66483:e14940d8
>   devices=/dev/loop/1,/dev/loop/2,/dev/loop/3
>  
>
Why does this show /dev/loop/1 instead of /dev/loop1 ?

># mdadm --stop /dev/md0
># mdadm --assemble --scan
>mdadm: no devices found for /dev/md0
>
>// ^^^^^^^^^^^^^^^^^^^^^^^^  ??? Why?
>
># mdadm --assemble /dev/md0 /dev/loop1 /dev/loop2 /dev/loop3
>mdadm: /dev/md0 has been started with 3 drives.
>
>// So far so good..
>
># mdadm --stop /dev/md0
># losetup -d /dev/loop3
># mdadm --assemble /dev/md0 /dev/loop1 /dev/loop2 /dev/loop3
>mdadm: no RAID superblock on /dev/loop7
>mdadm: /dev/loop7 has no superblock - assembly aborted
>  
>
Where's loop7 coming from all of a sudden? I thought you were using 
loop1, loop2, loop3

>// ^^^^^^^^^^^^^^^^^^^^^^^^  ??? It aborts :-(...
>// Doesn't an inactive loop device seem the same as a missing disk to MD?
>
># rm -f /dev/loop3
># mdadm --assemble /dev/md0 /dev/loop1 /dev/loop2 /dev/loop3
>mdadm: cannot open device /dev/loop7: No such file or directory
>mdadm: /dev/loop7 has no superblock - assembly aborted
>  
>
Once again, where is loop7 coming from?

>// ^^^^^^^^^^^^^^^^^^^^^^^^  ??? It aborts, just as above...
>
>Hm!
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>  
>

Re: RAID1 assembly requires manual "mdadm --run"

[Neil Brown]

On Friday July 8, molle.bestefich@gmail.com wrote:
> 
> So a clean RAID1 with a disk missing should start without --run, just
> like a clean RAID5 with a disk missing?

Anything degraded needs at least --run or --scan.

> 
> Nevermind, I'll try to reproduce it instead of asking too many questions.
> And I suck a bit at testing MD with loop devices, so if someone could
> pitch in and tell me what I'm doing wrong here, I'd appreciate it very
> much (-:
> 
> # mknod /dev/md0 b 9 0
> # dd if=/dev/zero of=test1 bs=1M count=100
> # dd if=/dev/zero of=test2 bs=1M count=100
> # dd if=/dev/zero of=test3 bs=1M count=100
> # losetup /dev/loop1 test1
> # losetup /dev/loop2 test2
> # losetup /dev/loop3 test3
> # mdadm --create /dev/md0 -l 1 -n 3 /dev/loop1 /dev/loop2 /dev/loop3
> mdadm: array /dev/md0 started.
> 
> # mdadm --detail --scan > /etc/mdadm.conf
> # cat /etc/mdadm.conf
> ARRAY /dev/md0 level=raid1 num-devices=3
>    UUID=1dcc972f:0b856580:05c66483:e14940d8
>    devices=/dev/loop/1,/dev/loop/2,/dev/loop/3

/dev/loop/N is reported instead of /dev/loopN because mdadm only has
major/minor device numbers to go on and scans /dev, returning the
first name it finds. 1.12.0 returns the shortest name it finds.


> 
> # mdadm --stop /dev/md0
> # mdadm --assemble --scan
> mdadm: no devices found for /dev/md0
> 
> // ^^^^^^^^^^^^^^^^^^^^^^^^  ??? Why?

The ARRAY line doesn't tell mdadm where to find the array.  It only
tells mdadm how to recognised the components when they are found(*).
The DEVICES line (which you don't have) tells mdadm where to find
components.
So:
  echo 'DEVICES /dev/loop?' > /etc/mdadm.conf
  mdadm --detail --scan >> /etc/mdadm.conf
  mdadm --stop /dev/md0
  mdadm --assemble --scan
should work

> 
> # mdadm --assemble /dev/md0 /dev/loop1 /dev/loop2 /dev/loop3
> mdadm: /dev/md0 has been started with 3 drives.
> 
> // So far so good..
> 
> # mdadm --stop /dev/md0
> # losetup -d /dev/loop3
> # mdadm --assemble /dev/md0 /dev/loop1 /dev/loop2 /dev/loop3
> mdadm: no RAID superblock on /dev/loop7
> mdadm: /dev/loop7 has no superblock - assembly aborted
> 
> // ^^^^^^^^^^^^^^^^^^^^^^^^  ??? It aborts :-(...
> // Doesn't an inactive loop device seem the same as a missing disk
> to MD?

I assume that is loop3, not loop7 ??

You have asked mdadm to assemble /dev/loop1 /dev/loop2 and /dev/loop3
into an array.  But it can tell that they are not all parts of the
same array (infact loop3 isn't part of an array at all).
The only parts of the array that you currently have are on /dev/loop1
and /dev/loop2. So you want to

  mdadm --assemble /dev/md0 --run /dev/loop1 /dev/loop2
(you need --run to convince mdadm that you know what you are doing).
Not that with /dev/loop3 no funcitoning,
  mdadm --assemble --scan
will still work.

NeilBrown


(*)
The mdadm.conf entry
> ARRAY /dev/md0 level=raid1 num-devices=3
>    UUID=1dcc972f:0b856580:05c66483:e14940d8
>    devices=/dev/loop/1,/dev/loop/2,/dev/loop/3

tells mdadm:
  You can recognise components of /dev/md0 as they have a raid
  superblock which records the level as 'raid1', the number of devices
  as '3', the UUID as 1dcc972f:0b856580:05c66483:e14940d8, and they
  will only be on one of the devices /dev/loop/1, /dev/loop/2, or
  /dev/loop/3.

Normally you will not want the 'devices=' section and mdadm 1.12.0
won't report if unless you give a '-v'.

NeilBrown

Re: RAID1 assembly requires manual "mdadm --run"

[Molle Bestefich]

Tyler wrote:
> Molle Bestefich wrote:
>> Neil Brown wrote:
>>> Molle Bestefich wrote:
>>>> Mitchell Laks wrote:
>>>>> However I think that raids should boot as long as they are
intact, as a matter
>>>>> of policy. Otherwise we lose our  ability to rely upon them for remote
>>>>> servers...
>>>>>
>>>>>
>>>> It does seem wrong that a RAID 5 starts OK with a disk missing, but a
>>>> RAID 1 fails.
>>>>
>>>> Perhaps MD is unable to tell which disk in the RAID 1 is the freshest
>>>> and therefore refuses to assemble any RAID 1's with disks missing?
>>>>
>>>>
>>> This doesn't sound right at all.
>>>
>>> "--run" is required to start a degraded array as a way of confirming
>>> to mdadm that you really have listed all the drives available.
>>> The normal way of starting an array at boot time is by describing the
>>> array (usually by UUID) in mdadm.conf and letting mdadm find the
>>> component devices with "mdadm --assemble --scan".
>>>
>>> This usage does not require --run.
>>>
>>> The only time there is a real reluctance to start a degraded array is
>>> when it is raid5/6 and it suffered an unclean shutdown.
>>> A dirty, degraded raid5/6 can have undetectably data corruption, and I
>>> really want you to be aware of that and not just assume that "because
>>> it started, the data must be OK".
>>>
>>>
>>
>> Sounds very sane.
>>
>> So a clean RAID1 with a disk missing should start without --run, just
>> like a clean RAID5 with a disk missing?
>>
>> Nevermind, I'll try to reproduce it instead of asking too many questions.
>> And I suck a bit at testing MD with loop devices, so if someone could
>> pitch in and tell me what I'm doing wrong here, I'd appreciate it very
>> much (-:
>>
>> # mknod /dev/md0 b 9 0
>> # dd if=/dev/zero of=test1 bs=1M count=100
>> # dd if=/dev/zero of=test2 bs=1M count=100
>> # dd if=/dev/zero of=test3 bs=1M count=100
>> # losetup /dev/loop1 test1
>> # losetup /dev/loop2 test2
>> # losetup /dev/loop3 test3
>> # mdadm --create /dev/md0 -l 1 -n 3 /dev/loop1 /dev/loop2 /dev/loop3
>> mdadm: array /dev/md0 started.
>>
>> # mdadm --detail --scan > /etc/mdadm.conf
>> # cat /etc/mdadm.conf
>> ARRAY /dev/md0 level=raid1 num-devices=3
>>    UUID=1dcc972f:0b856580:05c66483:e14940d8
>>    devices=/dev/loop/1,/dev/loop/2,/dev/loop/3
>>
>>
> Why does this show /dev/loop/1 instead of /dev/loop1 ?

Hm!  Didn't spot that.
This machine is using udev, so /dev/loopx is a symbolic link to /dev/loop/x.
I haven't got a clue why DM resolves the link..

But since mdadm --assemble --scan doesn't seem to work correctly using
this example, and I'm punching in loop device names anyway, it's less
important..

>> # mdadm --stop /dev/md0
>> # mdadm --assemble --scan
>> mdadm: no devices found for /dev/md0
>>
>> // ^^^^^^^^^^^^^^^^^^^^^^^^  ??? Why?
>>
>> # mdadm --assemble /dev/md0 /dev/loop1 /dev/loop2 /dev/loop3
>> mdadm: /dev/md0 has been started with 3 drives.
>>
>> // So far so good..
>>
>> # mdadm --stop /dev/md0
>> # losetup -d /dev/loop3
>> # mdadm --assemble /dev/md0 /dev/loop1 /dev/loop2 /dev/loop3
>> mdadm: no RAID superblock on /dev/loop7
>> mdadm: /dev/loop7 has no superblock - assembly aborted
>>
>>
> Where's loop7 coming from all of a sudden? I thought you were using
> loop1, loop2, loop3

Ehehe.. Oops.
I redid the test with different loop devices, just to be sure.

Copy/pasted from the wrong place in the terminal window.
Sorry for the confusion.  Just imagine that the 7 is really a 3 :-).

>> // ^^^^^^^^^^^^^^^^^^^^^^^^  ??? It aborts :-(...
>> // Doesn't an inactive loop device seem the same as a missing disk to MD?
>>
>> # rm -f /dev/loop3
>> # mdadm --assemble /dev/md0 /dev/loop1 /dev/loop2 /dev/loop3
>> mdadm: cannot open device /dev/loop7: No such file or directory
>> mdadm: /dev/loop7 has no superblock - assembly aborted
>>
>>
> Once again, where is loop7 coming from?

Same as above.

Re: RAID1 assembly requires manual "mdadm --run"

[Molle Bestefich]

Neil Brown wrote:
> On Friday July 8, molle.bestefich@gmail.com wrote:
> > So a clean RAID1 with a disk missing should start without --run, just
> > like a clean RAID5 with a disk missing?

> Not that with /dev/loop3 no funcitoning,
>   mdadm --assemble --scan
> will still work.

Super!
That was exactly the point of the test.

You're correct, when I try again with a DEVICE line and --assemble
--scan, everything works perfectly.

(Not sure why Mitchell Laks saw something else happen.)

Thanks a lot for taking the time to answer questions, the information
is much appreciated!