Can't mount /dev/md0 after stopping a synchronization

Can't mount /dev/md0 after stopping a synchronization

[Mike Garey]

I'm in the process of setting up a software RAID level 1 on debian
testing with two 160 gig drives on an Asus P4P800 motherboard with
Intel Pentium 4 CPU 3.00GHz and 512 megs of RAM.  I've been following
the instructions in /usr/share/doc/mdadm/rootraiddoc97.html, but after
rebooting to /dev/hdc1 (/dev/md0) and issuing the command "mdadm --add
/dev/md0 /dev/hda1", I was only getting 1636K/sec for the
synchronization, which was _way_ too slow (I forgot I had left a CDROM
drive on the secondary IDE channel, so /dev/hdc1 was running at UDMA2,
plus, I was using a 40 conductor cable instead of 80).  So I decided
to reboot the system and cancel the synchronization, thus destroying
everything on /dev/hda1 (but this didn't seem like an issue, since I
was able to boot from /dev/md0 aka /dev/hdc1).

So after stopping the sync process and rebooting, I was unable to boot
from hdc1 because I had forgotten to copy over my initrd with md/raid1
modules from /dev/hda1 to /dev/hdc1, and of course now /dev/hda1 is
destroyed, so I can't use the initrd-RAID that was previously on there
(and the initrd-RAID never existed on /dev/hdc1, since I created it on
/dev/hda1 _after_ I had issued the "mount /dev/md0 /mnt/md0; cp -axu /
/mnt/md0" commands - don't ask why).  So when I rebooted, I didn't
have md/raid1 support, and so /dev/md0 couldn't mount.. I ended up
building a new initrd with md/raid1 on another debian box, then
copying them to /dev/hdc1.  I've now tried installing grub on
/dev/hdc, disconnected /dev/hda, and attempted to boot.  I now get the
following message:

md: md driver 0.90.3 MAX_MD_DEVS=3D256, MD_SB_DISKS=3D27
md: bitmap version 4.39
md: raid1 personality registered as nr3
.
.
Begin: Mounting root file system...
Begin: Running /scripts/local-top
Done.
ALERT! /dev/md0 does not exist.  Dropping to a shell!

At which point I get dropped into busybox..  One thing I notice is
that although it shows the md module being loaded, it doesn't really
say anything about configuring/adding any RAID disks (which is used to
say, back when I had first booted into /dev/md0 (hdc1) before killing
/dev/hda1).  So now apparently /dev/md0 no longer exists, and I have
no idea how to create it.

In fstab on /dev/hcd1, I have:

/dev/md0 / ext3 defaults 0 0

and in /boot/grub/menu.lst I have:

title           Debian GNU/Linux, kernel 2.6.15-486 RAID
root            (hd0,0)
kernel          /boot/vmlinuz-2.6.15-486 root=3D/dev/md0 ro
initrd          /boot/initrd.img-2.6.15-486-RAID
savedefault
boot

Now, /dev/md0 was originally created by using:

mdadm --create /dev/md0 --level=3D1 --raid-disks=3D2 missing /dev/hdc1

can anybody tell me why /dev/md0 is no longer available on /dev/hdc1,
and if possible how to recreate it?

I should mention that after I cancelled the synchronization process,
and was unable to reboot into /dev/hdc1, I then mounted /dev/hdc1
through busybox, changed the Type of the disk from FD (linux raid) to
83 (linux), modified /etc/fstab and /boot/grub/menu.lst so it pointed
to /dev/hdc1 instead of /dev/md0.  I tried rebooting after changing
these options, and it gave me a message saying "Failed to mount root
file system" or something to that effect. Could changing the Type of
the disk or mounting /dev/hdc1 have caused /dev/md0 to become
corrupted/deleted?

If anybody has the time to read through my message
and give me some advice, I would very much appreciate it.  Thanks,

Mike

Re: Can't mount /dev/md0 after stopping a synchronization

[Mike Garey]

Just wanted to add a few more details/questions to my previous post..

In case I provided too much information in my previous email, here's a
condensed version:

1) got to "step 6.2 Add the first-disk to our existing RAID device" of
the rootraiddoc walkthrough
2) issued command "mdadm --add /dev/md0 /dev/hda1"
3) after getting terrible throughput, decided to stop the
synchronization process and reboot
4) failed to reboot from /dev/md0 (hdc1)
5) hda1 is now hosed, hdc1 contains a complete working copy of the
previous contents of /dev/hda1, but I'm unable to boot from it
("ALERT! /dev/md0 does not exist" message)
6) re-installed debian onto hda1 and rebooted from it.

Now, it seems as though I'm sort of back at "step 3.3 Create RAID
device".  I've tried to do the following while booted from my clean
debian install on /dev/hda1:

mkdir /mnt/md0
mount /dev/md0 /mnt/md0

but I get the message "mount: you must specify the filesystem type",
because I guess the system has no idea what /dev/md0 is.  So then I
tried the following (possibly risky?) command:

mdadm --create /dev/md0 --level=1 --raid-disks=2 --spare-devices 0
missing /dev/hdc1

but I get the message:

mdadm: /dev/hdc1 appears to contain an ext2fs file system
    size=158577472K  mtime=Tue Apr  4 18:21:09 2006
mdadm: /dev/hdc1 appears to be part of a raid array:
    level=1 devices=2 ctime=Mon Apr  3 13:41:33 2006
Continue creating array? n
mdadm: create aborted

It seems from the above message that mdadm --create would destroy all
the data on /dev/hdc1 (someone correct me if I'm wrong), rather than
simply creating /dev/md0.  If this is the case, should I be using
mdadm --assemble instead? (but if so, I need to assemble to a
pre-existing md device, in which case I don't have one)  What I'd like
to do is be able to mount /dev/hdc1 as /dev/md0 (as in "step 4.2 Mount
your RAID device."), and then hopefully be able to boot off of it, but
right now I'm confused about how to create /dev/md0 without destroying
any of my information on either /dev/hda1 or /dev/hdc1 (of course,
after the reboot, I don't mind destroying /dev/hda1).

the other alternative is of course to zero the superblocks on
/dev/hdc1, reboot from it as a normal drive, and then follow the
walkthrough from the beginning, but this time starting with /dev/hdc1
as the master drive.  I'd like to avoid this if possible, since I'd
really like to find out why I can't boot from /dev/md0 (hdc1), since
if this was not a test, and /dev/hda1 really did fail, I'd be in a
mess.  So I'd prefer to know how to recover from this, since it seems
that the position I'm in right now is essentiallly the same scenario
as if /dev/hda1 had failed, in which case I'd like to be prepared.

If anyone can make heads or tails of what I'm talking about, I would
greatly appreciate any information or suggestions.  Thanks in advance,

Mike

P.S. output of mdadm --examine is given below (output is the same for
/dev/hda1 and /dev/hdc1)

          Magic : a92b4efc
        Version : 00.90.03
           UUID : f1ab0f6d:6c5c4bf6:228dec79:3aa3582f
  Creation Time : Mon Apr  3 13:41:33 2006
     Raid Level : raid1
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Tue Apr  4 09:37:10 2006
          State : clean
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1
       Checksum : f63c0eb5 - correct
         Events : 0.19314


      Number   Major   Minor   RaidDevice State
this     1      22        1        1      active sync   /dev/hdc1

   0     0       0        0        0      removed
   1     1      22        1        1      active sync   /dev/hdc1
   2     2       3        1        2      spare   /dev/hda1


> I'm in the process of setting up a software RAID level 1 on debian
> testing with two 160 gig drives on an Asus P4P800 motherboard with
> Intel Pentium 4 CPU 3.00GHz and 512 megs of RAM.  I've been following
> the instructions in /usr/share/doc/mdadm/rootraiddoc97.html, but after
> rebooting to /dev/hdc1 (/dev/md0) and issuing the command "mdadm --add
> /dev/md0 /dev/hda1", I was only getting 1636K/sec for the
> synchronization, which was _way_ too slow (I forgot I had left a CDROM
> drive on the secondary IDE channel, so /dev/hdc1 was running at UDMA2,
> plus, I was using a 40 conductor cable instead of 80).  So I decided
> to reboot the system and cancel the synchronization, thus destroying
> everything on /dev/hda1 (but this didn't seem like an issue, since I
> was able to boot from /dev/md0 aka /dev/hdc1).
>
> So after stopping the sync process and rebooting, I was unable to boot
> from hdc1 because I had forgotten to copy over my initrd with md/raid1
> modules from /dev/hda1 to /dev/hdc1, and of course now /dev/hda1 is
> destroyed, so I can't use the initrd-RAID that was previously on there
> (and the initrd-RAID never existed on /dev/hdc1, since I created it on
> /dev/hda1 _after_ I had issued the "mount /dev/md0 /mnt/md0; cp -axu /
> /mnt/md0" commands - don't ask why).  So when I rebooted, I didn't
> have md/raid1 support, and so /dev/md0 couldn't mount.. I ended up
> building a new initrd with md/raid1 on another debian box, then
> copying them to /dev/hdc1.  I've now tried installing grub on
> /dev/hdc, disconnected /dev/hda, and attempted to boot.  I now get the
> following message:
>
> md: md driver 0.90.3 MAX_MD_DEVS=3D256, MD_SB_DISKS=3D27
> md: bitmap version 4.39
> md: raid1 personality registered as nr3
> .
> .
> Begin: Mounting root file system...
> Begin: Running /scripts/local-top
> Done.
> ALERT! /dev/md0 does not exist.  Dropping to a shell!
>
> At which point I get dropped into busybox..  One thing I notice is
> that although it shows the md module being loaded, it doesn't really
> say anything about configuring/adding any RAID disks (which is used to
> say, back when I had first booted into /dev/md0 (hdc1) before killing
> /dev/hda1).  So now apparently /dev/md0 no longer exists, and I have
> no idea how to create it.
>
> In fstab on /dev/hcd1, I have:
>
> /dev/md0 / ext3 defaults 0 0
>
> and in /boot/grub/menu.lst I have:
>
> title           Debian GNU/Linux, kernel 2.6.15-486 RAID
> root            (hd0,0)
> kernel          /boot/vmlinuz-2.6.15-486 root=3D/dev/md0 ro
> initrd          /boot/initrd.img-2.6.15-486-RAID
> savedefault
> boot
>
> Now, /dev/md0 was originally created by using:
>
> mdadm --create /dev/md0 --level=3D1 --raid-disks=3D2 missing /dev/hdc1
>
> can anybody tell me why /dev/md0 is no longer available on /dev/hdc1,
> and if possible how to recreate it?
>
> I should mention that after I cancelled the synchronization process,
> and was unable to reboot into /dev/hdc1, I then mounted /dev/hdc1
> through busybox, changed the Type of the disk from FD (linux raid) to
> 83 (linux), modified /etc/fstab and /boot/grub/menu.lst so it pointed
> to /dev/hdc1 instead of /dev/md0.  I tried rebooting after changing
> these options, and it gave me a message saying "Failed to mount root
> file system" or something to that effect. Could changing the Type of
> the disk or mounting /dev/hdc1 have caused /dev/md0 to become
> corrupted/deleted?
>
> If anybody has the time to read through my message
> and give me some advice, I would very much appreciate it.  Thanks,
>
> Mike

Re: Can't mount /dev/md0 after stopping a synchronization

[Troels Bang Jensen]

This isn't quite what you asked about, but the rootraiddoc97 document is 
kinda obsolete by now

-Debian Sarge has a new partitioner which can set up pretty advanced 
RAID configurations when installing. Just create RAID auto partitions on 
the disks and then create the arrays afterwards in the partitioner - 
it's far easier, and I've set up quite a few boot-on-RAID1 systems that way.

/Troels


Mike Garey wrote:
> Just wanted to add a few more details/questions to my previous post..
>
> In case I provided too much information in my previous email, here's a
> condensed version:
>
> 1) got to "step 6.2 Add the first-disk to our existing RAID device" of
> the rootraiddoc walkthrough
> 2) issued command "mdadm --add /dev/md0 /dev/hda1"
> 3) after getting terrible throughput, decided to stop the
> synchronization process and reboot
> 4) failed to reboot from /dev/md0 (hdc1)
> 5) hda1 is now hosed, hdc1 contains a complete working copy of the
> previous contents of /dev/hda1, but I'm unable to boot from it
> ("ALERT! /dev/md0 does not exist" message)
> 6) re-installed debian onto hda1 and rebooted from it.
>
> Now, it seems as though I'm sort of back at "step 3.3 Create RAID
> device".  I've tried to do the following while booted from my clean
> debian install on /dev/hda1:
>
> mkdir /mnt/md0
> mount /dev/md0 /mnt/md0
>
> but I get the message "mount: you must specify the filesystem type",
> because I guess the system has no idea what /dev/md0 is.  So then I
> tried the following (possibly risky?) command:
>
> mdadm --create /dev/md0 --level=1 --raid-disks=2 --spare-devices 0
> missing /dev/hdc1
>
> but I get the message:
>
> mdadm: /dev/hdc1 appears to contain an ext2fs file system
>     size=158577472K  mtime=Tue Apr  4 18:21:09 2006
> mdadm: /dev/hdc1 appears to be part of a raid array:
>     level=1 devices=2 ctime=Mon Apr  3 13:41:33 2006
> Continue creating array? n
> mdadm: create aborted
>
> It seems from the above message that mdadm --create would destroy all
> the data on /dev/hdc1 (someone correct me if I'm wrong), rather than
> simply creating /dev/md0.  If this is the case, should I be using
> mdadm --assemble instead? (but if so, I need to assemble to a
> pre-existing md device, in which case I don't have one)  What I'd like
> to do is be able to mount /dev/hdc1 as /dev/md0 (as in "step 4.2 Mount
> your RAID device."), and then hopefully be able to boot off of it, but
> right now I'm confused about how to create /dev/md0 without destroying
> any of my information on either /dev/hda1 or /dev/hdc1 (of course,
> after the reboot, I don't mind destroying /dev/hda1).
>
> the other alternative is of course to zero the superblocks on
> /dev/hdc1, reboot from it as a normal drive, and then follow the
> walkthrough from the beginning, but this time starting with /dev/hdc1
> as the master drive.  I'd like to avoid this if possible, since I'd
> really like to find out why I can't boot from /dev/md0 (hdc1), since
> if this was not a test, and /dev/hda1 really did fail, I'd be in a
> mess.  So I'd prefer to know how to recover from this, since it seems
> that the position I'm in right now is essentiallly the same scenario
> as if /dev/hda1 had failed, in which case I'd like to be prepared.
>
> If anyone can make heads or tails of what I'm talking about, I would
> greatly appreciate any information or suggestions.  Thanks in advance,
>
> Mike
>
> P.S. output of mdadm --examine is given below (output is the same for
> /dev/hda1 and /dev/hdc1)
>
>           Magic : a92b4efc
>         Version : 00.90.03
>            UUID : f1ab0f6d:6c5c4bf6:228dec79:3aa3582f
>   Creation Time : Mon Apr  3 13:41:33 2006
>      Raid Level : raid1
>    Raid Devices : 2
>   Total Devices : 2
> Preferred Minor : 0
>
>     Update Time : Tue Apr  4 09:37:10 2006
>           State : clean
>  Active Devices : 1
> Working Devices : 2
>  Failed Devices : 0
>   Spare Devices : 1
>        Checksum : f63c0eb5 - correct
>          Events : 0.19314
>
>
>       Number   Major   Minor   RaidDevice State
> this     1      22        1        1      active sync   /dev/hdc1
>
>    0     0       0        0        0      removed
>    1     1      22        1        1      active sync   /dev/hdc1
>    2     2       3        1        2      spare   /dev/hda1
>
>
>   
>> I'm in the process of setting up a software RAID level 1 on debian
>> testing with two 160 gig drives on an Asus P4P800 motherboard with
>> Intel Pentium 4 CPU 3.00GHz and 512 megs of RAM.  I've been following
>> the instructions in /usr/share/doc/mdadm/rootraiddoc97.html, but after
>> rebooting to /dev/hdc1 (/dev/md0) and issuing the command "mdadm --add
>> /dev/md0 /dev/hda1", I was only getting 1636K/sec for the
>> synchronization, which was _way_ too slow (I forgot I had left a CDROM
>> drive on the secondary IDE channel, so /dev/hdc1 was running at UDMA2,
>> plus, I was using a 40 conductor cable instead of 80).  So I decided
>> to reboot the system and cancel the synchronization, thus destroying
>> everything on /dev/hda1 (but this didn't seem like an issue, since I
>> was able to boot from /dev/md0 aka /dev/hdc1).
>>
>> So after stopping the sync process and rebooting, I was unable to boot
>> from hdc1 because I had forgotten to copy over my initrd with md/raid1
>> modules from /dev/hda1 to /dev/hdc1, and of course now /dev/hda1 is
>> destroyed, so I can't use the initrd-RAID that was previously on there
>> (and the initrd-RAID never existed on /dev/hdc1, since I created it on
>> /dev/hda1 _after_ I had issued the "mount /dev/md0 /mnt/md0; cp -axu /
>> /mnt/md0" commands - don't ask why).  So when I rebooted, I didn't
>> have md/raid1 support, and so /dev/md0 couldn't mount.. I ended up
>> building a new initrd with md/raid1 on another debian box, then
>> copying them to /dev/hdc1.  I've now tried installing grub on
>> /dev/hdc, disconnected /dev/hda, and attempted to boot.  I now get the
>> following message:
>>
>> md: md driver 0.90.3 MAX_MD_DEVS=3D256, MD_SB_DISKS=3D27
>> md: bitmap version 4.39
>> md: raid1 personality registered as nr3
>> .
>> .
>> Begin: Mounting root file system...
>> Begin: Running /scripts/local-top
>> Done.
>> ALERT! /dev/md0 does not exist.  Dropping to a shell!
>>
>> At which point I get dropped into busybox..  One thing I notice is
>> that although it shows the md module being loaded, it doesn't really
>> say anything about configuring/adding any RAID disks (which is used to
>> say, back when I had first booted into /dev/md0 (hdc1) before killing
>> /dev/hda1).  So now apparently /dev/md0 no longer exists, and I have
>> no idea how to create it.
>>
>> In fstab on /dev/hcd1, I have:
>>
>> /dev/md0 / ext3 defaults 0 0
>>
>> and in /boot/grub/menu.lst I have:
>>
>> title           Debian GNU/Linux, kernel 2.6.15-486 RAID
>> root            (hd0,0)
>> kernel          /boot/vmlinuz-2.6.15-486 root=3D/dev/md0 ro
>> initrd          /boot/initrd.img-2.6.15-486-RAID
>> savedefault
>> boot
>>
>> Now, /dev/md0 was originally created by using:
>>
>> mdadm --create /dev/md0 --level=3D1 --raid-disks=3D2 missing /dev/hdc1
>>
>> can anybody tell me why /dev/md0 is no longer available on /dev/hdc1,
>> and if possible how to recreate it?
>>
>> I should mention that after I cancelled the synchronization process,
>> and was unable to reboot into /dev/hdc1, I then mounted /dev/hdc1
>> through busybox, changed the Type of the disk from FD (linux raid) to
>> 83 (linux), modified /etc/fstab and /boot/grub/menu.lst so it pointed
>> to /dev/hdc1 instead of /dev/md0.  I tried rebooting after changing
>> these options, and it gave me a message saying "Failed to mount root
>> file system" or something to that effect. Could changing the Type of
>> the disk or mounting /dev/hdc1 have caused /dev/md0 to become
>> corrupted/deleted?
>>
>> If anybody has the time to read through my message
>> and give me some advice, I would very much appreciate it.  Thanks,
>>
>> Mike
>>     
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>   

Re: Can't mount /dev/md0 after stopping a synchronization

[Mike Garey]

On 4/5/06, Troels Bang Jensen <marvin@fnuck.dk> wrote:
> This isn't quite what you asked about, but the rootraiddoc97 document is
> kinda obsolete by now
>
> -Debian Sarge has a new partitioner which can set up pretty advanced
> RAID configurations when installing. Just create RAID auto partitions on
> the disks and then create the arrays afterwards in the partitioner -
> it's far easier, and I've set up quite a few boot-on-RAID1 systems that way.

Thanks for the suggestion Troels, I didn't realize the rootraiddoc was
obsolete (although it is about 2 years old now it seems).  In the
future, I think I'll try using the debian partitioner, but at the
moment, I'd still like to find out why I can't mount /dev/md0, as it's
rather disturbing, given the fact that if this happens in a production
system and I can't recover from it, I'm screwed.

I tried booting from /dev/hdc1 (as /dev/md0 in grub) using a 2.6.15
kernel with md and raid1 support built in and this is what I now get:

md: autodetecting raid arrays
md: autorun ...
md: considering hdc1 ...
md: adding hdc1 ...
md: created md0
md: bind:<hdc1>
raid1: RAID set md0 active with 1 out of 2 mirrors
md: ...autrun done.

Warning: unable to open an initial console
Input: AT translated set 2 keyboard as /class/input/input0

and then at this point, the system just hangs and nothing happens.  So
I seem to be getting closer.. If I try booting from a kernel without
raid1 and md support, but using an initrd with raid1/md modules, then
I get the "ALERT! /dev/md0 does not exist.  Dropping to a shell!"
message.  I can't understand why there would be any difference between
using a kernel with raid1/md support, or using an initrd image with
raid1/md support, but apparently there is.  If anyone else has any
suggestions, please keep them coming.

Regards,

Mike

Re: Can't mount /dev/md0 after stopping a synchronization

[Tuomas Leikola]

On 4/5/06, Mike Garey <random51k@gmail.com> wrote:
> I tried booting from /dev/hdc1 (as /dev/md0 in grub) using a 2.6.15
> kernel with md and raid1 support built in and this is what I now get:
>
> md: autodetecting raid arrays
> md: autorun ...
> md: considering hdc1 ...
> md: adding hdc1 ...
> md: created md0
> md: bind:<hdc1>
> raid1: RAID set md0 active with 1 out of 2 mirrors
> md: ...autrun done.
>
> Warning: unable to open an initial console
> Input: AT translated set 2 keyboard as /class/input/input0
>
> and then at this point, the system just hangs and nothing happens.  So
> I seem to be getting closer.. If I try booting from a kernel without
> raid1 and md support, but using an initrd with raid1/md modules, then
> I get the "ALERT! /dev/md0 does not exist.  Dropping to a shell!"
> message.  I can't understand why there would be any difference between
> using a kernel with raid1/md support, or using an initrd image with
> raid1/md support, but apparently there is.  If anyone else has any
> suggestions, please keep them coming.

Sounds like your initrd could use a command like

mdadm --assemble /dev/md0 /dev/hda1 /dev/hdc1

at some point before mounting the real rootfs. There are many cleaner
examples in the list archive, but that should do the trick. It seems
like your initrd-kernel doesn't autostart the raid for some reason
(config option?).

Note, you should never do any read/write access to the component disks
after creating the raid. I guess you know this already, but some
wording seemed suspect.

Can you specify more what is the problem with mounting md0? The log
snipped doesn't show any errors about that.

Re: Can't mount /dev/md0 after stopping a synchronization

[Mike Garey]

On 4/5/06, Tuomas Leikola <tuomas.leikola@gmail.com> wrote:
> On 4/5/06, Mike Garey <random51k@gmail.com> wrote:
> > I tried booting from /dev/hdc1 (as /dev/md0 in grub) using a 2.6.15
> > kernel with md and raid1 support built in and this is what I now get:
> >
> > md: autodetecting raid arrays
> > md: autorun ...
> > md: considering hdc1 ...
> > md: adding hdc1 ...
> > md: created md0
> > md: bind:<hdc1>
> > raid1: RAID set md0 active with 1 out of 2 mirrors
> > md: ...autrun done.
> >
> > Warning: unable to open an initial console
> > Input: AT translated set 2 keyboard as /class/input/input0
> >
> > and then at this point, the system just hangs and nothing happens.  So
> > I seem to be getting closer.. If I try booting from a kernel without
> > raid1 and md support, but using an initrd with raid1/md modules, then
> > I get the "ALERT! /dev/md0 does not exist.  Dropping to a shell!"
> > message.  I can't understand why there would be any difference between
> > using a kernel with raid1/md support, or using an initrd image with
> > raid1/md support, but apparently there is.  If anyone else has any
> > suggestions, please keep them coming.
>
> Sounds like your initrd could use a command like
>
> mdadm --assemble /dev/md0 /dev/hda1 /dev/hdc1
>
> at some point before mounting the real rootfs. There are many cleaner
> examples in the list archive, but that should do the trick. It seems
> like your initrd-kernel doesn't autostart the raid for some reason
> (config option?).

but wouldn't I need to have /dev/md0 available before doing mdadm
--assemble?  When booting from an initrd image, /dev/md0 is never
created for some reason..

> Note, you should never do any read/write access to the component disks
> after creating the raid. I guess you know this already, but some
> wording seemed suspect.

I take it you mean after the disks are synchronized, it's a bad thing
to write directly to /dev/hda1 or /dev/hdc1... In my case, I've
written directly to /dev/hdc1, but it's the only disk in the array,
and I'm going to be resyncing /dev/hda1 anyways, so it seems in this
instance it's okay?

> Can you specify more what is the problem with mounting md0? The log
> snipped doesn't show any errors about that.

after googling a bit more, I seem to have found a solution to my problem:

http://linux.derkeiler.com/Mailing-Lists/Debian/2005-07/2953.html

It seems that when I followed rootraiddoc.97, "Step 4.3 Copy your
Debian system to RAID device.", issuing the command "cp -axu /
/mnt/md0" didn't copy over my /dev directory, which was mounted on a
separate partition (tmpfs on /dev type tmpfs) and since I used the -x
switch to cp, it forced it to stay on the current file system (which I
thought was okay, since I figured /dev was populated dynamically by
udev anyways).

In any case, I simply booted from /dev/hda1, mounted /dev/hdc1, then
copied /dev/console from hda1 to hdc1.  After this, I was able to
disconnect hda and boot into /dev/md0 on hdc1.  I guess I should've
figured this out when I first ran into the "Warning: unable to open an
initial console" error when booting from a raid1/md enabled kernel,
but at that point I switched to using an initrd image and everything
seemed fine (how wrong I was! :)

So in conclusion, I'm still unable to boot from a kernel with an
initrd image containing the md and raid1 modules (I'm sure this has
something to do with udev not creating /dev/md0 properly), but I can
successfully boot from a raid1/md enabled kernel.  So after a very
large diversion, I'm back to my original problem of trying to decrease
the sync time.  I've now disconnected the CDROM drive from the
secondary ide channel, and I've replaced the 40 conductor cable with
an 80 conductor cable.  Hopefully it won't take me 25 hours to
synchronize the second time around!

Thanks to everyone who offered advice and suggestions, your help was
greatly appreciated (and possibly this thread may help some other poor
soul in the future, should they run into a similar situation).

Mike

Re: Can't mount /dev/md0 after stopping a synchronization

[Paul Clements]

Mike Garey wrote:

> I seem to be getting closer.. If I try booting from a kernel without
> raid1 and md support, but using an initrd with raid1/md modules, then
> I get the "ALERT! /dev/md0 does not exist.  Dropping to a shell!"
> message.  I can't understand why there would be any difference between
> using a kernel with raid1/md support, or using an initrd image with
> raid1/md support, but apparently there is.  If anyone else has any

Autodetection doesn't occur unless md is built into the kernel -- one of 
the reasons why using autodetection is becoming less and less popular. 
You're probably better off assembling the array from your initrd with 
some invocation of mdadm.

--
Paul

Re: Can't mount /dev/md0 after stopping a synchronization

[Mike Garey]

On 4/5/06, Paul Clements <paul.clements@steeleye.com> wrote:
> Mike Garey wrote:
>
> > I seem to be getting closer.. If I try booting from a kernel without
> > raid1 and md support, but using an initrd with raid1/md modules, then
> > I get the "ALERT! /dev/md0 does not exist.  Dropping to a shell!"
> > message.  I can't understand why there would be any difference between
> > using a kernel with raid1/md support, or using an initrd image with
> > raid1/md support, but apparently there is.  If anyone else has any
>
> Autodetection doesn't occur unless md is built into the kernel -- one of
> the reasons why using autodetection is becoming less and less popular.
> You're probably better off assembling the array from your initrd with
> some invocation of mdadm.

I've since tried building my own initrd using the mkinitramfs script
that comes with debian testing for kernel 2.6.15, as well as the
mkinitramfs from mdadm-2.4.  Before, I had only used mkinitramfs that
came with debian testing, and it didn't seem to include the mdadm
binary into the image, which would explain why it wasn't able to mount
the root filesystem (it also tries to execute the command "/sbin/mdrun
dev", which I guess should start the RAID array, but /sbin/mdrun
doesn't exist either - seems the built in mkinitramfs script doesn't
really work exactly as it should - either that, or I'm making a
mistake somewhere).  Anyways, after booting using my initramfs image
(and failling to mount /dev/md0), I was dropped into busybox and tried
the following:

/ # mdadm -Acpartitions --super-minor=0 --auto=part /dev/mda
md: md_d0 stopped.
md: bind<hdc1>
raid: raid set md_d0 active with 1 out of 2 mirrors
mdadm: /dev/mda has been started with 1 drive (out of 2).
/ # mount /dev/mda /root
EXT2-fs warning (device md_d0): ext2_fill_super: mounting ext3
filesystem as ext2

/# mount
none on /sys type sysfs (rw)
non on /proc type proc (rw,nodiratime)
udev on /dev type tmpfs (rw)
/dev/mda on /root type ext2 (rw,nogrpid)

so it seems with a little bit of tweaking, I'll be able to modify the
scripts to get /dev/md0 to mount as the root file system.  The only
thing that bothers me is the "EXT2-fs warning" about having the ext3
filesystem mounted as ext2.. Does anybody know why this is happening,
or how to fix this?  Also, having mounted the filesystem as ext2, will
I have caused any damage to it?  And should I be mounting my RAID
device as /dev/mda or /dev/md0?

On 4/5/06, Jim Klimov <klimov@2ka.mipt.ru> wrote:
> Hello Mike,
>
> I'm afraid I haven't seen your first messages in detail. Sorry, if I
> repeat the lines you already know by heart ;)
>
> Nevertheless, do you by chance have the ability to rebuild your
> kernel, or are you keen on making MD work from initrd and some stock
> kernel?

right now I'm using a custom built kernel with md and raid1 built into
it and it works fine (although I get the same EXT2-fs warning as with
the initramfs image early on in the boot process, but it seems to end
up being mounted as ext3 when the system finally boots).  As I
mentioned above, I'm now trying to get the initramfs version working,
since it'd be nice to have a solution that works with the standard
prepackaged kernel.


> If a custom kernel is ok, link the raid1 driver statically. In this
> case, if the root submirrors are primary partitions flagged with 0xfd
> filesystem type, the kernel will assemble these partitions to a mirror
> before mounting.
>
> You can also try the raid1 driver's parameters on kernel command line
> to force md-device naming and components, i.e.
> ... root=/dev/md0 md=0,/dev/hda1,/dev/hdc1
>
> And in case of mirroring i think you should always be able to mount
> the filesystems as they are, i.e.
> ... root=/dev/hda1 ro
>
> A pitfall is there, that /etc/fstab may require mounting other FSes
> from metadevices, not partitions. And if you mount read-write, then
> your submirrors will be out of sync and that's undetected by raid
> driver.

Re: Can't mount /dev/md0 after stopping a synchronization

[Tuomas Leikola]

> but wouldn't I need to have /dev/md0 available before doing mdadm
> --assemble?  When booting from an initrd image, /dev/md0 is never
> created for some reason..
>

Sorry, I didn't realize initrd adds to the mix.

> > Note, you should never do any read/write access to the component disks
> > after creating the raid. I guess you know this already, but some
> > wording seemed suspect.
>
> I take it you mean after the disks are synchronized, it's a bad thing
> to write directly to /dev/hda1 or /dev/hdc1... In my case, I've
> written directly to /dev/hdc1, but it's the only disk in the array,
> and I'm going to be resyncing /dev/hda1 anyways, so it seems in this
> instance it's okay?

Correct, unless the disks are already halfway synchronized. Didn't
check this, but IIRC modern-day md remembers how much is synchronized,
it sync is aborted (cleanly).

Glad your problem got solved!

BTW, if you're still having problems with sync speed, try fiddling around with

/proc/sys/dev/raid/speed_limit_min

as the md sync speed detection code is not foolproof (at least lvms as
component devices has fooled it on occasion).

Re: Can't mount /dev/md0 after stopping a synchronization

[Mike Garey]

> On 4/5/06, Jim Klimov <klimov@2ka.mipt.ru> wrote:
> > I'm afraid I haven't seen your first messages in detail. Sorry, if I
> > repeat the lines you already know by heart ;)
> >
> > Nevertheless, do you by chance have the ability to rebuild your
> > kernel, or are you keen on making MD work from initrd and some stock
> > kernel?
>
> right now I'm using a custom built kernel with md and raid1 built into
> it and it works fine (although I get the same EXT2-fs warning as with
> the initramfs image early on in the boot process, but it seems to end
> up being mounted as ext3 when the system finally boots).  As I
> mentioned above, I'm now trying to get the initramfs version working,
> since it'd be nice to have a solution that works with the standard
> prepackaged kernel.

Well, I figured out why I was getting the EXT2-fs warning: I had
forgotten to build ext3 support into the kernel..  I don't think I
damaged my disk though, cause it mounts the system as read-only at
first, and by the time it mounts it read-write, the ext3 module is
loaded.

I've also finally managed to get /dev/md0 mounted while booted from an
initramfs image.  I used the stock debian mkinitramfs script and for
some reason it decided to start working.  I really can't figure out
why I had so much trouble with it in the beginning, but it seemed to
start working after I had manually entered the command "mdadm
-Acpartitions --super-minor=0 --auto=part /dev/mda" after being
dropped into busybox - perhaps this changed the superblocks on
/dev/hdc1 or something that I'm not aware of..

On 4/7/06, Tuomas Leikola <tuomas.leikola@gmail.com> wrote:
> BTW, if you're still having problems with sync speed, try fiddling around with
>
> /proc/sys/dev/raid/speed_limit_min
>
> as the md sync speed detection code is not foolproof (at least lvms as
> component devices has fooled it on occasion).

I ended up changing the number in /proc/sys/dev/raid/speed_limit_min to
150000.  I also disconnected the CDROM from the secondary IDE channel,
replaced the 40 conductor cable with an 80 conductor cable and enabled
UDMA mode 5 on both my hard drives.  I now get the following when
syncing:

[12:44AM][mike@asterisk]% cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 hda1[2] hdc1[1]
  158577472 blocks [2/1] [_U]
  [>....................]  recovery =  0.2% (469056/158577472)
finish=78.6min speed=33504K/sec

So it seems I managed to get the sync time down from 25 hours to 78,
which of course is much more reasonable ;)

I have one last question though.. When I update /boot/grub/menu.lst
while booted from /dev/md0 with both disks available, does this file
get written to the MBR on both disks, or do I have to do this
manually?

Just wanted to say thanks to everybody that helped and offered advice,
I'm really glad that I finally managed to resolve this problem - even
with a backup, I still get very worried about messing with my data.

Mike

Re: Can't mount /dev/md0 after stopping a synchronization

[Tuomas Leikola]

On 4/8/06, Mike Garey <random51k@gmail.com> wrote:
> I have one last question though.. When I update /boot/grub/menu.lst
> while booted from /dev/md0 with both disks available, does this file
> get written to the MBR on both disks, or do I have to do this
> manually?
>

Grub's configuration lives on both mirrors, as it's in the filesystem,
not in MBR. At boot time, grub kinda "mounts" the filesystem and reads
the configuration from there. (grub doesn't understand the mirror, but
it doesn't need to)