Re: RAID50 boot problems

Re: RAID50 boot problems

[Alexander Zvyagin]

[-- Attachment #1: Type: text/plain, Size: 912 bytes --]

Hi,

I have almost working RAID50 (two RAID5 forms RAID0) in my
Linux-mint13. When booting the system always ends up in initramfs
shell, saying it cannot find the "root". When I type "cat
/proc/mdstat" I see that both RAID5 are available, but they did not
form RAID0. So I do "mdadm -A --scan" to finish the raid creation.
After that I can exit the initramfs shell and system finishes booting
(and works later) without any problems. Of course initramfs image is
in sync with root (/etc/mdadm/mdadm.conf file is the same in initramfs
and rootfs).
This is what my system is using:
kernel 3.5.0-26-generic
mdadm - v3.2.5 - 18th May 2012

To summarize, I have  RAID50 which _almost_ works. It just needs to be
pushed/kicked during the boot. Could you advise my how can I
automatize the process, so I do not need to type "mdadm -A
--scan;exit" every time I switch on my computer?

Thanks a lot in advance,
Alexander.

[-- Attachment #2: mdadm.conf --]
[-- Type: application/octet-stream, Size: 254 bytes --]

ARRAY /dev/md126 metadata=1.2 name=mint:2 UUID=d4627170:69b910b8:cf8e268b:83a62762
ARRAY /dev/md/mint:1 metadata=1.2 name=mint:1 UUID=f952099f:0f15aa9e:5d8c6ce9:bfa86e8f
ARRAY /dev/md3 metadata=1.2 name=az-desk:3 UUID=60cb6f56:38ee9e2d:360e3a9f:e8980f93

[-- Attachment #3: mdadm.examine --]
[-- Type: application/octet-stream, Size: 6672 bytes --]

/dev/sda2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : f952099f:0f15aa9e:5d8c6ce9:bfa86e8f
           Name : mint:1
  Creation Time : Sun Apr  7 20:04:59 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 622925131 (297.03 GiB 318.94 GB)
     Array Size : 934387200 (891.10 GiB 956.81 GB)
  Used Dev Size : 622924800 (297.03 GiB 318.94 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 54abc4d2:23409f91:fee39aa2:39d07220

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Apr 17 20:28:08 2013
       Checksum : e870f7ff - correct
         Events : 1231

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sdb2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : f952099f:0f15aa9e:5d8c6ce9:bfa86e8f
           Name : mint:1
  Creation Time : Sun Apr  7 20:04:59 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 622925131 (297.03 GiB 318.94 GB)
     Array Size : 934387200 (891.10 GiB 956.81 GB)
  Used Dev Size : 622924800 (297.03 GiB 318.94 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : c35d9e5f:e1ffe560:9c10b049:9771c128

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Apr 17 20:28:08 2013
       Checksum : f3f53828 - correct
         Events : 1231

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sdc2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : f952099f:0f15aa9e:5d8c6ce9:bfa86e8f
           Name : mint:1
  Creation Time : Sun Apr  7 20:04:59 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 622925131 (297.03 GiB 318.94 GB)
     Array Size : 934387200 (891.10 GiB 956.81 GB)
  Used Dev Size : 622924800 (297.03 GiB 318.94 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 12e4fe90:e9ae1caf:d9f61454:056207a1

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Apr 17 20:28:08 2013
       Checksum : f637442c - correct
         Events : 1231

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sdd2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : f952099f:0f15aa9e:5d8c6ce9:bfa86e8f
           Name : mint:1
  Creation Time : Sun Apr  7 20:04:59 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 622925131 (297.03 GiB 318.94 GB)
     Array Size : 934387200 (891.10 GiB 956.81 GB)
  Used Dev Size : 622924800 (297.03 GiB 318.94 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : c69a0620:6b05340c:fabfd028:fb64582d

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Apr 17 20:28:08 2013
       Checksum : 43631d7a - correct
         Events : 1231

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sde2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : d4627170:69b910b8:cf8e268b:83a62762
           Name : mint:2
  Creation Time : Sun Apr  7 20:04:41 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 622925131 (297.03 GiB 318.94 GB)
     Array Size : 934387200 (891.10 GiB 956.81 GB)
  Used Dev Size : 622924800 (297.03 GiB 318.94 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 7565be7b:b8663018:43e8cf59:e5ea1f89

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Apr 17 20:28:09 2013
       Checksum : 971fadbf - correct
         Events : 1423

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sdf2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : d4627170:69b910b8:cf8e268b:83a62762
           Name : mint:2
  Creation Time : Sun Apr  7 20:04:41 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 622925131 (297.03 GiB 318.94 GB)
     Array Size : 934387200 (891.10 GiB 956.81 GB)
  Used Dev Size : 622924800 (297.03 GiB 318.94 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 793a705d:606dfd59:81bb10ee:233d0679

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Apr 17 20:28:09 2013
       Checksum : 3ec5aee9 - correct
         Events : 1423

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sdg2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : d4627170:69b910b8:cf8e268b:83a62762
           Name : mint:2
  Creation Time : Sun Apr  7 20:04:41 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 622925131 (297.03 GiB 318.94 GB)
     Array Size : 934387200 (891.10 GiB 956.81 GB)
  Used Dev Size : 622924800 (297.03 GiB 318.94 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : c274c01d:66d6409f:9581b260:f46bd93c

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Apr 17 20:28:09 2013
       Checksum : 7ace471d - correct
         Events : 1423

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sdi2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : d4627170:69b910b8:cf8e268b:83a62762
           Name : mint:2
  Creation Time : Sun Apr  7 20:04:41 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 622925131 (297.03 GiB 318.94 GB)
     Array Size : 934387200 (891.10 GiB 956.81 GB)
  Used Dev Size : 622924800 (297.03 GiB 318.94 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : b503e602:da8427cc:40c34fc3:dc46e6da

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Apr 17 20:28:09 2013
       Checksum : 8d84a11a - correct
         Events : 1423

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing)

Re: RAID50 boot problems

[Alexander Zvyagin]

yes, I can do that. But I thought that mdadm should be able to
assemble my raid50 without any help from my side.

Let me rephrase the question. My raid50 system does not boot because ....
- I miss something in mdadm configuration
- mdadm cannot assemble (by itself) raid50 arrays (which is hard to believe)
- there is a bug somewhere

With best wishes,
Alexander Zvyagin.


On 17 April 2013 21:08, Tommy Apel <tommyapeldk@gmail.com> wrote:
> unpack you initfs and make the changes to the init file and pack it again.
>
> /Tommy
>
>
> 2013/4/17 Alexander Zvyagin <zvyagin.alexander@gmail.com>
>>
>> Hi,
>>
>> I have almost working RAID50 (two RAID5 forms RAID0) in my
>> Linux-mint13. When booting the system always ends up in initramfs
>> shell, saying it cannot find the "root". When I type "cat
>> /proc/mdstat" I see that both RAID5 are available, but they did not
>> form RAID0. So I do "mdadm -A --scan" to finish the raid creation.
>> After that I can exit the initramfs shell and system finishes booting
>> (and works later) without any problems. Of course initramfs image is
>> in sync with root (/etc/mdadm/mdadm.conf file is the same in initramfs
>> and rootfs).
>> This is what my system is using:
>> kernel 3.5.0-26-generic
>> mdadm - v3.2.5 - 18th May 2012
>>
>> To summarize, I have  RAID50 which _almost_ works. It just needs to be
>> pushed/kicked during the boot. Could you advise my how can I
>> automatize the process, so I do not need to type "mdadm -A
>> --scan;exit" every time I switch on my computer?
>>
>> Thanks a lot in advance,
>> Alexander.
>
>

Re: RAID50 boot problems

[Roy Sigurd Karlsbakk]

----- Opprinnelig melding -----
> Hi,
> 
> I have almost working RAID50 (two RAID5 forms RAID0) in my
> Linux-mint13. When booting the system always ends up in initramfs
> shell, saying it cannot find the "root". When I type "cat
> /proc/mdstat" I see that both RAID5 are available, but they did not
> form RAID0. So I do "mdadm -A --scan" to finish the raid creation.
> After that I can exit the initramfs shell and system finishes booting
> (and works later) without any problems. Of course initramfs image is
> in sync with root (/etc/mdadm/mdadm.conf file is the same in initramfs
> and rootfs).
> This is what my system is using:
> kernel 3.5.0-26-generic
> mdadm - v3.2.5 - 18th May 2012
> 
> To summarize, I have RAID50 which _almost_ works. It just needs to be
> pushed/kicked during the boot. Could you advise my how can I
> automatize the process, so I do not need to type "mdadm -A
> --scan;exit" every time I switch on my computer?

Interesting… I have an ubuntu 13.04 VM for just raid testing, and I tried this configuration. I created two RAID-5s here and then a RAID-0 on top of those. I yet haven't put a filesystem on it. I just wanted to see if I could reproduce your issue, and I can.

It seems what happens is it only runs an initial scan, and then just finding the raids on physical (or otherwise) disks. In my case, md9 won't be available before md0 and md1 are started. I solved this by adding a line to /etc/rc.local

# If using nested MD, such as RAID5+0, another scan is needed
mdadm --assemble --scan

See below for full logs.

roy

root@raidtest:~# mdadm --detail --scan >> /etc/mdadm/mdadm.conf
(edit mdadm.conf to remove old entries)
root@raidtest:~# update-initramfs -u 
update-initramfs: Generating /boot/initrd.img-3.8.0-18-generic
root@raidtest:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md9 : active raid0 md0[0] md1[1]
      4191232 blocks super 1.2 512k chunks
      
md1 : active raid5 vde[0] vdg[3] vdf[1]
      2096000 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      
md0 : active raid5 vdd[3] vdc[1] vdb[0]
      2096000 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      
unused devices: <none>

root@raidtest:~# reboot
...
root@raidtest:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 vdd[3] vdc[1] vdb[0]
      2096000 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      
md1 : active raid5 vdg[3] vde[0] vdf[1]
      2096000 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      
unused devices: <none>
root@raidtest:~# 

-- 
Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID50 boot problems

[Tommy Apel]

The thing is that in order for this to work you need to have the
stripe scanned after the two 5s are created, I seem to recall that if
you put the stripe last in your mdadm.conf it will come up after the
5s


/Tommy

2013/4/19 Roy Sigurd Karlsbakk <roy@karlsbakk.net>:
> ----- Opprinnelig melding -----
>> Hi,
>>
>> I have almost working RAID50 (two RAID5 forms RAID0) in my
>> Linux-mint13. When booting the system always ends up in initramfs
>> shell, saying it cannot find the "root". When I type "cat
>> /proc/mdstat" I see that both RAID5 are available, but they did not
>> form RAID0. So I do "mdadm -A --scan" to finish the raid creation.
>> After that I can exit the initramfs shell and system finishes booting
>> (and works later) without any problems. Of course initramfs image is
>> in sync with root (/etc/mdadm/mdadm.conf file is the same in initramfs
>> and rootfs).
>> This is what my system is using:
>> kernel 3.5.0-26-generic
>> mdadm - v3.2.5 - 18th May 2012
>>
>> To summarize, I have RAID50 which _almost_ works. It just needs to be
>> pushed/kicked during the boot. Could you advise my how can I
>> automatize the process, so I do not need to type "mdadm -A
>> --scan;exit" every time I switch on my computer?
>
> Interesting… I have an ubuntu 13.04 VM for just raid testing, and I tried this configuration. I created two RAID-5s here and then a RAID-0 on top of those. I yet haven't put a filesystem on it. I just wanted to see if I could reproduce your issue, and I can.
>
> It seems what happens is it only runs an initial scan, and then just finding the raids on physical (or otherwise) disks. In my case, md9 won't be available before md0 and md1 are started. I solved this by adding a line to /etc/rc.local
>
> # If using nested MD, such as RAID5+0, another scan is needed
> mdadm --assemble --scan
>
> See below for full logs.
>
> roy
>
> root@raidtest:~# mdadm --detail --scan >> /etc/mdadm/mdadm.conf
> (edit mdadm.conf to remove old entries)
> root@raidtest:~# update-initramfs -u
> update-initramfs: Generating /boot/initrd.img-3.8.0-18-generic
> root@raidtest:~# cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> md9 : active raid0 md0[0] md1[1]
>       4191232 blocks super 1.2 512k chunks
>
> md1 : active raid5 vde[0] vdg[3] vdf[1]
>       2096000 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
>
> md0 : active raid5 vdd[3] vdc[1] vdb[0]
>       2096000 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
>
> unused devices: <none>
>
> root@raidtest:~# reboot
> ...
> root@raidtest:~# cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> md0 : active raid5 vdd[3] vdc[1] vdb[0]
>       2096000 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
>
> md1 : active raid5 vdg[3] vde[0] vdf[1]
>       2096000 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
>
> unused devices: <none>
> root@raidtest:~#
>
> --
> Vennlige hilsener / Best regards
>
> roy
> --
> Roy Sigurd Karlsbakk
> (+47) 98013356
> roy@karlsbakk.net
> http://blogg.karlsbakk.net/
> GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
> --
> I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID50 boot problems

[Roy Sigurd Karlsbakk]

> # If using nested MD, such as RAID5+0, another scan is needed
> mdadm --assemble --scan

Btw, that probably won't be enough for you, since fstab etc is parsed before rc.local is run. I tried to fix it with doing

# If using nested MD, such as RAID5+0, another scan is needed
mdadm --assemble --scan
mount /testraid

…but for some reason the filesystem doesn't get mounted...

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID50 boot problems

[Roy Sigurd Karlsbakk]

----- Opprinnelig melding -----
> The thing is that in order for this to work you need to have the
> stripe scanned after the two 5s are created, I seem to recall that if
> you put the stripe last in your mdadm.conf it will come up after the
> 5s

I just tried to change the order in mdadm.conf, update-initramfs -u, reboot, same thing. md9 doesn't come up until a manual -A --scan

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID50 boot problems

[Alexander Zvyagin]

> I just tried to change the order in mdadm.conf, update-initramfs -u, reboot, same thing. md9 doesn't come up until a manual -A --scan

yes, I was trying to do that as well (I tried too many things, some of
them are already forgotten).

thanks for looking into my problem!
Alexander.

Re: RAID50 boot problems

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

On Wed, 17 Apr 2013 20:45:40 +0200
Alexander Zvyagin <zvyagin.alexander@gmail.com> wrote:

> I have almost working RAID50 (two RAID5 forms RAID0) in my
> Linux-mint13. When booting the system always ends up in initramfs
> shell, saying it cannot find the "root".

Post a full dmesg of a failed boot-up;

also try "rootdelay=10" kernel command line parameter.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: RAID50 boot problems

[Roy Sigurd Karlsbakk]

> On Wed, 17 Apr 2013 20:45:40 +0200
> Alexander Zvyagin <zvyagin.alexander@gmail.com> wrote:
> 
> > I have almost working RAID50 (two RAID5 forms RAID0) in my
> > Linux-mint13. When booting the system always ends up in initramfs
> > shell, saying it cannot find the "root".
> 
> Post a full dmesg of a failed boot-up;
> 
> also try "rootdelay=10" kernel command line parameter.

Please see http://paste.ubuntu.com/5721934/ for the full list, taken with network console. This is with rootdelay=10
-- 
Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID50 boot problems

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 987 bytes --]

On Fri, 19 Apr 2013 17:58:06 +0200 (CEST) Roy Sigurd Karlsbakk
<roy@karlsbakk.net> wrote:

> > On Wed, 17 Apr 2013 20:45:40 +0200
> > Alexander Zvyagin <zvyagin.alexander@gmail.com> wrote:
> > 
> > > I have almost working RAID50 (two RAID5 forms RAID0) in my
> > > Linux-mint13. When booting the system always ends up in initramfs
> > > shell, saying it cannot find the "root".
> > 
> > Post a full dmesg of a failed boot-up;
> > 
> > also try "rootdelay=10" kernel command line parameter.
> 
> Please see http://paste.ubuntu.com/5721934/ for the full list, taken with network console. This is with rootdelay=10

The "bind" messages are in random order so presumably udev running 'mdadm -I'
on each device as it appear to add it to an array.
However when the md0 and md1 devices appear, udev isn't being run on that.
So it looks like your udev rules file is wrong.
Find out which file(s) in /{etc,lib,usr/lib}/udev/rules.d mention mdadm and
post them.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID50 boot problems

[Roy Sigurd Karlsbakk]

> > Please see http://paste.ubuntu.com/5721934/ for the full list, taken
> > with network console. This is with rootdelay=10
> 
> The "bind" messages are in random order so presumably udev running
> 'mdadm -I'
> on each device as it appear to add it to an array.
> However when the md0 and md1 devices appear, udev isn't being run on
> that.
> So it looks like your udev rules file is wrong.
> Find out which file(s) in /{etc,lib,usr/lib}/udev/rules.d mention
> mdadm and
> post them.

/lib/udev/rules.d/64-md-raid.rules is here http://paste.ubuntu.com/5592227/
-- 
Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID50 boot problems

[Roy Sigurd Karlsbakk]

> > > Please see http://paste.ubuntu.com/5721934/ for the full list,
> > > taken
> > > with network console. This is with rootdelay=10
> >
> > The "bind" messages are in random order so presumably udev running
> > 'mdadm -I'
> > on each device as it appear to add it to an array.
> > However when the md0 and md1 devices appear, udev isn't being run on
> > that.
> > So it looks like your udev rules file is wrong.
> > Find out which file(s) in /{etc,lib,usr/lib}/udev/rules.d mention
> > mdadm and
> > post them.
> 
> /lib/udev/rules.d/64-md-raid.rules is here
> http://paste.ubuntu.com/5592227/

Bug tested positive also on Ubuntu Precise (12.04) and reported to https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1171945

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID50 boot problems

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1620 bytes --]

On Tue, 23 Apr 2013 19:34:19 +0200 (CEST) Roy Sigurd Karlsbakk
<roy@karlsbakk.net> wrote:

> > > > Please see http://paste.ubuntu.com/5721934/ for the full list,
> > > > taken
> > > > with network console. This is with rootdelay=10
> > >
> > > The "bind" messages are in random order so presumably udev running
> > > 'mdadm -I'
> > > on each device as it appear to add it to an array.
> > > However when the md0 and md1 devices appear, udev isn't being run on
> > > that.
> > > So it looks like your udev rules file is wrong.
> > > Find out which file(s) in /{etc,lib,usr/lib}/udev/rules.d mention
> > > mdadm and
> > > post them.
> > 
> > /lib/udev/rules.d/64-md-raid.rules is here
> > http://paste.ubuntu.com/5592227/
> 
> Bug tested positive also on Ubuntu Precise (12.04) and reported to https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1171945
> 
> Vennlige hilsener / Best regards
> 
>

This will run "mdadm --incremental $tempnode" on any device for which
ID_FS_TYPE is set to "linux_raid_member", which certainly seems reasonable.

What does:
   udevadm info --query=property --path=/dev/mdXXX | grep ID_FS_TYPE

report for the raid5 arrays?

Looking bug report I see that md0 and md1 have
   ID_FS_TYPE=linux_raid_member

So that should be working.

The fact that rootdelay=10 makes a difference suggests that it is
successfully assembling the raid0, but just taking a bit too long.
Maybe the script in the initrd needs "udevadm settle" just before it attempts
to mount.

Can you look inside the initrd and see if "udevadm settle" is used anywhere?

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID50 boot problems

[Roy Sigurd Karlsbakk]

> > > /lib/udev/rules.d/64-md-raid.rules is here
> > > http://paste.ubuntu.com/5592227/
> >
> > Bug tested positive also on Ubuntu Precise (12.04) and reported to
> > https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1171945
> 
> This will run "mdadm --incremental $tempnode" on any device for which
> ID_FS_TYPE is set to "linux_raid_member", which certainly seems
> reasonable.
> 
> What does:
> udevadm info --query=property --path=/dev/mdXXX | grep ID_FS_TYPE
> 
> report for the raid5 arrays?

root@raidtest:~# udevadm info --query=property --path=/dev/md0
device path not found
root@raidtest:~# udevadm info --query=property --path=/dev/md1
device path not found

> Looking bug report I see that md0 and md1 have
> ID_FS_TYPE=linux_raid_member
> 
> So that should be working.
> 
> The fact that rootdelay=10 makes a difference suggests that it is
> successfully assembling the raid0, but just taking a bit too long.
> Maybe the script in the initrd needs "udevadm settle" just before it
> attempts
> to mount.
> 
> Can you look inside the initrd and see if "udevadm settle" is used
> anywhere?

mdadm settle doesn't seem to be used there. I really don't know my way around the bootup scripts that well, so I'm unsure where to add it…
-- 
Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID50 boot problems

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1670 bytes --]

On Wed, 24 Apr 2013 13:19:47 +0200 (CEST) Roy Sigurd Karlsbakk
<roy@karlsbakk.net> wrote:

> > > > /lib/udev/rules.d/64-md-raid.rules is here
> > > > http://paste.ubuntu.com/5592227/
> > >
> > > Bug tested positive also on Ubuntu Precise (12.04) and reported to
> > > https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1171945
> > 
> > This will run "mdadm --incremental $tempnode" on any device for which
> > ID_FS_TYPE is set to "linux_raid_member", which certainly seems
> > reasonable.
> > 
> > What does:
> > udevadm info --query=property --path=/dev/mdXXX | grep ID_FS_TYPE
> > 
> > report for the raid5 arrays?
> 
> root@raidtest:~# udevadm info --query=property --path=/dev/md0
> device path not found
> root@raidtest:~# udevadm info --query=property --path=/dev/md1
> device path not found

Sorry, that should be --name, not --path.

> 
> > Looking bug report I see that md0 and md1 have
> > ID_FS_TYPE=linux_raid_member
> > 
> > So that should be working.
> > 
> > The fact that rootdelay=10 makes a difference suggests that it is
> > successfully assembling the raid0, but just taking a bit too long.
> > Maybe the script in the initrd needs "udevadm settle" just before it
> > attempts
> > to mount.
> > 
> > Can you look inside the initrd and see if "udevadm settle" is used
> > anywhere?
> 
> mdadm settle doesn't seem to be used there. I really don't know my way around the bootup scripts that well, so I'm unsure where to add it…

Can you put your initrd (or initramfs or whatever Ubuntu calls it) somewhere
that I can down load it?  Or just email it to me, it's probably only about
10Meg.

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID50 boot problems

[Roy Sigurd Karlsbakk]

> Sorry, that should be --name, not --path.

roy@raidtest:~$ cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 vdc[1] vdb[0] vdd[3]
      2096000 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      
md1 : active raid5 vdf[1] vdg[3] vde[0]
      2096000 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      
unused devices: <none>
roy@raidtest:~$ udevadm info --query=property --name=/dev/md0
DEVLINKS=/dev/disk/by-id/md-name-raidtest:0 /dev/disk/by-id/md-uuid-44285c9c:70d2769c:24009302:979f89fd
DEVNAME=/dev/md0
DEVPATH=/devices/virtual/block/md0
DEVTYPE=disk
ID_FS_LABEL=raidtest:9
ID_FS_LABEL_ENC=raidtest:9
ID_FS_TYPE=linux_raid_member
ID_FS_USAGE=raid
ID_FS_UUID=7ca2b28e-c483-5ef8-29ab-e952221fd4cc
ID_FS_UUID_ENC=7ca2b28e-c483-5ef8-29ab-e952221fd4cc
ID_FS_UUID_SUB=6bc3d0db-9521-c852-cbc2-35b5c398d566
ID_FS_UUID_SUB_ENC=6bc3d0db-9521-c852-cbc2-35b5c398d566
ID_FS_VERSION=1.2
MAJOR=9
MD_DEVICES=3
MD_LEVEL=raid5
MD_METADATA=1.2
MD_NAME=raidtest:0
MD_UUID=44285c9c:70d2769c:24009302:979f89fd
MINOR=0
SUBSYSTEM=block
UDEV_LOG=3
USEC_INITIALIZED=2756441
roy@raidtest:~$ udevadm info --query=property --name=/dev/md1
DEVLINKS=/dev/disk/by-id/md-name-raidtest:1 /dev/disk/by-id/md-uuid-8bd41f4b:080a965a:6b251afc:f4755b7a
DEVNAME=/dev/md1
DEVPATH=/devices/virtual/block/md1
DEVTYPE=disk
ID_FS_LABEL=raidtest:9
ID_FS_LABEL_ENC=raidtest:9
ID_FS_TYPE=linux_raid_member
ID_FS_USAGE=raid
ID_FS_UUID=7ca2b28e-c483-5ef8-29ab-e952221fd4cc
ID_FS_UUID_ENC=7ca2b28e-c483-5ef8-29ab-e952221fd4cc
ID_FS_UUID_SUB=b18fa133-148d-471d-4727-198072ad9dd9
ID_FS_UUID_SUB_ENC=b18fa133-148d-471d-4727-198072ad9dd9
ID_FS_VERSION=1.2
MAJOR=9
MD_DEVICES=3
MD_LEVEL=raid5
MD_METADATA=1.2
MD_NAME=raidtest:1
MD_UUID=8bd41f4b:080a965a:6b251afc:f4755b7a
MINOR=1
SUBSYSTEM=block
UDEV_LOG=3
USEC_INITIALIZED=2760291
roy@raidtest:~$ 

> Can you put your initrd (or initramfs or whatever Ubuntu calls it)
> somewhere
> that I can down load it? Or just email it to me, it's probably only
> about 10Meg.

Here are the ones for 12.04 x86 and 13.04 amd64. The same happen on both

12.04 http://karlsbakk.net/tmp/initrd.img-3.2.0-40-generic-pae
13.04 http://karlsbakk.net/tmp/initrd.img-3.8.0-19-generic

Thanks!

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID50 boot problems

[Roy Sigurd Karlsbakk]

> > Can you put your initrd (or initramfs or whatever Ubuntu calls it)
> > somewhere
> > that I can down load it? Or just email it to me, it's probably only
> > about 10Meg.
> 
> Here are the ones for 12.04 x86 and 13.04 amd64. The same happen on
> both
> 
> 12.04 http://karlsbakk.net/tmp/initrd.img-3.2.0-40-generic-pae
> 13.04 http://karlsbakk.net/tmp/initrd.img-3.8.0-19-generic

Did you have time to looke into this?

thanks

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID50 boot problems

[Dmitrijs Ledkovs]

On 24 April 2013 07:52, NeilBrown <neilb@suse.de> wrote:
> On Tue, 23 Apr 2013 19:34:19 +0200 (CEST) Roy Sigurd Karlsbakk
> <roy@karlsbakk.net> wrote:
>
>> > > > Please see http://paste.ubuntu.com/5721934/ for the full list,
>> > > > taken
>> > > > with network console. This is with rootdelay=10
>> > >
>> > > The "bind" messages are in random order so presumably udev running
>> > > 'mdadm -I'
>> > > on each device as it appear to add it to an array.
>> > > However when the md0 and md1 devices appear, udev isn't being run on
>> > > that.
>> > > So it looks like your udev rules file is wrong.
>> > > Find out which file(s) in /{etc,lib,usr/lib}/udev/rules.d mention
>> > > mdadm and
>> > > post them.
>> >
>> > /lib/udev/rules.d/64-md-raid.rules is here
>> > http://paste.ubuntu.com/5592227/
>>
>> Bug tested positive also on Ubuntu Precise (12.04) and reported to https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1171945
>>
>> Vennlige hilsener / Best regards
>>
>>
>
> This will run "mdadm --incremental $tempnode" on any device for which
> ID_FS_TYPE is set to "linux_raid_member", which certainly seems reasonable.
>
> What does:
>    udevadm info --query=property --path=/dev/mdXXX | grep ID_FS_TYPE
>
> report for the raid5 arrays?
>
> Looking bug report I see that md0 and md1 have
>    ID_FS_TYPE=linux_raid_member
>
> So that should be working.
>
> The fact that rootdelay=10 makes a difference suggests that it is
> successfully assembling the raid0, but just taking a bit too long.
> Maybe the script in the initrd needs "udevadm settle" just before it attempts
> to mount.
>
> Can you look inside the initrd and see if "udevadm settle" is used anywhere?
>

Yes, we do call and wait for udevadm to settle a few times, but it is
still too short and may not be long enough to detect nested raid
volumes and mount them properly in the correct order and non-degraded.
I have a few thoughts on using a strategy similar to that in dracut /
fedora to pass ids of the md arrays to assemble for rootfs device, and
keep trying to assemble the rest of mdadm "on best effort" basis
during boot.
That way I am also hoping to finally get rid of the dreaded "boot
degraded" boot option / question / prompt.
This is still just design in progress and hasn't been implemented yet.
I will be contacting this mailing list once I have something ready to
improve raid assembly in ubuntu.

Regards,

Dmitrijs.

Re: RAID50 boot problems

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2108 bytes --]

On Wed, 24 Apr 2013 14:29:39 +0200 (CEST) Roy Sigurd Karlsbakk
<roy@karlsbakk.net> wrote:

> > Sorry, that should be --name, not --path.
> 
> roy@raidtest:~$ cat /proc/mdstat 
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
> md0 : active raid5 vdc[1] vdb[0] vdd[3]
>       2096000 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
>       
> md1 : active raid5 vdf[1] vdg[3] vde[0]
>       2096000 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
>       
> unused devices: <none>
> roy@raidtest:~$ udevadm info --query=property --name=/dev/md0
.....
> ID_FS_TYPE=linux_raid_member
....
> roy@raidtest:~$ udevadm info --query=property --name=/dev/md1
...
> ID_FS_TYPE=linux_raid_member
....

So that all looks good.

> > Can you put your initrd (or initramfs or whatever Ubuntu calls it)
> > somewhere
> > that I can down load it? Or just email it to me, it's probably only
> > about 10Meg.
> 
> Here are the ones for 12.04 x86 and 13.04 amd64. The same happen on both
> 
> 12.04 http://karlsbakk.net/tmp/initrd.img-3.2.0-40-generic-pae
> 13.04 http://karlsbakk.net/tmp/initrd.img-3.8.0-19-generic
> 

Strangely etc/mdadm/mdadm.conf in the 12.04 image lists the 2 RAID5s and the
RAID0, but the file in 13.04 only lists to 2 RAID5s.  I don't think that
makes a difference, but it is strange.

scripts/local-premount/mdadm contains "wait_for_udev" so that should be
enough...

Wait a bit.  I just noticed that 64-md-raid.rules only runs 
   /sbin/mdadm --incremental $tempnode
on ACTION=="add".
The RAID5 arrays aren't ready immediately and you need to catch 
   ACTION=="change"
Yes, that's horrible and inconsistent but that is life.

It would be worth adding an extra line:

ACTION=="change", RUN+="/sbin/mdadm --incremental $tempnode"

I'm not sure how to do that.  Maybe just modify the file in the root
filesystem and  run mkinitrd or mkinitramfs or whatever the command is.

Though if that is the problem, then I cannot see what just setting a
rootdelay would help.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID50 boot problems

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3638 bytes --]

On Wed, 24 Apr 2013 23:44:20 +0100 Dmitrijs Ledkovs <xnox@debian.org> wrote:

> On 24 April 2013 07:52, NeilBrown <neilb@suse.de> wrote:
> > On Tue, 23 Apr 2013 19:34:19 +0200 (CEST) Roy Sigurd Karlsbakk
> > <roy@karlsbakk.net> wrote:
> >
> >> > > > Please see http://paste.ubuntu.com/5721934/ for the full list,
> >> > > > taken
> >> > > > with network console. This is with rootdelay=10
> >> > >
> >> > > The "bind" messages are in random order so presumably udev running
> >> > > 'mdadm -I'
> >> > > on each device as it appear to add it to an array.
> >> > > However when the md0 and md1 devices appear, udev isn't being run on
> >> > > that.
> >> > > So it looks like your udev rules file is wrong.
> >> > > Find out which file(s) in /{etc,lib,usr/lib}/udev/rules.d mention
> >> > > mdadm and
> >> > > post them.
> >> >
> >> > /lib/udev/rules.d/64-md-raid.rules is here
> >> > http://paste.ubuntu.com/5592227/
> >>
> >> Bug tested positive also on Ubuntu Precise (12.04) and reported to https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1171945
> >>
> >> Vennlige hilsener / Best regards
> >>
> >>
> >
> > This will run "mdadm --incremental $tempnode" on any device for which
> > ID_FS_TYPE is set to "linux_raid_member", which certainly seems reasonable.
> >
> > What does:
> >    udevadm info --query=property --path=/dev/mdXXX | grep ID_FS_TYPE
> >
> > report for the raid5 arrays?
> >
> > Looking bug report I see that md0 and md1 have
> >    ID_FS_TYPE=linux_raid_member
> >
> > So that should be working.
> >
> > The fact that rootdelay=10 makes a difference suggests that it is
> > successfully assembling the raid0, but just taking a bit too long.
> > Maybe the script in the initrd needs "udevadm settle" just before it attempts
> > to mount.
> >
> > Can you look inside the initrd and see if "udevadm settle" is used anywhere?
> >
> 
> Yes, we do call and wait for udevadm to settle a few times, but it is
> still too short and may not be long enough to detect nested raid
> volumes and mount them properly in the correct order and non-degraded.
> I have a few thoughts on using a strategy similar to that in dracut /
> fedora to pass ids of the md arrays to assemble for rootfs device, and
> keep trying to assemble the rest of mdadm "on best effort" basis
> during boot.
> That way I am also hoping to finally get rid of the dreaded "boot
> degraded" boot option / question / prompt.
> This is still just design in progress and hasn't been implemented yet.
> I will be contacting this mailing list once I have something ready to
> improve raid assembly in ubuntu.
> 

My current thinking is that the initramfs should *only* assemble arrays needed
to mount the root filesystems.  All other arrays should wait for root to be
mounted so that real /etc/mdadm.conf (or /etc/mdadm/mdadm.conf) can be
consulted.
This can be achieved by putting
  auto -all
in mdadm.conf on the initramfs, then listing the arrays that are needed.

I'm not convinced that your boot-degraded option is a bad thing.  Certainly
it should be optional so unattended boot is possible, and we should do our
best to minimise the number of times that it is consulted.  But there are
times when it is better to know that something is wrong, than to proceed and
do the wrong thing.

A particularly bad case is a RAID1 pair where one device failed a few days
ago.
If after a reboot the good device is missing (cable problem?) and the bad
device is visible, it could be best not to boot rather than to boot with an
old root based on the old  'failed' device.

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID50 boot problems

[Roy Sigurd Karlsbakk]

> My current thinking is that the initramfs should *only* assemble
> arrays needed
> to mount the root filesystems. All other arrays should wait for root
> to be
> mounted so that real /etc/mdadm.conf (or /etc/mdadm/mdadm.conf) can be
> consulted.
> This can be achieved by putting
> auto -all
> in mdadm.conf on the initramfs, then listing the arrays that are
> needed.
> 
> I'm not convinced that your boot-degraded option is a bad thing.
> Certainly
> it should be optional so unattended boot is possible, and we should do
> our
> best to minimise the number of times that it is consulted. But there
> are
> times when it is better to know that something is wrong, than to
> proceed and
> do the wrong thing.
> 
> A particularly bad case is a RAID1 pair where one device failed a few
> days
> ago.
> If after a reboot the good device is missing (cable problem?) and the
> bad
> device is visible, it could be best not to boot rather than to boot
> with an
> old root based on the old 'failed' device.


Anyone that knows more about this issue and how to fix it? I've filed a bug, but haven't gotten any more feedback

-- 
Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID50 boot problems

[Dmitrijs Ledkovs]

On 25 April 2013 00:44, NeilBrown <neilb@suse.de> wrote:
> On Wed, 24 Apr 2013 23:44:20 +0100 Dmitrijs Ledkovs <xnox@debian.org> wrote:
>
>> On 24 April 2013 07:52, NeilBrown <neilb@suse.de> wrote:
>> > On Tue, 23 Apr 2013 19:34:19 +0200 (CEST) Roy Sigurd Karlsbakk
>> > <roy@karlsbakk.net> wrote:
>> >
>> >> > > > Please see http://paste.ubuntu.com/5721934/ for the full list,
>> >> > > > taken
>> >> > > > with network console. This is with rootdelay=10
>> >> > >
>> >> > > The "bind" messages are in random order so presumably udev running
>> >> > > 'mdadm -I'
>> >> > > on each device as it appear to add it to an array.
>> >> > > However when the md0 and md1 devices appear, udev isn't being run on
>> >> > > that.
>> >> > > So it looks like your udev rules file is wrong.
>> >> > > Find out which file(s) in /{etc,lib,usr/lib}/udev/rules.d mention
>> >> > > mdadm and
>> >> > > post them.
>> >> >
>> >> > /lib/udev/rules.d/64-md-raid.rules is here
>> >> > http://paste.ubuntu.com/5592227/
>> >>
>> >> Bug tested positive also on Ubuntu Precise (12.04) and reported to https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1171945
>> >>
>> >> Vennlige hilsener / Best regards
>> >>
>> >>
>> >
>> > This will run "mdadm --incremental $tempnode" on any device for which
>> > ID_FS_TYPE is set to "linux_raid_member", which certainly seems reasonable.
>> >
>> > What does:
>> >    udevadm info --query=property --path=/dev/mdXXX | grep ID_FS_TYPE
>> >
>> > report for the raid5 arrays?
>> >
>> > Looking bug report I see that md0 and md1 have
>> >    ID_FS_TYPE=linux_raid_member
>> >
>> > So that should be working.
>> >
>> > The fact that rootdelay=10 makes a difference suggests that it is
>> > successfully assembling the raid0, but just taking a bit too long.
>> > Maybe the script in the initrd needs "udevadm settle" just before it attempts
>> > to mount.
>> >
>> > Can you look inside the initrd and see if "udevadm settle" is used anywhere?
>> >
>>
>> Yes, we do call and wait for udevadm to settle a few times, but it is
>> still too short and may not be long enough to detect nested raid
>> volumes and mount them properly in the correct order and non-degraded.
>> I have a few thoughts on using a strategy similar to that in dracut /
>> fedora to pass ids of the md arrays to assemble for rootfs device, and
>> keep trying to assemble the rest of mdadm "on best effort" basis
>> during boot.
>> That way I am also hoping to finally get rid of the dreaded "boot
>> degraded" boot option / question / prompt.
>> This is still just design in progress and hasn't been implemented yet.
>> I will be contacting this mailing list once I have something ready to
>> improve raid assembly in ubuntu.
>>
>
> My current thinking is that the initramfs should *only* assemble arrays needed
> to mount the root filesystems.  All other arrays should wait for root to be
> mounted so that real /etc/mdadm.conf (or /etc/mdadm/mdadm.conf) can be
> consulted.
> This can be achieved by putting
>   auto -all
> in mdadm.conf on the initramfs, then listing the arrays that are needed.
>

That's an easy way to do it. One property of Ubuntu's initramfs at the
moment is that they are more or less generic, one can reuse the same
one to boot similar machines.
I'm not sure if it was a requirement at any point, but many initramfs
& faster boot speed design choices led to that.
And ideally I'd like to keep that property. Hence I'd like to ideally
pass the arrays to be assembled as a kernel arg and not store it in
the initramfs.

> I'm not convinced that your boot-degraded option is a bad thing.  Certainly
> it should be optional so unattended boot is possible, and we should do our
> best to minimise the number of times that it is consulted.  But there are
> times when it is better to know that something is wrong, than to proceed and
> do the wrong thing.
>

Well, the default is to not boot-degraded and instead a configuration
question is asked upon installation if one would like to boot no
matter what.
(deb packages allow that via debconf, unlike usual rpm packages).

> A particularly bad case is a RAID1 pair where one device failed a few days
> ago.
> If after a reboot the good device is missing (cable problem?) and the bad
> device is visible, it could be best not to boot rather than to boot with an
> old root based on the old  'failed' device.
>

This is the ultimate reason for not booting degraded by default. But
unfortunately we still hit too many "false positive" degraded
scenarios as the moment, thus the opponents to the current strategy.

Regards,

Dmitrijs.