help: re-add needed after each reboot

help: re-add needed after each reboot

[D. Lin]

Hi,

I am running 3.2.0 (ubuntu 12.04). My raid5 has 3 disks (sd[cbd]1) and no
spares. The system boots off a non-raid device and auto discovers and
mounts the raid device.

For some reason, sdc1 is not automatically detected and is always missed
on each boot. I have to re-add it back. What could be causing this?

This did not happen while I was running 2.6.38.

Thanks,

Dong Lin


; mdadm --misc --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Fri Jul 22 22:39:40 2011
     Raid Level : raid5
     Array Size : 3907020800 (3726.03 GiB 4000.79 GB)
  Used Dev Size : 1953510400 (1863.01 GiB 2000.39 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Mon May  7 09:21:29 2012
          State : active
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : disc:0
           UUID : 4195aaa7:0ebd37ce:d4b9481e:1a68f4a8
         Events : 230784

    Number   Major   Minor   RaidDevice State
       4       8       33        0      active sync   /dev/sdc1
       2       8       49        1      active sync   /dev/sdd1
       3       8       17        2      active sync   /dev/sdb1


----------------------------

here is what happened on boot:

; dmesg |grep md
[    1.781917] md: linear personality registered for level -1
[    1.789184] md: multipath personality registered for level -4
[    1.797121] md: raid0 personality registered for level 0
[    1.801681] md: raid1 personality registered for level 1
[    1.820485] ata7: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xfa00
irq 14
[    1.820547] ata8: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xfa08
irq 15
[    1.920271] md: bind<sdb1>
[    1.962482] md: bind<sdd1>
[    2.300917] md: raid6 personality registered for level 6
[    2.300980] md: raid5 personality registered for level 5
[    2.301036] md: raid4 personality registered for level 4
[    2.306171] md: raid10 personality registered for level 10
[    2.329393] md/raid:md0: device sdd1 operational as raid disk 1
[    2.329450] md/raid:md0: device sdb1 operational as raid disk 2
[    2.329792] md/raid:md0: allocated 3228kB
[    2.329921] md/raid:md0: raid level 5 active with 2 out of 3 devices,
algorithm 2
[    2.330235] created bitmap (15 pages) for device md0
[    2.330706] md0: bitmap initialized from disk: read 1/1 pages, set 0 of
29809 bits
[    2.362404] md0: detected capacity change from 0 to 4000789299200
[    2.367990]  md0: unknown partition table



=======================================
partition tables:

; fdisk -l | egrep '^/dev/sd[cbd]'
/dev/sdb1              63  3907024064  1953512001   fd  Linux raid autodetect
/dev/sdc1              63  3907024064  1953512001   fd  Linux raid autodetect
/dev/sdd1              63  3907024064  1953512001   fd  Linux raid autodetect

Re: help: re-add needed after each reboot

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3844 bytes --]

On Mon, 7 May 2012 09:29:03 -0700 "D. Lin" <dlbulk-mllr@yahoo.com> wrote:

> Hi,
> 
> I am running 3.2.0 (ubuntu 12.04). My raid5 has 3 disks (sd[cbd]1) and no
> spares. The system boots off a non-raid device and auto discovers and
> mounts the raid device.
> 
> For some reason, sdc1 is not automatically detected and is always missed
> on each boot. I have to re-add it back. What could be causing this?
> 
> This did not happen while I was running 2.6.38.

Would it be equally accurate to say that it didn't happen when you were
running ubunutu 11.10??  My point is that maybe the important change is not
in the kernel.

What do you have in /etc/mdadm.conf ? (or maybe /etc/mdadm/mdadm.conf)
If you stop the array, then

  mdadm -Asvvv

what messages are generated, and what is the result?

NeilBrown


> 
> Thanks,
> 
> Dong Lin
> 
> 
> ; mdadm --misc --detail /dev/md0
> /dev/md0:
>         Version : 1.2
>   Creation Time : Fri Jul 22 22:39:40 2011
>      Raid Level : raid5
>      Array Size : 3907020800 (3726.03 GiB 4000.79 GB)
>   Used Dev Size : 1953510400 (1863.01 GiB 2000.39 GB)
>    Raid Devices : 3
>   Total Devices : 3
>     Persistence : Superblock is persistent
> 
>   Intent Bitmap : Internal
> 
>     Update Time : Mon May  7 09:21:29 2012
>           State : active
>  Active Devices : 3
> Working Devices : 3
>  Failed Devices : 0
>   Spare Devices : 0
> 
>          Layout : left-symmetric
>      Chunk Size : 512K
> 
>            Name : disc:0
>            UUID : 4195aaa7:0ebd37ce:d4b9481e:1a68f4a8
>          Events : 230784
> 
>     Number   Major   Minor   RaidDevice State
>        4       8       33        0      active sync   /dev/sdc1
>        2       8       49        1      active sync   /dev/sdd1
>        3       8       17        2      active sync   /dev/sdb1
> 
> 
> ----------------------------
> 
> here is what happened on boot:
> 
> ; dmesg |grep md
> [    1.781917] md: linear personality registered for level -1
> [    1.789184] md: multipath personality registered for level -4
> [    1.797121] md: raid0 personality registered for level 0
> [    1.801681] md: raid1 personality registered for level 1
> [    1.820485] ata7: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xfa00
> irq 14
> [    1.820547] ata8: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xfa08
> irq 15
> [    1.920271] md: bind<sdb1>
> [    1.962482] md: bind<sdd1>
> [    2.300917] md: raid6 personality registered for level 6
> [    2.300980] md: raid5 personality registered for level 5
> [    2.301036] md: raid4 personality registered for level 4
> [    2.306171] md: raid10 personality registered for level 10
> [    2.329393] md/raid:md0: device sdd1 operational as raid disk 1
> [    2.329450] md/raid:md0: device sdb1 operational as raid disk 2
> [    2.329792] md/raid:md0: allocated 3228kB
> [    2.329921] md/raid:md0: raid level 5 active with 2 out of 3 devices,
> algorithm 2
> [    2.330235] created bitmap (15 pages) for device md0
> [    2.330706] md0: bitmap initialized from disk: read 1/1 pages, set 0 of
> 29809 bits
> [    2.362404] md0: detected capacity change from 0 to 4000789299200
> [    2.367990]  md0: unknown partition table
> 
> 
> 
> =======================================
> partition tables:
> 
> ; fdisk -l | egrep '^/dev/sd[cbd]'
> /dev/sdb1              63  3907024064  1953512001   fd  Linux raid autodetect
> /dev/sdc1              63  3907024064  1953512001   fd  Linux raid autodetect
> /dev/sdd1              63  3907024064  1953512001   fd  Linux raid autodetect
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: help: re-add needed after each reboot

[D. Lin]

Thanks for your help.

>
> Would it be equally accurate to say that it didn't happen when you were
> running ubunutu 11.10??  My point is that maybe the important change is
> not in the kernel.
>

My system was running 11.04 when things worked. I skipped 11.10.


> What do you have in /etc/mdadm.conf ? (or maybe /etc/mdadm/mdadm.conf)

# egrep -v '^#|^$' /etc/mdadm/mdadm.conf
DEVICE partitions
CREATE owner=root group=disk mode=0660 auto=yes
HOMEHOST <system>


> If you stop the array, then
>
>   mdadm -Asvvv
>
> what messages are generated, and what is the result?
>

root@anguish:/# mdadm --misc --stop /dev/md0
mdadm: stopped /dev/md0

[195415.073137] md0: detected capacity change from 4000789299200 to 0
[195415.073150] md: md0 stopped.
[195415.073166] md: unbind<sdc1>
[195415.092175] md: export_rdev(sdc1)
[195415.092210] md: unbind<sdd1>
[195415.092413] md: export_rdev(sdd1)
[195415.092533] md: unbind<sdb1>
[195415.104245] md: export_rdev(sdb1)


# mdadm -Asvvv
mdadm: looking for devices for further assembly
mdadm: no RAID superblock on /dev/sdd
mdadm: no RAID superblock on /dev/sdc
mdadm: no RAID superblock on /dev/sdb
mdadm: /dev/sdd1 is identified as a member of /dev/md/disc:0, slot 1.
mdadm: /dev/sdc1 is identified as a member of /dev/md/disc:0, slot 0.
mdadm: /dev/sdb1 is identified as a member of /dev/md/disc:0, slot 2.
mdadm: added /dev/sdd1 to /dev/md/disc:0 as 1
mdadm: added /dev/sdb1 to /dev/md/disc:0 as 2
mdadm: added /dev/sdc1 to /dev/md/disc:0 as 0
mdadm: /dev/md/disc:0 has been started with 3 drives.
mdadm: looking for devices for further assembly
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/md/disc:0
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: cannot open device /dev/sdb: Device or resource busy

root@anguish:/# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md127 : active raid5 sdc1[4] sdb1[3] sdd1[2]
      3907020800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3]
[UUU]
      bitmap: 0/15 pages [0KB], 65536KB chunk

unused devices: <none>


please note that the md device is now md127, as opposed to md0. When
2.6.38 was running, md127 was auto discovered.


Now, I reboot the system. md0 is auto discovered and degraded.

[    1.649849] md: linear personality registered for level -1
[    1.658472] md: multipath personality registered for level -4
[    1.665077] md: raid0 personality registered for level 0
[    1.669855] md: raid1 personality registered for level 1
[    1.869347] md: bind<sdd1>
[    2.157297] md: bind<sdb1>
[    2.168850] md: raid6 personality registered for level 6
[    2.168923] md: raid5 personality registered for level 5
[    2.168999] md: raid4 personality registered for level 4
[    2.352210] md: raid10 personality registered for level 10
[    2.378106] md/raid:md0: device sdb1 operational as raid disk 2
[    2.378158] md/raid:md0: device sdd1 operational as raid disk 1
[    2.378503] md/raid:md0: allocated 3228kB
[    2.378629] md/raid:md0: raid level 5 active with 2 out of 3 devices,
algorithm 2
[    2.378939] created bitmap (15 pages) for device md0
[    2.379386] md0: bitmap initialized from disk: read 1/1 pages, set 0 of
29809 bits
[    2.401485] md0: detected capacity change from 0 to 4000789299200
[    2.410364]  md0: unknown partition table


$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : active raid5 sdb1[3] sdd1[2]
      3907020800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2]
[_UU]
      bitmap: 7/15 pages [28KB], 65536KB chunk

Re: help: re-add needed after each reboot

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 4546 bytes --]

On Mon, 7 May 2012 23:41:34 -0700 "D. Lin" <dlbulk-mllr@yahoo.com> wrote:

> Thanks for your help.
> 
> >
> > Would it be equally accurate to say that it didn't happen when you were
> > running ubunutu 11.10??  My point is that maybe the important change is
> > not in the kernel.
> >
> 
> My system was running 11.04 when things worked. I skipped 11.10.
> 
> 
> > What do you have in /etc/mdadm.conf ? (or maybe /etc/mdadm/mdadm.conf)
> 
> # egrep -v '^#|^$' /etc/mdadm/mdadm.conf
> DEVICE partitions
> CREATE owner=root group=disk mode=0660 auto=yes
> HOMEHOST <system>
> 
> 
> > If you stop the array, then
> >
> >   mdadm -Asvvv
> >
> > what messages are generated, and what is the result?
> >
> 
> root@anguish:/# mdadm --misc --stop /dev/md0
> mdadm: stopped /dev/md0
> 
> [195415.073137] md0: detected capacity change from 4000789299200 to 0
> [195415.073150] md: md0 stopped.
> [195415.073166] md: unbind<sdc1>
> [195415.092175] md: export_rdev(sdc1)
> [195415.092210] md: unbind<sdd1>
> [195415.092413] md: export_rdev(sdd1)
> [195415.092533] md: unbind<sdb1>
> [195415.104245] md: export_rdev(sdb1)
> 
> 
> # mdadm -Asvvv
> mdadm: looking for devices for further assembly
> mdadm: no RAID superblock on /dev/sdd
> mdadm: no RAID superblock on /dev/sdc
> mdadm: no RAID superblock on /dev/sdb
> mdadm: /dev/sdd1 is identified as a member of /dev/md/disc:0, slot 1.
> mdadm: /dev/sdc1 is identified as a member of /dev/md/disc:0, slot 0.
> mdadm: /dev/sdb1 is identified as a member of /dev/md/disc:0, slot 2.
> mdadm: added /dev/sdd1 to /dev/md/disc:0 as 1
> mdadm: added /dev/sdb1 to /dev/md/disc:0 as 2
> mdadm: added /dev/sdc1 to /dev/md/disc:0 as 0
> mdadm: /dev/md/disc:0 has been started with 3 drives.
> mdadm: looking for devices for further assembly
> mdadm: looking for devices for further assembly
> mdadm: no recogniseable superblock on /dev/md/disc:0
> mdadm: cannot open device /dev/sdd1: Device or resource busy
> mdadm: cannot open device /dev/sdd: Device or resource busy
> mdadm: cannot open device /dev/sdc1: Device or resource busy
> mdadm: cannot open device /dev/sdc: Device or resource busy
> mdadm: cannot open device /dev/sdb1: Device or resource busy
> mdadm: cannot open device /dev/sdb: Device or resource busy
> 
> root@anguish:/# cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md127 : active raid5 sdc1[4] sdb1[3] sdd1[2]
>       3907020800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3]
> [UUU]
>       bitmap: 0/15 pages [0KB], 65536KB chunk
> 
> unused devices: <none>
> 
> 
> please note that the md device is now md127, as opposed to md0. When
> 2.6.38 was running, md127 was auto discovered.
> 
> 
> Now, I reboot the system. md0 is auto discovered and degraded.
> 
> [    1.649849] md: linear personality registered for level -1
> [    1.658472] md: multipath personality registered for level -4
> [    1.665077] md: raid0 personality registered for level 0
> [    1.669855] md: raid1 personality registered for level 1
> [    1.869347] md: bind<sdd1>
> [    2.157297] md: bind<sdb1>
> [    2.168850] md: raid6 personality registered for level 6
> [    2.168923] md: raid5 personality registered for level 5
> [    2.168999] md: raid4 personality registered for level 4
> [    2.352210] md: raid10 personality registered for level 10
> [    2.378106] md/raid:md0: device sdb1 operational as raid disk 2
> [    2.378158] md/raid:md0: device sdd1 operational as raid disk 1
> [    2.378503] md/raid:md0: allocated 3228kB
> [    2.378629] md/raid:md0: raid level 5 active with 2 out of 3 devices,
> algorithm 2
> [    2.378939] created bitmap (15 pages) for device md0
> [    2.379386] md0: bitmap initialized from disk: read 1/1 pages, set 0 of
> 29809 bits
> [    2.401485] md0: detected capacity change from 0 to 4000789299200
> [    2.410364]  md0: unknown partition table
> 
> 
> $ cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md0 : active raid5 sdb1[3] sdd1[2]
>       3907020800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2]
> [_UU]
>       bitmap: 7/15 pages [28KB], 65536KB chunk
> 
>

Thanks for the extra detail.  Everything looks sensible so far.

Can you check the contents of mdadm.conf on the initrd please.
e.g.

   cd /tmp
   zcat /boot/initrd | cpio -idv
   cat etc/mdadm/mdadm.conf

something like that.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: help: re-add needed after each reboot

[D. Lin]

Thanks Neil.

>
> Thanks for the extra detail.  Everything looks sensible so far.
>
> Can you check the contents of mdadm.conf on the initrd please.
> e.g.
>
>    cd /tmp
>    zcat /boot/initrd | cpio -idv
>    cat etc/mdadm/mdadm.conf
>
> something like that.
>
> NeilBrown
>

content of /boot/initrd.img-3.2.0-24-generic:

; egrep -v '^#|^$' etc/mdadm/mdadm.conf
CREATE owner=root group=disk mode=0660 auto=yes
HOMEHOST <system>
ARRAY /dev/md/0 metadata=1.2 UUID=4195aaa7:0ebd37ce:d4b9481e:1a68f4a8
name=disc:0

Re: help: re-add needed after each reboot

[Bill Davidsen]

D. Lin wrote:
> Hi,
>
> I am running 3.2.0 (ubuntu 12.04). My raid5 has 3 disks (sd[cbd]1) and no
> spares. The system boots off a non-raid device and auto discovers and
> mounts the raid device.
>
> For some reason, sdc1 is not automatically detected and is always missed
> on each boot. I have to re-add it back. What could be causing this?
>
> This did not happen while I was running 2.6.38.
>
Unless you can get back to that kernel and the problem goes away, that's likely 
to be irrelevant.

If this only happens on power cycle I would suspect a slow spin-up on one drive, 
otherwise that's not it. Be any chance is that drive, and only that drive, set 
to spin down when not in use?

Check partition types. That shouldn't be an issue, but check anyway.

Have you run SMART on the drives? Ignore this idea if you don't see the problem 
when booting off the old kernel. Just because you changed software and the 
problem occurred, don't completely rule out hardware, or a subtle config problem.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot