Bugreport: mdmon will not start with latest git version when using ddf container

Bugreport: mdmon will not start with latest git version when using ddf container

[Albert Pauw]

Seems mdmon will not start when I create a ddf container with raid 
devices therein:

I am using this version from git:

mdadm - v3.3-rc2-20-gd81cc6a - 31st July 2013

on Centos 6.4, 2.6.32-358.14.1.el6.x86_64

I am using 5 loop devices of 512 MB each for disks.
I created a container with 5 disks:

mdadm -CR /dev/md127 -e ddf -l container -n 5 /dev/loop[1-5]

[root@moonlight ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md127 : inactive loop5[4](S) loop4[3](S) loop3[2](S) loop2[1](S) loop1[0](S)
       163840 blocks super external:ddf

unused devices: <none>

And created two raid sets in it, one with three disks in raid 5, the 
other a raid 1

[root@moonlight ~]# mdadm -CR /dev/md1 -l raid5 -n 3 /dev/md127
mdadm: array /dev/md1 started.
mdadm: failed to launch mdmon. Array remains readonly
[root@moonlight ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md1 : active (read-only) raid5 loop1[2] loop2[1] loop3[0]
       958464 blocks super external:/md127/1 level 5, 512k chunk, 
algorithm 2 [3/3] [UUU]
           resync=PENDING

md0 : active (read-only) raid1 loop4[1] loop5[0]
       479232 blocks super external:/md127/0 [2/2] [UU]
           resync=PENDING

md127 : inactive loop5[4](S) loop4[3](S) loop3[2](S) loop2[1](S) loop1[0](S)
       163840 blocks super external:ddf

unused devices: <none>

And adding another one, raid5 in my case, gives the same result for the 
next /dev/md device.

Unfortunately, they stay readonly, as mdmon won't start, so mdadm 
--readwrite /dev/md0, etc, won't work.

However, when I create these raid devices directly on top of the 
/dev/loop[1-5] devices, without using a ddf container, it works as 
advertised and the devices get (re)build/resynced.



Regards,

Albert

Re: Bugreport: mdmon will not start with latest git version when using ddf container

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2515 bytes --]

On Wed, 31 Jul 2013 20:14:18 +0200 Albert Pauw <albert.pauw@gmail.com> wrote:

> Seems mdmon will not start when I create a ddf container with raid 
> devices therein:
> 
> I am using this version from git:
> 
> mdadm - v3.3-rc2-20-gd81cc6a - 31st July 2013
> 
> on Centos 6.4, 2.6.32-358.14.1.el6.x86_64
> 
> I am using 5 loop devices of 512 MB each for disks.
> I created a container with 5 disks:
> 
> mdadm -CR /dev/md127 -e ddf -l container -n 5 /dev/loop[1-5]
> 
> [root@moonlight ~]# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [raid1]
> md127 : inactive loop5[4](S) loop4[3](S) loop3[2](S) loop2[1](S) loop1[0](S)
>        163840 blocks super external:ddf
> 
> unused devices: <none>
> 
> And created two raid sets in it, one with three disks in raid 5, the 
> other a raid 1
> 
> [root@moonlight ~]# mdadm -CR /dev/md1 -l raid5 -n 3 /dev/md127
> mdadm: array /dev/md1 started.
> mdadm: failed to launch mdmon. Array remains readonly
> [root@moonlight ~]# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [raid1]
> md1 : active (read-only) raid5 loop1[2] loop2[1] loop3[0]
>        958464 blocks super external:/md127/1 level 5, 512k chunk, 
> algorithm 2 [3/3] [UUU]
>            resync=PENDING
> 
> md0 : active (read-only) raid1 loop4[1] loop5[0]
>        479232 blocks super external:/md127/0 [2/2] [UU]
>            resync=PENDING
> 
> md127 : inactive loop5[4](S) loop4[3](S) loop3[2](S) loop2[1](S) loop1[0](S)
>        163840 blocks super external:ddf
> 
> unused devices: <none>
> 
> And adding another one, raid5 in my case, gives the same result for the 
> next /dev/md device.
> 
> Unfortunately, they stay readonly, as mdmon won't start, so mdadm 
> --readwrite /dev/md0, etc, won't work.
> 
> However, when I create these raid devices directly on top of the 
> /dev/loop[1-5] devices, without using a ddf container, it works as 
> advertised and the devices get (re)build/resynced.
> 

Thanks for the report.
This works when I try it so we must be doing something slightly differently.

When mdadm tries to run mdmon it looks in:
  - the same directory that 'mdadm' was found in
  - /sbin
  - the current directory.

Where are mdadm and mdmon installed?  What directory are you running from.
If we can't discover the problem from that information, please run
   strace -f -o /tmp/md-trace mdadm -CR /dev/md1 -l raid5 -n 3 /dev/md127

and post the resulting /tmp/md-trace file.

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Bugreport: mdmon will not start with latest git version when using ddf container

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 451 bytes --]

On Thu, 1 Aug 2013 07:18:53 +0200 Albert Pauw <albert.pauw@gmail.com> wrote:

> Hi Neil,
> 
> both mdadm and mdmon come from the same location, /sbin, and are the newly
> created one.
> 
> I've attached the strace file. Hopefully you can find any clues in there.
> 

Yes I can.

mdmon needs to create /run/mdadm but you don't have '/run'.
try:
 make clean
 make MAP_DIR=/var/run/mdadm
or maybe
 make MAP_DIR=/dev/.mdadm

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Bugreport: mdmon will not start with latest git version when using ddf container

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

On Thu, 1 Aug 2013 08:15:36 +0200 Albert Pauw <albert.pauw@gmail.com> wrote:

> Hi Neil,
> 
> used /var/run/mdadm, and yes it is working now.
> 
> Thanks,
> 
> Albert
> 
> P.S. Should this variable not set default to something like /var/run/mdadm
> or /dev?
> 

I believe that most distros are moving towards /run so I think that is the
best default.
I guess I could get the Makefile to check that the chosen directory exists...

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Bugreport: mdmon will not start with latest git version when using ddf container

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 611 bytes --]

On Thu, 1 Aug 2013 10:00:56 +0200 Albert Pauw <albert.pauw@gmail.com> wrote:

> Hi Neil,
> 
> just using standard Centos 6.4, looks like RedHat and its derivatives are
> not going the /run route for now.
> 
> Yes, checking if it exists should be good. If not, resort to /var/run, and
> when that does not exist, fall back to /dev.
> That last step may be a bit superfluous, but a very safe fallback.
> 

I rather not do too much guessing..
For now I've changed the Makefile so that if the relevant directory doesn't
exist it will fail and tell you how to avoid the failure.

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]