Accesses to not yet running array

Accesses to not yet running array

[Francis Moreau]

hi,

It looks like a process wait uninterruptibly when it tries to access
an array which is not running yet.

Furthermore once the process is waiting, I can't start/stop the array anymore.

So I need to reboot my system and be sure next time that any processes
don't try to access a not yet running array.

Is that expected ?

Thanks
-- 
Francis

Re: Accesses to not yet running array

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

On Tue, 20 Aug 2013 16:08:47 +0200 Francis Moreau <francis.moro@gmail.com>
wrote:

> hi,
> 
> It looks like a process wait uninterruptibly when it tries to access
> an array which is not running yet.
> 
> Furthermore once the process is waiting, I can't start/stop the array anymore.
> 
> So I need to reboot my system and be sure next time that any processes
> don't try to access a not yet running array.
> 
> Is that expected ?
>

No.

To be able to say more I would need lots more details.
What is the exact state of the array (cat /proc/mdstat ;mdadm -D ...)
Where is the process waiting (cat /proc/PID/stack)
What sort of "access"
How did the array get to the "not running yet" state?
Any kernel messages?
Anything else that might be relevant.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Accesses to not yet running array

[Francis Moreau]

Hello,

On Wed, Aug 21, 2013 at 1:06 AM, NeilBrown <neilb@suse.de> wrote:
> On Tue, 20 Aug 2013 16:08:47 +0200 Francis Moreau <francis.moro@gmail.com>
> wrote:
>
>> hi,
>>
>> It looks like a process wait uninterruptibly when it tries to access
>> an array which is not running yet.
>>
>> Furthermore once the process is waiting, I can't start/stop the array anymore.
>>
>> So I need to reboot my system and be sure next time that any processes
>> don't try to access a not yet running array.
>>
>> Is that expected ?
>>
>
> No.
>
> To be able to say more I would need lots more details.
> What is the exact state of the array (cat /proc/mdstat ;mdadm -D ...)
> Where is the process waiting (cat /proc/PID/stack)
> What sort of "access"
> How did the array get to the "not running yet" state?
> Any kernel messages?
> Anything else that might be relevant.
>

please find some details below:

# uname -r
3.9.5-301.fc19.x86_64

# mdadm --version
mdadm - v3.2.6 - 25th October 2012

# mdadm -I /dev/sda
mdadm: container /dev/md/ddf0 now has 1 device
mdadm: /dev/md/126 assembled with 1 device but not started

# cat /proc/mdstat
Personalities : [raid1]
md126 : inactive sda[0]
      8355840 blocks super external:/md127/0

md127 : inactive sda[0](S)
      32768 blocks super external:ddf

unused devices: <none>

# mdadm -R /dev/md126
mdadm: started /dev/md126

# cat /proc/mdstat
Personalities : [raid1]
md126 : active raid1 sda[0]
      8355840 blocks super external:/md127/0 [2/1] [U_]

md127 : inactive sda[0](S)
      32768 blocks super external:ddf

unused devices: <none>

# mount /dev/md126p3 /mnt
<stucks>

# ps aux | grep mount
root      1543  0.1  0.1 123404  1324 tty1     D+   09:43   0:00 mount
/dev/md126p3 /mnt

# cat /proc/1543/stack
[<ffffffff814ce285>] md_write_start+0xb5/0x1a0
[<ffffffffa0222b76>] make_request+0x46/0xc30 [raid1]
[<ffffffff814c4403>] md_make_request+0xd3/0x230
[<ffffffff812cdee2>] generic_make_request+0xc2/0x110
[<ffffffff812cdfa3>] submit_bio+0x73/0x160
[<ffffffff811ca544>] submit_bh+0x114/0x1e0
[<ffffffff811cb263>] __sync_dirty_buffer+0x53/0xe0
[<ffffffff811cb303>] sync_dirty_buffer+0x13/0x20
[<ffffffff8123dd38>] ext4_commit_super+0x198/0x230
[<ffffffff81240015>] ext4_setup_super+0x125/0x1a0
[<ffffffff8124353e>] ext4_fill_super+0x265e/0x2dc0
[<ffffffff8119cce5>] mount_bdev+0x1b5/0x1f0
[<ffffffff81232d35>] ext4_mount+0x15/0x20
[<ffffffff8119d5e9>] mount_fs+0x39/0x1b0
[<ffffffff811b679f>] vfs_kern_mount+0x5f/0xf0
[<ffffffff811b8a6e>] do_mount+0x23e/0xa20
[<ffffffff811b92d3>] sys_mount+0x83/0xc0
[<ffffffff8164e799>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

# mdadm -D /dev/md126
/dev/md126:
      Container : /dev/md/ddf0, member 0
     Raid Level : raid1
     Array Size : 8355840 (7.97 GiB 8.56 GB)
  Used Dev Size : 8355840 (7.97 GiB 8.56 GB)
   Raid Devices : 2
  Total Devices : 1

          State : active, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       0        0        1      removed

# mdadm -D /dev/md127
/dev/md127:
        Version : ddf
     Raid Level : container
  Total Devices : 1

Working Devices : 1

  Member Arrays : /dev/md/126

    Number   Major   Minor   RaidDevice

       0       8        0        -        /dev/sda

# mdadm -R /dev/md126
mdadm: failed to run array /dev/md126: Device or resource busy


Thanks
-- 
Francis

Re: Accesses to not yet running array

[Francis Moreau]

Hello,

[ adding Martin in CC since it seems related to DDF... ]

On Wed, Aug 21, 2013 at 3:57 PM, Francis Moreau <francis.moro@gmail.com> wrote:
> Hello,
>
> On Wed, Aug 21, 2013 at 1:06 AM, NeilBrown <neilb@suse.de> wrote:
>> On Tue, 20 Aug 2013 16:08:47 +0200 Francis Moreau <francis.moro@gmail.com>
>> wrote:
>>
>>> hi,
>>>
>>> It looks like a process wait uninterruptibly when it tries to access
>>> an array which is not running yet.
>>>
>>> Furthermore once the process is waiting, I can't start/stop the array anymore.
>>>
>>> So I need to reboot my system and be sure next time that any processes
>>> don't try to access a not yet running array.
>>>
>>> Is that expected ?
>>>
>>
>> No.
>>
>> To be able to say more I would need lots more details.
>> What is the exact state of the array (cat /proc/mdstat ;mdadm -D ...)
>> Where is the process waiting (cat /proc/PID/stack)
>> What sort of "access"
>> How did the array get to the "not running yet" state?
>> Any kernel messages?
>> Anything else that might be relevant.
>>
>
> please find some details below:
>
> # uname -r
> 3.9.5-301.fc19.x86_64
>
> # mdadm --version
> mdadm - v3.2.6 - 25th October 2012
>
> # mdadm -I /dev/sda
> mdadm: container /dev/md/ddf0 now has 1 device
> mdadm: /dev/md/126 assembled with 1 device but not started
>
> # cat /proc/mdstat
> Personalities : [raid1]
> md126 : inactive sda[0]
>       8355840 blocks super external:/md127/0
>
> md127 : inactive sda[0](S)
>       32768 blocks super external:ddf
>
> unused devices: <none>
>
> # mdadm -R /dev/md126
> mdadm: started /dev/md126
>
> # cat /proc/mdstat
> Personalities : [raid1]
> md126 : active raid1 sda[0]
>       8355840 blocks super external:/md127/0 [2/1] [U_]
>
> md127 : inactive sda[0](S)
>       32768 blocks super external:ddf
>
> unused devices: <none>
>
> # mount /dev/md126p3 /mnt
> <stucks>
>
> # ps aux | grep mount
> root      1543  0.1  0.1 123404  1324 tty1     D+   09:43   0:00 mount
> /dev/md126p3 /mnt
>
> # cat /proc/1543/stack
> [<ffffffff814ce285>] md_write_start+0xb5/0x1a0
> [<ffffffffa0222b76>] make_request+0x46/0xc30 [raid1]
> [<ffffffff814c4403>] md_make_request+0xd3/0x230
> [<ffffffff812cdee2>] generic_make_request+0xc2/0x110
> [<ffffffff812cdfa3>] submit_bio+0x73/0x160
> [<ffffffff811ca544>] submit_bh+0x114/0x1e0
> [<ffffffff811cb263>] __sync_dirty_buffer+0x53/0xe0
> [<ffffffff811cb303>] sync_dirty_buffer+0x13/0x20
> [<ffffffff8123dd38>] ext4_commit_super+0x198/0x230
> [<ffffffff81240015>] ext4_setup_super+0x125/0x1a0
> [<ffffffff8124353e>] ext4_fill_super+0x265e/0x2dc0
> [<ffffffff8119cce5>] mount_bdev+0x1b5/0x1f0
> [<ffffffff81232d35>] ext4_mount+0x15/0x20
> [<ffffffff8119d5e9>] mount_fs+0x39/0x1b0
> [<ffffffff811b679f>] vfs_kern_mount+0x5f/0xf0
> [<ffffffff811b8a6e>] do_mount+0x23e/0xa20
> [<ffffffff811b92d3>] sys_mount+0x83/0xc0
> [<ffffffff8164e799>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
>
> # mdadm -D /dev/md126
> /dev/md126:
>       Container : /dev/md/ddf0, member 0
>      Raid Level : raid1
>      Array Size : 8355840 (7.97 GiB 8.56 GB)
>   Used Dev Size : 8355840 (7.97 GiB 8.56 GB)
>    Raid Devices : 2
>   Total Devices : 1
>
>           State : active, degraded
>  Active Devices : 1
> Working Devices : 1
>  Failed Devices : 0
>   Spare Devices : 0
>
>     Number   Major   Minor   RaidDevice State
>        0       8        0        0      active sync   /dev/sda
>        1       0        0        1      removed
>
> # mdadm -D /dev/md127
> /dev/md127:
>         Version : ddf
>      Raid Level : container
>   Total Devices : 1
>
> Working Devices : 1
>
>   Member Arrays : /dev/md/126
>
>     Number   Major   Minor   RaidDevice
>
>        0       8        0        -        /dev/sda
>
> # mdadm -R /dev/md126
> mdadm: failed to run array /dev/md126: Device or resource busy
>

This issue doesn't exist if I'm using the linux native format 1.2.

Martin, any idea about this issue which seems related to DDF ?

Thanks
-- 
Francis

Re: Accesses to not yet running array

[Martin Wilck]

On 08/23/2013 12:00 PM, Francis Moreau wrote:

> [ adding Martin in CC since it seems related to DDF... ]

I reproduced it here. The problem is that mdmon isn't running. Simply
try running "mdmon /dev/md127", and the mount should proceed.

You'd avoid the problem by running "mdadm -IR /dev/sda" in the first place.

@Neil: This is unexpected, I think we need some code to start mdmon if
necessary in Manage_run. I considered coming up with a patch, but you
can do it much quicker, I think.

Martin

Re: Accesses to not yet running array

[Francis Moreau]

Hello Martin,

On Fri, Aug 23, 2013 at 9:09 PM, Martin Wilck <mwilck@arcor.de> wrote:
> On 08/23/2013 12:00 PM, Francis Moreau wrote:
>
>> [ adding Martin in CC since it seems related to DDF... ]
>
> I reproduced it here. The problem is that mdmon isn't running. Simply
> try running "mdmon /dev/md127", and the mount should proceed.
>

oh, ok, I should have noticed.

> You'd avoid the problem by running "mdadm -IR /dev/sda" in the first place.

Actually this is how arrays are assembled during boot on some
distributions (Fedora for example). They incorporate disks as they're
discovered and then they run the array (degraded or not). Therefore
booting with a missing disk doesn't work.

Thanks
-- 
Francis

Re: Accesses to not yet running array

[Francis Moreau]

On Sat, Aug 24, 2013 at 2:40 PM, Francis Moreau <francis.moro@gmail.com> wrote:
> Hello Martin,
>
> On Fri, Aug 23, 2013 at 9:09 PM, Martin Wilck <mwilck@arcor.de> wrote:
>> On 08/23/2013 12:00 PM, Francis Moreau wrote:
>>
>>> [ adding Martin in CC since it seems related to DDF... ]
>>
>> I reproduced it here. The problem is that mdmon isn't running. Simply
>> try running "mdmon /dev/md127", and the mount should proceed.
>>
>
> oh, ok, I should have noticed.
>
>> You'd avoid the problem by running "mdadm -IR /dev/sda" in the first place.
>
> Actually this is how arrays are assembled during boot on some
> distributions (Fedora for example).

I meant the sequence:

mdadm -I /dev/sda
mdadm -I /dev/sdb
mdadm -R /dev/mdxxx

is used by distribution.

-- 
Francis

Re: Accesses to not yet running array

[Martin Wilck]

On 08/24/2013 02:42 PM, Francis Moreau wrote:
> On Sat, Aug 24, 2013 at 2:40 PM, Francis Moreau <francis.moro@gmail.com> wrote:
>> Hello Martin,
>>
>> On Fri, Aug 23, 2013 at 9:09 PM, Martin Wilck <mwilck@arcor.de> wrote:
>>> On 08/23/2013 12:00 PM, Francis Moreau wrote:
>>>
>>>> [ adding Martin in CC since it seems related to DDF... ]
>>>
>>> I reproduced it here. The problem is that mdmon isn't running. Simply
>>> try running "mdmon /dev/md127", and the mount should proceed.
>>>
>>
>> oh, ok, I should have noticed.
>>
>>> You'd avoid the problem by running "mdadm -IR /dev/sda" in the first place.
>>
>> Actually this is how arrays are assembled during boot on some
>> distributions (Fedora for example).
> 
> I meant the sequence:
> 
> mdadm -I /dev/sda
> mdadm -I /dev/sdb
> mdadm -R /dev/mdxxx
> 
> is used by distribution.

CentOS 6 uses the following sequence:

mdadm -I /dev/sda
mdadm -I /dev/sdb
mdadm -I /dev/mdXXX

I don't see mdadm -R anywhere on CentOS.
You could change your udev rules to run "mdadm -IR /dev/mdXXX" instead.
Have you tried that?

I wonder if that woudn't be smarter anyway. Neil?

Martin

Re: Accesses to not yet running array

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1573 bytes --]

On Sat, 24 Aug 2013 20:41:19 +0200 Martin Wilck <mwilck@arcor.de> wrote:

> On 08/24/2013 02:42 PM, Francis Moreau wrote:
> > On Sat, Aug 24, 2013 at 2:40 PM, Francis Moreau <francis.moro@gmail.com> wrote:
> >> Hello Martin,
> >>
> >> On Fri, Aug 23, 2013 at 9:09 PM, Martin Wilck <mwilck@arcor.de> wrote:
> >>> On 08/23/2013 12:00 PM, Francis Moreau wrote:
> >>>
> >>>> [ adding Martin in CC since it seems related to DDF... ]
> >>>
> >>> I reproduced it here. The problem is that mdmon isn't running. Simply
> >>> try running "mdmon /dev/md127", and the mount should proceed.
> >>>
> >>
> >> oh, ok, I should have noticed.
> >>
> >>> You'd avoid the problem by running "mdadm -IR /dev/sda" in the first place.
> >>
> >> Actually this is how arrays are assembled during boot on some
> >> distributions (Fedora for example).
> > 
> > I meant the sequence:
> > 
> > mdadm -I /dev/sda
> > mdadm -I /dev/sdb
> > mdadm -R /dev/mdxxx
> > 
> > is used by distribution.
> 
> CentOS 6 uses the following sequence:
> 
> mdadm -I /dev/sda
> mdadm -I /dev/sdb
> mdadm -I /dev/mdXXX
> 
> I don't see mdadm -R anywhere on CentOS.
> You could change your udev rules to run "mdadm -IR /dev/mdXXX" instead.
> Have you tried that?
> 
> I wonder if that woudn't be smarter anyway. Neil?

I don't think you want the udev rules to run "mdadm -IR ...".

If 'mdadm -IR...' to be run at all, it should probably be
   mdadm -IRs

and it should happen after all attached busses have been scanned and udev has
processed all new devices.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Accesses to not yet running array

[Francis Moreau]

Hello Martin,

On Sat, Aug 24, 2013 at 8:41 PM, Martin Wilck <mwilck@arcor.de> wrote:
> On 08/24/2013 02:42 PM, Francis Moreau wrote:
>> On Sat, Aug 24, 2013 at 2:40 PM, Francis Moreau <francis.moro@gmail.com> wrote:
>>> Hello Martin,
>>>
>>> On Fri, Aug 23, 2013 at 9:09 PM, Martin Wilck <mwilck@arcor.de> wrote:
>>>> On 08/23/2013 12:00 PM, Francis Moreau wrote:
>>>>
>>>>> [ adding Martin in CC since it seems related to DDF... ]
>>>>
>>>> I reproduced it here. The problem is that mdmon isn't running. Simply
>>>> try running "mdmon /dev/md127", and the mount should proceed.
>>>>
>>>
>>> oh, ok, I should have noticed.
>>>
>>>> You'd avoid the problem by running "mdadm -IR /dev/sda" in the first place.
>>>
>>> Actually this is how arrays are assembled during boot on some
>>> distributions (Fedora for example).
>>
>> I meant the sequence:
>>
>> mdadm -I /dev/sda
>> mdadm -I /dev/sdb
>> mdadm -R /dev/mdxxx
>>
>> is used by distribution.
>
> CentOS 6 uses the following sequence:
>
> mdadm -I /dev/sda
> mdadm -I /dev/sdb
> mdadm -I /dev/mdXXX
>
> I don't see mdadm -R anywhere on CentOS.

After digging into the code of dracut, it seems that the bug (mdmon
not started by mdadm -R /dev/mdxxx on incomplete array) seems to be
known and a workaround exists.

Basically the script which starts MD arrays (slightly modified by me
to make it easier to read) does:

for md in /dev/md[0-9_]*; do

    ...

    [ $(cat $md/array_state) != inactive ] && continue

    mdadm --offroot -R $md

    [ $(cat $md/array_state) = inactive ] && continue

    # workaround for mdmon bug
    [ $(cat $md/degraded) -gt 0 ] && mdmon --offroot --takeover $md
done

In theory this should fix the bug but it doesn't (looks like I'm
really unlucky). Indeed at the time the script is run, the container
device doesn't exist in /dev. Therefore the workaround is only
executed on the MD array device. I can see the following message
during the boot: "mdmon: md126 is not a container - cannot monitor"

I tried to add "udevadm settle" before executing the workaround but
that doesn't help.

> You could change your udev rules to run "mdadm -IR /dev/mdXXX" instead.
> Have you tried that?

I tried and it works fine.

Thanks
-- 
Francis

Re: Accesses to not yet running array

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

On Fri, 23 Aug 2013 21:09:24 +0200 Martin Wilck <mwilck@arcor.de> wrote:

> On 08/23/2013 12:00 PM, Francis Moreau wrote:
> 
> > [ adding Martin in CC since it seems related to DDF... ]
> 
> I reproduced it here. The problem is that mdmon isn't running. Simply
> try running "mdmon /dev/md127", and the mount should proceed.
> 
> You'd avoid the problem by running "mdadm -IR /dev/sda" in the first place.
> 
> @Neil: This is unexpected, I think we need some code to start mdmon if
> necessary in Manage_run. I considered coming up with a patch, but you
> can do it much quicker, I think.
>

I've changed Manage_run() to call the same code that "mdadm -IRs" call, but
asking to just run the one array, not all arrays.
This should  do what is expected (I hope).

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Accesses to not yet running array

[Francis Moreau]

Hello Neil,

On Mon, Aug 26, 2013 at 7:30 AM, NeilBrown <neilb@suse.de> wrote:
> On Fri, 23 Aug 2013 21:09:24 +0200 Martin Wilck <mwilck@arcor.de> wrote:
>
>> On 08/23/2013 12:00 PM, Francis Moreau wrote:
>>
>> > [ adding Martin in CC since it seems related to DDF... ]
>>
>> I reproduced it here. The problem is that mdmon isn't running. Simply
>> try running "mdmon /dev/md127", and the mount should proceed.
>>
>> You'd avoid the problem by running "mdadm -IR /dev/sda" in the first place.
>>
>> @Neil: This is unexpected, I think we need some code to start mdmon if
>> necessary in Manage_run. I considered coming up with a patch, but you
>> can do it much quicker, I think.
>>
>
> I've changed Manage_run() to call the same code that "mdadm -IRs" call, but
> asking to just run the one array, not all arrays.
> This should  do what is expected (I hope).


Any chance to get the same fix for 3.2.x release ?

Thanks
-- 
Francis

Re: Accesses to not yet running array

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1258 bytes --]

On Mon, 26 Aug 2013 09:30:04 +0200 Francis Moreau <francis.moro@gmail.com>
wrote:

> Hello Neil,
> 
> On Mon, Aug 26, 2013 at 7:30 AM, NeilBrown <neilb@suse.de> wrote:
> > On Fri, 23 Aug 2013 21:09:24 +0200 Martin Wilck <mwilck@arcor.de> wrote:
> >
> >> On 08/23/2013 12:00 PM, Francis Moreau wrote:
> >>
> >> > [ adding Martin in CC since it seems related to DDF... ]
> >>
> >> I reproduced it here. The problem is that mdmon isn't running. Simply
> >> try running "mdmon /dev/md127", and the mount should proceed.
> >>
> >> You'd avoid the problem by running "mdadm -IR /dev/sda" in the first place.
> >>
> >> @Neil: This is unexpected, I think we need some code to start mdmon if
> >> necessary in Manage_run. I considered coming up with a patch, but you
> >> can do it much quicker, I think.
> >>
> >
> > I've changed Manage_run() to call the same code that "mdadm -IRs" call, but
> > asking to just run the one array, not all arrays.
> > This should  do what is expected (I hope).
> 
> 
> Any chance to get the same fix for 3.2.x release ?

I think it would be a lot easier to just remove the "mdadm -R ..." calls.
Where exactly are they?  In some /etc/init.d script?

You probably want "mdadm -IRs" there.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Accesses to not yet running array

[Francis Moreau]

On Mon, Aug 26, 2013 at 9:53 AM, NeilBrown <neilb@suse.de> wrote:
> On Mon, 26 Aug 2013 09:30:04 +0200 Francis Moreau <francis.moro@gmail.com>
> wrote:
>
>> Hello Neil,
>>
>> On Mon, Aug 26, 2013 at 7:30 AM, NeilBrown <neilb@suse.de> wrote:
>> > On Fri, 23 Aug 2013 21:09:24 +0200 Martin Wilck <mwilck@arcor.de> wrote:
>> >
>> >> On 08/23/2013 12:00 PM, Francis Moreau wrote:
>> >>
>> >> > [ adding Martin in CC since it seems related to DDF... ]
>> >>
>> >> I reproduced it here. The problem is that mdmon isn't running. Simply
>> >> try running "mdmon /dev/md127", and the mount should proceed.
>> >>
>> >> You'd avoid the problem by running "mdadm -IR /dev/sda" in the first place.
>> >>
>> >> @Neil: This is unexpected, I think we need some code to start mdmon if
>> >> necessary in Manage_run. I considered coming up with a patch, but you
>> >> can do it much quicker, I think.
>> >>
>> >
>> > I've changed Manage_run() to call the same code that "mdadm -IRs" call, but
>> > asking to just run the one array, not all arrays.
>> > This should  do what is expected (I hope).
>>
>>
>> Any chance to get the same fix for 3.2.x release ?
>
> I think it would be a lot easier to just remove the "mdadm -R ..." calls.
> Where exactly are they?  In some /etc/init.d script?

Nope, it's in the initrd, in my case generated by dracut.

Did you mean removing "mdadm -R ..." and add "mdadm -IRs ..." instead ?

>
> You probably want "mdadm -IRs" there.
>

Well, I thought there was a difference between running "mdadm -IRs"
and the fix you did in Manage_run() and your fix was preferred.

-- 
Francis