ISW fakeraid: fail to add back a disk previously removed

ISW fakeraid: fail to add back a disk previously removed

[Francis Moreau]

Hello,

I'm testing ISW fake-raid with mdadm 3.3 and I can't add back a disk
to a container previously marked as failed and removed.

Also I'm testing ISW inside a VM so I'm not sure my test case is valid.

So basically my ISW array was created with dmraid. I then installed my
system on this array and rebooted to switch to mdadm. So far so good
everything seems to work perfectly:

# export IMSM_NO_PLATFORM=1

# cat /proc/mdstat
Personalities : [raid1]
md126 : active raid1 sda[1] sdb[0]
      2092375 blocks super external:/md127/0 [2/2] [UU]

md127 : inactive sda[1](S) sdb[0](S)
      6306 blocks super external:imsm

unused devices: <none>

# mdadm -E /dev/sdb
/dev/sdb:
          Magic : Intel Raid ISM Cfg Sig.
        Version : 1.1.00
    Orig Family : ef92cdd9
[ ... ]

Now I'm marking sda as failed and remove it:

# mdadm --fail /dev/md126 sda
# mdadm --remove /dev/md127 sda

everything works fine until I add back sda to the container:

# mdadm --add /dev/md127 /dev/sda
mdadm: add /dev/sda

# cat /proc/mdstat
Personalities : [raid1]
md126 : active raid1 sdb[0]
      2092375 blocks super external:/md127/0 [2/1] [_U]

md127 : inactive sda[1](S) sdb[0](S)
      6306 blocks super external:imsm

sda is not added anymore to the array md126 and therefore no resynchro happens.

# mdadm -E /dev/sda
/dev/sda:
   MBR Magic : aa55
Partition[0] :      3612577 sectors at         2048 (type 83)
Partition[1] :       560133 sectors at      3616767 (type 05)

Could anybody tell me if that's expected and if so could you englight me ?

Thanks.
-- 
Francis

Re: ISW fakeraid: fail to add back a disk previously removed

[Francis Moreau]

Hi,


[ Adding Dan Williams in CC since it appears to be the author of the
intel fake raid support. ]


On Wed, Oct 23, 2013 at 11:24 AM, Francis Moreau <francis.moro@gmail.com> wrote:
> Hello,
>
> I'm testing ISW fake-raid with mdadm 3.3 and I can't add back a disk
> to a container previously marked as failed and removed.
>
> Also I'm testing ISW inside a VM so I'm not sure my test case is valid.
>
> So basically my ISW array was created with dmraid. I then installed my
> system on this array and rebooted to switch to mdadm. So far so good
> everything seems to work perfectly:
>
> # export IMSM_NO_PLATFORM=1
>
> # cat /proc/mdstat
> Personalities : [raid1]
> md126 : active raid1 sda[1] sdb[0]
>       2092375 blocks super external:/md127/0 [2/2] [UU]
>
> md127 : inactive sda[1](S) sdb[0](S)
>       6306 blocks super external:imsm
>
> unused devices: <none>
>
> # mdadm -E /dev/sdb
> /dev/sdb:
>           Magic : Intel Raid ISM Cfg Sig.
>         Version : 1.1.00
>     Orig Family : ef92cdd9
> [ ... ]
>
> Now I'm marking sda as failed and remove it:
>
> # mdadm --fail /dev/md126 sda
> # mdadm --remove /dev/md127 sda
>
> everything works fine until I add back sda to the container:
>
> # mdadm --add /dev/md127 /dev/sda
> mdadm: add /dev/sda
>
> # cat /proc/mdstat
> Personalities : [raid1]
> md126 : active raid1 sdb[0]
>       2092375 blocks super external:/md127/0 [2/1] [_U]
>
> md127 : inactive sda[1](S) sdb[0](S)
>       6306 blocks super external:imsm
>
> sda is not added anymore to the array md126 and therefore no resynchro happens.
>
> # mdadm -E /dev/sda
> /dev/sda:
>    MBR Magic : aa55
> Partition[0] :      3612577 sectors at         2048 (type 83)
> Partition[1] :       560133 sectors at      3616767 (type 05)
>
> Could anybody tell me if that's expected and if so could you englight me ?
>
> Thanks.
> --
> Francis



-- 
Francis

Re: ISW fakeraid: fail to add back a disk previously removed

[Francis Moreau]

Hi Neil,

So far now answer, I'm adding you in the thread since it seems a
critical issue, hope you don't mind.

Thanks

On Mon, Oct 28, 2013 at 7:18 AM, Francis Moreau <francis.moro@gmail.com> wrote:
> Hi,
>
>
> [ Adding Dan Williams in CC since it appears to be the author of the
> intel fake raid support. ]
>
>
> On Wed, Oct 23, 2013 at 11:24 AM, Francis Moreau <francis.moro@gmail.com> wrote:
>> Hello,
>>
>> I'm testing ISW fake-raid with mdadm 3.3 and I can't add back a disk
>> to a container previously marked as failed and removed.
>>
>> Also I'm testing ISW inside a VM so I'm not sure my test case is valid.
>>
>> So basically my ISW array was created with dmraid. I then installed my
>> system on this array and rebooted to switch to mdadm. So far so good
>> everything seems to work perfectly:
>>
>> # export IMSM_NO_PLATFORM=1
>>
>> # cat /proc/mdstat
>> Personalities : [raid1]
>> md126 : active raid1 sda[1] sdb[0]
>>       2092375 blocks super external:/md127/0 [2/2] [UU]
>>
>> md127 : inactive sda[1](S) sdb[0](S)
>>       6306 blocks super external:imsm
>>
>> unused devices: <none>
>>
>> # mdadm -E /dev/sdb
>> /dev/sdb:
>>           Magic : Intel Raid ISM Cfg Sig.
>>         Version : 1.1.00
>>     Orig Family : ef92cdd9
>> [ ... ]
>>
>> Now I'm marking sda as failed and remove it:
>>
>> # mdadm --fail /dev/md126 sda
>> # mdadm --remove /dev/md127 sda
>>
>> everything works fine until I add back sda to the container:
>>
>> # mdadm --add /dev/md127 /dev/sda
>> mdadm: add /dev/sda
>>
>> # cat /proc/mdstat
>> Personalities : [raid1]
>> md126 : active raid1 sdb[0]
>>       2092375 blocks super external:/md127/0 [2/1] [_U]
>>
>> md127 : inactive sda[1](S) sdb[0](S)
>>       6306 blocks super external:imsm
>>
>> sda is not added anymore to the array md126 and therefore no resynchro happens.
>>
>> # mdadm -E /dev/sda
>> /dev/sda:
>>    MBR Magic : aa55
>> Partition[0] :      3612577 sectors at         2048 (type 83)
>> Partition[1] :       560133 sectors at      3616767 (type 05)
>>
>> Could anybody tell me if that's expected and if so could you englight me ?
>>
>> Thanks.
>> --
>> Francis
>
>
>
> --
> Francis



-- 
Francis

RE: ISW fakeraid: fail to add back a disk previously removed

[Dorau, Lukasz]

[-- Attachment #1: Type: text/plain, Size: 2588 bytes --]

On Wednesday, November 06, 2013 10:06 AM Francis Moreau <francis.moro@gmail.com>
wrote:
> Hi Neil,
>
> So far now answer, I'm adding you in the thread since it seems a
> critical issue, hope you don't mind.
>
> Thanks
>
> On Mon, Oct 28, 2013 at 7:18 AM, Francis Moreau <francis.moro@gmail.com>
> wrote:
> > Hi,
> >
> >
> > [ Adding Dan Williams in CC since it appears to be the author of the
> > intel fake raid support. ]
> >
> >
> > On Wed, Oct 23, 2013 at 11:24 AM, Francis Moreau <francis.moro@gmail.com>
> wrote:
> >> Hello,
> >>
> >> I'm testing ISW fake-raid with mdadm 3.3 and I can't add back a disk
> >> to a container previously marked as failed and removed.
> >>
> >> Also I'm testing ISW inside a VM so I'm not sure my test case is valid.
> >>
> >> So basically my ISW array was created with dmraid. I then installed my
> >> system on this array and rebooted to switch to mdadm. So far so good
> >> everything seems to work perfectly:
> >>
> >> # export IMSM_NO_PLATFORM=1
> >>
> >> # cat /proc/mdstat
> >> Personalities : [raid1]
> >> md126 : active raid1 sda[1] sdb[0]
> >>       2092375 blocks super external:/md127/0 [2/2] [UU]
> >>
> >> md127 : inactive sda[1](S) sdb[0](S)
> >>       6306 blocks super external:imsm
> >>
> >> unused devices: <none>
> >>
> >> # mdadm -E /dev/sdb
> >> /dev/sdb:
> >>           Magic : Intel Raid ISM Cfg Sig.
> >>         Version : 1.1.00
> >>     Orig Family : ef92cdd9
> >> [ ... ]
> >>
> >> Now I'm marking sda as failed and remove it:
> >>
> >> # mdadm --fail /dev/md126 sda
> >> # mdadm --remove /dev/md127 sda
> >>
> >> everything works fine until I add back sda to the container:
> >>
> >> # mdadm --add /dev/md127 /dev/sda
> >> mdadm: add /dev/sda
> >>
> >> # cat /proc/mdstat
> >> Personalities : [raid1]
> >> md126 : active raid1 sdb[0]
> >>       2092375 blocks super external:/md127/0 [2/1] [_U]
> >>
> >> md127 : inactive sda[1](S) sdb[0](S)
> >>       6306 blocks super external:imsm
> >>
> >> sda is not added anymore to the array md126 and therefore no resynchro
> happens.
> >>
> >> # mdadm -E /dev/sda
> >> /dev/sda:
> >>    MBR Magic : aa55
> >> Partition[0] :      3612577 sectors at         2048 (type 83)
> >> Partition[1] :       560133 sectors at      3616767 (type 05)
> >>
> >> Could anybody tell me if that's expected and if so could you englight me ?
> >>

Hi Francis,

If you created a RAID array with mdadm, it would work correctly - recovery would 
start.
The IMSM metadata is not supported in dmraid anymore and mdadm does not have to 
work correctly with array created and started with dmraid.

Lukasz


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6582 bytes --]

Re: ISW fakeraid: fail to add back a disk previously removed

[Francis Moreau]

Hello Lukasz

On Wed, Nov 6, 2013 at 3:08 PM, Dorau, Lukasz <lukasz.dorau@intel.com> wrote:

 [...]

>
> If you created a RAID array with mdadm, it would work correctly - recovery would
> start.

Nope, I've got an identical behaviour if the array is created by mdadm (v3.3).

> The IMSM metadata is not supported in dmraid anymore and mdadm does not have to
> work correctly with array created and started with dmraid.
>

I initially created the array with dmraid because it claims to support
ISW format whereas mdadm supports IMSM. I wanted to be sure that ISW
format was also supported by mdadm if the 2 formats were different.

As you can see, I'm confused about ISW vs IMSM.

Are those 2 formats the same thing ?

Thanks.
-- 
Francis

RE: ISW fakeraid: fail to add back a disk previously removed

[Dorau, Lukasz]

On Friday, November 08, 2013 2:46 PM Francis Moreau <francis.moro@gmail.com> wrote:
> > If you created a RAID array with mdadm, it would work correctly - recovery would
> > start.
> 
> Nope, I've got an identical behaviour if the array is created by mdadm (v3.3).
> 

I have checked it twice with Fedora 19 and mdadm v3.3 and it works.
What OS are you running?

> As you can see, I'm confused about ISW vs IMSM.
> 
> Are those 2 formats the same thing ?
> 

ISW is a name of Intel metadata used by dmraid and IMSM is a name of Intel metadata used by mdadm, but ISW in dmraid has not been supported and developed for a couple of years, so ISW might be called an old version of IMSM.

Lukasz

Re: ISW fakeraid: fail to add back a disk previously removed

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1073 bytes --]

On Fri, 8 Nov 2013 14:33:24 +0000 "Dorau, Lukasz" <lukasz.dorau@intel.com>
wrote:

> On Friday, November 08, 2013 2:46 PM Francis Moreau <francis.moro@gmail.com> wrote:
> > > If you created a RAID array with mdadm, it would work correctly - recovery would
> > > start.
> > 
> > Nope, I've got an identical behaviour if the array is created by mdadm (v3.3).
> > 
> 
> I have checked it twice with Fedora 19 and mdadm v3.3 and it works.
> What OS are you running?
> 
> > As you can see, I'm confused about ISW vs IMSM.
> > 
> > Are those 2 formats the same thing ?
> > 
> 
> ISW is a name of Intel metadata used by dmraid and IMSM is a name of Intel metadata used by mdadm, but ISW in dmraid has not been supported and developed for a couple of years, so ISW might be called an old version of IMSM.
> 
> Lukasz

Unless we identify something in the metadata the dmraid writes which clearly
contravene the specification, we must assume there is a bug in mdadm.  I
haven't had a change to look at this properly yet but I hope to next week.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: ISW fakeraid: fail to add back a disk previously removed

[Dan Williams]

On Fri, Nov 8, 2013 at 5:45 AM, Francis Moreau <francis.moro@gmail.com> wrote:
> Hello Lukasz
>
> On Wed, Nov 6, 2013 at 3:08 PM, Dorau, Lukasz <lukasz.dorau@intel.com> wrote:
>
>  [...]
>
>>
>> If you created a RAID array with mdadm, it would work correctly - recovery would
>> start.
>
> Nope, I've got an identical behaviour if the array is created by mdadm (v3.3).

It should really work either way.  I gave it a shot with:

mdadm - v3.3-30-gf33a71f - 31st October 2013
mdadm - v3.2.5 - 18th May 2012

...and it seems ok with loop devices, so something more fundamental is wrong.

mdmon is in charge of finding new devices in the container and adding
them to the member arrays.  Can you confirm that mdmon is running when
the disk is failing to be added?

The IMSM format is sensitive to disk serial numbers, I'm curious if
your virtual disks are specifying unique serial numbers for your
devices?  In my above tests with loopback devices I am using
IMSM_DEVNAME_AS_SERIAL=1 for debug purposes you might try that as
well.

--
Dan

Re: ISW fakeraid: fail to add back a disk previously removed

[Francis Moreau]

Hello Dan,

On Fri, Nov 8, 2013 at 11:02 PM, Dan Williams <dan.j.williams@intel.com> wrote:
> On Fri, Nov 8, 2013 at 5:45 AM, Francis Moreau <francis.moro@gmail.com> wrote:
>> Hello Lukasz
>>
>> On Wed, Nov 6, 2013 at 3:08 PM, Dorau, Lukasz <lukasz.dorau@intel.com> wrote:
>>
>>  [...]
>>
>>>
>>> If you created a RAID array with mdadm, it would work correctly - recovery would
>>> start.
>>
>> Nope, I've got an identical behaviour if the array is created by mdadm (v3.3).
>
> It should really work either way.  I gave it a shot with:
>
> mdadm - v3.3-30-gf33a71f - 31st October 2013
> mdadm - v3.2.5 - 18th May 2012
>
> ...and it seems ok with loop devices, so something more fundamental is wrong.
>
> mdmon is in charge of finding new devices in the container and adding
> them to the member arrays.  Can you confirm that mdmon is running when
> the disk is failing to be added?
>
> The IMSM format is sensitive to disk serial numbers, I'm curious if
> your virtual disks are specifying unique serial numbers for your
> devices?  In my above tests with loopback devices I am using
> IMSM_DEVNAME_AS_SERIAL=1 for debug purposes you might try that as
> well.
>

I think I found something.

In my current settings, when mdadm creates a new RAID array with
foreign metadata, it starts mdmon by using systemd throught the
mdmon@.service. However in that case the new mdmon process doesn't see
the IMSM_NO_PLATFORM environment variable set because process started
by systemd are executed in a clean env.

To verify this, I created the array again by calling mdadm with
MDADM_NO_SYSTEMCTL env variable set and it worked as expected. I also
tested by using systemd and by adding "Environment=IMSM_NO_PLATFORM=1"
in the unit file and it worked too.

WDTY ?

Thanks
-- 
Francis