More ddf container woes

More ddf container woes

[Albert Pauw]

  Hi Neil,

I found some more trouble with the ddf code, separate from the stuff I 
mentioned before (which is still present in the version I used below).

Here's what I did and found:

Note: Updated mdadm from the git repository up to and including the 
commit "Manage:  be more careful about --add attempts."

Used six disks, sdb - sdg out of which I created a 5-disk container, 
leaving one disk unused for the moment:

mdadm -C /dev/md127 -l container -e ddf -n 5 /dev/sd[b-f]

Created two RAID sets in this container:

mdadm -C /dev/md0 -l 1 -n 2 /dev/md127
mdadm -C /dev/md1 -l 5 -n 3 /dev/md127

Note: At this moment, only one mdmon is running (mdmon md127)

After finishing creating both RAID sets, I fail two disks, one in each 
RAID set:

mdadm -f /dev/md0 /dev/sdb

mdadm -f /dev/md1 /dev/sdd

The first failed disk (sdb) is automatically removed from /dev/md0, but 
oddly enough the disk stays marked as
"active/Online" in the "mdadm -E /dev/md127" output, the second failed 
disk (sdd) gets marked [F] in the RAID 5
array, but NOT removed, only when I do a

mdmon --all

the failed disk in /dev/md1 is removed, this second failed disk IS 
marked "Failed" in the "mdadm -E output".

Note: Checking on the RAID arrays using "mdadm -D" they are both marked 
as "clean, degraded".

I now add a new empty clean disk (/dev/sdg) to the container, after 
which md1 (the RAID 5 set) is immediately starting to rebuild.
The RAID 1 set (md0), however, is set to "resync=DELAYED", very odd, 
because I only added one disk.

Looking at the output of /proc/mdstat i see that disk sdg (the new 
spare) is actually added to both RAID
arrays, and after finishing the rebuild of md1 the other RAID set (md0) 
is also rebuild, using the SAME spare disk (sdg).


Albert

Re: More ddf container woes

[Albert Pauw]

  More experiments with the same setup

On 03/10/11 09:34 AM, Albert Pauw wrote:
>  Hi Neil,
>
> I found some more trouble with the ddf code, separate from the stuff I 
> mentioned before (which is still present in the version I used below).
>
> Here's what I did and found:
>
> Note: Updated mdadm from the git repository up to and including the 
> commit "Manage:  be more careful about --add attempts."
>
> Used six disks, sdb - sdg out of which I created a 5-disk container, 
> leaving one disk unused for the moment:
>
> mdadm -C /dev/md127 -l container -e ddf -n 5 /dev/sd[b-f]
>
> Created two RAID sets in this container:
>
> mdadm -C /dev/md0 -l 1 -n 2 /dev/md127
> mdadm -C /dev/md1 -l 5 -n 3 /dev/md127
>
> Note: At this moment, only one mdmon is running (mdmon md127)
>
> After finishing creating both RAID sets, I fail two disks, one in each 
> RAID set:
>
> mdadm -f /dev/md0 /dev/sdb
>
> mdadm -f /dev/md1 /dev/sdd
>
> The first failed disk (sdb) is automatically removed from /dev/md0, 
> but oddly enough the disk stays marked as
> "active/Online" in the "mdadm -E /dev/md127" output, the second failed 
> disk (sdd) gets marked [F] in the RAID 5
> array, but NOT removed, only when I do a
>
> mdmon --all
>
> the failed disk in /dev/md1 is removed, this second failed disk IS 
> marked "Failed" in the "mdadm -E output".
>
> Note: Checking on the RAID arrays using "mdadm -D" they are both 
> marked as "clean, degraded".

I now failed the disk in reverse order, first the RAID5 set (md1), then 
the RAID 1 set (md0), and the behaviour was different.

Now both disks stay marked failed [F] in the subarrays (md0 and md1). 
Doing a "mdadm -E /dev/md127" shows all disks
"active/Online", so the container isn't told of the failure of the 
disks. Only after a "mdmon --all" both failed disks are removed
from their respective array. "mdadm -E /dev/md127" now shows both disks 
as failed, so the container knows about the failed disks now.

When I don't run "mdmon --all" and want to add a spare disk it fails 
with the message "mdadm: add failed for /dev/sdg: mdmon not running".

The rest of the response stays the same. Adding a clean new disk to the 
container makes both subarrays going into recovery with this
new disk, md1 first and after finishing this, md0 gets resynched (with 
the same disk!).

When I fail two disks of the RAID 5 set (md1), so the whole subarray is 
failed, and then add a spare disk to the container, only md0 (the RAID 1 
set)
picks it up, md1 doesn't get rebuild (and that's who it should be).

>
> I now add a new empty clean disk (/dev/sdg) to the container, after 
> which md1 (the RAID 5 set) is immediately starting to rebuild.
> The RAID 1 set (md0), however, is set to "resync=DELAYED", very odd, 
> because I only added one disk.
>
> Looking at the output of /proc/mdstat i see that disk sdg (the new 
> spare) is actually added to both RAID
> arrays, and after finishing the rebuild of md1 the other RAID set 
> (md0) is also rebuild, using the SAME spare disk (sdg).
>
>
> Albert
>

To sum it up, there are two problems here:

- A failed disk in a subarray isn't automatically removed and marked 
"Failed" in the container, although in some cases it does (see above).
Only after a manual "mdmon --all" will this take place.

- When two subarrays have failed disks, are degraded, but operational 
and I add a spare disk to the container, both will pick up the spare
disk for replacement. They won't do this in parallel, but in sequence, 
but nevertheless use the same disk.

Albert

Re: More ddf container woes

[NeilBrown]

On Fri, 11 Mar 2011 12:50:16 +0100 Albert Pauw <albert.pauw@gmail.com> wrote:

>   More experiments with the same setup
> 

Hi Albert,
 thanks again for this testing.

> To sum it up, there are two problems here:
> 
> - A failed disk in a subarray isn't automatically removed and marked 
> "Failed" in the container, although in some cases it does (see above).
> Only after a manual "mdmon --all" will this take place.

I think this is fixed in my devel-3.2 branch

   git://neil.brown.name/mdadm devel-3.2

Some aspects of it a fixed in the 'master' branch, but removing a
device properly from a container won't be fixed in 3.1.x, only 3.2.x

> 
> - When two subarrays have failed disks, are degraded, but operational 
> and I add a spare disk to the container, both will pick up the spare
> disk for replacement. They won't do this in parallel, but in sequence, 
> but nevertheless use the same disk.

I haven't fixed this yet, but can easily duplicate it.  There are a
couple of issues here that I need to think through before I get
it fixed properly.

Hopefully tomorrow.

Thanks,
NeilBrown


> 
> Albert
> 

Re: More ddf container woes

[Albert Pauw]

  Hi Neil,

thanks, yes I noticed with the new git stuff some problems are fixed now.

I noticed one more thing:

When I look at the end of the output of the "mdadm -E /dev/md127" output I
see it mentions the amount of phyiscal disks. When I fail a disk it is 
marked as
"active/Offline, Failed" which is good. When I remove it, the amount of 
physical
disks reported by the "mdadm -E" command stays the same, the RefNo is still
there, the Size is still there, the Device file is removed and the state 
is still
"active/Offline, Failed". The whole entry should be removed and the 
amount of
physical disks lowered by one.

When I re-add the failed disk (but NOT zeroed the superblock) the state 
is still
"active/Offline, Failed" but reused for resynching a failed RAID set.

Assuming that the failed state of a disk is also recorded in the 
superblock on the disk
three different possibilities are likely when adding a disk:

- A clean new disk, a new superblock is created with a new RefNo
- A failed disk is added, use the failed state and RefNo
- A good disk is added, possibly from a good RAID set, use this 
superblock with the
RefNo and status. Make it possible to reassemble the RAID set when all 
the disks
are added.

Thanks for the fixes so far,

regards,

Albert

On 03/14/11 09:02 AM, NeilBrown wrote:
> On Fri, 11 Mar 2011 12:50:16 +0100 Albert Pauw<albert.pauw@gmail.com>  wrote:
>
>>    More experiments with the same setup
>>
> Hi Albert,
>   thanks again for this testing.
>
>> To sum it up, there are two problems here:
>>
>> - A failed disk in a subarray isn't automatically removed and marked
>> "Failed" in the container, although in some cases it does (see above).
>> Only after a manual "mdmon --all" will this take place.
> I think this is fixed in my devel-3.2 branch
>
>     git://neil.brown.name/mdadm devel-3.2
>
> Some aspects of it a fixed in the 'master' branch, but removing a
> device properly from a container won't be fixed in 3.1.x, only 3.2.x
>
>> - When two subarrays have failed disks, are degraded, but operational
>> and I add a spare disk to the container, both will pick up the spare
>> disk for replacement. They won't do this in parallel, but in sequence,
>> but nevertheless use the same disk.
> I haven't fixed this yet, but can easily duplicate it.  There are a
> couple of issues here that I need to think through before I get
> it fixed properly.
>
> Hopefully tomorrow.
>
> Thanks,
> NeilBrown
>
>
>> Albert
>>
>

Re: More ddf container woes

[NeilBrown]

On Mon, 14 Mar 2011 10:00:17 +0100 Albert Pauw <albert.pauw@gmail.com> wrote:

>   Hi Neil,
> 
> thanks, yes I noticed with the new git stuff some problems are fixed now.
> 
> I noticed one more thing:
> 
> When I look at the end of the output of the "mdadm -E /dev/md127" output I
> see it mentions the amount of phyiscal disks. When I fail a disk it is 
> marked as
> "active/Offline, Failed" which is good. When I remove it, the amount of 
> physical
> disks reported by the "mdadm -E" command stays the same, the RefNo is still
> there, the Size is still there, the Device file is removed and the state 
> is still
> "active/Offline, Failed". The whole entry should be removed and the 
> amount ofen
> physical disks lowered by one.

Well... maybe.  Probably.

The DDF spec "requires" that there be an entry in the "physical disks"
table for every disk that is connected to the controller - whether failed
or not.
That makes some sense when you think about a hardware-RAID controller.
But how does that apply when DDF is running on a host system rather than
a RAID controller??
Maybe we should only remove them when they are physically unplugged??

There would probably be value in thinking through all of this a lot more
but for now I have arranged to remove any failed device that it not
part of an array (even a failed part).

You can find all of this in my git tree.  I decided to back-port the
code from devel-3.2 which deletes devices from the DDF when you remove
them from a container - so you should find the code in the 'master'
branch works as well as that in 'devel-3.2'.

I would appreciate any more testing results that you come up with.



> 
> When I re-add the failed disk (but NOT zeroed the superblock) the state 
> is still
> "active/Offline, Failed" but reused for resynching a failed RAID set.
> 
> Assuming that the failed state of a disk is also recorded in the 
> superblock on the disk
> three different possibilities are likely when adding a disk:
> 
> - A clean new disk, a new superblock is created with a new RefNo
> - A failed disk is added, use the failed state and RefNo
> - A good disk is added, possibly from a good RAID set, use this 
> superblock with the
> RefNo and status. Make it possible to reassemble the RAID set when all 
> the disks
> are added.

It currently seems to preserve the 'failed' state.  While that may
not be ideal, it is not clearly 'wrong' and can be worked-around
by zeroing the metadata.

So I plan to leave it as it is for the moment.

I hope to put a bit of time in to sorting out some of these more subtle
issues next week - so you could well see progress in the future ...
especially if you have a brilliant idea about how it *should* work and manage
to convince me :-)



> 
> Thanks for the fixes so far,

And thank you for your testing.

NeilBrown

Re: More ddf container woes

[Albert Pauw]

  Hi Neil,

I updated to the git version (devel) and tried my "old" tricks:

- Create a container with 5 disks
- Created two raid sets (raid 1 md0 and raid 5 md1) in this container

mdadm -E  /dev/md127 shows all disks active/Online

- Failed one disk in md0

mdadm -E /dev/md127 shows this disk as active/Offline, Failed

- Failed one disk in md1

mdadm -E /dev/md127 shows this disk as active/Offline, Failed

- Added a new spare disk to the container

mdadm -E /dev/md127 shows this new disk as active/Online, Rebuilding

this looks good, but although the container has six disks, the lastly failed
disk is missing, mdadm -E /dev/md127 only shows five disks (including 
the rebuilding one).

This time however, only one of the failed raid sets is rebuilding, so 
that fix is ok.

Here is another scenario with strange implications:

- Created a container with 6 disks

mdadm -E /dev/md127 shows all 6 disks as Global-Spare/Online

- Removed one of the disks, as I only needed 5

This time mdadm -e /dev/md127 shows six physical disks, one of which has 
no device

- Created two raid sets (raid 1 md0 and raid 5 md1) in this container

mdadm -E  /dev/md127 shows all disks active/Online, except the "empty 
entry" which stays
Global-Spare/Online

- I fail two disks, one in each raid array

mdadm -E /dev/md127 shows these two disks as active/Offline, Failed

- I add back the disk I removed earlier, it should fit into the empty 
slot of mdadm -E

mdadm -E /dev/md127 shows something very strange, namely
-> All disks are not set to Global-Spare/Online
-> All device files are removed from the slots in mdadm -E, except the 
newly added one,
which shows the correct device

Albert

On 03/15/11 05:43 AM, NeilBrown wrote:
> On Mon, 14 Mar 2011 10:00:17 +0100 Albert Pauw<albert.pauw@gmail.com>  wrote:
>
>>    Hi Neil,
>>
>> thanks, yes I noticed with the new git stuff some problems are fixed now.
>>
>> I noticed one more thing:
>>
>> When I look at the end of the output of the "mdadm -E /dev/md127" output I
>> see it mentions the amount of phyiscal disks. When I fail a disk it is
>> marked as
>> "active/Offline, Failed" which is good. When I remove it, the amount of
>> physical
>> disks reported by the "mdadm -E" command stays the same, the RefNo is still
>> there, the Size is still there, the Device file is removed and the state
>> is still
>> "active/Offline, Failed". The whole entry should be removed and the
>> amount ofen
>> physical disks lowered by one.
> Well... maybe.  Probably.
>
> The DDF spec "requires" that there be an entry in the "physical disks"
> table for every disk that is connected to the controller - whether failed
> or not.
> That makes some sense when you think about a hardware-RAID controller.
> But how does that apply when DDF is running on a host system rather than
> a RAID controller??
> Maybe we should only remove them when they are physically unplugged??
>
> There would probably be value in thinking through all of this a lot more
> but for now I have arranged to remove any failed device that it not
> part of an array (even a failed part).
>
> You can find all of this in my git tree.  I decided to back-port the
> code from devel-3.2 which deletes devices from the DDF when you remove
> them from a container - so you should find the code in the 'master'
> branch works as well as that in 'devel-3.2'.
>
> I would appreciate any more testing results that you come up with.
>
>
>
>> When I re-add the failed disk (but NOT zeroed the superblock) the state
>> is still
>> "active/Offline, Failed" but reused for resynching a failed RAID set.
>>
>> Assuming that the failed state of a disk is also recorded in the
>> superblock on the disk
>> three different possibilities are likely when adding a disk:
>>
>> - A clean new disk, a new superblock is created with a new RefNo
>> - A failed disk is added, use the failed state and RefNo
>> - A good disk is added, possibly from a good RAID set, use this
>> superblock with the
>> RefNo and status. Make it possible to reassemble the RAID set when all
>> the disks
>> are added.
> It currently seems to preserve the 'failed' state.  While that may
> not be ideal, it is not clearly 'wrong' and can be worked-around
> by zeroing the metadata.
>
> So I plan to leave it as it is for the moment.
>
> I hope to put a bit of time in to sorting out some of these more subtle
> issues next week - so you could well see progress in the future ...
> especially if you have a brilliant idea about how it *should* work and manage
> to convince me :-)
>
>
>
>> Thanks for the fixes so far,
> And thank you for your testing.
>
> NeilBrown
>

Re: More ddf container woes

[Albert Pauw]

  Hi Neil,

I noticed on your 3.1.5 announcement that there were ddf fixes as well.

I tried the stuff I mentioned before (see below), but those issues 
weren't fixed.

I hope you will have some time to look into this.

Regards,

Albert Pauw

-------- Original Message --------

  Hi Neil,

I updated to the git version (devel) and tried my "old" tricks:

- Create a container with 5 disks
- Created two raid sets (raid 1 md0 and raid 5 md1) in this container

mdadm -E  /dev/md127 shows all disks active/Online

- Failed one disk in md0

mdadm -E /dev/md127 shows this disk as active/Offline, Failed

- Failed one disk in md1

mdadm -E /dev/md127 shows this disk as active/Offline, Failed

- Added a new spare disk to the container

mdadm -E /dev/md127 shows this new disk as active/Online, Rebuilding

this looks good, but although the container has six disks, the lastly failed
disk is missing, mdadm -E /dev/md127 only shows five disks (including
the rebuilding one).

This time however, only one of the failed raid sets is rebuilding, so
that fix is ok.

Here is another scenario with strange implications:

- Created a container with 6 disks

mdadm -E /dev/md127 shows all 6 disks as Global-Spare/Online

- Removed one of the disks, as I only needed 5

This time mdadm -e /dev/md127 shows six physical disks, one of which has
no device

- Created two raid sets (raid 1 md0 and raid 5 md1) in this container

mdadm -E  /dev/md127 shows all disks active/Online, except the "empty
entry" which stays
Global-Spare/Online

- I fail two disks, one in each raid array

mdadm -E /dev/md127 shows these two disks as active/Offline, Failed

- I add back the disk I removed earlier, it should fit into the empty
slot of mdadm -E

mdadm -E /dev/md127 shows something very strange, namely
->  All disks are set to Global-Spare/Online
->  All device files are removed from the slots in mdadm -E, except the
newly added one,
which shows the correct device

Albert

Re: More ddf container woes

[NeilBrown]

On Wed, 23 Mar 2011 20:18:24 +0100 Albert Pauw <albert.pauw@gmail.com> wrote:

>   Hi Neil,
> 
> I noticed on your 3.1.5 announcement that there were ddf fixes as well.

Yes, they are mainly the fixes that you had helped me find and so already
knew about.

> 
> I tried the stuff I mentioned before (see below), but those issues 
> weren't fixed.
> 
> I hope you will have some time to look into this.

Hopefully, but probably not for a few weeks - I need to take a break from
mdadm and catch up with some other stuff.

I'll keep this message to refer back to when I do get back to mdadm.

Thanks for your help,

NeilBrown


> 
> Regards,
> 
> Albert Pauw
> 
> -------- Original Message --------
> 
>   Hi Neil,
> 
> I updated to the git version (devel) and tried my "old" tricks:
> 
> - Create a container with 5 disks
> - Created two raid sets (raid 1 md0 and raid 5 md1) in this container
> 
> mdadm -E  /dev/md127 shows all disks active/Online
> 
> - Failed one disk in md0
> 
> mdadm -E /dev/md127 shows this disk as active/Offline, Failed
> 
> - Failed one disk in md1
> 
> mdadm -E /dev/md127 shows this disk as active/Offline, Failed
> 
> - Added a new spare disk to the container
> 
> mdadm -E /dev/md127 shows this new disk as active/Online, Rebuilding
> 
> this looks good, but although the container has six disks, the lastly failed
> disk is missing, mdadm -E /dev/md127 only shows five disks (including
> the rebuilding one).
> 
> This time however, only one of the failed raid sets is rebuilding, so
> that fix is ok.
> 
> Here is another scenario with strange implications:
> 
> - Created a container with 6 disks
> 
> mdadm -E /dev/md127 shows all 6 disks as Global-Spare/Online
> 
> - Removed one of the disks, as I only needed 5
> 
> This time mdadm -e /dev/md127 shows six physical disks, one of which has
> no device
> 
> - Created two raid sets (raid 1 md0 and raid 5 md1) in this container
> 
> mdadm -E  /dev/md127 shows all disks active/Online, except the "empty
> entry" which stays
> Global-Spare/Online
> 
> - I fail two disks, one in each raid array
> 
> mdadm -E /dev/md127 shows these two disks as active/Offline, Failed
> 
> - I add back the disk I removed earlier, it should fit into the empty
> slot of mdadm -E
> 
> mdadm -E /dev/md127 shows something very strange, namely
> ->  All disks are set to Global-Spare/Online
> ->  All device files are removed from the slots in mdadm -E, except the
> newly added one,
> which shows the correct device
> 
> Albert
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 

Version 3.2.5 and ddf issues (bugreport)

[Albert Pauw]

Hi Neil,

After a hiatus of 1.5 year (busy with all sorts) I am back and tried the 
ddf code to see how things improved.

I build a VM Centos 6.3 system with 6 extra 1GB disks for testing.
I found several issues in the standard installed 3.2.3 version of mdadm 
relating to ddf, but installed the
3.2.5 version in order to work with recent code.

However, while version 3.2.3 is able to create a ddf container with 
raidsets in it, I found a problem with the 3.2.5 version.

After initially creating the container:

mdadm -C /dev/md127 -e ddf -l container /dev/sd[b-g]

which worked, I created a raid (1 or 5 it doesn't matter in this case) 
in it:

mdadm -C /dev/md0 -l raid5 -n 3 /dev/md127

However, it stays on resync=PENDING and readonly, and doesn't get build.

So I tried to set it to readwrite:

mdadm --readwrite  /dev/md0

Unfortunately, it stays on readonly and doesn't get build.

As said before, this did work in 3.2.3.

Are you already on this problem?

Kind regards,

Albert

P.S. My idea was, because the ddf code is not that much used, to create 
a test/validate script for ddf.

Re: Version 3.2.5 and ddf issues (bugreport)

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1771 bytes --]

On Sat, 28 Jul 2012 13:46:06 +0200 Albert Pauw <albert.pauw@gmail.com> wrote:

> Hi Neil,
> 
> After a hiatus of 1.5 year (busy with all sorts) I am back and tried the 
> ddf code to see how things improved.

Thanks!

> 
> I build a VM Centos 6.3 system with 6 extra 1GB disks for testing.
> I found several issues in the standard installed 3.2.3 version of mdadm 
> relating to ddf, but installed the
> 3.2.5 version in order to work with recent code.
> 
> However, while version 3.2.3 is able to create a ddf container with 
> raidsets in it, I found a problem with the 3.2.5 version.
> 
> After initially creating the container:
> 
> mdadm -C /dev/md127 -e ddf -l container /dev/sd[b-g]
> 
> which worked, I created a raid (1 or 5 it doesn't matter in this case) 
> in it:
> 
> mdadm -C /dev/md0 -l raid5 -n 3 /dev/md127
> 
> However, it stays on resync=PENDING and readonly, and doesn't get build.
> 
> So I tried to set it to readwrite:
> 
> mdadm --readwrite  /dev/md0
> 
> Unfortunately, it stays on readonly and doesn't get build.
> 
> As said before, this did work in 3.2.3.
> 
> Are you already on this problem?

It sounds like a problem with 'mdmon'.  mdmon needs to be running before the
array can become read-write.  mdadm should start mdmon automatically but
maybe it isn't.  Maybe it cannot find mdmon?

could you check if mdadm is running?  If it isn't run
   mdmon /dev/md127 &
and see if it starts working.

> 
> Kind regards,
> 
> Albert
> 
> P.S. My idea was, because the ddf code is not that much used, to create 
> a test/validate script for ddf.
> 
There is one 'ddf' test in my test suite and I think it is working.  I'm
happy to add more tests if you suggest or write some.

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Version 3.2.5 and ddf issues (bugreport)

[Albert Pauw]

On 07/31/2012 08:11 AM, NeilBrown wrote:
> On Sat, 28 Jul 2012 13:46:06 +0200 Albert Pauw <albert.pauw@gmail.com> wrote:
>
>> Hi Neil,
>>
>> After a hiatus of 1.5 year (busy with all sorts) I am back and tried the
>> ddf code to see how things improved.
> Thanks!
>
>> I build a VM Centos 6.3 system with 6 extra 1GB disks for testing.
>> I found several issues in the standard installed 3.2.3 version of mdadm
>> relating to ddf, but installed the
>> 3.2.5 version in order to work with recent code.
>>
>> However, while version 3.2.3 is able to create a ddf container with
>> raidsets in it, I found a problem with the 3.2.5 version.
>>
>> After initially creating the container:
>>
>> mdadm -C /dev/md127 -e ddf -l container /dev/sd[b-g]
>>
>> which worked, I created a raid (1 or 5 it doesn't matter in this case)
>> in it:
>>
>> mdadm -C /dev/md0 -l raid5 -n 3 /dev/md127
>>
>> However, it stays on resync=PENDING and readonly, and doesn't get build.
>>
>> So I tried to set it to readwrite:
>>
>> mdadm --readwrite  /dev/md0
>>
>> Unfortunately, it stays on readonly and doesn't get build.
>>
>> As said before, this did work in 3.2.3.
>>
>> Are you already on this problem?
> It sounds like a problem with 'mdmon'.  mdmon needs to be running before the
> array can become read-write.  mdadm should start mdmon automatically but
> maybe it isn't.  Maybe it cannot find mdmon?
>
> could you check if mdadm is running?  If it isn't run
>     mdmon /dev/md127 &
> and see if it starts working.
Hi Neil,

thanks for your reply. Yes, mdmon wasn't running. Couldn't get it 
running with a recompiled 3.2.5, the standard one which came with Centos 
(3.2.3) works fine, I assume the made some changes to the code? Anyway, 
I moved to my own laptop, running Fedora 16 and pulled mdadm frm git and 
recompiled. That works. I also used loop devices as disks.

Here is the first of my findings:

I created a container with six disks, disk 1-2 is a raid 1 device, disk 
3-6 are a raid 6 device.

Here is the table shown at the end of the mdadm -E command for the 
container:

  Physical Disks : 6
       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    65956a72    479232K /dev/loop1      active/Online

I now fail a disk (disk 0) and I get:

  Physical Disks : 6
       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    65956a72    479232K /dev/loop1      active/Offline, Failed

Then I removed the disk from the container:

  Physical Disks : 6
       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    65956a72    479232K                 active/Offline, 
Failed, Missing

Notice the active/Offline status, is this correct?

I added the disk back into the container, NO zero-superblock:

  Physical Disks : 6
       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    65956a72    479232K /dev/loop1      active/Offline, 
Failed, Missing

It stays active/Offline (this is now correct I assume), Failed (again 
correct if had failed before), but also still missing.

I remove the disk again, do a zero-superblock and add it again:

Physical Disks : 6
       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    ede51ba3    479232K /dev/loop1      active/Online, Rebuilding

This is correct, the disk is seen as a new disk and rebuilding starts.


Regards,

Albert

Re: Version 3.2.5 and ddf issues (bugreport)

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 5609 bytes --]

On Tue, 31 Jul 2012 10:46:26 +0200 Albert Pauw <albert.pauw@gmail.com> wrote:

> On 07/31/2012 08:11 AM, NeilBrown wrote:
> > On Sat, 28 Jul 2012 13:46:06 +0200 Albert Pauw <albert.pauw@gmail.com> wrote:
> >
> >> Hi Neil,
> >>
> >> After a hiatus of 1.5 year (busy with all sorts) I am back and tried the
> >> ddf code to see how things improved.
> > Thanks!
> >
> >> I build a VM Centos 6.3 system with 6 extra 1GB disks for testing.
> >> I found several issues in the standard installed 3.2.3 version of mdadm
> >> relating to ddf, but installed the
> >> 3.2.5 version in order to work with recent code.
> >>
> >> However, while version 3.2.3 is able to create a ddf container with
> >> raidsets in it, I found a problem with the 3.2.5 version.
> >>
> >> After initially creating the container:
> >>
> >> mdadm -C /dev/md127 -e ddf -l container /dev/sd[b-g]
> >>
> >> which worked, I created a raid (1 or 5 it doesn't matter in this case)
> >> in it:
> >>
> >> mdadm -C /dev/md0 -l raid5 -n 3 /dev/md127
> >>
> >> However, it stays on resync=PENDING and readonly, and doesn't get build.
> >>
> >> So I tried to set it to readwrite:
> >>
> >> mdadm --readwrite  /dev/md0
> >>
> >> Unfortunately, it stays on readonly and doesn't get build.
> >>
> >> As said before, this did work in 3.2.3.
> >>
> >> Are you already on this problem?
> > It sounds like a problem with 'mdmon'.  mdmon needs to be running before the
> > array can become read-write.  mdadm should start mdmon automatically but
> > maybe it isn't.  Maybe it cannot find mdmon?
> >
> > could you check if mdadm is running?  If it isn't run
> >     mdmon /dev/md127 &
> > and see if it starts working.
> Hi Neil,
> 
> thanks for your reply. Yes, mdmon wasn't running. Couldn't get it 
> running with a recompiled 3.2.5, the standard one which came with Centos 
> (3.2.3) works fine, I assume the made some changes to the code? Anyway, 
> I moved to my own laptop, running Fedora 16 and pulled mdadm frm git and 
> recompiled. That works. I also used loop devices as disks.
> 
> Here is the first of my findings:
> 
> I created a container with six disks, disk 1-2 is a raid 1 device, disk 
> 3-6 are a raid 6 device.
> 
> Here is the table shown at the end of the mdadm -E command for the 
> container:
> 
>   Physical Disks : 6
>        Number    RefNo      Size       Device      Type/State
>           0    06a5f547    479232K /dev/loop2      active/Online
>           1    47564acc    479232K /dev/loop3      active/Online
>           2    bf30692c    479232K /dev/loop5      active/Online
>           3    275d02f5    479232K /dev/loop4      active/Online
>           4    b0916b3f    479232K /dev/loop6      active/Online
>           5    65956a72    479232K /dev/loop1      active/Online
> 
> I now fail a disk (disk 0) and I get:
> 
>   Physical Disks : 6
>        Number    RefNo      Size       Device      Type/State
>           0    06a5f547    479232K /dev/loop2      active/Online
>           1    47564acc    479232K /dev/loop3      active/Online
>           2    bf30692c    479232K /dev/loop5      active/Online
>           3    275d02f5    479232K /dev/loop4      active/Online
>           4    b0916b3f    479232K /dev/loop6      active/Online
>           5    65956a72    479232K /dev/loop1      active/Offline, Failed
> 
> Then I removed the disk from the container:
> 
>   Physical Disks : 6
>        Number    RefNo      Size       Device      Type/State
>           0    06a5f547    479232K /dev/loop2      active/Online
>           1    47564acc    479232K /dev/loop3      active/Online
>           2    bf30692c    479232K /dev/loop5      active/Online
>           3    275d02f5    479232K /dev/loop4      active/Online
>           4    b0916b3f    479232K /dev/loop6      active/Online
>           5    65956a72    479232K                 active/Offline, 
> Failed, Missing
> 
> Notice the active/Offline status, is this correct?
> 
> I added the disk back into the container, NO zero-superblock:
> 
>   Physical Disks : 6
>        Number    RefNo      Size       Device      Type/State
>           0    06a5f547    479232K /dev/loop2      active/Online
>           1    47564acc    479232K /dev/loop3      active/Online
>           2    bf30692c    479232K /dev/loop5      active/Online
>           3    275d02f5    479232K /dev/loop4      active/Online
>           4    b0916b3f    479232K /dev/loop6      active/Online
>           5    65956a72    479232K /dev/loop1      active/Offline, 
> Failed, Missing
> 
> It stays active/Offline (this is now correct I assume), Failed (again 
> correct if had failed before), but also still missing.
> 
> I remove the disk again, do a zero-superblock and add it again:
> 
> Physical Disks : 6
>        Number    RefNo      Size       Device      Type/State
>           0    06a5f547    479232K /dev/loop2      active/Online
>           1    47564acc    479232K /dev/loop3      active/Online
>           2    bf30692c    479232K /dev/loop5      active/Online
>           3    275d02f5    479232K /dev/loop4      active/Online
>           4    b0916b3f    479232K /dev/loop6      active/Online
>           5    ede51ba3    479232K /dev/loop1      active/Online, Rebuilding
> 
> This is correct, the disk is seen as a new disk and rebuilding starts.
> 
> 
> Regards,
> 
> Albert

Hi Albert,

thanks for this and your other reports.
I won't be able to look at them for a while, but hopefully will get back to
you some time next week.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Version 3.2.5 and ddf issues (bugreport)

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 6452 bytes --]

On Tue, 31 Jul 2012 10:46:26 +0200 Albert Pauw <albert.pauw@gmail.com> wrote:

> On 07/31/2012 08:11 AM, NeilBrown wrote:
> > On Sat, 28 Jul 2012 13:46:06 +0200 Albert Pauw <albert.pauw@gmail.com> wrote:
> >
> >> Hi Neil,
> >>
> >> After a hiatus of 1.5 year (busy with all sorts) I am back and tried the
> >> ddf code to see how things improved.
> > Thanks!
> >
> >> I build a VM Centos 6.3 system with 6 extra 1GB disks for testing.
> >> I found several issues in the standard installed 3.2.3 version of mdadm
> >> relating to ddf, but installed the
> >> 3.2.5 version in order to work with recent code.
> >>
> >> However, while version 3.2.3 is able to create a ddf container with
> >> raidsets in it, I found a problem with the 3.2.5 version.
> >>
> >> After initially creating the container:
> >>
> >> mdadm -C /dev/md127 -e ddf -l container /dev/sd[b-g]
> >>
> >> which worked, I created a raid (1 or 5 it doesn't matter in this case)
> >> in it:
> >>
> >> mdadm -C /dev/md0 -l raid5 -n 3 /dev/md127
> >>
> >> However, it stays on resync=PENDING and readonly, and doesn't get build.
> >>
> >> So I tried to set it to readwrite:
> >>
> >> mdadm --readwrite  /dev/md0
> >>
> >> Unfortunately, it stays on readonly and doesn't get build.
> >>
> >> As said before, this did work in 3.2.3.
> >>
> >> Are you already on this problem?
> > It sounds like a problem with 'mdmon'.  mdmon needs to be running before the
> > array can become read-write.  mdadm should start mdmon automatically but
> > maybe it isn't.  Maybe it cannot find mdmon?
> >
> > could you check if mdadm is running?  If it isn't run
> >     mdmon /dev/md127 &
> > and see if it starts working.
> Hi Neil,
> 
> thanks for your reply. Yes, mdmon wasn't running. Couldn't get it 
> running with a recompiled 3.2.5, the standard one which came with Centos 
> (3.2.3) works fine, I assume the made some changes to the code? Anyway, 
> I moved to my own laptop, running Fedora 16 and pulled mdadm frm git and 
> recompiled. That works. I also used loop devices as disks.
> 
> Here is the first of my findings:
> 
> I created a container with six disks, disk 1-2 is a raid 1 device, disk 
> 3-6 are a raid 6 device.
> 
> Here is the table shown at the end of the mdadm -E command for the 
> container:
> 
>   Physical Disks : 6
>        Number    RefNo      Size       Device      Type/State
>           0    06a5f547    479232K /dev/loop2      active/Online
>           1    47564acc    479232K /dev/loop3      active/Online
>           2    bf30692c    479232K /dev/loop5      active/Online
>           3    275d02f5    479232K /dev/loop4      active/Online
>           4    b0916b3f    479232K /dev/loop6      active/Online
>           5    65956a72    479232K /dev/loop1      active/Online
> 
> I now fail a disk (disk 0) and I get:
> 
>   Physical Disks : 6
>        Number    RefNo      Size       Device      Type/State
>           0    06a5f547    479232K /dev/loop2      active/Online
>           1    47564acc    479232K /dev/loop3      active/Online
>           2    bf30692c    479232K /dev/loop5      active/Online
>           3    275d02f5    479232K /dev/loop4      active/Online
>           4    b0916b3f    479232K /dev/loop6      active/Online
>           5    65956a72    479232K /dev/loop1      active/Offline, Failed
> 
> Then I removed the disk from the container:
> 
>   Physical Disks : 6
>        Number    RefNo      Size       Device      Type/State
>           0    06a5f547    479232K /dev/loop2      active/Online
>           1    47564acc    479232K /dev/loop3      active/Online
>           2    bf30692c    479232K /dev/loop5      active/Online
>           3    275d02f5    479232K /dev/loop4      active/Online
>           4    b0916b3f    479232K /dev/loop6      active/Online
>           5    65956a72    479232K                 active/Offline, 
> Failed, Missing
> 
> Notice the active/Offline status, is this correct?

To be honest, I don't know.  The DDF spec doesn't really go into that sort of
detail, or at least I didn't find it.
Given that the device is Missing, it hardly seems to matter whether it is
Active or Spare or Foreign or Legacy.
I guess if it re-appears we want to know what it was ... maybe.

> 
> I added the disk back into the container, NO zero-superblock:
> 
>   Physical Disks : 6
>        Number    RefNo      Size       Device      Type/State
>           0    06a5f547    479232K /dev/loop2      active/Online
>           1    47564acc    479232K /dev/loop3      active/Online
>           2    bf30692c    479232K /dev/loop5      active/Online
>           3    275d02f5    479232K /dev/loop4      active/Online
>           4    b0916b3f    479232K /dev/loop6      active/Online
>           5    65956a72    479232K /dev/loop1      active/Offline, 
> Failed, Missing
> 
> It stays active/Offline (this is now correct I assume), Failed (again 
> correct if had failed before), but also still missing.

I found why this happens.  When I added code to support incremental assembly
of DDF arrays, I broke the ability to hot-add a device which happened to have
reasonably good looking metadata on it.  The best approach for now is to
--zero the device first.  I'll push out a patch which does just that.


> 
> I remove the disk again, do a zero-superblock and add it again:
> 
> Physical Disks : 6
>        Number    RefNo      Size       Device      Type/State
>           0    06a5f547    479232K /dev/loop2      active/Online
>           1    47564acc    479232K /dev/loop3      active/Online
>           2    bf30692c    479232K /dev/loop5      active/Online
>           3    275d02f5    479232K /dev/loop4      active/Online
>           4    b0916b3f    479232K /dev/loop6      active/Online
>           5    ede51ba3    479232K /dev/loop1      active/Online, Rebuilding
> 
> This is correct, the disk is seen as a new disk and rebuilding starts.
> 
> 
> Regards,
> 
> Albert

diff --git a/Manage.c b/Manage.c
index f83af65..7f27f74 100644
--- a/Manage.c
+++ b/Manage.c
@@ -786,6 +786,7 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
 			return -1;
 		}
 
+		Kill(dv->devname, NULL, 0, -1, 0);
 		dfd = dev_open(dv->devname, O_RDWR | O_EXCL|O_DIRECT);
 		if (mdmon_running(tst->container_dev))
 			tst->update_tail = &tst->updates;



Thanks,
NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]