[mdadm git pull] "--assemble --scan" support for imsm

[mdadm git pull] "--assemble --scan" support for imsm

[Dan Williams]

Hi Neil,

This update allows imsm containers and member arrays to be discovered
with '-Eb' and subsequently assembled with '-As'.

For example here is a test for assembling a 4 drive container with (1)
4-disk raid5 array plus one spare.

# echo "DEVICES /dev/loop[0-4]" > tmp/mdadm.conf

# mdadm -Ebsc tmp/mdadm.conf >> tmp/mdadm.conf

# cat tmp/mdadm.conf
DEVICES /dev/loop[0-4]
ARRAY /dev/imsm metadata=imsm auto=md UUID=b98f5dbe-aa859e7b-0e369b89-a80986d4
ARRAY /dev/md/r1 container=/dev/imsm member=0 auto=mdp UUID=3538e39c-b397c2e9-1aa031f9-2bc0eca4
   spares=1

# mdadm -Asvc tmp/mdadm.conf
mdadm: looking for devices for /dev/imsm
mdadm: /dev/loop4 is identified as a member of /dev/imsm, slot -1.
mdadm: /dev/loop3 is identified as a member of /dev/imsm, slot -1.
mdadm: /dev/loop2 is identified as a member of /dev/imsm, slot -1.
mdadm: /dev/loop1 is identified as a member of /dev/imsm, slot -1.
mdadm: /dev/loop0 is identified as a member of /dev/imsm, slot -1.
mdadm: added /dev/loop3 to /dev/imsm as -1
mdadm: added /dev/loop2 to /dev/imsm as -1
mdadm: added /dev/loop1 to /dev/imsm as -1
mdadm: added /dev/loop0 to /dev/imsm as -1
mdadm: added /dev/loop4 to /dev/imsm as -1
mdadm: Container /dev/imsm has been assembled with 5 drives
mdadm: looking to assemble member array 0 inside container /dev/imsm
mdadm: match found for member 0

I suppose brief_examine_super_imsm() can be updated to look at the
currently running kernel and not specify auto=mdp if the "extended
partition" capability is available.  Is there a quick way to check for
this capability?  You can see that I tagged this support as
"preliminary" as it does not attempt to address the multiple container
case, I left it alone for now as that solution depends on the device
name rework.

Please have a look.

Thanks,
Dan

The following changes since commit b01b06bda8dce132e6eb3c3826ad0f4b94ebdf43:
  NeilBrown (1):
        Merge branch 'master' into devel-3.0

are available in the git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/djbw/mdadm.git devel

Dan Williams (7):
      update copyright headers
      imsm: update metadata immediately on "add spare" events
      Examine: fix MD_DISK_SYNC is a bit not a flag
      imsm: return associated uuid for spares
      imsm: copy raid device info when associating spares
      imsm: include members in ->brief_examine
      Preliminary -As support for container member arrays

 Assemble.c    |   46 ++++++++++++++++++++++++++++++++++++++++++++++
 Examine.c     |    2 +-
 Incremental.c |    6 ++++++
 config.c      |    2 ++
 managemon.c   |   19 +++++++++++++++++++
 mdadm.h       |    1 +
 mdmon.c       |   28 +++++++++++++++++++---------
 mdmon.h       |   20 ++++++++++++++++++++
 monitor.c     |   19 +++++++++++++++++++
 sg_io.c       |    2 +-
 super-intel.c |   53 ++++++++++++++++++++++++++++++++++++++++++++---------
 util.c        |    9 +++++++++
 12 files changed, 187 insertions(+), 20 deletions(-)

Re: [mdadm git pull] "--assemble --scan" support for imsm

[Neil Brown]

On Tuesday October 28, dan.j.williams@intel.com wrote:
> Hi Neil,
> 
> This update allows imsm containers and member arrays to be discovered
> with '-Eb' and subsequently assembled with '-As'.

Thanks Dan.  There's definitely some useful stuff in here.
Some of it may be made a lot simpler by some revamping I hoping to get
done before the weekend...

> 
> For example here is a test for assembling a 4 drive container with (1)
> 4-disk raid5 array plus one spare.
> 
> # echo "DEVICES /dev/loop[0-4]" > tmp/mdadm.conf
> 
> # mdadm -Ebsc tmp/mdadm.conf >> tmp/mdadm.conf
> 
> # cat tmp/mdadm.conf
> DEVICES /dev/loop[0-4]
> ARRAY /dev/imsm metadata=imsm auto=md UUID=b98f5dbe-aa859e7b-0e369b89-a80986d4
> ARRAY /dev/md/r1 container=/dev/imsm member=0 auto=mdp UUID=3538e39c-b397c2e9-1aa031f9-2bc0eca4
>    spares=1

I don't like the use of "/dev/imsm" is here.  It obviously doesn't
scale to more that one imsm array, and while we all know that the
world doesn't need more than about 5 computers, I suspect someone out
there will find 1 IMSM array limiting :-)

My approach for the "/dev/imsm" in the first line is simple to discard
it.  I've changed mdadm to allow the device name to be missing from
mdadm.conf and to not generate in for imsm and ddf.  When "mdadm -As"
tries to assemble such a device it makes up a name, which may not be
stable between reboots.  But if you are using mount-by-label then this
shouldn't be a problem.  And that seems to be the way the world is
going.

The "container=/dev/imsm" is slightly harder to deal with.  Just
leaving that out means there is no direct linkage between the two
lines.  That might be a problem, except for the next point, which
somewhat makes it moot.

One change I'm introducing in 3.0 is that 'homehost' will never be
NULL.  It now defaults to <system> which expands via 'gethostname()'.
One consequence of this is that
  mdadm -As
will, after it has processed all it can find in mdadm.conf will
attempt auto-assembly of anything else it can find.  Things that
aren't identified as belonging to 'this host' will still be assembled,
but with a guaranteed unique name.
This means that
   mdadm -As
with an empty mdadm.conf will now assemble everything it can find.
Hopefully I'll get around to coding it so they are assembled
'read-auto'.

If you do
  mdadm -Es > /tmp/mdadm.conf
  mdadm -Asc/tmp/mdadm.conf

you will get one slight difference.  Every array will be assumed to
belong to 'this' host (because they are listed in our local
mdadm.conf)  and mdadm will be a little more generous in giving
meaningful names.
Also, the members of a container are local if the container is local,
so you don't really need to list the members in the output of "-Es".

I'll try to make sure it still works if the members are listed without
a "container=" setting.  I toyed with the idea of supporting
"container=previous" or similar.  It's a bit ugly though.

In short, I plan to take all you patches.  Remove the references to
"/dev/imsm" and then make it all "do the right thing".

BTW, I'm currently prohibiting names like "/dev/imsm".  You would need
to use "/dev/md/imsm".  How much would that bother you?
I'm not completely committed to this, but it is a lot easier to impose
a more uniform naming scheme.

> 
> I suppose brief_examine_super_imsm() can be updated to look at the
> currently running kernel and not specify auto=mdp if the "extended
> partition" capability is available.  Is there a quick way to check for
> this capability?  You can see that I tagged this support as
> "preliminary" as it does not attempt to address the multiple container
> case, I left it alone for now as that solution depends on the device
> name rework.

I'm tempted to allow the metadata to provide hints as to whether
partitions are needed.  So you wouldn't need "auto=mdp" in mdadm.conf
- super-intel.c would somehow say "use partitions" (and 'ddf' would
say "don't") and mdadm would do as it is told.
The only way to check for the extended-minor-space is to check the
kernel version.  I prefer to avoid doing that where possible as it
breaks when distros backport things, but I think in this case it is
justified (and if a distro did back-port the extended minor space
without updating mdadm it would all still work, just slightly
differently).

> 
> Please have a look.

I have. Thanks.

I will try to have something credible on top of it pushed out by
tomorrow evening (24 hours from now).  I might even call it
mdadm-3.0-devel2 (which I've been promising for a couple of weeks
without delivering).

Thanks,
NeilBrown

Re: [mdadm git pull] "--assemble --scan" support for imsm

[Dan Williams]

On Wed, Oct 29, 2008 at 2:19 AM, Neil Brown <neilb@suse.de> wrote:
> The "container=/dev/imsm" is slightly harder to deal with.  Just
> leaving that out means there is no direct linkage between the two
> lines.  That might be a problem, except for the next point, which
> somewhat makes it moot.
>
> One change I'm introducing in 3.0 is that 'homehost' will never be
> NULL.  It now defaults to <system> which expands via 'gethostname()'.
> One consequence of this is that
>  mdadm -As
> will, after it has processed all it can find in mdadm.conf will
> attempt auto-assembly of anything else it can find.  Things that
> aren't identified as belonging to 'this host' will still be assembled,
> but with a guaranteed unique name.
> This means that
>   mdadm -As
> with an empty mdadm.conf will now assemble everything it can find.
> Hopefully I'll get around to coding it so they are assembled
> 'read-auto'.
>
> If you do
>  mdadm -Es > /tmp/mdadm.conf
>  mdadm -Asc/tmp/mdadm.conf
>
> you will get one slight difference.  Every array will be assumed to
> belong to 'this' host (because they are listed in our local
> mdadm.conf)  and mdadm will be a little more generous in giving
> meaningful names.
> Also, the members of a container are local if the container is local,
> so you don't really need to list the members in the output of "-Es".
>
> I'll try to make sure it still works if the members are listed without
> a "container=" setting.  I toyed with the idea of supporting
> "container=previous" or similar.  It's a bit ugly though.
>

What about container=<container uuid> as we can never really discern
with certainty the name of the container device at
->brief_examine_super() time?

> In short, I plan to take all you patches.  Remove the references to
> "/dev/imsm" and then make it all "do the right thing".
>
> BTW, I'm currently prohibiting names like "/dev/imsm".  You would need
> to use "/dev/md/imsm".  How much would that bother you?
> I'm not completely committed to this, but it is a lot easier to impose
> a more uniform naming scheme.
>

I've been using /dev/imsm out of habit, but I can see how that might
cause problems with the /dev namespace.  /dev/md/imsm makes more
sense.

> I will try to have something credible on top of it pushed out by
> tomorrow evening (24 hours from now).  I might even call it
> mdadm-3.0-devel2 (which I've been promising for a couple of weeks
> without delivering).
>

Sounds good, thanks,
Dan

Re: [mdadm git pull] "--assemble --scan" support for imsm

[Neil Brown]

On Wednesday October 29, dan.j.williams@intel.com wrote:
> 
> What about container=<container uuid> as we can never really discern
> with certainty the name of the container device at
> ->brief_examine_super() time?

Yes, could do that.  It feels a bit ugly, but it probably makes sense.

But I've had another look at your 'provisional' thing and understand it
better and want to suggest some enhancements (Which might look like a
complete rewrite, but the important part of noticing the problem is
still yours :-)

You call Incremental_container from Assemble and that really isn't
right.  Each call to 'Assemble'  should assemble just one array, but
Incremental_container might assemble several.

Assemble() has the following basic structure:

 1/ walk through list of available devices determining which ones
    match the 'identity' of the required array.
 2/ Check the set of devices for consistency and make any updates that
    have been requested
 3/ Assemble the array from the selected devices, and start it.

With a slight variation for the 'auto-assemble' case wherein the
'identity' to match is taken from the first device that is found to be
part of an array that is not already assembled.

To fit the assembly of a specific member of a container into this
model, we need to have the 'container' in the list of available
devices.
If the identity specifies 'container=whatever' then we clearly select
all devices which match that.  You would expect at exactly one - the
container.  You would then need to call ->container_content on that
container and find the correct member array which matches the
'member=' specifier (or any other specifier there might be?)

Exactly how updates and "--force" are passed though would need to be
sorted out.
Then the devices in the selected array from ->container_content could
be passed to sysfs_add_disk and the array started.

Auto-assembly would discover that the first unused-so-far device was a
container, and would need to load the list of arrays and assemble the
first one that was not yet assembled.

I think that is the 'right' thing to do, but it seems like a lot of
work just for now, so I might leave it for a while and see if
something else occurs to me.

Thanks,
NeilBrown

Re: [mdadm git pull] "--assemble --scan" support for imsm

[Neil Brown]

On Wednesday October 29, neilb@suse.de wrote:
> 
> I will try to have something credible on top of it pushed out by
> tomorrow evening (24 hours from now).  I might even call it
> mdadm-3.0-devel2 (which I've been promising for a couple of weeks
> without delivering).


Well, I didn't quite make that, things always take longer than you
expect.
But I have pushed out a 'scratch-3.0' which is my current working
code.  Some of it will probably be revised which is why it isn't in
devel-3.0 yet.  But it is there to be looked at (or laughed at?) and
maybe tested a bit.

You might like to run with

 export MDADM_NO_UDEV=1

so that it doesn't depend on udev to create all the links, as some of
that doesn't quite work right yet.

NeilBrown

Re: [mdadm git pull] "--assemble --scan" support for imsm

[Dan Williams]

On Wed, 2008-10-29 at 20:42 -0700, Neil Brown wrote:
> To fit the assembly of a specific member of a container into this
> model, we need to have the 'container' in the list of available
> devices.
> If the identity specifies 'container=whatever' then we clearly select
> all devices which match that.  You would expect at exactly one - the
> container.  You would then need to call ->container_content on that
> container and find the correct member array which matches the
> 'member=' specifier (or any other specifier there might be?)
> 
> Exactly how updates and "--force" are passed though would need to be
> sorted out.
> Then the devices in the selected array from ->container_content could
> be passed to sysfs_add_disk and the array started.
> 
> Auto-assembly would discover that the first unused-so-far device was a
> container, and would need to load the list of arrays and assemble the
> first one that was not yet assembled.
> 

I had a go at this, and the result is pushed out to my scratch-devel
branch.  mdadm can now:

1/ Scan and assemble container members
mdadm -Es > mdadm.conf
cat mdadm.conf
ARRAY metadata=imsm auto=md UUID=88f0b659-966978a7-842f86c0-879a414d
ARRAY /dev/md/r1 container=88f0b659-966978a7-842f86c0-879a414d
   member=0 auto=mdp UUID=42d6de62-bd27b267-96e8e960-13efc3a6
ARRAY /dev/md/r2 container=88f0b659-966978a7-842f86c0-879a414d
   member=1 auto=mdp UUID=b0764c97-8b4fdf0e-0821862e-a80f1452
mdadm -Asc mdadm.conf
mdadm: Container /dev/md/imsm127 has been assembled with 6 drives
mdadm: Started /dev/md/r1 with 6 devices
mdadm: Started /dev/md/r2 with 6 devices

2/ Assemble member array by uuid
mdadm -A /dev/md/r1 /dev/md/imsm -a mdp -u 42d6de62-bd27b267-96e8e960-13efc3a6

3/ Auto-assemble member arrays as foreign arrays

This still does not handle updates or --force... it seems awkward to try
and support --force at anything other than the container level, but that
is the extent of my thinking so far.

The following changes since commit fb9253084ab0b9e4ac1113e4fc8f0b88d818d4f8:
  NeilBrown (1):
        mdopen:  fix up name parsing.

are available in the git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/djbw/mdadm.git scratch-devel

Dan Williams (21):
      Quiet uninitialized variable warnings
      Makefile: fixup install of udev rules
      imsm: match_home(), document why it is stubbed out
      Let symlinks to standard devices count as standard names
      imsm: display member array uuid in examine_super_imsm
      imsm: display container uuid in detail_super
      config: add 'containers' as a DEVICE keyword
      Add a 'container' flag to mddev_dev_t
      Assemble: check if the devlist entry is a container
      Assemble: let trustworthy be LOCAL when array is identified
      Assemble: replace 'info' with 'content'
      Assemble: teach it to look inside containers for matching arrays
      Assemble: discard non-container devices when not required
      cleanup an unused call to container2devname
      config: add containers to the default search list
      Assemble: revert preliminary -As support
      Copy container_dev in dup_super
      Assemble: factor out and call assemble_container_content
      fixup create_mddev
      Assemble: block attempts to reassemble container members
      Assemble: enable member array auto-assembly

 Assemble.c    |  428 ++++++++++++++++++++++++++++++++++++++-------------------
 Incremental.c |   99 +------------
 Makefile      |    4 +-
 config.c      |   46 ++++++-
 mdadm.c       |    1 +
 mdadm.h       |   17 ++-
 mdmon.c       |    9 ++
 mdopen.c      |    7 +-
 super-intel.c |   37 ++++--
 util.c        |  155 ++++++++++++++++++++-
 10 files changed, 542 insertions(+), 261 deletions(-)

Re: [mdadm git pull] "--assemble --scan" support for imsm

[Neil Brown]

On Sunday November 2, dan.j.williams@intel.com wrote:
> On Wed, 2008-10-29 at 20:42 -0700, Neil Brown wrote:
> > To fit the assembly of a specific member of a container into this
> > model, we need to have the 'container' in the list of available
> > devices.
> > If the identity specifies 'container=whatever' then we clearly select
> > all devices which match that.  You would expect at exactly one - the
> > container.  You would then need to call ->container_content on that
> > container and find the correct member array which matches the
> > 'member=' specifier (or any other specifier there might be?)
> > 
> > Exactly how updates and "--force" are passed though would need to be
> > sorted out.
> > Then the devices in the selected array from ->container_content could
> > be passed to sysfs_add_disk and the array started.
> > 
> > Auto-assembly would discover that the first unused-so-far device was a
> > container, and would need to load the list of arrays and assemble the
> > first one that was not yet assembled.
> > 
> 
> I had a go at this, and the result is pushed out to my scratch-devel
> branch.

Thanks.  I took your code, kept some bits, fixed some bits, rewrote
other bits and completely replaced some more :-)  And added some of my
code too.

I've just pushed the result to my devel-3.0 branch

  git pull git://neil.brown.name/mdadm devel-3.0

It get a clean "make everything" and the self tests all run (I hope -
it's late and I cannot wait for the final run).

Tomorrow I'm going to hack at the man page, and write some more test
scripts.

With the current code

  mdadm -As

should find and assemble all of your arrays.  And give fairly useful
names.

I haven't included your

   Let symlinks to standard devices count as standard names

patch because I don't think I agree with it, and I don't think it is
needed any more.

I also have left out your

     imsm: match_home(), document why it is stubbed out
    
because I don't understand.  Maybe some more words about how
'family_num' can distinguish between "Local" and "Foreign". ??

Thanks heaps,

NeilBrown
 

Re: [mdadm git pull] "--assemble --scan" support for imsm

[Dan Williams]

Neil Brown wrote:
> I haven't included your
> 
>    Let symlinks to standard devices count as standard names
> 
> patch because I don't think I agree with it, and I don't think it is
> needed any more.
> 

My concern was that:
for i in `seq 1 5`
do
	mdadm -As
	mdadm -Ss
done

...would create 5 different device numbers.  Your right it is better 
now, but between successive assemblies the device numbers still toggle 
between two different values.  So it's just a small cosmetic issue now. 
  Are their lifetime issues that preclude predictable device numbers?

> I also have left out your
> 
>      imsm: match_home(), document why it is stubbed out
>     
> because I don't understand.  Maybe some more words about how
> 'family_num' can distinguish between "Local" and "Foreign". ??
> 

...it doesn't.  'family_num' is a checksum that includes the serial 
numbers of the component disks when the array was first created.  So it 
can help determine when two arrays were created in different places, but 
the metadata can't identify which, if any, is "Local".

I'll redo the comment.

Thanks for the pull and the review,
Dan

Re: [mdadm git pull] "--assemble --scan" support for imsm

[Dan Williams]

[ added linux-raid with permission ]

On 8/16/2010 5:17 PM, K. Posern wrote:
> Dear Mr. Williams,
>
> I am a bit desperate that's why I dare to contact you like this:
> As I did not find any answer to my below 2 questions in man-pages, in
> the source-code (quick look with grep+less), on irc channels and of
> course with google, I am hoping that you can give me any hint or refer
> me to somebody who might.
>
> I adore Intels engagement for linux. Unfortunately is there no website
> (yet?): "intellinuxraid.org" (like there is: intellinuxgraphics.org).

There is no website.  My personal thought is that we should extend the 
content at raid.wiki.kernel.org.

> The only official information from intel I could dig up so fare states:
> imsm is supported with mdadm >=v3 and dmraid >=v1
>
> But I am trying to find out if it is possible to:
>
> a) Autoassemble the RAID (so avoid having an initrd)

No, the auto-assemble I mentioned in the pull request will not avoid 
having an initrd.  You are thinking of "in-kernel autodetect" which is 
not supported with the imsm metadata format (or any metadata format 
besides the md-v0.90 format).  In general assembling arrays in userspace 
is much more flexible which is why you see the in-kernel autodetect code 
has been put in a no-new-features/maintenance-only mode.

I know Fedora and OpenSUSE have added support for this configuration to 
their initrd tools.

> b) BOOT linux from this: grub2? lilo? ... anything?

One of the primary benefits of using the Intel(R) RST (mdadm-imsm) 
format is that you can use the platform bios option-rom to boot from 
raid.  So grub, grub2, lilo, or any other bootloader can boot from raid. 
  This is because the option-rom installs a bios int13 handler that 
allows the bootloader to read from a raid volume as if it were a 
standalone disk.

> About (a):
> The git pull request seems to indicate thate there is auto-assembly
> support for imsm software raids?!
> Is this true? - again: grep

The "mdadm -As" command will search all available disks and 
automatically assemble all the imsm arrays that it finds.

> What RAID-Levels (for an imsm fakeraid) are supported?

I think the 'fakeraid' moniker is tired and misplaced when referring to 
this new functionality in md/mdadm.  There is nothing fake about Linux 
software RAID.  I prefer "bios-raid" or even "software raid".

Current platforms ship support for raid levels 0, 1, 10, and 5.

> Especially: What about RAID-0?

Supported.

>
> About (b):
> According to "Kyron" and his wiki:
> http://wiki.neuralbs.com/index.php/Gentoo_Quick_Install_notes#GRUB2
> grub2 should support imsm (but grep for imsm in the Bazaar checkout does
> not yield any hit) <<< I contacted the grub2 mailing-list for clarification

Like I mentioned above, the bootloader does not need any special code to 
boot from a bios-raid array because it can use standard int13 services.

> I really hope you can help me. Most importantly with question (a).
>
>
> Yours sincerely,
>
> K. Posern
>

Regards,
Dan

RE: [mdadm git pull] "--assemble --scan" support for imsm

[Jiang, Dave]

> -----Original Message-----
> From: Williams, Dan J
> Sent: Tuesday, August 17, 2010 9:49 AM
> To: K. Posern
> Cc: linux-raid@vger.kernel.org; Jiang, Dave
> Subject: Re: [mdadm git pull] "--assemble --scan" support for imsm
> 
> [ added linux-raid with permission ]
> 
> On 8/16/2010 5:17 PM, K. Posern wrote:
> > Dear Mr. Williams,
> >
> > I am a bit desperate that's why I dare to contact you like this:
> > As I did not find any answer to my below 2 questions in man-pages, in
> > the source-code (quick look with grep+less), on irc channels and of
> > course with google, I am hoping that you can give me any hint or
> refer
> > me to somebody who might.
> >
> > I adore Intels engagement for linux. Unfortunately is there no
> website
> > (yet?): "intellinuxraid.org" (like there is: intellinuxgraphics.org).
> 
> There is no website.  My personal thought is that we should extend the
> content at raid.wiki.kernel.org.

I have updated raid.wiki with the external metadata information a few weeks back. Please take a look and let me know if additional information is required. Thanks!