Re: [[Patch mdadm] 2/5] Move the files mdmon opens into /dev/ to support handoff after pivotroot

[Dan Williams <dan.j.williams-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>]

>> 1/ If you take a look at native md superblock support you see that the
>> support code is duplicated between kernel-space and user space, having
>> it all handled in userspace means only one code base to maintain
>> (elegant aspect #1).
>
> Elegance is in the eye of the beholder.  More on that in a minute.
>

True, but let's agree that superblock formats are quirky, arbitrary
and all around inelegant.  Only needing to write that code once is at
the very least an aid to one's sanity.

>> 2/ The kernel can simply worry about the *mechanism* of providing raid
>> while all the assembly *policy* and support for any number of
>> superblock formats is relegated to where policy belongs (elegant
>> aspect #2).
>
> I would argue that dirty/clean state manipulation is *not* policy and
> *is* mechanism.  So, by your definition of what should be in the kernel
> combined with my definition of what dirty/clean state manipulation is,
> the solution is not only not elegant, it's flat incorrect.

You are conveniently blurring the lines between event generation and
event handling.  The kernel handles all the detail of detecting,
notifying and reaping the event.  The arbitrary superblock specific
actions that need to happen in response to that event are really not
very interesting to rest of the mechanism of providing raid.  You
could argue that I am conveniently drawing a line, and you would be
right.  There are convenient aspects of having this portion of the
solution in userspace which do not compromise the integrity of the
raid mechanism.

We can now also handle spare assignment policy, hot-plug policy,
corner case disagreements between a superblock's definition of a
"container", all without thrashing the kernel.

>
>> 2a/ This simply follows in the path of the design decision to not
>> support in-kernel auto-assembly of version-1 superblocks which started
>> the requirement to use an initramfs to boot software raid.  (this is a
>> not so elegant aspect because it mandates an initramfs to boot, but I
>> don't think a general purpose distro can ever get away from that
>> requirement).
>
> I'm fine with needing mdadm to assemble the device.  I'm not fine with
> needing mdmon once it's assembled.
>
>> I will say that needing to touch several software packages (kernel,
>> initramfs, initscripts, mdadm) to get imsm superblock support has
>> added some excitement to the process in the short term.  Long term I
>> think the elegant aspects of the decision will prove their worth.
>
> I will say that needing to touch multiple software packages might not be
> a bad thing, but think of *how* they had to be changed.  We had to add
> special exceptions for mdmon all over the place: kernel scheduler (for
> suspend/resume, mdmon can't be frozen like the rest of user space or
> else writing our suspend to disk image doesn't work), initramfs,
> initscripts after initramfs, initscripts on halt, SELinux.  In all these
> cases, we had to take something that we want to keep simple and add
> special case rules and exceptions for mdmon.  That pretty solidly says
> that while this arrangement may have been elegant for *you*, it was not
> elegant in the overall grand scheme of things.

No, nothing elegant about that, but I think you would agree this isn't
something we threw over the wall and walked away from.  Making mdmon
more convenient to handle is hopefully an obvious priority.  Yes, I
know you would like to see it die, but we are where we are.

>
> What would have been smart was to leave array creation, assembly,
> verfication, and modification to user space, but to put *all* of the
> raid mechanics, including superblock clean/dirty state processing and
> array shut down capabilities, in the kernel.  Had you done that, I would
> have called your solution elegant.
>
> It's at this point that I feel obliged to mention that, in terms of this
> whole big argument, the incremental map file has at least some amount of
> sense belonging in /dev, it's really the mdmon .pid and .sock files that
> don't, and those files wouldn't even exist had you designed things as I
> mention here.  It's the fact that you have two files per device that you
> should be placing in a specific place on the filesystem in order for
> them to be useful and adhere to standards yet the program they belong to
> needs to exist outside the context of any filesystem that I think is
> pretty strong evidence of the inelegance of this design.
>

This comment makes me see Neil's argument in a different light,
(hopefully I am not mischaracterizing it), but essentially we are
waiting for the standards to catch up with this new class of program.
FUSE, CUSE, and mdmon belong to a class of programs that move
traditionally exclusive kernel space functionality to userspace.
Debian's /lib/init/rw looks to be a response to this grey area of the
standards (not that I have any familiarity with the LSB).

--
Dan