handling mdmon in the initramfs

handling mdmon in the initramfs

[Dan Williams]

Hi,

As I learned from Hans and Harald at Plumbers, mdadm and mdmon currently 
have a few sharp edges when being handled in the initramfs environment. 
  In talking over some proposed fixes there was a question about the 
full set of requirements.  Here is a rundown of the problems and 
proposed solutions...

Problem 1: Ensuring mdmon is active while writes may be in flight
The kernel will block writes to member disks that have failed and all 
writes while the array is not in the 'active' state.  For these reasons 
mdmon is needed in the initramfs because some file systems write to the 
backing device, even when mounting read-only, to recover their journal.

However, once that is done Neil points out that mdmon will not be needed 
again until the filesystem is mounted read-write.  Even if the array 
goes degraded as a result of running the startup scripts the kernel will 
allow reads to pass, so we may not need rigid 100% mdmon coverage.

Two strategies for this situation are to stop mdmon after mounting the 
rootfs, or just let it be terminated as a result of starting a new 
instance from the final rootfs.  The latter approach brings up the 
question of how to communicate with the initramfs-mdmon-instance to make 
sure we do not end up with two mdmon instances servicing the same 
container.  The proposed solution here is to switch to 
abstract-namespace-sockets removing the need to drop a socket file.

Problem 2: Discovery / Assembly
Several issues have forced dracut to punt on using mdadm -I.  Instead 
dracut copies mdadm.conf to the initramfs and uses mdadm -As after a 
udevadm --settle.  One low hanging issue is the fact that non-rootfs 
arrays may only be partially assembled when dracut discovers and 
switches to the final rootfs.  Upon switching the in-progress map file 
is lost.  Moving /var/run/mdadm/map to /dev/.mdadm/map would appear to 
solve this issue.

There was also a report about an udev event storm during incremental 
assembly, but I am not clear on the sequence of events?

Any other missed requirements or problems with the proposed solutions?

Thanks,
Dan
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: handling mdmon in the initramfs

[Neil Brown]

On Thursday October 1, dan.j.williams-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org wrote:
> Hi,
> 
> As I learned from Hans and Harald at Plumbers, mdadm and mdmon currently 
> have a few sharp edges when being handled in the initramfs environment. 
>   In talking over some proposed fixes there was a question about the 
> full set of requirements.  Here is a rundown of the problems and 
> proposed solutions...

Thanks for this.

> 
> Problem 1: Ensuring mdmon is active while writes may be in flight
> The kernel will block writes to member disks that have failed and all 
> writes while the array is not in the 'active' state.  For these reasons 
> mdmon is needed in the initramfs because some file systems write to the 
> backing device, even when mounting read-only, to recover their journal.
> 
> However, once that is done Neil points out that mdmon will not be needed 
> again until the filesystem is mounted read-write.  Even if the array 
> goes degraded as a result of running the startup scripts the kernel will 
> allow reads to pass, so we may not need rigid 100% mdmon coverage.
> 
> Two strategies for this situation are to stop mdmon after mounting the 
> rootfs, or just let it be terminated as a result of starting a new 
> instance from the final rootfs.  The latter approach brings up the 
> question of how to communicate with the initramfs-mdmon-instance to make 
> sure we do not end up with two mdmon instances servicing the same 
> container.  The proposed solution here is to switch to 
> abstract-namespace-sockets removing the need to drop a socket file.

What exactly do you mean by "abstract-namespace-sockets"??

I would much rather just down mdmon before pivot_root while everything
is read-only, and start it up again afterwards.

> 
> Problem 2: Discovery / Assembly
> Several issues have forced dracut to punt on using mdadm -I.  Instead 
> dracut copies mdadm.conf to the initramfs and uses mdadm -As after a 
> udevadm --settle.  One low hanging issue is the fact that non-rootfs 
> arrays may only be partially assembled when dracut discovers and 
> switches to the final rootfs.  Upon switching the in-progress map file 
> is lost.  Moving /var/run/mdadm/map to /dev/.mdadm/map would appear to 
> solve this issue.

mdadm already uses /dev/.mdadm.map if /var/run is not writable.  So is
this a solved problem, or do we need some other way to force mdadm not
to use /var/run too early.

Thanks,
NeilBrown


> 
> There was also a report about an udev event storm during incremental 
> assembly, but I am not clear on the sequence of events?
> 
> Any other missed requirements or problems with the proposed solutions?
> 
> Thanks,
> Dan
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: handling mdmon in the initramfs

[Dan Williams]

Neil Brown wrote:
>> Two strategies for this situation are to stop mdmon after mounting the 
>> rootfs, or just let it be terminated as a result of starting a new 
>> instance from the final rootfs.  The latter approach brings up the 
>> question of how to communicate with the initramfs-mdmon-instance to make 
>> sure we do not end up with two mdmon instances servicing the same 
>> container.  The proposed solution here is to switch to 
>> abstract-namespace-sockets removing the need to drop a socket file.
> 
> What exactly do you mean by "abstract-namespace-sockets"??

Harald pointed me to this.  It's the hack that udev uses for its control 
socket [1].  You create a unix domain socket as usual but make the first 
character of the file name a NULL byte.

> 
> I would much rather just down mdmon before pivot_root while everything
> is read-only, and start it up again afterwards.
> 
>> Problem 2: Discovery / Assembly
>> Several issues have forced dracut to punt on using mdadm -I.  Instead 
>> dracut copies mdadm.conf to the initramfs and uses mdadm -As after a 
>> udevadm --settle.  One low hanging issue is the fact that non-rootfs 
>> arrays may only be partially assembled when dracut discovers and 
>> switches to the final rootfs.  Upon switching the in-progress map file 
>> is lost.  Moving /var/run/mdadm/map to /dev/.mdadm/map would appear to 
>> solve this issue.
> 
> mdadm already uses /dev/.mdadm.map if /var/run is not writable.  So is
> this a solved problem, or do we need some other way to force mdadm not
> to use /var/run too early.

So you are talking about commit cf3a3d78 [2].  I think the problem is 
that /var/run/mdadm/map is writable in dracut so we never fall back to 
/dev/.mdadm.map.  Any reason to not make /dev/.mdadm.map the first 
location to try?

Thanks,
Dan

[1]: 
http://git.kernel.org/?p=linux/hotplug/udev.git;a=blob;f=libudev/libudev-ctrl.c;h=e47b2b66#l65

[2]: http://neil.brown.name/git?p=mdadm;a=commitdiff;h=cf3a3d78

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: handling mdmon in the initramfs

[Neil Brown]

On Thursday October 1, dan.j.williams-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org wrote:
> Neil Brown wrote:
> >> Two strategies for this situation are to stop mdmon after mounting the 
> >> rootfs, or just let it be terminated as a result of starting a new 
> >> instance from the final rootfs.  The latter approach brings up the 
> >> question of how to communicate with the initramfs-mdmon-instance to make 
> >> sure we do not end up with two mdmon instances servicing the same 
> >> container.  The proposed solution here is to switch to 
> >> abstract-namespace-sockets removing the need to drop a socket file.
> > 
> > What exactly do you mean by "abstract-namespace-sockets"??
> 
> Harald pointed me to this.  It's the hack that udev uses for its control 
> socket [1].  You create a unix domain socket as usual but make the first 
> character of the file name a NULL byte.

ooohhhh..... that is soooo ugly!
And given that anyone can bind to any 'abstract' name, there is room
for a local denial-of-service attack there too.

I guess we could use it if we really had to, but I'd rather avoid it
if possible.

> 
> > 
> > I would much rather just down mdmon before pivot_root while everything
> > is read-only, and start it up again afterwards.
> > 
> >> Problem 2: Discovery / Assembly
> >> Several issues have forced dracut to punt on using mdadm -I.  Instead 
> >> dracut copies mdadm.conf to the initramfs and uses mdadm -As after a 
> >> udevadm --settle.  One low hanging issue is the fact that non-rootfs 
> >> arrays may only be partially assembled when dracut discovers and 
> >> switches to the final rootfs.  Upon switching the in-progress map file 
> >> is lost.  Moving /var/run/mdadm/map to /dev/.mdadm/map would appear to 
> >> solve this issue.
> > 
> > mdadm already uses /dev/.mdadm.map if /var/run is not writable.  So is
> > this a solved problem, or do we need some other way to force mdadm not
> > to use /var/run too early.
> 
> So you are talking about commit cf3a3d78 [2].  I think the problem is 
> that /var/run/mdadm/map is writable in dracut so we never fall back to 
> /dev/.mdadm.map.  Any reason to not make /dev/.mdadm.map the first 
> location to try?

Because (damn it) the file is not a device file, it is a
run-time-management file and so it belongs in /var/run.  If we just
put things wherever was convenient instead of where they belonged we
would end up with something like /proc!

If there is a writable /var/run in early boot, presumably it is a
tmpfs filesystem.
Would it not be simple to
   mount --bind /var/run /root/var/run
before the pivot_root, or maybe "mount --move"?
Then everything would benefit from a stable /var/run and nothing would
have to put silly inappropriate files in /dev.

On the other hand, if /var/run is not a tmpfs filesystem and is
normally part of the real /var, what is the justification for having a
writable /var/run in early boot?  Sounds like a recipe for confusion.

But if it really has to be that way, you could create symlinks from
/var/run/mdadm/map and /var/run/mdadm.map to /nowhere in the
early-boot /var/run, and that would force mdadm to use
/dev/.mdadm.map.

NeilBrown


> 
> Thanks,
> Dan
> 
> [1]: 
> http://git.kernel.org/?p=linux/hotplug/udev.git;a=blob;f=libudev/libudev-ctrl.c;h=e47b2b66#l65
> 
> [2]: http://neil.brown.name/git?p=mdadm;a=commitdiff;h=cf3a3d78
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: handling mdmon in the initramfs

[Kay Sievers]

On Fri, Oct 2, 2009 at 05:31, Neil Brown <neilb-l3A5Bk7waGM@public.gmane.org> wrote:
> On Thursday October 1, dan.j.williams-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org wrote:
>> Neil Brown wrote:
>> >> Two strategies for this situation are to stop mdmon after mounting the
>> >> rootfs, or just let it be terminated as a result of starting a new
>> >> instance from the final rootfs.  The latter approach brings up the
>> >> question of how to communicate with the initramfs-mdmon-instance to make
>> >> sure we do not end up with two mdmon instances servicing the same
>> >> container.  The proposed solution here is to switch to
>> >> abstract-namespace-sockets removing the need to drop a socket file.
>> >
>> > What exactly do you mean by "abstract-namespace-sockets"??
>>
>> Harald pointed me to this.  It's the hack that udev uses for its control
>> socket [1].  You create a unix domain socket as usual but make the first
>> character of the file name a NULL byte.
>
> ooohhhh..... that is soooo ugly!
> And given that anyone can bind to any 'abstract' name, there is room
> for a local denial-of-service attack there too.
>
> I guess we could use it if we really had to, but I'd rather avoid it
> if possible.

Right, you can only do that, if you are 100% sure, that the service
always runs and binds the socket before any other user can log in,
like udev does.

Kay
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: handling mdmon in the initramfs

[Hans de Goede]

Hi,

On 10/02/2009 01:23 AM, Dan Williams wrote:
> Hi,
>
> As I learned from Hans and Harald at Plumbers, mdadm and mdmon currently
> have a few sharp edges when being handled in the initramfs environment.
> In talking over some proposed fixes there was a question about the full
> set of requirements. Here is a rundown of the problems and proposed
> solutions...
>
> Problem 1: Ensuring mdmon is active while writes may be in flight
> The kernel will block writes to member disks that have failed and all
> writes while the array is not in the 'active' state. For these reasons
> mdmon is needed in the initramfs because some file systems write to the
> backing device, even when mounting read-only, to recover their journal.
>
> However, once that is done Neil points out that mdmon will not be needed
> again until the filesystem is mounted read-write. Even if the array goes
> degraded as a result of running the startup scripts the kernel will
> allow reads to pass, so we may not need rigid 100% mdmon coverage.
>

I'm not sure this is true, I had mdmon crashing on hand over from initramfs
-> real root (the malloc vs calloc thing) and IIRC, this causes to hang rc.sysinit
way before getting around the checking the filesystems. Notice that
checking the FS also requires R/W access!

This may have to do something with us calling "mdadm -As --run" from rc.sysinit
before checking the FS, maybe that wants to communicate with mdmon ?

> Two strategies for this situation are to stop mdmon after mounting the
> rootfs, or just let it be terminated as a result of starting a new
> instance from the final rootfs.

Ack, and I must say this is the solution I prefer, lets not try to play the
lets hope nothing needs mdmon before we restart it game, I've done too much
reboots of a hanging system due to mdmon crashing (about 70 I guess) to think
this is a good idea.

 > The latter approach brings up the
> question of how to communicate with the initramfs-mdmon-instance to make
> sure we do not end up with two mdmon instances servicing the same
> container. The proposed solution here is to switch to
> abstract-namespace-sockets removing the need to drop a socket file.
>
> Problem 2: Discovery / Assembly
> Several issues have forced dracut to punt on using mdadm -I. Instead
> dracut copies mdadm.conf to the initramfs and uses mdadm -As after a
> udevadm --settle. One low hanging issue is the fact that non-rootfs
> arrays may only be partially assembled when dracut discovers and
> switches to the final rootfs. Upon switching the in-progress map file is
> lost. Moving /var/run/mdadm/map to /dev/.mdadm/map would appear to solve
> this issue.
>
> There was also a report about an udev event storm during incremental
> assembly, but I am not clear on the sequence of events?
>

The problem is that assembly in general, causes a whole slew of udev change
events being emitted from the /dev/md# node. It would be nice if this could
be reduced somewhat. Esp as we do a "mdadm --detail --export" on each change
event. I've also seen the "mdadm --detail --export" not work (not return any
info) because (I think) the /dev/md# node was not ready yet.

Also see:
https://bugzilla.redhat.com/show_bug.cgi?id=523387

Note that the biggest problem is the partially assembled arrays when we
switch root though (and the "mdadm --detail --export" called from the udev
rules sometimes not working).

Regards,

Hans
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: handling mdmon in the initramfs

[Hans de Goede]

Hi,

On 10/02/2009 05:31 AM, Neil Brown wrote:

<snip>

>>>> Problem 2: Discovery / Assembly
>>>> Several issues have forced dracut to punt on using mdadm -I.  Instead
>>>> dracut copies mdadm.conf to the initramfs and uses mdadm -As after a
>>>> udevadm --settle.  One low hanging issue is the fact that non-rootfs
>>>> arrays may only be partially assembled when dracut discovers and
>>>> switches to the final rootfs.  Upon switching the in-progress map file
>>>> is lost.  Moving /var/run/mdadm/map to /dev/.mdadm/map would appear to
>>>> solve this issue.
>>>
>>> mdadm already uses /dev/.mdadm.map if /var/run is not writable.  So is
>>> this a solved problem, or do we need some other way to force mdadm not
>>> to use /var/run too early.
>>
>> So you are talking about commit cf3a3d78 [2].  I think the problem is
>> that /var/run/mdadm/map is writable in dracut so we never fall back to
>> /dev/.mdadm.map.  Any reason to not make /dev/.mdadm.map the first
>> location to try?
>
> Because (damn it) the file is not a device file, it is a
> run-time-management file and so it belongs in /var/run.  If we just
> put things wherever was convenient instead of where they belonged we
> would end up with something like /proc!
>
> If there is a writable /var/run in early boot, presumably it is a
> tmpfs filesystem.
> Would it not be simple to
>     mount --bind /var/run /root/var/run
> before the pivot_root, or maybe "mount --move"?
> Then everything would benefit from a stable /var/run and nothing would
> have to put silly inappropriate files in /dev.
>
> On the other hand, if /var/run is not a tmpfs filesystem and is
> normally part of the real /var, what is the justification for having a
> writable /var/run in early boot?  Sounds like a recipe for confusion.
>

There is no such thing in the initrd (a writable /var/run), but there is
a writable / (initrd == ramdisk), and mdadm will happily create /var/run
itself, if it would not do that, there would no issue.

Although I do wonder how later, when we do have a writable /var/run, mdadm
decides which file to use. Once it has used /dev/.mdadm.map once it should
keep on using that.

Regards,

Hans
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: handling mdmon in the initramfs

[Neil Brown]

On Friday October 2, hdegoede-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org wrote:
> Hi,
> 
> There is no such thing in the initrd (a writable /var/run), but there is
> a writable / (initrd == ramdisk), and mdadm will happily create /var/run
> itself, if it would not do that, there would no issue.

mdadm does not create directories (or at least not for the map file).
So if /var/run does not exist, it will use a file in /dev.

> 
> Although I do wonder how later, when we do have a writable /var/run, mdadm
> decides which file to use. Once it has used /dev/.mdadm.map once it should
> keep on using that.

It always tries the first of
  /var/run/mdadm/map  /var/run/mdadm.map /dev/.mdadm.map

it will normally read, then write.
So if /var/run doesn't exist, it will use /dev/.mdadm.map
Once /var/run/mdadm exists, it will
  read from /dev/.mdadm.map
  write to /var/run/mdadm/map

and from then on use /var/run/mdadm/map

So if the initramfs really does not contain /var/run, then this aspect
should "just work".

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: handling mdmon in the initramfs

[Hans de Goede]

Hi,

On 10/02/2009 09:39 AM, Neil Brown wrote:
> On Friday October 2, hdegoede-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org wrote:
>> Hi,
>>
>> There is no such thing in the initrd (a writable /var/run), but there is
>> a writable / (initrd == ramdisk), and mdadm will happily create /var/run
>> itself, if it would not do that, there would no issue.
>
> mdadm does not create directories (or at least not for the map file).
> So if /var/run does not exist, it will use a file in /dev.
>
>>
>> Although I do wonder how later, when we do have a writable /var/run, mdadm
>> decides which file to use. Once it has used /dev/.mdadm.map once it should
>> keep on using that.
>
> It always tries the first of
>    /var/run/mdadm/map  /var/run/mdadm.map /dev/.mdadm.map
>
> it will normally read, then write.
> So if /var/run doesn't exist, it will use /dev/.mdadm.map
> Once /var/run/mdadm exists, it will
>    read from /dev/.mdadm.map
>    write to /var/run/mdadm/map
>
> and from then on use /var/run/mdadm/map
>
> So if the initramfs really does not contain /var/run, then this aspect
> should "just work".
>

Ok,

Harald, I know we recently added some stuff to dracut with symlinks under
/var/run/mdadm. Since since then, we've opted to instead patch mdmon
to store its state under /dev/.mdadm (until a better fix is upstream), so the
symlink stuff can be removed and then the map stuff should just work.

Note that we will then still have issues with incremental assembly, as
dracut does incremental assembly, and if we then have partly assembled
arrays when switching over to /, rc.sysinit does a
mdadm -As --run, assembling everything in mdadm.conf, which will still
cause issues if some of them are already partly incremental assembled. But
I don't consider this a mdadm bug, but a distro one :)

Regards,

Hans
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html