initramfs: udev, hotplug, klibc and modprobe

initramfs: udev, hotplug, klibc and modprobe

[Jürg Billeter]

Hi

As I couldn't find a documented way how to use initramfs together with
udev, hotplug, klibc and modprobe, I've done some tests myself. I wanted
to enable booting a fully modular kernel without initrd and custom
hardware detection scripts on any linux-supported hardware (ide, scsi,
usb whatever).

Meanwhile it's gotten to work very well but I had to write some patches,
scripts, little programs myself and so I thought to write down how I
build my initramfs and I'd like to get comments from people actively
working on udev and related technology whether my implementation makes
sense or I've completely misunderstood something.

Following my rough build instructions, my files marked with a footnote

      * Build a fully modular kernel without initramfs (2.6.10)
      * Build klibc (0.196) and copy ash and the static utils to
        initramfs
              * [1] Patch to add a stripped down, klibc-compatible
                version of module-init-tools' modprobe to the build
      * Build udev (050) with klibc and copy it to initramfs
              * [2] Patch to shorten event timeout from 10s to 3s and
                set the expected hotplug seqnum start to 5 to avoid the
                event timeout as I've never seen a hotplug event with a
                seqnum < 5, am I missing some?
              * [3] Patch to add udevinitd and udevinitsend. These are
                modified versions of udevd and udevsend acting as a
                "caching daemon". They save events received during early
                userspace and resend them to late userspace udevd when
                ready. This enables late userspace to process all events
                (for example to load modules not included in initramfs)
                without using coldplugging and thelike.

-- 
Jürg Billeter <j@bitron.ch>



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

initramfs: udev, hotplug, klibc and modprobe

[Juerg Billeter]

Hi

[Ignore my previous message, I hit send way too early...]

As I couldn't find a documented way how to use initramfs together with
udev, hotplug, klibc and modprobe, I've done some tests myself. I wanted
to enable booting a fully modular kernel without initrd and custom
hardware detection scripts on any linux-supported hardware (ide, scsi,
usb whatever).

Meanwhile it's gotten to work very well but I had to write some patches,
scripts, little programs myself and so I thought to write down how I
build my initramfs and I'd like to get comments from people actively
working on udev and related technology whether my implementation makes
sense or I've completely misunderstood something.

Following my rough initramfs kernel build instructions, my files are
marked with a footnote

      * Build a fully modular kernel without initramfs (2.6.10)
      * Build klibc (0.196) and copy ash and the static utils to
        initramfs
              * [1] Patch to add a stripped down, klibc-compatible
                version of module-init-tools' modprobe to the build
      * Build udev (050) with klibc and copy it to initramfs
              * [2] Patch to shorten event timeout from 10s to 3s and
                set the expected hotplug seqnum start to 5 to avoid the
                event timeout as I've never seen a hotplug event with a
                seqnum < 5, am I missing some?
              * [3] Patch to add udevinitd and udevinitsend. These are
                modified versions of udevd and udevsend acting as a
                "caching daemon". They save events received during early
                userspace and resend them to late userspace udevd when
                ready. This enables late userspace to process all events
                (for example to load modules not included in initramfs)
                without using coldplugging and thelike. (udevinitsend
                gets called via hotplug.d)
      * [4] Copy my initial hotplug script to initramfs. This mounts
        /proc and /sys and sets /proc/sys/kernel/hotplug
      * [5] Copy my udevsend-wrapper script to initramfs. This makes
        sure that the hotplug events wait until AF_UNIX has been
        initialized by the kernel.
      * [6] Build my hotplug module loading utility for initramfs. This
        is a minimal replacement of the hotplug agents to improve speed
        and not depend on bash and other tools.
      * [7] Copy my init script to initramfs. This handles calling
        run-init to start late userspace.
      * [8] Copy my init.dev script to dev.d. This mounts the requested
        root filesystem as soon as available. (is able to process
        root=/dev/cdrom)
      * Copy the essential modules to initramfs (including modules.dep)
      * Create initramfs cpio and recreate bzImage with initramfs

The following commands remain to be executed when late userspace is ready:

      * killlall udevd # kill udevd that has been started in initramfs
      * echo /sbin/udevsend > /proc/sys/kernel/hotplug
      * udevd # start new udevd daemon
      * killall -USR1 udevinitd # signals udevinitd to resend the cached
        messages and exit afterwards

If someone wants to try these instructions and needs more specific
commands, you can download the build specification from
http://www.paldo.org/paldo/specs/linux-2.6.xml

All files can be found at
http://www.paldo.org/paldo/sources/linux-2.6/FILENAME

[1] klibc-0.196-modprobe-1.patch.bz2 
[2] udev-050-event-timeout-1.patch.bz2
[3] udev-050-initramfs-1.patch.bz2
[4] initramfs-hotplug-20041226
[5] initramfs-udevsend-wrapper-20041226
[6] initramfs-module.hotplug.c-20041226
[7] initramfs-init-20041227
[8] initramfs-init.dev-20041227

Comments welcome

Jürg

-- 
Jürg Billeter <juerg@paldo.org>



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Kay Sievers]

On Sun, 2005-01-09 at 23:25 +0100, Juerg Billeter wrote:
> As I couldn't find a documented way how to use initramfs together with
> udev, hotplug, klibc and modprobe, I've done some tests myself. I wanted
> to enable booting a fully modular kernel without initrd and custom
> hardware detection scripts on any linux-supported hardware (ide, scsi,
> usb whatever).
> 
> Meanwhile it's gotten to work very well but I had to write some patches,
> scripts, little programs myself and so I thought to write down how I
> build my initramfs and I'd like to get comments from people actively
> working on udev and related technology whether my implementation makes
> sense or I've completely misunderstood something.
> 
> Following my rough initramfs kernel build instructions, my files are
> marked with a footnote
> 
>       * Build a fully modular kernel without initramfs (2.6.10)
>       * Build klibc (0.196) and copy ash and the static utils to
>         initramfs
>               * [1] Patch to add a stripped down, klibc-compatible
>                 version of module-init-tools' modprobe to the build
>       * Build udev (050) with klibc and copy it to initramfs
>               * [2] Patch to shorten event timeout from 10s to 3s and
>                 set the expected hotplug seqnum start to 5 to avoid the
>                 event timeout as I've never seen a hotplug event with a
>                 seqnum < 5, am I missing some?

The next version of udevd will have a timeout of only 2 seconds during a
startup phase. This may not be needed then.

>               * [3] Patch to add udevinitd and udevinitsend. These are
>                 modified versions of udevd and udevsend acting as a
>                 "caching daemon". They save events received during early
>                 userspace and resend them to late userspace udevd when
>                 ready. This enables late userspace to process all events
>                 (for example to load modules not included in initramfs)
>                 without using coldplugging and thelike. (udevinitsend
>                 gets called via hotplug.d)

Nice idea. In a recent discussion, the question came up about a possible
combination of udevstart and coldplugging. It may be possible to
synthesize all the events from the information in sysfs? What do you
think about that in relation to your udevd "event queue".

Thanks,
Kay



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Alexander E. Patrakov]

Kay Sievers wrote:

>>               * [3] Patch to add udevinitd and udevinitsend. These are
>>                 modified versions of udevd and udevsend acting as a
>>                 "caching daemon". They save events received during early
>>                 userspace and resend them to late userspace udevd when
>>                 ready. This enables late userspace to process all events
>>                 (for example to load modules not included in initramfs)
>>                 without using coldplugging and thelike. (udevinitsend
>>                 gets called via hotplug.d)
> 
> Nice idea. In a recent discussion, the question came up about a possible
> combination of udevstart and coldplugging. It may be possible to
> synthesize all the events from the information in sysfs? What do you
> think about that in relation to your udevd "event queue".

See also my old alternative implementation:

http://bugs.linuxfromscratch.org/show_bug.cgi?id†8
http://archive.linuxfromscratch.org/mail-archives/lfs-hackers/2004-July/001729.html

-- 
Alexander E. Patrakov



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Jürg Billeter]

On Mit, 2005-01-12 at 10:07 +0500, Alexander E. Patrakov wrote:
> Kay Sievers wrote:
> 
> >>               * [3] Patch to add udevinitd and udevinitsend. These are
> >>                 modified versions of udevd and udevsend acting as a
> >>                 "caching daemon". They save events received during early
> >>                 userspace and resend them to late userspace udevd when
> >>                 ready. This enables late userspace to process all events
> >>                 (for example to load modules not included in initramfs)
> >>                 without using coldplugging and thelike. (udevinitsend
> >>                 gets called via hotplug.d)
> > 
> > Nice idea. In a recent discussion, the question came up about a possible
> > combination of udevstart and coldplugging. It may be possible to
> > synthesize all the events from the information in sysfs? What do you
> > think about that in relation to your udevd "event queue".
> 
> See also my old alternative implementation:
> 
> http://bugs.linuxfromscratch.org/show_bug.cgi?id†8
> http://archive.linuxfromscratch.org/mail-archives/lfs-hackers/2004-July/001729.html

I've looked at it at that time, too, but there seem to exist some
drawbacks:

- It doesn't support loading modules in initramfs and it seems that this
  couldn't be added easily as udevd holds back some events until the
  child process of previous events finished which never happens with
  the sleeping hotplug processes.
- It doesn't work with run-init which moves the /root mount to /. With
  your initramfs, the whole system will be run in chroot in a way that
  a root user can escape it and get back to the initramfs tmpfs which
  could be not that desirable.
- Some 100 processes test for /root/dev/null every second and will
  be sent in possibly a random order to udevd at once (probably not
  a major performance problem but we should consider it)

Jürg



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Jürg Billeter]

On Mit, 2005-01-12 at 03:37 +0100, Kay Sievers wrote:
> On Sun, 2005-01-09 at 23:25 +0100, Juerg Billeter wrote:
> >       * Build udev (050) with klibc and copy it to initramfs
> >               * [2] Patch to shorten event timeout from 10s to 3s and
> >                 set the expected hotplug seqnum start to 5 to avoid the
> >                 event timeout as I've never seen a hotplug event with a
> >                 seqnum < 5, am I missing some?
> 
> The next version of udevd will have a timeout of only 2 seconds during a
> startup phase. This may not be needed then.

That's nice. Do you know anything about hotplug events with a seqnum
less than 5? If they can never reach udev, maybe expected_seqnum should
be corrected although the better approach would be to "find" the lost
events.

> >               * [3] Patch to add udevinitd and udevinitsend. These are
> >                 modified versions of udevd and udevsend acting as a
> >                 "caching daemon". They save events received during early
> >                 userspace and resend them to late userspace udevd when
> >                 ready. This enables late userspace to process all events
> >                 (for example to load modules not included in initramfs)
> >                 without using coldplugging and thelike. (udevinitsend
> >                 gets called via hotplug.d)
> 
> Nice idea. In a recent discussion, the question came up about a possible
> combination of udevstart and coldplugging. It may be possible to
> synthesize all the events from the information in sysfs? What do you
> think about that in relation to your udevd "event queue".

If really all events could be synthesized including the order/seqnum and
one sets udevd's event queue on hold (to avoid race conditions between
newly generated events and not yet "read" events) this probably works
well. As there is no point in time a process can determine when there
are no further incoming hotplug events, so it knows it is safe to start
reconstruction / replay, there may still be race conditions, not sure.

My solution may be easier to implement / use when regarding possible
race conditions but it possibly has drawbacks, too, I don't know.

Thanks for your reply

Jürg




-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Harald Hoyer]

Kay Sievers wrote:
>>              * [3] Patch to add udevinitd and udevinitsend. These are
>>                modified versions of udevd and udevsend acting as a
>>                "caching daemon". They save events received during early
>>                userspace and resend them to late userspace udevd when
>>                ready. This enables late userspace to process all events
>>                (for example to load modules not included in initramfs)
>>                without using coldplugging and thelike. (udevinitsend
>>                gets called via hotplug.d)
> 
> 
> Nice idea. In a recent discussion, the question came up about a possible
> combination of udevstart and coldplugging. It may be possible to
> synthesize all the events from the information in sysfs? What do you
> think about that in relation to your udevd "event queue".

I really would like to see the hotplay replay since September :)
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id\x133841#c6

Basic problem for Fedora Core:
1. initramfs:
- modules have to be loaded, devices have to be created with udev
- the linuxrc does not know, when the devices are created, so it just calls
   udevstart several times after module loading (workaround)
- hotplug events are not processed completly, because "/" is not mounted yet

2. SysVinit:
- important hotplug events are missing, after "/" is mounted rw (e.g. loading
  of scsi modules after the scsi-adapter module was loaded in initramfs)


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Alexander E. Patrakov]

J?rg Billeter wrote:

> On Mit, 2005-01-12 at 10:07 +0500, Alexander E. Patrakov wrote:
>> Kay Sievers wrote:
>> 
>> >>               * [3] Patch to add udevinitd and udevinitsend. These are
>> >>                 modified versions of udevd and udevsend acting as a
>> >>                 "caching daemon". They save events received during
>> >>                 early userspace and resend them to late userspace
>> >>                 udevd when ready. This enables late userspace to
>> >>                 process all events (for example to load modules not
>> >>                 included in initramfs) without using coldplugging and
>> >>                 thelike. (udevinitsend gets called via hotplug.d)
>> > 
>> > Nice idea. In a recent discussion, the question came up about a
>> > possible combination of udevstart and coldplugging. It may be possible
>> > to synthesize all the events from the information in sysfs? What do you
>> > think about that in relation to your udevd "event queue".
>> 
>> See also my old alternative implementation:
>> 
>> http://bugs.linuxfromscratch.org/show_bug.cgi?id†8
>>
http://archive.linuxfromscratch.org/mail-archives/lfs-hackers/2004-July/001729.html
> 
> I've looked at it at that time, too, but there seem to exist some
> drawbacks:
> 
> - It doesn't support loading modules in initramfs and it seems that this
>   couldn't be added easily as udevd holds back some events until the
>   child process of previous events finished which never happens with
>   the sleeping hotplug processes.

It's still possible to extend my initramfs with a script that preloads a
predefined list of modules (like "modules" script in LFS 6.0). But it seems
that you are talking about something different - namely, loading modules
corresponding to devices found, like the original Hotplug package does.
This also seems possible to implement - just call the equivalent of the
original hotplug script before sleeping.

> - It doesn't work with run-init which moves the /root mount to /. With
>   your initramfs, the whole system will be run in chroot in a way that
>   a root user can escape it and get back to the initramfs tmpfs which
>   could be not that desirable.

Known problem. However, with run_init one cannot have guaranteed delivery of
hotplug events to the late userspace, just because a timeslice exists when
everything in initramfs has been erased, but move-mount hasn't yet
happened.

The situation might (I say "might", because I haven't investigated this
seriously) be better if run-init, instead of doing what it does now (erase
everything and then mount --bind and run init), forked itself. The parent
should create a new namespace (in order for /root/dev/null to remain a
valid path for already-spawned sleeping-hotplug processes), mount --bind,
signal the child, and execute the real init. The child, upon reception of
the signal, should do the cleanup.

Of course the waiting wrappers should also be modified to change the
namespace and redo the mount --bind, and also they should use a version of
the shell that doesn't rely upon "[" being a separate program (busybox has
such option; maybe it's even better to rewrite them in C).

I will investigate this further (and maybe come up with a revised initramfs
image) if you tell me to do so.

> - Some 100 processes test for /root/dev/null every second and will
>   be sent in possibly a random order to udevd at once (probably not
>   a major performance problem but we should consider it)

You won't believe me, but it _is_ a major performance problem if one uses a
dynamic translator like qemu in order to test this initramfs safely (exec
is very expensive since it requires the whole round of code translation).

As to your question concerning the event numbering that starts at 5 in your
case: you do miss some events. They start at 0. Probably this old kernel
patch is relevant: http://lkml.org/lkml/diff/2004/7/13/74/1 (I know that
there was a second patch shortly thereafter, that limits concurrency of
usermode helpers, but I can't find it now).

-- 
Alexander E. Patrakov
(don't CC: me, I read linux-hotplug-devel via gmane)



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Kay Sievers]

On Wed, 2005-01-12 at 09:24 +0100, Jürg Billeter wrote: 
> On Mit, 2005-01-12 at 03:37 +0100, Kay Sievers wrote:
> > On Sun, 2005-01-09 at 23:25 +0100, Juerg Billeter wrote:
> > >               * [3] Patch to add udevinitd and udevinitsend. These are
> > >                 modified versions of udevd and udevsend acting as a
> > >                 "caching daemon". They save events received during early
> > >                 userspace and resend them to late userspace udevd when
> > >                 ready. This enables late userspace to process all events
> > >                 (for example to load modules not included in initramfs)
> > >                 without using coldplugging and thelike. (udevinitsend
> > >                 gets called via hotplug.d)
> > 
> > Nice idea. In a recent discussion, the question came up about a possible
> > combination of udevstart and coldplugging. It may be possible to
> > synthesize all the events from the information in sysfs? What do you
> > think about that in relation to your udevd "event queue".
> 
> If really all events could be synthesized including the order/seqnum and
> one sets udevd's event queue on hold (to avoid race conditions between
> newly generated events and not yet "read" events) this probably works
> well. As there is no point in time a process can determine when there
> are no further incoming hotplug events, so it knows it is safe to start
> reconstruction / replay, there may still be race conditions, not sure.
> 
> My solution may be easier to implement / use when regarding possible
> race conditions but it possibly has drawbacks, too, I don't know.

On Wed, 2005-01-12 at 10:42 +0100, Harald Hoyer wrote:
I really would like to see the hotplay replay since September :) 
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id\x133841#c6

Ok, sounds reasonable to reply the events as a replacement to faking the
hotplug events for coldplugging and running udevstart.

How does the transition from udevinitd to the real udevd look like on
your box, Jürg? How are "real" events handled, traveling in during the
transition?
Maybe udevd should be started from initramfs and never be replaced by a
different one? We may just add a knob to the "real" udevd to hold the
event execution back until userspace asks to flush the queue and udevd
switches over to the current behavior.

Thanks,
Kay




-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Alexander E. Patrakov]

Kay Sievers wrote:

> Maybe udevd should be started from initramfs and never be replaced by a
> different one? We may just add a knob to the "real" udevd to hold the
> event execution back until userspace asks to flush the queue and udevd
> switches over to the current behavior.

At the first glance, that's a good idea. One drawback: one has to write the
plain-text INSTALL file and also think of the users who don't want udev and
prefer static /dev instead.

-- 
Alexander E. Patrakov



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Kay Sievers]

On Thu, 2005-01-13 at 08:50 +0500, Alexander E. Patrakov wrote:
> Kay Sievers wrote:
> 
> > Maybe udevd should be started from initramfs and never be replaced by a
> > different one? We may just add a knob to the "real" udevd to hold the
> > event execution back until userspace asks to flush the queue and udevd
> > switches over to the current behavior.
> 
> At the first glance, that's a good idea. 

Yes, sounds promising. Let's see what Jürg thinks, it's is very close to
his setup, maybe he can try it out...

> One drawback: one has to write the plain-text INSTALL file

This is not too hard, I think. :)

> and also think of the users who don't want udev and
> prefer static /dev instead.

Sure, but it doesn't make a lot of sense to think about what we can do
_with_ udev for people who don't want udev, right? :)

Thanks,
Kay



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Juerg Billeter]

Kay Sievers wrote:
>  How does the transition from udevinitd to the real udevd look like on
>  your box, Jürg? How are "real" events handled, traveling in during the
>  transition?

/proc/sys/kernel/hotplug gets set to /sbin/udevinitsend before entering
"real" userspace. This binary exists in both, initramfs and userspace
and acts almost like udevsend but sends the messages to udevinitd. So
messages sent during initializing real userspace will be sent directly
to udevinitd.
As soon as real userspace is ready, it kills the initramfs' udevd,
sets /proc/sys/kernel/hotplug to /sbin/udevsend and sends USR1 signal to
udevinitd which flushes the queue by sending the messages to the "real"
udevd.

>  Maybe udevd should be started from initramfs and never be replaced by a
>  different one? We may just add a knob to the "real" udevd to hold the
>  event execution back until userspace asks to flush the queue and udevd
>  switches over to the current behavior.

Yes, this looks sane but you need to keep in mind that the early hotplug
events need to be handled twice, once in early userspace (to load
modules required for mounting rootfs) and once in real userspace. So
udevd can't just hold the event execution back, it has to execute and
keep a separate queue.

A problem coming to my mind right now is that udevd doesn't know
anything about the overmount of the rootfs and AFAIK it can't access the
new/real rootfs without chroot/chdir at the right moment (between
mounting the new rootfs to /root and before mount move /root to /) or am
I mistaken here?

BTW: It should probably be possible to have different versions of udev
in the initramfs archive in the kernel and in userspace, at the moment I
suppose this wouldn't work with my setup as udevd seems to check the
version number in the messages.

Jürg



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Kay Sievers]

On Thu, 2005-01-13 at 15:34 +0100, Juerg Billeter wrote:
> Kay Sievers wrote:
> >  How does the transition from udevinitd to the real udevd look like on
> >  your box, Jürg? How are "real" events handled, traveling in during the
> >  transition?
> 
> /proc/sys/kernel/hotplug gets set to /sbin/udevinitsend before entering
> "real" userspace. This binary exists in both, initramfs and userspace
> and acts almost like udevsend but sends the messages to udevinitd. So
> messages sent during initializing real userspace will be sent directly
> to udevinitd.
> As soon as real userspace is ready, it kills the initramfs' udevd,
> sets /proc/sys/kernel/hotplug to /sbin/udevsend and sends USR1 signal to
> udevinitd which flushes the queue by sending the messages to the "real"
> udevd.

Ok, but how do we handle possible unfinished processes that want to send
to the initudevd socket while we are killing the initudevd. I'm pretty
sure that people find a way to create s setup with hundreds of processes
still waiting. :)

How do we handle the events traveling in before we flush the queue to
the real udevd? Timeout?

> >  Maybe udevd should be started from initramfs and never be replaced by a
> >  different one? We may just add a knob to the "real" udevd to hold the
> >  event execution back until userspace asks to flush the queue and udevd
> >  switches over to the current behavior.
> 
> Yes, this looks sane but you need to keep in mind that the early hotplug
> events need to be handled twice, once in early userspace (to load
> modules required for mounting rootfs) and once in real userspace. So
> udevd can't just hold the event execution back, it has to execute and
> keep a separate queue.

I think initramfs should use the hotplug multiplexer and call the normal
udev binary directly to create the nodes along with initudevd to queue
the events.

> A problem coming to my mind right now is that udevd doesn't know
> anything about the overmount of the rootfs and AFAIK it can't access the
> new/real rootfs without chroot/chdir at the right moment (between
> mounting the new rootfs to /root and before mount move /root to /) or am
> I mistaken here?

Don't know. :) If needed we can do this along with the transition from
"queue only" to "normal" behavior, right?

> BTW: It should probably be possible to have different versions of udev
> in the initramfs archive in the kernel and in userspace, at the moment I
> suppose this wouldn't work with my setup as udevd seems to check the
> version number in the messages.

Yes, we once had a binary struct with the values which made the magic
necessary. Now we have only a serialized environment. If we end with two
versions of udevd we may just think about removing the magic check.

Thanks,
Kay




-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Juerg Billeter]

On Don, 2005-01-13 at 16:21 +0100, Kay Sievers wrote:
> On Thu, 2005-01-13 at 15:34 +0100, Juerg Billeter wrote:
> > Kay Sievers wrote:
> > >  How does the transition from udevinitd to the real udevd look like on
> > >  your box, Jürg? How are "real" events handled, traveling in during the
> > >  transition?
> > 
> > /proc/sys/kernel/hotplug gets set to /sbin/udevinitsend before entering
> > "real" userspace. This binary exists in both, initramfs and userspace
> > and acts almost like udevsend but sends the messages to udevinitd. So
> > messages sent during initializing real userspace will be sent directly
> > to udevinitd.
> > As soon as real userspace is ready, it kills the initramfs' udevd,
> > sets /proc/sys/kernel/hotplug to /sbin/udevsend and sends USR1 signal to
> > udevinitd which flushes the queue by sending the messages to the "real"
> > udevd.
> 
> Ok, but how do we handle possible unfinished processes that want to send
> to the initudevd socket while we are killing the initudevd. I'm pretty
> sure that people find a way to create s setup with hundreds of processes
> still waiting. :)
> 
> How do we handle the events traveling in before we flush the queue to
> the real udevd? Timeout?

If we set /proc/sys/kernel/hotplug to the real udevsend before sending
the signal to udevinitd, udevd should be able to correctly reorder the
messages coming in from the two sources (udevsend and udevinitd). To
assure not losing any messages from udevinitsend it should suffice to
let udevinitd forward all messages as long as select() indicates there
is data waiting.

> > >  Maybe udevd should be started from initramfs and never be replaced by a
> > >  different one? We may just add a knob to the "real" udevd to hold the
> > >  event execution back until userspace asks to flush the queue and udevd
> > >  switches over to the current behavior.
> > 
> > Yes, this looks sane but you need to keep in mind that the early hotplug
> > events need to be handled twice, once in early userspace (to load
> > modules required for mounting rootfs) and once in real userspace. So
> > udevd can't just hold the event execution back, it has to execute and
> > keep a separate queue.
> 
> I think initramfs should use the hotplug multiplexer and call the normal
> udev binary directly to create the nodes along with initudevd to queue
> the events.

Would be a possibility but I don't see any advantages over using udevd
in initramfs, too - and it may create problems/races due to not
reordering the incoming messages.

> > A problem coming to my mind right now is that udevd doesn't know
> > anything about the overmount of the rootfs and AFAIK it can't access the
> > new/real rootfs without chroot/chdir at the right moment (between
> > mounting the new rootfs to /root and before mount move /root to /) or am
> > I mistaken here?
> 
> Don't know. :) If needed we can do this along with the transition from
> "queue only" to "normal" behavior, right?

AFAIK it is already too late then. It has to be done before mount
move /root to /, but the "normal" behavior will be enabled after
sysvinit or whatever has started.

Jürg



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Hannes Reinecke]

Kay Sievers wrote:
> On Thu, 2005-01-13 at 15:34 +0100, Juerg Billeter wrote:
> 
>>Kay Sievers wrote:
>>
>>> How does the transition from udevinitd to the real udevd look like on
>>> your box, Jürg? How are "real" events handled, traveling in during the
>>> transition?
>>
>>/proc/sys/kernel/hotplug gets set to /sbin/udevinitsend before entering
>>"real" userspace. This binary exists in both, initramfs and userspace
>>and acts almost like udevsend but sends the messages to udevinitd. So
>>messages sent during initializing real userspace will be sent directly
>>to udevinitd.
>>As soon as real userspace is ready, it kills the initramfs' udevd,
>>sets /proc/sys/kernel/hotplug to /sbin/udevsend and sends USR1 signal to
>>udevinitd which flushes the queue by sending the messages to the "real"
>>udevd.
> 
> 
> Ok, but how do we handle possible unfinished processes that want to send
> to the initudevd socket while we are killing the initudevd. I'm pretty
> sure that people find a way to create s setup with hundreds of processes
> still waiting. :)
> 
Oh, for sure.

> How do we handle the events traveling in before we flush the queue to
> the real udevd? Timeout?
> 
No, we should do it with a staged approach:

- Fire up udevd
- send USR1 to udevinitd
- udevinitd sends magic "Hey, this is udevsend talking to you" message
   to udevd
- udevd stops executing events in response to that magic message
- set /proc/sys/kernel/hotplug to udevsend
- udevinitd sends events to udevd
- any events sent by the 'real' udevsend will be queued
- once udevinitd has sent all events to udevd, it sends a magic
   "I'm done with" message to udevd
- udevd starts processing events.

This way, we won't lose events plus all events will be executed in-order.

> 
>>> Maybe udevd should be started from initramfs and never be replaced by a
>>> different one? We may just add a knob to the "real" udevd to hold the
>>> event execution back until userspace asks to flush the queue and udevd
>>> switches over to the current behavior.
>>
>>Yes, this looks sane but you need to keep in mind that the early hotplug
>>events need to be handled twice, once in early userspace (to load
>>modules required for mounting rootfs) and once in real userspace. So
>>udevd can't just hold the event execution back, it has to execute and
>>keep a separate queue.
> 
> 
> I think initramfs should use the hotplug multiplexer and call the normal
> udev binary directly to create the nodes along with initudevd to queue
> the events.
> 
Ah well. The old problem.
In doing this you have to make sure to copy the _entire_ hotplug 
subsystem plus all called scripts and programs to initramfs.
Plus any configuration file used by any of those.
And you have to make sure that any change in those configuration file is 
being reflected to the initramfs, too.

Still looking for proper ways of solving this. In the meantime, I opted 
for using a minimal hotplug version in initramfs: just set 
/proc/sys/kernel/hotplug to 'udev' to have all device nodes created and 
load all required modules by hand.

> 
>>A problem coming to my mind right now is that udevd doesn't know
>>anything about the overmount of the rootfs and AFAIK it can't access the
>>new/real rootfs without chroot/chdir at the right moment (between
>>mounting the new rootfs to /root and before mount move /root to /) or am
>>I mistaken here?
> 
> 
> Don't know. :) If needed we can do this along with the transition from
> "queue only" to "normal" behavior, right?
> 
That will be quite hard, as udevd in initramfs is started as a child of 
(Hmm, child of what indeed. One should check.). When doing the chroot() 
thing to the real rootfs the new namespace is only available to all 
children of init(), which the udevd in initramfs is certainly not.
So the udevd would have to know where the rootfs is mounted and do a 
chroot() itself.
Can't say I like this, as the mount point of the rootfs is not really 
defined and can be just anything.
So having two distinct processes, one for initramfs and another one for 
the real thing is IMHO a better idea.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke			hare@suse.de
SuSE Linux AG				S390 & zSeries
Maxfeldstraße 5				+49 911 74053 688
90409 Nürnberg				http://www.suse.de


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Kay Sievers]

On Fri, 2005-01-14 at 10:29 +0100, Hannes Reinecke wrote:
> Kay Sievers wrote:
> > How do we handle the events traveling in before we flush the queue to
> > the real udevd? Timeout?
> > 
> No, we should do it with a staged approach:
> 
> - Fire up udevd
> - send USR1 to udevinitd
> - udevinitd sends magic "Hey, this is udevsend talking to you" message
>    to udevd
> - udevd stops executing events in response to that magic message

If udevd already executes events before we get the initrams events
flushed in, we execute them not in the right order. Is this ok? 
Shouldn't we just queue _all_ events until we got the initramfs ones?

> - set /proc/sys/kernel/hotplug to udevsend
> - udevinitd sends events to udevd
> - any events sent by the 'real' udevsend will be queued
> - once udevinitd has sent all events to udevd, it sends a magic
>    "I'm done with" message to udevd
> - udevd starts processing events.
> 
> This way, we won't lose events plus all events will be executed in-order.

Seems possible at least, to collect all events from the very beginning
on and replay them from real userspace without the need for coldplug or
udevstart. Sounds nice! We can even load-limit from udevd here. :)

> > I think initramfs should use the hotplug multiplexer and call the normal
> > udev binary directly to create the nodes along with initudevd to queue
> > the events.
> > 
> Ah well. The old problem.
> In doing this you have to make sure to copy the _entire_ hotplug 
> subsystem plus all called scripts and programs to initramfs.
> Plus any configuration file used by any of those.
> And you have to make sure that any change in those configuration file is 
> being reflected to the initramfs, too.
> 
> Still looking for proper ways of solving this. In the meantime, I opted 
> for using a minimal hotplug version in initramfs: just set 
> /proc/sys/kernel/hotplug to 'udev' to have all device nodes created and 
> load all required modules by hand.

But we need to collect the events here with initudevd now. initudevd can
execute udev just like the real udevd does today, but keep the queue for
a later flush into the real udevd.
 
> >>A problem coming to my mind right now is that udevd doesn't know
> >>anything about the overmount of the rootfs and AFAIK it can't access the
> >>new/real rootfs without chroot/chdir at the right moment (between
> >>mounting the new rootfs to /root and before mount move /root to /) or am
> >>I mistaken here?
> > 
> > Don't know. :) If needed we can do this along with the transition from
> > "queue only" to "normal" behavior, right?
> > 
> That will be quite hard, as udevd in initramfs is started as a child of 
> (Hmm, child of what indeed. One should check.). When doing the chroot() 
> thing to the real rootfs the new namespace is only available to all 
> children of init(), which the udevd in initramfs is certainly not.
> So the udevd would have to know where the rootfs is mounted and do a 
> chroot() itself.
> Can't say I like this, as the mount point of the rootfs is not really 
> defined and can be just anything.
> So having two distinct processes, one for initramfs and another one for 
> the real thing is IMHO a better idea.

Yeah, sounds reasonable to have two version and replace the initudevd
process.

Thanks,
Kay



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Hannes Reinecke]

Kay Sievers wrote:
> On Fri, 2005-01-14 at 10:29 +0100, Hannes Reinecke wrote:
> 
>>Kay Sievers wrote:
>>
>>>How do we handle the events traveling in before we flush the queue to
>>>the real udevd? Timeout?
>>>
>>
>>No, we should do it with a staged approach:
>>
>>- Fire up udevd
>>- send USR1 to udevinitd
>>- udevinitd sends magic "Hey, this is udevsend talking to you" message
>>   to udevd
>>- udevd stops executing events in response to that magic message
> 
> 
> If udevd already executes events before we get the initrams events
> flushed in, we execute them not in the right order. Is this ok? 
> Shouldn't we just queue _all_ events until we got the initramfs ones?
> 
> 

It's a bit tricky as we don't know on startup whether we'll get any 
initramfs events. If the user doesn't use udevinitd (or it fails or 
whatever) we'll be waiting forever.

>>- set /proc/sys/kernel/hotplug to udevsend
>>- udevinitd sends events to udevd
>>- any events sent by the 'real' udevsend will be queued
>>- once udevinitd has sent all events to udevd, it sends a magic
>>   "I'm done with" message to udevd
>>- udevd starts processing events.
>>
>>This way, we won't lose events plus all events will be executed in-order.
> 
> 
> Seems possible at least, to collect all events from the very beginning
> on and replay them from real userspace without the need for coldplug or
> udevstart. Sounds nice! We can even load-limit from udevd here. :)
> 
Well, not quite, as udevsend will still be started from the kernel. But 
yes, load-limiting is a definite should-have.

>>>I think initramfs should use the hotplug multiplexer and call the normal
>>>udev binary directly to create the nodes along with initudevd to queue
>>>the events.
>>>
>>
>>Ah well. The old problem.
>>In doing this you have to make sure to copy the _entire_ hotplug 
>>subsystem plus all called scripts and programs to initramfs.
>>Plus any configuration file used by any of those.
>>And you have to make sure that any change in those configuration file is 
>>being reflected to the initramfs, too.
>>
>>Still looking for proper ways of solving this. In the meantime, I opted 
>>for using a minimal hotplug version in initramfs: just set 
>>/proc/sys/kernel/hotplug to 'udev' to have all device nodes created and 
>>load all required modules by hand.
> 
> 
> But we need to collect the events here with initudevd now. initudevd can
> execute udev just like the real udevd does today, but keep the queue for
> a later flush into the real udevd.
>  
Yeah, sure.
We'll have to do the queueing bit anyway.
(Currently it's done in script-logic, not nice).

Cheers,

Hannes
-- 
Dr. Hannes Reinecke			hare@suse.de
SuSE Linux AG				S390 & zSeries
Maxfeldstraße 5				+49 911 74053 688
90409 Nürnberg				http://www.suse.de


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: initramfs: udev, hotplug, klibc and modprobe

[Kay Sievers]

On Fri, 2005-01-14 at 11:27 +0100, Hannes Reinecke wrote:
> Kay Sievers wrote:
> > On Fri, 2005-01-14 at 10:29 +0100, Hannes Reinecke wrote:
> > 
> >>Kay Sievers wrote:
> >>
> >>>How do we handle the events traveling in before we flush the queue to
> >>>the real udevd? Timeout?
> >>>
> >>
> >>No, we should do it with a staged approach:
> >>
> >>- Fire up udevd
> >>- send USR1 to udevinitd
> >>- udevinitd sends magic "Hey, this is udevsend talking to you" message
> >>   to udevd
> >>- udevd stops executing events in response to that magic message
> > 
> > 
> > If udevd already executes events before we get the initrams events
> > flushed in, we execute them not in the right order. Is this ok? 
> > Shouldn't we just queue _all_ events until we got the initramfs ones?
> > 
> It's a bit tricky as we don't know on startup whether we'll get any 
> initramfs events. If the user doesn't use udevinitd (or it fails or 
> whatever) we'll be waiting forever.

Can't we read the state from /proc/sys/kernel/hotplug which is still set
to something different as /sbin/hotplug?

> >>- set /proc/sys/kernel/hotplug to udevsend
> >>- udevinitd sends events to udevd
> >>- any events sent by the 'real' udevsend will be queued
> >>- once udevinitd has sent all events to udevd, it sends a magic
> >>   "I'm done with" message to udevd
> >>- udevd starts processing events.
> >>
> >>This way, we won't lose events plus all events will be executed in-order.
> > 
> > 
> > Seems possible at least, to collect all events from the very beginning
> > on and replay them from real userspace without the need for coldplug or
> > udevstart. Sounds nice! We can even load-limit from udevd here. :)
> > 
> Well, not quite, as udevsend will still be started from the kernel. But 
> yes, load-limiting is a definite should-have.

This will not really harm, I expect. udevsend exits very fast if it can
send the event and if it can't send it is sleeping and not running. This
is unlikely to cause problems. The hundreds of shells in "R" are the
problem, right?

Thanks,
Kay



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel