[Buildroot] [PATCH] linux: don't automatically set uevent_helper with mdev /dev management

[Buildroot] [PATCH] linux: don't automatically set uevent_helper with mdev /dev management

[Thomas De Schampheleire]

When mdev /dev management is chosen in the buildroot configuration, the
Linux configuration is updated automatically to set option
CONFIG_UEVENT_HELPER_PATH to "/sbin/mdev". However, the help text of this
option explicitly recommends not setting this option due to large
performance impact during boot (experienced first hand by the reporter ?nd
author).

The mdev startup script S10mdev already sets the helper during userspace
boot, which will make sure mdev is working correctly.

Fixes bug #6596: https://bugs.busybox.net/show_bug.cgi?id=6596

Reported-by: Andreas Koop <andreas.koop@zf.com>
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
 linux/linux.mk |  2 --
 1 files changed, 0 insertions(+), 2 deletions(-)

diff --git a/linux/linux.mk b/linux/linux.mk
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -186,8 +186,6 @@ define LINUX_CONFIGURE_CMDS
 	$(if $(BR2_ROOTFS_DEVICE_CREATION_STATIC),,
 		$(call KCONFIG_ENABLE_OPT,CONFIG_DEVTMPFS,$(@D)/.config)
 		$(call KCONFIG_ENABLE_OPT,CONFIG_DEVTMPFS_MOUNT,$(@D)/.config))
-	$(if $(BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_MDEV),
-		$(call KCONFIG_SET_OPT,CONFIG_UEVENT_HELPER_PATH,\"/sbin/mdev\",$(@D)/.config))
 	$(if $(BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_UDEV),
 		$(call KCONFIG_ENABLE_OPT,CONFIG_INOTIFY_USER,$(@D)/.config))
 	$(if $(BR2_PACKAGE_KTAP),

[Buildroot] [PATCH] linux: don't automatically set uevent_helper with mdev /dev management

[Yann E. MORIN]

Thomas, All,

On 2014-02-07 09:20 +0100, Thomas De Schampheleire spake thusly:
> When mdev /dev management is chosen in the buildroot configuration, the
> Linux configuration is updated automatically to set option
> CONFIG_UEVENT_HELPER_PATH to "/sbin/mdev". However, the help text of this
> option explicitly recommends not setting this option due to large
> performance impact during boot (experienced first hand by the reporter ?nd
> author).
> 
> The mdev startup script S10mdev already sets the helper during userspace
> boot, which will make sure mdev is working correctly.
> 
> Fixes bug #6596: https://bugs.busybox.net/show_bug.cgi?id=6596
> 
> Reported-by: Andreas Koop <andreas.koop@zf.com>
> Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] linux: don't automatically set uevent_helper with mdev /dev management

[Peter Korsgaard]

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:

 > When mdev /dev management is chosen in the buildroot configuration, the
 > Linux configuration is updated automatically to set option
 > CONFIG_UEVENT_HELPER_PATH to "/sbin/mdev". However, the help text of this
 > option explicitly recommends not setting this option due to large
 > performance impact during boot (experienced first hand by the reporter ?nd
 > author).

 > The mdev startup script S10mdev already sets the helper during userspace
 > boot, which will make sure mdev is working correctly.

 > Fixes bug #6596: https://bugs.busybox.net/show_bug.cgi?id=6596

 > Reported-by: Andreas Koop <andreas.koop@zf.com>
 > Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

Hmm, but the the coldplug handling with 'mdev -s' afaik doesn't handle
module or firmware loading, so those features will break from this.

I would be interested in knowing why hotplug is so much slower than
coldplug. Do you have any idea about how many times mdev gets executed
during bootup?

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] linux: don't automatically set uevent_helper with mdev /dev management

[Thomas De Schampheleire]

Peter Korsgaard <jacmet@uclibc.org> schreef:
>>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:
>
> > When mdev /dev management is chosen in the buildroot configuration, the
> > Linux configuration is updated automatically to set option
> > CONFIG_UEVENT_HELPER_PATH to "/sbin/mdev". However, the help text of this
> > option explicitly recommends not setting this option due to large
> > performance impact during boot (experienced first hand by the reporter ?nd
> > author).
>
> > The mdev startup script S10mdev already sets the helper during userspace
> > boot, which will make sure mdev is working correctly.
>
> > Fixes bug #6596: https://bugs.busybox.net/show_bug.cgi?id=6596
>
> > Reported-by: Andreas Koop <andreas.koop@zf.com>
> > Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
>
>Hmm, but the the coldplug handling with 'mdev -s' afaik doesn't handle
>module or firmware loading, so those features will break from this.

Isn't this handled by the installation of the hotplug helper from
S10mdev, after which mdev -s is run?
I.e. everything that happened before S10mdev is caught by 'mdev -s'
and everything after is handled by the hotplug helper, right?

>
>I would be interested in knowing why hotplug is so much slower than
>coldplug. Do you have any idea about how many times mdev gets executed
>during bootup?
>

No, I don't know. I do know that on one particular arm-based single
core system, the time to enter userspace increased from 2 seconds to
10 seconds due to this hotplug helper.

Best regards,
Thomas

[Buildroot] [PATCH] linux: don't automatically set uevent_helper with mdev /dev management

[Peter Korsgaard]

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:

Hi,

 >> Hmm, but the the coldplug handling with 'mdev -s' afaik doesn't handle
 >> module or firmware loading, so those features will break from this.

 > Isn't this handled by the installation of the hotplug helper from
 > S10mdev, after which mdev -s is run?

Yes, if the events happen after that line, but not if they happen
before.

 > I.e. everything that happened before S10mdev is caught by 'mdev -s'
 > and everything after is handled by the hotplug helper, right?

No, mdev -s only scans /sys for devices and create the corresponding
device nodes (already done by devtmpfs) and execute whatever commands
you have defined, it afaik doesn't handle module load or firmware requests.


 >> I would be interested in knowing why hotplug is so much slower than
 >> coldplug. Do you have any idea about how many times mdev gets executed
 >> during bootup?
 >> 

 > No, I don't know. I do know that on one particular arm-based single
 > core system, the time to enter userspace increased from 2 seconds to
 > 10 seconds due to this hotplug helper.

Funky. I personally have never used mdev, but that sounds pretty
extreme.


-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] linux: don't automatically set uevent_helper with mdev /dev management

[Thomas De Schampheleire]

Hi Peter,

On Fri, Feb 7, 2014 at 10:05 PM, Peter Korsgaard <jacmet@uclibc.org> wrote:
>>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:
>
> Hi,
>
>  >> Hmm, but the the coldplug handling with 'mdev -s' afaik doesn't handle
>  >> module or firmware loading, so those features will break from this.
>
>  > Isn't this handled by the installation of the hotplug helper from
>  > S10mdev, after which mdev -s is run?
>
> Yes, if the events happen after that line, but not if they happen
> before.
>
>  > I.e. everything that happened before S10mdev is caught by 'mdev -s'
>  > and everything after is handled by the hotplug helper, right?
>
> No, mdev -s only scans /sys for devices and create the corresponding
> device nodes (already done by devtmpfs) and execute whatever commands
> you have defined, it afaik doesn't handle module load or firmware requests.

When you say 'module load', the only relevant consequence of that is
the creation of nodes in /sys that may need corresponding entries in
/dev right? So this is covered by the S10mdev script, as far as I see.
What else would happen in 'module load' that mdev does not cover?

I'm not familiar enough with firmware requests to talk about that.

However, the kernel configuration says: "This should not be used
today, because usual systems create many events at bootup or device
discovery in a very short time frame. One forked process per event can
create so many processes that it creates a high system load, or on
smaller systems it is known to create out-of-memory situations during
bootup."

So then I wonder, how is it supposed to work for module load and
firmware loading if you can't use this option. Or put another way: are
you sure we need to set the uevent_helper_path for that?

>
>
>  >> I would be interested in knowing why hotplug is so much slower than
>  >> coldplug. Do you have any idea about how many times mdev gets executed
>  >> during bootup?
>  >>
>
>  > No, I don't know. I do know that on one particular arm-based single
>  > core system, the time to enter userspace increased from 2 seconds to
>  > 10 seconds due to this hotplug helper.
>
> Funky. I personally have never used mdev, but that sounds pretty
> extreme.

It's a large increase yes, but it's what we measured.
Another system (4-core powerpc) had about 1-2 seconds increase.

Best regards,
Thomas

[Buildroot] [PATCH] linux: don't automatically set uevent_helper with mdev /dev management

[Peter Korsgaard]

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:

Hi,

 >> No, mdev -s only scans /sys for devices and create the corresponding
 >> device nodes (already done by devtmpfs) and execute whatever commands
 >> you have defined, it afaik doesn't handle module load or firmware requests.

 > When you say 'module load', the only relevant consequence of that is
 > the creation of nodes in /sys that may need corresponding entries in
 > /dev right? So this is covered by the S10mdev script, as far as I see.
 > What else would happen in 'module load' that mdev does not cover?

I mean module load as in automatic loading of kernel modules
modprobe. The kernel generates hotplug events with the needed modalias
that mdev can pass to modprobe, but our mdev.conf is not setup to do that.

Here's a blog post about it:

https://frankpzh.wordpress.com/2011/04/16/kmod-udev-and-modprobe/

 > However, the kernel configuration says: "This should not be used
 > today, because usual systems create many events at bootup or device
 > discovery in a very short time frame. One forked process per event can
 > create so many processes that it creates a high system load, or on
 > smaller systems it is known to create out-of-memory situations during
 > bootup."

Yes, the "modern" way to send these events (E.G. what udev is using) is
afaik through netlink instead of forking a process for each, but if you
want to handle these events through mdev then there's not really any way
around it.


 > So then I wonder, how is it supposed to work for module load and
 > firmware loading if you can't use this option. Or put another way: are
 > you sure we need to set the uevent_helper_path for that?

If you want to be notified of hotplug events (and don't use netlink),
then you have to set it.

Out of interest I did a quick test with qemu_arm_versatile_defconfig and
the following patch:

diff -u package/busybox/S10mdev output/target/etc/init.d/S10mdev
--- package/busybox/S10mdev     2013-10-24 09:55:30.270408593 +0200
+++ output/target/etc/init.d/S10mdev    2014-02-08 21:26:48.450713447 +0100
@@ -7,6 +7,8 @@
   start)
        echo "Starting mdev..."
        echo /sbin/mdev >/proc/sys/kernel/hotplug
+       echo -n "seqnum: "
+       cat /sys/kernel/uevent_seqnum
        /sbin/mdev -s
        ;;
   stop)

And I see the kernel triggers 357 hotplug events before we get
here. Looking at the pid number after bootup and login that also seems
to match (428 with CONFIG_UEVENT_HELPER_PATH="/sbin/mdev" and 73 when
set to the empty string), but I don't see any significant boot speed
difference here.

So, I'm not saying I don't believe you or that it perhaps wouldn't be
better to handle the initial hotplug events with mdev, just that it less
"correct" - But mdev is afaik inherently racy, so perhaps not doing it
would be the "least bad" approach (and gives us same behaviour if kernel
isn't built by buildroot).

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] linux: don't automatically set uevent_helper with mdev /dev management

[Thomas De Schampheleire]

Hi Peter,

On Sat, Feb 8, 2014 at 9:51 PM, Peter Korsgaard <jacmet@uclibc.org> wrote:
>>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:
>
> Hi,
>
>  >> No, mdev -s only scans /sys for devices and create the corresponding
>  >> device nodes (already done by devtmpfs) and execute whatever commands
>  >> you have defined, it afaik doesn't handle module load or firmware requests.
>
>  > When you say 'module load', the only relevant consequence of that is
>  > the creation of nodes in /sys that may need corresponding entries in
>  > /dev right? So this is covered by the S10mdev script, as far as I see.
>  > What else would happen in 'module load' that mdev does not cover?
>
> I mean module load as in automatic loading of kernel modules
> modprobe. The kernel generates hotplug events with the needed modalias
> that mdev can pass to modprobe, but our mdev.conf is not setup to do that.
>
> Here's a blog post about it:
>
> https://frankpzh.wordpress.com/2011/04/16/kmod-udev-and-modprobe/
>
>  > However, the kernel configuration says: "This should not be used
>  > today, because usual systems create many events at bootup or device
>  > discovery in a very short time frame. One forked process per event can
>  > create so many processes that it creates a high system load, or on
>  > smaller systems it is known to create out-of-memory situations during
>  > bootup."
>
> Yes, the "modern" way to send these events (E.G. what udev is using) is
> afaik through netlink instead of forking a process for each, but if you
> want to handle these events through mdev then there's not really any way
> around it.
>
>
>  > So then I wonder, how is it supposed to work for module load and
>  > firmware loading if you can't use this option. Or put another way: are
>  > you sure we need to set the uevent_helper_path for that?
>
> If you want to be notified of hotplug events (and don't use netlink),
> then you have to set it.
>
> Out of interest I did a quick test with qemu_arm_versatile_defconfig and
> the following patch:
>
> diff -u package/busybox/S10mdev output/target/etc/init.d/S10mdev
> --- package/busybox/S10mdev     2013-10-24 09:55:30.270408593 +0200
> +++ output/target/etc/init.d/S10mdev    2014-02-08 21:26:48.450713447 +0100
> @@ -7,6 +7,8 @@
>    start)
>         echo "Starting mdev..."
>         echo /sbin/mdev >/proc/sys/kernel/hotplug
> +       echo -n "seqnum: "
> +       cat /sys/kernel/uevent_seqnum
>         /sbin/mdev -s
>         ;;
>    stop)
>
> And I see the kernel triggers 357 hotplug events before we get
> here. Looking at the pid number after bootup and login that also seems
> to match (428 with CONFIG_UEVENT_HELPER_PATH="/sbin/mdev" and 73 when
> set to the empty string), but I don't see any significant boot speed
> difference here.
>
> So, I'm not saying I don't believe you or that it perhaps wouldn't be
> better to handle the initial hotplug events with mdev, just that it less
> "correct" - But mdev is afaik inherently racy, so perhaps not doing it
> would be the "least bad" approach (and gives us same behaviour if kernel
> isn't built by buildroot).
>

Thanks for the clarifications and test.
An alternative is to provide an option in buildroot to select the
behavior: either the current behavior (install mdev as hotplug helper
by setting the kernel config) or the proposed behavior in this patch
by removing that.
Then every project can make its own tradeoff...

What do you think of that?

Thanks,
Thomas

[Buildroot] [PATCH] linux: don't automatically set uevent_helper with mdev /dev management

[Peter Korsgaard]

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:

Hi,

 > Thanks for the clarifications and test.
 > An alternative is to provide an option in buildroot to select the
 > behavior: either the current behavior (install mdev as hotplug helper
 > by setting the kernel config) or the proposed behavior in this patch
 > by removing that.
 > Then every project can make its own tradeoff...

 > What do you think of that?

I don't like it. We had several people report the issue, so I think the
best way forward is to apply your patch. If people want the extra
features of mdev-from-bootup they can always tweak their kernel config
to set it.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] linux: don't automatically set uevent_helper with mdev /dev management

[Peter Korsgaard]

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:

 > When mdev /dev management is chosen in the buildroot configuration, the
 > Linux configuration is updated automatically to set option
 > CONFIG_UEVENT_HELPER_PATH to "/sbin/mdev". However, the help text of this
 > option explicitly recommends not setting this option due to large
 > performance impact during boot (experienced first hand by the reporter ?nd
 > author).

 > The mdev startup script S10mdev already sets the helper during userspace
 > boot, which will make sure mdev is working correctly.

 > Fixes bug #6596: https://bugs.busybox.net/show_bug.cgi?id=6596

 > Reported-by: Andreas Koop <andreas.koop@zf.com>
 > Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

Committed, thanks.

-- 
Bye, Peter Korsgaard