Making udev emit a signal when it is done loading modules

Making udev emit a signal when it is done loading modules

[Hans de Goede]

Dear udev developers,

We (me and some kernel devs mostly) would like to add support to
the kernel for userspace telling the kernel that it is done with
the *initial* loading of modules, with the purpose of cleaning up
(disabling) unused harware resources like e.g. regulators and
clocks.

Currently the kernel does this cleanup just before it starts init
(which may very well be init from a ramdisk). In some cases this
is too early really, because later on a module may get loaded
which needs this resources, these resources will then get turned
on again by the loaded driver, and most of the time this is not
an issue, but sometimes it is.

I realize very well that there is no magic moment where udev is
really ever done loading modules, but the case which we want to
support only involves devices which are *already enumerated*, but
may not yet have a driver loaded, when udev starts. We would like
udev to emit a signal (ABI to be discussed) when it is done
trying to load modules for everything which was already enumerated
when it starts, iow when there are no new device events pending
anymore when udev does its initial hotplug replay.

So the question to you is would you be willing to include such
functionality in udev ?  Note this signal would need to be emitted
when udev from the real rootfs is done with the initial module
loading, as the real rootfs may very well have more modules
available then the initrd.

###

With the generic story above told let me also give the concrete
example / problem which has let to me asking this (note this has
been brought up before on various kernel lists, it is a
re-occuring theme, this is just an example really) :

The problem at hand is a sata connector which also has a sata-power
connector on an embedded (ish) board where the sata-power is
controlled through a gpio. The sata-power connector is modeled
in devicetree as a power-supply and this supply gets controlled
by the ahci_platform driver.

The disk power may very well have already been turned on by the
bootloader, so we add a regulator-boot-on property to the regulator
node in devicetree to make sure that it is left untouched when the
regulator driver loads. If the ahci_platform driver is build into
the kernel, it will then take control of the regulator and
everything works well.

If however the ahci_platform driver is a module, then as soon as
the kernel is ready to start init, unused regulators are turned off
and the disk looses its power while spinning and ends up doing an
emergency heads park. Then as soon as the ahci_platform module
loads the disk is spin up again. This is not good (quite bad
actually) for the lifetime of the disk.

Regards,

Hans
_______________________________________________
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: Making udev emit a signal when it is done loading modules

[Greg KH]

On Sat, Jan 17, 2015 at 09:44:00AM +0100, Hans de Goede wrote:
> Dear udev developers,
> 
> We (me and some kernel devs mostly) would like to add support to
> the kernel for userspace telling the kernel that it is done with
> the *initial* loading of modules, with the purpose of cleaning up
> (disabling) unused harware resources like e.g. regulators and
> clocks.

But you don't "know" when that happens.  Especially with discoverable
busses (PCI, USB, etc.), you know this :)

> Currently the kernel does this cleanup just before it starts init
> (which may very well be init from a ramdisk). In some cases this
> is too early really, because later on a module may get loaded
> which needs this resources, these resources will then get turned
> on again by the loaded driver, and most of the time this is not
> an issue, but sometimes it is.
> 
> I realize very well that there is no magic moment where udev is
> really ever done loading modules, but the case which we want to
> support only involves devices which are *already enumerated*, but
> may not yet have a driver loaded, when udev starts. We would like
> udev to emit a signal (ABI to be discussed) when it is done
> trying to load modules for everything which was already enumerated
> when it starts, iow when there are no new device events pending
> anymore when udev does its initial hotplug replay.

The kernel doesn't even "know" when this type of thing is, how can udev
know this?

> So the question to you is would you be willing to include such
> functionality in udev ?  Note this signal would need to be emitted
> when udev from the real rootfs is done with the initial module
> loading, as the real rootfs may very well have more modules
> available then the initrd.
> 
> ###
> 
> With the generic story above told let me also give the concrete
> example / problem which has let to me asking this (note this has
> been brought up before on various kernel lists, it is a
> re-occuring theme, this is just an example really) :
> 
> The problem at hand is a sata connector which also has a sata-power
> connector on an embedded (ish) board where the sata-power is
> controlled through a gpio. The sata-power connector is modeled
> in devicetree as a power-supply and this supply gets controlled
> by the ahci_platform driver.
> 
> The disk power may very well have already been turned on by the
> bootloader, so we add a regulator-boot-on property to the regulator
> node in devicetree to make sure that it is left untouched when the
> regulator driver loads. If the ahci_platform driver is build into
> the kernel, it will then take control of the regulator and
> everything works well.
> 
> If however the ahci_platform driver is a module, then as soon as
> the kernel is ready to start init, unused regulators are turned off
> and the disk looses its power while spinning and ends up doing an
> emergency heads park.

What turns off the power in this situation?  The kernel?  Or userspace?
Don't you have control of this?

Have you tried to even create a patch that could do this type of thing
to udev to see if it is even possible?

thanks,

greg k-h
_______________________________________________
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Making udev emit a signal when it is done loading modules

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 993 bytes --]

On Sat, Jan 17, 2015 at 05:56:30AM -0800, Greg KH wrote:
> On Sat, Jan 17, 2015 at 09:44:00AM +0100, Hans de Goede wrote:

> > If however the ahci_platform driver is a module, then as soon as
> > the kernel is ready to start init, unused regulators are turned off
> > and the disk looses its power while spinning and ends up doing an
> > emergency heads park.

> What turns off the power in this situation?  The kernel?  Or userspace?
> Don't you have control of this?

The kernel turns off the power in late_initcall because it noticed that
nothing is using the regulator, it does this because otherwise we're
just left with whatever the bootloader had which will may well be
wasting power on unused rails due to inflexible PMICs (we do similar
things for other subsystems like clocks).  This works fine for most use
cases since in embedded systems anything critical tends to end up built
in.

There is configurability to allow things to be flagged as always on so
they never get turned off.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Making udev emit a signal when it is done loading modules

[Hans de Goede]

Hi,

On 17-01-15 14:56, Greg KH wrote:
> On Sat, Jan 17, 2015 at 09:44:00AM +0100, Hans de Goede wrote:
>> Dear udev developers,
>>
>> We (me and some kernel devs mostly) would like to add support to
>> the kernel for userspace telling the kernel that it is done with
>> the *initial* loading of modules, with the purpose of cleaning up
>> (disabling) unused harware resources like e.g. regulators and
>> clocks.
>
> But you don't "know" when that happens.  Especially with discoverable
> busses (PCI, USB, etc.), you know this :)

Right, I'm just bringing this up because it keeps coming back as
a possible solution to hardware resources getting turned off
too early during various kernel discussions, so I thought that as
this is a re-occuring theme, at a minimum someone should discuss
this with the udev people.

IOW I'm merely the manager, and you know what they say about
messengers ...  :)

>
>> Currently the kernel does this cleanup just before it starts init
>> (which may very well be init from a ramdisk). In some cases this
>> is too early really, because later on a module may get loaded
>> which needs this resources, these resources will then get turned
>> on again by the loaded driver, and most of the time this is not
>> an issue, but sometimes it is.
>>
>> I realize very well that there is no magic moment where udev is
>> really ever done loading modules, but the case which we want to
>> support only involves devices which are *already enumerated*, but
>> may not yet have a driver loaded, when udev starts. We would like
>> udev to emit a signal (ABI to be discussed) when it is done
>> trying to load modules for everything which was already enumerated
>> when it starts, iow when there are no new device events pending
>> anymore when udev does its initial hotplug replay.
>
> The kernel doesn't even "know" when this type of thing is, how can udev
> know this?

This is why I clearly limited this to *already enumerated* devices,
udev will know when it has exhausted whatever the kernel has enumerated
before it started, because at one point it will stop getting uevents
when replaying. This does not mean enumeration is really complete in any
way, it just means that modprobe has been run for any *already enumerated*
devices.

Most of these problems are on embedded (ish) systems, and there it is
a good bet that the troublesome device is enumerated already, since it
likely is devicetree instantiated.


>
>> So the question to you is would you be willing to include such
>> functionality in udev ?  Note this signal would need to be emitted
>> when udev from the real rootfs is done with the initial module
>> loading, as the real rootfs may very well have more modules
>> available then the initrd.
>>
>> ###
>>
>> With the generic story above told let me also give the concrete
>> example / problem which has let to me asking this (note this has
>> been brought up before on various kernel lists, it is a
>> re-occuring theme, this is just an example really) :
>>
>> The problem at hand is a sata connector which also has a sata-power
>> connector on an embedded (ish) board where the sata-power is
>> controlled through a gpio. The sata-power connector is modeled
>> in devicetree as a power-supply and this supply gets controlled
>> by the ahci_platform driver.
>>
>> The disk power may very well have already been turned on by the
>> bootloader, so we add a regulator-boot-on property to the regulator
>> node in devicetree to make sure that it is left untouched when the
>> regulator driver loads. If the ahci_platform driver is build into
>> the kernel, it will then take control of the regulator and
>> everything works well.
>>
>> If however the ahci_platform driver is a module, then as soon as
>> the kernel is ready to start init, unused regulators are turned off
>> and the disk looses its power while spinning and ends up doing an
>> emergency heads park.
>
> What turns off the power in this situation?  The kernel?  Or userspace?

The kernel.

> Don't you have control of this?

To some degree, this can be controller by a driver, and a driver can
stop this turning off, but atm this means that the driver MUST be builtin,
that is what we would like to fix.

> Have you tried to even create a patch that could do this type of thing
> to udev to see if it is even possible?

No, because as you indicated at the start of your mail I know this
area pretty well and I know this a somewhat controversial proposal,
so IMHO there is no use in spending time on this without at least
some buy-in of involved people.

Regards,

Hans
_______________________________________________
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Making udev emit a signal when it is done loading modules

[jonsmirl-Re5JQEeQqe8AvxtiuMwx3w]

On Sat, Jan 17, 2015 at 8:56 AM, Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote:
> On Sat, Jan 17, 2015 at 09:44:00AM +0100, Hans de Goede wrote:
>> Dear udev developers,
>>
>> We (me and some kernel devs mostly) would like to add support to
>> the kernel for userspace telling the kernel that it is done with
>> the *initial* loading of modules, with the purpose of cleaning up
>> (disabling) unused harware resources like e.g. regulators and
>> clocks.
>
> But you don't "know" when that happens.  Especially with discoverable
> busses (PCI, USB, etc.), you know this :)

Consider the ARM multi-arch kernels where everything is a loadable
device driver.

1) BIOS does basic system setup
2) Core kernel boots with very few drivers.
3) Late-init calls runs - does power/clock clean up and turns everything off
--everything in system blinks---
4) Now the modules start loading and devices all get turned back on.
These are loadable modules for hardware that is known to be present
since it is listed in the device tree.
-- better place to be doing power/clock clean up
5) Doesn't matter now if you load a module an hour later, you wanted
the clock/power turned off for that hour.

I know of two ways to fix this interval where the system blinks....
1) An enumerated protection scheme. Device tree lists out all of the
clock/regulators/etc... that it doesn't want the power cleanup turning
off. Then it hopes that a driver gets loaded that uses these devices.
If not those clock/regulators stay on forever. This is fragile and
requires coordination with the BIOS.
2) in my opinion power clean is happening too early. In needs to moved
after the initial phase of loading modules for device tree enumerated
devices.

Power/clock clean up does not need to be at an exact point in system
boot. It is just a clean up to reduce power consumption from things
the BIOS turned on that kernel does not use. It is usually not fatal
if it runs at the wrong point, but it makes everything blink when it
runs too early.

For example the automatic clean up could be removed from late_init
calls and turned into something that could be triggered from a script.
Then runs that script action late in the boot. Or if udev is smart
enough it could trigger cleanup after it has loaded modules for all
devices it is aware of. But any way you look at it, cleaning up in
late-init call for a multi-arch kernel is the too early.

>
>> Currently the kernel does this cleanup just before it starts init
>> (which may very well be init from a ramdisk). In some cases this
>> is too early really, because later on a module may get loaded
>> which needs this resources, these resources will then get turned
>> on again by the loaded driver, and most of the time this is not
>> an issue, but sometimes it is.
>>
>> I realize very well that there is no magic moment where udev is
>> really ever done loading modules, but the case which we want to
>> support only involves devices which are *already enumerated*, but
>> may not yet have a driver loaded, when udev starts. We would like
>> udev to emit a signal (ABI to be discussed) when it is done
>> trying to load modules for everything which was already enumerated
>> when it starts, iow when there are no new device events pending
>> anymore when udev does its initial hotplug replay.
>
> The kernel doesn't even "know" when this type of thing is, how can udev
> know this?
>
>> So the question to you is would you be willing to include such
>> functionality in udev ?  Note this signal would need to be emitted
>> when udev from the real rootfs is done with the initial module
>> loading, as the real rootfs may very well have more modules
>> available then the initrd.
>>
>> ###
>>
>> With the generic story above told let me also give the concrete
>> example / problem which has let to me asking this (note this has
>> been brought up before on various kernel lists, it is a
>> re-occuring theme, this is just an example really) :
>>
>> The problem at hand is a sata connector which also has a sata-power
>> connector on an embedded (ish) board where the sata-power is
>> controlled through a gpio. The sata-power connector is modeled
>> in devicetree as a power-supply and this supply gets controlled
>> by the ahci_platform driver.
>>
>> The disk power may very well have already been turned on by the
>> bootloader, so we add a regulator-boot-on property to the regulator
>> node in devicetree to make sure that it is left untouched when the
>> regulator driver loads. If the ahci_platform driver is build into
>> the kernel, it will then take control of the regulator and
>> everything works well.
>>
>> If however the ahci_platform driver is a module, then as soon as
>> the kernel is ready to start init, unused regulators are turned off
>> and the disk looses its power while spinning and ends up doing an
>> emergency heads park.
>
> What turns off the power in this situation?  The kernel?  Or userspace?
> Don't you have control of this?
>
> Have you tried to even create a patch that could do this type of thing
> to udev to see if it is even possible?
>
> thanks,
>
> greg k-h
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



-- 
Jon Smirl
jonsmirl-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [systemd-devel] Making udev emit a signal when it is done loading modules

[David Herrmann]

Hi

On Sat, Jan 17, 2015 at 9:44 AM, Hans de Goede <hdegoede@redhat.com> wrote:
> Dear udev developers,
>
> We (me and some kernel devs mostly) would like to add support to
> the kernel for userspace telling the kernel that it is done with
> the *initial* loading of modules, with the purpose of cleaning up
> (disabling) unused harware resources like e.g. regulators and
> clocks.
>
> Currently the kernel does this cleanup just before it starts init
> (which may very well be init from a ramdisk). In some cases this
> is too early really, because later on a module may get loaded
> which needs this resources, these resources will then get turned
> on again by the loaded driver, and most of the time this is not
> an issue, but sometimes it is.
>
> I realize very well that there is no magic moment where udev is
> really ever done loading modules, but the case which we want to
> support only involves devices which are *already enumerated*, but
> may not yet have a driver loaded, when udev starts. We would like
> udev to emit a signal (ABI to be discussed) when it is done
> trying to load modules for everything which was already enumerated
> when it starts, iow when there are no new device events pending
> anymore when udev does its initial hotplug replay.
>
> So the question to you is would you be willing to include such
> functionality in udev ?  Note this signal would need to be emitted
> when udev from the real rootfs is done with the initial module
> loading, as the real rootfs may very well have more modules
> available then the initrd.

We spent quite some time making module loading 'on demand'. This
means, udev is by no means the only one loading modules. Furthermore,
udev might broadcast events which is reacted on by external listeners.
We cannot rely on them *not* loading modules.

A notion of "settled" would be handy in a lot of situations, but I
doubt that we can enforce it in udev.

Thanks
David

Re: Making udev emit a signal when it is done loading modules

[Zbigniew Jędrzejewski-Szmek]

On Sat, Jan 17, 2015 at 09:44:00AM +0100, Hans de Goede wrote:
> We would like
> udev to emit a signal (ABI to be discussed) when it is done
> trying to load modules for everything which was already enumerated
> when it starts, iow when there are no new device events pending
> anymore when udev does its initial hotplug replay.
I think you can just create a unit like:

# disable-new-hardware.service
[Unit]
After=systemd-udev-settle.service systemd-modules-load.service
Wants=systemd-udev-settle.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/ping-the-kernel

> So the question to you is would you be willing to include such
> functionality in udev?
I don't think udevd has enough knowledge. But a systemd unit like
the one above should work.

Zbyszek
_______________________________________________
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Making udev emit a signal when it is done loading modules

[Kay Sievers]

On Sat, Jan 17, 2015 at 5:03 PM, Zbigniew Jędrzejewski-Szmek
<zbyszek@in.waw.pl> wrote:
> On Sat, Jan 17, 2015 at 09:44:00AM +0100, Hans de Goede wrote:
>> We would like
>> udev to emit a signal (ABI to be discussed) when it is done
>> trying to load modules for everything which was already enumerated
>> when it starts, iow when there are no new device events pending
>> anymore when udev does its initial hotplug replay.
> I think you can just create a unit like:
>
> # disable-new-hardware.service
> [Unit]
> After=systemd-udev-settle.service systemd-modules-load.service
> Wants=systemd-udev-settle.service
>
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStart=/usr/local/bin/ping-the-kernel
>
>> So the question to you is would you be willing to include such
>> functionality in udev?
> I don't think udevd has enough knowledge. But a systemd unit like
> the one above should work.

No, nothing new, maintained or properly working tool should use or
rely on udevadm settle. Settle is just a dirty hack to make broken
legacy tools work until they disappear.

Kay

Re: Making udev emit a signal when it is done loading modules

[Lennart Poettering]

On Sat, 17.01.15 17:03, Zbigniew Jędrzejewski-Szmek (zbyszek@in.waw.pl) wrote:

> On Sat, Jan 17, 2015 at 09:44:00AM +0100, Hans de Goede wrote:
> > We would like
> > udev to emit a signal (ABI to be discussed) when it is done
> > trying to load modules for everything which was already enumerated
> > when it starts, iow when there are no new device events pending
> > anymore when udev does its initial hotplug replay.
> I think you can just create a unit like:
> 
> # disable-new-hardware.service
> [Unit]
> After=systemd-udev-settle.service systemd-modules-load.service
> Wants=systemd-udev-settle.service
> 
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStart=/usr/local/bin/ping-the-kernel
> 
> > So the question to you is would you be willing to include such
> > functionality in udev?
> I don't think udevd has enough knowledge. But a systemd unit like
> the one above should work.

To clarify this: if people do this, then this pulls in
systemd-udev-settle.service, which slows down boot. Every service that
does that is hence a majour source of slowness. 

It's a hack to use this, not a solution. 

Lennart

-- 
Lennart Poettering, Red Hat
_______________________________________________
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Making udev emit a signal when it is done loading modules

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 791 bytes --]

On Wed, Jan 28, 2015 at 02:38:52AM +0100, Lennart Poettering wrote:

> To clarify this: if people do this, then this pulls in
> systemd-udev-settle.service, which slows down boot. Every service that
> does that is hence a majour source of slowness. 

> It's a hack to use this, not a solution. 

Well, yes.  There aren't really any good solutions with our event driven
model - we never "finish" booting, we just get to a point where nothing
has been happening for a while.  I have been thinking that we need to
just admit that properly and do something timer based - have a timer
that gets reset every time we instantiate something, then do all our
"end of boot" actions when nothing happened for a while.  It's not
elegant but I don't think elegant is a realistic goal here.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]