The case of udev and the missing /dev/input/mice

The case of udev and the missing /dev/input/mice

[Scott James Remnant]

[-- Attachment #1: Type: text/plain, Size: 4843 bytes --]

Background: in the upcoming Ubuntu 5.10 we've been having some problems
with /dev/input/mice not being created on startup despite the "mousedev"
module being hard-loaded early in the boot sequence.
(http://bugzilla.ubuntu.com/show_bug.cgi?id=12915 for those interested).

Debian has had similar problems too (http://bugs.debian.org/317333) and
found that starting udevd earlier manually seemed to fix it.


After much debugging, I've finally figured out what's going on ... it's
a bit of a story, but here goes...


Your system boots up and gets to the S:S20modules-init-tools stage,
that's where we read /etc/modules and modprobe the modules in order.
Now modprobe is basically just a kernel request, and these days tends to
return pretty quicky to userspace without blocking for everything to
happen.

Deep Black Magic happens inside the kernel, and once it's done it
generates a series of hotplug events which it passes back to userspace
through two means; by running the program specified
in /proc/sys/kernel/hotplug with interesting environment; and also
through a netlink socket.

/proc/sys/kernel/hotplug is "udevsend", a tool that gathers up this
environment and sends it over a local socket to the "udevd" process that
marshals all of these events.  If there's no daemon listening it tries
to start one up, and will retry sending the event for a while until it
gets to the other end.

Now we have a whole bunch of udevsend processes all run at pretty much
the same time, all of these try to start up udevd and all of the udevd
processes try to bind to the local socket to receive events on.  One of
them wins, the rest die and go away.  A little time passes by which time
all of the running udevsend will have dispatched their event to this
udevd that will marshal them.

This udevd _also_ begins listening on the netlink socket, as it's a
better way to get events from the kernel than having it execute
something which mucks around with IPC to get it to us.

Meanwhile the kernel is happily generating both /proc/sys/kernel/hotplug
and netlink events for what's happening on the box, in fact it's been
doing this all the time udevd has been getting its clothes on.

If the module sequence loaded is something like "psmouse, mousedev, ...,
lp" (exactly as it is in breezy machines that have been upgraded from
warty/hoary[0]) you may find that the first netlink event you receive is
actually for the printer port.

But that's ok, we had udevsend events for the rest...

Well, that's the theory; sadly here's the practice.

On receiving the netlink event for the printer port, udevd disables
receipt of any "sequence numbered" events from udevsend (ie. those that
will almost certainly be duplicated over the netlink socket).
Unfortunately this means all the udevsend events we're about to receive
from the processes that backed off a second or so while fighting over
who got to start udevd[1].

These udevsend processes deliver their events to udevd, which cheerfully
ignores them because it thinks it's going to get another copy over the
netlink socket any second now.  Unfortunately the netlink event has
already been and gone, and we just ignored an event we weren't supposed
to.


The two problems as I see them are:

1) The fact that receiving a netlink event disables sequence numbered
   udevsend events, when there's already code to deal with de-duping
   events anyway.  Is there actually any need for this additional check,
   can't we just queue both events and have them ignored by
   msg_queue_insert() ?

2) That this ignoring of events is done at receipt, rather than in queue
   order.  This means that the "later" parport_pc netlink event is able
   to disable queueing of udevsend events with a lower sequence number.

I can envisage that #1 is necessary in case the time between receiving
the udevsend and netlink event is so long that we've already processed
and removed one of the events by the time the second is queued.  In
which case the problem becomes fixing #2, however unless the kernel
promises strict ordering of events over the netlink socket (which I
doubt, otherwise it wouldn't need sequence numbers), we can't assume
that we've received all of the pre-netlink events we are going to.

I suspect the right solution is actually to implement history of what
events we've already processed, and de-dupe them that way; rather than
ignoring messages on receipt.

Scott

[0] A common "fix" has been to simply install breezy fresh; this happens
    to change the /etc/modules order slightly and thus hide the bug.
[1] And if we deliberately start udevd before we begin any of this
    module loading, it sees the netlink event, and thus again hides the 
    bug.
-- 
Scott James Remnant
scott@ubuntu.com

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The case of udev and the missing /dev/input/mice

[Kay Sievers]

On Wed, Sep 21, 2005 at 03:38:06AM +0100, Scott James Remnant wrote:
> Background: in the upcoming Ubuntu 5.10 we've been having some problems
> with /dev/input/mice not being created on startup despite the "mousedev"
> module being hard-loaded early in the boot sequence.
> (http://bugzilla.ubuntu.com/show_bug.cgi?id\x12915 for those interested).
> 
> Debian has had similar problems too (http://bugs.debian.org/317333) and
> found that starting udevd earlier manually seemed to fix it.

Yes, that's a good way to fix it.

> After much debugging, I've finally figured out what's going on ... it's
> a bit of a story, but here goes...

Great, we finally have an idea why this happens. Thanks for finding that
out.

> On receiving the netlink event for the printer port, udevd disables
> receipt of any "sequence numbered" events from udevsend (ie. those that
> will almost certainly be duplicated over the netlink socket).
> Unfortunately this means all the udevsend events we're about to receive
> from the processes that backed off a second or so while fighting over
> who got to start udevd[1].
> 
> These udevsend processes deliver their events to udevd, which cheerfully
> ignores them because it thinks it's going to get another copy over the
> netlink socket any second now.  Unfortunately the netlink event has
> already been and gone, and we just ignored an event we weren't supposed
> to.
> 
> 
> The two problems as I see them are:
> 
> 1) The fact that receiving a netlink event disables sequence numbered
>    udevsend events, when there's already code to deal with de-duping
>    events anyway.  Is there actually any need for this additional check,
>    can't we just queue both events and have them ignored by
>    msg_queue_insert() ?
> 
> 2) That this ignoring of events is done at receipt, rather than in queue
>    order.  This means that the "later" parport_pc netlink event is able
>    to disable queueing of udevsend events with a lower sequence number.
> 
> I can envisage that #1 is necessary in case the time between receiving
> the udevsend and netlink event is so long that we've already processed
> and removed one of the events by the time the second is queued.

Yes, that was the reason for ignoring the incoming messages.

> In which case the problem becomes fixing #2, however unless the kernel
> promises strict ordering of events over the netlink socket (which I
> doubt, otherwise it wouldn't need sequence numbers)

Netlink events are always in the right order. The SEQNUM is only needed
for the forked events.

> we can't assume
> that we've received all of the pre-netlink events we are going to.

Right, as "/proc/sys/kernel/hotplug" events are forked processes, you will
never know when and in which order they will arrive.

> I suspect the right solution is actually to implement history of what
> events we've already processed, and de-dupe them that way; rather than
> ignoring messages on receipt.

We could just accept all events with a lower sequence number as the first
netlink event's one, that may fix it.

The "right solution" is to start udevd as one of the first things
after taking over control from the kernel. This way you will only catch
the events for the last "non driver core" subsystem, the input layer.

At the time the input layer is fixed, the need for udevsend will
completely go away and /proc/sys/kernel/hotplug should be disabled
when taking over control from the kernel - it is only needed in
initramfs.
After input is fixed, the whole event reordering and timeout handling
will be removed from udevd and we need to start udevd manually anyway.

Kay


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: The case of udev and the missing /dev/input/mice

[Olivier Blin]

Kay Sievers <kay.sievers@vrfy.org> writes:

> At the time the input layer is fixed, the need for udevsend will
> completely go away and /proc/sys/kernel/hotplug should be disabled
> when taking over control from the kernel - it is only needed in
> initramfs.
> After input is fixed, the whole event reordering and timeout handling
> will be removed from udevd and we need to start udevd manually anyway.

Isn't it fixed now? Or not merged yet?
http://lkml.org/lkml/2005/9/15/59

-- 
Olivier Blin
Mandriva



-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: The case of udev and the missing /dev/input/mice

[Greg KH]

On Wed, Sep 21, 2005 at 01:19:35PM +0200, Olivier Blin wrote:
> Kay Sievers <kay.sievers@vrfy.org> writes:
> 
> > At the time the input layer is fixed, the need for udevsend will
> > completely go away and /proc/sys/kernel/hotplug should be disabled
> > when taking over control from the kernel - it is only needed in
> > initramfs.
> > After input is fixed, the whole event reordering and timeout handling
> > will be removed from udevd and we need to start udevd manually anyway.
> 
> Isn't it fixed now? Or not merged yet?
> http://lkml.org/lkml/2005/9/15/59

Not merged at all, we still are working on fixing up the driver core to
handle this properly...

thanks,

greg k-h


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: The case of udev and the missing /dev/input/mice

[Scott James Remnant]

[-- Attachment #1: Type: text/plain, Size: 1224 bytes --]

On Wed, 2005-09-21 at 11:51 +0200, Kay Sievers wrote:

> On Wed, Sep 21, 2005 at 03:38:06AM +0100, Scott James Remnant wrote:
> > Background: in the upcoming Ubuntu 5.10 we've been having some problems
> > with /dev/input/mice not being created on startup despite the "mousedev"
> > module being hard-loaded early in the boot sequence.
> > (http://bugzilla.ubuntu.com/show_bug.cgi?id=12915 for those interested).
> > 
> > Debian has had similar problems too (http://bugs.debian.org/317333) and
> > found that starting udevd earlier manually seemed to fix it.
> 
> Yes, that's a good way to fix it.
> 
One thing I'd like to see changed in udevd is to move the
init_udevd_socket() and init_uevent_netlink_sock() calls to above the
daemonization; that way when you call "udevd --daemon" from the init
script, you *know* that the next command may cause a netlink event.

Right now there's an unknown amount of time between calling "udevd
--daemon" and being able to safely "modprobe".

This'd also mean that udevd could exit with an error status if it's
unable to create the necessary sockets; rather than the child exiting
and the status being lost.

Scott
-- 
Scott James Remnant
scott@ubuntu.com

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The case of udev and the missing /dev/input/mice

[Dmitry Torokhov]

On 9/21/05, Greg KH <greg@kroah.com> wrote:
> On Wed, Sep 21, 2005 at 01:19:35PM +0200, Olivier Blin wrote:
> > Kay Sievers <kay.sievers@vrfy.org> writes:
> >
> > > At the time the input layer is fixed, the need for udevsend will
> > > completely go away and /proc/sys/kernel/hotplug should be disabled
> > > when taking over control from the kernel - it is only needed in
> > > initramfs.
> > > After input is fixed, the whole event reordering and timeout handling
> > > will be removed from udevd and we need to start udevd manually anyway.
> >
> > Isn't it fixed now? Or not merged yet?
> > http://lkml.org/lkml/2005/9/15/59
>
> Not merged at all, we still are working on fixing up the driver core to
> handle this properly...
>

I think while driver core is being fixed we should go "Suse way" and
add top-level input_dev class and merge input changes. Hovewer there
will be compatibility issues with current installation (as there will
be compatibility problems once we fix driver core but I'd rather force
users to upgrade once)...

Greg, would you accept a patch that would allow a class override its
subsustem as far as hotplug is concerned? This way "input_dev" class
would set its name to "input" allowing using existing setups.

--
Dmitry


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: The case of udev and the missing /dev/input/mice

[Greg KH]

On Wed, Sep 21, 2005 at 10:33:19AM -0500, Dmitry Torokhov wrote:
> On 9/21/05, Greg KH <greg@kroah.com> wrote:
> > On Wed, Sep 21, 2005 at 01:19:35PM +0200, Olivier Blin wrote:
> > > Kay Sievers <kay.sievers@vrfy.org> writes:
> > >
> > > > At the time the input layer is fixed, the need for udevsend will
> > > > completely go away and /proc/sys/kernel/hotplug should be disabled
> > > > when taking over control from the kernel - it is only needed in
> > > > initramfs.
> > > > After input is fixed, the whole event reordering and timeout handling
> > > > will be removed from udevd and we need to start udevd manually anyway.
> > >
> > > Isn't it fixed now? Or not merged yet?
> > > http://lkml.org/lkml/2005/9/15/59
> >
> > Not merged at all, we still are working on fixing up the driver core to
> > handle this properly...
> >
> 
> I think while driver core is being fixed we should go "Suse way" and
> add top-level input_dev class and merge input changes. Hovewer there
> will be compatibility issues with current installation (as there will
> be compatibility problems once we fix driver core but I'd rather force
> users to upgrade once)...
> 
> Greg, would you accept a patch that would allow a class override its
> subsustem as far as hotplug is concerned? This way "input_dev" class
> would set its name to "input" allowing using existing setups.

Ick, no way.  Let's fix this properly, there's no real need to do
something like this right now.

thanks,

greg k-h


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: The case of udev and the missing /dev/input/mice

[Greg KH]

On Wed, Sep 21, 2005 at 08:49:43AM -0700, Greg KH wrote:
> > Greg, would you accept a patch that would allow a class override its
> > subsustem as far as hotplug is concerned? This way "input_dev" class
> > would set its name to "input" allowing using existing setups.

Actually, it would be nice if you could respin your patches that convert
all of the input devices to be dynamically created.  That would work
just fine today with no sysfs changes, right?

That way, any future changes we do would be much smaller, and could be
done on top of your patchset.

Or am I missing some deep dependencies in your patchset?

thanks,

greg k-h


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: The case of udev and the missing /dev/input/mice

[Dmitry Torokhov]

On 9/21/05, Greg KH <greg@kroah.com> wrote:
> On Wed, Sep 21, 2005 at 08:49:43AM -0700, Greg KH wrote:
> > > Greg, would you accept a patch that would allow a class override its
> > > subsustem as far as hotplug is concerned? This way "input_dev" class
> > > would set its name to "input" allowing using existing setups.
>
> Actually, it would be nice if you could respin your patches that convert
> all of the input devices to be dynamically created.  That would work
> just fine today with no sysfs changes, right?
>
> That way, any future changes we do would be much smaller, and could be
> done on top of your patchset.
>
> Or am I missing some deep dependencies in your patchset?
>

Well, no ;) I was trying to make them separable. I will resping the
dynamic allocation patches, but I would also like to get sysfs part
going - there are bunch of locking issues (or rather absence of
locking ;) ) in input core and I would like to fix it but I don't want
to redo it later to account for sysfs rules. That's why I am pushing
to get some kind of sysfs resolution. Again, that "name override"
would go away later, I do not propose it as a permanant solution.

--
Dmitry


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: The case of udev and the missing /dev/input/mice

[Greg KH]

On Wed, Sep 21, 2005 at 11:11:21AM -0500, Dmitry Torokhov wrote:
> On 9/21/05, Greg KH <greg@kroah.com> wrote:
> > On Wed, Sep 21, 2005 at 08:49:43AM -0700, Greg KH wrote:
> > > > Greg, would you accept a patch that would allow a class override its
> > > > subsustem as far as hotplug is concerned? This way "input_dev" class
> > > > would set its name to "input" allowing using existing setups.
> >
> > Actually, it would be nice if you could respin your patches that convert
> > all of the input devices to be dynamically created.  That would work
> > just fine today with no sysfs changes, right?
> >
> > That way, any future changes we do would be much smaller, and could be
> > done on top of your patchset.
> >
> > Or am I missing some deep dependencies in your patchset?
> >
> 
> Well, no ;) I was trying to make them separable. I will resping the
> dynamic allocation patches,

Great.

> but I would also like to get sysfs part going - there are bunch of
> locking issues (or rather absence of locking ;) ) in input core and I
> would like to fix it but I don't want to redo it later to account for
> sysfs rules.

I agree.

> That's why I am pushing to get some kind of sysfs resolution. Again,
> that "name override" would go away later, I do not propose it as a
> permanant solution.

I have to give a talk about this next week, so I want to have this done
by then so I have something to talk about :)

So don't worry, I am activly working on it, I have the basics done now,
just trying to get it all to work without oopsing in odd places...

thanks,

greg k-h


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: The case of udev and the missing /dev/input/mice

[Dmitry Torokhov]

On 9/21/05, Greg KH <greg@kroah.com> wrote:
>
> I have to give a talk about this next week, so I want to have this done
> by then so I have something to talk about :)
>
> So don't worry, I am activly working on it, I have the basics done now,
> just trying to get it all to work without oopsing in odd places...
>

Just remember - release often, release early ;)

Btw, I see you picked up my "call hotplug earlier" patch but not "pass
class_interface" to add()/remove()". Do you have an objection for the
later? (It will break I2O unless you have my patch that removes usage
of class_interface from there, but other than that it should be fine).

--
Dmitry


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: The case of udev and the missing /dev/input/mice

[Greg KH]

On Wed, Sep 21, 2005 at 11:46:31AM -0500, Dmitry Torokhov wrote:
> On 9/21/05, Greg KH <greg@kroah.com> wrote:
> >
> > I have to give a talk about this next week, so I want to have this done
> > by then so I have something to talk about :)
> >
> > So don't worry, I am activly working on it, I have the basics done now,
> > just trying to get it all to work without oopsing in odd places...
> >
> 
> Just remember - release often, release early ;)

Hey, you can always see the current state of my tree, along with this
work-in-progress on kernel.org in:
	pub/linux/kernel/people/gregkh/gregkh/patches
if you really are brave :)

> Btw, I see you picked up my "call hotplug earlier" patch but not "pass
> class_interface" to add()/remove()". Do you have an objection for the
> later? (It will break I2O unless you have my patch that removes usage
> of class_interface from there, but other than that it should be fine).

I'm reworking the class_interface stuff right now, so I was going to
hold off on this to see if it's needed.

Well, why do you need it?  You want to have a "generic" class interface?
Why not individual different ones for every interface type you want to
use?

thanks,

greg k-h


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: The case of udev and the missing /dev/input/mice

[Dmitry Torokhov]

On 9/21/05, Greg KH <greg@kroah.com> wrote:
> On Wed, Sep 21, 2005 at 11:46:31AM -0500, Dmitry Torokhov wrote:
> > On 9/21/05, Greg KH <greg@kroah.com> wrote:
> > >
> > > I have to give a talk about this next week, so I want to have this done
> > > by then so I have something to talk about :)
> > >
> > > So don't worry, I am activly working on it, I have the basics done now,
> > > just trying to get it all to work without oopsing in odd places...
> > >
> >
> > Just remember - release often, release early ;)
>
> Hey, you can always see the current state of my tree, along with this
> work-in-progress on kernel.org in:
>        pub/linux/kernel/people/gregkh/gregkh/patches
> if you really are brave :)
>
> > Btw, I see you picked up my "call hotplug earlier" patch but not "pass
> > class_interface" to add()/remove()". Do you have an objection for the
> > later? (It will break I2O unless you have my patch that removes usage
> > of class_interface from there, but other than that it should be fine).
>
> I'm reworking the class_interface stuff right now, so I was going to
> hold off on this to see if it's needed.
>
> Well, why do you need it?  You want to have a "generic" class interface?
> Why not individual different ones for every interface type you want to
> use?
>

For input I want to do some actions every time I connect a new
interface to the device and I want to have it done in core. So when a
new interface registers core installs add()/remove() handler which
then calls interface's connect/disconnect handlers. The other way
around would be to program every interface to call into input-core
exported function to finalize connect/disconnect which is not clean
IMHO. I generally prefer the core to control the operation, instead of
leaving it to individual drivers.

> thanks,
>
> greg k-h
>


--
Dmitry


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel