Report: Threaded udevd

Report: Threaded udevd

[Alan Jenkins]

As hinted earlier, I'm working on a threaded version of udevd.  The idea
is to processes events in threads, instead of forking a process for each
event.  I'm offended by how much cpu time udev takes during boot, and
this inefficiency seems to be the biggest culprit.

Patches will be posted as replies.

= Result =

I've succeeded in reducing the overall cpu usage of the udevd process by
45% on my EeePC.  This is measured by oprofile using "udevadm trigger"
on a running system.

Unfortunately, bootchart shows it only reduces the real time by 0.6
seconds out of a total of 2.4 seconds in the udev initscript - thats
25%.  The difference is mainly due to 160 calls to modprobe.  This is
using the recently released modprobe release 3.5, which has optimised
binary index files.  Future optimisations might include a "modprobe daemon".

= Caveats =

I've finished polishing this first version and fixed all the bugs I
found, but there could be some more :).  In particular, I still need to
verify that all the library functions it calls are reentrant.  The
libudev calls should be fine, but I'm not certain that all the standard
C library calls are safe.

It depends on pthreads being available - at the moment they are
compulsory.  It should be straightforward to make threading optional at
build time though.

To build with pthreads I use "acx_pthreads.m4", which is licensed under
the GPL3 with a special autoconf exception.  I think the intent of the
license is to make the file itself is GPL3, but leave the derived
configure script freely distributable.  It sounds ok, but I wouldn't
merge it without a more authoritative answer.

Timeouts are treated differently, and have not been tested.  Instead of
setting a timeout for the event as a whole, each external program is run
with a timeout.  If the timeout expires, the program is treated as if it
failed - instead of just aborting the event process.

= Justification for udev-exec =

External programs are run from a "udev-exec" subprocess.  This was
measured as reducing the fork() overhead.  To reduce overheads further,
it allowed me to use of vfork() without temporarily suspending all the
threads.  The commands to run are passed over a unix datagram socket;
pipes for stdout / stderr pipes are passed as ancillary data.  This
udev-exec accounts for over half the code.  The message passing
shouldn't impose too much overhead.  I think udev-exec should be used
even if threading has been disabled, otherwise there would be too many
ugly #ifdefs.

udev-exec also avoids running external programs with extra file
descriptors - the problem "close on exec" was designed to solve.  I
don't think I could insist on a kernel recent enough to provide "close
on exec" support.  Plus I don't think "close on exec" helps with pipes
used to communicate with child processes.  Without this, the Ubuntu
"watershed" command seemed to complain about extra file descriptors.

Lastly, udev-exec helped with implementing timeouts - event threads just
call select() with a timeout, waiting to read the exitcode of the
command from a pipe.  None of the wait() family of commands include a
timeout.  alarm() wouldn't work because it's per-process.  Without
udev-exec, I would have to create posix timers with callbacks to signal
the correct thread - and disable SA_RESTART before waiting.

Re: Report: Threaded udevd

[Scott James Remnant]

[-- Attachment #1: Type: text/plain, Size: 2590 bytes --]

On Wed, 2008-10-22 at 15:10 +0100, Alan Jenkins wrote:

> == Justification for udev-exec ==
> 
> External programs are run from a "udev-exec" subprocess.  This was
> measured as reducing the fork() overhead.  To reduce overheads further,
> it allowed me to use of vfork() without temporarily suspending all the
> threads.  The commands to run are passed over a unix datagram socket;
> pipes for stdout / stderr pipes are passed as ancillary data.  This
> udev-exec accounts for over half the code.  The message passing
> shouldn't impose too much overhead.  I think udev-exec should be used
> even if threading has been disabled, otherwise there would be too many
> ugly #ifdefs.
> 
This sounds odd to me; udev-exec still needs to fork() for each child,
and there's a cost of passing the command over the socket -- so how is
this cheaper than just fork()ing inside udevd?

> udev-exec also avoids running external programs with extra file
> descriptors - the problem "close on exec" was designed to solve.  I
> don't think I could insist on a kernel recent enough to provide "close
> on exec" support.  Plus I don't think "close on exec" helps with pipes
> used to communicate with child processes.  Without this, the Ubuntu
> "watershed" command seemed to complain about extra file descriptors.
> 
I see no problem with depending on the very latest kernels; using a
latest udev version goes hand-in-hand with using a latest kernel.

The pipe2() syscall allows you to specify flags for the returned file
descriptors:

	pipe2 (&fds, O_CLOEXEC)

> Lastly, udev-exec helped with implementing timeouts - event threads just
> call select() with a timeout, waiting to read the exitcode of the
> command from a pipe.  None of the wait() family of commands include a
> timeout.  alarm() wouldn't work because it's per-process.  Without
> udev-exec, I would have to create posix timers with callbacks to signal
> the correct thread - and disable SA_RESTART before waiting.
> 
Ironically, I have a long-standing proposal to fix this ;)  I've just
got to get around to persuading a kernel developer to do it, or do it
myself.

The idea was you'd do:

  fd = waitfd (P_PID, pid, 0);

  FD_SET (fd, &readfds);

  select (fd + 1, &readfds, NULL, NULL, &timeout);

  read (fd, &siginfo, sizeof (struct siginfo_t));

ie. a version of waitid() that returns the data through a socket - then
all daemon main loop events could be done with select() [we already have
signalfd(), timerfd_*(), etc.)

Scott
-- 
Scott James Remnant
scott@canonical.com

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Report: Threaded udevd

[Marco d'Itri]

[-- Attachment #1: Type: text/plain, Size: 274 bytes --]

On Oct 22, Scott James Remnant <scott@canonical.com> wrote:

> I see no problem with depending on the very latest kernels; using a
> latest udev version goes hand-in-hand with using a latest kernel.
I do, since this tends to badly mess upgrades...

-- 
ciao,
Marco

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Report: Threaded udevd

[Alan Jenkins]

Scott James Remnant wrote:
> On Wed, 2008-10-22 at 15:10 +0100, Alan Jenkins wrote:
>
>   
>> = Justification for udev-exec =
>>
>> External programs are run from a "udev-exec" subprocess.  This was
>> measured as reducing the fork() overhead.  To reduce overheads further,
>> it allowed me to use of vfork() without temporarily suspending all the
>> threads.  The commands to run are passed over a unix datagram socket;
>> pipes for stdout / stderr pipes are passed as ancillary data.  This
>> udev-exec accounts for over half the code.  The message passing
>> shouldn't impose too much overhead.  I think udev-exec should be used
>> even if threading has been disabled, otherwise there would be too many
>> ugly #ifdefs.
>>
>>     
> This sounds odd to me; udev-exec still needs to fork() for each child,
> and there's a cost of passing the command over the socket -- so how is
> this cheaper than just fork()ing inside udevd?
>
>   

My guess was that it avoids cloning the mappings for the thread stacks. 
I can test this again and compare the two profiles.  There was
definitely a significant difference, but I should have another look and
find out why.

I also measured an improvement from using vfork().  I don't know how
that horror interacts with threading, and I don't want to find out :).

>> udev-exec also avoids running external programs with extra file
>> descriptors - the problem "close on exec" was designed to solve.  I
>> don't think I could insist on a kernel recent enough to provide "close
>> on exec" support.  Plus I don't think "close on exec" helps with pipes
>> used to communicate with child processes.  Without this, the Ubuntu
>> "watershed" command seemed to complain about extra file descriptors.
>>
>>     
> I see no problem with depending on the very latest kernels; using a
> latest udev version goes hand-in-hand with using a latest kernel.
>
>   

Hmm.  I guess libudev ought to use it either way.  (libudev can test if
the flag is supported; I don't think it needs the new system calls).

> The pipe2() syscall allows you to specify flags for the returned file
> descriptors:
>
> 	pipe2 (&fds, O_CLOEXEC)
>   

Ah!  I see how it's supposed to be used now.

pipe2 (&fds, O_CLOEXEC)
if (fork() = 0) {
   dup2(&fds[WRITE_END], STDOUT_FILENO);
   /* insert call to fcntl to remove O_CLOEXEC */
   exec(program_name);
...

I was missing the call to fcntl().  It's doesn't really work without it :).

>> Lastly, udev-exec helped with implementing timeouts - event threads just
>> call select() with a timeout, waiting to read the exitcode of the
>> command from a pipe.  None of the wait() family of commands include a
>> timeout.  alarm() wouldn't work because it's per-process.  Without
>> udev-exec, I would have to create posix timers with callbacks to signal
>> the correct thread - and disable SA_RESTART before waiting.
>>
>>     
> Ironically, I have a long-standing proposal to fix this ;)  I've just
> got to get around to persuading a kernel developer to do it, or do it
> myself.
>
> The idea was you'd do:
>
>   fd = waitfd (P_PID, pid, 0);
>
>   FD_SET (fd, &readfds);
>
>   select (fd + 1, &readfds, NULL, NULL, &timeout);
>
>   read (fd, &siginfo, sizeof (struct siginfo_t));
>
> ie. a version of waitid() that returns the data through a socket - then
> all daemon main loop events could be done with select() [we already have
> signalfd(), timerfd_*(), etc.)
>   

I saw a similar complaint by Bernstein when I looked up the self-pipe
trick.  "Of course, the Right Thing would be to have fork() return a
file descriptor".  FWIW, wayback says the page was put up in 2000.

Thanks for the comments
Alan

Re: Report: Threaded udevd

[Scott James Remnant]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

On Wed, 2008-10-22 at 17:12 +0100, Alan Jenkins wrote:

> > The pipe2() syscall allows you to specify flags for the returned file
> > descriptors:
> >
> > 	pipe2 (&fds, O_CLOEXEC)
> >   
> 
> Ah!  I see how it's supposed to be used now.
> 
> pipe2 (&fds, O_CLOEXEC)
> if (fork() == 0) {
>    dup2(&fds[WRITE_END], STDOUT_FILENO);
>    /* insert call to fcntl to remove O_CLOEXEC */
>    exec(program_name);
> 
You don't need the call to fcntl.  From the dup2(2) manpage:

       The two descriptors do not share file descriptor flags  (the  close-on-
       exec  flag).  The close-on-exec flag (FD_CLOEXEC; see fcntl(2)) for the
       duplicate descriptor is off.

So after you dup2() the writing end to standard output, standard output
will not be closed on exec.

Scott
-- 
Scott James Remnant
scott@canonical.com

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Report: Threaded udevd

[Greg KH]

On Wed, Oct 22, 2008 at 05:15:31PM +0200, Marco d'Itri wrote:
> On Oct 22, Scott James Remnant <scott@canonical.com> wrote:
> 
> > I see no problem with depending on the very latest kernels; using a
> > latest udev version goes hand-in-hand with using a latest kernel.
> I do, since this tends to badly mess upgrades...

How?  Can't you just specify a minimum kernel version in your packaging
system for the latest version of udev?  That's what we do for other
distros...

thanks,

greg k-h

Re: Report: Threaded udevd

[Marco d'Itri]

[-- Attachment #1: Type: text/plain, Size: 1140 bytes --]

On Oct 22, Greg KH <greg@kroah.com> wrote:

> > > I see no problem with depending on the very latest kernels; using a
> > > latest udev version goes hand-in-hand with using a latest kernel.
> > I do, since this tends to badly mess upgrades...
> How?  Can't you just specify a minimum kernel version in your packaging
> system for the latest version of udev?  That's what we do for other
> distros...
The problem is handling upgrades between two different major revisions
of a distribution. e.g. Debian 3.1 shipped udev 56 and kernel 2.6.8,
while Debian 4.0 shipped udev 105 and kernel 2.6.18.
udev 105 did not work with kernel 2.6.18 and udev 56 did not work with
kernel 2.6.8, so the udev installation script had to jump hoops to not
start the new udev, make sure that a suitable updated kernel was
installed at the same time and recommend the administrator to reboot
ASAP.
And if for some reason the new kernel does not work, the user will be
left with a mostly broken (no working udev) system.
This is why I would really really like to see kernel 2.6.26 supported by
udev for the next few years.

-- 
ciao,
Marco

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Report: Threaded udevd

[Scott James Remnant]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

On Wed, 2008-10-22 at 19:02 +0200, Marco d'Itri wrote:

> On Oct 22, Greg KH <greg@kroah.com> wrote:
> 
> > > > I see no problem with depending on the very latest kernels; using a
> > > > latest udev version goes hand-in-hand with using a latest kernel.
> > > I do, since this tends to badly mess upgrades...
> > How?  Can't you just specify a minimum kernel version in your packaging
> > system for the latest version of udev?  That's what we do for other
> > distros...
> The problem is handling upgrades between two different major revisions
> 
We just don't restart the kernel or udev until after the upgrade is
done :)

Scott
-- 
Scott James Remnant
scott@canonical.com

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Report: Threaded udevd

[Marco d'Itri]

[-- Attachment #1: Type: text/plain, Size: 268 bytes --]

On Oct 22, Scott James Remnant <scott@canonical.com> wrote:

> We just don't restart the kernel or udev until after the upgrade is
> done :)
Which may not work well if the new configuration files are not
compatible enough with the old udev.

-- 
ciao,
Marco

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Report: Threaded udevd

[Karl O. Pinc]

On 10/22/2008 11:24:53 AM, Greg KH wrote:
> On Wed, Oct 22, 2008 at 05:15:31PM +0200, Marco d'Itri wrote:
> > On Oct 22, Scott James Remnant <scott@canonical.com> wrote:
> >
> > > I see no problem with depending on the very latest kernels; using
> a
> > > latest udev version goes hand-in-hand with using a latest kernel.
> > I do, since this tends to badly mess upgrades...

FWIW, I've a perspective as an end-user....

There have been several times when I've got hardware that's
too new.  I get it to work by installing a newer kernel,
either one I build or something backported.  (FWIW I tend to
use Deiban.)

Aside from having to learn how to build and install a custom
udev to go with my newer kernel, I'd have to learn how to
install and uninstall the new udev depending upon
which kernel I'm running.

If udev is to be closely tied to the kernel version
it'd be nice if:

A) Packagers packaged udev with the kernel.  (No doubt causing
problems for those who don't use udev for whatever reason.)

B) The boot process automatically used the correct udev to
go with the kernel chosen at boot time.

In the ideal world newer kernels would work with older
userspace binaries for at least a few years.  Otherwise
it soon becomes impossible for a mere mortal to
run newer kernel versions because the system integration
involved gets out of control.  I occasionally provide
support on irc and I already feel close to the edge
when it comes to telling somebody what they need to
do to try a newer kernel.  It would be sad to have to
tell people "you'll have to wait months, at least, for
the distro to integrate the latest code before you can
run Linux on your shiny new hardware".  (It always
seems to be the network hardware that's a problem,
which sorta makes Linux a non-starter.)

Regards,

Karl <kop@meme.com>
Free Software:  "You don't pay back, you pay forward."
                  -- Robert A. Heinlein

Re: Report: Threaded udevd

[Marco d'Itri]

On Oct 23, "Karl O. Pinc" <kop@meme.com> wrote:

> B) The boot process automatically used the correct udev to
> go with the kernel chosen at boot time.
Not practical, this would require two sets of configuration files.

> In the ideal world newer kernels would work with older
> userspace binaries for at least a few years.  Otherwise
This is the only sane solution, but it should be true for modern kernels
unless you disable the compatibility support.

-- 
ciao,
Marco

Re: Report: Threaded udevd

[Karl O. Pinc]

On 10/23/2008 11:29:47 AM, Marco d'Itri wrote:
> On Oct 23, "Karl O. Pinc" <kop@meme.com> wrote:
> 
> > B) The boot process automatically used the correct udev to
> > go with the kernel chosen at boot time.
> Not practical, this would require two sets of configuration files.

Yah.  I figured.  I want a pony too.  ;-)

Just thought I'd report from the trenches.

Karl <kop@meme.com>
Free Software:  "You don't pay back, you pay forward."
                  -- Robert A. Heinlein