Re: Recommendation for activating a deferred module init in the kernel

Re: Recommendation for activating a deferred module init in the kernel

[Tim Bird]

Jörn Engel wrote:
> On Tue, 17 June 2008 11:23:18 -0700, Tim Bird wrote:
>> I'm not that happy using an ioctl for this trigger.  What is
>> the preferred method of activating a kernel feature like this?
>> I presume something in /proc or /sys, but I'm not sure.
> 
> I personally would be unhappy with any kind of interface for this.  It
> would be much nicer to make it transparent and still get the benefits.
> One option would be to start a kernel thread for the initialization and
> renice it to 19 or so.

That's an interesting idea. I'm pretty sure the product guys want
an explicit trigger, so they can make sure they've got the main
application well underway before this deferred initialization occurs.

> 
> If you want an explicit trigger, you could either hook into init_post()
> or have hooks in the open functions of drivers with deferred
> initialization.

This would presumably require multiple calls (one to the open of
each deferred module).  I would still need a trigger for the memory
free operation, unless I hardcode the order of the opening and just
"know" that the last one should free the memory.  I'll have to see
if all the modules being loaded like this have open()s.

Thanks for the ideas!
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: Recommendation for activating a deferred module init in the kernel

[Tim Bird]

Tim Bird wrote:
>> If you want an explicit trigger, you could either hook into init_post()
>> or have hooks in the open functions of drivers with deferred
>> initialization.
> 
> This would presumably require multiple calls (one to the open of
> each deferred module).  I would still need a trigger for the memory
> free operation, unless I hardcode the order of the opening and just
> "know" that the last one should free the memory.  I'll have to see
> if all the modules being loaded like this have open()s.

Sorry - I responded too quickly.  I'm not sure I follow the
original suggestion.  How would I call the open function of
a module that is not initialized yet?
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: Recommendation for activating a deferred module init in the kernel

[Jörn Engel]

On Tue, 17 June 2008 12:55:31 -0700, Tim Bird wrote:
> 
> Sorry - I responded too quickly.  I'm not sure I follow the
> original suggestion.  How would I call the open function of
> a module that is not initialized yet?

Hmm, good point.  I guess that suggestion has just failed the reality
test.

Jörn

-- 
You ain't got no problem, Jules. I'm on the motherfucker. Go back in
there, chill them niggers out and wait for the Wolf, who should be
coming directly.
-- Marsellus Wallace

Re: Recommendation for activating a deferred module init in the kernel

[Josh Boyer]

On Tue, 17 Jun 2008 22:23:19 +0200
Jörn Engel <joern@logfs.org> wrote:

> On Tue, 17 June 2008 12:55:31 -0700, Tim Bird wrote:
> > 
> > Sorry - I responded too quickly.  I'm not sure I follow the
> > original suggestion.  How would I call the open function of
> > a module that is not initialized yet?
> 
> Hmm, good point.  I guess that suggestion has just failed the reality
> test.

Have a simple module that calls the initialization functions for
anything with a deferred init.  Then you can open that and use an ioctl
to...

I think if we try really really hard, we can make this a Rube-Goldberg
Machine!

josh

Re: Recommendation for activating a deferred module init in the kernel

[Stefan Richter]

>> On Tue, 17 June 2008 12:55:31 -0700, Tim Bird wrote:
>>> Sorry - I responded too quickly.  I'm not sure I follow the
>>> original suggestion.  How would I call the open function of
>>> a module that is not initialized yet?

You will be able to open a character device file as soon as cdev_init() 
finished... when the Big Kernel Lock is not being taken around 
file_operations.open() anymore.

On Tue, 17 Jun 2008 11:28:29 -0700, Tim Bird wrote:
| One technique
| that we are forward-porting from an old kernel (and that I thought I
| might work on mainlining) is to compile modules statically into the
| kernel, but defer their initialization until after boot time.
...
| One of the main sub-systems that we defer initialization of this
| way is USB, and this saves quite a bit of time.  (Of course the
| same, or slightly more CPU cycles are eventually used during
| bootup time.  But this lets us get to user space quicker so we
| can start user-visible applications faster.)

What if you don't defer module initialization, but merely device probing?

$ ls /sys/bus/*/drivers_autoprobe
/sys/bus/acpi/drivers_autoprobe
/sys/bus/firewire/drivers_autoprobe
/sys/bus/i2c/drivers_autoprobe
/sys/bus/ide/drivers_autoprobe
/sys/bus/pci/drivers_autoprobe
/sys/bus/pci_express/drivers_autoprobe
/sys/bus/platform/drivers_autoprobe
/sys/bus/pnp/drivers_autoprobe
/sys/bus/scsi/drivers_autoprobe
/sys/bus/serio/drivers_autoprobe
/sys/bus/usb/drivers_autoprobe

If you set /sys/bus/foo/drivers_autoprobe to 0 (default is 1), then a 
/sys/bus/foo/drivers/bar will not be bound to devices.  You can trigger 
driver--device binding later per device by writing a device's bus ID 
into /sys/bus/foo/drivers/bar/bind, or by writing into 
/sys/bus/foo/drivers_probe (I guess; I only used the per-device way so far).

Now, since you want to do this with statically linked drivers, i.e. need 
to prevent probing before userspace and sysfs are up and running, you 
probably need to modify the bus types or/and the driver core so that the 
the less vital buses have drivers_autoprobe off by default.

On the other hand, maybe you want to have probes of some PCI drivers 
executed but not the probes of some other PCI drivers; uhci_hcd for 
example.  I guess you could achieve that by modifying 
drivers/pci/pci-driver.c::pci_bus_match().  E.g. add a sysfs attribute 
to pci-driver.c which, as long as containing its initial value, lets 
bus_match skip certain unwanted drivers (or match only whitelisted 
drivers).  Later, userspace writes into the extra sysfs attribute and 
into the standard driver core sysfs attributes to trigger the deferred 
driver probes.
-- 
Stefan Richter
-=====-==--- -==- =--=-
http://arcgraph.de/sr/

Re: Recommendation for activating a deferred module init in the kernel

[Johannes Stezenbach]

On Wed, Jun 18, 2008 at 12:48:27AM +0200, Stefan Richter wrote:
>>> On Tue, 17 June 2008 12:55:31 -0700, Tim Bird wrote:
> On Tue, 17 Jun 2008 11:28:29 -0700, Tim Bird wrote:
> | One of the main sub-systems that we defer initialization of this
> | way is USB, and this saves quite a bit of time.  (Of course the
> | same, or slightly more CPU cycles are eventually used during
> | bootup time.  But this lets us get to user space quicker so we
> | can start user-visible applications faster.)
>
> What if you don't defer module initialization, but merely device probing?
...
> If you set /sys/bus/foo/drivers_autoprobe to 0 (default is 1), then a  
> /sys/bus/foo/drivers/bar will not be bound to devices.  You can trigger  
> driver--device binding later per device by writing a device's bus ID  
> into /sys/bus/foo/drivers/bar/bind, or by writing into  
> /sys/bus/foo/drivers_probe (I guess; I only used the per-device way so 
> far).

I think the USB bus enumeration can take significant time:
recognize a device is connected, turn on bus power, try
to read descriptors (bus powered devices might be slow to
respond after power up). And this will happen even with
drivers_autoprobe == 0, right?
OTOH I think just calling the module init function when no
devices are present on the bus doesn't need much time.

If you could delay the enumeration it would not be neccessary
to mess with drivers_autoprobe. However, I don't know enough
about USB so I don't know how to do it...


Johannes

Re: Recommendation for activating a deferred module init in the kernel

[Stefan Richter]

Johannes Stezenbach wrote:
> I think the USB bus enumeration can take significant time:
> recognize a device is connected, turn on bus power, try
> to read descriptors (bus powered devices might be slow to
> respond after power up). And this will happen even with
> drivers_autoprobe == 0, right?

Probably... I don't know which particular steps happen in the USB core 
before upper layer drivers are bound.  Not binding the [eou]hci-hcd PCI 
driver would certainly be more effective.
-- 
Stefan Richter
-=====-==--- -==- =--=-
http://arcgraph.de/sr/

Re: Recommendation for activating a deferred module init in the kernel

[Johannes Stezenbach]

On Wed, Jun 18, 2008, Stefan Richter wrote:
> Johannes Stezenbach wrote:
>> I think the USB bus enumeration can take significant time:
>> recognize a device is connected, turn on bus power, try
>> to read descriptors (bus powered devices might be slow to
>> respond after power up). And this will happen even with
>> drivers_autoprobe == 0, right?
>
> Probably... I don't know which particular steps happen in the USB core  
> before upper layer drivers are bound.  Not binding the [eou]hci-hcd PCI  
> driver would certainly be more effective.

Well, in embedded systems you often don't have a PCI bus,
but platform devices. Maybe it's as simple as delaying
the USB platform_device_register() call, I don't know.

Johannes

Re: Recommendation for activating a deferred module init in the kernel

[Jörn Engel]

On Tue, 17 June 2008 12:52:22 -0700, Tim Bird wrote:
> Jörn Engel wrote:
> > On Tue, 17 June 2008 11:23:18 -0700, Tim Bird wrote:
> >> I'm not that happy using an ioctl for this trigger.  What is
> >> the preferred method of activating a kernel feature like this?
> >> I presume something in /proc or /sys, but I'm not sure.
> > 
> > I personally would be unhappy with any kind of interface for this.  It
> > would be much nicer to make it transparent and still get the benefits.
> > One option would be to start a kernel thread for the initialization and
> > renice it to 19 or so.
> 
> That's an interesting idea. I'm pretty sure the product guys want
> an explicit trigger, so they can make sure they've got the main
> application well underway before this deferred initialization occurs.

Well, there should be a way to ensure this doesn't hog the cpu at all -
unless it is idle or someone is actually waiting for the initialization
to finish.  Not sure if nice 19 is good enough for that.

> > If you want an explicit trigger, you could either hook into init_post()
> > or have hooks in the open functions of drivers with deferred
> > initialization.
> 
> This would presumably require multiple calls (one to the open of
> each deferred module).  I would still need a trigger for the memory
> free operation, unless I hardcode the order of the opening and just
> "know" that the last one should free the memory.  I'll have to see
> if all the modules being loaded like this have open()s.

If you want to keep things simple - and I believe initially you should -
you can simply do all initializations in one go.  Something like this:

int foo_open(...)
{
	wait_for_deferred_init();
	...
}

static DECLARE_COMPLETION(init_complete);

void wait_for_deferred_init(void)
{
	static atomic_t in_progress = ATOMIC_INIT(-1);

	if (!atomic_inc_not_zero(in_progress) {
		wait_for_completion(init_complete);
		return;
	}

	for (all deferred initcalls)
		foo_init();

	complete(init_complete);
	free_memory();
}

Jörn

-- 
Anything that can go wrong, will.
-- Finagle's Law

Re: Recommendation for activating a deferred module init in the kernel

[Tim Bird]

Jim Freeman wrote:
> Run modprobe?  Have it do just the _init bits without a load/link
> of the actual module text?

Interesting...  Maybe I could overload sys_init_module().  I'll take
a look at this.
  -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================