Recommendation for activating a deferred module init in the kernel

Recommendation for activating a deferred module init in the kernel

[Tim Bird]

Hi all,

I am working with a product team on bootup time issues.  One technique
that we are forward-porting from an old kernel (and that I thought I
might work on mainlining) is to compile modules statically into the kernel,
but defer their initialization until after boot time.

Normally, module init routines are declared with module_init(), which,
when they are statically linked into the kernel, uses the macro
__define_initcall() to add an entry to a special init segment.
This is called during bootup by do_initcall() (in init/main.c),
and the memory where the function resides is eventually freed.

There are several phases of initcalls (see include/linux/init.h),
including core, postcore, arch, subsys, fs, rootfs, device, and late.

In our modification, we:
 1) add another phase for deferred initcalls
 2) modify the module_init macro definition to use that phase, for the
 modules we wish to defer initialization of
 3) modify the free code to not free the memory until later
 4) add an ioctl to trigger the deferred initialization (and memory free)

One of the main sub-systems that we defer initialization of this
way is USB, and this saves quite a bit of time.  (Of course the
same, or slightly more CPU cycles are eventually used during
bootup time.  But this lets us get to user space quicker so we
can start user-visible applications faster.)

I'm not that happy using an ioctl for this trigger.  What is
the preferred method of activating a kernel feature like this?
I presume something in /proc or /sys, but I'm not sure.

Also, is there something like this in the kernel now that I'm
missing?

Any advice or comments would be welcome.

Thanks,
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: Recommendation for activating a deferred module init in the kernel

[David VomLehn]

Tim Bird wrote:
> Hi all,
> 
> I am working with a product team on bootup time issues.  One technique that we
> are forward-porting from an old kernel (and that I thought I might work on
> mainlining) is to compile modules statically into the kernel, but defer their
> initialization until after boot time.
...
> One of the main sub-systems that we defer initialization of this way is USB,
> and this saves quite a bit of time.  (Of course the same, or slightly more CPU
> cycles are eventually used during bootup time.  But this lets us get to user
> space quicker so we can start user-visible applications faster.)
> 
> I'm not that happy using an ioctl for this trigger.  What is the preferred
> method of activating a kernel feature like this? I presume something in /proc
> or /sys, but I'm not sure.

 From your description, it seems as though you always want to do the
initialization, you just may want to delay it until well after starting up user
space. If something like this were performance-critical, an ioctl might be 
justified, but this would be a one-time trigger. I think that making this as 
closely analogous to loading a kernel module as possible is a good conceptual 
approach. Since you might choose, at run time, whether or not to load a kernel 
module, it makes sense to do this on a device-by-device basis. To me, that 
suggests doing something in /sys.

My first impression was that this was an awful kludge, but drawing the parallel 
to loadable modules makes me not only happier, but also leads me into wondering 
if there shouldn't be a generic framework for supporting this. So, instead of 
using module_init, there might be some other macro that indicated that this 
driver was to be initialized in a deferred, and optional, fashion.

-- 
David VomLehn, dvomlehn@cisco.com
The opinions expressed herein are likely mine, but might not be my employer's...




     - - - - -                              Cisco                            - - - - -         
This e-mail and any attachments may contain information which is confidential, 
proprietary, privileged or otherwise protected by law. The information is solely 
intended for the named addressee (or a person responsible for delivering it to 
the addressee). If you are not the intended recipient of this message, you are 
not authorized to read, print, retain, copy or disseminate this message or any 
part of it. If you have received this e-mail in error, please notify the sender 
immediately by return e-mail and delete it from your computer.

Re: Recommendation for activating a deferred module init in the kernel

[Jörn Engel]

On Tue, 17 June 2008 11:23:18 -0700, Tim Bird wrote:
> 
> I'm not that happy using an ioctl for this trigger.  What is
> the preferred method of activating a kernel feature like this?
> I presume something in /proc or /sys, but I'm not sure.

I personally would be unhappy with any kind of interface for this.  It
would be much nicer to make it transparent and still get the benefits.
One option would be to start a kernel thread for the initialization and
renice it to 19 or so.

If you want an explicit trigger, you could either hook into init_post()
or have hooks in the open functions of drivers with deferred
initialization.  Obviously you need to wait for completion here anyway,
so adding a trigger wouldn't be too expensive.

Jörn

-- 
Joern's library part 13:
http://www.chip-architect.com/
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Recommendation for activating a deferred module init in the kernel

[Jim Freeman]

On Tue, Jun 17, 2008 at 09:07:51PM +0200, J??rn Engel wrote:
> On Tue, 17 June 2008 11:23:18 -0700, Tim Bird wrote:
> > 
> > I'm not that happy using an ioctl for this trigger.  What is
> > the preferred method of activating a kernel feature like this?
> > I presume something in /proc or /sys, but I'm not sure.
> 
> I personally would be unhappy with any kind of interface for this.  It
> would be much nicer to make it transparent and still get the benefits.
> One option would be to start a kernel thread for the initialization and
> renice it to 19 or so.
> 
> If you want an explicit trigger, you could either hook into init_post()
> or have hooks in the open functions of drivers with deferred
> initialization.  Obviously you need to wait for completion here anyway,
> so adding a trigger wouldn't be too expensive.

Run modprobe?  Have it do just the _init bits without a load/link
of the actual module text?


> J??rn
> 
> -- 
> Joern's library part 13:
> http://www.chip-architect.com/

Re: Recommendation for activating a deferred module init in the kernel

[Tim Bird]

Jörn Engel wrote:
> On Tue, 17 June 2008 11:23:18 -0700, Tim Bird wrote:
>> I'm not that happy using an ioctl for this trigger.  What is
>> the preferred method of activating a kernel feature like this?
>> I presume something in /proc or /sys, but I'm not sure.
> 
> I personally would be unhappy with any kind of interface for this.  It
> would be much nicer to make it transparent and still get the benefits.
> One option would be to start a kernel thread for the initialization and
> renice it to 19 or so.

That's an interesting idea. I'm pretty sure the product guys want
an explicit trigger, so they can make sure they've got the main
application well underway before this deferred initialization occurs.

> 
> If you want an explicit trigger, you could either hook into init_post()
> or have hooks in the open functions of drivers with deferred
> initialization.

This would presumably require multiple calls (one to the open of
each deferred module).  I would still need a trigger for the memory
free operation, unless I hardcode the order of the opening and just
"know" that the last one should free the memory.  I'll have to see
if all the modules being loaded like this have open()s.

Thanks for the ideas!
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Recommendation for activating a deferred module init in the kernel

[Tim Bird]

Tim Bird wrote:
>> If you want an explicit trigger, you could either hook into init_post()
>> or have hooks in the open functions of drivers with deferred
>> initialization.
> 
> This would presumably require multiple calls (one to the open of
> each deferred module).  I would still need a trigger for the memory
> free operation, unless I hardcode the order of the opening and just
> "know" that the last one should free the memory.  I'll have to see
> if all the modules being loaded like this have open()s.

Sorry - I responded too quickly.  I'm not sure I follow the
original suggestion.  How would I call the open function of
a module that is not initialized yet?
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: Recommendation for activating a deferred module init in the kernel

[Tim Bird]

Jim Freeman wrote:
> Run modprobe?  Have it do just the _init bits without a load/link
> of the actual module text?

Interesting...  Maybe I could overload sys_init_module().  I'll take
a look at this.
  -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: Recommendation for activating a deferred module init in the kernel

[Jörn Engel]

On Tue, 17 June 2008 12:52:22 -0700, Tim Bird wrote:
> Jörn Engel wrote:
> > On Tue, 17 June 2008 11:23:18 -0700, Tim Bird wrote:
> >> I'm not that happy using an ioctl for this trigger.  What is
> >> the preferred method of activating a kernel feature like this?
> >> I presume something in /proc or /sys, but I'm not sure.
> > 
> > I personally would be unhappy with any kind of interface for this.  It
> > would be much nicer to make it transparent and still get the benefits.
> > One option would be to start a kernel thread for the initialization and
> > renice it to 19 or so.
> 
> That's an interesting idea. I'm pretty sure the product guys want
> an explicit trigger, so they can make sure they've got the main
> application well underway before this deferred initialization occurs.

Well, there should be a way to ensure this doesn't hog the cpu at all -
unless it is idle or someone is actually waiting for the initialization
to finish.  Not sure if nice 19 is good enough for that.

> > If you want an explicit trigger, you could either hook into init_post()
> > or have hooks in the open functions of drivers with deferred
> > initialization.
> 
> This would presumably require multiple calls (one to the open of
> each deferred module).  I would still need a trigger for the memory
> free operation, unless I hardcode the order of the opening and just
> "know" that the last one should free the memory.  I'll have to see
> if all the modules being loaded like this have open()s.

If you want to keep things simple - and I believe initially you should -
you can simply do all initializations in one go.  Something like this:

int foo_open(...)
{
	wait_for_deferred_init();
	...
}

static DECLARE_COMPLETION(init_complete);

void wait_for_deferred_init(void)
{
	static atomic_t in_progress = ATOMIC_INIT(-1);

	if (!atomic_inc_not_zero(in_progress) {
		wait_for_completion(init_complete);
		return;
	}

	for (all deferred initcalls)
		foo_init();

	complete(init_complete);
	free_memory();
}

Jörn

-- 
Anything that can go wrong, will.
-- Finagle's Law
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Recommendation for activating a deferred module init in the kernel

[Jörn Engel]

On Tue, 17 June 2008 12:55:31 -0700, Tim Bird wrote:
> 
> Sorry - I responded too quickly.  I'm not sure I follow the
> original suggestion.  How would I call the open function of
> a module that is not initialized yet?

Hmm, good point.  I guess that suggestion has just failed the reality
test.

Jörn

-- 
You ain't got no problem, Jules. I'm on the motherfucker. Go back in
there, chill them niggers out and wait for the Wolf, who should be
coming directly.
-- Marsellus Wallace
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Recommendation for activating a deferred module init in the kernel

[Josh Boyer]

On Tue, 17 Jun 2008 22:23:19 +0200
Jörn Engel <joern@logfs.org> wrote:

> On Tue, 17 June 2008 12:55:31 -0700, Tim Bird wrote:
> > 
> > Sorry - I responded too quickly.  I'm not sure I follow the
> > original suggestion.  How would I call the open function of
> > a module that is not initialized yet?
> 
> Hmm, good point.  I guess that suggestion has just failed the reality
> test.

Have a simple module that calls the initialization functions for
anything with a deferred init.  Then you can open that and use an ioctl
to...

I think if we try really really hard, we can make this a Rube-Goldberg
Machine!

josh
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Recommendation for activating a deferred module init in the kernel

[Stefan Richter]

>> On Tue, 17 June 2008 12:55:31 -0700, Tim Bird wrote:
>>> Sorry - I responded too quickly.  I'm not sure I follow the
>>> original suggestion.  How would I call the open function of
>>> a module that is not initialized yet?

You will be able to open a character device file as soon as cdev_init() 
finished... when the Big Kernel Lock is not being taken around 
file_operations.open() anymore.

On Tue, 17 Jun 2008 11:28:29 -0700, Tim Bird wrote:
| One technique
| that we are forward-porting from an old kernel (and that I thought I
| might work on mainlining) is to compile modules statically into the
| kernel, but defer their initialization until after boot time.
...
| One of the main sub-systems that we defer initialization of this
| way is USB, and this saves quite a bit of time.  (Of course the
| same, or slightly more CPU cycles are eventually used during
| bootup time.  But this lets us get to user space quicker so we
| can start user-visible applications faster.)

What if you don't defer module initialization, but merely device probing?

$ ls /sys/bus/*/drivers_autoprobe
/sys/bus/acpi/drivers_autoprobe
/sys/bus/firewire/drivers_autoprobe
/sys/bus/i2c/drivers_autoprobe
/sys/bus/ide/drivers_autoprobe
/sys/bus/pci/drivers_autoprobe
/sys/bus/pci_express/drivers_autoprobe
/sys/bus/platform/drivers_autoprobe
/sys/bus/pnp/drivers_autoprobe
/sys/bus/scsi/drivers_autoprobe
/sys/bus/serio/drivers_autoprobe
/sys/bus/usb/drivers_autoprobe

If you set /sys/bus/foo/drivers_autoprobe to 0 (default is 1), then a 
/sys/bus/foo/drivers/bar will not be bound to devices.  You can trigger 
driver--device binding later per device by writing a device's bus ID 
into /sys/bus/foo/drivers/bar/bind, or by writing into 
/sys/bus/foo/drivers_probe (I guess; I only used the per-device way so far).

Now, since you want to do this with statically linked drivers, i.e. need 
to prevent probing before userspace and sysfs are up and running, you 
probably need to modify the bus types or/and the driver core so that the 
the less vital buses have drivers_autoprobe off by default.

On the other hand, maybe you want to have probes of some PCI drivers 
executed but not the probes of some other PCI drivers; uhci_hcd for 
example.  I guess you could achieve that by modifying 
drivers/pci/pci-driver.c::pci_bus_match().  E.g. add a sysfs attribute 
to pci-driver.c which, as long as containing its initial value, lets 
bus_match skip certain unwanted drivers (or match only whitelisted 
drivers).  Later, userspace writes into the extra sysfs attribute and 
into the standard driver core sysfs attributes to trigger the deferred 
driver probes.
-- 
Stefan Richter
-=====-==--- -==- =--=-
http://arcgraph.de/sr/

Re: Recommendation for activating a deferred module init in the kernel

[Johannes Stezenbach]

On Wed, Jun 18, 2008 at 12:48:27AM +0200, Stefan Richter wrote:
>>> On Tue, 17 June 2008 12:55:31 -0700, Tim Bird wrote:
> On Tue, 17 Jun 2008 11:28:29 -0700, Tim Bird wrote:
> | One of the main sub-systems that we defer initialization of this
> | way is USB, and this saves quite a bit of time.  (Of course the
> | same, or slightly more CPU cycles are eventually used during
> | bootup time.  But this lets us get to user space quicker so we
> | can start user-visible applications faster.)
>
> What if you don't defer module initialization, but merely device probing?
...
> If you set /sys/bus/foo/drivers_autoprobe to 0 (default is 1), then a  
> /sys/bus/foo/drivers/bar will not be bound to devices.  You can trigger  
> driver--device binding later per device by writing a device's bus ID  
> into /sys/bus/foo/drivers/bar/bind, or by writing into  
> /sys/bus/foo/drivers_probe (I guess; I only used the per-device way so 
> far).

I think the USB bus enumeration can take significant time:
recognize a device is connected, turn on bus power, try
to read descriptors (bus powered devices might be slow to
respond after power up). And this will happen even with
drivers_autoprobe == 0, right?
OTOH I think just calling the module init function when no
devices are present on the bus doesn't need much time.

If you could delay the enumeration it would not be neccessary
to mess with drivers_autoprobe. However, I don't know enough
about USB so I don't know how to do it...


Johannes

Re: Recommendation for activating a deferred module init in the kernel

[Stefan Richter]

Johannes Stezenbach wrote:
> I think the USB bus enumeration can take significant time:
> recognize a device is connected, turn on bus power, try
> to read descriptors (bus powered devices might be slow to
> respond after power up). And this will happen even with
> drivers_autoprobe == 0, right?

Probably... I don't know which particular steps happen in the USB core 
before upper layer drivers are bound.  Not binding the [eou]hci-hcd PCI 
driver would certainly be more effective.
-- 
Stefan Richter
-=====-==--- -==- =--=-
http://arcgraph.de/sr/

Re: Recommendation for activating a deferred module init in the kernel

[Gilad Ben-Yossef]

[Resending due to reject from vger mail server. My apologies if you got 
this twice ]


Hi,


Tim Bird wrote:

> I am working with a product team on bootup time issues.  One technique
> that we are forward-porting from an old kernel (and that I thought I
> might work on mainlining) is to compile modules statically into the kernel,
> but defer their initialization until after boot time.
>   
This may sound like a stupid question, but why are you compiling the 
modules statically?

I mean, it sounds to me like the perfect way to do what you want is to 
compile the modules dynamically (you can store them in a an in kernel 
initramfs if you want to keep them attached to the kernel binary for 
ease of maintenance) and simply call a script that calls modprobe when 
it's OK to load them?

Sounds a hell of lot simpler to me...

Gilad

-- 
Gilad Ben-Yossef 
Chief Coffee Drinker

Codefidence Ltd.
The code is free, your time isn't.(TM)

Web:    http://codefidence.com
Email:  gilad@codefidence.com
Office: +972-8-9316883 ext. 201
Fax:    +972-8-9316885
Mobile: +972-52-8260388

	Q: How many NSA agents does it take to replace a lightbulb?
	A: dSva7DrYiY24yeTItKyyogFXD5gRuoRqPNQ9v6WCLLywZPINlu!

Re: Recommendation for activating a deferred module init in the kernel

[David Woodhouse]

On Wed, 2008-06-18 at 09:47 +0300, Gilad Ben-Yossef wrote:
> This may sound like a stupid question, but why are you compiling the 
> modules statically?

I wondered that.

One potential reason to avoid modules is that they waste RAM -- you have
to allocate an integral number of pages for each one, which means an
average of 2KiB wasted for each module you load. Although that isn't
much, it's not zero either. It might be possible to optimise that by
'packing' module allocations, if you're careful about it.

Also, on some architectures modules have historically been less
efficient because when kernel and module text are too far apart in the
virtual address space, you have to insert trampolines for all calls
between them. I believe this used to be the case on ARM, but it got
fixed by moving stuff around? It's only v850 which still does this
AFAICT.

Do we actually manage to free inittext/initdata from modules on all
architectures now? The comments in <linux/init.h> suggest not, but I
think they lie -- it looks like like kernel/module.c will handle that
entirely generically for sections named ".init*" -- it'll allocate space
for those sections in a separate module_alloc() call, and free them
itself. (Do bear this in mind, if you take the above suggestion about
packing in module_alloc().)

So the only real reason I can see to avoid modules in the _current_
kernel would be the wasted RAM, which should be something we can
address. Tim, have I missed something? What _were_ your reasons for
avoiding modules, and can we do something about those instead of trying
to defer the initialisation of in-kernel code?

Delaying init of certain modules seems like a poor man's substitute for
a properly multi-threaded startup....

-- 
dwmw2

Re: Recommendation for activating a deferred module init in the kernel

[David Woodhouse]

On Wed, 2008-06-18 at 09:20 +0100, David Woodhouse wrote:
> So the only real reason I can see to avoid modules in the _current_
> kernel would be the wasted RAM, which should be something we can
> address. Tim, have I missed something? 

... like the time it takes to actually load modules and do the
relocations, perhaps? You did say you were doing it to improve boot
time. But this is the stuff you were happy to _defer_, so presumably not
in your fast path?

If that _is_ a real concern, then given the observation¹ that most
systems load exactly the same modules in exactly the same order every
time they boot... has anyone looked at doing 'prelink' for modules? 

-- 
dwmw2

¹ which I just pulled out of my arse, admittedly. But I think it's true :)

--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Recommendation for activating a deferred module init in the kernel

[Adrian Bunk]

On Wed, Jun 18, 2008 at 09:20:22AM +0100, David Woodhouse wrote:
> On Wed, 2008-06-18 at 09:47 +0300, Gilad Ben-Yossef wrote:
> > This may sound like a stupid question, but why are you compiling the 
> > modules statically?
> 
> I wondered that.
> 
> One potential reason to avoid modules is that they waste RAM -- you have
> to allocate an integral number of pages for each one, which means an
> average of 2KiB wasted for each module you load. Although that isn't
> much, it's not zero either. It might be possible to optimise that by
> 'packing' module allocations, if you're careful about it.
>...
> So the only real reason I can see to avoid modules in the _current_
> kernel would be the wasted RAM, which should be something we can
> address.
>...

You miss the size increase imposed by CONFIG_MODULES=y.

E.g. setting CONFIG_MODULES=y in the arm collie_defconfig will
increase the size of vmlinux by 14% (sic).

I haven't investigated why it takes that much space, but stuff like 
kernel/module.o taking 23kB and each EXPORT_SYMBOL requiring a few
bytes simply cannot be completely eliminated.

> dwmw2

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Recommendation for activating a deferred module init in the kernel

[Geert Uytterhoeven]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1804 bytes --]

On Wed, 18 Jun 2008, Adrian Bunk wrote:
> On Wed, Jun 18, 2008 at 09:20:22AM +0100, David Woodhouse wrote:
> > On Wed, 2008-06-18 at 09:47 +0300, Gilad Ben-Yossef wrote:
> > > This may sound like a stupid question, but why are you compiling the 
> > > modules statically?
> > 
> > I wondered that.
> > 
> > One potential reason to avoid modules is that they waste RAM -- you have
> > to allocate an integral number of pages for each one, which means an
> > average of 2KiB wasted for each module you load. Although that isn't
> > much, it's not zero either. It might be possible to optimise that by
> > 'packing' module allocations, if you're careful about it.
> >...
> > So the only real reason I can see to avoid modules in the _current_
> > kernel would be the wasted RAM, which should be something we can
> > address.
> >...
> 
> You miss the size increase imposed by CONFIG_MODULES=y.
> 
> E.g. setting CONFIG_MODULES=y in the arm collie_defconfig will
> increase the size of vmlinux by 14% (sic).
> 
> I haven't investigated why it takes that much space, but stuff like 
> kernel/module.o taking 23kB and each EXPORT_SYMBOL requiring a few
> bytes simply cannot be completely eliminated.

Sounds like we need a tool that strips out the unneeded symbols, given a list
of modules?

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Techsoft Centre
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453
Fax:      +32 (0)2 700 8622
E-mail:   Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/

Sony Technology and Software Centre Europe
A division of Sony Service Centre (Europe) N.V.
Registered office: Technologielaan 7 · B-1840 Londerzeel · Belgium
VAT BE 0413.825.160 · RPR Brussels
Fortis 293-0376800-10 GEBA-BE-BB

Re: Recommendation for activating a deferred module init in the kernel

[Johannes Stezenbach]

On Wed, Jun 18, 2008, Stefan Richter wrote:
> Johannes Stezenbach wrote:
>> I think the USB bus enumeration can take significant time:
>> recognize a device is connected, turn on bus power, try
>> to read descriptors (bus powered devices might be slow to
>> respond after power up). And this will happen even with
>> drivers_autoprobe == 0, right?
>
> Probably... I don't know which particular steps happen in the USB core  
> before upper layer drivers are bound.  Not binding the [eou]hci-hcd PCI  
> driver would certainly be more effective.

Well, in embedded systems you often don't have a PCI bus,
but platform devices. Maybe it's as simple as delaying
the USB platform_device_register() call, I don't know.

Johannes

Re: Recommendation for activating a deferred module init in the kernel

[David Woodhouse]

On Wed, 2008-06-18 at 10:57 +0200, Geert Uytterhoeven wrote:
> On Wed, 18 Jun 2008, Adrian Bunk wrote:
> > On Wed, Jun 18, 2008 at 09:20:22AM +0100, David Woodhouse wrote:
> > > On Wed, 2008-06-18 at 09:47 +0300, Gilad Ben-Yossef wrote:
> > > > This may sound like a stupid question, but why are you compiling the 
> > > > modules statically?
> > > 
> > > I wondered that.
> > > 
> > > One potential reason to avoid modules is that they waste RAM -- you have
> > > to allocate an integral number of pages for each one, which means an
> > > average of 2KiB wasted for each module you load. Although that isn't
> > > much, it's not zero either. It might be possible to optimise that by
> > > 'packing' module allocations, if you're careful about it.
> > >...
> > > So the only real reason I can see to avoid modules in the _current_
> > > kernel would be the wasted RAM, which should be something we can
> > > address.
> > >...
> > 
> > You miss the size increase imposed by CONFIG_MODULES=y.
> > 
> > E.g. setting CONFIG_MODULES=y in the arm collie_defconfig will
> > increase the size of vmlinux by 14% (sic).
> > 
> > I haven't investigated why it takes that much space, but stuff like 
> > kernel/module.o taking 23kB and each EXPORT_SYMBOL requiring a few
> > bytes simply cannot be completely eliminated.
> 
> Sounds like we need a tool that strips out the unneeded symbols, given a list
> of modules?

And do the --gc-sections step again after that... :)

-- 
dwmw2

Re: Recommendation for activating a deferred module init in the kernel

[Adrian Bunk]

On Wed, Jun 18, 2008 at 10:59:50AM +0100, David Woodhouse wrote:
> On Wed, 2008-06-18 at 10:57 +0200, Geert Uytterhoeven wrote:
> > On Wed, 18 Jun 2008, Adrian Bunk wrote:
> > > On Wed, Jun 18, 2008 at 09:20:22AM +0100, David Woodhouse wrote:
> > > > On Wed, 2008-06-18 at 09:47 +0300, Gilad Ben-Yossef wrote:
> > > > > This may sound like a stupid question, but why are you compiling the 
> > > > > modules statically?
> > > > 
> > > > I wondered that.
> > > > 
> > > > One potential reason to avoid modules is that they waste RAM -- you have
> > > > to allocate an integral number of pages for each one, which means an
> > > > average of 2KiB wasted for each module you load. Although that isn't
> > > > much, it's not zero either. It might be possible to optimise that by
> > > > 'packing' module allocations, if you're careful about it.
> > > >...
> > > > So the only real reason I can see to avoid modules in the _current_
> > > > kernel would be the wasted RAM, which should be something we can
> > > > address.
> > > >...
> > > 
> > > You miss the size increase imposed by CONFIG_MODULES=y.
> > > 
> > > E.g. setting CONFIG_MODULES=y in the arm collie_defconfig will
> > > increase the size of vmlinux by 14% (sic).
> > > 
> > > I haven't investigated why it takes that much space, but stuff like 
> > > kernel/module.o taking 23kB and each EXPORT_SYMBOL requiring a few
> > > bytes simply cannot be completely eliminated.
> > 
> > Sounds like we need a tool that strips out the unneeded symbols, given a list
> > of modules?
> 
> And do the --gc-sections step again after that... :)

But even after all optimizations CONFIG_MODULES=y will still cause a 
significant additional cost [1] when thinking in the dimensions of 
Tim's "the 30 or so Linux-tiny patches that I use get me about 110k of 
reductions.  For me, this is about 5% of my kernel size" statement in 
another thread.

> dwmw2

cu
Adrian

[1] as I said, kernel/module.o alone takes 23kB

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Recommendation for activating a deferred module init in the kernel

[David Woodhouse]

On Wed, 2008-06-18 at 13:33 +0300, Adrian Bunk wrote:
> But even after all optimizations CONFIG_MODULES=y will still cause a 
> significant additional cost [1] when thinking in the dimensions of 
> Tim's "the 30 or so Linux-tiny patches that I use get me about 110k of 
> reductions.  For me, this is about 5% of my kernel size" statement in 
> another thread.

Yes. If the system in question is currently being built without
CONFIG_MODULES, that's fairly much a no-go for my alternative
suggestion.

-- 
dwmw2

Re: Recommendation for activating a deferred module init in the kernel

[Geert Uytterhoeven]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1742 bytes --]

On Wed, 18 Jun 2008, Adrian Bunk wrote:
> On Wed, Jun 18, 2008 at 10:59:50AM +0100, David Woodhouse wrote:
> > On Wed, 2008-06-18 at 10:57 +0200, Geert Uytterhoeven wrote:
> > > On Wed, 18 Jun 2008, Adrian Bunk wrote:
> > > > You miss the size increase imposed by CONFIG_MODULES=y.
> > > > 
> > > > E.g. setting CONFIG_MODULES=y in the arm collie_defconfig will
> > > > increase the size of vmlinux by 14% (sic).
> > > > 
> > > > I haven't investigated why it takes that much space, but stuff like 
> > > > kernel/module.o taking 23kB and each EXPORT_SYMBOL requiring a few
> > > > bytes simply cannot be completely eliminated.
> > > 
> > > Sounds like we need a tool that strips out the unneeded symbols, given a list
> > > of modules?
> > 
> > And do the --gc-sections step again after that... :)
> 
> But even after all optimizations CONFIG_MODULES=y will still cause a 
> significant additional cost [1] when thinking in the dimensions of 
> Tim's "the 30 or so Linux-tiny patches that I use get me about 110k of 
> reductions.  For me, this is about 5% of my kernel size" statement in 
> another thread.
> 
> [1] as I said, kernel/module.o alone takes 23kB

Which could be freed after the last module has been loaded?

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Techsoft Centre
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453
Fax:      +32 (0)2 700 8622
E-mail:   Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/

Sony Technology and Software Centre Europe
A division of Sony Service Centre (Europe) N.V.
Registered office: Technologielaan 7 · B-1840 Londerzeel · Belgium
VAT BE 0413.825.160 · RPR Brussels
Fortis 293-0376800-10 GEBA-BE-BB

Re: Recommendation for activating a deferred module init in the kernel

[Amol Lad]

>
> I am working with a product team on bootup time issues.  One technique
> that we are forward-porting from an old kernel (and that I thought I
> might work on mainlining) is to compile modules statically into the kernel,
> but defer their initialization until after boot time.


I landed in same problem sometime back with USB and tried two approaches

1. Dirty approach of deferring loading of all usb modules till you see
application startup "from user point of view" is done. I then just
call into a script which loads all the usb modules "in parallel" with
rest of app initialization

2. The usb modules that I load are usbcore, host controller driver,
scsi_mod, sd_mod and usb_storage. I noted that it's actually scsi
enumeration that takes a long time; usb enumeration was much faster. I
compiled usbcore and host controller driver statically in the kernel,
enabled CONFIG_HOTPLUG and did rest of the module loading in a script
invoked from /sbin/hotplug.

Amol

Re: Recommendation for activating a deferred module init in the kernel

[Nicolas Pitre]

On Wed, 18 Jun 2008, Geert Uytterhoeven wrote:

> On Wed, 18 Jun 2008, Adrian Bunk wrote:
> > On Wed, Jun 18, 2008 at 10:59:50AM +0100, David Woodhouse wrote:
> > > On Wed, 2008-06-18 at 10:57 +0200, Geert Uytterhoeven wrote:
> > > > On Wed, 18 Jun 2008, Adrian Bunk wrote:
> > > > > You miss the size increase imposed by CONFIG_MODULES=y.
> > > > > 
> > > > > E.g. setting CONFIG_MODULES=y in the arm collie_defconfig will
> > > > > increase the size of vmlinux by 14% (sic).
> > > > > 
> > > > > I haven't investigated why it takes that much space, but stuff like 
> > > > > kernel/module.o taking 23kB and each EXPORT_SYMBOL requiring a few
> > > > > bytes simply cannot be completely eliminated.
> > > > 
> > > > Sounds like we need a tool that strips out the unneeded symbols, given a list
> > > > of modules?
> > > 
> > > And do the --gc-sections step again after that... :)
> > 
> > But even after all optimizations CONFIG_MODULES=y will still cause a 
> > significant additional cost [1] when thinking in the dimensions of 
> > Tim's "the 30 or so Linux-tiny patches that I use get me about 110k of 
> > reductions.  For me, this is about 5% of my kernel size" statement in 
> > another thread.
> > 
> > [1] as I said, kernel/module.o alone takes 23kB
> 
> Which could be freed after the last module has been loaded?

Even if you free the module loading infrastructure afterwards, you still 
have to carry it in flash for both the kernel part and the user part.  
And the whole process (having to sort out module loading in user space, 
etc.) is still more complex than not having to load modules at all. Many 
embedded designs have fixed hardware configuration and purpose which 
makes dynamic module loading an unnecessary bloat.

Having the kernel span a parallel thread with a lower priority to 
initialize drivers could be a much simpler solution in terms of size and 
speed.  This way the init process could run early, and devices would 
become available eventually just as if corresponding modules were 
loaded.

The only difficulty is to properly determine how to distinguish between 
drivers that have to be initialized early in order to mount the root fs 
and the others that can wait.


Nicolas

Re: Recommendation for activating a deferred module init in the kernel

[Tim Bird]

Gilad Ben-Yossef wrote:
> This may sound like a stupid question, but why are you compiling the
> modules statically?
> 
> I mean, it sounds to me like the perfect way to do what you want is to
> compile the modules dynamically (you can store them in a an in kernel
> initramfs if you want to keep them attached to the kernel binary for
> ease of maintenance) and simply call a script that calls modprobe when
> it's OK to load them?

I'm not positive, but I think we link them statically for size and
time reasons.  Statically linking the modules means we can eliminate
the module load code from the kernel, and also save ourselves the ELF
sections (in each module) which hold relocation information, etc.
Also, insmod-ing (or modprobe-ing) the modules takes extra time
overhead for the symbol relocation step.  Since this would be
deferred in our case, this might not be a big deal, but it does
add to the total length of time required to boot.

> Sounds a hell of lot simpler to me...
Our patch isn't that big or complicated now.  I just didn't like
the ioctl().  Replacing the ioctl with some other lightweight
mechanism shouldn't complicate the code too much.

I'm not sure requiring a shell interpreter (to run the script)
a script, a program (modprobe) and a whole new set of memory
operations (to handle the initramfs), is simpler than a single
ioctl. (It might be, in the non-embedded case, where the
presence of these things is taken for granted.)

 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: Recommendation for activating a deferred module init in the kernel

[Tim Bird]

Gilad Ben-Yossef wrote:
> Well, seeing as both modprobe and a minimal shell are part of busybox
> which is included in over 90%+ of Linux based embedded systems and that
> the script is trivial, not to mention that you can just have the
> application call modprobe directly, just as it will be calling ioctl()
> in your case, thereby negating the need for both script and shell at
> all, I do believe that complexity wise my solution still has some merit.

I agree.  When you say "have the application call modprobe directly",
I'm not sure I understand what you mean.  Are you talking about a call
to the kernel (a syscall) or a library function?  The kernel has the
syscall sys_init_module(), which I'm considering using.  Is there some
mobprobe library call that might make sense to use?

Thanks for the feedback.
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: Recommendation for activating a deferred module init in the kernel

[Gilad Ben-Yossef]

Tim Bird wrote:

> Gilad Ben-Yossef wrote:
>   
>> Well, seeing as both modprobe and a minimal shell are part of busybox
>> which is included in over 90%+ of Linux based embedded systems and that
>> the script is trivial, not to mention that you can just have the
>> application call modprobe directly, just as it will be calling ioctl()
>> in your case, thereby negating the need for both script and shell at
>> all, I do believe that complexity wise my solution still has some merit.
>>     
>
> I agree.  When you say "have the application call modprobe directly",
> I'm not sure I understand what you mean.  Are you talking about a call
> to the kernel (a syscall) or a library function?  The kernel has the
> syscall sys_init_module(), which I'm considering using.  Is there some
> mobprobe library call that might make sense to use?
>
>   
I simply meant that you can fork and exec modprobe itself (or use 
system() but that
would require a working shell). This would "save" the need for a 
separate script and a shell.

I guess "invoking" would have been more suitable word then "calling".

The only downside I see of calling the sys_init_module syscall directly 
is that it
doesn't do any of the dependency tracking that modprobe does, so it's more
a insmod replacement then a modprobe one, but I doubt this matters at 
all in an
embedded system anyway.


Do people here think a shared library implementing modprobe would be 
useful?

I ran into such a need a couple of times myself and does not look 
difficult to do, but does
anyone else here thinks it will be useful?

Gilad

-- 
Gilad Ben-Yossef 
Chief Coffee Drinker

Codefidence Ltd.
The code is free, your time isn't.(TM)

Web:    http://codefidence.com
Email:  gilad@codefidence.com
Office: +972-8-9316883 ext. 201
Fax:    +972-8-9316885
Mobile: +972-52-8260388

	Q: How many NSA agents does it take to replace a lightbulb?
	A: dSva7DrYiY24yeTItKyyogFXD5gRuoRqPNQ9v6WCLLywZPINlu!

Re: Recommendation for activating a deferred module init in the kernel

[Tim Bird]

Gilad Ben-Yossef wrote:
> Tim Bird wrote:
>> I agree.  When you say "have the application call modprobe directly",
>> I'm not sure I understand what you mean.
>
> I simply meant that you can fork and exec modprobe itself (or use
> system() but that
> would require a working shell). This would "save" the need for a
> separate script and a shell.

Well, this would explain why I didn't follow your original
point.  I thought you were using the word "modprobe" as a placeholder
for some other module-installation-related concept.  In all
my years of working with embedded Linux, I have never used
modprobe in a target device.  (And I avoid insmod whenever I can).
Sorry for my confusion.

> The only downside I see of calling the sys_init_module syscall directly
> is that it
> doesn't do any of the dependency tracking that modprobe does, so it's more
> a insmod replacement then a modprobe one, but I doubt this matters at
> all in an
> embedded system anyway.
It may just be my own blind spot, but I can't think of a good
reason to do such dependency tracking in an embedded device.
It is a sad state of affairs if the product developers don't
know the module dependencies for their own products.

> 
> Do people here think a shared library implementing modprobe would be
> useful?
Speaking from my own experience, not for embedded.

 -- Tim


=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: Recommendation for activating a deferred module init in the kernel

[Gilad Ben-Yossef]

Tim Bird wrote:

> Gilad Ben-Yossef wrote:
>   
>> Tim Bird wrote:
>>     
>>> I agree.  When you say "have the application call modprobe directly",
>>> I'm not sure I understand what you mean.
>>>       
>> I simply meant that you can fork and exec modprobe itself (or use
>> system() but that
>> would require a working shell). This would "save" the need for a
>> separate script and a shell.
>>     
>
> Well, this would explain why I didn't follow your original
> point.  I thought you were using the word "modprobe" as a placeholder
> for some other module-installation-related concept.  

Well, modprobe could just as well be insmod, but the basic idea is still 
the same.
>   
>> The only downside I see of calling the sys_init_module syscall directly
>> is that it
>> doesn't do any of the dependency tracking that modprobe does, so it's more
>> a insmod replacement then a modprobe one, but I doubt this matters at
>> all in an
>> embedded system anyway.
>>     
> It may just be my own blind spot, but I can't think of a good
> reason to do such dependency tracking in an embedded device.
> It is a sad state of affairs if the product developers don't
> know the module dependencies for their own products.
>   
For custom built modules written in house , I agree - but think of one 
of those devices that has an OTG USB port, the kind that you can one day 
connect it to a computer to play the part of a USB storage gadget and 
the next day hook a  it to up to a serial USB connected GSM modem.

It's crazy to have all that functionality loaded into the device RAM at 
all times and the using modprobe to track the dependencies of the module 
makes just as much sense as it does for your laptop.

I guess the term "embedded" gets kind of blurry with these things, but I 
also think Linux adoption in the embedded world is driven by this trend.


Cheers,
Gilad

-- 
Gilad Ben-Yossef 
Chief Coffee Drinker

Codefidence Ltd.
The code is free, your time isn't.(TM)

Web:    http://codefidence.com
Email:  gilad@codefidence.com
Office: +972-8-9316883 ext. 201
Fax:    +972-8-9316885
Mobile: +972-52-8260388

	Q: How many NSA agents does it take to replace a lightbulb?
	A: dSva7DrYiY24yeTItKyyogFXD5gRuoRqPNQ9v6WCLLywZPINlu!