[PATCH] PART1: Proposed init & module changes for 2.5

[PATCH] PART1: Proposed init & module changes for 2.5

[Rusty Russell]

Highlights:
	o Unified boot/module parameters, with type checking.
		Using PARAM(xxx, type, perms) and family.

	o Simpler in-kernel module loading
		insmod can now be written in ~20 lines.

	o Nicer internal module interface
		request_module("%s", foo)
		request_module_start/stop wrappers
		module_get()/module_put(), symbol_get()/symbol_put()

	o Warm fuzzy bleeding edge feel

Todo:
	(1) Incorporate non-ppc architectures (x86, sparc, sparc64,
	    ppc64 already written, not in patch).

	(2) Convert parts of the kernel *I* don't use.

	(3) Use kbuild 2.5, so in-kernel params don't all get called
	    "unknown.xxx".

	(4) Update documentation.

	(5) CONFIG_MODVERSIONS replacement and check for kernel version.

	(6) Patch in proc/sys, so param entries can appear there
	    (this is what the final arg to PARAM is for).

	(7) Write more sophisticated modprobe:
		o /etc/modules.conf
		o Fallback to old modutils if old system.
		o --force to override MODVERSIONS and kernel version.
		o Save parameters from /proc on unloading.

	(8) rmmod should not remove if busy (unless rmmod -f).

	(9) Convert everyone to new safe module insert usage.

Get it from:
	http://netfilter.samba.org/diary/module-init-patch-001.bz2
	http://netfilter.filewatcher.org/diary/module-init-patch-001.bz2
	http://netfilter.gnumonks.org/diary/module-init-patch-001.bz2
(46885 bytes)

And v. v. simple insmod/rmmod/modprobe from:
	http://netfilter.samba.org/diary/module-init-tools-001.bz2
	http://netfilter.filewatcher.org/diary/module-init-tools-001.bz2
	http://netfilter.gnumonks.org/diary/module-init-tools-001.bz2
(33181 bytes)

HACKERS QUICKREF:

OLD:	int parse_foo(char *s) { ... return 1; }
	__setup("foo=", parse_foo);
NEW:	int parse_foo(char *s, struct kernel_param *kp) { ... return 0; }
	PARAM_CALL(foo, parse_foo, NULL);
COMMENTS:
	This will match "foo=" only.  kb->name == "foo", kp->arg ==
	third arg of PARAM_CALL.

OLD:	MODULE_PARM("foo", &foo, "i")
NEW:	PARAM(foo, int, S_IRUGO);
COMMENTS:
	foo must be in scope (and in this case, of type int) or the
	compiler will catch it.
	S_IRUGO means that it will be accessible through proc (use 000
	if this is not useful, or maybe S_IWUSR if root can change it
	at any time).
	Use PARAM_NAMED to give a different name, PARAM_ARRAY for array.

OLD:	inter_module_*
NEW:	EXPORT_SYMBOL, symbol_get(foo) & symbol_put(foo)
COMMENTS:
	These are now typesafe, so foo declaration must be in scope.

OLD:	if (foo->owner) __MOD_INC_USE_COUNT(foo->owner);
NEW:	module_get(foo->owner);

OLD:	module_init(initfn)
NEW:	init_and_startcall(initfn, startfn);
COMMENTS:
	Please use a semicolon at the end.  Modulable code should
	transition to a two-stage init (initfn sets everything up
	and may fail, and startfn which exposes the module to the rest
	of the kernel and can't fail).  Some modules only need
	initfn or startfn, in which case use initcall() or startcall().

OLD:	if (try_inc_mod_count(foo->owner))
NEW:	module_get(foo->owner)

OLD:	int initfn(void) { ...; return 0; }
	module_init(initfn)
NEW:	void initfn(void) { ...; }
	bootcall(initfn);
COMMENTS:
	This is for code which can never be a module: we ignore the
	return value anyway, so why have one (panic() if you must).

OLD:	void exitfn(void) { ...; }
	module_exit(void)
NEW:	int stopfn(void) { ...; return 0; }
	void exitfn(void) { ...; }
	stopcall(stopfn);
	exitcall(exitfn);
COMMENTS:
	If there are neither, then module is not unloadable.  This is
	perfectly OK.  If stopfn returns 0 (otherwise it should be
	-errno) it must have deregistered itself from the rest of the
	kernel (ie. module count can never increase again), but still
	be usable to anyone using it currently.  Once exitfn is
	called, it is guaranteed to be unused.

Enjoy!
Rusty.
--
Premature optmztion is rt of all evl. --DK

Re: [PATCH] PART1: Proposed init & module changes for 2.5

[David Cinege]

On Sunday 23 September 2001 2:37, Rusty Russell wrote:

Russ,

How about implementing MBS? (Bootloader module loading. IE as implmented in 
GRUB) so we can finally have completely modular kernels?

Dave

Re: [PATCH] PART1: Proposed init & module changes for 2.5

[Ingo Oeser]

Hi,

I like the PARAM() part, but still have some question. Hope you
can clarify.

On Sun, Sep 23, 2001 at 04:37:43PM +1000, Rusty Russell wrote:
> OLD:	module_init(initfn)
> NEW:	init_and_startcall(initfn, startfn);
> COMMENTS:
> 	Please use a semicolon at the end.  Modulable code should
> 	transition to a two-stage init (initfn sets everything up
> 	and may fail, and startfn which exposes the module to the rest
> 	of the kernel and can't fail).  Some modules only need
> 	initfn or startfn, in which case use initcall() or startcall().
 
Why separating them?

> OLD:	void exitfn(void) { ...; }
> 	module_exit(void)
> NEW:	int stopfn(void) { ...; return 0; }
> 	void exitfn(void) { ...; }
> 	stopcall(stopfn);
> 	exitcall(exitfn);
> COMMENTS:
> 	If there are neither, then module is not unloadable.  This is
> 	perfectly OK.  If stopfn returns 0 (otherwise it should be
> 	-errno) it must have deregistered itself from the rest of the
> 	kernel (ie. module count can never increase again), but still
> 	be usable to anyone using it currently.  Once exitfn is
> 	called, it is guaranteed to be unused.

Same question here: Why you changed this? 

What is meant with "usable to anyone using it currently"? 
Does it mean the variables of it can still be read/written? 
Can the startcall or the initcall still be called after stopcall? 

(Important if we rely on zero initialization, which we do in many
modules).

Can IO to ports still be issued and "transactions" still be
finished after stopcall has been issued?

Thanks and Regards

Ingo Oeser
-- 
In der Wunschphantasie vieler Mann-Typen [ist die Frau] unsigned und
operatorvertraeglich. --- Dietz Proepper in dasr

Re: [PATCH] PART1: Proposed init & module changes for 2.5

[Rusty Russell]

In message <20010923124336.D30515@nightmaster.csn.tu-chemnitz.de> you write:
> On Sun, Sep 23, 2001 at 04:37:43PM +1000, Rusty Russell wrote:
> > OLD:	module_init(initfn)
> > NEW:	init_and_startcall(initfn, startfn);
> > COMMENTS:
> > 	Please use a semicolon at the end.  Modulable code should
> > 	transition to a two-stage init (initfn sets everything up
> > 	and may fail, and startfn which exposes the module to the rest
> > 	of the kernel and can't fail).  Some modules only need
> > 	initfn or startfn, in which case use initcall() or startcall().
>  
> Why separating them?

Hi Ingo,

	Glad you like the PARAM stuff.  To answer your question,
consider the following standard case in module initialization:

	int init(void)
	{
		int ret;

		ret = register_something();
		if (ret != 0) return ret;

		ret = register_something_else();
		if (ret != 0) {
			unregister_something();
			return ret;
		}
		return 0;
	}

Imagine register_something() succeeds, but register_something_else()
fails.  Someone could be accessing the module (through
register_something()): what do we do?  We can't free the module and
return the error to insmod...

Mind you, many modules currently ignore errors anyway...

> > OLD:	void exitfn(void) { ...; }
> > 	module_exit(void)
> > NEW:	int stopfn(void) { ...; return 0; }
> > 	void exitfn(void) { ...; }
> > 	stopcall(stopfn);
> > 	exitcall(exitfn);
> > COMMENTS:
> > 	If there are neither, then module is not unloadable.  This is
> > 	perfectly OK.  If stopfn returns 0 (otherwise it should be
> > 	-errno) it must have deregistered itself from the rest of the
> > 	kernel (ie. module count can never increase again), but still
> > 	be usable to anyone using it currently.  Once exitfn is
> > 	called, it is guaranteed to be unused.
> 
> Same question here: Why you changed this? 
> 
> What is meant with "usable to anyone using it currently"? 

Someone could still have a reference to the module when stopfn is
called.  If you register an interrupt handler, say, there could be
someone running it on another CPU right now.

> Does it mean the variables of it can still be read/written? 

Yes: after stopfn, no NEW things can access the module, but it must be
still usable for anyone who got access to the module before.

> Can the startcall or the initcall still be called after stopcall? 

No: at this stage the module will have to be reloaded, exactly because
of assumptions like zero-initialization.

> Can IO to ports still be issued and "transactions" still be
> finished after stopcall has been issued?

Yes... The module is still "live", it's just you must unregister
anything which might allow new accesses (eg. proc entries,
filesystems, IRQ handlers, etc).  Sometime after the module count
falls to zero, exitfn will be called and the module actually freed.

Hope that clarifies,
Rusty.
--
Premature optmztion is rt of all evl. --DK

Re: [PATCH] PART1: Proposed init & module changes for 2.5

[Rusty Russell]

On Sun, 23 Sep 2001 03:12:54 -0400
David Cinege <dcinege@psychosis.com> wrote:

> On Sunday 23 September 2001 2:37, Rusty Russell wrote:
> 
> Russ,
> 
> How about implementing MBS? (Bootloader module loading. IE as implmented in 
> GRUB) so we can finally have completely modular kernels?

Hi Dave,

	Why duplicate the module code in the boot loader?  Surely a tiny initrd 
or equiv is a better option here?

Rusty.
--
New mailer, beware.

Re: [PATCH] PART1: Proposed init & module changes for 2.5

[Keith Owens]

On Mon, 24 Sep 2001 08:42:25 +1000, 
Rusty Russell <rusty@rustcorp.com.au> wrote:
>In message <20010923124336.D30515@nightmaster.csn.tu-chemnitz.de> you write:
>> Can the startcall or the initcall still be called after stopcall? 
>
>No: at this stage the module will have to be reloaded, exactly because
>of assumptions like zero-initialization.

When we discussed this at linux.conf.au in Sydney, we agreed that we
could call startfn after stopfn to handle the quiesce unload algorithm.
That handles the rmmod race without exporting mod use count to
everything, i.e.

  rmmod
  if use count == 0, call stopfn()
    synchronize_kernel()
    if use count == 0, exitfn()
    if use count != 0, startfn()

That catches the race where a second cpu has entered the module but not
done MOD_INC_USECOUNT yet.  The module is now in use but stopfn has
been called, the best thing to do is accept that rmmod lost the race
and reinstate the module.

Re: [PATCH] PART1: Proposed init & module changes for 2.5

[David Cinege]

On Sunday 23 September 2001 20:09, you wrote:
> On Sun, 23 Sep 2001 03:12:54 -0400
>
> David Cinege <dcinege@psychosis.com> wrote:
> > On Sunday 23 September 2001 2:37, Rusty Russell wrote:
> >
> > Russ,
> >
> > How about implementing MBS? (Bootloader module loading. IE as implmented
> > in GRUB) so we can finally have completely modular kernels?
>
> Hi Dave,
>
> 	Why duplicate the module code in the boot loader?  Surely a tiny initrd
> or equiv is a better option here?

Russ,

No. It's much more restrictive. IMO the JOB of the boot loader is to provide 
minimal lowest level (arch dependant) device access to load and initialise 
the kernel, with enough user control to manipulate how the kernel
is initialised. GRUB does a fantastic job of this on x86. It's the first 
bootloader I've seen 'get it right'.

I don't suggest implementing the complete module loading in the boot loader,
(I'm still not even sure I really like it in the kernel.  ; > )
only loading the modules themselves into memory for the kernel access during 
it's exec. (Like an initrd image is loaded) If a boot loader wants to 
understand and use modules.dep or do on the fly dependency checking, or have 
kernel userland utils to update it's conifg intelligently, that's it's 
business. I only want to see a way modules can be preloaded before kernel 
exec.

This has already been designed and is called the Multi Boot Standard.
You can read the spec in the GRUB source archive.
	http://www.gnu.org/software/grub/
It been around a few years. It's sad only HURD(?) is using it,

Why an initrd solution doesn't even touch on the problem:
You have a kernel.
	It's scsi layer and scsi device drivers are all modules.

You have a system.
	It contains a single scsi drive.
	The kernel and modules reside on that drive.

Now the impossible exercise.
	Get that machine the boot.

All the parts are there but you can't. Your only option is to use an initrd 
that has the required modules *merged* into it. This is just disgusting and 
silly. It's also impossible to make that initrd on that machine without a 
suplementary system. (Yes there are other ways around this by why not just 
KISS???)

We ALREADY have the boot loader and spec designed to allow reloading
modules into memory this way. (GRUB) It should not be very hard to implement 
this from the pre-existing Linux intird code to read the modules out. It's 
also easy for other boot loaders to use their initrd code to implement the 
functionality. (Remember when you think about it's usefulness put it in the 
perspective of bootloaders that can read from a filesystem. IE GRUB and 
Syslinux.)

Please let's do this already...
I look forward to one day reaching the point it's *reasonable* to have a 
completely modular kernel and that any device drivers (even boot devices) can 
be upgraded without having to update the entire kernel.

If you've never played with GRUB yet, you can grab a nice raw floppy image 
from my ftp site.  ftp://ftp.psychosis.com/linux/grub

Dave

Re: [PATCH] PART1: Proposed init & module changes for 2.5

[Rusty Russell]

In message <4103.1001293289@kao2.melbourne.sgi.com> you write:
> On Mon, 24 Sep 2001 08:42:25 +1000, 
> Rusty Russell <rusty@rustcorp.com.au> wrote:
> >In message <20010923124336.D30515@nightmaster.csn.tu-chemnitz.de> you write:
> >> Can the startcall or the initcall still be called after stopcall? 
> >
> >No: at this stage the module will have to be reloaded, exactly because
> >of assumptions like zero-initialization.
> 
> When we discussed this at linux.conf.au in Sydney, we agreed that we
> could call startfn after stopfn to handle the quiesce unload algorithm.
> That handles the rmmod race without exporting mod use count to
> everything, i.e.
> 
>   rmmod
>   if use count == 0, call stopfn()
>     synchronize_kernel()
>     if use count == 0, exitfn()
>     if use count != 0, startfn()
> 
> That catches the race where a second cpu has entered the module but not
> done MOD_INC_USECOUNT yet.  The module is now in use but stopfn has
> been called, the best thing to do is accept that rmmod lost the race
> and reinstate the module.

Hi Keith!

	Yes, this was my original intention, but the more I thought
about it, the more I wondered if it was worth it.  In fact, my patch
adds CONFIG_MODULE_UNLOAD, because most people don't actually need it,
and I don't really care if most modules are not unloadable.  Loading
modules makes sense.  Unloading them is pretty much an optimization.

	The reason I eventually decided to split initialization in
this version anyway, was that so many drivers get it wrong!  Things
like registering interrupt handlers before they have set up their
internal state, etc.  Splitting it into two functions is merely a
mechanism to get them to think about things a little harder 8)

	  And honestly, I got distracted by the PARAM stuff 8)

Cheers!
Rusty.
--
Premature optmztion is rt of all evl. --DK

Re: [PATCH] PART1: Proposed init & module changes for 2.5

[Keith Owens]

On Mon, 24 Sep 2001 14:35:54 +1000, 
Rusty Russell <rusty@rustcorp.com.au> wrote:
>In message <4103.1001293289@kao2.melbourne.sgi.com> you write:
>> When we discussed this at linux.conf.au in Sydney, we agreed that we
>> could call startfn after stopfn to handle the quiesce unload algorithm.
>> That handles the rmmod race without exporting mod use count to
>> everything, i.e.
>
>	Yes, this was my original intention, but the more I thought
>about it, the more I wondered if it was worth it.  In fact, my patch
>adds CONFIG_MODULE_UNLOAD, because most people don't actually need it,
>and I don't really care if most modules are not unloadable.  Loading
>modules makes sense.  Unloading them is pretty much an optimization.
>
>	The reason I eventually decided to split initialization in
>this version anyway, was that so many drivers get it wrong!  Things
>like registering interrupt handlers before they have set up their
>internal state, etc.  Splitting it into two functions is merely a
>mechanism to get them to think about things a little harder 8)

I absolutely agree that the split between probe and register needs to
be done.  The difference between calling startfn after stopfn and
forbidding that sequence is the difference between

  rmmod
    loses race
    return OK
    module is still in use, user is confused
 
and

  rmmod
    loses race
    calls startfn
    returns failure
    module is still in use, user is happy

Allowing startfn after stopfn gives a more consistent view of the
operation.

Re: [PATCH] PART1: Proposed init & module changes for 2.5

[Rusty Russell]

In message <E15lMrA-0006ny-00@schizo.psychosis.com> you write:
> is initialised. GRUB does a fantastic job of this on x86. It's the first 
> bootloader I've seen 'get it right'.
> 
> I don't suggest implementing the complete module loading in the boot loader,
> (I'm still not even sure I really like it in the kernel.  ; > )

I was hoping someone would say this.  (Note that my userspace tools
are not as complete as the current ones, so take a pinch of salt):

Old kernel:
$ wc -l kernel/module.c
   1250 kernel/module.c
$ ls -l /sbin/insmod
-rwxr-xr-x    1 root     root        99824 Aug 30 09:35 /sbin/insmod
$ find modutils-2.4.1/ -name '*.[ch]' | xargs cat | wc -l
  20612

New kernel:
$ wc -l working-pmac-module/kernel/module.c         
    922 working-pmac-module/kernel/module.c
$ wc -l working-pmac-module/arch/ppc/kernel/module.c
    253 working-pmac-module/arch/ppc/kernel/module.c
$ ls -l modult-init-tools/insmod
-rwxrwxr-x    1 rusty    rusty        5240 Sep 24 15:23 insmod
$ find module-init-tools/ -name '*.[ch]' | xargs cat | wc -l
    661

Convinced yet?

> only loading the modules themselves into memory for the kernel access during 
> it's exec. (Like an initrd image is loaded) If a boot loader wants to 
> understand and use modules.dep or do on the fly dependency checking, or have 
> kernel userland utils to update it's conifg intelligently, that's it's 
> business. I only want to see a way modules can be preloaded before kernel 
> exec.

Why not just put them in the initrd image?  The kernel is not going to
use the entire module as it is, so it will have to copy it anyway...

> All the parts are there but you can't. Your only option is to use an initrd 
> that has the required modules *merged* into it. This is just disgusting and 
> silly.

Um, how do you get the modules into grub then?  If it can read ext2 at
boot, then it could create an initrd for me.  Otherwise, how is it
different from creating an initrd?

It'd be easy to implement now.  But it won't gain us anything, since
we'll all be using initrd-tng in 2.5 anyway.

Cheers,
Rusty.
PS.  Am big fan of LRP: it rocks.
--
Premature optmztion is rt of all evl. --DK

Re: [PATCH] PART1: Proposed init & module changes for 2.5

[Rusty Russell]

On Mon, 24 Sep 2001 14:54:09 +1000
Keith Owens <kaos@ocs.com.au> wrote:
>   rmmod
>     loses race
>     return OK
>     module is still in use, user is confused

Ah, I see... no, it's currently:

   rmmod
      loses race
      waits for module count to hit zero

This can be desirable behavior (rmmod -f).  It's not good if you wanted
auto-module-unloading, but then, we've never wanted to take the hit to have
a pagable kernel, and I don't think we should do so.

Cheers,
Rusty.

Re: [PATCH] PART1: Proposed init & module changes for 2.5

[Andrzej Krzysztofowicz]

"Rusty Russell wrote:"
> > I don't suggest implementing the complete module loading in the boot loader,
> > (I'm still not even sure I really like it in the kernel.  ; > )
[...]
> > only loading the modules themselves into memory for the kernel access during 
> > it's exec. (Like an initrd image is loaded) If a boot loader wants to 
> > understand and use modules.dep or do on the fly dependency checking, or have 
> > kernel userland utils to update it's conifg intelligently, that's it's 
> > business. I only want to see a way modules can be preloaded before kernel 
> > exec.
> 
> Why not just put them in the initrd image?  The kernel is not going to
> use the entire module as it is, so it will have to copy it anyway...

Somebody wants to modularize all binary formats, including binfmt_elf ... ?

Andrzej
-- 
=======================================================================
  Andrzej M. Krzysztofowicz               ankry@mif.pg.gda.pl
  phone (48)(58) 347 14 61
Faculty of Applied Phys. & Math.,   Technical University of Gdansk

Re: [PATCH] PART1: Proposed init & module changes for 2.5

[David Cinege]

On Monday 24 September 2001 1:31, Rusty Russell wrote:

> Convinced yet?

If it's cleaner in operation then the current system, yes.
What we've got now just seems so ugly.

> Why not just put them in the initrd image?  

Because images are not dynamic.  : P 

> Um, how do you get the modules into grub then?  If it can read ext2 at
> boot, then it could create an initrd for me.  Otherwise, how is it
> different from creating an initrd?

Yep you're still using LILO...(that still does that absolute block
offset shit.) GRUB is like syslinux in that it can read FS. But not just
ext2. It also knows reiserfs, fat, bsd, and some more. It also has a complete 
interactive commandline. Is more of a 'boot shell'. I gave you a like. Go try 
it. : >

Only the GRUB loader itself sits on an sbsolute sector.
So via the ominopotent symlink, you would need to do...nothing
after you install a new kernel to boot. With initrd, you would have
to make another initrd. But actually you would need one for every
kernel. (I've kept up to 6 active on my box at one time...)
This sucks...and it's why we all compile everything we really need static.

BTW if your initrd images gets corrupted or lost, you are paddling up fecal 
creek with your hands.  Let's not forget how much fun it is if you get a 
library mismatch on your initrd boot utils. Hope you compiled them all
static.

Loading system critcal kernel modules should be as straight forward
as the feature I'm requesting. The boot loader puts it in memory. The kernel 
loads it from memory. It's all good.

> It'd be easy to implement now.  But it won't gain us anything, since
> we'll all be using initrd-tng in 2.5 anyway.

Oh jesus...what have I missed? (I've been off the list a long time)

> Cheers,
> Rusty.
> PS.  Am big fan of LRP: it rocks.

I hope so...you're still imortalized in the POSIXness script. ; >
LRP is unfortunatly dated to the point it doesn't really rock right
now. : < It's simply cool.  : > If I ever get to work on LRP v5.0... (Hell I 
never got to release V4.0... : P) it will rock like nothing else ever has.

But since you bring up LRP, let's get down into the minimal space issues.
It's a living hell tring to maintain kernels in an OS distibution and a 
primary reason is that you have to play roulette deciding what to make
static. Then when you get it good, you have to fuck it all up to squeeze
it onto a boot disk. Adding what I want fixes this quite well. 

With the work I've done with LRP and embedded systems, I doubt many people
out there have played with the initrd more then me. If you haven't noticed
I'm back on the list because I just posted a new version of initrd dynamic
with tmpfs. It's a lifesaver for initrd, since now the FS 'creatation' is 
100% dynamic the the 'image' can be multiple tar.gz archives.

However as clean as it is, it's still not something I would want
to load critical kernel parts for boot.

Saying one should use only initrd for this is like saying the kernel
should not handle starting MD devices at all. 

As a programming ethic it sounds good but I think few programmer have
actually used some of these ideals in 'real world' system
adminstration. Try finding yourself staring at a mission critical server
that you now can not get to boot at 03:00 and you can't get
out to the net. That sort of experience changes your perspective. Been there.
Many times. No fun.

So just code this feature for me. Please. Make Dave happy.  ; >


Dave

-- 
The time is now 22:19 (Totalitarian)  -  http://www.ccops.org/clock.html