[PATCH 2/3] Better interface for hooking early initcalls.

[PATCH 2/3] Better interface for hooking early initcalls.

[Eduard - Gabriel Munteanu]

Added early initcall (pre-SMP) support, using an identical interface to
that of regular initcalls. Functions called from do_pre_smp_initcalls()
could be converted to use this cleaner interface.

This is required by CPU hotplug, because early users have to register
notifiers before going SMP. One such CPU hotplug user is the relay
interface with buffer-only channels, which needs to register such a
notifier, to be usable in early code. This in turn is used by kmemtrace.

Signed-off-by: Eduard - Gabriel Munteanu <eduard.munteanu@linux360.ro>
---
 include/asm-generic/vmlinux.lds.h |    2 ++
 include/linux/init.h              |    7 +++++++
 init/main.c                       |   13 +++++++++++--
 3 files changed, 20 insertions(+), 2 deletions(-)

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index f054778..4a318e5 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -326,6 +326,8 @@
 	}
 
 #define INITCALLS							\
+	*(.initcallearly.init)						\
+	__early_initcall_end = .;					\
   	*(.initcall0.init)						\
   	*(.initcall0s.init)						\
   	*(.initcall1.init)						\
diff --git a/include/linux/init.h b/include/linux/init.h
index 21d658c..bd12a4c 100644
--- a/include/linux/init.h
+++ b/include/linux/init.h
@@ -170,6 +170,13 @@ extern void (*late_time_init)(void);
 	__attribute__((__section__(".initcall" level ".init"))) = fn
 
 /*
+ * Early initcalls run before initializing SMP.
+ *
+ * Only for built-in code, not modules.
+ */
+#define early_initcall(fn)		__define_initcall("early",fn,early)
+
+/*
  * A "pure" initcall has no dependencies on anything else, and purely
  * initializes variables that couldn't be statically initialized.
  *
diff --git a/init/main.c b/init/main.c
index f7fb200..bd5df3e 100644
--- a/init/main.c
+++ b/init/main.c
@@ -736,13 +736,13 @@ static void __init do_one_initcall(initcall_t fn)
 }
 
 
-extern initcall_t __initcall_start[], __initcall_end[];
+extern initcall_t __initcall_start[], __initcall_end[], __early_initcall_end[];
 
 static void __init do_initcalls(void)
 {
 	initcall_t *call;
 
-	for (call = __initcall_start; call < __initcall_end; call++)
+	for (call = __early_initcall_end; call < __initcall_end; call++)
 		do_one_initcall(*call);
 
 	/* Make sure there is no pending stuff from the initcall sequence */
@@ -775,6 +775,14 @@ static int __init nosoftlockup_setup(char *str)
 }
 __setup("nosoftlockup", nosoftlockup_setup);
 
+static void __init __do_pre_smp_initcalls(void)
+{
+	initcall_t *call;
+
+	for (call = __initcall_start; call < __early_initcall_end; call++)
+		(*call)();
+}
+
 static void __init do_pre_smp_initcalls(void)
 {
 	extern int spawn_ksoftirqd(void);
@@ -856,6 +864,7 @@ static int __init kernel_init(void * unused)
 
 	smp_prepare_cpus(setup_max_cpus);
 
+	__do_pre_smp_initcalls();
 	do_pre_smp_initcalls();
 
 	smp_init();
-- 
1.5.5.4

Re: [PATCH 2/3] Better interface for hooking early initcalls.

[Mathieu Desnoyers]

* Eduard - Gabriel Munteanu (eduard.munteanu@linux360.ro) wrote:
> Added early initcall (pre-SMP) support, using an identical interface to
> that of regular initcalls. Functions called from do_pre_smp_initcalls()
> could be converted to use this cleaner interface.
> 
> This is required by CPU hotplug, because early users have to register
> notifiers before going SMP. One such CPU hotplug user is the relay
> interface with buffer-only channels, which needs to register such a
> notifier, to be usable in early code. This in turn is used by kmemtrace.
> 

I am not sure it's worth it trying to define a generic "early" initcall,
since definition of "how early it is" may change with time.

Currently, it's earlier than SMP init, but later on, it could become
earlier than mm init. If there are only few users of this, and given
that they must be designed "knowing" how early they are initialized wrt
other subsystems, I think it would make sense to call them directly from
the init code without putting them in a "early initcall" category.

> Signed-off-by: Eduard - Gabriel Munteanu <eduard.munteanu@linux360.ro>
 .... 
> -extern initcall_t __initcall_start[], __initcall_end[];
> +extern initcall_t __initcall_start[], __initcall_end[], __early_initcall_end[];
>  
>  static void __init do_initcalls(void)
>  {
>  	initcall_t *call;
>  
> -	for (call = __initcall_start; call < __initcall_end; call++)
> +	for (call = __early_initcall_end; call < __initcall_end; call++)
>  		do_one_initcall(*call);
>  
>  	/* Make sure there is no pending stuff from the initcall sequence */
> @@ -775,6 +775,14 @@ static int __init nosoftlockup_setup(char *str)
>  }
>  __setup("nosoftlockup", nosoftlockup_setup);
>  
> +static void __init __do_pre_smp_initcalls(void)
> +{
> +	initcall_t *call;
> +
> +	for (call = __initcall_start; call < __early_initcall_end; call++)
> +		(*call)();

why not do_one_initcall(*call); ?

Mathieu

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [PATCH 2/3] Better interface for hooking early initcalls.

[Eduard - Gabriel Munteanu]

On Tue, 17 Jun 2008 10:07:42 -0400
Mathieu Desnoyers <compudj@krystal.dyndns.org> wrote:

> I am not sure it's worth it trying to define a generic "early"
> initcall, since definition of "how early it is" may change with time.
> 
> Currently, it's earlier than SMP init, but later on, it could become
> earlier than mm init. If there are only few users of this, and given
> that they must be designed "knowing" how early they are initialized
> wrt other subsystems, I think it would make sense to call them
> directly from the init code without putting them in a "early
> initcall" category.

I designed this to be "same as other core initcalls, but before SMP".
This isn't a replacement we could use for very early code, such as
kmem_cache_init(). Take into account that CPU hotplug requires this, as
stated in the docs, I quote:
> You need to call register_cpu_notifier() from your init function.
> Init functions could be of two types: 
> 1. early init (init function called when only the boot processor is online).
> 2. late init (init function called _after_ all the CPUs are online).

> why not do_one_initcall(*call); ?

I haven't actually tried do_one_initcall() at that point, but it seemed
it messed up with preemption and IRQs. Will check and see if causes any
problems and resubmit if it works. But you do have a point, debugging
should be doable for these initcalls too.
 
> Mathieu
> 

Re: [PATCH 2/3] Better interface for hooking early initcalls.

[Vegard Nossum]

On Tue, Jun 17, 2008 at 4:07 PM, Mathieu Desnoyers
<compudj@krystal.dyndns.org> wrote:
> * Eduard - Gabriel Munteanu (eduard.munteanu@linux360.ro) wrote:
>> Added early initcall (pre-SMP) support, using an identical interface to
>> that of regular initcalls. Functions called from do_pre_smp_initcalls()
>> could be converted to use this cleaner interface.
>>
>> This is required by CPU hotplug, because early users have to register
>> notifiers before going SMP. One such CPU hotplug user is the relay
>> interface with buffer-only channels, which needs to register such a
>> notifier, to be usable in early code. This in turn is used by kmemtrace.
>>
>
> I am not sure it's worth it trying to define a generic "early" initcall,
> since definition of "how early it is" may change with time.
>
> Currently, it's earlier than SMP init, but later on, it could become
> earlier than mm init. If there are only few users of this, and given
> that they must be designed "knowing" how early they are initialized wrt
> other subsystems, I think it would make sense to call them directly from
> the init code without putting them in a "early initcall" category.

If this existed already, I'd use it for kmemcheck. We need to hook
into what happens just before SMP is initialized in order to set
maxcpus = 1 depending on some kernel parameter. Right now it's called
directly from do_pre_smp_initcalls():

@@ -779,6 +780,7 @@ static void __init do_pre_smp_initcalls(void)
 {
        extern int spawn_ksoftirqd(void);

+       kmemcheck_init();
        migration_init();
        spawn_ksoftirqd();
        if (!nosoftlockup)

With proposed patch, we wouldn't have to touch this file at all, so
for what it's worth, consider it acked by me.


Vegard

-- 
"The animistic metaphor of the bug that maliciously sneaked in while
the programmer was not looking is intellectually dishonest as it
disguises that the error is the programmer's own creation."
	-- E. W. Dijkstra, EWD1036

[PATCH 2/3] Better interface for hooking early initcalls.

[Eduard - Gabriel Munteanu]

Added early initcall (pre-SMP) support, using an identical interface to
that of regular initcalls. Functions called from do_pre_smp_initcalls()
could be converted to use this cleaner interface.

This is required by CPU hotplug, because early users have to register
notifiers before going SMP. One such CPU hotplug user is the relay
interface with buffer-only channels, which needs to register such a
notifier, to be usable in early code. This in turn is used by kmemtrace.

Signed-off-by: Eduard - Gabriel Munteanu <eduard.munteanu@linux360.ro>
---
 include/asm-generic/vmlinux.lds.h |    2 ++
 include/linux/init.h              |    7 +++++++
 init/main.c                       |   13 +++++++++++--
 3 files changed, 20 insertions(+), 2 deletions(-)

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index f054778..4a318e5 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -326,6 +326,8 @@
 	}
 
 #define INITCALLS							\
+	*(.initcallearly.init)						\
+	__early_initcall_end = .;					\
   	*(.initcall0.init)						\
   	*(.initcall0s.init)						\
   	*(.initcall1.init)						\
diff --git a/include/linux/init.h b/include/linux/init.h
index 21d658c..bd12a4c 100644
--- a/include/linux/init.h
+++ b/include/linux/init.h
@@ -170,6 +170,13 @@ extern void (*late_time_init)(void);
 	__attribute__((__section__(".initcall" level ".init"))) = fn
 
 /*
+ * Early initcalls run before initializing SMP.
+ *
+ * Only for built-in code, not modules.
+ */
+#define early_initcall(fn)		__define_initcall("early",fn,early)
+
+/*
  * A "pure" initcall has no dependencies on anything else, and purely
  * initializes variables that couldn't be statically initialized.
  *
diff --git a/init/main.c b/init/main.c
index f7fb200..bd5df3e 100644
--- a/init/main.c
+++ b/init/main.c
@@ -736,13 +736,13 @@ static void __init do_one_initcall(initcall_t fn)
 }
 
 
-extern initcall_t __initcall_start[], __initcall_end[];
+extern initcall_t __initcall_start[], __initcall_end[], __early_initcall_end[];
 
 static void __init do_initcalls(void)
 {
 	initcall_t *call;
 
-	for (call = __initcall_start; call < __initcall_end; call++)
+	for (call = __early_initcall_end; call < __initcall_end; call++)
 		do_one_initcall(*call);
 
 	/* Make sure there is no pending stuff from the initcall sequence */
@@ -775,6 +775,14 @@ static int __init nosoftlockup_setup(char *str)
 }
 __setup("nosoftlockup", nosoftlockup_setup);
 
+static void __init __do_pre_smp_initcalls(void)
+{
+	initcall_t *call;
+
+	for (call = __initcall_start; call < __early_initcall_end; call++)
+		(*call)();
+}
+
 static void __init do_pre_smp_initcalls(void)
 {
 	extern int spawn_ksoftirqd(void);
@@ -856,6 +864,7 @@ static int __init kernel_init(void * unused)
 
 	smp_prepare_cpus(setup_max_cpus);
 
+	__do_pre_smp_initcalls();
 	do_pre_smp_initcalls();
 
 	smp_init();
-- 
1.5.5.4

Re: [PATCH 2/3] Better interface for hooking early initcalls.

[Pekka Enberg]

Hi Eduard,

On Sat, Jun 21, 2008 at 5:10 PM, Eduard - Gabriel Munteanu
<eduard.munteanu@linux360.ro> wrote:
> Added early initcall (pre-SMP) support, using an identical interface to
> that of regular initcalls. Functions called from do_pre_smp_initcalls()
> could be converted to use this cleaner interface.

As there aren't that many functions called from do_pre_smp_initcalls()
, I'd suggest you just go ahead and convert them to get this patch
merged to mainline.

Re: [PATCH 2/3] Better interface for hooking early initcalls.

[Eduard - Gabriel Munteanu]

On Sun, 22 Jun 2008 22:22:00 +0300
"Pekka Enberg" <penberg@cs.helsinki.fi> wrote:

> Hi Eduard,

Hi,
 
> As there aren't that many functions called from do_pre_smp_initcalls()
> , I'd suggest you just go ahead and convert them to get this patch
> merged to mainline.

Will do, good idea. However, I feel like these should be two different
patches. The first adds the new interface, the second converts existing
users to the new interface. Will resubmit the patch series soon, along
with the conversion patch.


	Eduard

Re: [PATCH 2/3] Better interface for hooking early initcalls.

[Vegard Nossum]

On Sun, Jun 22, 2008 at 10:38 PM, Eduard - Gabriel Munteanu
<eduard.munteanu@linux360.ro> wrote:
> Will do, good idea. However, I feel like these should be two different
> patches. The first adds the new interface, the second converts existing
> users to the new interface. Will resubmit the patch series soon, along
> with the conversion patch.

By the way -- may I ask why it is called early_initcall() and not
pre_smp_initcall()? I think it makes sense to encode the exact purpose
of the initcall into the name. "early" is not very explicit. This is
just my opinion :-) Thanks.


Vegard

-- 
"The animistic metaphor of the bug that maliciously sneaked in while
the programmer was not looking is intellectually dishonest as it
disguises that the error is the programmer's own creation."
	-- E. W. Dijkstra, EWD1036

Re: [PATCH 2/3] Better interface for hooking early initcalls.

[Pekka Enberg]

On Sun, Jun 22, 2008 at 11:43 PM, Vegard Nossum <vegard.nossum@gmail.com> wrote:
> By the way -- may I ask why it is called early_initcall() and not
> pre_smp_initcall()? I think it makes sense to encode the exact purpose
> of the initcall into the name. "early" is not very explicit. This is
> just my opinion :-) Thanks.

Sounds good to me.

Re: [PATCH 2/3] Better interface for hooking early initcalls.

[Eduard - Gabriel Munteanu]

On Sun, 22 Jun 2008 22:43:55 +0200
"Vegard Nossum" <vegard.nossum@gmail.com> wrote:
 
> By the way -- may I ask why it is called early_initcall() and not
> pre_smp_initcall()? I think it makes sense to encode the exact purpose
> of the initcall into the name. "early" is not very explicit. This is
> just my opinion :-) Thanks.

CPU hotplug docs name these (inexisting at that time) initcalls
'early', so I went on implementing 'early initcalls'. However, I don't
have strong feelings for either of these two names, so this could
change.

> Vegard
> 

Re: [PATCH 2/3] Better interface for hooking early initcalls.

[Eduard - Gabriel Munteanu]

On Sun, 22 Jun 2008 23:45:57 +0300
"Pekka Enberg" <penberg@cs.helsinki.fi> wrote:

> On Sun, Jun 22, 2008 at 11:43 PM, Vegard Nossum
> <vegard.nossum@gmail.com> wrote:
> > By the way -- may I ask why it is called early_initcall() and not
> > pre_smp_initcall()? I think it makes sense to encode the exact
> > purpose of the initcall into the name. "early" is not very
> > explicit. This is just my opinion :-) Thanks.
> 
> Sounds good to me.

Which one? :-)

I mean I could change the name, just let me know which one is better.
One encodes the purpose, while the other is consistent with CPU hotplug
stuff. Not that we couldn't change CPU hotplug docs to mention this,
this is an idea too.