Re: [RFC PATCH 0/6] module, kbuild: Faster boot with custom kernel.

[Rusty Russell <rusty@rustcorp.com.au>]

On Friday 20 February 2009 08:29:48 Kay Sievers wrote:
> Further testing revealed, if I only comment out the stop_machine()
> preparation, which is used in an error case, I get almost the same
> improvement, even with the original mutex in place. Without the mutex
> it's still a bit better, maybe it would be much better if we have more
> CPUs, but all the long delays are gone only with removing the
> stop_machine() preparation.

Hmm, interesting.  The reason that reducing the lock coverage had this effect
is because stop_machine_create() just bumps a refcount if someone is already
between ...create() and ...destroy().

So, now we've found the problem, let's fix it, then re-visit mutex reduction.

module: don't use stop_machine on module load

Kay Sievers <kay.sievers@vrfy.org> discovered that boot times are slowed
by about half a second because all the stop_machine_create() calls,
and he only probes about 40 modules (I have 125 loaded on this laptop).

We only do stop_machine_create() so we can unlink the module if
something goes wrong, but it's overkill (and buggy anyway: if
stop_machine_create() fails we still call stop_machine_destroy()).

Since we are only protecting against kallsyms (esp. oops) walking the
list, synchronize_sched() is sufficient (synchronize_rcu() is probably
sufficient, but we're not in a hurry).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

diff --git a/kernel/module.c b/kernel/module.c
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -1881,12 +1881,6 @@ static noinline struct module *load_modu
 	if (len > 64 * 1024 * 1024 || (hdr = vmalloc(len)) == NULL)
 		return ERR_PTR(-ENOMEM);
 
-	/* Create stop_machine threads since the error path relies on
-	 * a non-failing stop_machine call. */
-	err = stop_machine_create();
-	if (err)
-		goto free_hdr;
-
 	if (copy_from_user(hdr, umod, len) != 0) {
 		err = -EFAULT;
 		goto free_hdr;
@@ -2271,12 +2265,13 @@ static noinline struct module *load_modu
 	/* Get rid of temporary copy */
 	vfree(hdr);
 
-	stop_machine_destroy();
 	/* Done! */
 	return mod;
 
  unlink:
-	stop_machine(__unlink_module, mod, NULL);
+	/* Unlink carefully: kallsyms could be walking list. */
+	__list_del(&mod->list);
+	synchronize_sched();
 	module_arch_cleanup(mod);
  cleanup:
 	kobject_del(&mod->mkobj.kobj);
@@ -2297,7 +2292,6 @@ static noinline struct module *load_modu
 	kfree(args);
  free_hdr:
 	vfree(hdr);
-	stop_machine_destroy();
 	return ERR_PTR(err);
 
  truncated: