[PATCH 2/2] hotplug resource limitation

[PATCH 2/2] hotplug resource limitation

[Hannes Reinecke]

[-- Attachment #1: Type: text/plain, Size: 1117 bytes --]

Hi all,
this is the second patch to implement hotplug resource limitation 
(relative to 2.6.7-rc2-mm2).

In some cases it is preferrable to adapt the number of concurrent 
hotplug processes on the fly in addition to set the number statically 
during boot. Additionally, it might be required to disable hotplug / 
kmod event delivery altogether for debugging purposes (e.g. if a module 
loaded automatically is crashing the machine).

This patch implements a new sysctl variable kernel.khelper_max (with 
proc interface /proc/sys/kernel/khelper_max) which holds the max # of 
concurrently running khelper threads. Setting this variable to '0' 
disables hotplug event delivery altogether. Events will not be lost but 
rather queued within the kernel (i.e. threads are cloned which wait on a 
semaphore to become free). This allows for a replay of missed events 
once the variable is increased again.

Further explanation on top of the patch.

Please apply.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke			hare@suse.de
SuSE Linux AG				S390 & zSeries
Maxfeldstraße 5				+49 911 74053 688
90409 Nürnberg				http://www.suse.de

[-- Attachment #2: khelper-mm2-dynamic.patch --]
[-- Type: text/x-patch, Size: 8095 bytes --]

Restrict max number of concurrent hotplug processes (Part 2)

The max number of concurrent hotplug / kmod events might need to be
changed at runtime. This also allows for disabling the delivery of
hotplug / kmod events altogether for debugging reasons and later
enabling them again, with a complete replay of all queued events.

The runtime variable is implemented as a new sysctl variable, which
resets the global variable 'khelper_max' to the new value. Care has to
be taken if the variable is decreased and some processes are already
queued / blocked: As it's not possible to ensure that a call to down()
will not be blocked we take the other route of simply not calling up()
on exit from an already running process if the variable has been
changed during runtime of the process.

The patch alse needed to change request_module() to dynamically
allocate the process arguments, as now theses arguments might be
needed by a delayed execution of the kmod thread.

Signed-off-by: Hannes Reinecke <hare@suse.de>

diff -pur linux-2.6.8-rc2-mm2/include/linux/sysctl.h linux-2.6.8-rc2-mm2.hotplug/include/linux/sysctl.h
--- linux-2.6.8-rc2-mm2/include/linux/sysctl.h	2004-08-03 14:58:14.000000000 +0200
+++ linux-2.6.8-rc2-mm2.hotplug/include/linux/sysctl.h	2004-08-05 16:33:02.000000000 +0200
@@ -136,6 +136,7 @@ enum
 	KERN_UNKNOWN_NMI_PANIC=66, /* int: unknown nmi panic flag */
 	KERN_INTERACTIVE=67,	/* interactive tasks can have cpu bursts */
 	KERN_COMPUTE=68,	/* adjust timeslices for a compute server */
+	KERN_KHELPER_MAX=69,	/* max # of concurrent khelper processes */
 };
 
 
@@ -784,6 +785,8 @@ extern int proc_doulongvec_minmax(ctl_ta
 				  void __user *, size_t *);
 extern int proc_doulongvec_ms_jiffies_minmax(ctl_table *table, int,
 				      struct file *, void __user *, size_t *);
+extern int proc_dointvec_khelper(ctl_table *, int, struct file *,
+				 void __user *, size_t *);
 
 extern int do_sysctl (int __user *name, int nlen,
 		      void __user *oldval, size_t __user *oldlenp,
diff -pur linux-2.6.8-rc2-mm2/kernel/kmod.c linux-2.6.8-rc2-mm2.hotplug/kernel/kmod.c
--- linux-2.6.8-rc2-mm2/kernel/kmod.c	2004-08-05 17:27:53.000000000 +0200
+++ linux-2.6.8-rc2-mm2.hotplug/kernel/kmod.c	2004-08-05 17:14:26.000000000 +0200
@@ -47,6 +47,7 @@ extern int max_threads;
 
 static struct workqueue_struct *khelper_wq;
 int khelper_max = 50;
+static int khelper_diff;
 static struct semaphore khelper_sem;
 
 #ifdef CONFIG_KMOD
@@ -75,11 +76,11 @@ int request_module(const char *fmt, ...)
 	va_list args;
 	char module_name[MODULE_NAME_LEN];
 	int ret;
-	char *argv[] = { modprobe_path, "-q", "--", module_name, NULL };
-	static char *envp[] = { "HOME=/",
-				"TERM=linux",
-				"PATH=/sbin:/usr/sbin:/bin:/usr/bin",
-				NULL };
+	char *argv_buffer = NULL;
+	char *envp_buffer = NULL;
+	char *scratch = NULL;
+	char **argv = NULL;
+	char **envp = NULL;
 
 	va_start(args, fmt);
 	ret = vsnprintf(module_name, MODULE_NAME_LEN, fmt, args);
@@ -87,6 +88,51 @@ int request_module(const char *fmt, ...)
 	if (ret >= MODULE_NAME_LEN)
 		return -ENAMETOOLONG;
 
+	/*
+	 * Use kmalloc for argv and envp as they might be delayed.
+	 */
+	argv_buffer = kmalloc(512,GFP_KERNEL);
+	if (!argv_buffer)
+		return -ENOMEM;
+	memset(argv_buffer,0x0,512);
+
+	envp_buffer = kmalloc(128,GFP_KERNEL);
+	if (!envp_buffer) {
+		kfree(argv_buffer);
+		return -ENOMEM;
+	}
+	memset(envp_buffer,0x0, 128);
+
+	scratch = argv_buffer;
+	argv = (char **)scratch;
+	scratch += sizeof(char *) * 5;
+	argv[0] = scratch;
+	strcpy(scratch, modprobe_path);
+	scratch += strlen(scratch) + 1;
+	argv[1] = scratch;
+	strcpy(scratch, "-q");
+	scratch += 3;
+	argv[2] = scratch;
+	strcpy(scratch, "--");
+	scratch += 3;
+	argv[3] = scratch;
+	strcpy(scratch, module_name);
+	argv[4] = NULL;
+
+	scratch = envp_buffer;
+	envp = (char **)scratch;
+	scratch += sizeof(char *) * 4;
+	envp[0] = scratch;
+	strcpy(scratch, "HOME=/");
+	scratch += strlen("HOME=/") + 1;
+	envp[1] = scratch;
+	strcpy(scratch, "TERM=linux");
+	scratch += strlen("TERM=linux") + 1;
+	envp[2] = scratch;
+	strcpy(scratch,"PATH=/sbin:/usr/sbin:/bin:/usr/bin");
+	
+	envp[3] = NULL;
+
 	/* If modprobe needs a service that is in a module, we get a recursive
 	 * loop.  Limit the number of running kmod threads to max_threads/2 or
 	 * MAX_KMOD_CONCURRENT, whichever is the smaller.  A cleaner method
@@ -102,7 +148,7 @@ int request_module(const char *fmt, ...)
 	 * Resource checking is now implemented in 
 	 * call_usermodehelper --hare
 	 */
-	ret = call_usermodehelper(modprobe_path, argv, envp, 1);
+	ret = call_usermodehelper(argv[0], argv, envp, 1);
 	return ret;
 }
 EXPORT_SYMBOL(request_module);
@@ -203,7 +249,7 @@ static int wait_for_helper(void *data)
 		printk(KERN_INFO "khelper: delay event %s (current %d, max %d)\n",
 		       ev_descr, atomic_read(&khelper_sem.count), khelper_max);
 #endif
-		if (wait == 0) {
+		if (khelper_max < 5 || wait == 0) {
 			/* Queueing is for async events only */
 			wait = -1;
 			sub_info->retval = 0;
@@ -285,7 +331,14 @@ static int wait_for_helper(void *data)
 		kfree(stored_info.envp);
 	}
 
-	up(&khelper_sem);
+	if (khelper_diff > 0) {
+#ifdef DEBUG_KHELPER
+		printk(KERN_INFO "khelper: decrease semaphore\n");
+#endif
+		khelper_diff--;
+	} else {
+		up(&khelper_sem);
+	}
 
 	return 0;
 }
@@ -363,6 +416,48 @@ void __init usermodehelper_init(void)
 	sema_init(&khelper_sem, khelper_max);
 }
 
+/* 
+ * Worker to adapt the number of khelper processes.
+ * down() might block, so we need a separate thread ...
+ */
+int khelper_modify_number(int diff)
+{
+	if (khelper_max > 0) {
+		printk(KERN_INFO "khelper: max %d concurrent processes\n",
+		       khelper_max);
+	} else {
+		printk(KERN_INFO "khelper: delaying events\n");
+	}
+
+	if (diff > 0) {
+		while (diff > 0) {
+			up(&khelper_sem);
+			diff--;
+		}
+	} else {
+		/* 
+		 * Note that we cannot call down() directly
+		 * as this would block.
+		 * So we first test whether we would block
+		 * and instruct the running processes to not
+		 * call up() instead.
+		 */
+		while (diff < 0) {
+			if (down_trylock(&khelper_sem)) {
+				khelper_diff++;
+			} else {
+#ifdef DEBUG_KHELPER
+				printk(KERN_INFO "khelper: decrease semaphore\n");
+#endif
+			}
+			diff++;
+		}
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(khelper_modify_number);
+
 /*
  * Sanity check for khelper_max is done in usermodehelper_init,
  * as at this point of time the system is not fully initialised.
diff -pur linux-2.6.8-rc2-mm2/kernel/sysctl.c linux-2.6.8-rc2-mm2.hotplug/kernel/sysctl.c
--- linux-2.6.8-rc2-mm2/kernel/sysctl.c	2004-08-03 14:58:15.000000000 +0200
+++ linux-2.6.8-rc2-mm2.hotplug/kernel/sysctl.c	2004-08-05 16:51:43.000000000 +0200
@@ -64,6 +64,7 @@ extern int sysctl_lower_zone_protection;
 extern int min_free_kbytes;
 extern int printk_ratelimit_jiffies;
 extern int printk_ratelimit_burst;
+extern int khelper_max;
 
 #if defined(CONFIG_X86_LOCAL_APIC) && defined(__i386__)
 int unknown_nmi_panic;
@@ -414,6 +415,14 @@ static ctl_table kern_table[] = {
 		.proc_handler	= &proc_dostring,
 		.strategy	= &sysctl_string,
 	},
+	{
+		.ctl_name	= KERN_KHELPER_MAX,
+		.procname	= "khelper_max",
+		.data		= &khelper_max,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec_khelper,
+	},
 #endif
 #ifdef CONFIG_CHR_DEV_SG
 	{
@@ -1952,6 +1961,27 @@ int proc_dointvec_userhz_jiffies(ctl_tab
 		    	    do_proc_dointvec_userhz_jiffies_conv,NULL);
 }
 
+extern int khelper_modify_number(int diff);
+
+int proc_dointvec_khelper(ctl_table *table, int write, struct file *filp,
+			  void __user *buffer, size_t *lenp)
+{
+	int maxval = max_threads / 2;
+	struct do_proc_dointvec_minmax_conv_param param = {
+		.min = (int *) &zero,
+		.max = (int *) &maxval,
+	};
+	int oldval = khelper_max, retval;
+
+	retval = do_proc_dointvec(table, write, filp, buffer, lenp,
+				  do_proc_dointvec_minmax_conv, &param);
+	
+	if (khelper_max != oldval)
+	    khelper_modify_number(khelper_max - oldval);
+	return retval;
+}
+
+
 #else /* CONFIG_PROC_FS */
 
 int proc_dostring(ctl_table *table, int write, struct file *filp,

Re: [PATCH 2/2] hotplug resource limitation

[Greg KH]

On Mon, Aug 09, 2004 at 03:07:15PM +0200, Hannes Reinecke wrote:
> Hi all,
> this is the second patch to implement hotplug resource limitation 
> (relative to 2.6.7-rc2-mm2).
> 
> In some cases it is preferrable to adapt the number of concurrent 
> hotplug processes on the fly in addition to set the number statically 
> during boot.

Why?  This should be "auto-tuning".  We don't want to provide
yet-another-knob-for-people-to-tweak-from-userspace, right?

> Additionally, it might be required to disable hotplug / 
> kmod event delivery altogether for debugging purposes (e.g. if a module 
> loaded automatically is crashing the machine).

Ugh, that's just not a good thing at all.  You can do that by running:
	echo /bin/true > /proc/sys/kernel/hotplug
today if you have to.  I don't like the ability to stop the kernel from
running properly, like this patch allows you to.

thanks,

greg k-h

Re: [PATCH 2/2] hotplug resource limitation

[Hannes Reinecke]

Greg KH wrote:
> On Mon, Aug 09, 2004 at 03:07:15PM +0200, Hannes Reinecke wrote:
> 
>>Hi all,
>>this is the second patch to implement hotplug resource limitation 
>>(relative to 2.6.7-rc2-mm2).
>>
>>In some cases it is preferrable to adapt the number of concurrent 
>>hotplug processes on the fly in addition to set the number statically 
>>during boot.
> 
> 
> Why?  This should be "auto-tuning".  We don't want to provide
> yet-another-knob-for-people-to-tweak-from-userspace, right?
> 
In principle you're right. But as we don't really now how much resources 
the installed hotplug program will require I don't really see another way.

> 
>>Additionally, it might be required to disable hotplug / 
>>kmod event delivery altogether for debugging purposes (e.g. if a module 
>>loaded automatically is crashing the machine).
> 
> 
> Ugh, that's just not a good thing at all.  You can do that by running:
> 	echo /bin/true > /proc/sys/kernel/hotplug
> today if you have to.  I don't like the ability to stop the kernel from
> running properly, like this patch allows you to.
> 
But then you'll lose all events which happen in the meantime.

The dynamic setting is mainly intended for two scenarios:
- Booting. You can disable all events on boot with the kernel 
commandline and re-enable them once your hotplug subsystem is up and 
running. This way you can handle all (ok, all devices appearing in 
sysfs) device with hotplug events; there is no need to regenerate / fake 
all events which might have been missed. Currently you need two sets of 
scripts, one for configuring all devices until hotplug is running and 
another one used by hotplug.
- Debugging. Especially laptops or legacy free machines do have the 
problem that hotplug scripts might misconfigure the machine, e.g. by 
loading the wrong module. Of course it's possible to keep this 'disable 
hotplug events on request' (which should event available during boot) 
entirely in userspace. But it would increase the size of /sbin/hotplug 
(and hence the running time) by quite a bit; and this would bite us on 
every hotplug event generated resulting in a general slowdown of the 
hotplug subsystem. If we can stop the hotplug event delivery from within 
the kernel, we can keep the main hotplug script as small as possible 
while retaining the functionality of temporarily disabling all hotplug 
events.

What we could do, though, is to implement two semaphores, one for kmod 
calls and another one for hotplug calls. Then we can queue or event 
delay all hotplug events while any kmod call would just be blocked until 
the semaphore is free again.

Better approach?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke			hare@suse.de
SuSE Linux AG				S390 & zSeries
Maxfeldstraße 5				+49 911 74053 688
90409 Nürnberg				http://www.suse.de

Re: [PATCH 2/2] hotplug resource limitation

[Greg KH]

On Tue, Aug 10, 2004 at 10:10:02AM +0200, Hannes Reinecke wrote:
> Greg KH wrote:
> >On Mon, Aug 09, 2004 at 03:07:15PM +0200, Hannes Reinecke wrote:
> >
> >>Hi all,
> >>this is the second patch to implement hotplug resource limitation 
> >>(relative to 2.6.7-rc2-mm2).
> >>
> >>In some cases it is preferrable to adapt the number of concurrent 
> >>hotplug processes on the fly in addition to set the number statically 
> >>during boot.
> >
> >
> >Why?  This should be "auto-tuning".  We don't want to provide
> >yet-another-knob-for-people-to-tweak-from-userspace, right?
> >
> In principle you're right. But as we don't really now how much resources 
> the installed hotplug program will require I don't really see another way.

But who is going to know how to tweak such a knob?  How will an admin
know they should reduce the number of outstanding processes?

> >>Additionally, it might be required to disable hotplug / 
> >>kmod event delivery altogether for debugging purposes (e.g. if a module 
> >>loaded automatically is crashing the machine).
> >
> >
> >Ugh, that's just not a good thing at all.  You can do that by running:
> >	echo /bin/true > /proc/sys/kernel/hotplug
> >today if you have to.  I don't like the ability to stop the kernel from
> >running properly, like this patch allows you to.
> >
> But then you'll lose all events which happen in the meantime.

Yup.

> The dynamic setting is mainly intended for two scenarios:
> - Booting. You can disable all events on boot with the kernel 
> commandline and re-enable them once your hotplug subsystem is up and 
> running. This way you can handle all (ok, all devices appearing in 
> sysfs) device with hotplug events; there is no need to regenerate / fake 
> all events which might have been missed. Currently you need two sets of 
> scripts, one for configuring all devices until hotplug is running and 
> another one used by hotplug.

Or just put hotplug into your initramfs to catch all of the events from
the beginning of kernel time.

Actually, we currently only have 1 set of scripts that do the hotplug
work.  We have an additional one that makes some "fake" hotplug events
at init time to catch the events that we were not able to catch.  So we
still only have 1 main code path.

> - Debugging. Especially laptops or legacy free machines do have the 
> problem that hotplug scripts might misconfigure the machine, e.g. by 
> loading the wrong module. Of course it's possible to keep this 'disable 
> hotplug events on request' (which should event available during boot) 
> entirely in userspace. But it would increase the size of /sbin/hotplug 
> (and hence the running time) by quite a bit; and this would bite us on 
> every hotplug event generated resulting in a general slowdown of the 
> hotplug subsystem.

I'd be very supprised if you could actually measure the hit of adding 2
more lines of bash to those hotplug scripts.  They aren't exactly the
fastest thing in the world in the first place, nor do they need to be.

> If we can stop the hotplug event delivery from within the kernel, we
> can keep the main hotplug script as small as possible while retaining
> the functionality of temporarily disabling all hotplug events.

So, add kernel complexity in order to reduce userspace complexity?  I
don't think that argument is going to win here :)

> What we could do, though, is to implement two semaphores, one for kmod 
> calls and another one for hotplug calls. Then we can queue or event 
> delay all hotplug events while any kmod call would just be blocked until 
> the semaphore is free again.
> 
> Better approach?

Drop the whole thing?  I really don't see the need for this part of the
patch (part 1 makes sense for some limited memory machines like the s390
boxes.)  But I'm always open to further convincing...

thanks,

greg k-h

Re: [PATCH 2/2] hotplug resource limitation

[Hannes Reinecke]

Greg KH wrote:
> On Tue, Aug 10, 2004 at 10:10:02AM +0200, Hannes Reinecke wrote:
> 
>>Greg KH wrote:
>>
>>>On Mon, Aug 09, 2004 at 03:07:15PM +0200, Hannes Reinecke wrote:
>>>
>>>
>>>>Hi all,
>>>>this is the second patch to implement hotplug resource limitation 
>>>>(relative to 2.6.7-rc2-mm2).
>>>>
>>>>In some cases it is preferrable to adapt the number of concurrent 
>>>>hotplug processes on the fly in addition to set the number statically 
>>>>during boot.
>>>
>>>
>>>Why?  This should be "auto-tuning".  We don't want to provide
>>>yet-another-knob-for-people-to-tweak-from-userspace, right?
>>>
>>
>>In principle you're right. But as we don't really now how much resources 
>>the installed hotplug program will require I don't really see another way.
> 
> 
> But who is going to know how to tweak such a knob?  How will an admin
> know they should reduce the number of outstanding processes?
> 
Documentation?

> 
>>>>Additionally, it might be required to disable hotplug / 
>>>>kmod event delivery altogether for debugging purposes (e.g. if a module 
>>>>loaded automatically is crashing the machine).
>>>
>>>
>>>Ugh, that's just not a good thing at all.  You can do that by running:
>>>	echo /bin/true > /proc/sys/kernel/hotplug
>>>today if you have to.  I don't like the ability to stop the kernel from
>>>running properly, like this patch allows you to.
>>>
>>
>>But then you'll lose all events which happen in the meantime.
> 
> 
> Yup.
> 
> 
>>The dynamic setting is mainly intended for two scenarios:
>>- Booting. You can disable all events on boot with the kernel 
>>commandline and re-enable them once your hotplug subsystem is up and 
>>running. This way you can handle all (ok, all devices appearing in 
>>sysfs) device with hotplug events; there is no need to regenerate / fake 
>>all events which might have been missed. Currently you need two sets of 
>>scripts, one for configuring all devices until hotplug is running and 
>>another one used by hotplug.
> 
> 
> Or just put hotplug into your initramfs to catch all of the events from
> the beginning of kernel time.
> 
Unfortunately this does not quite work as of today.
Open issues:
- udp sockets are not available during early stages, making it 
impossible to use udevsend/udevd
- No initial hook is given, which would allow for a setup of the 
initramfs (e.g. mounting of /dev on ramfs). Relying to catch event with 
SEQNUM 0 is dodgy (as it might well be that we never get this one); 
checking on every event ditto (as one might want to use 'udev' as 
hotplug program).
- Synchronisation of configuration files is hard to solve; either you 
have to update the initramfs every time after a device has been 
reconfigured or you do some sort of auto-detection, in which case you 
have to pack in every module etc. into the initramfs.

> Actually, we currently only have 1 set of scripts that do the hotplug
> work.  We have an additional one that makes some "fake" hotplug events
> at init time to catch the events that we were not able to catch.  So we
> still only have 1 main code path.
> 
But IMHO the main advantage of having initramfs is that we can get rid 
of this 'coldplug' / fake hotplug event stuff as we're able to catch all 
events. Otherwise I don't really see the point in initramfs.

> 
>>- Debugging. Especially laptops or legacy free machines do have the 
>>problem that hotplug scripts might misconfigure the machine, e.g. by 
>>loading the wrong module. Of course it's possible to keep this 'disable 
>>hotplug events on request' (which should event available during boot) 
>>entirely in userspace. But it would increase the size of /sbin/hotplug 
>>(and hence the running time) by quite a bit; and this would bite us on 
>>every hotplug event generated resulting in a general slowdown of the 
>>hotplug subsystem.
> 
> 
> I'd be very supprised if you could actually measure the hit of adding 2
> more lines of bash to those hotplug scripts.  They aren't exactly the
> fastest thing in the world in the first place, nor do they need to be.
> 
But we want to keep the initial 'hotplug' script as small/fast as 
possible, as this script is called by _every_ event.
Any subsequent script (e.g. agents) do not really matter, as they are 
called much more infrequently (here on my local desktop machine I have a 
ration of 10:1).

> 
>>If we can stop the hotplug event delivery from within the kernel, we
>>can keep the main hotplug script as small as possible while retaining
>>the functionality of temporarily disabling all hotplug events.
> 
> 
> So, add kernel complexity in order to reduce userspace complexity?  I
> don't think that argument is going to win here :)
> 
Unfair argument :-).

Cheers,

Hannes
-- 
Dr. Hannes Reinecke			hare@suse.de
SuSE Linux AG				S390 & zSeries
Maxfeldstraße 5				+49 911 74053 688
90409 Nürnberg				http://www.suse.de

Re: [PATCH 2/2] hotplug resource limitation

[Olaf Dabrunz]

On 16-Aug-04, Hannes Reinecke wrote:
> Greg KH wrote:
> >On Tue, Aug 10, 2004 at 10:10:02AM +0200, Hannes Reinecke wrote:
> >
> >>Greg KH wrote:
> >>
> >>>On Mon, Aug 09, 2004 at 03:07:15PM +0200, Hannes Reinecke wrote:
> >I'd be very supprised if you could actually measure the hit of adding 2
> >more lines of bash to those hotplug scripts.  They aren't exactly the
> >fastest thing in the world in the first place, nor do they need to be.

On several architectures (most notably s390, but also IPF and PPC) the
load generated by hotplug during boot can in actual practice become
quite high. The bus scans can generate events of the order of 100 to
1000.

These machines are more I/O-focused and handle many devices in parallel.
So having a quick and lean hotplug becomes very desirable.

Since these machines also tend to be used in HA situations, problem
recovery time and therefore reboot time becomes an essential factor to
decrease periods of unavailability.

> But we want to keep the initial 'hotplug' script as small/fast as 
> possible, as this script is called by _every_ event.
> Any subsequent script (e.g. agents) do not really matter, as they are 
> called much more infrequently (here on my local desktop machine I have a 
> ration of 10:1).

-- 
Olaf Dabrunz (od/odabrunz), SUSE Linux AG, Nürnberg