[PATCH v2 0/5] notifier error injection

[PATCH v2 0/5] notifier error injection

[Akinobu Mita]

This provides the ability to inject artifical errors to the following
notifier chain callbacks. It is useful to test the error handling of
notifier call chain failures.

 * CPU notifier
 * PM notifier
 * memory hotplug notifier
 * powerpc pSeries reconfig notifier

We already have cpu-notifier-error-inject.ko module providing such
functionality for the CPU notifier.  However, this patch set adds
common basic functions and consistent debugfs interface for them.

Each of the files in the directory represents an event which can be
failed and contains the error code.  If the notifier call chain should
be failed with some events notified, write the error code to the files.

Example: Inject CPU offline error (-1 == -EPERM)

	# cd /sys/kernel/debug/cpu-notifier-error-inject
	# echo -1 > CPU_UP_PREPARE
	# echo 0 > /sys/devices/system/cpu/cpu1/online
	bash: echo: write error: Operation not permitted

Changelog:

* v2
- "PM: Improve error code of pm_notifier_call_chain()" is now in -next
- "debugfs: add debugfs_create_int" is dropped
- put a comment in err_inject_notifier_block_init()
- only allow valid errno to be injected (-MAX_ERRNO <= errno <= 0)
- improve Kconfig help text
- make CONFIG_PM_NOTIFIER_ERROR_INJECTION visible even if PM_DEBUG is disabled
- make CONFIG_PM_NOTIFIER_ERROR_INJECTION default if PM_DEBUG is enabled

Akinobu Mita (5):
  fault-injection: notifier error injection
  cpu: CPU notifier error injection
  PM: PM notifier error injection
  memory: memory notifier error injection
  powerpc: pSeries reconfig notifier error injection

 Documentation/feature-removal-schedule.txt |    8 +++
 arch/powerpc/platforms/pseries/reconfig.c  |   31 +++++++++++
 drivers/base/memory.c                      |   29 ++++++++++
 include/linux/notifier.h                   |   25 +++++++++
 kernel/cpu.c                               |   31 +++++++++++
 kernel/notifier.c                          |   81 ++++++++++++++++++++++++++++
 kernel/power/main.c                        |   30 ++++++++++
 lib/Kconfig.debug                          |   71 ++++++++++++++++++++++++
 8 files changed, 306 insertions(+), 0 deletions(-)

-- 
1.7.4.4

[PATCH v2 1/5] fault-injection: notifier error injection

[Akinobu Mita]

The notifier error injection provides the ability to inject artifical
errors to specified notifier chain callbacks. It is useful to test
the error handling of notifier call chain failures.

This adds common basic functions to define which type of events can be
fail and to initialize the debugfs interface to control what error code
should be returned and which event should be failed.

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Pavel Machek <pavel@ucw.cz>
Cc: "Rafael J. Wysocki" <rjw@sisk.pl>
Cc: linux-pm@lists.linux-foundation.org
Cc: Greg Kroah-Hartman <gregkh@suse.de>
Cc: linux-mm@kvack.org
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: linuxppc-dev@lists.ozlabs.org
---
* v2
- put a comment in err_inject_notifier_block_init()
- only allow valid errno to be injected (-MAX_ERRNO <= errno <= 0)

 include/linux/notifier.h |   25 ++++++++++++++
 kernel/notifier.c        |   81 ++++++++++++++++++++++++++++++++++++++++++++++
 lib/Kconfig.debug        |   11 ++++++
 3 files changed, 117 insertions(+), 0 deletions(-)

diff --git a/include/linux/notifier.h b/include/linux/notifier.h
index c0688b0..51882d6 100644
--- a/include/linux/notifier.h
+++ b/include/linux/notifier.h
@@ -278,5 +278,30 @@ extern struct blocking_notifier_head reboot_notifier_list;
 #define VT_UPDATE		0x0004 /* A bigger update occurred */
 #define VT_PREWRITE		0x0005 /* A char is about to be written to the console */
 
+#ifdef CONFIG_NOTIFIER_ERROR_INJECTION
+
+struct err_inject_notifier_action {
+	unsigned long val;
+	int error;
+	const char *name;
+};
+
+#define ERR_INJECT_NOTIFIER_ACTION(action)	\
+	.name = #action, .val = (action),
+
+struct err_inject_notifier_block {
+	struct notifier_block nb;
+	struct dentry *dir;
+	struct err_inject_notifier_action actions[];
+	/* The last slot must be terminated with zero sentinel */
+};
+
+extern int err_inject_notifier_block_init(struct err_inject_notifier_block *enb,
+				const char *name, int priority);
+extern void err_inject_notifier_block_cleanup(
+				struct err_inject_notifier_block *enb);
+
+#endif /* CONFIG_NOTIFIER_ERROR_INJECTION */
+
 #endif /* __KERNEL__ */
 #endif /* _LINUX_NOTIFIER_H */
diff --git a/kernel/notifier.c b/kernel/notifier.c
index 2488ba7..8824ae3 100644
--- a/kernel/notifier.c
+++ b/kernel/notifier.c
@@ -5,6 +5,7 @@
 #include <linux/rcupdate.h>
 #include <linux/vmalloc.h>
 #include <linux/reboot.h>
+#include <linux/debugfs.h>
 
 /*
  *	Notifier list for kernel code which wants to be called
@@ -584,3 +585,83 @@ int unregister_die_notifier(struct notifier_block *nb)
 	return atomic_notifier_chain_unregister(&die_chain, nb);
 }
 EXPORT_SYMBOL_GPL(unregister_die_notifier);
+
+#ifdef CONFIG_NOTIFIER_ERROR_INJECTION
+
+static int debugfs_errno_set(void *data, u64 val)
+{
+	*(int *)data = clamp_t(int, val, -MAX_ERRNO, 0);
+	return 0;
+}
+
+static int debugfs_errno_get(void *data, u64 *val)
+{
+	*val = *(int *)data;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_errno, debugfs_errno_get, debugfs_errno_set,
+			"%lld\n");
+
+static struct dentry *debugfs_create_errno(const char *name, mode_t mode,
+				struct dentry *parent, int *value)
+{
+	return debugfs_create_file(name, mode, parent, value, &fops_errno);
+}
+
+static int err_inject_notifier_callback(struct notifier_block *nb,
+				unsigned long val, void *p)
+{
+	int err = 0;
+	struct err_inject_notifier_block *enb =
+		container_of(nb, struct err_inject_notifier_block, nb);
+	struct err_inject_notifier_action *action;
+
+	for (action = enb->actions; action->name; action++) {
+		if (action->val == val) {
+			err = action->error;
+			break;
+		}
+	}
+	if (err) {
+		printk(KERN_INFO "Injecting error (%d) to %s\n",
+			err, action->name);
+	}
+
+	return notifier_from_errno(err);
+}
+
+int err_inject_notifier_block_init(struct err_inject_notifier_block *enb,
+				const char *name, int priority)
+{
+	struct err_inject_notifier_action *action;
+	mode_t mode = S_IFREG | S_IRUSR | S_IWUSR;
+
+	enb->nb.notifier_call = err_inject_notifier_callback;
+	enb->nb.priority = priority;
+
+	enb->dir = debugfs_create_dir(name, NULL);
+	if (!enb->dir)
+		return -ENOMEM;
+
+	for (action = enb->actions; action->name; action++) {
+		/*
+		 * Create debugfs r/w file containing action->error. If
+		 * notifier call chain is called with action->val, it will
+		 * fail with the error code
+		 */
+		if (!debugfs_create_errno(action->name, mode, enb->dir,
+					&action->error)) {
+			debugfs_remove_recursive(enb->dir);
+			return -ENOMEM;
+		}
+	}
+	return 0;
+}
+
+void err_inject_notifier_block_cleanup(struct err_inject_notifier_block *enb)
+{
+	debugfs_remove_recursive(enb->dir);
+}
+
+#endif /* CONFIG_NOTIFIER_ERROR_INJECTION */
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index dd373c8..8c6ce7e 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1018,6 +1018,17 @@ config LKDTM
 	Documentation on how to use the module can be found in
 	Documentation/fault-injection/provoke-crashes.txt
 
+config NOTIFIER_ERROR_INJECTION
+	bool "Notifier error injection"
+	depends on DEBUG_KERNEL
+	select DEBUG_FS
+	help
+	  This option provides the ability to inject artifical errors to
+	  specified notifier chain callbacks. It is useful to test the error
+	  handling of notifier call chain failures.
+
+	  Say N if unsure.
+
 config CPU_NOTIFIER_ERROR_INJECT
 	tristate "CPU notifier error injection module"
 	depends on HOTPLUG_CPU && DEBUG_KERNEL
-- 
1.7.4.4

[PATCH v2 2/5] cpu: CPU notifier error injection

[Akinobu Mita]

This provides the ability to inject artifical errors to CPU notifier
chain callbacks.  It is controlled through debugfs interface under
/sys/kernel/debug/cpu-notifier-error-inject/

Each of the files in the directory represents an event which can be
failed and contains the error code.  If the notifier call chain should
be failed with some events notified, write the error code to the files.

Example1: inject CPU offline error (-1 == -EPERM)

	# echo -1 > /sys/kernel/debug/cpu-notifier-error-inject/CPU_UP_PREPARE
	# echo 0 > /sys/devices/system/cpu/cpu1/online
	bash: echo: write error: Operation not permitted

Example2: inject CPU online error (-2 == -ENOENT)

	# echo -2 > /sys/kernel/debug/cpu-notifier-error-inject/CPU_DOWN_PREPARE
	# echo 1 > /sys/devices/system/cpu/cpu1/online
	bash: echo: write error: No such file or directory

This also schedules cpu-notifier-error-inject.ko module for removal in
favor of more generic interface.

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
---
* v2
- improve Kconfig help text

 Documentation/feature-removal-schedule.txt |    8 +++++++
 kernel/cpu.c                               |   31 ++++++++++++++++++++++++++++
 lib/Kconfig.debug                          |   20 ++++++++++++++++++
 3 files changed, 59 insertions(+), 0 deletions(-)

diff --git a/Documentation/feature-removal-schedule.txt b/Documentation/feature-removal-schedule.txt
index 72e2384..cf0b4b7 100644
--- a/Documentation/feature-removal-schedule.txt
+++ b/Documentation/feature-removal-schedule.txt
@@ -583,3 +583,11 @@ Why:	Superseded by the UVCIOC_CTRL_QUERY ioctl.
 Who:	Laurent Pinchart <laurent.pinchart@ideasonboard.com>
 
 ----------------------------
+
+What:	Support for CPU_NOTIFIER_ERROR_INJECT (cpu-notifier-error-inject.ko)
+When:	3.3
+Why:	Replaced with more generic debugfs interface in
+	/sys/kernel/debug/cpu-notifier-error-inject/
+Who:	Akinobu Mita <akinobu.mita@gmail.com>
+
+----------------------------
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 12b7458..c326f4d 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -170,6 +170,37 @@ void __ref unregister_cpu_notifier(struct notifier_block *nb)
 }
 EXPORT_SYMBOL(unregister_cpu_notifier);
 
+#ifdef CONFIG_CPU_NOTIFIER_ERROR_INJECTION
+
+static struct err_inject_notifier_block err_inject_cpu_notifier = {
+	.actions = {
+		{ ERR_INJECT_NOTIFIER_ACTION(CPU_UP_PREPARE) },
+		{ ERR_INJECT_NOTIFIER_ACTION(CPU_UP_PREPARE_FROZEN) },
+		{ ERR_INJECT_NOTIFIER_ACTION(CPU_DOWN_PREPARE) },
+		{ ERR_INJECT_NOTIFIER_ACTION(CPU_DOWN_PREPARE_FROZEN) },
+		{}
+	}
+};
+
+static int __init err_inject_cpu_notifier_init(void)
+{
+	int err;
+
+	err = err_inject_notifier_block_init(&err_inject_cpu_notifier,
+				"cpu-notifier-error-inject", -1);
+	if (err)
+		return err;
+
+	err = register_hotcpu_notifier(&err_inject_cpu_notifier.nb);
+	if (err)
+		err_inject_notifier_block_cleanup(&err_inject_cpu_notifier);
+
+	return err;
+}
+late_initcall(err_inject_cpu_notifier_init);
+
+#endif /* CONFIG_CPU_NOTIFIER_ERROR_INJECTION */
+
 static inline void check_for_tasks(int cpu)
 {
 	struct task_struct *p;
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 8c6ce7e..d1bbedd 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1029,6 +1029,26 @@ config NOTIFIER_ERROR_INJECTION
 
 	  Say N if unsure.
 
+config CPU_NOTIFIER_ERROR_INJECTION
+	bool "CPU notifier error injection"
+	depends on HOTPLUG_CPU && NOTIFIER_ERROR_INJECTION
+	help
+	  This option provides the ability to inject artifical errors to
+	  CPU notifier chain callbacks.  It is controlled through debugfs
+	  interface under /sys/kernel/debug/cpu-notifier-error-inject/
+
+	  Each of the files in the directory represents an event which can be
+	  failed and contains the error code.  If the notifier call chain
+	  should be failed with some events notified, write the error code to
+	  the files.
+
+	  Example: Inject CPU offline error (-1 == -EPERM)
+
+	  # cd /sys/kernel/debug/cpu-notifier-error-inject
+	  # echo -1 > CPU_UP_PREPARE
+	  # echo 0 > /sys/devices/system/cpu/cpu1/online
+	  bash: echo: write error: Operation not permitted
+
 config CPU_NOTIFIER_ERROR_INJECT
 	tristate "CPU notifier error injection module"
 	depends on HOTPLUG_CPU && DEBUG_KERNEL
-- 
1.7.4.4

Re: [PATCH v2 2/5] cpu: CPU notifier error injection

[Américo Wang]

On Mon, Jul 18, 2011 at 9:16 AM, Akinobu Mita <akinobu.mita@gmail.com> wrote:
...
> diff --git a/Documentation/feature-removal-schedule.txt b/Documentation/feature-removal-schedule.txt
> index 72e2384..cf0b4b7 100644
> --- a/Documentation/feature-removal-schedule.txt
> +++ b/Documentation/feature-removal-schedule.txt
> @@ -583,3 +583,11 @@ Why:       Superseded by the UVCIOC_CTRL_QUERY ioctl.
>  Who:   Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>
>  ----------------------------
> +
> +What:  Support for CPU_NOTIFIER_ERROR_INJECT (cpu-notifier-error-inject.ko)
> +When:  3.3
> +Why:   Replaced with more generic debugfs interface in
> +       /sys/kernel/debug/cpu-notifier-error-inject/
> +Who:   Akinobu Mita <akinobu.mita@gmail.com>

Hi, Akinobu,

I think it is okay to just remove it by this patchset, no need to defer it.

Re: [PATCH v2 2/5] cpu: CPU notifier error injection

[Akinobu Mita]

2011/7/18 Américo Wang <xiyou.wangcong@gmail.com>:
> On Mon, Jul 18, 2011 at 9:16 AM, Akinobu Mita <akinobu.mita@gmail.com> wrote:
> ...
>> diff --git a/Documentation/feature-removal-schedule.txt b/Documentation/feature-removal-schedule.txt
>> index 72e2384..cf0b4b7 100644
>> --- a/Documentation/feature-removal-schedule.txt
>> +++ b/Documentation/feature-removal-schedule.txt
>> @@ -583,3 +583,11 @@ Why:       Superseded by the UVCIOC_CTRL_QUERY ioctl.
>>  Who:   Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>>
>>  ----------------------------
>> +
>> +What:  Support for CPU_NOTIFIER_ERROR_INJECT (cpu-notifier-error-inject.ko)
>> +When:  3.3
>> +Why:   Replaced with more generic debugfs interface in
>> +       /sys/kernel/debug/cpu-notifier-error-inject/
>> +Who:   Akinobu Mita <akinobu.mita@gmail.com>
>
> Hi, Akinobu,
>
> I think it is okay to just remove it by this patchset, no need to defer it.

The only drawback is that new machanism has no way to specify notifier
priority and the priority is hard-coded to be -1.  So it can only inject
error at the tail of notifier chain.

My plan to fix that was the kernel parameters.  But it's not flexible
as it needs to reboot for changing the priority.

So I'm now considering keeping this kernel module style instead of
initializing in late_initcall().

[PATCH v2 3/5] PM: PM notifier error injection

[Akinobu Mita]

This provides the ability to inject artifical errors to PM notifier
chain callbacks.  It is controlled through debugfs interface under
/sys/kernel/debug/pm-notifier-error-inject/

Each of the files in the directory represents an event which can be
failed and contains the error code.  If the notifier call chain should
be failed with some events notified, write the error code to the files.

Example: Inject PM suspend error (-12 = -ENOMEM)

	# cd /sys/kernel/debug/pm-notifier-error-inject
	# echo -12 > PM_SUSPEND_PREPARE
	# echo mem > /sys/power/state
	bash: echo: write error: Cannot allocate memory

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Pavel Machek <pavel@ucw.cz>
Cc: "Rafael J. Wysocki" <rjw@sisk.pl>
Cc: linux-pm@lists.linux-foundation.org
---
* v2
- improve Kconfig help text
- make CONFIG_PM_NOTIFIER_ERROR_INJECTION visible even if PM_DEBUG is disabled
- make CONFIG_PM_NOTIFIER_ERROR_INJECTION default if PM_DEBUG is enabled

 kernel/power/main.c |   30 ++++++++++++++++++++++++++++++
 lib/Kconfig.debug   |   16 ++++++++++++++++
 2 files changed, 46 insertions(+), 0 deletions(-)

diff --git a/kernel/power/main.c b/kernel/power/main.c
index 6c601f8..04b3774 100644
--- a/kernel/power/main.c
+++ b/kernel/power/main.c
@@ -42,6 +42,36 @@ int pm_notifier_call_chain(unsigned long val)
 	return notifier_to_errno(ret);
 }
 
+#ifdef CONFIG_PM_NOTIFIER_ERROR_INJECTION
+
+static struct err_inject_notifier_block err_inject_pm_notifier = {
+	.actions = {
+		{ ERR_INJECT_NOTIFIER_ACTION(PM_HIBERNATION_PREPARE) },
+		{ ERR_INJECT_NOTIFIER_ACTION(PM_SUSPEND_PREPARE) },
+		{ ERR_INJECT_NOTIFIER_ACTION(PM_RESTORE_PREPARE) },
+		{}
+	}
+};
+
+static int __init err_inject_pm_notifier_init(void)
+{
+	int err;
+
+	err = err_inject_notifier_block_init(&err_inject_pm_notifier,
+				"pm-notifier-error-inject", -1);
+	if (err)
+		return err;
+
+	err = register_pm_notifier(&err_inject_pm_notifier.nb);
+	if (err)
+		err_inject_notifier_block_cleanup(&err_inject_pm_notifier);
+
+	return err;
+}
+late_initcall(err_inject_pm_notifier_init);
+
+#endif /* CONFIG_PM_NOTIFIER_ERROR_INJECTION */
+
 /* If set, devices may be suspended and resumed asynchronously. */
 int pm_async_enabled = 1;
 
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index d1bbedd..e5671ba 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1049,6 +1049,22 @@ config CPU_NOTIFIER_ERROR_INJECTION
 	  # echo 0 > /sys/devices/system/cpu/cpu1/online
 	  bash: echo: write error: Operation not permitted
 
+config PM_NOTIFIER_ERROR_INJECTION
+	bool "PM notifier error injection"
+	depends on PM && NOTIFIER_ERROR_INJECTION
+	default PM_DEBUG
+	help
+	  This option provides the ability to inject artifical errors to
+	  PM notifier chain callbacks.  It is controlled through debugfs
+	  interface.
+
+	  Example: Inject PM suspend error (-12 = -ENOMEM)
+
+	  # cd /sys/kernel/debug/pm-notifier-error-inject
+	  # echo -12 > PM_SUSPEND_PREPARE
+	  # echo mem > /sys/power/state
+	  bash: echo: write error: Cannot allocate memory
+
 config CPU_NOTIFIER_ERROR_INJECT
 	tristate "CPU notifier error injection module"
 	depends on HOTPLUG_CPU && DEBUG_KERNEL
-- 
1.7.4.4

[PATCH v2 4/5] memory: memory notifier error injection

[Akinobu Mita]

This provides the ability to inject artifical errors to memory hotplug
notifier chain callbacks.  It is controlled through debugfs interface
under /sys/kernel/debug/memory-notifier-error-inject/

Each of the files in the directory represents an event which can be
failed and contains the error code.  If the notifier call chain should
be failed with some events notified, write the error code to the files.

Example: Inject memory hotplug offline error (-12 == -ENOMEM)

	# cd /sys/kernel/debug/memory-notifier-error-inject
	# echo -12 > MEM_GOING_OFFLINE
	# echo offline > /sys/devices/system/memory/memoryXXX/state
	bash: echo: write error: Cannot allocate memory

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Greg Kroah-Hartman <gregkh@suse.de>
Cc: linux-mm@kvack.org
---
* v2
- improve Kconfig help text

 drivers/base/memory.c |   29 +++++++++++++++++++++++++++++
 lib/Kconfig.debug     |   15 +++++++++++++++
 2 files changed, 44 insertions(+), 0 deletions(-)

diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index 9f9b235..5b7430f 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -89,6 +89,35 @@ void unregister_memory_isolate_notifier(struct notifier_block *nb)
 }
 EXPORT_SYMBOL(unregister_memory_isolate_notifier);
 
+#ifdef CONFIG_MEMORY_NOTIFIER_ERROR_INJECTION
+
+static struct err_inject_notifier_block err_inject_memory_notifier = {
+	.actions = {
+		{ ERR_INJECT_NOTIFIER_ACTION(MEM_GOING_ONLINE) },
+		{ ERR_INJECT_NOTIFIER_ACTION(MEM_GOING_OFFLINE) },
+		{}
+	}
+};
+
+static int __init err_inject_memory_notifier_init(void)
+{
+	int err;
+
+	err = err_inject_notifier_block_init(&err_inject_memory_notifier,
+				"memory-notifier-error-inject", -1);
+	if (err)
+		return err;
+
+	err = register_memory_notifier(&err_inject_memory_notifier.nb);
+	if (err)
+		err_inject_notifier_block_cleanup(&err_inject_memory_notifier);
+
+	return err;
+}
+late_initcall(err_inject_memory_notifier_init);
+
+#endif /* CONFIG_MEMORY_NOTIFIER_ERROR_INJECTION */
+
 /*
  * register_memory - Setup a sysfs device for a memory block
  */
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index e5671ba..8f5c380 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1065,6 +1065,21 @@ config PM_NOTIFIER_ERROR_INJECTION
 	  # echo mem > /sys/power/state
 	  bash: echo: write error: Cannot allocate memory
 
+config MEMORY_NOTIFIER_ERROR_INJECTION
+	bool "Memory hotplug notifier error injection"
+	depends on MEMORY_HOTPLUG_SPARSE && NOTIFIER_ERROR_INJECTION
+	help
+	  This option provides the ability to inject artifical errors to
+	  memory hotplug notifier chain callbacks.  It is controlled through
+	  debugfs interface.
+
+	  Example: Inject memory hotplug offline error (-12 == -ENOMEM)
+
+	  # cd /sys/kernel/debug/memory-notifier-error-inject
+	  # echo -12 > MEM_GOING_OFFLINE
+	  # echo offline > /sys/devices/system/memory/memoryXXX/state
+	  bash: echo: write error: Cannot allocate memory
+
 config CPU_NOTIFIER_ERROR_INJECT
 	tristate "CPU notifier error injection module"
 	depends on HOTPLUG_CPU && DEBUG_KERNEL
-- 
1.7.4.4

[PATCH v2 5/5] powerpc: pSeries reconfig notifier error injection

[Akinobu Mita]

This provides the ability to inject artifical errors to pSeries reconfig
notifier chain callbacks.  It is controlled through debugfs interface
under /sys/kernel/debug/pSeries-reconfig-notifier-error-inject/

Each of the files in the directory represents an event which can be
failed and contains the error code.  If the notifier call chain should
be failed with some events notified, write the error code to the files.

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: linuxppc-dev@lists.ozlabs.org
---
- Nothing changed from v1

 arch/powerpc/platforms/pseries/reconfig.c |   31 +++++++++++++++++++++++++++++
 lib/Kconfig.debug                         |    9 ++++++++
 2 files changed, 40 insertions(+), 0 deletions(-)

diff --git a/arch/powerpc/platforms/pseries/reconfig.c b/arch/powerpc/platforms/pseries/reconfig.c
index 168651a..31d9b0f 100644
--- a/arch/powerpc/platforms/pseries/reconfig.c
+++ b/arch/powerpc/platforms/pseries/reconfig.c
@@ -117,6 +117,37 @@ int pSeries_reconfig_notify(unsigned long action, void *p)
 	return notifier_to_errno(err);
 }
 
+#ifdef CONFIG_PSERIES_RECONFIG_NOTIFIER_ERROR_INJECTION
+
+static struct err_inject_notifier_block err_inject_reconfig_nb = {
+	.actions = {
+		{ ERR_INJECT_NOTIFIER_ACTION(PSERIES_RECONFIG_ADD) },
+		{ ERR_INJECT_NOTIFIER_ACTION(PSERIES_RECONFIG_REMOVE) },
+		{ ERR_INJECT_NOTIFIER_ACTION(PSERIES_DRCONF_MEM_ADD) },
+		{ ERR_INJECT_NOTIFIER_ACTION(PSERIES_DRCONF_MEM_REMOVE) },
+		{}
+	}
+};
+
+static int __init err_inject_reconfig_notifier_init(void)
+{
+	int err;
+
+	err = err_inject_notifier_block_init(&err_inject_reconfig_nb,
+			"pSeries-reconfig-notifier-error-inject", -1);
+	if (err)
+		return err;
+
+	err = pSeries_reconfig_notifier_register(&err_inject_reconfig_nb.nb);
+	if (err)
+		err_inject_notifier_block_cleanup(&err_inject_reconfig_nb);
+
+	return err;
+}
+late_initcall(err_inject_reconfig_notifier_init);
+
+#endif /* CONFIG_PSERIES_RECONFIG_NOTIFIER_ERROR_INJECTION */
+
 static int pSeries_reconfig_add_node(const char *path, struct property *proplist)
 {
 	struct device_node *np;
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 8f5c380..d713000 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1080,6 +1080,15 @@ config MEMORY_NOTIFIER_ERROR_INJECTION
 	  # echo offline > /sys/devices/system/memory/memoryXXX/state
 	  bash: echo: write error: Cannot allocate memory
 
+config PSERIES_RECONFIG_NOTIFIER_ERROR_INJECTION
+	bool "pSeries reconfig notifier error injection"
+	depends on PPC_PSERIES && NOTIFIER_ERROR_INJECTION
+	help
+	  This option provides the ability to inject artifical errors to
+	  pSeries reconfig notifier chain callbacks.  It is controlled
+	  through debugfs interface under
+	  /sys/kernel/debug/pSeries-reconfig-notifier-error-inject/
+
 config CPU_NOTIFIER_ERROR_INJECT
 	tristate "CPU notifier error injection module"
 	depends on HOTPLUG_CPU && DEBUG_KERNEL
-- 
1.7.4.4

Re: [PATCH v2 0/5] notifier error injection

[Andrew Morton]

On Mon, 18 Jul 2011 10:16:01 +0900
Akinobu Mita <akinobu.mita@gmail.com> wrote:

> This provides the ability to inject artifical errors to the following
> notifier chain callbacks. It is useful to test the error handling of
> notifier call chain failures.

That all looks very nice, but I wonder how many people are actually
using these things.  The injection framework itself doesn't seem to
have had a lot of uptake.

I wonder if we could improve things by adding an easy-to-run testing
script which identifies all the available error-injection inputs,
exercises them and then reports on the result?

Such a script would logically reside under ./tests/fault-injection/,
but we still don't have a tests/ directory.  Which perhaps tells us
something ;)

Re: [PATCH v2 0/5] notifier error injection

[Américo Wang]

On Fri, Jul 22, 2011 at 5:37 AM, Andrew Morton
<akpm@linux-foundation.org> wrote:
...
> Such a script would logically reside under ./tests/fault-injection/,
> but we still don't have a tests/ directory.  Which perhaps tells us
> something ;)

We have tools/testing/. :)

Re: [PATCH v2 0/5] notifier error injection

[Akinobu Mita]

2011/7/22 Andrew Morton <akpm@linux-foundation.org>:
> On Mon, 18 Jul 2011 10:16:01 +0900
> Akinobu Mita <akinobu.mita@gmail.com> wrote:
>
>> This provides the ability to inject artifical errors to the following
>> notifier chain callbacks. It is useful to test the error handling of
>> notifier call chain failures.
>
> That all looks very nice, but I wonder how many people are actually
> using these things.  The injection framework itself doesn't seem to
> have had a lot of uptake.
>
> I wonder if we could improve things by adding an easy-to-run testing
> script which identifies all the available error-injection inputs,
> exercises them and then reports on the result?
>
> Such a script would logically reside under ./tests/fault-injection/,
> but we still don't have a tests/ directory.  Which perhaps tells us
> something ;)

Sounds goood. I'll create shell script that detects hot-pluggable CPU
or memory block and runs quick tests.

BTW, the patchset has been rewritten to be kernel modules rather than
initializing at late_initcall()s.  The reason for switching to kernel
modules is to provide an easy way to specify notifier priority as
module parameter.  I'll send newer patchset soon.