[patch 1/7] documentation and scripts

[patch 1/7] documentation and scripts

[Akinobu Mita]

[-- Attachment #1: doc.patch --]
[-- Type: text/plain, Size: 8405 bytes --]

From: Akinobu Mita <akinobu.mita@gmail.com>

This patch set provides some fault-injection capabilities.

- kmalloc() failures

- alloc_pages() failures

- disk IO errors

We can see what really happens if those failures happen.

In order to enable these fault-injection capabilities:

1. Enable relevant config options (CONFIG_FAILSLAB, CONFIG_PAGE_ALLOC,
   CONFIG_MAKE_REQUEST) and if you want to configure them via debugfs,
   enable CONFIG_FAULT_INJECTION_DEBUG_FS.

2. Build and boot with this kernel

3. Configure fault-injection capabilities behavior by boot option or debugfs

   - Boot option

     failslab=
     fail_page_alloc=
     fail_make_request=

   - Debugfs

     /debug/failslab/*
     /debug/fail_page_alloc/*
     /debug/fail_make_request/*

   Please refer to the Documentation/fault-injection/fault-injection.txt
   for details.

4. See what really happens.

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: Don Mullis <dwm@meer.net>

 Documentation/fault-injection/failcmd.sh          |    4 
 Documentation/fault-injection/failmodule.sh       |   31 +++
 Documentation/fault-injection/fault-injection.txt |  180 ++++++++++++++++++++++
 Documentation/kernel-parameters.txt               |    7 
 4 files changed, 222 insertions(+)

Index: work-fault-inject/Documentation/fault-injection/failcmd.sh
===================================================================
--- /dev/null
+++ work-fault-inject/Documentation/fault-injection/failcmd.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+echo 1 > /proc/self/make-it-fail
+exec $*
Index: work-fault-inject/Documentation/fault-injection/failmodule.sh
===================================================================
--- /dev/null
+++ work-fault-inject/Documentation/fault-injection/failmodule.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+#
+# Usage: failmodule <failname> <modulename> [stacktrace-depth]
+#
+#	<failname>: "failslab", "fail_alloc_page", or "fail_make_request"
+#
+#	<modulename>: module name that you want to inject faults.
+#
+#	[stacktrace-depth]: the maximum number of stacktrace walking allowed
+#
+
+STACKTRACE_DEPTH=5
+if [ $# -gt 2 ]; then
+	STACKTRACE_DEPTH=$3
+fi
+
+if [ ! -d /debug/$1 ]; then
+	echo "Fault-injection $1 does not exist" >&2
+	exit 1
+fi
+if [ ! -d /sys/module/$2 ]; then
+	echo "Module $2 does not exist" >&2
+	exit 1
+fi
+
+# Disable any fault injection
+echo 0 > /debug/$1/stacktrace-depth
+
+echo `cat /sys/module/$2/sections/.text` > /debug/$1/address-start
+echo `cat /sys/module/$2/sections/.exit.text` > /debug/$1/address-end
+echo $STACKTRACE_DEPTH > /debug/$1/stacktrace-depth
Index: work-fault-inject/Documentation/fault-injection/fault-injection.txt
===================================================================
--- /dev/null
+++ work-fault-inject/Documentation/fault-injection/fault-injection.txt
@@ -0,0 +1,180 @@
+Fault injection capabilities infrastructure
+===========================================
+
+See also drivers/md/faulty.c and "every_nth" module option for scsi_debug.
+
+
+Available fault injection capabilities
+--------------------------------------
+
+o failslab
+
+  injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
+
+o fail_page_alloc
+
+  injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
+
+o fail_make_request
+
+  injects disk IO errors on permitted devices by
+  /sys/block/<device>/make-it-fail or
+  /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
+
+Configure fault-injection capabilities behavior
+-----------------------------------------------
+
+o debugfs entries
+
+fault-inject-debugfs kernel module provides some debugfs entries for runtime
+configuration of fault-injection capabilities.
+
+- /debug/*/probability:
+
+	likelihood of failure injection, in percent.
+
+- /debug/*/interval:
+
+	specifies the interval between failures, for calls to
+	should_fail() that pass all the other tests.
+
+	Note that if you enable this, by setting interval>1, you will
+	probably want to set probability=100.
+
+- /debug/*/times:
+
+	specifies how many times failures may happen at most.
+	A value of -1 means "no limit".
+
+- /debug/*/space:
+
+	specifies an initial resource "budget", decremented by "size"
+	on each call to should_fail(,size).  Failure injection is
+	suppressed until "space" reaches zero.
+
+- /debug/*/verbose
+
+	specifies the verbosity of the messages when failure is injected.
+	A value of '0' means no any messages, setting it to '1' will
+	print just to tell failure happened, and '2' will also
+	print call trace.  This is useful to debug the problems revealed by
+	fault injection capabilities.
+
+- /debug/*/task-filter:
+
+	A value of '0' disables filtering by process.
+	Any positive value limits failures to only processes indicated by
+	/proc/<pid>/make-it-fail==1.
+
+- /debug/*/stacktrace-depth:
+
+	specifies the maximum stacktrace depth walked during search
+	for a caller within [address-start,address-end).
+
+- /debug/*/address-start:
+- /debug/*/address-end:
+
+	specifies the range of virtual addresses tested during
+	stacktrace walking.  Failure is injected only if some caller
+	in the walked stacktrace lies within this range.
+
+o Boot option
+
+In order to inject faults while debugfs is not available (early boot time),
+use the boot option:
+
+    failslab=
+    fail_page_alloc=
+    fail_make_request=<interval>,<probability>,<space>,<times>
+
+How to add new fault injection capability
+-----------------------------------------
+
+o #include <linux/fault-inject.h>
+
+o define the fault attributes
+
+  DEFINE_FAULT_INJECTION(name);
+
+  Please see the definition of struct fault_attr in fault-inject.h
+  for details.
+
+o provide the way to configure fault attributes
+
+- boot option
+
+  If you need to enable the fault injection capability from boot time, you can
+  provide boot option to configure it. There is a helper function for it.
+
+  setup_fault_attr(attr, str);
+
+- debugfs entries
+
+  failslab, fail_page_alloc, and fail_make_request use this way.
+  There is a helper function for it.
+
+  init_fault_attr_entries(entries, attr, name);
+  void cleanup_fault_attr_entries(entries);
+
+- module parameters
+
+  If the scope of the fault injection capability is limited to a
+  single kernel module, it is better to provide module parameters to
+  configure the fault attributes.
+
+o add a hook to insert failures
+
+  should_fail() returns 1 when failures should happen.
+
+	should_fail(attr,size);
+
+Application Examples
+--------------------
+
+o inject slab allocation failures into module init/cleanup code
+
+------------------------------------------------------------------------------
+#!/bin/bash
+
+FAILCMD=Documentation/fault-injection/failcmd.sh
+
+echo Y > /debug/failslab/task-filter
+echo 10 > /debug/failslab/probability
+echo -1 > /debug/failslab/times
+
+oops()
+{
+	dmesg | grep BUG > /dev/null 2>&1 
+}
+
+find /lib/modules/`uname -r` -name '*.ko' -exec basename {} .ko \; |
+	while read i && ! oops
+	do
+		echo inserting $i...
+		bash $FAILCMD modprobe $i
+	done
+
+lsmod | awk '{ if ($3 == 0) { print $1 } }' |
+	while read i && ! oops
+	do
+		echo removing $i...
+		bash $FAILCMD modprobe -r $i
+	done
+
+------------------------------------------------------------------------------
+
+o inject slab allocation failures only for a specific module
+
+------------------------------------------------------------------------------
+#!/bin/bash
+
+FAILMOD=Documentation/fault-injection/failmodule.sh
+
+echo injecting errors into the module $1...
+
+modprobe $1
+bash $FAILMOD failslab $1 10
+echo 25 > /debug/failslab/probability
+
+------------------------------------------------------------------------------
+
Index: work-fault-inject/Documentation/kernel-parameters.txt
===================================================================
--- work-fault-inject.orig/Documentation/kernel-parameters.txt
+++ work-fault-inject/Documentation/kernel-parameters.txt
@@ -545,6 +545,13 @@ and is between 256 and 4096 characters. 
 	eurwdt=		[HW,WDT] Eurotech CPU-1220/1410 onboard watchdog.
 			Format: <io>[,<irq>]
 
+	failslab=
+	fail_page_alloc=
+	fail_make_request=[KNL]
+			General fault injection mechanism.
+			Format: <interval>,<probability>,<space>,<times>
+			See also /Documentation/fault-injection/.
+
 	fd_mcs=		[HW,SCSI]
 			See header of drivers/scsi/fd_mcs.c.
 

--

[patch 2/7] fault-injection capabilities infrastructure

[Akinobu Mita]

[-- Attachment #1: should-fail.patch --]
[-- Type: text/plain, Size: 8675 bytes --]

From: Akinobu Mita <akinobu.mita@gmail.com>

This patch provides base functions for implement fault-injection
capabilities.

- Lightweight random simulator is taken from crasher module for SUSE kernel

- The function should_fail() is taken from failmalloc-1.0
  (http://www.nongnu.org/failmalloc/)

Cc: okuji@enbug.org
Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: Don Mullis <dwm@meer.net>

 include/linux/fault-inject.h |   69 ++++++++++++++
 lib/Kconfig.debug            |   12 ++
 lib/Makefile                 |    1 
 lib/fault-inject.c           |  207 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 289 insertions(+)

Index: work-fault-inject/lib/Kconfig.debug
===================================================================
--- work-fault-inject.orig/lib/Kconfig.debug
+++ work-fault-inject/lib/Kconfig.debug
@@ -469,3 +469,15 @@ config LKDTM
 
 	Documentation on how to use the module can be found in
 	drivers/misc/lkdtm.c
+
+config FAULT_INJECTION
+	bool
+
+config FAULT_INJECTION_DEBUG_FS
+	bool "debugfs entries for fault-injection capabilities"
+	depends on FAULT_INJECTION && SYSFS
+	select DEBUG_FS
+	help
+	  This option enables to configure fault-injection capabilities via
+	  debugfs entries.
+
Index: work-fault-inject/lib/Makefile
===================================================================
--- work-fault-inject.orig/lib/Makefile
+++ work-fault-inject/lib/Makefile
@@ -56,6 +56,7 @@ obj-$(CONFIG_AUDIT_GENERIC) += audit.o
 obj-$(CONFIG_STATISTICS) += statistic.o
 
 obj-$(CONFIG_SWIOTLB) += swiotlb.o
+obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o
 
 lib-$(CONFIG_GENERIC_BUG) += bug.o
 
Index: work-fault-inject/include/linux/fault-inject.h
===================================================================
--- /dev/null
+++ work-fault-inject/include/linux/fault-inject.h
@@ -0,0 +1,69 @@
+#ifndef _LINUX_FAULT_INJECT_H
+#define _LINUX_FAULT_INJECT_H
+
+#ifdef CONFIG_FAULT_INJECTION
+
+#include <linux/types.h>
+#include <linux/debugfs.h>
+#include <asm/atomic.h>
+
+/*
+ * For explanation of the elements of this struct, see
+ * Documentation/fault-injection/fault-injection.txt
+ */
+struct fault_attr {
+	unsigned long probability;
+	unsigned long interval;
+	atomic_t times;
+	atomic_t space;
+	unsigned long verbose;
+
+	unsigned long count;
+
+#ifdef CONFIG_FAULT_INJECTION_DEBUG_FS
+
+	struct {
+		struct dentry *dir;
+
+		struct dentry *probability_file;
+		struct dentry *interval_file;
+		struct dentry *times_file;
+		struct dentry *space_file;
+		struct dentry *verbose_file;
+	} entries;
+
+#endif
+};
+
+#define DEFINE_FAULT_ATTR(name)					\
+	struct fault_attr name = {				\
+		.interval = 1,					\
+		.times = ATOMIC_INIT(1),			\
+	}
+
+int setup_fault_attr(struct fault_attr *attr, char *str);
+void should_fail_srandom(unsigned long entropy);
+int should_fail(struct fault_attr *attr, ssize_t size);
+
+#ifdef CONFIG_FAULT_INJECTION_DEBUG_FS
+
+int init_fault_attr_entries(struct fault_attr *attr, const char *name);
+void cleanup_fault_attr_entries(struct fault_attr *attr);
+
+#else /* CONFIG_FAULT_INJECTION_DEBUG_FS */
+
+static inline int init_fault_attr_entries(struct fault_attr *attr,
+					  const char *name)
+{
+	return -ENODEV;
+}
+
+static inline void cleanup_fault_attr_entries(struct fault_attr *attr)
+{
+}
+
+#endif /* CONFIG_FAULT_INJECTION_DEBUG_FS */
+
+#endif /* CONFIG_FAULT_INJECTION */
+
+#endif /* _LINUX_FAULT_INJECT_H */
Index: work-fault-inject/lib/fault-inject.c
===================================================================
--- /dev/null
+++ work-fault-inject/lib/fault-inject.c
@@ -0,0 +1,207 @@
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/random.h>
+#include <linux/stat.h>
+#include <linux/types.h>
+#include <linux/fs.h>
+#include <linux/module.h>
+#include <linux/fault-inject.h>
+
+int setup_fault_attr(struct fault_attr *attr, char *str)
+{
+	unsigned long probability;
+	unsigned long interval;
+	int times;
+	int space;
+
+	/* "<interval>,<probability>,<space>,<times>" */
+	if (sscanf(str, "%lu,%lu,%d,%d",
+			&interval, &probability, &space, &times) < 4) {
+		printk(KERN_WARNING
+			"FAULT_INJECTION: failed to parse arguments\n");
+		return 0;
+	}
+
+	attr->probability = probability;
+	attr->interval = interval;
+	atomic_set(&attr->times, times);
+	atomic_set(&attr->space, space);
+
+	return 1;
+}
+
+#define failure_probability(attr)	(attr)->probability
+#define failure_interval(attr)		(attr)->interval
+#define max_failures(attr)		(attr)->times
+#define current_space(attr)		(attr)->space
+#define atomic_dec_not_zero(v)		atomic_add_unless((v), -1, 0)
+
+static unsigned long rand_seed = 152L;
+
+static unsigned long should_fail_random(void)
+{
+	rand_seed = rand_seed * 690690L+1;
+	return rand_seed ^ jiffies;
+}
+
+void should_fail_srandom(unsigned long entropy)
+{
+	rand_seed ^= entropy;
+	should_fail_random();
+}
+
+static void fail_dump(struct fault_attr *attr)
+{
+	if (attr->verbose > 0)
+		printk(KERN_NOTICE "FAULT_INJECTION: forcing a failure\n");
+	if (attr->verbose > 1)
+		dump_stack();
+}
+
+/*
+ * This code is stolen from failmalloc-1.0
+ * http://www.nongnu.org/failmalloc/
+ */
+
+int should_fail(struct fault_attr *attr, ssize_t size)
+{
+	if (atomic_read(&max_failures(attr)) == 0)
+		return 0;
+
+	if (atomic_read(&current_space(attr)) > size) {
+		atomic_sub(size, &current_space(attr));
+		return 0;
+	}
+
+	if (failure_interval(attr) > 1) {
+		attr->count++;
+		if (attr->count % failure_interval(attr))
+			return 0;
+	}
+
+	if (failure_probability(attr) > should_fail_random() % 100)
+		goto fail;
+
+	return 0;
+
+fail:
+	fail_dump(attr);
+
+	if (atomic_read(&max_failures(attr)) != -1)
+		atomic_dec_not_zero(&max_failures(attr));
+
+	return 1;
+}
+
+#ifdef CONFIG_FAULT_INJECTION_DEBUG_FS
+
+static void debugfs_ul_set(void *data, u64 val)
+{
+	*(unsigned long *)data = val;
+}
+
+static u64 debugfs_ul_get(void *data)
+{
+	return *(unsigned long *)data;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_ul, debugfs_ul_get, debugfs_ul_set, "%llu\n");
+
+static struct dentry *debugfs_create_ul(const char *name, mode_t mode,
+				struct dentry *parent, unsigned long *value)
+{
+	return debugfs_create_file(name, mode, parent, value, &fops_ul);
+}
+
+static void debugfs_atomic_t_set(void *data, u64 val)
+{
+	atomic_set((atomic_t *)data, val);
+}
+
+static u64 debugfs_atomic_t_get(void *data)
+{
+	return atomic_read((atomic_t *)data);
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_atomic_t, debugfs_atomic_t_get,
+			debugfs_atomic_t_set, "%lld\n");
+
+static struct dentry *debugfs_create_atomic_t(const char *name, mode_t mode,
+				struct dentry *parent, atomic_t *value)
+{
+	return debugfs_create_file(name, mode, parent, value, &fops_atomic_t);
+}
+
+void cleanup_fault_attr_entries(struct fault_attr *attr)
+{
+	if (attr->entries.dir) {
+		if (attr->entries.probability_file) {
+			debugfs_remove(attr->entries.probability_file);
+			attr->entries.probability_file = NULL;
+		}
+		if (attr->entries.interval_file) {
+			debugfs_remove(attr->entries.interval_file);
+			attr->entries.interval_file = NULL;
+		}
+		if (attr->entries.times_file) {
+			debugfs_remove(attr->entries.times_file);
+			attr->entries.times_file = NULL;
+		}
+		if (attr->entries.space_file) {
+			debugfs_remove(attr->entries.space_file);
+			attr->entries.space_file = NULL;
+		}
+		if (attr->entries.verbose_file) {
+			debugfs_remove(attr->entries.verbose_file);
+			attr->entries.verbose_file = NULL;
+		}
+		debugfs_remove(attr->entries.dir);
+		attr->entries.dir = NULL;
+	}
+}
+
+int init_fault_attr_entries(struct fault_attr *attr, const char *name)
+{
+	mode_t mode = S_IFREG | S_IRUSR | S_IWUSR;
+	struct dentry *dir;
+	struct dentry *file;
+
+	memset(&attr->entries, 0, sizeof(attr->entries));
+
+	dir = debugfs_create_dir(name, NULL);
+	if (!dir)
+		goto fail;
+	attr->entries.dir = dir;
+
+	file = debugfs_create_ul("probability", mode, dir, &attr->probability);
+	if (!file)
+		goto fail;
+	attr->entries.probability_file = file;
+
+	file = debugfs_create_ul("interval", mode, dir, &attr->interval);
+	if (!file)
+		goto fail;
+	attr->entries.interval_file = file;
+
+	file = debugfs_create_atomic_t("times", mode, dir, &attr->times);
+	if (!file)
+		goto fail;
+	attr->entries.times_file = file;
+
+	file = debugfs_create_atomic_t("space", mode, dir, &attr->space);
+	if (!file)
+		goto fail;
+	attr->entries.space_file = file;
+
+	file = debugfs_create_ul("verbose", mode, dir, &attr->verbose);
+	if (!file)
+		goto fail;
+	attr->entries.verbose_file = file;
+
+	return 0;
+fail:
+	cleanup_fault_attr_entries(attr);
+	return -ENOMEM;
+}
+
+#endif /* CONFIG_FAULT_INJECTION_DEBUG_FS */

--

[patch 3/7] fault-injection capability for kmalloc

[Akinobu Mita]

[-- Attachment #1: failslab.patch --]
[-- Type: text/plain, Size: 3052 bytes --]

From: Akinobu Mita <akinobu.mita@gmail.com>

This patch provides fault-injection capability for kmalloc.

Boot option:

failslab=<interval>,<probability>,<space>,<times>

	<interval> -- specifies the interval of failures.

	<probability> -- specifies how often it should fail in percent.

	<space> -- specifies the size of free space where memory can be
		   allocated safely in bytes.

	<times> -- specifies how many times failures may happen at most.

Debugfs:

/debug/failslab/interval
/debug/failslab/probability
/debug/failslab/specifies
/debug/failslab/times

Example:

	failslab=10,100,0,-1

slab allocation (kmalloc(), kmem_cache_alloc(),..) fails once per 10 times.

Cc: Pekka Enberg <penberg@cs.helsinki.fi>
Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>

 lib/Kconfig.debug |    7 +++++++
 mm/slab.c         |   43 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 50 insertions(+)

Index: work-fault-inject/mm/slab.c
===================================================================
--- work-fault-inject.orig/mm/slab.c
+++ work-fault-inject/mm/slab.c
@@ -107,6 +107,7 @@
 #include	<linux/mempolicy.h>
 #include	<linux/mutex.h>
 #include	<linux/rtmutex.h>
+#include	<linux/fault-inject.h>
 
 #include	<asm/uaccess.h>
 #include	<asm/cacheflush.h>
@@ -3075,12 +3076,54 @@ static void *cache_alloc_debugcheck_afte
 #define cache_alloc_debugcheck_after(a,b,objp,d) (objp)
 #endif
 
+#ifdef CONFIG_FAILSLAB
+
+static DEFINE_FAULT_ATTR(failslab);
+
+static int __init setup_failslab(char *str)
+{
+	should_fail_srandom(jiffies);
+
+	return setup_fault_attr(&failslab, str);
+}
+__setup("failslab=", setup_failslab);
+
+static int should_failslab(struct kmem_cache *cachep, gfp_t flags)
+{
+	if ((flags & __GFP_NOFAIL) || cachep == &cache_cache)
+		return 0;
+
+	return should_fail(&failslab, obj_size(cachep));
+}
+
+static int __init failslab_debugfs(void)
+{
+	should_fail_srandom(jiffies);
+
+	return init_fault_attr_entries(&failslab, "failslab");
+}
+
+late_initcall(failslab_debugfs);
+
+#else /* CONFIG_FAILSLAB */
+
+static inline int should_failslab(struct kmem_cache *cachep, gfp_t flags)
+{
+	return 0;
+}
+
+#endif /* CONFIG_FAILSLAB */
+
 static inline void *____cache_alloc(struct kmem_cache *cachep, gfp_t flags)
 {
 	void *objp;
 	struct array_cache *ac;
 
 	check_irq_off();
+
+	if (should_failslab(cachep, flags))
+		return NULL;
+
 	ac = cpu_cache_get(cachep);
 	if (likely(ac->avail)) {
 		STATS_INC_ALLOCHIT(cachep);
Index: work-fault-inject/lib/Kconfig.debug
===================================================================
--- work-fault-inject.orig/lib/Kconfig.debug
+++ work-fault-inject/lib/Kconfig.debug
@@ -473,6 +473,13 @@ config LKDTM
 config FAULT_INJECTION
 	bool
 
+config FAILSLAB
+	bool "fault-injection capabilitiy for kmalloc"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION
+	help
+	  This option provides fault-injection capabilitiy for kmalloc.
+
 config FAULT_INJECTION_DEBUG_FS
 	bool "debugfs entries for fault-injection capabilities"
 	depends on FAULT_INJECTION && SYSFS

--

[patch 4/7] fault-injection capability for alloc_pages()

[Akinobu Mita]

[-- Attachment #1: fail_alloc_pages.patch --]
[-- Type: text/plain, Size: 3286 bytes --]

From: Akinobu Mita <akinobu.mita@gmail.com>

This patch provides fault-injection capability for alloc_pages()

Boot option:

fail_page_alloc=<interval>,<probability>,<space>,<times>

	<interval> -- specifies the interval of failures.

	<probability> -- specifies how often it should fail in percent.

	<space> -- specifies the size of free space where memory can be
		   allocated safely in pages.

	<times> -- specifies how many times failures may happen at most.

Debugfs:

/debug/fail_page_alloc/interval
/debug/fail_page_alloc/probability
/debug/fail_page_alloc/specifies
/debug/fail_page_alloc/times

Example:

	fail_page_alloc=10,100,0,-1

The page allocation (alloc_pages(), ...) fails once per 10 times.

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>

 lib/Kconfig.debug |    7 +++++++
 mm/page_alloc.c   |   42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 49 insertions(+)

Index: work-fault-inject/lib/Kconfig.debug
===================================================================
--- work-fault-inject.orig/lib/Kconfig.debug
+++ work-fault-inject/lib/Kconfig.debug
@@ -480,6 +480,13 @@ config FAILSLAB
 	help
 	  This option provides fault-injection capabilitiy for kmalloc.
 
+config FAIL_PAGE_ALLOC
+	bool "fault-injection capabilitiy for alloc_pages()"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION
+	help
+	  This option provides fault-injection capabilitiy for alloc_pages().
+
 config FAULT_INJECTION_DEBUG_FS
 	bool "debugfs entries for fault-injection capabilities"
 	depends on FAULT_INJECTION && SYSFS
Index: work-fault-inject/mm/page_alloc.c
===================================================================
--- work-fault-inject.orig/mm/page_alloc.c
+++ work-fault-inject/mm/page_alloc.c
@@ -39,6 +39,7 @@
 #include <linux/stop_machine.h>
 #include <linux/sort.h>
 #include <linux/pfn.h>
+#include <linux/fault-inject.h>
 
 #include <asm/tlbflush.h>
 #include <asm/div64.h>
@@ -906,6 +907,44 @@ failed:
 #define ALLOC_HIGH		0x20 /* __GFP_HIGH set */
 #define ALLOC_CPUSET		0x40 /* check for correct cpuset */
 
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+
+static DEFINE_FAULT_ATTR(fail_page_alloc);
+
+static int __init setup_fail_page_alloc(char *str)
+{
+	should_fail_srandom(jiffies);
+
+	return setup_fault_attr(&fail_page_alloc, str);
+}
+__setup("fail_page_alloc=", setup_fail_page_alloc);
+
+static int should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
+{
+	if (gfp_mask & __GFP_NOFAIL)
+		return 0;
+
+	return should_fail(&fail_page_alloc, 1 << order);
+}
+
+static int __init fail_page_alloc_debugfs(void)
+{
+	should_fail_srandom(jiffies);
+
+	return init_fault_attr_entries(&fail_page_alloc, "fail_page_alloc");
+}
+
+late_initcall(fail_page_alloc_debugfs);
+
+#else /* CONFIG_FAIL_PAGE_ALLOC */
+
+static inline int should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
+{
+	return 0;
+}
+
+#endif /* CONFIG_FAIL_PAGE_ALLOC */
+
 /*
  * Return 1 if free pages are above 'mark'. This takes into account the order
  * of the allocation.
@@ -1058,6 +1097,9 @@ __alloc_pages(gfp_t gfp_mask, unsigned i
 
 	might_sleep_if(wait);
 
+	if (should_fail_alloc_page(gfp_mask, order))
+		return NULL;
+
 restart:
 	page = get_page_from_freelist(gfp_mask|__GFP_HARDWALL, order,
 				zonelist, ALLOC_WMARK_LOW|ALLOC_CPUSET);

--

[patch 5/7] fault-injection capability for disk IO

[Akinobu Mita]

[-- Attachment #1: fail_make_request.patch --]
[-- Type: text/plain, Size: 6616 bytes --]

From: Akinobu Mita <akinobu.mita@gmail.com>

This patch provides fault-injection capability for disk IO.

Boot option:

fail_make_request=<probability>,<interval>,<space>,<times>

	<interval> -- specifies the interval of failures.

	<probability> -- specifies how often it should fail in percent.

	<space> -- specifies the size of free space where disk IO can be issued
		   safely in bytes.

	<times> -- specifies how many times failures may happen at most.

Debugfs:

/debug/fail_make_request/interval
/debug/fail_make_request/probability
/debug/fail_make_request/specifies
/debug/fail_make_request/times

Example:

	fail_make_request=10,100,0,-1
	echo 1 > /sys/blocks/hda/hda1/make-it-fail

generic_make_request() on /dev/hda1 fails once per 10 times.

Cc: Jens Axboe <axboe@suse.de>
Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>

 block/genhd.c         |   31 +++++++++++++++++++++++++++++++
 block/ll_rw_blk.c     |   43 +++++++++++++++++++++++++++++++++++++++++++
 fs/partitions/check.c |   27 +++++++++++++++++++++++++++
 include/linux/genhd.h |    4 ++++
 lib/Kconfig.debug     |    7 +++++++
 5 files changed, 112 insertions(+)

Index: work-fault-inject/block/ll_rw_blk.c
===================================================================
--- work-fault-inject.orig/block/ll_rw_blk.c
+++ work-fault-inject/block/ll_rw_blk.c
@@ -28,6 +28,7 @@
 #include <linux/interrupt.h>
 #include <linux/cpu.h>
 #include <linux/blktrace_api.h>
+#include <linux/fault-inject.h>
 
 /*
  * for max sense size
@@ -3050,6 +3051,45 @@ static void handle_bad_sector(struct bio
 	set_bit(BIO_EOF, &bio->bi_flags);
 }
 
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static DEFINE_FAULT_ATTR(fail_make_request);
+
+static int __init setup_fail_make_request(char *str)
+{
+	should_fail_srandom(jiffies);
+
+	return setup_fault_attr(&fail_make_request, str);
+}
+__setup("fail_make_request=", setup_fail_make_request);
+
+static int should_fail_request(struct bio *bio)
+{
+	if ((bio->bi_bdev->bd_disk->flags & GENHD_FL_FAIL) ||
+	    (bio->bi_bdev->bd_part && bio->bi_bdev->bd_part->make_it_fail))
+		return should_fail(&fail_make_request, bio->bi_size);
+
+	return 0;
+}
+
+static int __init fail_make_request_debugfs(void)
+{
+	should_fail_srandom(jiffies);
+
+	return init_fault_attr_entries(&fail_make_request, "fail_make_request");
+}
+
+late_initcall(fail_make_request_debugfs);
+
+#else /* CONFIG_FAIL_MAKE_REQUEST */
+
+static inline int should_fail_request(struct bio *bio)
+{
+	return 0;
+}
+
+#endif /* CONFIG_FAIL_MAKE_REQUEST */
+
 /**
  * generic_make_request: hand a buffer to its device driver for I/O
  * @bio:  The bio describing the location in memory and on the device.
@@ -3134,6 +3174,9 @@ end_io:
 		if (unlikely(test_bit(QUEUE_FLAG_DEAD, &q->queue_flags)))
 			goto end_io;
 
+		if (should_fail_request(bio))
+			goto end_io;
+
 		/*
 		 * If this device has partitions, remap block n
 		 * of partition p to block n+start(p) of the disk.
Index: work-fault-inject/lib/Kconfig.debug
===================================================================
--- work-fault-inject.orig/lib/Kconfig.debug
+++ work-fault-inject/lib/Kconfig.debug
@@ -487,6 +487,13 @@ config FAIL_PAGE_ALLOC
 	help
 	  This option provides fault-injection capabilitiy for alloc_pages().
 
+config FAIL_MAKE_REQUEST
+	bool "fault-injection capabilitiy for disk IO"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION 
+	help
+	  This option provides fault-injection capabilitiy to disk IO.
+
 config FAULT_INJECTION_DEBUG_FS
 	bool "debugfs entries for fault-injection capabilities"
 	depends on FAULT_INJECTION && SYSFS
Index: work-fault-inject/block/genhd.c
===================================================================
--- work-fault-inject.orig/block/genhd.c
+++ work-fault-inject/block/genhd.c
@@ -417,6 +417,34 @@ static struct disk_attribute disk_attr_s
 	.show	= disk_stats_read
 };
 
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static ssize_t disk_fail_store(struct gendisk * disk,
+			       const char *buf, size_t count)
+{
+	int i;
+
+	if (count > 0 && sscanf(buf, "%d", &i) > 0) {
+		if (i == 0)
+			disk->flags &= ~GENHD_FL_FAIL;
+		else
+			disk->flags |= GENHD_FL_FAIL;
+	}
+
+	return count;
+}
+static ssize_t disk_fail_read(struct gendisk * disk, char *page)
+{
+	return sprintf(page, "%d\n", disk->flags & GENHD_FL_FAIL ? 1 : 0);
+}
+static struct disk_attribute disk_attr_fail = {
+	.attr = {.name = "make-it-fail", .mode = S_IRUGO | S_IWUSR },
+	.store	= disk_fail_store,
+	.show	= disk_fail_read
+};
+
+#endif
+
 static struct attribute * default_attrs[] = {
 	&disk_attr_uevent.attr,
 	&disk_attr_dev.attr,
@@ -424,6 +452,9 @@ static struct attribute * default_attrs[
 	&disk_attr_removable.attr,
 	&disk_attr_size.attr,
 	&disk_attr_stat.attr,
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	&disk_attr_fail.attr,
+#endif
 	NULL,
 };
 
Index: work-fault-inject/include/linux/genhd.h
===================================================================
--- work-fault-inject.orig/include/linux/genhd.h
+++ work-fault-inject/include/linux/genhd.h
@@ -83,6 +83,9 @@ struct hd_struct {
 	struct kobject *holder_dir;
 	unsigned ios[2], sectors[2];	/* READs and WRITEs */
 	int policy, partno;
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	int make_it_fail;
+#endif
 };
 
 #define GENHD_FL_REMOVABLE			1
@@ -90,6 +93,7 @@ struct hd_struct {
 #define GENHD_FL_CD				8
 #define GENHD_FL_UP				16
 #define GENHD_FL_SUPPRESS_PARTITION_INFO	32
+#define GENHD_FL_FAIL				64
 
 struct disk_stats {
 	unsigned long sectors[2];	/* READs and WRITEs */
Index: work-fault-inject/fs/partitions/check.c
===================================================================
--- work-fault-inject.orig/fs/partitions/check.c
+++ work-fault-inject/fs/partitions/check.c
@@ -276,12 +276,39 @@ static struct part_attribute part_attr_s
 	.show	= part_stat_read
 };
 
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static ssize_t part_fail_store(struct hd_struct * p,
+			       const char *buf, size_t count)
+{
+	int i;
+
+	if (count > 0 && sscanf(buf, "%d", &i) > 0)
+		p->make_it_fail = (i == 0) ? 0 : 1;
+
+	return count;
+}
+static ssize_t part_fail_read(struct hd_struct * p, char *page)
+{
+	return sprintf(page, "%d\n", p->make_it_fail);
+}
+static struct part_attribute part_attr_fail = {
+	.attr = {.name = "make-it-fail", .mode = S_IRUGO | S_IWUSR },
+	.store	= part_fail_store,
+	.show	= part_fail_read
+};
+
+#endif
+
 static struct attribute * default_attrs[] = {
 	&part_attr_uevent.attr,
 	&part_attr_dev.attr,
 	&part_attr_start.attr,
 	&part_attr_size.attr,
 	&part_attr_stat.attr,
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	&part_attr_fail.attr,
+#endif
 	NULL,
 };
 

--

[patch 6/7] process filtering for fault-injection capabilities

[Akinobu Mita]

[-- Attachment #1: process-filter.patch --]
[-- Type: text/plain, Size: 5821 bytes --]

From: Akinobu Mita <akinobu.mita@gmail.com>

This patch provides process filtering feature.
The process filter allows failing only permitted processes
by /proc/<pid>/make-it-fail

Please see the example that demostrates how to inject slab allocation
failures into module init/cleanup code
in Documentation/fault-injection/fault-injection.txt

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>

 fs/proc/base.c               |   65 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/fault-inject.h |    2 +
 include/linux/sched.h        |    3 +
 lib/fault-inject.c           |   19 ++++++++++++
 4 files changed, 89 insertions(+)

Index: work-fault-inject/fs/proc/base.c
===================================================================
--- work-fault-inject.orig/fs/proc/base.c
+++ work-fault-inject/fs/proc/base.c
@@ -776,6 +776,65 @@ static struct file_operations proc_login
 };
 #endif
 
+#ifdef CONFIG_FAULT_INJECTION
+static ssize_t proc_fault_inject_read(struct file * file, char __user * buf,
+				      size_t count, loff_t *ppos)
+{
+	struct task_struct *task = get_proc_task(file->f_dentry->d_inode);
+	char buffer[PROC_NUMBUF];
+	size_t len;
+	int make_it_fail;
+	loff_t __ppos = *ppos;
+
+	if (!task)
+		return -ESRCH;
+	make_it_fail = task->make_it_fail;
+	put_task_struct(task);
+
+	len = snprintf(buffer, sizeof(buffer), "%i\n", make_it_fail);
+	if (__ppos >= len)
+		return 0;
+	if (count > len-__ppos)
+		count = len-__ppos;
+	if (copy_to_user(buf, buffer + __ppos, count))
+		return -EFAULT;
+	*ppos = __ppos + count;
+	return count;
+}
+
+static ssize_t proc_fault_inject_write(struct file * file,
+			const char __user * buf, size_t count, loff_t *ppos)
+{
+	struct task_struct *task;
+	char buffer[PROC_NUMBUF], *end;
+	int make_it_fail;
+
+	if (!capable(CAP_SYS_RESOURCE))
+		return -EPERM;
+	memset(buffer, 0, sizeof(buffer));
+	if (count > sizeof(buffer) - 1)
+		count = sizeof(buffer) - 1;
+	if (copy_from_user(buffer, buf, count))
+		return -EFAULT;
+	make_it_fail = simple_strtol(buffer, &end, 0);
+	if (*end == '\n')
+		end++;
+	task = get_proc_task(file->f_dentry->d_inode);
+	if (!task)
+		return -ESRCH;
+	task->make_it_fail = make_it_fail;
+	put_task_struct(task);
+	if (end - buffer == 0)
+		return -EIO;
+	return end - buffer;
+}
+
+static struct file_operations proc_fault_inject_operations = {
+	.read		= proc_fault_inject_read,
+	.write		= proc_fault_inject_write,
+};
+#endif
+
 #ifdef CONFIG_SECCOMP
 static ssize_t seccomp_read(struct file *file, char __user *buf,
 			    size_t count, loff_t *ppos)
@@ -1788,6 +1847,9 @@ static struct pid_entry tgid_base_stuff[
 #ifdef CONFIG_AUDITSYSCALL
 	REG("loginuid",   S_IWUSR|S_IRUGO, loginuid),
 #endif
+#ifdef CONFIG_FAULT_INJECTION
+	REG("make-it-fail", S_IRUGO|S_IWUSR, fault_inject),
+#endif
 };
 
 static int proc_tgid_base_readdir(struct file * filp,
@@ -2062,6 +2124,9 @@ static struct pid_entry tid_base_stuff[]
 #ifdef CONFIG_AUDITSYSCALL
 	REG("loginuid",  S_IWUSR|S_IRUGO, loginuid),
 #endif
+#ifdef CONFIG_FAULT_INJECTION
+	REG("make-it-fail", S_IRUGO|S_IWUSR, fault_inject),
+#endif
 };
 
 static int proc_tid_base_readdir(struct file * filp,
Index: work-fault-inject/include/linux/sched.h
===================================================================
--- work-fault-inject.orig/include/linux/sched.h
+++ work-fault-inject/include/linux/sched.h
@@ -1051,6 +1051,9 @@ struct task_struct {
 #ifdef	CONFIG_TASK_DELAY_ACCT
 	struct task_delay_info *delays;
 #endif
+#ifdef CONFIG_FAULT_INJECTION
+	int make_it_fail;
+#endif
 };
 
 static inline pid_t process_group(struct task_struct *tsk)
Index: work-fault-inject/include/linux/fault-inject.h
===================================================================
--- work-fault-inject.orig/include/linux/fault-inject.h
+++ work-fault-inject/include/linux/fault-inject.h
@@ -17,6 +17,7 @@ struct fault_attr {
 	atomic_t times;
 	atomic_t space;
 	unsigned long verbose;
+	u32 task_filter;
 
 	unsigned long count;
 
@@ -30,6 +31,7 @@ struct fault_attr {
 		struct dentry *times_file;
 		struct dentry *space_file;
 		struct dentry *verbose_file;
+		struct dentry *task_filter_file;
 	} entries;
 
 #endif
Index: work-fault-inject/lib/fault-inject.c
===================================================================
--- work-fault-inject.orig/lib/fault-inject.c
+++ work-fault-inject/lib/fault-inject.c
@@ -5,6 +5,7 @@
 #include <linux/types.h>
 #include <linux/fs.h>
 #include <linux/module.h>
+#include <linux/interrupt.h>
 #include <linux/fault-inject.h>
 
 int setup_fault_attr(struct fault_attr *attr, char *str)
@@ -58,6 +59,11 @@ static void fail_dump(struct fault_attr 
 		dump_stack();
 }
 
+static int fail_task(struct fault_attr *attr, struct task_struct *task)
+{
+	return !in_interrupt() && task->make_it_fail;
+}
+
 /*
  * This code is stolen from failmalloc-1.0
  * http://www.nongnu.org/failmalloc/
@@ -65,6 +71,9 @@ static void fail_dump(struct fault_attr 
 
 int should_fail(struct fault_attr *attr, ssize_t size)
 {
+	if (attr->task_filter && !fail_task(attr, current))
+		return 0;
+
 	if (atomic_read(&max_failures(attr)) == 0)
 		return 0;
 
@@ -155,6 +164,10 @@ void cleanup_fault_attr_entries(struct f
 			debugfs_remove(attr->entries.verbose_file);
 			attr->entries.verbose_file = NULL;
 		}
+		if (attr->entries.task_filter_file) {
+			debugfs_remove(attr->entries.task_filter_file);
+			attr->entries.task_filter_file = NULL;
+		}
 		debugfs_remove(attr->entries.dir);
 		attr->entries.dir = NULL;
 	}
@@ -198,6 +211,12 @@ int init_fault_attr_entries(struct fault
 		goto fail;
 	attr->entries.verbose_file = file;
 
+	file = debugfs_create_bool("task-filter", mode, dir,
+				   &attr->task_filter);
+	if (!file)
+		goto fail;
+	attr->entries.task_filter_file = file;
+
 	return 0;
 fail:
 	cleanup_fault_attr_entries(attr);

--

[patch 7/7] stacktrace filtering for fault-injection capabilities

[Akinobu Mita]

[-- Attachment #1: module-filter.patch --]
[-- Type: text/plain, Size: 6112 bytes --]

From: Akinobu Mita <akinobu.mita@gmail.com>

This patch provides stacktrace filtering feature.
The stacktrace filter allows failing only for the caller you are
interested in.

stacktrace filter is enabled by setting the value of
/debugfs/*/stacktrace-depth more than 0.
and specify the range of the virtual address
by the /debugfs/*/address-start and /debugfs/*/address-end

Please see the example that demostrates how to inject slab allocation
failures only for a specific module
in Documentation/fault-injection/fault-injection.txt

Cc: Valdis.Kletnieks@vt.edu
Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: Don Mullis <dwm@meer.net>

 include/linux/fault-inject.h |    7 ++
 lib/Kconfig.debug            |    2 
 lib/fault-inject.c           |  111 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 120 insertions(+)

Index: work-fault-inject/lib/fault-inject.c
===================================================================
--- work-fault-inject.orig/lib/fault-inject.c
+++ work-fault-inject/lib/fault-inject.c
@@ -6,6 +6,9 @@
 #include <linux/fs.h>
 #include <linux/module.h>
 #include <linux/interrupt.h>
+#include <linux/unwind.h>
+#include <linux/stacktrace.h>
+#include <linux/kallsyms.h>
 #include <linux/fault-inject.h>
 
 int setup_fault_attr(struct fault_attr *attr, char *str)
@@ -64,6 +67,82 @@ static int fail_task(struct fault_attr *
 	return !in_interrupt() && task->make_it_fail;
 }
 
+static int fail_any_address(struct fault_attr *attr)
+{
+	return (attr->address_start == 0 && attr->address_end == ULONG_MAX);
+}
+
+#ifdef CONFIG_STACK_UNWIND
+
+static asmlinkage int fail_stacktrace_callback(struct unwind_frame_info *info,
+						void *arg)
+{
+	int depth;
+	struct fault_attr *attr = arg;
+
+	for (depth = 0; depth < attr->stacktrace_depth
+			&& unwind(info) == 0 && UNW_PC(info); depth++) {
+		if (arch_unw_user_mode(info))
+			break;
+		if (attr->address_start <= UNW_PC(info) &&
+			       UNW_PC(info) < attr->address_end)
+			return 1;
+	}
+	return 0;
+}
+
+static int fail_stacktrace(struct fault_attr *attr)
+{
+	struct unwind_frame_info info;
+
+	return unwind_init_running(&info, fail_stacktrace_callback, attr);
+}
+
+#elif defined(CONFIG_STACKTRACE)
+
+#define MAX_STACK_TRACE_DEPTH 10
+
+static int fail_stacktrace(struct fault_attr *attr)
+{
+	struct stack_trace trace;
+	int depth = attr->stacktrace_depth;
+	unsigned long entries[MAX_STACK_TRACE_DEPTH];
+	int n;
+
+	if (depth == 0)
+		return 0;
+
+	trace.nr_entries = 0;
+	trace.entries = entries;
+	trace.max_entries = (depth < MAX_STACK_TRACE_DEPTH) ?
+				depth : MAX_STACK_TRACE_DEPTH;
+	trace.skip = 1;
+	trace.all_contexts = 0;
+
+	save_stack_trace(&trace, NULL);
+	for (n = 0; n < trace.nr_entries; n++)
+		if (attr->address_start <= entries[n] &&
+			       entries[n] < attr->address_end)
+			return 1;
+	return 0;
+}
+
+#else
+
+static inline int fail_stacktrace(struct fault_attr *attr)
+{
+	static int firsttime = 1;
+
+	if (firsttime) {
+		printk(KERN_WARNING
+		"This architecture does not implement save_stack_trace()"\n");
+		firsttime = 0;
+	}
+	return 0;
+}
+
+#endif
+
 /*
  * This code is stolen from failmalloc-1.0
  * http://www.nongnu.org/failmalloc/
@@ -74,6 +153,9 @@ int should_fail(struct fault_attr *attr,
 	if (attr->task_filter && !fail_task(attr, current))
 		return 0;
 
+	if (!fail_any_address(attr) && !fail_stacktrace(attr))
+		return 0;
+
 	if (atomic_read(&max_failures(attr)) == 0)
 		return 0;
 
@@ -168,6 +250,18 @@ void cleanup_fault_attr_entries(struct f
 			debugfs_remove(attr->entries.task_filter_file);
 			attr->entries.task_filter_file = NULL;
 		}
+		if (attr->entries.stacktrace_depth_file) {
+			debugfs_remove(attr->entries.stacktrace_depth_file);
+			attr->entries.stacktrace_depth_file = NULL;
+		}
+		if (attr->entries.address_start_file) {
+			debugfs_remove(attr->entries.address_start_file);
+			attr->entries.address_start_file = NULL;
+		}
+		if (attr->entries.address_end_file) {
+			debugfs_remove(attr->entries.address_end_file);
+			attr->entries.address_end_file = NULL;
+		}
 		debugfs_remove(attr->entries.dir);
 		attr->entries.dir = NULL;
 	}
@@ -217,6 +311,23 @@ int init_fault_attr_entries(struct fault
 		goto fail;
 	attr->entries.task_filter_file = file;
 
+	file = debugfs_create_ul("stacktrace-depth", mode, dir,
+				   &attr->stacktrace_depth);
+	if (!file)
+		goto fail;
+	attr->entries.stacktrace_depth_file = file;
+
+	file = debugfs_create_ul("address-start", mode, dir,
+				   &attr->address_start);
+	if (!file)
+		goto fail;
+	attr->entries.address_start_file = file;
+
+	file = debugfs_create_ul("address-end", mode, dir, &attr->address_end);
+	if (!file)
+		goto fail;
+	attr->entries.address_end_file = file;
+
 	return 0;
 fail:
 	cleanup_fault_attr_entries(attr);
Index: work-fault-inject/include/linux/fault-inject.h
===================================================================
--- work-fault-inject.orig/include/linux/fault-inject.h
+++ work-fault-inject/include/linux/fault-inject.h
@@ -18,6 +18,9 @@ struct fault_attr {
 	atomic_t space;
 	unsigned long verbose;
 	u32 task_filter;
+	unsigned long stacktrace_depth;
+	unsigned long address_start;
+	unsigned long address_end;
 
 	unsigned long count;
 
@@ -32,6 +35,9 @@ struct fault_attr {
 		struct dentry *space_file;
 		struct dentry *verbose_file;
 		struct dentry *task_filter_file;
+		struct dentry *stacktrace_depth_file;
+		struct dentry *address_start_file;
+		struct dentry *address_end_file;
 	} entries;
 
 #endif
@@ -41,6 +47,7 @@ struct fault_attr {
 	struct fault_attr name = {				\
 		.interval = 1,					\
 		.times = ATOMIC_INIT(1),			\
+		.address_end = ULONG_MAX,			\
 	}
 
 int setup_fault_attr(struct fault_attr *attr, char *str);
Index: work-fault-inject/lib/Kconfig.debug
===================================================================
--- work-fault-inject.orig/lib/Kconfig.debug
+++ work-fault-inject/lib/Kconfig.debug
@@ -472,6 +472,8 @@ config LKDTM
 
 config FAULT_INJECTION
 	bool
+	select STACKTRACE
+	select FRAME_POINTER
 
 config FAILSLAB
 	bool "fault-injection capabilitiy for kmalloc"

--

Re: [patch 3/7] fault-injection capability for kmalloc

[Pekka Enberg]

On 10/12/06, Akinobu Mita <akinobu.mita@gmail.com> wrote:
> From: Akinobu Mita <akinobu.mita@gmail.com>
>
> This patch provides fault-injection capability for kmalloc.

The slab bits look ok to me.

                                      Pekka

Re: [patch 2/7] fault-injection capabilities infrastructure

[Andrew Morton]

On Thu, 12 Oct 2006 16:43:07 +0900
Akinobu Mita <akinobu.mita@gmail.com> wrote:

> From: Akinobu Mita <akinobu.mita@gmail.com>
> 
> This patch provides base functions for implement fault-injection
> capabilities.
> 
> - Lightweight random simulator is taken from crasher module for SUSE kernel

heh, another one.

Please switch over to carta_random32() for now.  Later we'll probably be
removing carta_random32() and adding random32(), but I can take care of
that.


> +#define failure_probability(attr)	(attr)->probability
> +#define failure_interval(attr)		(attr)->interval
> +#define max_failures(attr)		(attr)->times
> +#define current_space(attr)		(attr)->space
> +#define atomic_dec_not_zero(v)		atomic_add_unless((v), -1, 0)

Please remove these macros and simply open-code these operations at each
callsite.

Re: [patch 5/7] fault-injection capability for disk IO

[Andrew Morton]

On Thu, 12 Oct 2006 16:43:10 +0900
Akinobu Mita <akinobu.mita@gmail.com> wrote:

> @@ -3134,6 +3174,9 @@ end_io:
>  		if (unlikely(test_bit(QUEUE_FLAG_DEAD, &q->queue_flags)))
>  			goto end_io;
>  
> +		if (should_fail_request(bio))
> +			goto end_io;
> +

hm, so we simulate IO errors by failing at make_request() time rather than
at end_that_request_last()-time, which is where IO errors would usually be
reported.

If we're testing the filesystem/VFS/etc layers then that's pretty much
equivalent.  But perhaps there's an argument for doing both.

Jens, could you have a think about it please?

Re: [patch 7/7] stacktrace filtering for fault-injection capabilities

[Andrew Morton]

On Thu, 12 Oct 2006 16:43:12 +0900
Akinobu Mita <akinobu.mita@gmail.com> wrote:

> From: Akinobu Mita <akinobu.mita@gmail.com>
> 
> This patch provides stacktrace filtering feature.
> The stacktrace filter allows failing only for the caller you are
> interested in.
> 
> stacktrace filter is enabled by setting the value of
> /debugfs/*/stacktrace-depth more than 0.
> and specify the range of the virtual address
> by the /debugfs/*/address-start and /debugfs/*/address-end
> 
> Please see the example that demostrates how to inject slab allocation
> failures only for a specific module
> in Documentation/fault-injection/fault-injection.txt

I read the documentation but I still don't understand this feature.  What
does the stacktrace actually do?  It gets stored somewhere and displayed
later?  What's it all for?

> --- work-fault-inject.orig/lib/Kconfig.debug
> +++ work-fault-inject/lib/Kconfig.debug
> @@ -472,6 +472,8 @@ config LKDTM
>  
>  config FAULT_INJECTION
>  	bool
> +	select STACKTRACE
> +	select FRAME_POINTER
>  
>  config FAILSLAB
>  	bool "fault-injection capabilitiy for kmalloc"
> 

Is the selection of FRAME_POINTER really needed?  The fancy new unwinder
is supposed to be able to handle frame-pointerless unwinding?

Re: [patch 1/7] documentation and scripts

[Andrew Morton]

On Thu, 12 Oct 2006 16:43:06 +0900
Akinobu Mita <akinobu.mita@gmail.com> wrote:

> +- /debug/*/probability:
> +
> +	likelihood of failure injection, in percent.

The fact that this is a percentage worries me.  When I was playing around
with this sort of thing several years ago I found that even
one-failure-per-thousand was a very high error rate for some testcases. 
This interface would force a minimum failure rate of one-per-hundred, which
is terribly high.

So I wonder if it'd be better to make this have units of "one millionth",
or simply make this tunable "1/(probability of failure)".  So setting it to
1,000,000 gives you one failure per million calls, on average.

Re: [patch 4/7] fault-injection capability for alloc_pages()

[Andrew Morton]

On Thu, 12 Oct 2006 16:43:09 +0900
Akinobu Mita <akinobu.mita@gmail.com> wrote:

> @@ -1058,6 +1097,9 @@ __alloc_pages(gfp_t gfp_mask, unsigned i
>  
>  	might_sleep_if(wait);
>  
> +	if (should_fail_alloc_page(gfp_mask, order))
> +		return NULL;

In previous work I've done on this I've found that allowing
application-initiated allocations to fail is a right pain: all of userspace
gets all unreliable and applications die all the time.

I realise that it's possible to limit the failures to a particular process,
but it's also possible to let the allocations fail for _all_ processes, in
which case this problem will hurt.

What I found was a reasonable fix for this problem was to limit the
failures to those requests which did not have __GFP_HIGHMEM set.  That way,
userspace allocations work, but kernel-internal allocations are subject to
failures.

That might be worth adding as an additional tunable?

Re: [patch 5/7] fault-injection capability for disk IO

[Jens Axboe]

On Thu, Oct 12 2006, Andrew Morton wrote:
> On Thu, 12 Oct 2006 16:43:10 +0900
> Akinobu Mita <akinobu.mita@gmail.com> wrote:
> 
> > @@ -3134,6 +3174,9 @@ end_io:
> >  		if (unlikely(test_bit(QUEUE_FLAG_DEAD, &q->queue_flags)))
> >  			goto end_io;
> >  
> > +		if (should_fail_request(bio))
> > +			goto end_io;
> > +
> 
> hm, so we simulate IO errors by failing at make_request() time rather than
> at end_that_request_last()-time, which is where IO errors would usually be
> reported.
> 
> If we're testing the filesystem/VFS/etc layers then that's pretty much
> equivalent.  But perhaps there's an argument for doing both.
> 
> Jens, could you have a think about it please?

For the io submitter and above layers, it's 100% identical to doing it
at end_that_request_first() time (which I suspect is what you meant). So
the above tests those layers only, which is fine if that is what you
want. When we have timeout timers at the block layer (moved from lower
layers), we could do more fancy stuff and test lower layers of the io
stack as well more easily.

-- 
Jens Axboe

Re: [patch 6/7] process filtering for fault-injection capabilities

[Don Mullis]

On Thu, 2006-10-12 at 16:43 +0900, Akinobu Mita wrote:
> This patch provides process filtering feature.
> The process filter allows failing only permitted processes
> by /proc/<pid>/make-it-fail

Akinobu: Toward the end of the previous round of review, we had 
the following exchange:
        
        On Tue, 2006-09-19 at 17:05 +0800, Akinobu Mita wrote:
        On Mon, Sep 18, 2006 at 10:54:51PM -0700, Don Mullis wrote:
        > > Add functionality to the process_filter variable: A negative argument
        > > injects failures for only for pid==-process_filter, thereby permitting
        > > per-process failures from boot time.
        > > 
        > 
        > Is it better to add new filter for this purpose?
        > Because someone may want to filter by tgid instead of pid.
        > 
        > - positive value is for task->pid
        > - nevative value is for task->tgid
        
        Your idea sounds good to me.


So naturally I'm wondering why the functionality was dropped.
An application I had in mind was to identify which of the boot-time
calls to the slab allocator must not fail but are not yet marked
__GFP_NOFAIL (some experimentation showed that for pid 1 there are
lots of these).

Andrew: Would such an exercise would be worth the effort?

Re: [patch 1/7] documentation and scripts

[Akinobu Mita]

On Thu, Oct 12, 2006 at 02:37:13PM -0700, Andrew Morton wrote:

> So I wonder if it'd be better to make this have units of "one millionth",
> or simply make this tunable "1/(probability of failure)".  So setting it to
> 1,000,000 gives you one failure per million calls, on average.

/debug/*/interval is available for this purpose.
The combination of below commands gives one failure per million calls.

# echo 1000000 > /debug/failslab/interval
# echo 100 > /debug/failslab/probability

Re: [patch 4/7] fault-injection capability for alloc_pages()

[Akinobu Mita]

On Thu, Oct 12, 2006 at 02:40:42PM -0700, Andrew Morton wrote:

> What I found was a reasonable fix for this problem was to limit the
> failures to those requests which did not have __GFP_HIGHMEM set.  That way,
> userspace allocations work, but kernel-internal allocations are subject to
> failures.

This is what I want especally when I ran the script in fault-inject.txt.
This would be quite useful.

Re: [patch 7/7] stacktrace filtering for fault-injection capabilities

[Akinobu Mita]

On Thu, Oct 12, 2006 at 02:20:04PM -0700, Andrew Morton wrote:

> I read the documentation but I still don't understand this feature.  What
> does the stacktrace actually do?  It gets stored somewhere and displayed
> later?  What's it all for?

For example someone may want to inject kmalloc()/kmem_cache_alloc()
failures into only e100 module. they want to inject not only direct
kmalloc() call, but also indirect allocation, too.

- e100_poll --> netif_receive_skb --> packet_rcv_spkt --> skb_clone
  --> kmem_cache_alloc

This patch enables to detect function calls like this by stacktrace
and inject failures. The script
Documentaion/fault-injection/failmodule.sh
helps it.

The range of text section of loaded e100 is expected to be
[/sys/module/e100/sections/.text, /sys/module/e100/sections/.exit.text)

So failmodule.sh stores these values into /debug/failslab/address-start
and /debug/failslab/address-end.

> > --- work-fault-inject.orig/lib/Kconfig.debug
> > +++ work-fault-inject/lib/Kconfig.debug
> > @@ -472,6 +472,8 @@ config LKDTM
> >  
> >  config FAULT_INJECTION
> >  	bool
> > +	select STACKTRACE
> > +	select FRAME_POINTER
> >  
> >  config FAILSLAB
> >  	bool "fault-injection capabilitiy for kmalloc"
> > 
> 
> Is the selection of FRAME_POINTER really needed?  The fancy new unwinder
> is supposed to be able to handle frame-pointerless unwinding?

As I wrote in another reply, There are two type of implementation of
this stacktrace filter.

- using STACKTRACE + FRAME_POINTER
- using new unwinder (STACK_UNWIND)

The stacktrace with using new unwinder without FRAME_POINTER is much
slower than STACKTRACE + FRAME_POINTER.

Re: [patch 7/7] stacktrace filtering for fault-injection capabilities

[Akinobu Mita]

2006/10/14, Akinobu Mita <akinobu.mita@gmail.com>:

> > > --- work-fault-inject.orig/lib/Kconfig.debug
> > > +++ work-fault-inject/lib/Kconfig.debug
> > > @@ -472,6 +472,8 @@ config LKDTM
> > >
> > >  config FAULT_INJECTION
> > >     bool
> > > +   select STACKTRACE
> > > +   select FRAME_POINTER
> > >
> > >  config FAILSLAB
> > >     bool "fault-injection capabilitiy for kmalloc"
> > >
> >
> > Is the selection of FRAME_POINTER really needed?  The fancy new unwinder
> > is supposed to be able to handle frame-pointerless unwinding?
>
> As I wrote in another reply, There are two type of implementation of
> this stacktrace filter.
>
> - using STACKTRACE + FRAME_POINTER
> - using new unwinder (STACK_UNWIND)
>
> The stacktrace with using new unwinder without FRAME_POINTER is much
> slower than STACKTRACE + FRAME_POINTER.
>

Maybe I should drop new unwinder support for now.

Re: [patch 6/7] process filtering for fault-injection capabilities

[Andrew Morton]

On Fri, 13 Oct 2006 10:28:54 -0700
Don Mullis <dwm@meer.net> wrote:

> On Thu, 2006-10-12 at 16:43 +0900, Akinobu Mita wrote:
> > This patch provides process filtering feature.
> > The process filter allows failing only permitted processes
> > by /proc/<pid>/make-it-fail
> 
> Akinobu: Toward the end of the previous round of review, we had 
> the following exchange:
>         
>         On Tue, 2006-09-19 at 17:05 +0800, Akinobu Mita wrote:
>         On Mon, Sep 18, 2006 at 10:54:51PM -0700, Don Mullis wrote:
>         > > Add functionality to the process_filter variable: A negative argument
>         > > injects failures for only for pid==-process_filter, thereby permitting
>         > > per-process failures from boot time.
>         > > 
>         > 
>         > Is it better to add new filter for this purpose?
>         > Because someone may want to filter by tgid instead of pid.
>         > 
>         > - positive value is for task->pid
>         > - nevative value is for task->tgid
>         
>         Your idea sounds good to me.
> 
> 
> So naturally I'm wondering why the functionality was dropped.
> An application I had in mind was to identify which of the boot-time
> calls to the slab allocator must not fail but are not yet marked
> __GFP_NOFAIL (some experimentation showed that for pid 1 there are
> lots of these).
> 
> Andrew: Would such an exercise would be worth the effort?
> 

If we're looking for unchecked boot-time allocation failures then I'd say
no, there's not much point in adding code to check for these.  We tend to
assume that the machine has enough memory to boot the kernel and initial
userspace.

That being said, some boot-time allocations are in code which could also
have been compiled into a module, so we do want to be checking those,
because we do care about memory-allocation failures at modprobe time.  But
we can check for these by building the relevant driver as a module and then
testing it.



And the "if it's positive it's a pid, if it's negative it's a tgid"
interface is rather unpleasant - if we're going to do this we should use
separate control files, or use something like

	echo "pid=1234" > /proc/process_filter
	echo "tgid=4321" > /proc/process_filter

Re: [patch 1/7] documentation and scripts

[Andrew Morton]

On Sat, 14 Oct 2006 02:47:24 +0900
Akinobu Mita <akinobu.mita@gmail.com> wrote:

> On Thu, Oct 12, 2006 at 02:37:13PM -0700, Andrew Morton wrote:
> 
> > So I wonder if it'd be better to make this have units of "one millionth",
> > or simply make this tunable "1/(probability of failure)".  So setting it to
> > 1,000,000 gives you one failure per million calls, on average.
> 
> /debug/*/interval is available for this purpose.
> The combination of below commands gives one failure per million calls.
> 
> # echo 1000000 > /debug/failslab/interval
> # echo 100 > /debug/failslab/probability

Oh.  What are the units of "interval"?

(I find it's nice to put the units in the actual filename if practical -
it's self-documenting)

Re: [patch 7/7] stacktrace filtering for fault-injection capabilities

[Andrew Morton]

On Sat, 14 Oct 2006 03:00:39 +0900
Akinobu Mita <akinobu.mita@gmail.com> wrote:

> On Thu, Oct 12, 2006 at 02:20:04PM -0700, Andrew Morton wrote:
> 
> > I read the documentation but I still don't understand this feature.  What
> > does the stacktrace actually do?  It gets stored somewhere and displayed
> > later?  What's it all for?
> 
> For example someone may want to inject kmalloc()/kmem_cache_alloc()
> failures into only e100 module. they want to inject not only direct
> kmalloc() call, but also indirect allocation, too.
> 
> - e100_poll --> netif_receive_skb --> packet_rcv_spkt --> skb_clone
>   --> kmem_cache_alloc
> 
> This patch enables to detect function calls like this by stacktrace
> and inject failures. The script
> Documentaion/fault-injection/failmodule.sh
> helps it.
> 
> The range of text section of loaded e100 is expected to be
> [/sys/module/e100/sections/.text, /sys/module/e100/sections/.exit.text)
> 
> So failmodule.sh stores these values into /debug/failslab/address-start
> and /debug/failslab/address-end.

Oh I see.  So you walk up the stack and if any caller falls between those
two addresses, we enable the fault-injector.   Fair enough.

Re: [patch 7/7] stacktrace filtering for fault-injection capabilities

[Andrew Morton]

On Sat, 14 Oct 2006 03:12:23 +0900
"Akinobu Mita" <akinobu.mita@gmail.com> wrote:

> 2006/10/14, Akinobu Mita <akinobu.mita@gmail.com>:
> 
> > > > --- work-fault-inject.orig/lib/Kconfig.debug
> > > > +++ work-fault-inject/lib/Kconfig.debug
> > > > @@ -472,6 +472,8 @@ config LKDTM
> > > >
> > > >  config FAULT_INJECTION
> > > >     bool
> > > > +   select STACKTRACE
> > > > +   select FRAME_POINTER
> > > >
> > > >  config FAILSLAB
> > > >     bool "fault-injection capabilitiy for kmalloc"
> > > >
> > >
> > > Is the selection of FRAME_POINTER really needed?  The fancy new unwinder
> > > is supposed to be able to handle frame-pointerless unwinding?
> >
> > As I wrote in another reply, There are two type of implementation of
> > this stacktrace filter.
> >
> > - using STACKTRACE + FRAME_POINTER
> > - using new unwinder (STACK_UNWIND)
> >
> > The stacktrace with using new unwinder without FRAME_POINTER is much
> > slower than STACKTRACE + FRAME_POINTER.
> >
> 
> Maybe I should drop new unwinder support for now.

I'd say it should stay.  It'll be a lot more accurate once everything is
sorted out.  And hopefully the new unwinder will get sped up.

Plus I don't think performance matters a lot here: if you want to test the
e100 driver with fault injection, the whole test would take about three
seconds.  After that, you've exercised every code path in the driver.  So
if enabling range-based fault injection increases that to 300 seconds, it is
still practical.

[patch 1/7] documentation and scripts

[Akinobu Mita]

[-- Attachment #1: doc.patch --]
[-- Type: text/plain, Size: 9378 bytes --]

From: Akinobu Mita <akinobu.mita@gmail.com>

This patch set provides some fault-injection capabilities.

- kmalloc() failures

- alloc_pages() failures

- disk IO errors

We can see what really happens if those failures happen.

In order to enable these fault-injection capabilities:

1. Enable relevant config options (CONFIG_FAILSLAB, CONFIG_PAGE_ALLOC,
   CONFIG_MAKE_REQUEST) and if you want to configure them via debugfs,
   enable CONFIG_FAULT_INJECTION_DEBUG_FS.

2. Build and boot with this kernel

3. Configure fault-injection capabilities behavior by boot option or debugfs

   - Boot option

     failslab=
     fail_page_alloc=
     fail_make_request=

   - Debugfs

     /debug/failslab/*
     /debug/fail_page_alloc/*
     /debug/fail_make_request/*

   Please refer to the Documentation/fault-injection/fault-injection.txt
   for details.

4. See what really happens.

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: Don Mullis <dwm@meer.net>

 Documentation/fault-injection/failcmd.sh          |    4 
 Documentation/fault-injection/failmodule.sh       |   31 +++
 Documentation/fault-injection/fault-injection.txt |  223 ++++++++++++++++++++++
 Documentation/kernel-parameters.txt               |    7 
 4 files changed, 265 insertions(+)

Index: 2.6-rc/Documentation/fault-injection/failcmd.sh
===================================================================
--- /dev/null
+++ 2.6-rc/Documentation/fault-injection/failcmd.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+echo 1 > /proc/self/make-it-fail
+exec $*
Index: 2.6-rc/Documentation/fault-injection/failmodule.sh
===================================================================
--- /dev/null
+++ 2.6-rc/Documentation/fault-injection/failmodule.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+#
+# Usage: failmodule <failname> <modulename> [stacktrace-depth]
+#
+#	<failname>: "failslab", "fail_alloc_page", or "fail_make_request"
+#
+#	<modulename>: module name that you want to inject faults.
+#
+#	[stacktrace-depth]: the maximum number of stacktrace walking allowed
+#
+
+STACKTRACE_DEPTH=5
+if [ $# -gt 2 ]; then
+	STACKTRACE_DEPTH=$3
+fi
+
+if [ ! -d /debug/$1 ]; then
+	echo "Fault-injection $1 does not exist" >&2
+	exit 1
+fi
+if [ ! -d /sys/module/$2 ]; then
+	echo "Module $2 does not exist" >&2
+	exit 1
+fi
+
+# Disable any fault injection
+echo 0 > /debug/$1/stacktrace-depth
+
+echo `cat /sys/module/$2/sections/.text` > /debug/$1/address-start
+echo `cat /sys/module/$2/sections/.exit.text` > /debug/$1/address-end
+echo $STACKTRACE_DEPTH > /debug/$1/stacktrace-depth
Index: 2.6-rc/Documentation/fault-injection/fault-injection.txt
===================================================================
--- /dev/null
+++ 2.6-rc/Documentation/fault-injection/fault-injection.txt
@@ -0,0 +1,223 @@
+Fault injection capabilities infrastructure
+===========================================
+
+See also drivers/md/faulty.c and "every_nth" module option for scsi_debug.
+
+
+Available fault injection capabilities
+--------------------------------------
+
+o failslab
+
+  injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
+
+o fail_page_alloc
+
+  injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
+
+o fail_make_request
+
+  injects disk IO errors on permitted devices by
+  /sys/block/<device>/make-it-fail or
+  /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
+
+Configure fault-injection capabilities behavior
+-----------------------------------------------
+
+o debugfs entries
+
+fault-inject-debugfs kernel module provides some debugfs entries for runtime
+configuration of fault-injection capabilities.
+
+- /debug/*/probability:
+
+	likelihood of failure injection, in percent.
+	Format: <percent>
+
+	Note that one-failure-per-handred is a very high error rate
+	for some testcases. Please set probably=100 and configure
+	/debug/*/interval for such testcases.
+
+- /debug/*/interval:
+
+	specifies the interval between failures, for calls to
+	should_fail() that pass all the other tests.
+
+	Note that if you enable this, by setting interval>1, you will
+	probably want to set probability=100.
+
+- /debug/*/times:
+
+	specifies how many times failures may happen at most.
+	A value of -1 means "no limit".
+
+- /debug/*/space:
+
+	specifies an initial resource "budget", decremented by "size"
+	on each call to should_fail(,size).  Failure injection is
+	suppressed until "space" reaches zero.
+
+- /debug/*/verbose
+
+	Format: { 0 | 1 | 2 }
+	specifies the verbosity of the messages when failure is injected.
+	We default to 0 (no extra messages), setting it to '1' will
+	print only to tell failure happened, '2' will print call trace too -
+	it is useful to debug the problems revealed by fault injection
+	capabilities.
+
+- /debug/*/task-filter:
+
+	Format: { 0 | 1 }
+	A value of '0' disables filtering by process (default).
+	Any positive value limits failures to only processes indicated by
+	/proc/<pid>/make-it-fail==1.
+
+- /debug/*/address-start:
+- /debug/*/address-end:
+
+	specifies the range of virtual addresses tested during
+	stacktrace walking.  Failure is injected only if some caller
+	in the walked stacktrace lies within this range.
+	Default is [0,ULONG_MAX) (whole of virtual address space).
+
+- /debug/*/stacktrace-depth:
+
+	specifies the maximum stacktrace depth walked during search
+	for a caller within [address-start,address-end).
+
+- /debug/failslab/ignore-gfp-highmem:
+- /debug/fail_page_alloc/ignore-gfp-highmem:
+
+	Format: { 0 | 1 }
+	default is 0, setting it to '1' won't inject failures into
+	highmem/user allocations.
+
+- /debug/failslab/ignore-gfp-wait:
+- /debug/fail_page_alloc/ignore-gfp-wait:
+
+	Format: { 0 | 1 }
+	default is 0, setting it to '1' will inject failures
+	only into non-sleep allocations (GFP_ATOMIC allocations).
+
+o Boot option
+
+In order to inject faults while debugfs is not available (early boot time),
+use the boot option:
+
+	failslab=
+	fail_page_alloc=
+	fail_make_request=<interval>,<probability>,<space>,<times>
+
+How to add new fault injection capability
+-----------------------------------------
+
+o #include <linux/fault-inject.h>
+
+o define the fault attributes
+
+  DECLARE_FAULT_INJECTION(name);
+
+  Please see the definition of struct fault_attr in fault-inject.h
+  for details.
+
+o provide the way to configure fault attributes
+
+- boot option
+
+  If you need to enable the fault injection capability from boot time, you can
+  provide boot option to configure it. There is a helper function for it.
+
+  setup_fault_attr(attr, str);
+
+- debugfs entries
+
+  failslab, fail_page_alloc, and fail_make_request use this way.
+  There is a helper function for it.
+
+  init_fault_attr_entries(entries, attr, name);
+  void cleanup_fault_attr_entries(entries);
+
+- module parameters
+
+  If the scope of the fault injection capability is limited to a
+  single kernel module, it is better to provide module parameters to
+  configure the fault attributes.
+
+o add a hook to insert failures
+
+  should_fail() returns 1 when failures should happen.
+
+	should_fail(attr,size);
+
+Application Examples
+--------------------
+
+o inject slab allocation failures into module init/cleanup code
+
+------------------------------------------------------------------------------
+#!/bin/bash
+
+FAILCMD=Documentation/fault-injection/failcmd.sh
+BLACKLIST="root_plug evbug"
+
+FAILNAME=failslab
+echo Y > /debug/$FAILNAME/task-filter
+echo 10 > /debug/$FAILNAME/probability
+echo 100 > /debug/$FAILNAME/interval
+echo -1 > /debug/$FAILNAME/times
+echo 2 > /debug/$FAILNAME/verbose
+echo 1 > /debug/$FAILNAME/ignore-gfp-highmem
+echo 1 > /debug/$FAILNAME/ignore-gfp-wait
+
+blacklist()
+{
+	echo $BLACKLIST | grep $1 > /dev/null 2>&1
+}
+
+oops()
+{
+	dmesg | grep BUG > /dev/null 2>&1 
+}
+
+find /lib/modules/`uname -r` -name '*.ko' -exec basename {} .ko \; |
+	while read i
+	do
+		oops && exit 1
+
+		if ! blacklist $i
+		then
+			echo inserting $i...
+			bash $FAILCMD modprobe $i
+		fi
+	done
+
+lsmod | awk '{ if ($3 == 0) { print $1 } }' |
+	while read i
+	do
+		oops && exit 1
+
+		if ! blacklist $i
+		then
+			echo removing $i...
+			bash $FAILCMD modprobe -r $i
+		fi
+	done
+
+------------------------------------------------------------------------------
+
+o inject slab allocation failures only for a specific module
+
+------------------------------------------------------------------------------
+#!/bin/bash
+
+FAILMOD=Documentation/fault-injection/failmodule.sh
+
+echo injecting errors into the module $1...
+
+modprobe $1
+bash $FAILMOD failslab $1 10
+echo 25 > /debug/failslab/probability
+
+------------------------------------------------------------------------------
+
Index: 2.6-rc/Documentation/kernel-parameters.txt
===================================================================
--- 2.6-rc.orig/Documentation/kernel-parameters.txt
+++ 2.6-rc/Documentation/kernel-parameters.txt
@@ -544,6 +544,13 @@ and is between 256 and 4096 characters. 
 	eurwdt=		[HW,WDT] Eurotech CPU-1220/1410 onboard watchdog.
 			Format: <io>[,<irq>]
 
+	failslab=
+	fail_page_alloc=
+	fail_make_request=[KNL]
+			General fault injection mechanism.
+			Format: <interval>,<probability>,<space>,<times>
+			See also /Documentation/fault-injection/.
+
 	fd_mcs=		[HW,SCSI]
 			See header of drivers/scsi/fd_mcs.c.
 

--