[patch 0/8] fault-injection capabilities (v3)

[patch 0/8] fault-injection capabilities (v3)

[Akinobu Mita]

Changes from last version

- add stacktrace filter
- drop process-filter from boot-time setup
- build fixes from Don Mullis
- add documentation and some scripts

---

This patch set provides some fault-injection capabilities.

- kmalloc() failures

- alloc_pages() failures

- disk IO errors

We can see what really happens if those failures happen.

In order to enable these fault-injection capabilities:

1. Enable relevant config options (CONFIG_FAILSLAB, CONFIG_PAGE_ALLOC,
   CONFIG_MAKE_REQUEST) and runtime configuration kernel module
   (CONFIG_FAULT_INJECTION_DEBUGFS)

2. build and boot with this kernel

3. modprobe fault-inject-debugfs

4. configure fault-injection capabilities behavior by debugfs

For example about kmalloc failures:

/debug/failslab/probability

	specifies how often it should fail in percent.

/debug/failslab/interval

	specifies the interval of failures.

/debug/failslab/times

	specifies how many times failures may happen at most.

/debug/failslab/space

	specifies the size of free space where memory can be allocated
	safely in bytes.

/debug/failslab/process-filter

	specifies whether process filter is enabled or not.
	it allows failing only permitted processes by /proc/<pid>/make-it-fail

/debug/failslab/stacktrace-depth

	specifies the maximum stacktrace depth walking allowed.
	a value '0' means stacktrace filter is disabled.

/debug/failslab/address-start
/debug/failslab/address-end

	specifies the range of virtual address.
	it allows failing only if the stacktrace hits in this range.

5. see what really happens.

[patch 1/8] documentation and scripts

[Akinobu Mita]

[-- Attachment #1: doc.patch --]
[-- Type: text/plain, Size: 6315 bytes --]

This patch provides the documentation and helper/testing scripts for
fault-injection capabilities.

Signed-off-by: Akinobu Mita <mita@miraclelinux.com>

 Documentation/fault-injection/failcmd.sh          |    4 
 Documentation/fault-injection/failmodule.sh       |   32 ++++
 Documentation/fault-injection/fault-injection.txt |  150 ++++++++++++++++++++++
 3 files changed, 186 insertions(+)

Index: work-shouldfail/Documentation/fault-injection/failcmd.sh
===================================================================
--- /dev/null
+++ work-shouldfail/Documentation/fault-injection/failcmd.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+echo 1 > /proc/self/make-it-fail
+exec $*
Index: work-shouldfail/Documentation/fault-injection/failmodule.sh
===================================================================
--- /dev/null
+++ work-shouldfail/Documentation/fault-injection/failmodule.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+#
+# Usage: failmodule <failname> <modulename> [stacktrace-depth]
+#
+#	<failname>: "failslab", "fail_alloc_page", or "fail_make_request"
+#
+#	<modulename>: module name that you want to inject faults.
+#
+#	[stacktrace-depth]: the maximum number of stacktrace walking allowed
+#
+
+STACKTRACE_DEPTH=5
+if [ $# -gt 2 ]; then
+	STACKTRACE_DEPTH=$3
+fi
+
+if [ ! -d /debug/$1 ]; then
+	echo "Fault-injection $1 does not exist" >&2
+	exit 1
+fi
+if [ ! -d /sys/module/$2 ]; then
+	echo "Module $2 does not exist" >&2
+	exit 1
+fi
+
+# Disable stacktrace filter at first
+echo 0 > /debug/$1/address-end
+echo 1 > /debug/$1/stacktrace-depth
+
+echo `cat /sys/module/$2/sections/.text` > /debug/$1/address-start
+echo `cat /sys/module/$2/sections/.exit.text` > /debug/$1/address-end
+echo $STACKTRACE_DEPTH > /debug/$1/stacktrace-depth
Index: work-shouldfail/Documentation/fault-injection/fault-injection.txt
===================================================================
--- /dev/null
+++ work-shouldfail/Documentation/fault-injection/fault-injection.txt
@@ -0,0 +1,150 @@
+Fault injection capabilities infrastructure
+===========================================
+
+Available fault injection capabilities
+--------------------------------------
+
+o failslab
+
+  injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
+
+o fail_page_alloc
+
+  injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
+
+o fail_make_request
+
+  injects disk IO errors on permitted devices by /sys/block/<device>/make-it-fail
+  or /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
+
+Configure fault-injection capabilities behavior
+-----------------------------------------------
+
+Example for failslab:
+
+o debugfs entries
+
+fault-inject-debugfs kernel module provides some debugfs entries for runtime
+configuration for fault-injection capabilities.
+
+- /debug/failslab/probability:
+
+	specifies how often it should fail in percent.
+
+- /debug/failslab/interval:
+
+	specifies the interval of failures.
+
+- /debug/failslab/times:
+
+	specifies how many times failures may happen at most.
+
+- /debug/failslab/space:
+
+	specifies the size of free space where memory can be allocated
+	safely in bytes.
+
+- /debug/failslab/process-filter:
+
+	specifies whether the process filter is enabled or not.
+	It allows failing only permitted processes by /proc/<pid>/make-it-fail
+
+- /debug/failslab/stacktrace-depth:
+
+	specifies the maximum stacktrace depth walking allowed.
+	A value '0' means stacktrace filter is disabled.
+
+- /debug/failslab/address-start:
+- /debug/failslab/address-end:
+
+	specifies the range of virtual address.
+	It allows failing only if the stacktrace hits in this range.
+
+o Boot option
+
+In order to inject faults while debugfs is not available (early boot time),
+We can use boot option.
+
+- failslab=<interval>,<probability>,<space>,<times>
+
+How to add new fault injection capability
+-----------------------------------------
+
+o #include <linux/fault-inject.h>
+
+o define the fault attributes
+
+  DEFINE_FAULT_INJECTION(name);
+
+  Please see the definition of struct fault_attr in fault-inject.h
+  for the detail.
+
+o provide the way to configure fault attributes
+
+- boot option
+
+  If you need to enable the fault injection capability ealier boot time,
+  you can provide boot option to configure it. There is a helper function for it.
+
+  setup_fault_attr(attr, str);
+
+- module parameters
+
+  If the scope of the fault injection capability is limited by a kernel module,
+  It is better to provide module parameters to configure the member of fault
+  attributes.
+
+- debugfs entries
+
+  failslab, fail_page_alloc, and fail_make_request use this way.
+
+  But now there is no helper functions to provides debugfs entries for
+  fault injection capabilities. Please refer to lib/fault-inject-debugfs.c
+  to know how to do. And please try not to add new one into
+  fault-inject-debugfs module.
+
+  Because failslab, fail_page_alloc, and fail_make_request are used ealier
+  boot time before debugfs is available and the slab allocator,
+  the page allocator, and the block layer cannot be built as module.
+
+o add a hook to insert failures
+
+  should_fail() returns 1 when failures should happen.
+
+  should_fail(attr);
+
+Tests
+-----
+
+o inject slab allocation failures into module init/cleanup code
+
+------------------------------------------------------------------------------
+#!/bin/bash
+
+FAILCMD=Documentation/fault-injection/failcmd.sh
+
+modprobe fault-inject-debugfs
+echo Y > /debug/failslab/process-filter
+echo 25 > /debug/failslab/probability
+
+find /lib/modules/`uname -r` -name '*.ko' -exec basename {} .ko \; \
+	| while read i; do bash $FAILCMD modprobe $i;done
+
+lsmod | awk '{ if ($3 == 0) { system("bash $FAILCMD modprobe -r " $1) } }'
+
+------------------------------------------------------------------------------
+
+o inject slab allocation failures only for a specific module
+
+------------------------------------------------------------------------------
+#!/bin/bash
+
+FAILMOD=Documentation/fault-injection/failmodule.sh
+
+modprobe fault-inject-debugfs
+modprobe your-test-module
+bash $FAILMOD failslab your-test-module 10
+echo 25 > /debug/failslab/probability
+
+------------------------------------------------------------------------------
+

--

[patch 2/8] fault-injection capabilities infrastructure

[Akinobu Mita]

[-- Attachment #1: should-fail.patch --]
[-- Type: text/plain, Size: 4629 bytes --]

This patch provides some functions for implement fault-injection
capabilities.

- Lightweight random simulator is taken from crasher module for SUSE kernel
- The function should_fail() is taken from failmalloc-1.0
  (http://www.nongnu.org/failmalloc/)

Cc: okuji@enbug.org
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>

 include/linux/fault-inject.h |   41 ++++++++++++++++++++
 lib/Kconfig.debug            |    4 ++
 lib/Makefile                 |    1 
 lib/fault-inject.c           |   84 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 130 insertions(+)

Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -368,3 +368,7 @@ config RCU_TORTURE_TEST
 	  at boot time (you probably don't).
 	  Say M if you want the RCU torture tests to build as a module.
 	  Say N if you are unsure.
+
+config FAULT_INJECTION
+	bool
+
Index: work-shouldfail/lib/Makefile
===================================================================
--- work-shouldfail.orig/lib/Makefile
+++ work-shouldfail/lib/Makefile
@@ -52,6 +52,7 @@ obj-$(CONFIG_SMP) += percpu_counter.o
 obj-$(CONFIG_AUDIT_GENERIC) += audit.o
 
 obj-$(CONFIG_SWIOTLB) += swiotlb.o
+obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o
 
 hostprogs-y	:= gen_crc32table
 clean-files	:= crc32table.h
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
--- /dev/null
+++ work-shouldfail/include/linux/fault-inject.h
@@ -0,0 +1,41 @@
+#ifndef _LINUX_FAULT_INJECT_H
+#define _LINUX_FAULT_INJECT_H
+
+#ifdef CONFIG_FAULT_INJECTION
+
+#include <linux/types.h>
+#include <asm/atomic.h>
+
+struct fault_attr {
+
+	/* how often it should fail in percent. */
+	unsigned long probability;
+
+	/* the interval of failures. */
+	unsigned long interval;
+
+	/*
+	 * how many times failures may happen at most.
+	 * A value of '-1' means infinity.
+	 */
+	atomic_t times;
+
+	/*
+	 * the size of free space where memory can be allocated safely.
+	 * A value of '0' means infinity.
+	 */
+	atomic_t space;
+
+	unsigned long count;
+};
+
+#define DEFINE_FAULT_ATTR(name) \
+	struct fault_attr name = { .times = ATOMIC_INIT(-1), }
+
+int setup_fault_attr(struct fault_attr *attr, char *str);
+void should_fail_srandom(unsigned long entropy);
+int should_fail(struct fault_attr *attr, ssize_t size);
+
+#endif /* CONFIG_FAULT_INJECTION */
+
+#endif /* _LINUX_FAULT_INJECT_H */
Index: work-shouldfail/lib/fault-inject.c
===================================================================
--- /dev/null
+++ work-shouldfail/lib/fault-inject.c
@@ -0,0 +1,84 @@
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/random.h>
+#include <linux/stat.h>
+#include <linux/types.h>
+#include <linux/fs.h>
+#include <linux/module.h>
+#include <linux/fault-inject.h>
+
+int setup_fault_attr(struct fault_attr *attr, char *str)
+{
+	unsigned long probability;
+	unsigned long interval;
+	int times;
+	int space;
+
+	/* "<interval>,<probability>,<space>,<times>" */
+	if (sscanf(str, "%lu,%lu,%d,%d",
+			&interval, &probability, &space, &times) < 4) {
+		printk(KERN_WARNING "SHOULD_FAIL: failed to parse arguments\n");
+		return 0;
+	}
+
+	attr->probability = probability;
+	attr->interval = interval;
+	atomic_set(&attr->times, times);
+	atomic_set(&attr->space, space);
+
+	return 1;
+}
+
+#define failure_probability(attr)	(attr)->probability
+#define failure_interval(attr)		(attr)->interval
+#define max_failures(attr)		(attr)->times
+#define current_space(attr)		(attr)->space
+#define atomic_dec_not_zero(v)		atomic_add_unless((v), -1, 0)
+
+static unsigned long rand_seed = 152L;
+
+static unsigned long should_fail_random(void)
+{
+	rand_seed = rand_seed * 690690L+1;
+	return rand_seed ^ jiffies;
+}
+
+void should_fail_srandom(unsigned long entropy)
+{
+	rand_seed ^= entropy;
+	should_fail_random();
+}
+
+/*
+ * This code is stolen from failmalloc-1.0
+ * http://www.nongnu.org/failmalloc/
+ */
+
+int should_fail(struct fault_attr *attr, ssize_t size)
+{
+	if (atomic_read(&max_failures(attr)) == 0)
+		return 0;
+
+	if (atomic_read(&current_space(attr)) > size) {
+		atomic_sub(size, &current_space(attr));
+		return 0;
+	}
+
+	if (failure_interval(attr) > 1) {
+		attr->count++;
+		if (attr->count % failure_interval(attr))
+			return 0;
+	}
+
+	if (failure_probability(attr) > should_fail_random() % 100)
+		goto fail;
+
+	return 0;
+
+fail:
+
+	if (atomic_read(&max_failures(attr)) != -1)
+		atomic_dec_not_zero(&max_failures(attr));
+
+	return 1;
+}

--

[patch 3/8] fault-injection capability for kmalloc

[Akinobu Mita]

[-- Attachment #1: failslab.patch --]
[-- Type: text/plain, Size: 3246 bytes --]

This patch provides fault-injection capability for kmalloc.

Boot option:

failslab=<interval>,<probability>,<space>,<times>

	<interval> -- specifies the interval of failures.

	<probability> -- specifies how often it should fail in percent.

	<space> -- specifies the size of free space where memory can be
		   allocated safely in bytes.

	<times> -- specifies how many times failures may happen at most.

Example:

	failslab=10,100,0,-1

slab allocation (kmalloc(), kmem_cache_alloc(),..) fails once per 10 times.

Cc: Pekka Enberg <penberg@cs.helsinki.fi>
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>

 include/linux/fault-inject.h |    4 ++++
 lib/Kconfig.debug            |    7 +++++++
 mm/slab.c                    |   34 ++++++++++++++++++++++++++++++++++
 3 files changed, 45 insertions(+)

Index: work-shouldfail/mm/slab.c
===================================================================
--- work-shouldfail.orig/mm/slab.c
+++ work-shouldfail/mm/slab.c
@@ -108,6 +108,7 @@
 #include	<linux/mempolicy.h>
 #include	<linux/mutex.h>
 #include	<linux/rtmutex.h>
+#include	<linux/fault-inject.h>
 
 #include	<asm/uaccess.h>
 #include	<asm/cacheflush.h>
@@ -2963,11 +2964,44 @@ static void *cache_alloc_debugcheck_afte
 #define cache_alloc_debugcheck_after(a,b,objp,d) (objp)
 #endif
 
+#ifdef CONFIG_FAILSLAB
+
+static DEFINE_FAULT_ATTR(failslab_attr);
+
+static int __init setup_failslab(char *str)
+{
+	should_fail_srandom(jiffies);
+	return setup_fault_attr(&failslab_attr, str);
+}
+__setup("failslab=", setup_failslab);
+
+struct fault_attr *failslab = &failslab_attr;
+EXPORT_SYMBOL_GPL(failslab);
+
+static int should_failslab(struct kmem_cache *cachep, gfp_t flags)
+{
+	if ((flags & __GFP_NOFAIL) || cachep == &cache_cache)
+		return 0;
+	return should_fail(failslab, obj_size(cachep));
+}
+
+#else
+
+static inline int should_failslab(struct kmem_cache *cachep, gfp_t flags)
+{
+	return 0;
+}
+
+#endif
+
 static inline void *____cache_alloc(struct kmem_cache *cachep, gfp_t flags)
 {
 	void *objp;
 	struct array_cache *ac;
 
+	if (should_failslab(cachep, flags))
+		return NULL;
+
 #ifdef CONFIG_NUMA
 	if (unlikely(current->flags & (PF_SPREAD_SLAB | PF_MEMPOLICY))) {
 		objp = alternate_node_alloc(cachep, flags);
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -372,3 +372,10 @@ config RCU_TORTURE_TEST
 config FAULT_INJECTION
 	bool
 
+config FAILSLAB
+	bool "fault-injection capabilitiy for kmalloc"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION
+	help
+	  This option provides fault-injection capabilitiy for kmalloc.
+
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
--- work-shouldfail.orig/include/linux/fault-inject.h
+++ work-shouldfail/include/linux/fault-inject.h
@@ -36,6 +36,10 @@ int setup_fault_attr(struct fault_attr *
 void should_fail_srandom(unsigned long entropy);
 int should_fail(struct fault_attr *attr, ssize_t size);
 
+#ifdef CONFIG_FAILSLAB
+extern struct fault_attr *failslab;
+#endif
+
 #endif /* CONFIG_FAULT_INJECTION */
 
 #endif /* _LINUX_FAULT_INJECT_H */

--

[patch 4/8] fault-injection capability for alloc_pages()

[Akinobu Mita]

[-- Attachment #1: fail_alloc_pages.patch --]
[-- Type: text/plain, Size: 3205 bytes --]

This patch provides fault-injection capability for alloc_pages()

Boot option:

fail_page_alloc=<interval>,<probability>,<space>,<times>

	<interval> -- specifies the interval of failures.

	<probability> -- specifies how often it should fail in percent.

	<space> -- specifies the size of free space where memory can be
		   allocated safely in pages.

	<times> -- specifies how many times failures may happen at most.

Example:

	fail_page_alloc=10,100,0,-1

The page allocation (alloc_pages(), ...) fails once per 10 times.

Signed-off-by: Akinobu Mita <mita@miraclelinux.com>

 include/linux/fault-inject.h |    3 +++
 lib/Kconfig.debug            |    7 +++++++
 mm/page_alloc.c              |   35 +++++++++++++++++++++++++++++++++++
 3 files changed, 45 insertions(+)

Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -379,3 +379,10 @@ config FAILSLAB
 	help
 	  This option provides fault-injection capabilitiy for kmalloc.
 
+config FAIL_PAGE_ALLOC
+	bool "fault-injection capabilitiy for alloc_pages()"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION
+	help
+	  This option provides fault-injection capabilitiy for alloc_pages().
+
Index: work-shouldfail/mm/page_alloc.c
===================================================================
--- work-shouldfail.orig/mm/page_alloc.c
+++ work-shouldfail/mm/page_alloc.c
@@ -37,6 +37,7 @@
 #include <linux/vmalloc.h>
 #include <linux/mempolicy.h>
 #include <linux/stop_machine.h>
+#include <linux/fault-inject.h>
 
 #include <asm/tlbflush.h>
 #include <asm/div64.h>
@@ -903,6 +904,37 @@ get_page_from_freelist(gfp_t gfp_mask, u
 	return page;
 }
 
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+
+static DEFINE_FAULT_ATTR(fail_page_alloc_attr);
+
+static int __init setup_fail_page_alloc(char *str)
+{
+	should_fail_srandom(jiffies);
+	return setup_fault_attr(&fail_page_alloc_attr, str);
+}
+__setup("fail_page_alloc=", setup_fail_page_alloc);
+
+struct fault_attr *fail_page_alloc = &fail_page_alloc_attr;
+EXPORT_SYMBOL_GPL(fail_page_alloc);
+
+static int should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
+{
+	if (gfp_mask & __GFP_NOFAIL)
+		return 0;
+
+	return should_fail(fail_page_alloc, 1 << order);
+}
+
+#else
+
+static inline int should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
+{
+	return 0;
+}
+
+#endif
+
 /*
  * This is the 'heart' of the zoned buddy allocator.
  */
@@ -921,6 +953,9 @@ __alloc_pages(gfp_t gfp_mask, unsigned i
 
 	might_sleep_if(wait);
 
+	if (should_fail_alloc_page(gfp_mask, order))
+		return NULL;
+
 restart:
 	z = zonelist->zones;  /* the list of zones suitable for gfp_mask */
 
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
--- work-shouldfail.orig/include/linux/fault-inject.h
+++ work-shouldfail/include/linux/fault-inject.h
@@ -39,6 +39,9 @@ int should_fail(struct fault_attr *attr,
 #ifdef CONFIG_FAILSLAB
 extern struct fault_attr *failslab;
 #endif
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+extern struct fault_attr *fail_page_alloc;
+#endif
 
 #endif /* CONFIG_FAULT_INJECTION */
 

--

[patch 5/8] fault-injection capability for disk IO

[Akinobu Mita]

[-- Attachment #1: fail_make_request.patch --]
[-- Type: text/plain, Size: 6655 bytes --]

This patch provides fault-injection capability for disk IO.

Boot option:

fail_make_request=<probability>,<interval>,<space>,<times>

	<interval> -- specifies the interval of failures.

	<probability> -- specifies how often it should fail in percent.

	<space> -- specifies the size of free space where disk IO can be issued
		   safely in bytes.

	<times> -- specifies how many times failures may happen at most.

Example:

	fail_make_request=10,100,0,-1
	echo 1 > /sys/blocks/hda/hda1/make-it-fail

generic_make_request() on /dev/hda1 fails once per 10 times.

Cc: Jens Axboe <axboe@suse.de>
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>

 block/genhd.c                |   31 +++++++++++++++++++++++++++++++
 block/ll_rw_blk.c            |   35 +++++++++++++++++++++++++++++++++++
 fs/partitions/check.c        |   27 +++++++++++++++++++++++++++
 include/linux/fault-inject.h |    3 +++
 include/linux/genhd.h        |    4 ++++
 lib/Kconfig.debug            |    7 +++++++
 6 files changed, 107 insertions(+)

Index: work-shouldfail/block/ll_rw_blk.c
===================================================================
--- work-shouldfail.orig/block/ll_rw_blk.c
+++ work-shouldfail/block/ll_rw_blk.c
@@ -28,6 +28,7 @@
 #include <linux/interrupt.h>
 #include <linux/cpu.h>
 #include <linux/blktrace_api.h>
+#include <linux/fault-inject.h>
 
 /*
  * for max sense size
@@ -2993,6 +2994,37 @@ static void handle_bad_sector(struct bio
 	set_bit(BIO_EOF, &bio->bi_flags);
 }
 
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static DEFINE_FAULT_ATTR(fail_make_request_attr);
+
+static int __init setup_fail_make_request(char *str)
+{
+	should_fail_srandom(jiffies);
+	return setup_fault_attr(&fail_make_request_attr, str);
+}
+__setup("fail_make_request=", setup_fail_make_request);
+
+struct fault_attr *fail_make_request = &fail_make_request_attr;
+EXPORT_SYMBOL_GPL(fail_make_request);
+
+static int should_fail_request(struct bio *bio)
+{
+	if ((bio->bi_bdev->bd_disk->flags & GENHD_FL_FAIL) ||
+	    (bio->bi_bdev->bd_part && bio->bi_bdev->bd_part->make_it_fail))
+		return should_fail(fail_make_request, bio->bi_size);
+
+	return 0;
+}
+#else
+
+static inline int should_fail_request(struct bio *bio)
+{
+	return 0;
+}
+
+#endif
+
 /**
  * generic_make_request: hand a buffer to its device driver for I/O
  * @bio:  The bio describing the location in memory and on the device.
@@ -3077,6 +3109,9 @@ end_io:
 		if (unlikely(test_bit(QUEUE_FLAG_DEAD, &q->queue_flags)))
 			goto end_io;
 
+		if (should_fail_request(bio))
+			goto end_io;
+
 		/*
 		 * If this device has partitions, remap block n
 		 * of partition p to block n+start(p) of the disk.
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -386,3 +386,10 @@ config FAIL_PAGE_ALLOC
 	help
 	  This option provides fault-injection capabilitiy for alloc_pages().
 
+config FAIL_MAKE_REQUEST
+	bool "fault-injection capabilitiy for disk IO"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION 
+	help
+	  This option provides fault-injection capabilitiy to disk IO.
+
Index: work-shouldfail/block/genhd.c
===================================================================
--- work-shouldfail.orig/block/genhd.c
+++ work-shouldfail/block/genhd.c
@@ -412,6 +412,34 @@ static struct disk_attribute disk_attr_s
 	.show	= disk_stats_read
 };
 
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static ssize_t disk_fail_store(struct gendisk * disk,
+			       const char *buf, size_t count)
+{
+	int i;
+
+	if (count > 0 && sscanf(buf, "%d", &i) > 0) {
+		if (i == 0)
+			disk->flags &= ~GENHD_FL_FAIL;
+		else
+			disk->flags |= GENHD_FL_FAIL;
+	}
+
+	return count;
+}
+static ssize_t disk_fail_read(struct gendisk * disk, char *page)
+{
+	return sprintf(page, "%d\n", disk->flags & GENHD_FL_FAIL ? 1 : 0);
+}
+static struct disk_attribute disk_attr_fail = {
+	.attr = {.name = "make-it-fail", .mode = S_IRUGO | S_IWUSR },
+	.store	= disk_fail_store,
+	.show	= disk_fail_read
+};
+
+#endif
+
 static struct attribute * default_attrs[] = {
 	&disk_attr_uevent.attr,
 	&disk_attr_dev.attr,
@@ -419,6 +447,9 @@ static struct attribute * default_attrs[
 	&disk_attr_removable.attr,
 	&disk_attr_size.attr,
 	&disk_attr_stat.attr,
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	&disk_attr_fail.attr,
+#endif
 	NULL,
 };
 
Index: work-shouldfail/include/linux/genhd.h
===================================================================
--- work-shouldfail.orig/include/linux/genhd.h
+++ work-shouldfail/include/linux/genhd.h
@@ -81,6 +81,9 @@ struct hd_struct {
 	struct kobject *holder_dir;
 	unsigned ios[2], sectors[2];	/* READs and WRITEs */
 	int policy, partno;
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	int make_it_fail;
+#endif
 };
 
 #define GENHD_FL_REMOVABLE			1
@@ -88,6 +91,7 @@ struct hd_struct {
 #define GENHD_FL_CD				8
 #define GENHD_FL_UP				16
 #define GENHD_FL_SUPPRESS_PARTITION_INFO	32
+#define GENHD_FL_FAIL				64
 
 struct disk_stats {
 	unsigned long sectors[2];	/* READs and WRITEs */
Index: work-shouldfail/fs/partitions/check.c
===================================================================
--- work-shouldfail.orig/fs/partitions/check.c
+++ work-shouldfail/fs/partitions/check.c
@@ -265,12 +265,39 @@ static struct part_attribute part_attr_s
 	.show	= part_stat_read
 };
 
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static ssize_t part_fail_store(struct hd_struct * p,
+			       const char *buf, size_t count)
+{
+	int i;
+
+	if (count > 0 && sscanf(buf, "%d", &i) > 0)
+		p->make_it_fail = (i == 0) ? 0 : 1;
+
+	return count;
+}
+static ssize_t part_fail_read(struct hd_struct * p, char *page)
+{
+	return sprintf(page, "%d\n", p->make_it_fail);
+}
+static struct part_attribute part_attr_fail = {
+	.attr = {.name = "make-it-fail", .mode = S_IRUGO | S_IWUSR },
+	.store	= part_fail_store,
+	.show	= part_fail_read
+};
+
+#endif
+
 static struct attribute * default_attrs[] = {
 	&part_attr_uevent.attr,
 	&part_attr_dev.attr,
 	&part_attr_start.attr,
 	&part_attr_size.attr,
 	&part_attr_stat.attr,
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	&part_attr_fail.attr,
+#endif
 	NULL,
 };
 
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
--- work-shouldfail.orig/include/linux/fault-inject.h
+++ work-shouldfail/include/linux/fault-inject.h
@@ -42,6 +42,9 @@ extern struct fault_attr *failslab;
 #ifdef CONFIG_FAIL_PAGE_ALLOC
 extern struct fault_attr *fail_page_alloc;
 #endif
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+extern struct fault_attr *fail_make_request;
+#endif
 
 #endif /* CONFIG_FAULT_INJECTION */
 

--

[patch 6/8] debugfs entries for configuration

[Akinobu Mita]

[-- Attachment #1: knobs.patch --]
[-- Type: text/plain, Size: 6668 bytes --]

This kernel module provides debugfs entries to enable to configure
fault-injection capabilities for failslab, fail_page_alloc, and
fail_make_request.

The slab allocator, the page allocator, and the block layer are initalized
before debugfs is available. and failslab, fail_page_alloc, and
fail_make_request are also enabled at the same time.
So I put the initalization and cleanup for these debugfs entries into
this kernel module.

This module provides the following entries so that we can configure
by writing these files.

/debug/
|-- fail_make_request
|   |-- interval
|   |-- probability
|   |-- space
|   `-- times
|-- fail_page_alloc
|   |-- interval
|   |-- probability
|   |-- space
|   `-- times
`-- failslab
    |-- interval
    |-- probability
    |-- space
    `-- times

Signed-off-by: Akinobu Mita <mita@miraclelinux.com>

 lib/Kconfig.debug          |    8 ++
 lib/Makefile               |    1 
 lib/fault-inject-debugfs.c |  179 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 188 insertions(+)

Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -393,3 +393,11 @@ config FAIL_MAKE_REQUEST
 	help
 	  This option provides fault-injection capabilitiy to disk IO.
 
+config FAULT_INJECTION_DEBUGFS 
+	tristate "runtime configuration for fault-injection capabilities"
+	depends on DEBUG_KERNEL && SYSFS && FAULT_INJECTION
+	select DEBUG_FS
+	help
+	  This option provides kernel module that provides runtime
+	  configuration interface by debugfs.
+
Index: work-shouldfail/lib/fault-inject-debugfs.c
===================================================================
--- /dev/null
+++ work-shouldfail/lib/fault-inject-debugfs.c
@@ -0,0 +1,179 @@
+#include <linux/module.h>
+#include <linux/fault-inject.h>
+#include <linux/debugfs.h>
+
+struct fault_attr_entries {
+	struct dentry *dir;
+	struct dentry *probability_file;
+	struct dentry *interval_file;
+	struct dentry *times_file;
+	struct dentry *space_file;
+	struct dentry *process_filter_file;
+};
+
+static void debugfs_ul_set(void *data, u64 val)
+{
+	*(unsigned long *)data = val;
+}
+
+static u64 debugfs_ul_get(void *data)
+{
+	return *(unsigned long *)data;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_ul, debugfs_ul_get, debugfs_ul_set, "%llu\n");
+
+static struct dentry *debugfs_create_ul(const char *name, mode_t mode,
+				struct dentry *parent, unsigned long *value)
+{
+	return debugfs_create_file(name, mode, parent, value, &fops_ul);
+}
+
+static void debugfs_atomic_t_set(void *data, u64 val)
+{
+	atomic_set((atomic_t *)data, val);
+}
+
+static u64 debugfs_atomic_t_get(void *data)
+{
+	return atomic_read((atomic_t *)data);
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_atomic_t, debugfs_atomic_t_get,
+			debugfs_atomic_t_set, "%lld\n");
+
+static struct dentry *debugfs_create_atomic_t(const char *name, mode_t mode,
+				struct dentry *parent, atomic_t *value)
+{
+	return debugfs_create_file(name, mode, parent, value, &fops_atomic_t);
+}
+
+static void cleanup_fault_attr_entries(struct fault_attr_entries *entries)
+{
+	if (entries->dir) {
+		if (entries->probability_file) {
+			debugfs_remove(entries->probability_file);
+			entries->probability_file = NULL;
+		}
+		if (entries->interval_file) {
+			debugfs_remove(entries->interval_file);
+			entries->interval_file = NULL;
+		}
+		if (entries->times_file) {
+			debugfs_remove(entries->times_file);
+			entries->times_file = NULL;
+		}
+		if (entries->space_file) {
+			debugfs_remove(entries->space_file);
+			entries->space_file = NULL;
+		}
+		if (entries->process_filter_file) {
+			debugfs_remove(entries->process_filter_file);
+			entries->process_filter_file = NULL;
+		}
+		debugfs_remove(entries->dir);
+		entries->dir = NULL;
+	}
+}
+
+static int init_fault_attr_entries(struct fault_attr_entries *entries,
+				struct fault_attr *attr, const char *name)
+{
+	mode_t mode = S_IFREG | S_IRUSR | S_IWUSR;
+	struct dentry *dir;
+	struct dentry *file;
+
+	memset(entries, 0, sizeof(*entries));
+
+	dir = debugfs_create_dir(name, NULL);
+	if (!dir)
+		goto fail;
+	entries->dir = dir;
+
+	file = debugfs_create_ul("probability", mode, dir, &attr->probability);
+	if (!file)
+		goto fail;
+	entries->probability_file = file;
+
+	file = debugfs_create_ul("interval", mode, dir, &attr->interval);
+	if (!file)
+		goto fail;
+	entries->interval_file = file;
+
+	file = debugfs_create_atomic_t("times", mode, dir, &attr->times);
+	if (!file)
+		goto fail;
+	entries->times_file = file;
+
+	file = debugfs_create_atomic_t("space", mode, dir, &attr->space);
+	if (!file)
+		goto fail;
+	entries->space_file = file;
+
+	file = debugfs_create_bool("process-filter", mode, dir,
+				   &attr->process_filter);
+	if (!file)
+		goto fail;
+	entries->process_filter_file = file;
+
+	return 0;
+fail:
+	cleanup_fault_attr_entries(entries);
+	return -ENOMEM;
+}
+
+#ifdef CONFIG_FAILSLAB
+static struct fault_attr_entries failslab_entries;
+#endif
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+static struct fault_attr_entries fail_page_alloc_entries;
+#endif
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+static struct fault_attr_entries fail_make_request_entries;
+#endif
+
+static void cleanup_entries(void)
+{
+#ifdef CONFIG_FAILSLAB
+	cleanup_fault_attr_entries(&failslab_entries);
+#endif
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+	cleanup_fault_attr_entries(&fail_page_alloc_entries);
+#endif
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	cleanup_fault_attr_entries(&fail_make_request_entries);
+#endif
+}
+
+static int init_entries(void)
+{
+	int err;
+
+#ifdef CONFIG_FAILSLAB
+	err = init_fault_attr_entries(&failslab_entries, failslab, "failslab");
+	if (err)
+		goto fail;
+#endif
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+	err = init_fault_attr_entries(&fail_page_alloc_entries, fail_page_alloc,
+				     "fail_page_alloc");
+	if (err)
+		goto fail;
+#endif
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	err = init_fault_attr_entries(&fail_make_request_entries,
+				     fail_make_request, "fail_make_request");
+	if (err)
+		goto fail;
+#endif
+
+	return 0;
+fail:
+	cleanup_entries();
+
+	return err;
+}
+
+module_init(init_entries);
+module_exit(cleanup_entries);
+MODULE_LICENSE("GPL");
Index: work-shouldfail/lib/Makefile
===================================================================
--- work-shouldfail.orig/lib/Makefile
+++ work-shouldfail/lib/Makefile
@@ -53,6 +53,7 @@ obj-$(CONFIG_AUDIT_GENERIC) += audit.o
 
 obj-$(CONFIG_SWIOTLB) += swiotlb.o
 obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o
+obj-$(CONFIG_FAULT_INJECTION_DEBUGFS) += fault-inject-debugfs.o
 
 hostprogs-y	:= gen_crc32table
 clean-files	:= crc32table.h

--

[patch 7/8] process filtering for fault-injection capabilities

[Akinobu Mita]

[-- Attachment #1: process-filter.patch --]
[-- Type: text/plain, Size: 5872 bytes --]

This patch provides process filtering feature.
The process filter allows failing only permitted processes
by /proc/<pid>/make-it-fail

Please see the example that demostrates how to inject slab allocation
failures into module init/cleanup code
in Documentation/fault-injection/fault-injection.txt

Signed-off-by: Akinobu Mita <mita@miraclelinux.com>

 fs/proc/base.c               |   77 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/fault-inject.h |    3 +
 include/linux/sched.h        |    3 +
 lib/fault-inject.c           |   13 +++++++
 4 files changed, 96 insertions(+)

Index: work-shouldfail/fs/proc/base.c
===================================================================
--- work-shouldfail.orig/fs/proc/base.c
+++ work-shouldfail/fs/proc/base.c
@@ -138,6 +138,9 @@ enum pid_directory_inos {
 #endif
 	PROC_TGID_OOM_SCORE,
 	PROC_TGID_OOM_ADJUST,
+#ifdef CONFIG_FAULT_INJECTION
+	PROC_TGID_FAULT_INJECTION,
+#endif
 	PROC_TID_INO,
 	PROC_TID_STATUS,
 	PROC_TID_MEM,
@@ -181,6 +184,9 @@ enum pid_directory_inos {
 #endif
 	PROC_TID_OOM_SCORE,
 	PROC_TID_OOM_ADJUST,
+#ifdef CONFIG_FAULT_INJECTION
+	PROC_TID_FAULT_INJECTION,
+#endif
 
 	/* Add new entries before this */
 	PROC_TID_FD_DIR = 0x8000,	/* 0x8000-0xffff */
@@ -240,6 +246,9 @@ static struct pid_entry tgid_base_stuff[
 #ifdef CONFIG_AUDITSYSCALL
 	E(PROC_TGID_LOGINUID, "loginuid", S_IFREG|S_IWUSR|S_IRUGO),
 #endif
+#ifdef CONFIG_FAULT_INJECTION
+	E(PROC_TGID_FAULT_INJECTION, "make-it-fail", S_IFREG|S_IWUSR|S_IRUGO),
+#endif
 	{0,0,NULL,0}
 };
 static struct pid_entry tid_base_stuff[] = {
@@ -282,6 +291,9 @@ static struct pid_entry tid_base_stuff[]
 #ifdef CONFIG_AUDITSYSCALL
 	E(PROC_TID_LOGINUID, "loginuid", S_IFREG|S_IWUSR|S_IRUGO),
 #endif
+#ifdef CONFIG_FAULT_INJECTION
+	E(PROC_TID_FAULT_INJECTION, "make-it-fail", S_IFREG|S_IWUSR|S_IRUGO),
+#endif
 	{0,0,NULL,0}
 };
 
@@ -992,6 +1004,65 @@ static struct file_operations proc_login
 };
 #endif
 
+#ifdef CONFIG_FAULT_INJECTION
+static ssize_t proc_fault_inject_read(struct file * file, char __user * buf,
+				      size_t count, loff_t *ppos)
+{
+	struct task_struct *task = get_proc_task(file->f_dentry->d_inode);
+	char buffer[PROC_NUMBUF];
+	size_t len;
+	int make_it_fail;
+	loff_t __ppos = *ppos;
+
+	if (!task)
+		return -ESRCH;
+	make_it_fail = task->make_it_fail;
+	put_task_struct(task);
+
+	len = snprintf(buffer, sizeof(buffer), "%i\n", make_it_fail);
+	if (__ppos >= len)
+		return 0;
+	if (count > len-__ppos)
+		count = len-__ppos;
+	if (copy_to_user(buf, buffer + __ppos, count))
+		return -EFAULT;
+	*ppos = __ppos + count;
+	return count;
+}
+
+static ssize_t proc_fault_inject_write(struct file * file,
+			const char __user * buf, size_t count, loff_t *ppos)
+{
+	struct task_struct *task;
+	char buffer[PROC_NUMBUF], *end;
+	int make_it_fail;
+
+	if (!capable(CAP_SYS_RESOURCE))
+		return -EPERM;
+	memset(buffer, 0, sizeof(buffer));
+	if (count > sizeof(buffer) - 1)
+		count = sizeof(buffer) - 1;
+	if (copy_from_user(buffer, buf, count))
+		return -EFAULT;
+	make_it_fail = simple_strtol(buffer, &end, 0);
+	if (*end == '\n')
+		end++;
+	task = get_proc_task(file->f_dentry->d_inode);
+	if (!task)
+		return -ESRCH;
+	task->make_it_fail = make_it_fail;
+	put_task_struct(task);
+	if (end - buffer == 0)
+		return -EIO;
+	return end - buffer;
+}
+
+static struct file_operations proc_fault_inject_operations = {
+	.read		= proc_fault_inject_read,
+	.write		= proc_fault_inject_write,
+};
+#endif
+
 #ifdef CONFIG_SECCOMP
 static ssize_t seccomp_read(struct file *file, char __user *buf,
 			    size_t count, loff_t *ppos)
@@ -1834,6 +1905,12 @@ static struct dentry *proc_pident_lookup
 			inode->i_fop = &proc_loginuid_operations;
 			break;
 #endif
+#ifdef CONFIG_FAULT_INJECTION
+		case PROC_TID_FAULT_INJECTION:
+		case PROC_TGID_FAULT_INJECTION:
+			inode->i_fop = &proc_fault_inject_operations;
+			break;
+#endif
 		default:
 			printk("procfs: impossible type (%d)",p->type);
 			iput(inode);
Index: work-shouldfail/include/linux/sched.h
===================================================================
--- work-shouldfail.orig/include/linux/sched.h
+++ work-shouldfail/include/linux/sched.h
@@ -996,6 +996,9 @@ struct task_struct {
 #ifdef	CONFIG_TASK_DELAY_ACCT
 	struct task_delay_info *delays;
 #endif
+#ifdef CONFIG_FAULT_INJECTION
+	int make_it_fail;
+#endif
 };
 
 static inline pid_t process_group(struct task_struct *tsk)
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
--- work-shouldfail.orig/include/linux/fault-inject.h
+++ work-shouldfail/include/linux/fault-inject.h
@@ -27,6 +27,9 @@ struct fault_attr {
 	atomic_t space;
 
 	unsigned long count;
+
+	/* A value of '0' means process filter is disabled. */
+	u32 process_filter;
 };
 
 #define DEFINE_FAULT_ATTR(name) \
Index: work-shouldfail/lib/fault-inject.c
===================================================================
--- work-shouldfail.orig/lib/fault-inject.c
+++ work-shouldfail/lib/fault-inject.c
@@ -5,6 +5,7 @@
 #include <linux/types.h>
 #include <linux/fs.h>
 #include <linux/module.h>
+#include <linux/interrupt.h>
 #include <linux/fault-inject.h>
 
 int setup_fault_attr(struct fault_attr *attr, char *str)
@@ -49,6 +50,15 @@ void should_fail_srandom(unsigned long e
 	should_fail_random();
 }
 
+static int fail_process(struct fault_attr *attr, struct task_struct *task)
+{
+	/* process filter is disabled */
+	if (!attr->process_filter)
+		return 1;
+
+	return !in_interrupt() && task->make_it_fail;
+}
+
 /*
  * This code is stolen from failmalloc-1.0
  * http://www.nongnu.org/failmalloc/
@@ -56,6 +66,9 @@ void should_fail_srandom(unsigned long e
 
 int should_fail(struct fault_attr *attr, ssize_t size)
 {
+	if (!fail_process(attr, current))
+		return 0;
+
 	if (atomic_read(&max_failures(attr)) == 0)
 		return 0;
 

--

[patch 8/8] stacktrace filtering for fault-injection capabilities

[Akinobu Mita]

[-- Attachment #1: module-filter.patch --]
[-- Type: text/plain, Size: 6116 bytes --]

This patch provides stacktrace filtering feature.
The stacktrace filter allows failing only for the caller you are
interested in.

stacktrace filter is enabled by setting the value of
/debugfs/*/stacktrace-depth more than 0.
and specify the range of the virtual address
by the /debugfs/*/address-start and /debugfs/*/address-end

Please see the example that demostrates how to inject slab allocation
failures only for a specific module
in Documentation/fault-injection/fault-injection.txt

Cc: Valdis.Kletnieks@vt.edu
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>

 include/linux/fault-inject.h |   15 +++++++++
 lib/Kconfig.debug            |    2 +
 lib/fault-inject-debugfs.c   |   32 +++++++++++++++++++
 lib/fault-inject.c           |   71 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 120 insertions(+)

Index: work-shouldfail/lib/fault-inject.c
===================================================================
--- work-shouldfail.orig/lib/fault-inject.c
+++ work-shouldfail/lib/fault-inject.c
@@ -6,6 +6,9 @@
 #include <linux/fs.h>
 #include <linux/module.h>
 #include <linux/interrupt.h>
+#include <linux/unwind.h>
+#include <linux/stacktrace.h>
+#include <linux/kallsyms.h>
 #include <linux/fault-inject.h>
 
 int setup_fault_attr(struct fault_attr *attr, char *str)
@@ -59,6 +62,72 @@ static int fail_process(struct fault_att
 	return !in_interrupt() && task->make_it_fail;
 }
 
+#ifdef CONFIG_STACK_UNWIND
+
+static asmlinkage int fail_stacktrace_callback(struct unwind_frame_info *info,
+						void *arg)
+{
+	int depth;
+	struct fault_attr *attr = arg;
+
+	for (depth = 0; depth < attr->stacktrace_depth
+			&& unwind(info) == 0 && UNW_PC(info); depth++) {
+		if (arch_unw_user_mode(info))
+			break;
+		if (attr->address_start <= UNW_PC(info) &&
+			       UNW_PC(info) < attr->address_end)
+			return 1;
+	}
+	return 0;
+}
+
+static int fail_stacktrace(struct fault_attr *attr)
+{
+	struct unwind_frame_info info;
+
+	/* stacktrace filter is disabled */
+	if (attr->stacktrace_depth == 0)
+		return 1;
+
+	return unwind_init_running(&info, fail_stacktrace_callback, attr);
+}
+
+#elif defined(CONFIG_STACKTRACE)
+
+#include <linux/stacktrace.h>
+
+#define MAX_STACK_TRACE_DEPTH 10
+
+static int fail_stacktrace(struct fault_attr *attr)
+{
+	struct stack_trace trace;
+	int depth = attr->stacktrace_depth;
+	unsigned long entries[MAX_STACK_TRACE_DEPTH];
+	int n;
+
+	/* stacktrace filter is disabled */
+	if (depth == 0)
+		return 1;
+
+	trace.nr_entries = 0;
+	trace.entries = entries;
+	trace.max_entries = (depth < MAX_STACK_TRACE_DEPTH) ?
+				depth : MAX_STACK_TRACE_DEPTH;
+
+	save_stack_trace(&trace, NULL, 0, 1);
+	for (n = 0; n < trace.nr_entries; n++)
+		if (attr->address_start <= entries[n] &&
+			       entries[n] < attr->address_end)
+			return 1;
+	return 0;
+}
+
+#else
+
+#define fail_stacktrace(attr)	(0)
+
+#endif
+
 /*
  * This code is stolen from failmalloc-1.0
  * http://www.nongnu.org/failmalloc/
@@ -68,6 +137,8 @@ int should_fail(struct fault_attr *attr,
 {
 	if (!fail_process(attr, current))
 		return 0;
+	if (!fail_stacktrace(attr))
+		return 0;
 
 	if (atomic_read(&max_failures(attr)) == 0)
 		return 0;
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
--- work-shouldfail.orig/include/linux/fault-inject.h
+++ work-shouldfail/include/linux/fault-inject.h
@@ -30,6 +30,21 @@ struct fault_attr {
 
 	/* A value of '0' means process filter is disabled. */
 	u32 process_filter;
+
+	/*
+	 * maximam number of stacktrace depth walking allowed
+	 * A value of '0' means stacktrace filter is disabled.
+	 */
+	unsigned long stacktrace_depth;
+
+	/*
+	 * If stacktrace_depth is enabled, it allows failing only when it
+	 * has been called from the virtual addresses in the range
+	 * 'address_start' to 'address_end-1'
+	 */ 
+	unsigned long address_start;
+	unsigned long address_end;
+
 };
 
 #define DEFINE_FAULT_ATTR(name) \
Index: work-shouldfail/lib/fault-inject-debugfs.c
===================================================================
--- work-shouldfail.orig/lib/fault-inject-debugfs.c
+++ work-shouldfail/lib/fault-inject-debugfs.c
@@ -9,6 +9,9 @@ struct fault_attr_entries {
 	struct dentry *times_file;
 	struct dentry *space_file;
 	struct dentry *process_filter_file;
+	struct dentry *stacktrace_depth_file;
+	struct dentry *address_start_file;
+	struct dentry *address_end_file;
 };
 
 static void debugfs_ul_set(void *data, u64 val)
@@ -71,6 +74,18 @@ static void cleanup_fault_attr_entries(s
 			debugfs_remove(entries->process_filter_file);
 			entries->process_filter_file = NULL;
 		}
+		if (entries->stacktrace_depth_file) {
+			debugfs_remove(entries->stacktrace_depth_file);
+			entries->stacktrace_depth_file = NULL;
+		}
+		if (entries->address_start_file) {
+			debugfs_remove(entries->address_start_file);
+			entries->address_start_file = NULL;
+		}
+		if (entries->address_end_file) {
+			debugfs_remove(entries->address_end_file);
+			entries->address_end_file = NULL;
+		}
 		debugfs_remove(entries->dir);
 		entries->dir = NULL;
 	}
@@ -116,6 +131,23 @@ static int init_fault_attr_entries(struc
 		goto fail;
 	entries->process_filter_file = file;
 
+	file = debugfs_create_ul("stacktrace-depth", mode, dir,
+				   &attr->stacktrace_depth);
+	if (!file)
+		goto fail;
+	entries->stacktrace_depth_file = file;
+
+	file = debugfs_create_ul("address-start", mode, dir,
+				   &attr->address_start);
+	if (!file)
+		goto fail;
+	entries->address_start_file = file;
+
+	file = debugfs_create_ul("address-end", mode, dir, &attr->address_end);
+	if (!file)
+		goto fail;
+	entries->address_end_file = file;
+
 	return 0;
 fail:
 	cleanup_fault_attr_entries(entries);
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -371,6 +371,8 @@ config RCU_TORTURE_TEST
 
 config FAULT_INJECTION
 	bool
+	select STACKTRACE
+	select FRAME_POINTER
 
 config FAILSLAB
 	bool "fault-injection capabilitiy for kmalloc"

--

Re: [patch 1/8] documentation and scripts

[Don Mullis]

Add reference to similar functionality in drivers/md/faulty.c .

Clarify debugfs descriptions.

Add an entry for fault-injection to 
Documentation/kernel-parameters.txt .


Signed-off-by: Don Mullis <dwm@meer.net>

---
 Documentation/fault-injection/fault-injection.txt |   85 ++++++++++------------
 Documentation/kernel-parameters.txt               |    7 +
 2 files changed, 47 insertions(+), 45 deletions(-)

Index: linux-2.6.17/Documentation/fault-injection/fault-injection.txt
===================================================================
--- linux-2.6.17.orig/Documentation/fault-injection/fault-injection.txt
+++ linux-2.6.17/Documentation/fault-injection/fault-injection.txt
@@ -1,6 +1,9 @@
 Fault injection capabilities infrastructure
 ===========================================
 
+See also drivers/md/faulty.c .
+
+
 Available fault injection capabilities
 --------------------------------------
 
@@ -14,58 +17,49 @@ o fail_page_alloc
 
 o fail_make_request
 
-  injects disk IO errors on permitted devices by /sys/block/<device>/make-it-fail
-  or /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
+  injects disk IO errors on permitted devices by
+  /sys/block/<device>/make-it-fail or
+  /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
 
 Configure fault-injection capabilities behavior
 -----------------------------------------------
 
-Example for failslab:
-
 o debugfs entries
 
 fault-inject-debugfs kernel module provides some debugfs entries for runtime
-configuration for fault-injection capabilities.
-
-- /debug/failslab/probability:
+configuration of fault-injection capabilities.
 
-	specifies how often it should fail in percent.
+- /debug/*/probability:
 
-- /debug/failslab/interval:
-
-	specifies the interval of failures.
-
-- /debug/failslab/times:
-
-	specifies how many times failures may happen at most.
+	likelihood of failure injection, in percent.
 
-- /debug/failslab/space:
+- /debug/*/interval:
 
-	specifies the size of free space where memory can be allocated
-	safely in bytes.
+	specifies the interval between failures, for calls to
+	should_fail() that pass all the other tests.
 
-- /debug/failslab/process-filter:
+	Note that if you enable this, by setting interval>1, you will
+	probably want to set probability=100.
 
-	specifies whether the process filter is enabled or not.
-	It allows failing only permitted processes by /proc/<pid>/make-it-fail
+- /debug/*/times:
 
-- /debug/failslab/stacktrace-depth:
-
-	specifies the maximum stacktrace depth walking allowed.
-	A value '0' means stacktrace filter is disabled.
+	specifies how many times failures may happen at most.
+	A value of -1 means "no limit".
 
-- /debug/failslab/address-start:
-- /debug/failslab/address-end:
+- /debug/*/space:
 
-	specifies the range of virtual address.
-	It allows failing only if the stacktrace hits in this range.
+	specifies an initial resource "budget", decremented by "size"
+	on each call to should_fail(,size).  Failure injection is
+	suppressed until "space" reaches zero.
 
 o Boot option
 
 In order to inject faults while debugfs is not available (early boot time),
-We can use boot option.
+use the boot option:
 
-- failslab=<interval>,<probability>,<space>,<times>
+    failslab=
+    fail_page_alloc=
+    fail_make_request=<interval>,<probability>,<space>,<times>
 
 How to add new fault injection capability
 -----------------------------------------
@@ -77,44 +71,42 @@ o define the fault attributes
   DEFINE_FAULT_INJECTION(name);
 
   Please see the definition of struct fault_attr in fault-inject.h
-  for the detail.
+  for details.
 
 o provide the way to configure fault attributes
 
 - boot option
 
-  If you need to enable the fault injection capability ealier boot time,
+  If you need to enable the fault injection capability from boot time,
   you can provide boot option to configure it. There is a helper function for it.
 
   setup_fault_attr(attr, str);
 
 - module parameters
 
-  If the scope of the fault injection capability is limited by a kernel module,
-  It is better to provide module parameters to configure the member of fault
-  attributes.
+  If the scope of the fault injection capability is limited to a
+  single kernel module, it is better to provide module parameters to
+  configure the fault attributes.
 
 - debugfs entries
 
   failslab, fail_page_alloc, and fail_make_request use this way.
-
-  But now there is no helper functions to provides debugfs entries for
+  But now there is no helper functions to provide debugfs entries for
   fault injection capabilities. Please refer to lib/fault-inject-debugfs.c
   to know how to do. And please try not to add new one into
   fault-inject-debugfs module.
-
-  Because failslab, fail_page_alloc, and fail_make_request are used ealier
-  boot time before debugfs is available and the slab allocator,
-  the page allocator, and the block layer cannot be built as module.
+  Because failslab, fail_page_alloc, and fail_make_request are used from
+  boot time, before debugfs is available, the slab allocator,
+  the page allocator, and the block layer cannot be built as modules.
 
 o add a hook to insert failures
 
   should_fail() returns 1 when failures should happen.
 
-  should_fail(attr);
+	should_fail(attr,size);
 
-Tests
------
+Application Examples
+--------------------
 
 o inject slab allocation failures into module init/cleanup code
 
@@ -148,3 +140,6 @@ echo 25 > /debug/failslab/probability
 
 ------------------------------------------------------------------------------
 
+
+
+
Index: linux-2.6.17/Documentation/kernel-parameters.txt
===================================================================
--- linux-2.6.17.orig/Documentation/kernel-parameters.txt
+++ linux-2.6.17/Documentation/kernel-parameters.txt
@@ -544,6 +544,13 @@ running once the system is up.
 	eurwdt=		[HW,WDT] Eurotech CPU-1220/1410 onboard watchdog.
 			Format: <io>[,<irq>]
 
+	failslab=
+	fail_page_alloc=
+	fail_make_request=[KNL]
+			General fault injection mechanism.
+			Format: <interval>,<probability>,<space>,<times>
+			See also /Documentation/fault-injection/.
+
 	fd_mcs=		[HW,SCSI]
 			See header of drivers/scsi/fd_mcs.c.
 

Re: [patch 2/8] fault-injection capabilities infrastructure

[Don Mullis]

Replace individual structure element comments with reference
to Documentation/fault-injection/fault-injection.txt

Init "interval" to 1 (smallest useful value).
Init "times" to 1 rather than -1 (infinity), for fewer 
accidental system lockups.


Signed-off-by: Don Mullis <dwm@meer.net>

---
 include/linux/fault-inject.h |   20 +++++---------------
 1 file changed, 5 insertions(+), 15 deletions(-)

Index: linux-2.6.17/include/linux/fault-inject.h
===================================================================
--- linux-2.6.17.orig/include/linux/fault-inject.h
+++ linux-2.6.17/include/linux/fault-inject.h
@@ -6,31 +6,21 @@
 #include <linux/types.h>
 #include <asm/atomic.h>
 
+/*
+ * For explanation of the elements of this struct, see
+ * Documentation/fault-injection/fault-injection.txt
+ */
 struct fault_attr {
-
-	/* how often it should fail in percent. */
 	unsigned long probability;
-
-	/* the interval of failures. */
 	unsigned long interval;
-
-	/*
-	 * how many times failures may happen at most.
-	 * A value of '-1' means infinity.
-	 */
 	atomic_t times;
-
-	/*
-	 * the size of free space where memory can be allocated safely.
-	 * A value of '0' means infinity.
-	 */
 	atomic_t space;
 
 	unsigned long count;
 };
 
 #define DEFINE_FAULT_ATTR(name) \
-	struct fault_attr name = { .times = ATOMIC_INIT(-1), }
+	struct fault_attr name = { .interval=1, .times = ATOMIC_INIT(1), }
 
 int setup_fault_attr(struct fault_attr *attr, char *str);
 void should_fail_srandom(unsigned long entropy);

Re: [patch 6/8] debugfs entries for configuration

[Don Mullis]

Factor out process-filter functionality, move later in series to
process-filter-specific patch.


Signed-off-by: Don Mullis <dwm@meer.net>

---
 lib/fault-inject-debugfs.c |   11 -----------
 1 file changed, 11 deletions(-)

Index: linux-2.6.17/lib/fault-inject-debugfs.c
===================================================================
--- linux-2.6.17.orig/lib/fault-inject-debugfs.c
+++ linux-2.6.17/lib/fault-inject-debugfs.c
@@ -8,7 +8,6 @@ struct fault_attr_entries {
 	struct dentry *interval_file;
 	struct dentry *times_file;
 	struct dentry *space_file;
-	struct dentry *process_filter_file;
 };
 
 static void debugfs_ul_set(void *data, u64 val)
@@ -67,10 +66,6 @@ static void cleanup_fault_attr_entries(s
 			debugfs_remove(entries->space_file);
 			entries->space_file = NULL;
 		}
-		if (entries->process_filter_file) {
-			debugfs_remove(entries->process_filter_file);
-			entries->process_filter_file = NULL;
-		}
 		debugfs_remove(entries->dir);
 		entries->dir = NULL;
 	}
@@ -110,12 +105,6 @@ static int init_fault_attr_entries(struc
 		goto fail;
 	entries->space_file = file;
 
-	file = debugfs_create_bool("process-filter", mode, dir,
-				   &attr->process_filter);
-	if (!file)
-		goto fail;
-	entries->process_filter_file = file;
-
 	return 0;
 fail:
 	cleanup_fault_attr_entries(entries);

Re: [patch 7/8] process filtering for fault-injection capabilities

[Don Mullis]

Add functionality to the process_filter variable: A negative argument
injects failures for only for pid==-process_filter, thereby permitting
per-process failures from boot time.

Restore process-filter arg to kernel command line.

Reintroduce process_filter_file addition to debugfs,
factored out earlier in series.

Add printk, called upon each failure injection.


Signed-off-by: Don Mullis <dwm@meer.net>

---
 Documentation/fault-injection/fault-injection.txt |    9 +++++++
 include/linux/fault-inject.h                      |    1 
 lib/fault-inject-debugfs.c                        |   11 ++++++++
 lib/fault-inject.c                                |   27 ++++++++++++++++------
 4 files changed, 40 insertions(+), 8 deletions(-)

Index: linux-2.6.17/include/linux/fault-inject.h
===================================================================
--- linux-2.6.17.orig/include/linux/fault-inject.h
+++ linux-2.6.17/include/linux/fault-inject.h
@@ -15,11 +15,8 @@ struct fault_attr {
 	unsigned long interval;
 	atomic_t times;
 	atomic_t space;
-
 	unsigned long count;
-
-	/* A value of '0' means process filter is disabled. */
-	u32 process_filter;
+	atomic_t process_filter;
 };
 
 #define DEFINE_FAULT_ATTR(name) \
Index: linux-2.6.17/lib/fault-inject.c
===================================================================
--- linux-2.6.17.orig/lib/fault-inject.c
+++ linux-2.6.17/lib/fault-inject.c
@@ -14,11 +14,12 @@ int setup_fault_attr(struct fault_attr *
 	unsigned long interval;
 	int times;
 	int space;
+	int process_filter;
 
-	/* "<interval>,<probability>,<space>,<times>" */
-	if (sscanf(str, "%lu,%lu,%d,%d",
-			&interval, &probability, &space, &times) < 4) {
-		printk(KERN_WARNING "SHOULD_FAIL: failed to parse arguments\n");
+	/* "<interval>,<probability>,<process-filter>,<space>,<times>" */
+	if (sscanf(str, "%lu,%lu,%d,%d,%d",
+			&interval, &probability, &process_filter, &space, &times) < 4) {
+		printk(KERN_WARNING "FAULT_INJECTION: failed to parse arguments\n");
 		return 0;
 	}
 
@@ -26,6 +27,7 @@ int setup_fault_attr(struct fault_attr *
 	attr->interval = interval;
 	atomic_set(&attr->times, times);
 	atomic_set(&attr->space, space);
+	atomic_set(&attr->process_filter, process_filter);
 
 	return 1;
 }
@@ -53,10 +55,20 @@ void should_fail_srandom(unsigned long e
 static int fail_process(struct fault_attr *attr, struct task_struct *task)
 {
 	/* process filter is disabled */
-	if (!attr->process_filter)
+	if (atomic_read(&attr->process_filter)==0)
 		return 1;
 
-	return !in_interrupt() && task->make_it_fail;
+	/* controlled by /proc/<pid>/make-it-fail, but boolean not set */
+	if (atomic_read(&attr->process_filter)>0 && !task->make_it_fail)
+		return 0;
+
+	/* enabled for single pid, but no match with pid */
+	if (atomic_read(&attr->process_filter)<0 &&
+	    -atomic_read(&attr->process_filter)!=task->pid)
+		return 0;
+
+	/* XX  Are we sure we never want calls from interrupt context to fail? */
+	return !in_interrupt();
 }
 
 /*
@@ -89,7 +101,8 @@ int should_fail(struct fault_attr *attr,
 	return 0;
 
 fail:
-
+	printk(KERN_NOTICE "FAULT_INJECTION: forcing a failure, pid==%d\n",
+	       current->pid);
 	if (atomic_read(&max_failures(attr)) != -1)
 		atomic_dec_not_zero(&max_failures(attr));
 
Index: linux-2.6.17/Documentation/fault-injection/fault-injection.txt
===================================================================
--- linux-2.6.17.orig/Documentation/fault-injection/fault-injection.txt
+++ linux-2.6.17/Documentation/fault-injection/fault-injection.txt
@@ -52,6 +52,15 @@ configuration of fault-injection capabil
 	on each call to should_fail(,size).  Failure injection is
 	suppressed until "space" reaches zero.
 
+- /debug/*/process-filter:
+
+	filter failures by pid.
+	A value of '0' disables filtering by process.
+	Any positive value limits failures to only processes indicated by
+	/proc/<pid>/make-it-fail==1.
+	A negative value means that failures are enabled for
+	pid==-process_filter irrespective of /proc/<pid>/make-it-fail.
+
 o Boot option
 
 In order to inject faults while debugfs is not available (early boot time),
Index: linux-2.6.17/lib/fault-inject-debugfs.c
===================================================================
--- linux-2.6.17.orig/lib/fault-inject-debugfs.c
+++ linux-2.6.17/lib/fault-inject-debugfs.c
@@ -8,6 +8,7 @@ struct fault_attr_entries {
 	struct dentry *interval_file;
 	struct dentry *times_file;
 	struct dentry *space_file;
+	struct dentry *process_filter_file;
 };
 
 static void debugfs_ul_set(void *data, u64 val)
@@ -66,6 +67,10 @@ static void cleanup_fault_attr_entries(s
 			debugfs_remove(entries->space_file);
 			entries->space_file = NULL;
 		}
+		if (entries->process_filter_file) {
+			debugfs_remove(entries->process_filter_file);
+			entries->process_filter_file = NULL;
+		}
 		debugfs_remove(entries->dir);
 		entries->dir = NULL;
 	}
@@ -105,6 +110,12 @@ static int init_fault_attr_entries(struc
 		goto fail;
 	entries->space_file = file;
 
+	file = debugfs_create_atomic_t("process-filter", mode, dir,
+				       &attr->process_filter);
+	if (!file)
+		goto fail;
+	entries->process_filter_file = file;
+
 	return 0;
 fail:
 	cleanup_fault_attr_entries(entries);

Re: [patch 7/8] process filtering for fault-injection capabilities

[Don Mullis]

Temporary fix-up so that next patch:
"[patch 8/8] stacktrace filtering for fault-injection capabilities"
applies cleanly.


Signed-off-by: Don Mullis <dwm@meer.net>

---
 include/linux/fault-inject.h |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Index: linux-2.6.17/include/linux/fault-inject.h
===================================================================
--- linux-2.6.17.orig/include/linux/fault-inject.h
+++ linux-2.6.17/include/linux/fault-inject.h
@@ -16,7 +16,9 @@ struct fault_attr {
 	atomic_t times;
 	atomic_t space;
 	unsigned long count;
-	atomic_t process_filter;
+
+	/* A value of '0' means process filter is disabled. */
+	u32 process_filter;
 };
 
 #define DEFINE_FAULT_ATTR(name) \

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

[Don Mullis]

Undo temporary fix-up for clean application of patch
"[patch 8/8] stacktrace filtering for fault-injection capabilities".

Fix bug in !(CONFIG_STACK_UNWIND || CONFIG_STACKTRACE) case, based on
code inspection only.  Anyone with a non-i386, -x86_64, -s390 willing
to test this?

Reintroduce stacktrace-specific documentation, factored out earlier.


Signed-off-by: Don Mullis <dwm@meer.net>

---
 Documentation/fault-injection/fault-injection.txt |   13 +++++++++++++
 include/linux/fault-inject.h                      |    2 +-
 lib/fault-inject.c                                |    2 +-
 3 files changed, 15 insertions(+), 2 deletions(-)

Index: linux-2.6.17/lib/fault-inject.c
===================================================================
--- linux-2.6.17.orig/lib/fault-inject.c
+++ linux-2.6.17/lib/fault-inject.c
@@ -136,7 +136,7 @@ static int fail_stacktrace(struct fault_
 
 #else
 
-#define fail_stacktrace(attr)	(0)
+#define fail_stacktrace(attr)	(1)
 
 #endif
 
Index: linux-2.6.17/Documentation/fault-injection/fault-injection.txt
===================================================================
--- linux-2.6.17.orig/Documentation/fault-injection/fault-injection.txt
+++ linux-2.6.17/Documentation/fault-injection/fault-injection.txt
@@ -61,6 +61,19 @@ configuration of fault-injection capabil
 	A negative value means that failures are enabled for
 	pid==-process_filter irrespective of /proc/<pid>/make-it-fail.
 
+- /debug/*/stacktrace-depth:
+
+	specifies the maximum stacktrace depth walked during search
+	for a caller within [address-start,address-end).  A value of 0
+	disables the stacktrace filter.
+
+- /debug/*/address-start:
+- /debug/*/address-end:
+
+	specifies the range of virtual addresses tested during
+	stacktrace walking.  Failure is injected only if some caller
+	in the walked stacktrace lies within this range.
+
 o Boot option
 
 In order to inject faults while debugfs is not available (early boot time),
Index: linux-2.6.17/include/linux/fault-inject.h
===================================================================
--- linux-2.6.17.orig/include/linux/fault-inject.h
+++ linux-2.6.17/include/linux/fault-inject.h
@@ -16,7 +16,7 @@ struct fault_attr {
 	atomic_t times;
 	atomic_t space;
 	unsigned long count;
-	u32 process_filter;
+	atomic_t process_filter;
 	unsigned long stacktrace_depth;
 	unsigned long address_start;
 	unsigned long address_end;

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

[Don Mullis]

Push fault-injection entries earlier in the list, so that they appear
nested under DEBUG_KERNEL in menuconfig/xconfig.


Signed-off-by: Don Mullis <dwm@meer.net>

---
 lib/Kconfig.debug |   70 +++++++++++++++++++++++++++---------------------------
 1 file changed, 35 insertions(+), 35 deletions(-)

Index: linux-2.6.17/lib/Kconfig.debug
===================================================================
--- linux-2.6.17.orig/lib/Kconfig.debug
+++ linux-2.6.17/lib/Kconfig.debug
@@ -294,6 +294,41 @@ config DEBUG_INFO
 
 	  If unsure, say N.
 
+config FAULT_INJECTION
+	bool
+	depends on DEBUG_KERNEL
+	select STACKTRACE
+	select FRAME_POINTER
+
+config FAILSLAB
+	bool "fault-injection capability for kmalloc"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION
+	help
+	  This option provides fault-injection capability for kmalloc.
+
+config FAIL_PAGE_ALLOC
+	bool "fault-injection capability for alloc_pages()"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION
+	help
+	  This option provides fault-injection capability for alloc_pages().
+
+config FAIL_MAKE_REQUEST
+	bool "fault-injection capability for disk IO"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION
+	help
+	  This option provides fault-injection capability to disk IO.
+
+config FAULT_INJECTION_DEBUGFS
+	tristate "runtime configuration for fault-injection capabilities"
+	depends on DEBUG_KERNEL && SYSFS && FAULT_INJECTION
+	select DEBUG_FS
+	help
+	  This option provides kernel module that provides runtime
+	  configuration interface by debugfs.
+
 config DEBUG_FS
 	bool "Debug Filesystem"
 	depends on SYSFS
@@ -368,38 +403,3 @@ config RCU_TORTURE_TEST
 	  at boot time (you probably don't).
 	  Say M if you want the RCU torture tests to build as a module.
 	  Say N if you are unsure.
-
-config FAULT_INJECTION
-	bool
-	select STACKTRACE
-	select FRAME_POINTER
-
-config FAILSLAB
-	bool "fault-injection capabilitiy for kmalloc"
-	depends on DEBUG_KERNEL
-	select FAULT_INJECTION
-	help
-	  This option provides fault-injection capabilitiy for kmalloc.
-
-config FAIL_PAGE_ALLOC
-	bool "fault-injection capabilitiy for alloc_pages()"
-	depends on DEBUG_KERNEL
-	select FAULT_INJECTION
-	help
-	  This option provides fault-injection capabilitiy for alloc_pages().
-
-config FAIL_MAKE_REQUEST
-	bool "fault-injection capabilitiy for disk IO"
-	depends on DEBUG_KERNEL
-	select FAULT_INJECTION 
-	help
-	  This option provides fault-injection capabilitiy to disk IO.
-
-config FAULT_INJECTION_DEBUGFS 
-	tristate "runtime configuration for fault-injection capabilities"
-	depends on DEBUG_KERNEL && SYSFS && FAULT_INJECTION
-	select DEBUG_FS
-	help
-	  This option provides kernel module that provides runtime
-	  configuration interface by debugfs.
-

Re: [patch 1/8] documentation and scripts

[Akinobu Mita]

On Mon, Sep 18, 2006 at 10:50:51PM -0700, Don Mullis wrote:
> Add reference to similar functionality in drivers/md/faulty.c .
> 
> Clarify debugfs descriptions.
> 
> Add an entry for fault-injection to 
> Documentation/kernel-parameters.txt .

Thank you for fixing my poor English and good description.
I merged this patch.

Re: [patch 2/8] fault-injection capabilities infrastructure

[Akinobu Mita]

On Mon, Sep 18, 2006 at 10:50:54PM -0700, Don Mullis wrote:
> Replace individual structure element comments with reference
> to Documentation/fault-injection/fault-injection.txt
> 
> Init "interval" to 1 (smallest useful value).
> Init "times" to 1 rather than -1 (infinity), for fewer 
> accidental system lockups.
> 

This patch also applied with small coding style fix.

>  #define DEFINE_FAULT_ATTR(name) \
> -	struct fault_attr name = { .times = ATOMIC_INIT(-1), }
> +	struct fault_attr name = { .interval=1, .times = ATOMIC_INIT(1), }

	struct fault_attr name = { .interval = 1, .times = ATOMIC_INIT(1), }

Re: [patch 7/8] process filtering for fault-injection capabilities

[Akinobu Mita]

On Mon, Sep 18, 2006 at 10:54:51PM -0700, Don Mullis wrote:
> Add functionality to the process_filter variable: A negative argument
> injects failures for only for pid==-process_filter, thereby permitting
> per-process failures from boot time.
> 

Is it better to add new filter for this purpose?
Because someone may want to filter by tgid instead of pid.

- positive value is for task->pid
- nevative value is for task->tgid

> Add printk, called upon each failure injection.
> 

This printk() will be very useful. But it is better to make
configurable, and the pid is not reliable in interrupt context.

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

[Akinobu Mita]

On Mon, Sep 18, 2006 at 10:56:28PM -0700, Don Mullis wrote:
> Fix bug in !(CONFIG_STACK_UNWIND || CONFIG_STACKTRACE) case, based on
> code inspection only.  Anyone with a non-i386, -x86_64, -s390 willing
> to test this?

Yes, it didn't work completely without this patch on other architectures.
Thank you.

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

[Akinobu Mita]

On Mon, Sep 18, 2006 at 10:57:51PM -0700, Don Mullis wrote:
> Push fault-injection entries earlier in the list, so that they appear
> nested under DEBUG_KERNEL in menuconfig/xconfig.

Disabling the option Kernel debugging can hide all config options
realated to fault-injection without this patch.

Do I misunderstand something?

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

[Don Mullis]

On Tue, 2006-09-19 at 17:09 +0800, Akinobu Mita wrote:
> On Mon, Sep 18, 2006 at 10:57:51PM -0700, Don Mullis wrote:
> > Push fault-injection entries earlier in the list, so that they appear
> > nested under DEBUG_KERNEL in menuconfig/xconfig.
> 
> Disabling the option Kernel debugging can hide all config options
> realated to fault-injection without this patch.
> 
> Do I misunderstand something?
> 

There's no problem with hiding per se, but rather with the indentation
level.  It's most natural for the user to have dependent options
indented under their "parent".  For an example, in "menuconfig" try
setting "Compile the kernel with frame unwind information";
notice that "Stack unwind support" appears immediately underneath
it, indented.  The indentation reminds the user why "Stack unwind
support" has appeared.

Note that several of the pre-existing, non-fault-injection options
under "Kernel debugging", are also broken in this way.

Re: [patch 7/8] process filtering for fault-injection capabilities

[Don Mullis]

On Tue, 2006-09-19 at 17:05 +0800, Akinobu Mita wrote:
> On Mon, Sep 18, 2006 at 10:54:51PM -0700, Don Mullis wrote:
> > Add functionality to the process_filter variable: A negative argument
> > injects failures for only for pid==-process_filter, thereby permitting
> > per-process failures from boot time.
> > 
> 
> Is it better to add new filter for this purpose?
> Because someone may want to filter by tgid instead of pid.
> 
> - positive value is for task->pid
> - nevative value is for task->tgid

Your idea sounds good to me.

> > Add printk, called upon each failure injection.
> > 
> 
> This printk() will be very useful. But it is better to make
> configurable, and the pid is not reliable in interrupt context.

Okay.  We do need some output now, for testing.

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

[Akinobu Mita]

On Tue, Sep 19, 2006 at 10:35:27AM -0700, Don Mullis wrote:
> There's no problem with hiding per se, but rather with the indentation
> level.  It's most natural for the user to have dependent options
> indented under their "parent".  For an example, in "menuconfig" try
> setting "Compile the kernel with frame unwind information";
> notice that "Stack unwind support" appears immediately underneath
> it, indented.  The indentation reminds the user why "Stack unwind
> support" has appeared.
> 

I see.

> Note that several of the pre-existing, non-fault-injection options
> under "Kernel debugging", are also broken in this way.

Perhaps you can move UNWIND_INFO, STACK_UNWIND, and DEBUG_FS
entries ealier in the list. It improves improve appearance for
other DEBUG_KERNEL dependent config options like (DEBUG_VM,
FRAME_POINTER, ...).

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

[Don Mullis]

On Wed, 2006-09-20 at 21:39 +0800, Akinobu Mita wrote:
> Perhaps you can move UNWIND_INFO, STACK_UNWIND, and DEBUG_FS
> entries ealier in the list. It improves improve appearance for
> other DEBUG_KERNEL dependent config options like (DEBUG_VM,
> FRAME_POINTER, ...).

Okay, I will prepare such a janitorial patch.

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

[Andi Kleen]

> Index: work-fault-inject/lib/Kconfig.debug
> ===================================================================
> --- work-fault-inject.orig/lib/Kconfig.debug
> +++ work-fault-inject/lib/Kconfig.debug
> @@ -401,6 +401,8 @@ config LKDTM
>  
>  config FAULT_INJECTION
>  	bool
> +	select STACKTRACE
> +	select FRAME_POINTER

I'm afraid this won't work fully reliably on i386/x86-64 at least. The problem 
is that if even with frame pointers the new unwinder will try its unwinding
first and if it gets stuck it will log the fallback entries into the stack
trace buffer too. And those fallback entries can be randomly everything left over
from previous stack traces. Then the fault injection will be more
like Russian roulette and could randomly hit other code paths too.

To make this work there would need to be some way to turn off fallback
for these particular stack traces. 

Or maybe just always use frame pointer
there, but even that will likely not help because the few places
where the dwarf2 unwinder still gets stuck (usually assembly code) 
are usually broken with frame pointers too.

-Andi