* [patch 1/8] documentation and scripts
2006-09-14 10:20 [patch 0/8] fault-injection capabilities (v3) Akinobu Mita
@ 2006-09-14 10:20 ` Akinobu Mita
2006-09-19 5:50 ` Don Mullis
2006-09-14 10:20 ` [patch 2/8] fault-injection capabilities infrastructure Akinobu Mita
` (6 subsequent siblings)
7 siblings, 1 reply; 25+ messages in thread
From: Akinobu Mita @ 2006-09-14 10:20 UTC (permalink / raw)
To: linux-kernel; +Cc: ak, akpm, Don Mullis, Akinobu Mita
[-- Attachment #1: doc.patch --]
[-- Type: text/plain, Size: 6315 bytes --]
This patch provides the documentation and helper/testing scripts for
fault-injection capabilities.
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>
Documentation/fault-injection/failcmd.sh | 4
Documentation/fault-injection/failmodule.sh | 32 ++++
Documentation/fault-injection/fault-injection.txt | 150 ++++++++++++++++++++++
3 files changed, 186 insertions(+)
Index: work-shouldfail/Documentation/fault-injection/failcmd.sh
===================================================================
--- /dev/null
+++ work-shouldfail/Documentation/fault-injection/failcmd.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+echo 1 > /proc/self/make-it-fail
+exec $*
Index: work-shouldfail/Documentation/fault-injection/failmodule.sh
===================================================================
--- /dev/null
+++ work-shouldfail/Documentation/fault-injection/failmodule.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+#
+# Usage: failmodule <failname> <modulename> [stacktrace-depth]
+#
+# <failname>: "failslab", "fail_alloc_page", or "fail_make_request"
+#
+# <modulename>: module name that you want to inject faults.
+#
+# [stacktrace-depth]: the maximum number of stacktrace walking allowed
+#
+
+STACKTRACE_DEPTH=5
+if [ $# -gt 2 ]; then
+ STACKTRACE_DEPTH=$3
+fi
+
+if [ ! -d /debug/$1 ]; then
+ echo "Fault-injection $1 does not exist" >&2
+ exit 1
+fi
+if [ ! -d /sys/module/$2 ]; then
+ echo "Module $2 does not exist" >&2
+ exit 1
+fi
+
+# Disable stacktrace filter at first
+echo 0 > /debug/$1/address-end
+echo 1 > /debug/$1/stacktrace-depth
+
+echo `cat /sys/module/$2/sections/.text` > /debug/$1/address-start
+echo `cat /sys/module/$2/sections/.exit.text` > /debug/$1/address-end
+echo $STACKTRACE_DEPTH > /debug/$1/stacktrace-depth
Index: work-shouldfail/Documentation/fault-injection/fault-injection.txt
===================================================================
--- /dev/null
+++ work-shouldfail/Documentation/fault-injection/fault-injection.txt
@@ -0,0 +1,150 @@
+Fault injection capabilities infrastructure
+===========================================
+
+Available fault injection capabilities
+--------------------------------------
+
+o failslab
+
+ injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
+
+o fail_page_alloc
+
+ injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
+
+o fail_make_request
+
+ injects disk IO errors on permitted devices by /sys/block/<device>/make-it-fail
+ or /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
+
+Configure fault-injection capabilities behavior
+-----------------------------------------------
+
+Example for failslab:
+
+o debugfs entries
+
+fault-inject-debugfs kernel module provides some debugfs entries for runtime
+configuration for fault-injection capabilities.
+
+- /debug/failslab/probability:
+
+ specifies how often it should fail in percent.
+
+- /debug/failslab/interval:
+
+ specifies the interval of failures.
+
+- /debug/failslab/times:
+
+ specifies how many times failures may happen at most.
+
+- /debug/failslab/space:
+
+ specifies the size of free space where memory can be allocated
+ safely in bytes.
+
+- /debug/failslab/process-filter:
+
+ specifies whether the process filter is enabled or not.
+ It allows failing only permitted processes by /proc/<pid>/make-it-fail
+
+- /debug/failslab/stacktrace-depth:
+
+ specifies the maximum stacktrace depth walking allowed.
+ A value '0' means stacktrace filter is disabled.
+
+- /debug/failslab/address-start:
+- /debug/failslab/address-end:
+
+ specifies the range of virtual address.
+ It allows failing only if the stacktrace hits in this range.
+
+o Boot option
+
+In order to inject faults while debugfs is not available (early boot time),
+We can use boot option.
+
+- failslab=<interval>,<probability>,<space>,<times>
+
+How to add new fault injection capability
+-----------------------------------------
+
+o #include <linux/fault-inject.h>
+
+o define the fault attributes
+
+ DEFINE_FAULT_INJECTION(name);
+
+ Please see the definition of struct fault_attr in fault-inject.h
+ for the detail.
+
+o provide the way to configure fault attributes
+
+- boot option
+
+ If you need to enable the fault injection capability ealier boot time,
+ you can provide boot option to configure it. There is a helper function for it.
+
+ setup_fault_attr(attr, str);
+
+- module parameters
+
+ If the scope of the fault injection capability is limited by a kernel module,
+ It is better to provide module parameters to configure the member of fault
+ attributes.
+
+- debugfs entries
+
+ failslab, fail_page_alloc, and fail_make_request use this way.
+
+ But now there is no helper functions to provides debugfs entries for
+ fault injection capabilities. Please refer to lib/fault-inject-debugfs.c
+ to know how to do. And please try not to add new one into
+ fault-inject-debugfs module.
+
+ Because failslab, fail_page_alloc, and fail_make_request are used ealier
+ boot time before debugfs is available and the slab allocator,
+ the page allocator, and the block layer cannot be built as module.
+
+o add a hook to insert failures
+
+ should_fail() returns 1 when failures should happen.
+
+ should_fail(attr);
+
+Tests
+-----
+
+o inject slab allocation failures into module init/cleanup code
+
+------------------------------------------------------------------------------
+#!/bin/bash
+
+FAILCMD=Documentation/fault-injection/failcmd.sh
+
+modprobe fault-inject-debugfs
+echo Y > /debug/failslab/process-filter
+echo 25 > /debug/failslab/probability
+
+find /lib/modules/`uname -r` -name '*.ko' -exec basename {} .ko \; \
+ | while read i; do bash $FAILCMD modprobe $i;done
+
+lsmod | awk '{ if ($3 == 0) { system("bash $FAILCMD modprobe -r " $1) } }'
+
+------------------------------------------------------------------------------
+
+o inject slab allocation failures only for a specific module
+
+------------------------------------------------------------------------------
+#!/bin/bash
+
+FAILMOD=Documentation/fault-injection/failmodule.sh
+
+modprobe fault-inject-debugfs
+modprobe your-test-module
+bash $FAILMOD failslab your-test-module 10
+echo 25 > /debug/failslab/probability
+
+------------------------------------------------------------------------------
+
--
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [patch 1/8] documentation and scripts
2006-09-14 10:20 ` [patch 1/8] documentation and scripts Akinobu Mita
@ 2006-09-19 5:50 ` Don Mullis
2006-09-19 9:01 ` Akinobu Mita
0 siblings, 1 reply; 25+ messages in thread
From: Don Mullis @ 2006-09-19 5:50 UTC (permalink / raw)
To: Akinobu Mita; +Cc: linux-kernel, ak, akpm
Add reference to similar functionality in drivers/md/faulty.c .
Clarify debugfs descriptions.
Add an entry for fault-injection to
Documentation/kernel-parameters.txt .
Signed-off-by: Don Mullis <dwm@meer.net>
---
Documentation/fault-injection/fault-injection.txt | 85 ++++++++++------------
Documentation/kernel-parameters.txt | 7 +
2 files changed, 47 insertions(+), 45 deletions(-)
Index: linux-2.6.17/Documentation/fault-injection/fault-injection.txt
===================================================================
--- linux-2.6.17.orig/Documentation/fault-injection/fault-injection.txt
+++ linux-2.6.17/Documentation/fault-injection/fault-injection.txt
@@ -1,6 +1,9 @@
Fault injection capabilities infrastructure
===========================================
+See also drivers/md/faulty.c .
+
+
Available fault injection capabilities
--------------------------------------
@@ -14,58 +17,49 @@ o fail_page_alloc
o fail_make_request
- injects disk IO errors on permitted devices by /sys/block/<device>/make-it-fail
- or /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
+ injects disk IO errors on permitted devices by
+ /sys/block/<device>/make-it-fail or
+ /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
Configure fault-injection capabilities behavior
-----------------------------------------------
-Example for failslab:
-
o debugfs entries
fault-inject-debugfs kernel module provides some debugfs entries for runtime
-configuration for fault-injection capabilities.
-
-- /debug/failslab/probability:
+configuration of fault-injection capabilities.
- specifies how often it should fail in percent.
+- /debug/*/probability:
-- /debug/failslab/interval:
-
- specifies the interval of failures.
-
-- /debug/failslab/times:
-
- specifies how many times failures may happen at most.
+ likelihood of failure injection, in percent.
-- /debug/failslab/space:
+- /debug/*/interval:
- specifies the size of free space where memory can be allocated
- safely in bytes.
+ specifies the interval between failures, for calls to
+ should_fail() that pass all the other tests.
-- /debug/failslab/process-filter:
+ Note that if you enable this, by setting interval>1, you will
+ probably want to set probability=100.
- specifies whether the process filter is enabled or not.
- It allows failing only permitted processes by /proc/<pid>/make-it-fail
+- /debug/*/times:
-- /debug/failslab/stacktrace-depth:
-
- specifies the maximum stacktrace depth walking allowed.
- A value '0' means stacktrace filter is disabled.
+ specifies how many times failures may happen at most.
+ A value of -1 means "no limit".
-- /debug/failslab/address-start:
-- /debug/failslab/address-end:
+- /debug/*/space:
- specifies the range of virtual address.
- It allows failing only if the stacktrace hits in this range.
+ specifies an initial resource "budget", decremented by "size"
+ on each call to should_fail(,size). Failure injection is
+ suppressed until "space" reaches zero.
o Boot option
In order to inject faults while debugfs is not available (early boot time),
-We can use boot option.
+use the boot option:
-- failslab=<interval>,<probability>,<space>,<times>
+ failslab=
+ fail_page_alloc=
+ fail_make_request=<interval>,<probability>,<space>,<times>
How to add new fault injection capability
-----------------------------------------
@@ -77,44 +71,42 @@ o define the fault attributes
DEFINE_FAULT_INJECTION(name);
Please see the definition of struct fault_attr in fault-inject.h
- for the detail.
+ for details.
o provide the way to configure fault attributes
- boot option
- If you need to enable the fault injection capability ealier boot time,
+ If you need to enable the fault injection capability from boot time,
you can provide boot option to configure it. There is a helper function for it.
setup_fault_attr(attr, str);
- module parameters
- If the scope of the fault injection capability is limited by a kernel module,
- It is better to provide module parameters to configure the member of fault
- attributes.
+ If the scope of the fault injection capability is limited to a
+ single kernel module, it is better to provide module parameters to
+ configure the fault attributes.
- debugfs entries
failslab, fail_page_alloc, and fail_make_request use this way.
-
- But now there is no helper functions to provides debugfs entries for
+ But now there is no helper functions to provide debugfs entries for
fault injection capabilities. Please refer to lib/fault-inject-debugfs.c
to know how to do. And please try not to add new one into
fault-inject-debugfs module.
-
- Because failslab, fail_page_alloc, and fail_make_request are used ealier
- boot time before debugfs is available and the slab allocator,
- the page allocator, and the block layer cannot be built as module.
+ Because failslab, fail_page_alloc, and fail_make_request are used from
+ boot time, before debugfs is available, the slab allocator,
+ the page allocator, and the block layer cannot be built as modules.
o add a hook to insert failures
should_fail() returns 1 when failures should happen.
- should_fail(attr);
+ should_fail(attr,size);
-Tests
------
+Application Examples
+--------------------
o inject slab allocation failures into module init/cleanup code
@@ -148,3 +140,6 @@ echo 25 > /debug/failslab/probability
------------------------------------------------------------------------------
+
+
+
Index: linux-2.6.17/Documentation/kernel-parameters.txt
===================================================================
--- linux-2.6.17.orig/Documentation/kernel-parameters.txt
+++ linux-2.6.17/Documentation/kernel-parameters.txt
@@ -544,6 +544,13 @@ running once the system is up.
eurwdt= [HW,WDT] Eurotech CPU-1220/1410 onboard watchdog.
Format: <io>[,<irq>]
+ failslab=
+ fail_page_alloc=
+ fail_make_request=[KNL]
+ General fault injection mechanism.
+ Format: <interval>,<probability>,<space>,<times>
+ See also /Documentation/fault-injection/.
+
fd_mcs= [HW,SCSI]
See header of drivers/scsi/fd_mcs.c.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [patch 1/8] documentation and scripts
2006-09-19 5:50 ` Don Mullis
@ 2006-09-19 9:01 ` Akinobu Mita
0 siblings, 0 replies; 25+ messages in thread
From: Akinobu Mita @ 2006-09-19 9:01 UTC (permalink / raw)
To: Don Mullis; +Cc: linux-kernel, ak, akpm
On Mon, Sep 18, 2006 at 10:50:51PM -0700, Don Mullis wrote:
> Add reference to similar functionality in drivers/md/faulty.c .
>
> Clarify debugfs descriptions.
>
> Add an entry for fault-injection to
> Documentation/kernel-parameters.txt .
Thank you for fixing my poor English and good description.
I merged this patch.
^ permalink raw reply [flat|nested] 25+ messages in thread
* [patch 2/8] fault-injection capabilities infrastructure
2006-09-14 10:20 [patch 0/8] fault-injection capabilities (v3) Akinobu Mita
2006-09-14 10:20 ` [patch 1/8] documentation and scripts Akinobu Mita
@ 2006-09-14 10:20 ` Akinobu Mita
2006-09-19 5:50 ` Don Mullis
2006-09-14 10:20 ` [patch 3/8] fault-injection capability for kmalloc Akinobu Mita
` (5 subsequent siblings)
7 siblings, 1 reply; 25+ messages in thread
From: Akinobu Mita @ 2006-09-14 10:20 UTC (permalink / raw)
To: linux-kernel; +Cc: ak, akpm, Don Mullis, okuji, Akinobu Mita
[-- Attachment #1: should-fail.patch --]
[-- Type: text/plain, Size: 4629 bytes --]
This patch provides some functions for implement fault-injection
capabilities.
- Lightweight random simulator is taken from crasher module for SUSE kernel
- The function should_fail() is taken from failmalloc-1.0
(http://www.nongnu.org/failmalloc/)
Cc: okuji@enbug.org
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>
include/linux/fault-inject.h | 41 ++++++++++++++++++++
lib/Kconfig.debug | 4 ++
lib/Makefile | 1
lib/fault-inject.c | 84 +++++++++++++++++++++++++++++++++++++++++++
4 files changed, 130 insertions(+)
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -368,3 +368,7 @@ config RCU_TORTURE_TEST
at boot time (you probably don't).
Say M if you want the RCU torture tests to build as a module.
Say N if you are unsure.
+
+config FAULT_INJECTION
+ bool
+
Index: work-shouldfail/lib/Makefile
===================================================================
--- work-shouldfail.orig/lib/Makefile
+++ work-shouldfail/lib/Makefile
@@ -52,6 +52,7 @@ obj-$(CONFIG_SMP) += percpu_counter.o
obj-$(CONFIG_AUDIT_GENERIC) += audit.o
obj-$(CONFIG_SWIOTLB) += swiotlb.o
+obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o
hostprogs-y := gen_crc32table
clean-files := crc32table.h
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
--- /dev/null
+++ work-shouldfail/include/linux/fault-inject.h
@@ -0,0 +1,41 @@
+#ifndef _LINUX_FAULT_INJECT_H
+#define _LINUX_FAULT_INJECT_H
+
+#ifdef CONFIG_FAULT_INJECTION
+
+#include <linux/types.h>
+#include <asm/atomic.h>
+
+struct fault_attr {
+
+ /* how often it should fail in percent. */
+ unsigned long probability;
+
+ /* the interval of failures. */
+ unsigned long interval;
+
+ /*
+ * how many times failures may happen at most.
+ * A value of '-1' means infinity.
+ */
+ atomic_t times;
+
+ /*
+ * the size of free space where memory can be allocated safely.
+ * A value of '0' means infinity.
+ */
+ atomic_t space;
+
+ unsigned long count;
+};
+
+#define DEFINE_FAULT_ATTR(name) \
+ struct fault_attr name = { .times = ATOMIC_INIT(-1), }
+
+int setup_fault_attr(struct fault_attr *attr, char *str);
+void should_fail_srandom(unsigned long entropy);
+int should_fail(struct fault_attr *attr, ssize_t size);
+
+#endif /* CONFIG_FAULT_INJECTION */
+
+#endif /* _LINUX_FAULT_INJECT_H */
Index: work-shouldfail/lib/fault-inject.c
===================================================================
--- /dev/null
+++ work-shouldfail/lib/fault-inject.c
@@ -0,0 +1,84 @@
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/random.h>
+#include <linux/stat.h>
+#include <linux/types.h>
+#include <linux/fs.h>
+#include <linux/module.h>
+#include <linux/fault-inject.h>
+
+int setup_fault_attr(struct fault_attr *attr, char *str)
+{
+ unsigned long probability;
+ unsigned long interval;
+ int times;
+ int space;
+
+ /* "<interval>,<probability>,<space>,<times>" */
+ if (sscanf(str, "%lu,%lu,%d,%d",
+ &interval, &probability, &space, ×) < 4) {
+ printk(KERN_WARNING "SHOULD_FAIL: failed to parse arguments\n");
+ return 0;
+ }
+
+ attr->probability = probability;
+ attr->interval = interval;
+ atomic_set(&attr->times, times);
+ atomic_set(&attr->space, space);
+
+ return 1;
+}
+
+#define failure_probability(attr) (attr)->probability
+#define failure_interval(attr) (attr)->interval
+#define max_failures(attr) (attr)->times
+#define current_space(attr) (attr)->space
+#define atomic_dec_not_zero(v) atomic_add_unless((v), -1, 0)
+
+static unsigned long rand_seed = 152L;
+
+static unsigned long should_fail_random(void)
+{
+ rand_seed = rand_seed * 690690L+1;
+ return rand_seed ^ jiffies;
+}
+
+void should_fail_srandom(unsigned long entropy)
+{
+ rand_seed ^= entropy;
+ should_fail_random();
+}
+
+/*
+ * This code is stolen from failmalloc-1.0
+ * http://www.nongnu.org/failmalloc/
+ */
+
+int should_fail(struct fault_attr *attr, ssize_t size)
+{
+ if (atomic_read(&max_failures(attr)) == 0)
+ return 0;
+
+ if (atomic_read(¤t_space(attr)) > size) {
+ atomic_sub(size, ¤t_space(attr));
+ return 0;
+ }
+
+ if (failure_interval(attr) > 1) {
+ attr->count++;
+ if (attr->count % failure_interval(attr))
+ return 0;
+ }
+
+ if (failure_probability(attr) > should_fail_random() % 100)
+ goto fail;
+
+ return 0;
+
+fail:
+
+ if (atomic_read(&max_failures(attr)) != -1)
+ atomic_dec_not_zero(&max_failures(attr));
+
+ return 1;
+}
--
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [patch 2/8] fault-injection capabilities infrastructure
2006-09-14 10:20 ` [patch 2/8] fault-injection capabilities infrastructure Akinobu Mita
@ 2006-09-19 5:50 ` Don Mullis
2006-09-19 9:03 ` Akinobu Mita
0 siblings, 1 reply; 25+ messages in thread
From: Don Mullis @ 2006-09-19 5:50 UTC (permalink / raw)
To: Akinobu Mita; +Cc: linux-kernel, ak, akpm, okuji
Replace individual structure element comments with reference
to Documentation/fault-injection/fault-injection.txt
Init "interval" to 1 (smallest useful value).
Init "times" to 1 rather than -1 (infinity), for fewer
accidental system lockups.
Signed-off-by: Don Mullis <dwm@meer.net>
---
include/linux/fault-inject.h | 20 +++++---------------
1 file changed, 5 insertions(+), 15 deletions(-)
Index: linux-2.6.17/include/linux/fault-inject.h
===================================================================
--- linux-2.6.17.orig/include/linux/fault-inject.h
+++ linux-2.6.17/include/linux/fault-inject.h
@@ -6,31 +6,21 @@
#include <linux/types.h>
#include <asm/atomic.h>
+/*
+ * For explanation of the elements of this struct, see
+ * Documentation/fault-injection/fault-injection.txt
+ */
struct fault_attr {
-
- /* how often it should fail in percent. */
unsigned long probability;
-
- /* the interval of failures. */
unsigned long interval;
-
- /*
- * how many times failures may happen at most.
- * A value of '-1' means infinity.
- */
atomic_t times;
-
- /*
- * the size of free space where memory can be allocated safely.
- * A value of '0' means infinity.
- */
atomic_t space;
unsigned long count;
};
#define DEFINE_FAULT_ATTR(name) \
- struct fault_attr name = { .times = ATOMIC_INIT(-1), }
+ struct fault_attr name = { .interval=1, .times = ATOMIC_INIT(1), }
int setup_fault_attr(struct fault_attr *attr, char *str);
void should_fail_srandom(unsigned long entropy);
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [patch 2/8] fault-injection capabilities infrastructure
2006-09-19 5:50 ` Don Mullis
@ 2006-09-19 9:03 ` Akinobu Mita
0 siblings, 0 replies; 25+ messages in thread
From: Akinobu Mita @ 2006-09-19 9:03 UTC (permalink / raw)
To: Don Mullis; +Cc: linux-kernel, ak, akpm, okuji
On Mon, Sep 18, 2006 at 10:50:54PM -0700, Don Mullis wrote:
> Replace individual structure element comments with reference
> to Documentation/fault-injection/fault-injection.txt
>
> Init "interval" to 1 (smallest useful value).
> Init "times" to 1 rather than -1 (infinity), for fewer
> accidental system lockups.
>
This patch also applied with small coding style fix.
> #define DEFINE_FAULT_ATTR(name) \
> - struct fault_attr name = { .times = ATOMIC_INIT(-1), }
> + struct fault_attr name = { .interval=1, .times = ATOMIC_INIT(1), }
struct fault_attr name = { .interval = 1, .times = ATOMIC_INIT(1), }
^ permalink raw reply [flat|nested] 25+ messages in thread
* [patch 3/8] fault-injection capability for kmalloc
2006-09-14 10:20 [patch 0/8] fault-injection capabilities (v3) Akinobu Mita
2006-09-14 10:20 ` [patch 1/8] documentation and scripts Akinobu Mita
2006-09-14 10:20 ` [patch 2/8] fault-injection capabilities infrastructure Akinobu Mita
@ 2006-09-14 10:20 ` Akinobu Mita
2006-09-14 10:20 ` [patch 4/8] fault-injection capability for alloc_pages() Akinobu Mita
` (4 subsequent siblings)
7 siblings, 0 replies; 25+ messages in thread
From: Akinobu Mita @ 2006-09-14 10:20 UTC (permalink / raw)
To: linux-kernel; +Cc: ak, akpm, Don Mullis, Pekka Enberg, Akinobu Mita
[-- Attachment #1: failslab.patch --]
[-- Type: text/plain, Size: 3246 bytes --]
This patch provides fault-injection capability for kmalloc.
Boot option:
failslab=<interval>,<probability>,<space>,<times>
<interval> -- specifies the interval of failures.
<probability> -- specifies how often it should fail in percent.
<space> -- specifies the size of free space where memory can be
allocated safely in bytes.
<times> -- specifies how many times failures may happen at most.
Example:
failslab=10,100,0,-1
slab allocation (kmalloc(), kmem_cache_alloc(),..) fails once per 10 times.
Cc: Pekka Enberg <penberg@cs.helsinki.fi>
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>
include/linux/fault-inject.h | 4 ++++
lib/Kconfig.debug | 7 +++++++
mm/slab.c | 34 ++++++++++++++++++++++++++++++++++
3 files changed, 45 insertions(+)
Index: work-shouldfail/mm/slab.c
===================================================================
--- work-shouldfail.orig/mm/slab.c
+++ work-shouldfail/mm/slab.c
@@ -108,6 +108,7 @@
#include <linux/mempolicy.h>
#include <linux/mutex.h>
#include <linux/rtmutex.h>
+#include <linux/fault-inject.h>
#include <asm/uaccess.h>
#include <asm/cacheflush.h>
@@ -2963,11 +2964,44 @@ static void *cache_alloc_debugcheck_afte
#define cache_alloc_debugcheck_after(a,b,objp,d) (objp)
#endif
+#ifdef CONFIG_FAILSLAB
+
+static DEFINE_FAULT_ATTR(failslab_attr);
+
+static int __init setup_failslab(char *str)
+{
+ should_fail_srandom(jiffies);
+ return setup_fault_attr(&failslab_attr, str);
+}
+__setup("failslab=", setup_failslab);
+
+struct fault_attr *failslab = &failslab_attr;
+EXPORT_SYMBOL_GPL(failslab);
+
+static int should_failslab(struct kmem_cache *cachep, gfp_t flags)
+{
+ if ((flags & __GFP_NOFAIL) || cachep == &cache_cache)
+ return 0;
+ return should_fail(failslab, obj_size(cachep));
+}
+
+#else
+
+static inline int should_failslab(struct kmem_cache *cachep, gfp_t flags)
+{
+ return 0;
+}
+
+#endif
+
static inline void *____cache_alloc(struct kmem_cache *cachep, gfp_t flags)
{
void *objp;
struct array_cache *ac;
+ if (should_failslab(cachep, flags))
+ return NULL;
+
#ifdef CONFIG_NUMA
if (unlikely(current->flags & (PF_SPREAD_SLAB | PF_MEMPOLICY))) {
objp = alternate_node_alloc(cachep, flags);
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -372,3 +372,10 @@ config RCU_TORTURE_TEST
config FAULT_INJECTION
bool
+config FAILSLAB
+ bool "fault-injection capabilitiy for kmalloc"
+ depends on DEBUG_KERNEL
+ select FAULT_INJECTION
+ help
+ This option provides fault-injection capabilitiy for kmalloc.
+
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
--- work-shouldfail.orig/include/linux/fault-inject.h
+++ work-shouldfail/include/linux/fault-inject.h
@@ -36,6 +36,10 @@ int setup_fault_attr(struct fault_attr *
void should_fail_srandom(unsigned long entropy);
int should_fail(struct fault_attr *attr, ssize_t size);
+#ifdef CONFIG_FAILSLAB
+extern struct fault_attr *failslab;
+#endif
+
#endif /* CONFIG_FAULT_INJECTION */
#endif /* _LINUX_FAULT_INJECT_H */
--
^ permalink raw reply [flat|nested] 25+ messages in thread* [patch 4/8] fault-injection capability for alloc_pages()
2006-09-14 10:20 [patch 0/8] fault-injection capabilities (v3) Akinobu Mita
` (2 preceding siblings ...)
2006-09-14 10:20 ` [patch 3/8] fault-injection capability for kmalloc Akinobu Mita
@ 2006-09-14 10:20 ` Akinobu Mita
2006-09-14 10:20 ` [patch 5/8] fault-injection capability for disk IO Akinobu Mita
` (3 subsequent siblings)
7 siblings, 0 replies; 25+ messages in thread
From: Akinobu Mita @ 2006-09-14 10:20 UTC (permalink / raw)
To: linux-kernel; +Cc: ak, akpm, Don Mullis, Akinobu Mita
[-- Attachment #1: fail_alloc_pages.patch --]
[-- Type: text/plain, Size: 3205 bytes --]
This patch provides fault-injection capability for alloc_pages()
Boot option:
fail_page_alloc=<interval>,<probability>,<space>,<times>
<interval> -- specifies the interval of failures.
<probability> -- specifies how often it should fail in percent.
<space> -- specifies the size of free space where memory can be
allocated safely in pages.
<times> -- specifies how many times failures may happen at most.
Example:
fail_page_alloc=10,100,0,-1
The page allocation (alloc_pages(), ...) fails once per 10 times.
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>
include/linux/fault-inject.h | 3 +++
lib/Kconfig.debug | 7 +++++++
mm/page_alloc.c | 35 +++++++++++++++++++++++++++++++++++
3 files changed, 45 insertions(+)
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -379,3 +379,10 @@ config FAILSLAB
help
This option provides fault-injection capabilitiy for kmalloc.
+config FAIL_PAGE_ALLOC
+ bool "fault-injection capabilitiy for alloc_pages()"
+ depends on DEBUG_KERNEL
+ select FAULT_INJECTION
+ help
+ This option provides fault-injection capabilitiy for alloc_pages().
+
Index: work-shouldfail/mm/page_alloc.c
===================================================================
--- work-shouldfail.orig/mm/page_alloc.c
+++ work-shouldfail/mm/page_alloc.c
@@ -37,6 +37,7 @@
#include <linux/vmalloc.h>
#include <linux/mempolicy.h>
#include <linux/stop_machine.h>
+#include <linux/fault-inject.h>
#include <asm/tlbflush.h>
#include <asm/div64.h>
@@ -903,6 +904,37 @@ get_page_from_freelist(gfp_t gfp_mask, u
return page;
}
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+
+static DEFINE_FAULT_ATTR(fail_page_alloc_attr);
+
+static int __init setup_fail_page_alloc(char *str)
+{
+ should_fail_srandom(jiffies);
+ return setup_fault_attr(&fail_page_alloc_attr, str);
+}
+__setup("fail_page_alloc=", setup_fail_page_alloc);
+
+struct fault_attr *fail_page_alloc = &fail_page_alloc_attr;
+EXPORT_SYMBOL_GPL(fail_page_alloc);
+
+static int should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
+{
+ if (gfp_mask & __GFP_NOFAIL)
+ return 0;
+
+ return should_fail(fail_page_alloc, 1 << order);
+}
+
+#else
+
+static inline int should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
+{
+ return 0;
+}
+
+#endif
+
/*
* This is the 'heart' of the zoned buddy allocator.
*/
@@ -921,6 +953,9 @@ __alloc_pages(gfp_t gfp_mask, unsigned i
might_sleep_if(wait);
+ if (should_fail_alloc_page(gfp_mask, order))
+ return NULL;
+
restart:
z = zonelist->zones; /* the list of zones suitable for gfp_mask */
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
--- work-shouldfail.orig/include/linux/fault-inject.h
+++ work-shouldfail/include/linux/fault-inject.h
@@ -39,6 +39,9 @@ int should_fail(struct fault_attr *attr,
#ifdef CONFIG_FAILSLAB
extern struct fault_attr *failslab;
#endif
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+extern struct fault_attr *fail_page_alloc;
+#endif
#endif /* CONFIG_FAULT_INJECTION */
--
^ permalink raw reply [flat|nested] 25+ messages in thread* [patch 5/8] fault-injection capability for disk IO
2006-09-14 10:20 [patch 0/8] fault-injection capabilities (v3) Akinobu Mita
` (3 preceding siblings ...)
2006-09-14 10:20 ` [patch 4/8] fault-injection capability for alloc_pages() Akinobu Mita
@ 2006-09-14 10:20 ` Akinobu Mita
2006-09-14 10:20 ` [patch 6/8] debugfs entries for configuration Akinobu Mita
` (2 subsequent siblings)
7 siblings, 0 replies; 25+ messages in thread
From: Akinobu Mita @ 2006-09-14 10:20 UTC (permalink / raw)
To: linux-kernel; +Cc: ak, akpm, Don Mullis, Jens Axboe, Akinobu Mita
[-- Attachment #1: fail_make_request.patch --]
[-- Type: text/plain, Size: 6655 bytes --]
This patch provides fault-injection capability for disk IO.
Boot option:
fail_make_request=<probability>,<interval>,<space>,<times>
<interval> -- specifies the interval of failures.
<probability> -- specifies how often it should fail in percent.
<space> -- specifies the size of free space where disk IO can be issued
safely in bytes.
<times> -- specifies how many times failures may happen at most.
Example:
fail_make_request=10,100,0,-1
echo 1 > /sys/blocks/hda/hda1/make-it-fail
generic_make_request() on /dev/hda1 fails once per 10 times.
Cc: Jens Axboe <axboe@suse.de>
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>
block/genhd.c | 31 +++++++++++++++++++++++++++++++
block/ll_rw_blk.c | 35 +++++++++++++++++++++++++++++++++++
fs/partitions/check.c | 27 +++++++++++++++++++++++++++
include/linux/fault-inject.h | 3 +++
include/linux/genhd.h | 4 ++++
lib/Kconfig.debug | 7 +++++++
6 files changed, 107 insertions(+)
Index: work-shouldfail/block/ll_rw_blk.c
===================================================================
--- work-shouldfail.orig/block/ll_rw_blk.c
+++ work-shouldfail/block/ll_rw_blk.c
@@ -28,6 +28,7 @@
#include <linux/interrupt.h>
#include <linux/cpu.h>
#include <linux/blktrace_api.h>
+#include <linux/fault-inject.h>
/*
* for max sense size
@@ -2993,6 +2994,37 @@ static void handle_bad_sector(struct bio
set_bit(BIO_EOF, &bio->bi_flags);
}
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static DEFINE_FAULT_ATTR(fail_make_request_attr);
+
+static int __init setup_fail_make_request(char *str)
+{
+ should_fail_srandom(jiffies);
+ return setup_fault_attr(&fail_make_request_attr, str);
+}
+__setup("fail_make_request=", setup_fail_make_request);
+
+struct fault_attr *fail_make_request = &fail_make_request_attr;
+EXPORT_SYMBOL_GPL(fail_make_request);
+
+static int should_fail_request(struct bio *bio)
+{
+ if ((bio->bi_bdev->bd_disk->flags & GENHD_FL_FAIL) ||
+ (bio->bi_bdev->bd_part && bio->bi_bdev->bd_part->make_it_fail))
+ return should_fail(fail_make_request, bio->bi_size);
+
+ return 0;
+}
+#else
+
+static inline int should_fail_request(struct bio *bio)
+{
+ return 0;
+}
+
+#endif
+
/**
* generic_make_request: hand a buffer to its device driver for I/O
* @bio: The bio describing the location in memory and on the device.
@@ -3077,6 +3109,9 @@ end_io:
if (unlikely(test_bit(QUEUE_FLAG_DEAD, &q->queue_flags)))
goto end_io;
+ if (should_fail_request(bio))
+ goto end_io;
+
/*
* If this device has partitions, remap block n
* of partition p to block n+start(p) of the disk.
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -386,3 +386,10 @@ config FAIL_PAGE_ALLOC
help
This option provides fault-injection capabilitiy for alloc_pages().
+config FAIL_MAKE_REQUEST
+ bool "fault-injection capabilitiy for disk IO"
+ depends on DEBUG_KERNEL
+ select FAULT_INJECTION
+ help
+ This option provides fault-injection capabilitiy to disk IO.
+
Index: work-shouldfail/block/genhd.c
===================================================================
--- work-shouldfail.orig/block/genhd.c
+++ work-shouldfail/block/genhd.c
@@ -412,6 +412,34 @@ static struct disk_attribute disk_attr_s
.show = disk_stats_read
};
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static ssize_t disk_fail_store(struct gendisk * disk,
+ const char *buf, size_t count)
+{
+ int i;
+
+ if (count > 0 && sscanf(buf, "%d", &i) > 0) {
+ if (i == 0)
+ disk->flags &= ~GENHD_FL_FAIL;
+ else
+ disk->flags |= GENHD_FL_FAIL;
+ }
+
+ return count;
+}
+static ssize_t disk_fail_read(struct gendisk * disk, char *page)
+{
+ return sprintf(page, "%d\n", disk->flags & GENHD_FL_FAIL ? 1 : 0);
+}
+static struct disk_attribute disk_attr_fail = {
+ .attr = {.name = "make-it-fail", .mode = S_IRUGO | S_IWUSR },
+ .store = disk_fail_store,
+ .show = disk_fail_read
+};
+
+#endif
+
static struct attribute * default_attrs[] = {
&disk_attr_uevent.attr,
&disk_attr_dev.attr,
@@ -419,6 +447,9 @@ static struct attribute * default_attrs[
&disk_attr_removable.attr,
&disk_attr_size.attr,
&disk_attr_stat.attr,
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+ &disk_attr_fail.attr,
+#endif
NULL,
};
Index: work-shouldfail/include/linux/genhd.h
===================================================================
--- work-shouldfail.orig/include/linux/genhd.h
+++ work-shouldfail/include/linux/genhd.h
@@ -81,6 +81,9 @@ struct hd_struct {
struct kobject *holder_dir;
unsigned ios[2], sectors[2]; /* READs and WRITEs */
int policy, partno;
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+ int make_it_fail;
+#endif
};
#define GENHD_FL_REMOVABLE 1
@@ -88,6 +91,7 @@ struct hd_struct {
#define GENHD_FL_CD 8
#define GENHD_FL_UP 16
#define GENHD_FL_SUPPRESS_PARTITION_INFO 32
+#define GENHD_FL_FAIL 64
struct disk_stats {
unsigned long sectors[2]; /* READs and WRITEs */
Index: work-shouldfail/fs/partitions/check.c
===================================================================
--- work-shouldfail.orig/fs/partitions/check.c
+++ work-shouldfail/fs/partitions/check.c
@@ -265,12 +265,39 @@ static struct part_attribute part_attr_s
.show = part_stat_read
};
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static ssize_t part_fail_store(struct hd_struct * p,
+ const char *buf, size_t count)
+{
+ int i;
+
+ if (count > 0 && sscanf(buf, "%d", &i) > 0)
+ p->make_it_fail = (i == 0) ? 0 : 1;
+
+ return count;
+}
+static ssize_t part_fail_read(struct hd_struct * p, char *page)
+{
+ return sprintf(page, "%d\n", p->make_it_fail);
+}
+static struct part_attribute part_attr_fail = {
+ .attr = {.name = "make-it-fail", .mode = S_IRUGO | S_IWUSR },
+ .store = part_fail_store,
+ .show = part_fail_read
+};
+
+#endif
+
static struct attribute * default_attrs[] = {
&part_attr_uevent.attr,
&part_attr_dev.attr,
&part_attr_start.attr,
&part_attr_size.attr,
&part_attr_stat.attr,
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+ &part_attr_fail.attr,
+#endif
NULL,
};
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
--- work-shouldfail.orig/include/linux/fault-inject.h
+++ work-shouldfail/include/linux/fault-inject.h
@@ -42,6 +42,9 @@ extern struct fault_attr *failslab;
#ifdef CONFIG_FAIL_PAGE_ALLOC
extern struct fault_attr *fail_page_alloc;
#endif
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+extern struct fault_attr *fail_make_request;
+#endif
#endif /* CONFIG_FAULT_INJECTION */
--
^ permalink raw reply [flat|nested] 25+ messages in thread* [patch 6/8] debugfs entries for configuration
2006-09-14 10:20 [patch 0/8] fault-injection capabilities (v3) Akinobu Mita
` (4 preceding siblings ...)
2006-09-14 10:20 ` [patch 5/8] fault-injection capability for disk IO Akinobu Mita
@ 2006-09-14 10:20 ` Akinobu Mita
2006-09-19 5:52 ` Don Mullis
2006-09-14 10:20 ` [patch 7/8] process filtering for fault-injection capabilities Akinobu Mita
2006-09-14 10:20 ` [patch 8/8] stacktrace " Akinobu Mita
7 siblings, 1 reply; 25+ messages in thread
From: Akinobu Mita @ 2006-09-14 10:20 UTC (permalink / raw)
To: linux-kernel; +Cc: ak, akpm, Don Mullis, Akinobu Mita
[-- Attachment #1: knobs.patch --]
[-- Type: text/plain, Size: 6668 bytes --]
This kernel module provides debugfs entries to enable to configure
fault-injection capabilities for failslab, fail_page_alloc, and
fail_make_request.
The slab allocator, the page allocator, and the block layer are initalized
before debugfs is available. and failslab, fail_page_alloc, and
fail_make_request are also enabled at the same time.
So I put the initalization and cleanup for these debugfs entries into
this kernel module.
This module provides the following entries so that we can configure
by writing these files.
/debug/
|-- fail_make_request
| |-- interval
| |-- probability
| |-- space
| `-- times
|-- fail_page_alloc
| |-- interval
| |-- probability
| |-- space
| `-- times
`-- failslab
|-- interval
|-- probability
|-- space
`-- times
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>
lib/Kconfig.debug | 8 ++
lib/Makefile | 1
lib/fault-inject-debugfs.c | 179 +++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 188 insertions(+)
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -393,3 +393,11 @@ config FAIL_MAKE_REQUEST
help
This option provides fault-injection capabilitiy to disk IO.
+config FAULT_INJECTION_DEBUGFS
+ tristate "runtime configuration for fault-injection capabilities"
+ depends on DEBUG_KERNEL && SYSFS && FAULT_INJECTION
+ select DEBUG_FS
+ help
+ This option provides kernel module that provides runtime
+ configuration interface by debugfs.
+
Index: work-shouldfail/lib/fault-inject-debugfs.c
===================================================================
--- /dev/null
+++ work-shouldfail/lib/fault-inject-debugfs.c
@@ -0,0 +1,179 @@
+#include <linux/module.h>
+#include <linux/fault-inject.h>
+#include <linux/debugfs.h>
+
+struct fault_attr_entries {
+ struct dentry *dir;
+ struct dentry *probability_file;
+ struct dentry *interval_file;
+ struct dentry *times_file;
+ struct dentry *space_file;
+ struct dentry *process_filter_file;
+};
+
+static void debugfs_ul_set(void *data, u64 val)
+{
+ *(unsigned long *)data = val;
+}
+
+static u64 debugfs_ul_get(void *data)
+{
+ return *(unsigned long *)data;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_ul, debugfs_ul_get, debugfs_ul_set, "%llu\n");
+
+static struct dentry *debugfs_create_ul(const char *name, mode_t mode,
+ struct dentry *parent, unsigned long *value)
+{
+ return debugfs_create_file(name, mode, parent, value, &fops_ul);
+}
+
+static void debugfs_atomic_t_set(void *data, u64 val)
+{
+ atomic_set((atomic_t *)data, val);
+}
+
+static u64 debugfs_atomic_t_get(void *data)
+{
+ return atomic_read((atomic_t *)data);
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_atomic_t, debugfs_atomic_t_get,
+ debugfs_atomic_t_set, "%lld\n");
+
+static struct dentry *debugfs_create_atomic_t(const char *name, mode_t mode,
+ struct dentry *parent, atomic_t *value)
+{
+ return debugfs_create_file(name, mode, parent, value, &fops_atomic_t);
+}
+
+static void cleanup_fault_attr_entries(struct fault_attr_entries *entries)
+{
+ if (entries->dir) {
+ if (entries->probability_file) {
+ debugfs_remove(entries->probability_file);
+ entries->probability_file = NULL;
+ }
+ if (entries->interval_file) {
+ debugfs_remove(entries->interval_file);
+ entries->interval_file = NULL;
+ }
+ if (entries->times_file) {
+ debugfs_remove(entries->times_file);
+ entries->times_file = NULL;
+ }
+ if (entries->space_file) {
+ debugfs_remove(entries->space_file);
+ entries->space_file = NULL;
+ }
+ if (entries->process_filter_file) {
+ debugfs_remove(entries->process_filter_file);
+ entries->process_filter_file = NULL;
+ }
+ debugfs_remove(entries->dir);
+ entries->dir = NULL;
+ }
+}
+
+static int init_fault_attr_entries(struct fault_attr_entries *entries,
+ struct fault_attr *attr, const char *name)
+{
+ mode_t mode = S_IFREG | S_IRUSR | S_IWUSR;
+ struct dentry *dir;
+ struct dentry *file;
+
+ memset(entries, 0, sizeof(*entries));
+
+ dir = debugfs_create_dir(name, NULL);
+ if (!dir)
+ goto fail;
+ entries->dir = dir;
+
+ file = debugfs_create_ul("probability", mode, dir, &attr->probability);
+ if (!file)
+ goto fail;
+ entries->probability_file = file;
+
+ file = debugfs_create_ul("interval", mode, dir, &attr->interval);
+ if (!file)
+ goto fail;
+ entries->interval_file = file;
+
+ file = debugfs_create_atomic_t("times", mode, dir, &attr->times);
+ if (!file)
+ goto fail;
+ entries->times_file = file;
+
+ file = debugfs_create_atomic_t("space", mode, dir, &attr->space);
+ if (!file)
+ goto fail;
+ entries->space_file = file;
+
+ file = debugfs_create_bool("process-filter", mode, dir,
+ &attr->process_filter);
+ if (!file)
+ goto fail;
+ entries->process_filter_file = file;
+
+ return 0;
+fail:
+ cleanup_fault_attr_entries(entries);
+ return -ENOMEM;
+}
+
+#ifdef CONFIG_FAILSLAB
+static struct fault_attr_entries failslab_entries;
+#endif
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+static struct fault_attr_entries fail_page_alloc_entries;
+#endif
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+static struct fault_attr_entries fail_make_request_entries;
+#endif
+
+static void cleanup_entries(void)
+{
+#ifdef CONFIG_FAILSLAB
+ cleanup_fault_attr_entries(&failslab_entries);
+#endif
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+ cleanup_fault_attr_entries(&fail_page_alloc_entries);
+#endif
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+ cleanup_fault_attr_entries(&fail_make_request_entries);
+#endif
+}
+
+static int init_entries(void)
+{
+ int err;
+
+#ifdef CONFIG_FAILSLAB
+ err = init_fault_attr_entries(&failslab_entries, failslab, "failslab");
+ if (err)
+ goto fail;
+#endif
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+ err = init_fault_attr_entries(&fail_page_alloc_entries, fail_page_alloc,
+ "fail_page_alloc");
+ if (err)
+ goto fail;
+#endif
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+ err = init_fault_attr_entries(&fail_make_request_entries,
+ fail_make_request, "fail_make_request");
+ if (err)
+ goto fail;
+#endif
+
+ return 0;
+fail:
+ cleanup_entries();
+
+ return err;
+}
+
+module_init(init_entries);
+module_exit(cleanup_entries);
+MODULE_LICENSE("GPL");
Index: work-shouldfail/lib/Makefile
===================================================================
--- work-shouldfail.orig/lib/Makefile
+++ work-shouldfail/lib/Makefile
@@ -53,6 +53,7 @@ obj-$(CONFIG_AUDIT_GENERIC) += audit.o
obj-$(CONFIG_SWIOTLB) += swiotlb.o
obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o
+obj-$(CONFIG_FAULT_INJECTION_DEBUGFS) += fault-inject-debugfs.o
hostprogs-y := gen_crc32table
clean-files := crc32table.h
--
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [patch 6/8] debugfs entries for configuration
2006-09-14 10:20 ` [patch 6/8] debugfs entries for configuration Akinobu Mita
@ 2006-09-19 5:52 ` Don Mullis
0 siblings, 0 replies; 25+ messages in thread
From: Don Mullis @ 2006-09-19 5:52 UTC (permalink / raw)
To: Akinobu Mita; +Cc: linux-kernel, ak, akpm
Factor out process-filter functionality, move later in series to
process-filter-specific patch.
Signed-off-by: Don Mullis <dwm@meer.net>
---
lib/fault-inject-debugfs.c | 11 -----------
1 file changed, 11 deletions(-)
Index: linux-2.6.17/lib/fault-inject-debugfs.c
===================================================================
--- linux-2.6.17.orig/lib/fault-inject-debugfs.c
+++ linux-2.6.17/lib/fault-inject-debugfs.c
@@ -8,7 +8,6 @@ struct fault_attr_entries {
struct dentry *interval_file;
struct dentry *times_file;
struct dentry *space_file;
- struct dentry *process_filter_file;
};
static void debugfs_ul_set(void *data, u64 val)
@@ -67,10 +66,6 @@ static void cleanup_fault_attr_entries(s
debugfs_remove(entries->space_file);
entries->space_file = NULL;
}
- if (entries->process_filter_file) {
- debugfs_remove(entries->process_filter_file);
- entries->process_filter_file = NULL;
- }
debugfs_remove(entries->dir);
entries->dir = NULL;
}
@@ -110,12 +105,6 @@ static int init_fault_attr_entries(struc
goto fail;
entries->space_file = file;
- file = debugfs_create_bool("process-filter", mode, dir,
- &attr->process_filter);
- if (!file)
- goto fail;
- entries->process_filter_file = file;
-
return 0;
fail:
cleanup_fault_attr_entries(entries);
^ permalink raw reply [flat|nested] 25+ messages in thread
* [patch 7/8] process filtering for fault-injection capabilities
2006-09-14 10:20 [patch 0/8] fault-injection capabilities (v3) Akinobu Mita
` (5 preceding siblings ...)
2006-09-14 10:20 ` [patch 6/8] debugfs entries for configuration Akinobu Mita
@ 2006-09-14 10:20 ` Akinobu Mita
2006-09-19 5:54 ` Don Mullis
2006-09-19 5:55 ` Don Mullis
2006-09-14 10:20 ` [patch 8/8] stacktrace " Akinobu Mita
7 siblings, 2 replies; 25+ messages in thread
From: Akinobu Mita @ 2006-09-14 10:20 UTC (permalink / raw)
To: linux-kernel; +Cc: ak, akpm, Don Mullis, Akinobu Mita
[-- Attachment #1: process-filter.patch --]
[-- Type: text/plain, Size: 5872 bytes --]
This patch provides process filtering feature.
The process filter allows failing only permitted processes
by /proc/<pid>/make-it-fail
Please see the example that demostrates how to inject slab allocation
failures into module init/cleanup code
in Documentation/fault-injection/fault-injection.txt
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>
fs/proc/base.c | 77 +++++++++++++++++++++++++++++++++++++++++++
include/linux/fault-inject.h | 3 +
include/linux/sched.h | 3 +
lib/fault-inject.c | 13 +++++++
4 files changed, 96 insertions(+)
Index: work-shouldfail/fs/proc/base.c
===================================================================
--- work-shouldfail.orig/fs/proc/base.c
+++ work-shouldfail/fs/proc/base.c
@@ -138,6 +138,9 @@ enum pid_directory_inos {
#endif
PROC_TGID_OOM_SCORE,
PROC_TGID_OOM_ADJUST,
+#ifdef CONFIG_FAULT_INJECTION
+ PROC_TGID_FAULT_INJECTION,
+#endif
PROC_TID_INO,
PROC_TID_STATUS,
PROC_TID_MEM,
@@ -181,6 +184,9 @@ enum pid_directory_inos {
#endif
PROC_TID_OOM_SCORE,
PROC_TID_OOM_ADJUST,
+#ifdef CONFIG_FAULT_INJECTION
+ PROC_TID_FAULT_INJECTION,
+#endif
/* Add new entries before this */
PROC_TID_FD_DIR = 0x8000, /* 0x8000-0xffff */
@@ -240,6 +246,9 @@ static struct pid_entry tgid_base_stuff[
#ifdef CONFIG_AUDITSYSCALL
E(PROC_TGID_LOGINUID, "loginuid", S_IFREG|S_IWUSR|S_IRUGO),
#endif
+#ifdef CONFIG_FAULT_INJECTION
+ E(PROC_TGID_FAULT_INJECTION, "make-it-fail", S_IFREG|S_IWUSR|S_IRUGO),
+#endif
{0,0,NULL,0}
};
static struct pid_entry tid_base_stuff[] = {
@@ -282,6 +291,9 @@ static struct pid_entry tid_base_stuff[]
#ifdef CONFIG_AUDITSYSCALL
E(PROC_TID_LOGINUID, "loginuid", S_IFREG|S_IWUSR|S_IRUGO),
#endif
+#ifdef CONFIG_FAULT_INJECTION
+ E(PROC_TID_FAULT_INJECTION, "make-it-fail", S_IFREG|S_IWUSR|S_IRUGO),
+#endif
{0,0,NULL,0}
};
@@ -992,6 +1004,65 @@ static struct file_operations proc_login
};
#endif
+#ifdef CONFIG_FAULT_INJECTION
+static ssize_t proc_fault_inject_read(struct file * file, char __user * buf,
+ size_t count, loff_t *ppos)
+{
+ struct task_struct *task = get_proc_task(file->f_dentry->d_inode);
+ char buffer[PROC_NUMBUF];
+ size_t len;
+ int make_it_fail;
+ loff_t __ppos = *ppos;
+
+ if (!task)
+ return -ESRCH;
+ make_it_fail = task->make_it_fail;
+ put_task_struct(task);
+
+ len = snprintf(buffer, sizeof(buffer), "%i\n", make_it_fail);
+ if (__ppos >= len)
+ return 0;
+ if (count > len-__ppos)
+ count = len-__ppos;
+ if (copy_to_user(buf, buffer + __ppos, count))
+ return -EFAULT;
+ *ppos = __ppos + count;
+ return count;
+}
+
+static ssize_t proc_fault_inject_write(struct file * file,
+ const char __user * buf, size_t count, loff_t *ppos)
+{
+ struct task_struct *task;
+ char buffer[PROC_NUMBUF], *end;
+ int make_it_fail;
+
+ if (!capable(CAP_SYS_RESOURCE))
+ return -EPERM;
+ memset(buffer, 0, sizeof(buffer));
+ if (count > sizeof(buffer) - 1)
+ count = sizeof(buffer) - 1;
+ if (copy_from_user(buffer, buf, count))
+ return -EFAULT;
+ make_it_fail = simple_strtol(buffer, &end, 0);
+ if (*end == '\n')
+ end++;
+ task = get_proc_task(file->f_dentry->d_inode);
+ if (!task)
+ return -ESRCH;
+ task->make_it_fail = make_it_fail;
+ put_task_struct(task);
+ if (end - buffer == 0)
+ return -EIO;
+ return end - buffer;
+}
+
+static struct file_operations proc_fault_inject_operations = {
+ .read = proc_fault_inject_read,
+ .write = proc_fault_inject_write,
+};
+#endif
+
#ifdef CONFIG_SECCOMP
static ssize_t seccomp_read(struct file *file, char __user *buf,
size_t count, loff_t *ppos)
@@ -1834,6 +1905,12 @@ static struct dentry *proc_pident_lookup
inode->i_fop = &proc_loginuid_operations;
break;
#endif
+#ifdef CONFIG_FAULT_INJECTION
+ case PROC_TID_FAULT_INJECTION:
+ case PROC_TGID_FAULT_INJECTION:
+ inode->i_fop = &proc_fault_inject_operations;
+ break;
+#endif
default:
printk("procfs: impossible type (%d)",p->type);
iput(inode);
Index: work-shouldfail/include/linux/sched.h
===================================================================
--- work-shouldfail.orig/include/linux/sched.h
+++ work-shouldfail/include/linux/sched.h
@@ -996,6 +996,9 @@ struct task_struct {
#ifdef CONFIG_TASK_DELAY_ACCT
struct task_delay_info *delays;
#endif
+#ifdef CONFIG_FAULT_INJECTION
+ int make_it_fail;
+#endif
};
static inline pid_t process_group(struct task_struct *tsk)
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
--- work-shouldfail.orig/include/linux/fault-inject.h
+++ work-shouldfail/include/linux/fault-inject.h
@@ -27,6 +27,9 @@ struct fault_attr {
atomic_t space;
unsigned long count;
+
+ /* A value of '0' means process filter is disabled. */
+ u32 process_filter;
};
#define DEFINE_FAULT_ATTR(name) \
Index: work-shouldfail/lib/fault-inject.c
===================================================================
--- work-shouldfail.orig/lib/fault-inject.c
+++ work-shouldfail/lib/fault-inject.c
@@ -5,6 +5,7 @@
#include <linux/types.h>
#include <linux/fs.h>
#include <linux/module.h>
+#include <linux/interrupt.h>
#include <linux/fault-inject.h>
int setup_fault_attr(struct fault_attr *attr, char *str)
@@ -49,6 +50,15 @@ void should_fail_srandom(unsigned long e
should_fail_random();
}
+static int fail_process(struct fault_attr *attr, struct task_struct *task)
+{
+ /* process filter is disabled */
+ if (!attr->process_filter)
+ return 1;
+
+ return !in_interrupt() && task->make_it_fail;
+}
+
/*
* This code is stolen from failmalloc-1.0
* http://www.nongnu.org/failmalloc/
@@ -56,6 +66,9 @@ void should_fail_srandom(unsigned long e
int should_fail(struct fault_attr *attr, ssize_t size)
{
+ if (!fail_process(attr, current))
+ return 0;
+
if (atomic_read(&max_failures(attr)) == 0)
return 0;
--
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [patch 7/8] process filtering for fault-injection capabilities
2006-09-14 10:20 ` [patch 7/8] process filtering for fault-injection capabilities Akinobu Mita
@ 2006-09-19 5:54 ` Don Mullis
2006-09-19 9:05 ` Akinobu Mita
2006-09-19 5:55 ` Don Mullis
1 sibling, 1 reply; 25+ messages in thread
From: Don Mullis @ 2006-09-19 5:54 UTC (permalink / raw)
To: Akinobu Mita; +Cc: linux-kernel, ak, akpm
Add functionality to the process_filter variable: A negative argument
injects failures for only for pid==-process_filter, thereby permitting
per-process failures from boot time.
Restore process-filter arg to kernel command line.
Reintroduce process_filter_file addition to debugfs,
factored out earlier in series.
Add printk, called upon each failure injection.
Signed-off-by: Don Mullis <dwm@meer.net>
---
Documentation/fault-injection/fault-injection.txt | 9 +++++++
include/linux/fault-inject.h | 1
lib/fault-inject-debugfs.c | 11 ++++++++
lib/fault-inject.c | 27 ++++++++++++++++------
4 files changed, 40 insertions(+), 8 deletions(-)
Index: linux-2.6.17/include/linux/fault-inject.h
===================================================================
--- linux-2.6.17.orig/include/linux/fault-inject.h
+++ linux-2.6.17/include/linux/fault-inject.h
@@ -15,11 +15,8 @@ struct fault_attr {
unsigned long interval;
atomic_t times;
atomic_t space;
-
unsigned long count;
-
- /* A value of '0' means process filter is disabled. */
- u32 process_filter;
+ atomic_t process_filter;
};
#define DEFINE_FAULT_ATTR(name) \
Index: linux-2.6.17/lib/fault-inject.c
===================================================================
--- linux-2.6.17.orig/lib/fault-inject.c
+++ linux-2.6.17/lib/fault-inject.c
@@ -14,11 +14,12 @@ int setup_fault_attr(struct fault_attr *
unsigned long interval;
int times;
int space;
+ int process_filter;
- /* "<interval>,<probability>,<space>,<times>" */
- if (sscanf(str, "%lu,%lu,%d,%d",
- &interval, &probability, &space, ×) < 4) {
- printk(KERN_WARNING "SHOULD_FAIL: failed to parse arguments\n");
+ /* "<interval>,<probability>,<process-filter>,<space>,<times>" */
+ if (sscanf(str, "%lu,%lu,%d,%d,%d",
+ &interval, &probability, &process_filter, &space, ×) < 4) {
+ printk(KERN_WARNING "FAULT_INJECTION: failed to parse arguments\n");
return 0;
}
@@ -26,6 +27,7 @@ int setup_fault_attr(struct fault_attr *
attr->interval = interval;
atomic_set(&attr->times, times);
atomic_set(&attr->space, space);
+ atomic_set(&attr->process_filter, process_filter);
return 1;
}
@@ -53,10 +55,20 @@ void should_fail_srandom(unsigned long e
static int fail_process(struct fault_attr *attr, struct task_struct *task)
{
/* process filter is disabled */
- if (!attr->process_filter)
+ if (atomic_read(&attr->process_filter)==0)
return 1;
- return !in_interrupt() && task->make_it_fail;
+ /* controlled by /proc/<pid>/make-it-fail, but boolean not set */
+ if (atomic_read(&attr->process_filter)>0 && !task->make_it_fail)
+ return 0;
+
+ /* enabled for single pid, but no match with pid */
+ if (atomic_read(&attr->process_filter)<0 &&
+ -atomic_read(&attr->process_filter)!=task->pid)
+ return 0;
+
+ /* XX Are we sure we never want calls from interrupt context to fail? */
+ return !in_interrupt();
}
/*
@@ -89,7 +101,8 @@ int should_fail(struct fault_attr *attr,
return 0;
fail:
-
+ printk(KERN_NOTICE "FAULT_INJECTION: forcing a failure, pid==%d\n",
+ current->pid);
if (atomic_read(&max_failures(attr)) != -1)
atomic_dec_not_zero(&max_failures(attr));
Index: linux-2.6.17/Documentation/fault-injection/fault-injection.txt
===================================================================
--- linux-2.6.17.orig/Documentation/fault-injection/fault-injection.txt
+++ linux-2.6.17/Documentation/fault-injection/fault-injection.txt
@@ -52,6 +52,15 @@ configuration of fault-injection capabil
on each call to should_fail(,size). Failure injection is
suppressed until "space" reaches zero.
+- /debug/*/process-filter:
+
+ filter failures by pid.
+ A value of '0' disables filtering by process.
+ Any positive value limits failures to only processes indicated by
+ /proc/<pid>/make-it-fail==1.
+ A negative value means that failures are enabled for
+ pid==-process_filter irrespective of /proc/<pid>/make-it-fail.
+
o Boot option
In order to inject faults while debugfs is not available (early boot time),
Index: linux-2.6.17/lib/fault-inject-debugfs.c
===================================================================
--- linux-2.6.17.orig/lib/fault-inject-debugfs.c
+++ linux-2.6.17/lib/fault-inject-debugfs.c
@@ -8,6 +8,7 @@ struct fault_attr_entries {
struct dentry *interval_file;
struct dentry *times_file;
struct dentry *space_file;
+ struct dentry *process_filter_file;
};
static void debugfs_ul_set(void *data, u64 val)
@@ -66,6 +67,10 @@ static void cleanup_fault_attr_entries(s
debugfs_remove(entries->space_file);
entries->space_file = NULL;
}
+ if (entries->process_filter_file) {
+ debugfs_remove(entries->process_filter_file);
+ entries->process_filter_file = NULL;
+ }
debugfs_remove(entries->dir);
entries->dir = NULL;
}
@@ -105,6 +110,12 @@ static int init_fault_attr_entries(struc
goto fail;
entries->space_file = file;
+ file = debugfs_create_atomic_t("process-filter", mode, dir,
+ &attr->process_filter);
+ if (!file)
+ goto fail;
+ entries->process_filter_file = file;
+
return 0;
fail:
cleanup_fault_attr_entries(entries);
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [patch 7/8] process filtering for fault-injection capabilities
2006-09-19 5:54 ` Don Mullis
@ 2006-09-19 9:05 ` Akinobu Mita
2006-09-19 17:38 ` Don Mullis
0 siblings, 1 reply; 25+ messages in thread
From: Akinobu Mita @ 2006-09-19 9:05 UTC (permalink / raw)
To: Don Mullis; +Cc: linux-kernel, ak, akpm
On Mon, Sep 18, 2006 at 10:54:51PM -0700, Don Mullis wrote:
> Add functionality to the process_filter variable: A negative argument
> injects failures for only for pid==-process_filter, thereby permitting
> per-process failures from boot time.
>
Is it better to add new filter for this purpose?
Because someone may want to filter by tgid instead of pid.
- positive value is for task->pid
- nevative value is for task->tgid
> Add printk, called upon each failure injection.
>
This printk() will be very useful. But it is better to make
configurable, and the pid is not reliable in interrupt context.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [patch 7/8] process filtering for fault-injection capabilities
2006-09-19 9:05 ` Akinobu Mita
@ 2006-09-19 17:38 ` Don Mullis
0 siblings, 0 replies; 25+ messages in thread
From: Don Mullis @ 2006-09-19 17:38 UTC (permalink / raw)
To: Akinobu Mita; +Cc: linux-kernel, ak, akpm
On Tue, 2006-09-19 at 17:05 +0800, Akinobu Mita wrote:
> On Mon, Sep 18, 2006 at 10:54:51PM -0700, Don Mullis wrote:
> > Add functionality to the process_filter variable: A negative argument
> > injects failures for only for pid==-process_filter, thereby permitting
> > per-process failures from boot time.
> >
>
> Is it better to add new filter for this purpose?
> Because someone may want to filter by tgid instead of pid.
>
> - positive value is for task->pid
> - nevative value is for task->tgid
Your idea sounds good to me.
> > Add printk, called upon each failure injection.
> >
>
> This printk() will be very useful. But it is better to make
> configurable, and the pid is not reliable in interrupt context.
Okay. We do need some output now, for testing.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [patch 7/8] process filtering for fault-injection capabilities
2006-09-14 10:20 ` [patch 7/8] process filtering for fault-injection capabilities Akinobu Mita
2006-09-19 5:54 ` Don Mullis
@ 2006-09-19 5:55 ` Don Mullis
1 sibling, 0 replies; 25+ messages in thread
From: Don Mullis @ 2006-09-19 5:55 UTC (permalink / raw)
To: Akinobu Mita; +Cc: linux-kernel, ak, akpm
Temporary fix-up so that next patch:
"[patch 8/8] stacktrace filtering for fault-injection capabilities"
applies cleanly.
Signed-off-by: Don Mullis <dwm@meer.net>
---
include/linux/fault-inject.h | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
Index: linux-2.6.17/include/linux/fault-inject.h
===================================================================
--- linux-2.6.17.orig/include/linux/fault-inject.h
+++ linux-2.6.17/include/linux/fault-inject.h
@@ -16,7 +16,9 @@ struct fault_attr {
atomic_t times;
atomic_t space;
unsigned long count;
- atomic_t process_filter;
+
+ /* A value of '0' means process filter is disabled. */
+ u32 process_filter;
};
#define DEFINE_FAULT_ATTR(name) \
^ permalink raw reply [flat|nested] 25+ messages in thread
* [patch 8/8] stacktrace filtering for fault-injection capabilities
2006-09-14 10:20 [patch 0/8] fault-injection capabilities (v3) Akinobu Mita
` (6 preceding siblings ...)
2006-09-14 10:20 ` [patch 7/8] process filtering for fault-injection capabilities Akinobu Mita
@ 2006-09-14 10:20 ` Akinobu Mita
2006-09-19 5:56 ` Don Mullis
2006-09-19 5:57 ` Don Mullis
7 siblings, 2 replies; 25+ messages in thread
From: Akinobu Mita @ 2006-09-14 10:20 UTC (permalink / raw)
To: linux-kernel; +Cc: ak, akpm, Don Mullis, Valdis.Kletnieks, Akinobu Mita
[-- Attachment #1: module-filter.patch --]
[-- Type: text/plain, Size: 6116 bytes --]
This patch provides stacktrace filtering feature.
The stacktrace filter allows failing only for the caller you are
interested in.
stacktrace filter is enabled by setting the value of
/debugfs/*/stacktrace-depth more than 0.
and specify the range of the virtual address
by the /debugfs/*/address-start and /debugfs/*/address-end
Please see the example that demostrates how to inject slab allocation
failures only for a specific module
in Documentation/fault-injection/fault-injection.txt
Cc: Valdis.Kletnieks@vt.edu
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>
include/linux/fault-inject.h | 15 +++++++++
lib/Kconfig.debug | 2 +
lib/fault-inject-debugfs.c | 32 +++++++++++++++++++
lib/fault-inject.c | 71 +++++++++++++++++++++++++++++++++++++++++++
4 files changed, 120 insertions(+)
Index: work-shouldfail/lib/fault-inject.c
===================================================================
--- work-shouldfail.orig/lib/fault-inject.c
+++ work-shouldfail/lib/fault-inject.c
@@ -6,6 +6,9 @@
#include <linux/fs.h>
#include <linux/module.h>
#include <linux/interrupt.h>
+#include <linux/unwind.h>
+#include <linux/stacktrace.h>
+#include <linux/kallsyms.h>
#include <linux/fault-inject.h>
int setup_fault_attr(struct fault_attr *attr, char *str)
@@ -59,6 +62,72 @@ static int fail_process(struct fault_att
return !in_interrupt() && task->make_it_fail;
}
+#ifdef CONFIG_STACK_UNWIND
+
+static asmlinkage int fail_stacktrace_callback(struct unwind_frame_info *info,
+ void *arg)
+{
+ int depth;
+ struct fault_attr *attr = arg;
+
+ for (depth = 0; depth < attr->stacktrace_depth
+ && unwind(info) == 0 && UNW_PC(info); depth++) {
+ if (arch_unw_user_mode(info))
+ break;
+ if (attr->address_start <= UNW_PC(info) &&
+ UNW_PC(info) < attr->address_end)
+ return 1;
+ }
+ return 0;
+}
+
+static int fail_stacktrace(struct fault_attr *attr)
+{
+ struct unwind_frame_info info;
+
+ /* stacktrace filter is disabled */
+ if (attr->stacktrace_depth == 0)
+ return 1;
+
+ return unwind_init_running(&info, fail_stacktrace_callback, attr);
+}
+
+#elif defined(CONFIG_STACKTRACE)
+
+#include <linux/stacktrace.h>
+
+#define MAX_STACK_TRACE_DEPTH 10
+
+static int fail_stacktrace(struct fault_attr *attr)
+{
+ struct stack_trace trace;
+ int depth = attr->stacktrace_depth;
+ unsigned long entries[MAX_STACK_TRACE_DEPTH];
+ int n;
+
+ /* stacktrace filter is disabled */
+ if (depth == 0)
+ return 1;
+
+ trace.nr_entries = 0;
+ trace.entries = entries;
+ trace.max_entries = (depth < MAX_STACK_TRACE_DEPTH) ?
+ depth : MAX_STACK_TRACE_DEPTH;
+
+ save_stack_trace(&trace, NULL, 0, 1);
+ for (n = 0; n < trace.nr_entries; n++)
+ if (attr->address_start <= entries[n] &&
+ entries[n] < attr->address_end)
+ return 1;
+ return 0;
+}
+
+#else
+
+#define fail_stacktrace(attr) (0)
+
+#endif
+
/*
* This code is stolen from failmalloc-1.0
* http://www.nongnu.org/failmalloc/
@@ -68,6 +137,8 @@ int should_fail(struct fault_attr *attr,
{
if (!fail_process(attr, current))
return 0;
+ if (!fail_stacktrace(attr))
+ return 0;
if (atomic_read(&max_failures(attr)) == 0)
return 0;
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
--- work-shouldfail.orig/include/linux/fault-inject.h
+++ work-shouldfail/include/linux/fault-inject.h
@@ -30,6 +30,21 @@ struct fault_attr {
/* A value of '0' means process filter is disabled. */
u32 process_filter;
+
+ /*
+ * maximam number of stacktrace depth walking allowed
+ * A value of '0' means stacktrace filter is disabled.
+ */
+ unsigned long stacktrace_depth;
+
+ /*
+ * If stacktrace_depth is enabled, it allows failing only when it
+ * has been called from the virtual addresses in the range
+ * 'address_start' to 'address_end-1'
+ */
+ unsigned long address_start;
+ unsigned long address_end;
+
};
#define DEFINE_FAULT_ATTR(name) \
Index: work-shouldfail/lib/fault-inject-debugfs.c
===================================================================
--- work-shouldfail.orig/lib/fault-inject-debugfs.c
+++ work-shouldfail/lib/fault-inject-debugfs.c
@@ -9,6 +9,9 @@ struct fault_attr_entries {
struct dentry *times_file;
struct dentry *space_file;
struct dentry *process_filter_file;
+ struct dentry *stacktrace_depth_file;
+ struct dentry *address_start_file;
+ struct dentry *address_end_file;
};
static void debugfs_ul_set(void *data, u64 val)
@@ -71,6 +74,18 @@ static void cleanup_fault_attr_entries(s
debugfs_remove(entries->process_filter_file);
entries->process_filter_file = NULL;
}
+ if (entries->stacktrace_depth_file) {
+ debugfs_remove(entries->stacktrace_depth_file);
+ entries->stacktrace_depth_file = NULL;
+ }
+ if (entries->address_start_file) {
+ debugfs_remove(entries->address_start_file);
+ entries->address_start_file = NULL;
+ }
+ if (entries->address_end_file) {
+ debugfs_remove(entries->address_end_file);
+ entries->address_end_file = NULL;
+ }
debugfs_remove(entries->dir);
entries->dir = NULL;
}
@@ -116,6 +131,23 @@ static int init_fault_attr_entries(struc
goto fail;
entries->process_filter_file = file;
+ file = debugfs_create_ul("stacktrace-depth", mode, dir,
+ &attr->stacktrace_depth);
+ if (!file)
+ goto fail;
+ entries->stacktrace_depth_file = file;
+
+ file = debugfs_create_ul("address-start", mode, dir,
+ &attr->address_start);
+ if (!file)
+ goto fail;
+ entries->address_start_file = file;
+
+ file = debugfs_create_ul("address-end", mode, dir, &attr->address_end);
+ if (!file)
+ goto fail;
+ entries->address_end_file = file;
+
return 0;
fail:
cleanup_fault_attr_entries(entries);
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -371,6 +371,8 @@ config RCU_TORTURE_TEST
config FAULT_INJECTION
bool
+ select STACKTRACE
+ select FRAME_POINTER
config FAILSLAB
bool "fault-injection capabilitiy for kmalloc"
--
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [patch 8/8] stacktrace filtering for fault-injection capabilities
2006-09-14 10:20 ` [patch 8/8] stacktrace " Akinobu Mita
@ 2006-09-19 5:56 ` Don Mullis
2006-09-19 9:08 ` Akinobu Mita
2006-09-19 5:57 ` Don Mullis
1 sibling, 1 reply; 25+ messages in thread
From: Don Mullis @ 2006-09-19 5:56 UTC (permalink / raw)
To: Akinobu Mita; +Cc: linux-kernel, ak, akpm, Valdis.Kletnieks
Undo temporary fix-up for clean application of patch
"[patch 8/8] stacktrace filtering for fault-injection capabilities".
Fix bug in !(CONFIG_STACK_UNWIND || CONFIG_STACKTRACE) case, based on
code inspection only. Anyone with a non-i386, -x86_64, -s390 willing
to test this?
Reintroduce stacktrace-specific documentation, factored out earlier.
Signed-off-by: Don Mullis <dwm@meer.net>
---
Documentation/fault-injection/fault-injection.txt | 13 +++++++++++++
include/linux/fault-inject.h | 2 +-
lib/fault-inject.c | 2 +-
3 files changed, 15 insertions(+), 2 deletions(-)
Index: linux-2.6.17/lib/fault-inject.c
===================================================================
--- linux-2.6.17.orig/lib/fault-inject.c
+++ linux-2.6.17/lib/fault-inject.c
@@ -136,7 +136,7 @@ static int fail_stacktrace(struct fault_
#else
-#define fail_stacktrace(attr) (0)
+#define fail_stacktrace(attr) (1)
#endif
Index: linux-2.6.17/Documentation/fault-injection/fault-injection.txt
===================================================================
--- linux-2.6.17.orig/Documentation/fault-injection/fault-injection.txt
+++ linux-2.6.17/Documentation/fault-injection/fault-injection.txt
@@ -61,6 +61,19 @@ configuration of fault-injection capabil
A negative value means that failures are enabled for
pid==-process_filter irrespective of /proc/<pid>/make-it-fail.
+- /debug/*/stacktrace-depth:
+
+ specifies the maximum stacktrace depth walked during search
+ for a caller within [address-start,address-end). A value of 0
+ disables the stacktrace filter.
+
+- /debug/*/address-start:
+- /debug/*/address-end:
+
+ specifies the range of virtual addresses tested during
+ stacktrace walking. Failure is injected only if some caller
+ in the walked stacktrace lies within this range.
+
o Boot option
In order to inject faults while debugfs is not available (early boot time),
Index: linux-2.6.17/include/linux/fault-inject.h
===================================================================
--- linux-2.6.17.orig/include/linux/fault-inject.h
+++ linux-2.6.17/include/linux/fault-inject.h
@@ -16,7 +16,7 @@ struct fault_attr {
atomic_t times;
atomic_t space;
unsigned long count;
- u32 process_filter;
+ atomic_t process_filter;
unsigned long stacktrace_depth;
unsigned long address_start;
unsigned long address_end;
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [patch 8/8] stacktrace filtering for fault-injection capabilities
2006-09-19 5:56 ` Don Mullis
@ 2006-09-19 9:08 ` Akinobu Mita
0 siblings, 0 replies; 25+ messages in thread
From: Akinobu Mita @ 2006-09-19 9:08 UTC (permalink / raw)
To: Don Mullis; +Cc: linux-kernel, ak, akpm, Valdis.Kletnieks
On Mon, Sep 18, 2006 at 10:56:28PM -0700, Don Mullis wrote:
> Fix bug in !(CONFIG_STACK_UNWIND || CONFIG_STACKTRACE) case, based on
> code inspection only. Anyone with a non-i386, -x86_64, -s390 willing
> to test this?
Yes, it didn't work completely without this patch on other architectures.
Thank you.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [patch 8/8] stacktrace filtering for fault-injection capabilities
2006-09-14 10:20 ` [patch 8/8] stacktrace " Akinobu Mita
2006-09-19 5:56 ` Don Mullis
@ 2006-09-19 5:57 ` Don Mullis
2006-09-19 9:09 ` Akinobu Mita
1 sibling, 1 reply; 25+ messages in thread
From: Don Mullis @ 2006-09-19 5:57 UTC (permalink / raw)
To: Akinobu Mita; +Cc: linux-kernel, ak, akpm, Valdis.Kletnieks
Push fault-injection entries earlier in the list, so that they appear
nested under DEBUG_KERNEL in menuconfig/xconfig.
Signed-off-by: Don Mullis <dwm@meer.net>
---
lib/Kconfig.debug | 70 +++++++++++++++++++++++++++---------------------------
1 file changed, 35 insertions(+), 35 deletions(-)
Index: linux-2.6.17/lib/Kconfig.debug
===================================================================
--- linux-2.6.17.orig/lib/Kconfig.debug
+++ linux-2.6.17/lib/Kconfig.debug
@@ -294,6 +294,41 @@ config DEBUG_INFO
If unsure, say N.
+config FAULT_INJECTION
+ bool
+ depends on DEBUG_KERNEL
+ select STACKTRACE
+ select FRAME_POINTER
+
+config FAILSLAB
+ bool "fault-injection capability for kmalloc"
+ depends on DEBUG_KERNEL
+ select FAULT_INJECTION
+ help
+ This option provides fault-injection capability for kmalloc.
+
+config FAIL_PAGE_ALLOC
+ bool "fault-injection capability for alloc_pages()"
+ depends on DEBUG_KERNEL
+ select FAULT_INJECTION
+ help
+ This option provides fault-injection capability for alloc_pages().
+
+config FAIL_MAKE_REQUEST
+ bool "fault-injection capability for disk IO"
+ depends on DEBUG_KERNEL
+ select FAULT_INJECTION
+ help
+ This option provides fault-injection capability to disk IO.
+
+config FAULT_INJECTION_DEBUGFS
+ tristate "runtime configuration for fault-injection capabilities"
+ depends on DEBUG_KERNEL && SYSFS && FAULT_INJECTION
+ select DEBUG_FS
+ help
+ This option provides kernel module that provides runtime
+ configuration interface by debugfs.
+
config DEBUG_FS
bool "Debug Filesystem"
depends on SYSFS
@@ -368,38 +403,3 @@ config RCU_TORTURE_TEST
at boot time (you probably don't).
Say M if you want the RCU torture tests to build as a module.
Say N if you are unsure.
-
-config FAULT_INJECTION
- bool
- select STACKTRACE
- select FRAME_POINTER
-
-config FAILSLAB
- bool "fault-injection capabilitiy for kmalloc"
- depends on DEBUG_KERNEL
- select FAULT_INJECTION
- help
- This option provides fault-injection capabilitiy for kmalloc.
-
-config FAIL_PAGE_ALLOC
- bool "fault-injection capabilitiy for alloc_pages()"
- depends on DEBUG_KERNEL
- select FAULT_INJECTION
- help
- This option provides fault-injection capabilitiy for alloc_pages().
-
-config FAIL_MAKE_REQUEST
- bool "fault-injection capabilitiy for disk IO"
- depends on DEBUG_KERNEL
- select FAULT_INJECTION
- help
- This option provides fault-injection capabilitiy to disk IO.
-
-config FAULT_INJECTION_DEBUGFS
- tristate "runtime configuration for fault-injection capabilities"
- depends on DEBUG_KERNEL && SYSFS && FAULT_INJECTION
- select DEBUG_FS
- help
- This option provides kernel module that provides runtime
- configuration interface by debugfs.
-
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [patch 8/8] stacktrace filtering for fault-injection capabilities
2006-09-19 5:57 ` Don Mullis
@ 2006-09-19 9:09 ` Akinobu Mita
2006-09-19 17:35 ` Don Mullis
0 siblings, 1 reply; 25+ messages in thread
From: Akinobu Mita @ 2006-09-19 9:09 UTC (permalink / raw)
To: Don Mullis; +Cc: linux-kernel, ak, akpm, Valdis.Kletnieks
On Mon, Sep 18, 2006 at 10:57:51PM -0700, Don Mullis wrote:
> Push fault-injection entries earlier in the list, so that they appear
> nested under DEBUG_KERNEL in menuconfig/xconfig.
Disabling the option Kernel debugging can hide all config options
realated to fault-injection without this patch.
Do I misunderstand something?
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [patch 8/8] stacktrace filtering for fault-injection capabilities
2006-09-19 9:09 ` Akinobu Mita
@ 2006-09-19 17:35 ` Don Mullis
2006-09-20 13:39 ` Akinobu Mita
0 siblings, 1 reply; 25+ messages in thread
From: Don Mullis @ 2006-09-19 17:35 UTC (permalink / raw)
To: Akinobu Mita; +Cc: linux-kernel, ak, akpm, Valdis.Kletnieks
On Tue, 2006-09-19 at 17:09 +0800, Akinobu Mita wrote:
> On Mon, Sep 18, 2006 at 10:57:51PM -0700, Don Mullis wrote:
> > Push fault-injection entries earlier in the list, so that they appear
> > nested under DEBUG_KERNEL in menuconfig/xconfig.
>
> Disabling the option Kernel debugging can hide all config options
> realated to fault-injection without this patch.
>
> Do I misunderstand something?
>
There's no problem with hiding per se, but rather with the indentation
level. It's most natural for the user to have dependent options
indented under their "parent". For an example, in "menuconfig" try
setting "Compile the kernel with frame unwind information";
notice that "Stack unwind support" appears immediately underneath
it, indented. The indentation reminds the user why "Stack unwind
support" has appeared.
Note that several of the pre-existing, non-fault-injection options
under "Kernel debugging", are also broken in this way.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [patch 8/8] stacktrace filtering for fault-injection capabilities
2006-09-19 17:35 ` Don Mullis
@ 2006-09-20 13:39 ` Akinobu Mita
2006-09-20 20:02 ` Don Mullis
0 siblings, 1 reply; 25+ messages in thread
From: Akinobu Mita @ 2006-09-20 13:39 UTC (permalink / raw)
To: Don Mullis; +Cc: linux-kernel, ak, akpm, Valdis.Kletnieks
On Tue, Sep 19, 2006 at 10:35:27AM -0700, Don Mullis wrote:
> There's no problem with hiding per se, but rather with the indentation
> level. It's most natural for the user to have dependent options
> indented under their "parent". For an example, in "menuconfig" try
> setting "Compile the kernel with frame unwind information";
> notice that "Stack unwind support" appears immediately underneath
> it, indented. The indentation reminds the user why "Stack unwind
> support" has appeared.
>
I see.
> Note that several of the pre-existing, non-fault-injection options
> under "Kernel debugging", are also broken in this way.
Perhaps you can move UNWIND_INFO, STACK_UNWIND, and DEBUG_FS
entries ealier in the list. It improves improve appearance for
other DEBUG_KERNEL dependent config options like (DEBUG_VM,
FRAME_POINTER, ...).
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [patch 8/8] stacktrace filtering for fault-injection capabilities
2006-09-20 13:39 ` Akinobu Mita
@ 2006-09-20 20:02 ` Don Mullis
0 siblings, 0 replies; 25+ messages in thread
From: Don Mullis @ 2006-09-20 20:02 UTC (permalink / raw)
To: Akinobu Mita; +Cc: linux-kernel, ak, akpm, Valdis.Kletnieks
On Wed, 2006-09-20 at 21:39 +0800, Akinobu Mita wrote:
> Perhaps you can move UNWIND_INFO, STACK_UNWIND, and DEBUG_FS
> entries ealier in the list. It improves improve appearance for
> other DEBUG_KERNEL dependent config options like (DEBUG_VM,
> FRAME_POINTER, ...).
Okay, I will prepare such a janitorial patch.
^ permalink raw reply [flat|nested] 25+ messages in thread