[RFC 0/1] nvme: Add NVMe LBA Fault Injection

[RFC 0/1] nvme: Add NVMe LBA Fault Injection

[Alan Adamson]

I originally submitted this just before the Holidays so many probably missed it, so
I'm reposting.

It has been requested that the NVMe fault injector be able to inject faults when accessing
specific Logical Block Addresses (LBA).  This patch is built onto the existing fault
injector to provide that ability to specify a LBA when setting up an injection fault.

If the fault injector is enabled, and the nvme driver is ready to send an IO or Admin command
to NVMe hardware, the function should_fail() is called to determine if a fault should
be injected instead of it being allowed to normally be issued.  The should_fail() function
uses the fault injector attributes: probability, times, space, and interval to determine
if a fault should be injected.  If it is determined a fault is to be injected, the request
is immediately completed using the attribute: status.  If no fault is to be injected,
the request is allowed to be issued to the hardware and completed normally.

If a LBA fault is setup, should_fail() is only called if the LBA falls within the range of
blocks of a NVMe READ or WRITE command. If it does not fall with the range, should_fail()
is not called and no fault would be injected.  If the LBA does fall within the range,
should_fail() is called to determine if a fault should be injected.

The LBA fault injector could be extended to also check command types (READ/WRITE).

Steps to setup a fault on a LBA access.

1. Setup the attributes: probability, times, and status according to
the original NVMe fault injector.

echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability
echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times
echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/status

2. Specify one or more LBAs: To configure a LBA, set the lba-inject-set
attribute to the LBA.

echo 0x22 > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-set

3. Enable the LBA injector: by setting the lba-inject-enable attribute
to on.

echo on > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-enable

To display the LBAs configured:

cat /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-show
LBA  Status
22   1

To stop injecting a fault on a LBA:

echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-clear
echo 0x22 > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-set

Alan Adamson (1):
  nvme: Add NVMe LBA Fault Injection

 .../fault-injection/nvme-fault-injection.rst  |  28 ++++
 drivers/nvme/host/core.c                      |   4 +-
 drivers/nvme/host/fault_inject.c              | 145 ++++++++++++++++--
 drivers/nvme/host/nvme.h                      |  16 +-
 4 files changed, 178 insertions(+), 15 deletions(-)

-- 
2.39.3

[RFC 1/1] nvme: Add NVMe LBA Fault Injection

[Alan Adamson]

Add support for injecting a fault when reading or writing a specific
Logical Block Address (LBA).

Steps to setup a fault on a LBA access.

1. Setup the attributes: probability, times, and status according to
the original NVMe fault injector.

echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability
echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times
echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/status

2. Specify one or more LBAs: To configure a LBA, set the lba-inject-set
attribute to the LBA.

echo 0x22 > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-set

3. Enable the LBA injector: by setting the lba-inject-enable attribute
to on.

echo on > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-enable

To display the LBAs configured:

cat /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-show
LBA  Status
22   1

To stop injecting a fault on a LBA:

echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-clear
echo 0x22 > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-set

Signed-off-by: Alan Adamson <alan.adamson@oracle.com>
---
 .../fault-injection/nvme-fault-injection.rst  |  28 ++++
 drivers/nvme/host/core.c                      |   4 +-
 drivers/nvme/host/fault_inject.c              | 145 ++++++++++++++++--
 drivers/nvme/host/nvme.h                      |  16 +-
 4 files changed, 178 insertions(+), 15 deletions(-)

diff --git a/Documentation/fault-injection/nvme-fault-injection.rst b/Documentation/fault-injection/nvme-fault-injection.rst
index 1d4427890d75..e17f49886fc4 100644
--- a/Documentation/fault-injection/nvme-fault-injection.rst
+++ b/Documentation/fault-injection/nvme-fault-injection.rst
@@ -176,3 +176,31 @@ Message from dmesg::
    secondary_startup_64+0xa4/0xb0
   nvme nvme0: Could not set queue count (16385)
   nvme nvme0: IO queues not created
+
+Example 4: Inject an error when reading or writing Logical Block Addrss (LBA) 0x22
+----------------------------------------------------------------------------------
+
+::
+
+  echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times
+  echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability
+  echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/status
+  echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/dont_retry
+  echo 0x22 > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-set
+  cat /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-show
+  LBA	Status
+  21	1
+
+  echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-enable
+  dd if=/dev/nvme0n1 of=/dev/null iflag=direct count=100
+  dd: error reading '/dev/nvme0n1': Operation not supported
+  33+0 records in
+  33+0 records out
+  16896 bytes (17 kB, 16 KiB) copied, 0.00711112 s, 2.4 MB/s
+
+  echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-clear
+  echo 0x22 > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-set
+  cat /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-show
+  LBA   Status
+
+  echo 0 > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-enable
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 8ebdfd623e0f..435ef8b475c0 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -3681,7 +3681,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 		nvme_add_ns_cdev(ns);
 
 	nvme_mpath_add_disk(ns, info->anagrpid);
-	nvme_fault_inject_init(&ns->fault_inject, ns->disk->disk_name);
+	nvme_fault_inject_init(&ns->fault_inject, ns->disk->disk_name, 1);
 
 	return;
 
@@ -4548,7 +4548,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 	dev_pm_qos_update_user_latency_tolerance(ctrl->device,
 		min(default_ps_max_latency_us, (unsigned long)S32_MAX));
 
-	nvme_fault_inject_init(&ctrl->fault_inject, dev_name(ctrl->device));
+	nvme_fault_inject_init(&ctrl->fault_inject, dev_name(ctrl->device), 0);
 	nvme_mpath_init_ctrl(ctrl);
 	ret = nvme_auth_init_ctrl(ctrl);
 	if (ret)
diff --git a/drivers/nvme/host/fault_inject.c b/drivers/nvme/host/fault_inject.c
index 1ba10a5c656d..81107ab66e21 100644
--- a/drivers/nvme/host/fault_inject.c
+++ b/drivers/nvme/host/fault_inject.c
@@ -5,6 +5,8 @@
  * Copyright (c) 2018, Oracle and/or its affiliates
  */
 
+#include <linux/kernel.h>
+#include <linux/module.h>
 #include <linux/moduleparam.h>
 #include "nvme.h"
 
@@ -15,8 +17,88 @@ static DECLARE_FAULT_ATTR(fail_default_attr);
 static char *fail_request;
 module_param(fail_request, charp, 0000);
 
+static void nvme_free_lba_block(struct lba_blk *lba_block)
+{
+	list_del(&lba_block->node);
+	kfree(lba_block);
+}
+
+static void nvme_free_all_lba_blocks(struct nvme_fault_inject *fault_inj)
+{
+	struct lba_blk *lba_block;
+	struct list_head *list_ptr, *tmp;
+
+	list_for_each_prev_safe(list_ptr, tmp, &fault_inj->lba_list) {
+		lba_block = list_entry(list_ptr, struct lba_blk, node);
+		list_del(list_ptr);
+		kfree(lba_block);
+	}
+}
+
+static int nvme_lba_inject_set(void *data, u64 val)
+{
+	struct nvme_fault_inject *fault_inj = data;
+	struct lba_blk *lba_block;
+	int lba;
+	int ret = 0;
+
+	lba =  val;
+	mutex_lock(&fault_inj->lba_block_lock);
+	if (fault_inj->lba_clear) {
+		list_for_each_entry(lba_block,
+		    &fault_inj->lba_list, node) {
+			if (lba_block->lba == lba) {
+				nvme_free_lba_block(lba_block);
+				fault_inj->lba_clear = 0;
+				mutex_unlock(&fault_inj->lba_block_lock);
+				return ret;
+			}
+		}
+		fault_inj->lba_clear = 0;
+		mutex_unlock(&fault_inj->lba_block_lock);
+		return ret;
+	}
+
+	list_for_each_entry(lba_block, &fault_inj->lba_list, node) {
+		if (lba_block->lba == lba) {
+			lba_block->status = fault_inj->status;
+			mutex_unlock(&fault_inj->lba_block_lock);
+			return ret;
+		}
+	}
+
+	lba_block = kmalloc(sizeof(struct lba_blk), GFP_KERNEL);
+	lba_block->lba = lba;
+	lba_block->status = fault_inj->status;
+
+	list_add(&lba_block->node, &fault_inj->lba_list);
+
+	mutex_unlock(&fault_inj->lba_block_lock);
+	return ret;
+}
+
+static int nvme_lba_inject_show(struct seq_file *file, void *data)
+{
+	struct nvme_fault_inject *fault_inj = file->private;
+	struct lba_blk *lba_block;
+
+	seq_printf(file, "LBA	Status\n");
+	mutex_lock(&fault_inj->lba_block_lock);
+	list_for_each_entry(lba_block, &fault_inj->lba_list, node) {
+		seq_printf(file, "%llx	%x\n",
+		    lba_block->lba, lba_block->status);
+	}
+	mutex_unlock(&fault_inj->lba_block_lock);
+
+	return 0;
+}
+
+DEFINE_SHOW_ATTRIBUTE(nvme_lba_inject);
+DEFINE_DEBUGFS_ATTRIBUTE(nvme_lba_inject_set_fops, NULL,
+			 nvme_lba_inject_set, "%llu\n");
+
 void nvme_fault_inject_init(struct nvme_fault_inject *fault_inj,
-			    const char *dev_name)
+			    const char *dev_name, bool lba_inject)
 {
 	struct dentry *dir, *parent;
 	struct fault_attr *attr = &fault_inj->attr;
@@ -41,15 +123,34 @@ void nvme_fault_inject_init(struct nvme_fault_inject *fault_inj,
 	}
 	fault_inj->parent = parent;
 
+	INIT_LIST_HEAD(&fault_inj->lba_list);
+
 	/* create debugfs for status code and dont_retry */
 	fault_inj->status = NVME_SC_INVALID_OPCODE;
 	fault_inj->dont_retry = true;
+	fault_inj->lba_clear = 0;
+	fault_inj->lba_inject_enable = false;
+	mutex_init(&fault_inj->lba_block_lock);
+
 	debugfs_create_x16("status", 0600, dir,	&fault_inj->status);
 	debugfs_create_bool("dont_retry", 0600, dir, &fault_inj->dont_retry);
+	if (lba_inject) {
+		debugfs_create_bool("lba-inject-clear", 0600, dir, &fault_inj->lba_clear);
+		debugfs_create_bool("lba-inject-enable", 0600, dir,
+		    &fault_inj->lba_inject_enable);
+		debugfs_create_file("lba-inject-show", 0600, dir,
+		    fault_inj, &nvme_lba_inject_fops);
+		debugfs_create_file_unsafe("lba-inject-set", 0600, dir,
+		    fault_inj, &nvme_lba_inject_set_fops);
+	}
 }
 
 void nvme_fault_inject_fini(struct nvme_fault_inject *fault_inject)
 {
+        mutex_lock(&fault_inject->lba_block_lock);
+	nvme_free_all_lba_blocks(fault_inject);
+        mutex_unlock(&fault_inject->lba_block_lock);
+
 	/* remove debugfs directories */
 	debugfs_remove_recursive(fault_inject->parent);
 }
@@ -58,25 +159,47 @@ void nvme_should_fail(struct request *req)
 {
 	struct gendisk *disk = req->q->disk;
 	struct nvme_fault_inject *fault_inject = NULL;
+	struct nvme_ns *ns = NULL;
+	struct lba_blk *lba_block = NULL;
 	u16 status;
 
 	if (disk) {
-		struct nvme_ns *ns = disk->private_data;
-
+		ns = disk->private_data;
 		if (ns)
 			fault_inject = &ns->fault_inject;
 		else
 			WARN_ONCE(1, "No namespace found for request\n");
-	} else {
+	} else
 		fault_inject = &nvme_req(req)->ctrl->fault_inject;
-	}
 
-	if (fault_inject && should_fail(&fault_inject->attr, 1)) {
-		/* inject status code and DNR bit */
-		status = fault_inject->status;
-		if (fault_inject->dont_retry)
-			status |= NVME_SC_DNR;
-		nvme_req(req)->status =	status;
+	if (!fault_inject || !should_fail(&fault_inject->attr, 1))
+		return;
+
+	mutex_lock(&fault_inject->lba_block_lock);
+
+	if (fault_inject->lba_inject_enable) {
+		u64 req_lba_start, req_lba_end;
+
+		req_lba_start = blk_rq_pos(req);
+		req_lba_end = req_lba_start + blk_rq_sectors(req) - 1;
+
+		list_for_each_entry(lba_block, &fault_inject->lba_list, node) {
+			if ((lba_block->lba >= req_lba_start) &&
+			    (lba_block->lba <= req_lba_end))
+				goto inject_fault;
+		}
+
+		mutex_unlock(&fault_inject->lba_block_lock);
+		return;
 	}
+
+inject_fault:
+	mutex_unlock(&fault_inject->lba_block_lock);
+
+	/* inject status code and DNR bit */
+	status = fault_inject->status;
+	if (fault_inject->dont_retry)
+		status |= NVME_SC_DNR;
+	nvme_req(req)->status =	status;
 }
 EXPORT_SYMBOL_GPL(nvme_should_fail);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index e7411dac00f7..447ce832613c 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -16,6 +16,7 @@
 #include <linux/rcupdate.h>
 #include <linux/wait.h>
 #include <linux/t10-pi.h>
+#include <linux/list.h>
 
 #include <trace/events/block.h>
 
@@ -245,7 +246,11 @@ struct nvme_fault_inject {
 	struct fault_attr attr;
 	struct dentry *parent;
 	bool dont_retry;	/* DNR, do not retry */
+	bool lba_clear;
+	bool lba_inject_enable;
 	u16 status;		/* status code */
+	struct mutex		lba_block_lock;
+	struct list_head	lba_list;
 #endif
 };
 
@@ -484,6 +489,13 @@ enum nvme_ns_features {
 	NVME_NS_DEAC,		/* DEAC bit in Write Zeores supported */
 };
 
+typedef struct lba_blk {
+	u64			lba;
+	u16			status;
+	u32			times;
+	struct list_head	node;
+} lba_blk_t;
+
 struct nvme_ns {
 	struct list_head list;
 
@@ -621,12 +633,12 @@ static inline void nvme_print_device_info(struct nvme_ctrl *ctrl)
 
 #ifdef CONFIG_FAULT_INJECTION_DEBUG_FS
 void nvme_fault_inject_init(struct nvme_fault_inject *fault_inj,
-			    const char *dev_name);
+			    const char *dev_name, bool lba_inject);
 void nvme_fault_inject_fini(struct nvme_fault_inject *fault_inject);
 void nvme_should_fail(struct request *req);
 #else
 static inline void nvme_fault_inject_init(struct nvme_fault_inject *fault_inj,
-					  const char *dev_name)
+					  const char *dev_name, bool lba_inject)
 {
 }
 static inline void nvme_fault_inject_fini(struct nvme_fault_inject *fault_inj)
-- 
2.39.3

Re: [RFC 0/1] nvme: Add NVMe LBA Fault Injection

[Christoph Hellwig]

On Tue, Jan 16, 2024 at 03:27:27PM -0800, Alan Adamson wrote:
> It has been requested that the NVMe fault injector be able to inject faults when accessing
> specific Logical Block Addresses (LBA).

Curious, but who has requested this?  Because injecting errors really
isn't the drivers job.

Re: [RFC 0/1] nvme: Add NVMe LBA Fault Injection

[alan.adamson]

On 1/17/24 11:24 PM, Christoph Hellwig wrote:
> On Tue, Jan 16, 2024 at 03:27:27PM -0800, Alan Adamson wrote:
>> It has been requested that the NVMe fault injector be able to inject faults when accessing
>> specific Logical Block Addresses (LBA).
> Curious, but who has requested this?  Because injecting errors really
> isn't the drivers job.


It's an application (database) that is requesting it for their error 
handling testing.

Alan

Re: [RFC 0/1] nvme: Add NVMe LBA Fault Injection

[Keith Busch]

On Thu, Jan 18, 2024 at 09:02:43AM -0800, alan.adamson@oracle.com wrote:
> 
> On 1/17/24 11:24 PM, Christoph Hellwig wrote:
> > On Tue, Jan 16, 2024 at 03:27:27PM -0800, Alan Adamson wrote:
> > > It has been requested that the NVMe fault injector be able to inject faults when accessing
> > > specific Logical Block Addresses (LBA).
> > Curious, but who has requested this?  Because injecting errors really
> > isn't the drivers job.
> 
> 
> It's an application (database) that is requesting it for their error
> handling testing.

Going out on a limb here... This seems so obscure to burden a driver to
synthesize. Could we just give a hook for a bpf override and you can
totally go for whatever scenario you can imagine with a script?

---
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 50818dbcfa1ae..df87a63335aa8 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -381,9 +381,22 @@ static inline void nvme_end_req_zoned(struct request *req)
 	}
 }
 
+#ifdef CONFIG_BPF_KPROBE_OVERRIDE
+static noinline blk_status_t nvme_req_status(struct nvme_request *req)
+{
+	return nvme_error_status(req->status);
+}
+ALLOW_ERROR_INJECTION(nvme_req_status, ERRNO);
+#else
+static inline blk_status_t nvme_req_status(struct nvme_request *req)
+{
+	return nvme_error_status(req->status);
+}
+#endif
+
 static inline void nvme_end_req(struct request *req)
 {
-	blk_status_t status = nvme_error_status(nvme_req(req)->status);
+	blk_status_t status = nvme_req_status(nvme_req(req));
 
 	if (unlikely(nvme_req(req)->status && !(req->rq_flags & RQF_QUIET)))
 		nvme_log_error(req);
--

Re: [RFC 0/1] nvme: Add NVMe LBA Fault Injection

[Christoph Hellwig]

On Thu, Jan 18, 2024 at 09:02:43AM -0800, alan.adamson@oracle.com wrote:
>
> On 1/17/24 11:24 PM, Christoph Hellwig wrote:
>> On Tue, Jan 16, 2024 at 03:27:27PM -0800, Alan Adamson wrote:
>>> It has been requested that the NVMe fault injector be able to inject faults when accessing
>>> specific Logical Block Addresses (LBA).
>> Curious, but who has requested this?  Because injecting errors really
>> isn't the drivers job.
>
>
> It's an application (database) that is requesting it for their error 
> handling testing.

Well, how about they then insert it into the real or virtual hardware.
The Linux nvme driver isn't really an error injection framework.

Re: [RFC 0/1] nvme: Add NVMe LBA Fault Injection

[alan.adamson]

On 1/23/24 1:05 AM, Christoph Hellwig wrote:
> On Thu, Jan 18, 2024 at 09:02:43AM -0800, alan.adamson@oracle.com wrote:
>> On 1/17/24 11:24 PM, Christoph Hellwig wrote:
>>> On Tue, Jan 16, 2024 at 03:27:27PM -0800, Alan Adamson wrote:
>>>> It has been requested that the NVMe fault injector be able to inject faults when accessing
>>>> specific Logical Block Addresses (LBA).
>>> Curious, but who has requested this?  Because injecting errors really
>>> isn't the drivers job.
>>
>> It's an application (database) that is requesting it for their error
>> handling testing.
> Well, how about they then insert it into the real or virtual hardware.
> The Linux nvme driver isn't really an error injection framework.
>
Sorry if you receive multiple of these, I'm having email/sending issues.


I get it, but there is already an injection framework in place for nvme. 
Is there no plan to improve it?

Alan

Re: [RFC 0/1] nvme: Add NVMe LBA Fault Injection

[Christoph Hellwig]

On Tue, Jan 23, 2024 at 09:25:51AM -0800, alan.adamson@oracle.com wrote:
> I get it, but there is already an injection framework in place for nvme. Is 
> there no plan to improve it?

Injecting fake I/O error really isn't the driver job.  For block
I/O we could do it the block layer or DM, but that's not something
to add to random drivers.  Error injection makes sense for testing
the driver itself, not applications.

Re: [RFC 0/1] nvme: Add NVMe LBA Fault Injection

[alan.adamson]

On 1/24/24 1:04 AM, Christoph Hellwig wrote:
> On Tue, Jan 23, 2024 at 09:25:51AM -0800, alan.adamson@oracle.com wrote:
>> I get it, but there is already an injection framework in place for nvme. Is
>> there no plan to improve it?
> Injecting fake I/O error really isn't the driver job.  For block
> I/O we could do it the block layer or DM, but that's not something
> to add to random drivers.  Error injection makes sense for testing
> the driver itself, not applications.

Thanks, I'll look at the block layer for this.  Do you think adding 
error injection to qemu-nvme makes sense?

Alan

Re: [RFC 0/1] nvme: Add NVMe LBA Fault Injection

[Keith Busch]

On Wed, Jan 24, 2024 at 08:52:54AM -0800, alan.adamson@oracle.com wrote:
> 
> Thanks, I'll look at the block layer for this.  Do you think adding error
> injection to qemu-nvme makes sense?

I frequently added custom error injection to qemu, though it was always
pretty hacky so I never upstreamed anything. Klaus may be interested in
getting a feature like that integrated upstream if you're able to put
something together.

Re: [RFC 0/1] nvme: Add NVMe LBA Fault Injection

[alan.adamson]

On 1/24/24 8:59 AM, Keith Busch wrote:
> On Wed, Jan 24, 2024 at 08:52:54AM -0800, alan.adamson@oracle.com wrote:
>> Thanks, I'll look at the block layer for this.  Do you think adding error
>> injection to qemu-nvme makes sense?
> I frequently added custom error injection to qemu, though it was always
> pretty hacky so I never upstreamed anything. Klaus may be interested in
> getting a feature like that integrated upstream if you're able to put
> something together.

Thanks, I'll put something together.

Alan

[RFC 0/1] nvme: Add NVMe LBA Fault Injection

[Alan Adamson]

It has been requested that the NVMe fault injector be able to inject faults when accessing
specific Logical Block Addresses (LBA).  This patch is built onto the existing fault
injector to provide that ability to specify a LBA when setting up an injection fault.

If the fault injector is enabled, and the nvme driver is ready to send an IO or Admin command
to NVMe hardware, the function should_fail() is called to determine if a fault should
be injected instead of it being allowed to normally be issued.  The should_fail() function
uses the fault injector attributes: probability, times, space, and interval to determine
if a fault should be injected.  If it is determined a fault is to be injected, the request
is immediately completed using the attribute: status.  If no fault is to be injected,
the request is allowed to be issued to the hardware and completed normally.

If a LBA fault is setup, should_fail() is only called if the LBA falls within the range of
blocks of a NVMe READ or WRITE command. If it does not fall with the range, should_fail()
is not called and no fault would be injected.  If the LBA does fall within the range,
should_fail() is called to determine if a fault should be injected.

The LBA fault injector could be extended to also check command types (READ/WRITE).

Steps to setup a fault on a LBA access.

1. Setup the attributes: probability, times, and status according to
the original NVMe fault injector.

echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability
echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times
echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/status

2. Specify one or more LBAs: To configure a LBA, set the lba-inject-set
attribute to the LBA.

echo 0x22 > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-set

3. Enable the LBA injector: by setting the lba-inject-enable attribute
to on.

echo on > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-enable

To display the LBAs configured:

cat /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-show
LBA  Status
22   1

To stop injecting a fault on a LBA:

echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-clear
echo 0x22 > /sys/kernel/debug/nvme0n1/fault_inject/lba-inject-set

Alan Adamson (1):
  nvme: Add NVMe LBA Fault Injection

 .../fault-injection/nvme-fault-injection.rst  |  28 ++++
 drivers/nvme/host/core.c                      |   4 +-
 drivers/nvme/host/fault_inject.c              | 145 ++++++++++++++++--
 drivers/nvme/host/nvme.h                      |  16 +-
 4 files changed, 178 insertions(+), 15 deletions(-)

-- 
2.39.3