[Qemu-devel] [PATCH] docs: add blkdebug block driver documentation

[Qemu-devel] [PATCH] docs: add blkdebug block driver documentation

[Stefan Hajnoczi]

The blkdebug block driver is undocumented.  Documenting it is worthwhile
since it offers powerful error injection features that are used by
qemu-iotests test cases.

This document will make it easier for people to learn about and use
blkdebug.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 docs/blkdebug.txt | 142 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 142 insertions(+)
 create mode 100644 docs/blkdebug.txt

diff --git a/docs/blkdebug.txt b/docs/blkdebug.txt
new file mode 100644
index 0000000..7e616e0
--- /dev/null
+++ b/docs/blkdebug.txt
@@ -0,0 +1,142 @@
+Block I/O error injection using blkdebug
+----------------------------------------
+The blkdebug block driver is a rule-based error injection engine.  It can be
+used to exercise error code paths in block drivers including ENOSPC (out of
+space) and EIO.
+
+This document gives an overview of the features available in blkdebug.
+
+Background
+----------
+Block drivers have many error code paths that handle I/O errors.  Image formats
+are especially complex since metadata I/O errors during cluster allocation or
+while updating tables happen halfway through request processing and require
+discipline to keep image files consistent.
+
+Error injection allows test cases to trigger I/O errors at specific points.
+This way, all error paths can be tested to make sure they are correct.
+
+Rules
+-----
+The blkdebug block driver takes a list of "rules" that tell the error injection
+engine when to fail an I/O request.
+
+Each I/O request is evaluated against the rules.  If a rule matches the request
+then its "action" is executed.
+
+Rules can be placed in a .ini file:
+
+  $ cat blkdebug.conf
+  [inject-error]
+  event = "read_aio"
+  errno = "28"
+
+This rule fails all aio read requests with ENOSPC (28).
+
+Invoke QEMU as follows:
+
+  $ qemu-system-x86_64
+        -drive if=none,cache=none,file=blkdebug:blkdebug.conf:test.img,id=drive0 \
+        -device virtio-blk-pci,drive=drive0,id=virtio-blk-pci0
+
+Rules support the following attributes:
+
+  event - which type of operation to match (e.g. read_aio, write_aio,
+	  flush_to_os, flush_to_disk).  See the "Events" section for
+          information on events.
+
+  state - (optional) the engine must be in this state number in order for this
+	  rule to match.  See the "State transitions" section for information
+          on states.
+
+  errno - the POSIX errno value to return when a request matches this rule
+
+  sector - (optional) a sector number that the request must overlap in order to
+           match this rule
+
+  once - (optional, default "off") only execute this action on the first
+         matching request
+
+  immediately - (optional, default "off") return a NULL BlockDriverAIOCB
+		pointer and fail without an errno instead.  This exercises the
+		code path where BlockDriverAIOCB fails and the caller's
+                BlockDriverCompletionFunc is not invoked.
+
+Events
+------
+Block drivers provide information about the type of I/O request they are about
+to make so rules can match specific types of requests.  For example, the qcow2
+block driver tells blkdebug when it accesses the L1 table so rules can match
+only L1 table accesses and not other metadata or guest data requests.
+
+The core events are:
+
+  read_aio - guest data read
+
+  write_aio - guest data write
+
+  flush_to_os - write out unwritten block driver state (e.g. cached metadata)
+
+  flush_to_disk - flush the host block device's disk cache
+
+See block/blkdebug.c:event_names[] for the list of available events.  You may
+need to grep block driver source code to understand the meaning of specific
+events.
+
+State transitions
+-----------------
+There are cases where more power is needed to match a particular I/O request in
+a longer sequence of requests.  For example:
+
+  write_aio
+  flush_to_disk
+  write_aio
+
+How do we match the 2nd write_aio but not the first?  This is where state
+transitions come in.
+
+The error injection engine has an integer called the "state" that always starts
+initialized to 1.  Rules can be conditional on the state and they can
+transition to a new state.
+
+For example, to match the 2nd write_aio:
+
+  [set-state]
+  event = "write_aio"
+  state = "1"
+  new_state = "2"
+
+  [inject-error]
+  event = "write_aio"
+  state = "2"
+  errno = "5"
+
+The first write_aio request matches the set-state rule and transitions from
+state 0 to state 1.  Once state 1 has been entered, the set-state rule no
+longer matches since it required state 0.  But the inject-error rule now
+matches the next write_aio request and injects EIO (5).
+
+State transition rules support the following attributes:
+
+  event - which type of operation to match (e.g. read_aio, write_aio,
+	  flush_to_os, flush_to_disk).  See the "Events" section for
+          information on events.
+
+  state - (optional) the engine must be in this state number in order for this
+	  rule to match
+
+  new_state - transition to this state number
+
+Suspend and resume
+------------------
+Exercising code paths in block drivers may require specific ordering amongst
+concurrent requests.  The "breakpoint" feature allows requests to be halted on
+a blkdebug event and resumed later.  This makes it possible to achieve
+deterministic ordering when multiple requests are in flight.
+
+Breakpoints on blkdebug events are associated with a user-defined "tag" string.
+This tag serves as an identifier by which the request can be resumed at a later
+point.
+
+See the qemu-io(1) break, resume, remove_break, and wait_break commands for
+details.
-- 
1.9.3

Re: [Qemu-devel] [PATCH] docs: add blkdebug block driver documentation

[Paolo Bonzini]

Il 22/09/2014 11:20, Stefan Hajnoczi ha scritto:
> The blkdebug block driver is undocumented.  Documenting it is worthwhile
> since it offers powerful error injection features that are used by
> qemu-iotests test cases.
> 
> This document will make it easier for people to learn about and use
> blkdebug.

Just one small comment below...

> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  docs/blkdebug.txt | 142 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 142 insertions(+)
>  create mode 100644 docs/blkdebug.txt
> 
> diff --git a/docs/blkdebug.txt b/docs/blkdebug.txt
> new file mode 100644
> index 0000000..7e616e0
> --- /dev/null
> +++ b/docs/blkdebug.txt
> @@ -0,0 +1,142 @@
> +Block I/O error injection using blkdebug
> +----------------------------------------
> +The blkdebug block driver is a rule-based error injection engine.  It can be
> +used to exercise error code paths in block drivers including ENOSPC (out of
> +space) and EIO.
> +
> +This document gives an overview of the features available in blkdebug.
> +
> +Background
> +----------
> +Block drivers have many error code paths that handle I/O errors.  Image formats
> +are especially complex since metadata I/O errors during cluster allocation or
> +while updating tables happen halfway through request processing and require
> +discipline to keep image files consistent.
> +
> +Error injection allows test cases to trigger I/O errors at specific points.
> +This way, all error paths can be tested to make sure they are correct.
> +
> +Rules
> +-----
> +The blkdebug block driver takes a list of "rules" that tell the error injection
> +engine when to fail an I/O request.
> +
> +Each I/O request is evaluated against the rules.  If a rule matches the request
> +then its "action" is executed.
> +
> +Rules can be placed in a .ini file:

Rules can be placed in a configuration file; the configuration file
follows the same .ini-like format used by QEMU's -readconfig option, and
each section of the file represents a rule.

The following configuration file defines a single rule:

> 
> +
> +  $ cat blkdebug.conf
> +  [inject-error]
> +  event = "read_aio"
> +  errno = "28"
> +


Paolo

Re: [Qemu-devel] [PATCH] docs: add blkdebug block driver documentation

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1917 bytes --]

On 09/22/2014 03:20 AM, Stefan Hajnoczi wrote:
> The blkdebug block driver is undocumented.  Documenting it is worthwhile
> since it offers powerful error injection features that are used by
> qemu-iotests test cases.
> 
> This document will make it easier for people to learn about and use
> blkdebug.
> 
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  docs/blkdebug.txt | 142 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 142 insertions(+)
>  create mode 100644 docs/blkdebug.txt
> 
> diff --git a/docs/blkdebug.txt b/docs/blkdebug.txt
> new file mode 100644
> index 0000000..7e616e0
> --- /dev/null
> +++ b/docs/blkdebug.txt
> @@ -0,0 +1,142 @@

No worse than many other files in this directory for omitting copyright
and license information (which means it inherits the default of GPLv2+
from the top-level).

> +
> +Rules can be placed in a .ini file:

Choice of article depends on whether you pronounce it "a dot-innie" or
"an eye-en-eye" file.

> +
> +  $ cat blkdebug.conf

This file suffix is not '.ini'.  So the idea is more that it is a
configuration file that uses '.ini' style parsing, and not that it is an
actual '.ini' file.

> +  [inject-error]
> +  event = "read_aio"
> +  errno = "28"
> +
> +This rule fails all aio read requests with ENOSPC (28).

Eeesh - we are really tying things to host-dependent errno values?
Might be worth a paragraph explaining that ENOSPC is not always 28.  Or
a design improvement to the setup to allow symbolic naming of the
desired actions. Or both.

> +
> +  errno - the POSIX errno value to return when a request matches this rule
> +

Again, POSIX doesn't proscribe errno values, just symbolic names; and
those names differ in values across different OS.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 539 bytes --]