[Qemu-devel] [PATCH v2] docs: add blkdebug block driver documentation

[Qemu-devel] [PATCH v2] docs: add blkdebug block driver documentation

[Stefan Hajnoczi]

The blkdebug block driver is undocumented.  Documenting it is worthwhile
since it offers powerful error injection features that are used by
qemu-iotests test cases.

This document will make it easier for people to learn about and use
blkdebug.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
v2:
 * Added GPL v2 or later license and Red Hat copyright [Eric]
 * Expanded ini rules file explanation [Paolo]
 * Added note that errno values depend on the host [Eric]

 docs/blkdebug.txt | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 155 insertions(+)
 create mode 100644 docs/blkdebug.txt

diff --git a/docs/blkdebug.txt b/docs/blkdebug.txt
new file mode 100644
index 0000000..4034c82
--- /dev/null
+++ b/docs/blkdebug.txt
@@ -0,0 +1,155 @@
+Block I/O error injection using blkdebug
+----------------------------------------
+Copyright (C) 2014 Red Hat Inc
+
+This work is licensed under the terms of the GNU GPL, version 2 or later.  See
+the COPYING file in the top-level directory.
+
+The blkdebug block driver is a rule-based error injection engine.  It can be
+used to exercise error code paths in block drivers including ENOSPC (out of
+space) and EIO.
+
+This document gives an overview of the features available in blkdebug.
+
+Background
+----------
+Block drivers have many error code paths that handle I/O errors.  Image formats
+are especially complex since metadata I/O errors during cluster allocation or
+while updating tables happen halfway through request processing and require
+discipline to keep image files consistent.
+
+Error injection allows test cases to trigger I/O errors at specific points.
+This way, all error paths can be tested to make sure they are correct.
+
+Rules
+-----
+The blkdebug block driver takes a list of "rules" that tell the error injection
+engine when to fail an I/O request.
+
+Each I/O request is evaluated against the rules.  If a rule matches the request
+then its "action" is executed.
+
+Rules can be placed in a configuration file; the configuration file
+follows the same .ini-like format used by QEMU's -readconfig option, and
+each section of the file represents a rule.
+
+The following configuration file defines a single rule:
+
+  $ cat blkdebug.conf
+  [inject-error]
+  event = "read_aio"
+  errno = "28"
+
+This rule fails all aio read requests with ENOSPC (28).  Note that the errno
+value depends on the host.  On Linux, see
+/usr/include/asm-generic/errno-base.h for errno values.
+
+Invoke QEMU as follows:
+
+  $ qemu-system-x86_64
+        -drive if=none,cache=none,file=blkdebug:blkdebug.conf:test.img,id=drive0 \
+        -device virtio-blk-pci,drive=drive0,id=virtio-blk-pci0
+
+Rules support the following attributes:
+
+  event - which type of operation to match (e.g. read_aio, write_aio,
+	  flush_to_os, flush_to_disk).  See the "Events" section for
+          information on events.
+
+  state - (optional) the engine must be in this state number in order for this
+	  rule to match.  See the "State transitions" section for information
+          on states.
+
+  errno - the numeric errno value to return when a request matches this rule.
+          The errno values depend on the host since the numeric values are not
+          standarized in the POSIX specification.
+
+  sector - (optional) a sector number that the request must overlap in order to
+           match this rule
+
+  once - (optional, default "off") only execute this action on the first
+         matching request
+
+  immediately - (optional, default "off") return a NULL BlockDriverAIOCB
+		pointer and fail without an errno instead.  This exercises the
+		code path where BlockDriverAIOCB fails and the caller's
+                BlockDriverCompletionFunc is not invoked.
+
+Events
+------
+Block drivers provide information about the type of I/O request they are about
+to make so rules can match specific types of requests.  For example, the qcow2
+block driver tells blkdebug when it accesses the L1 table so rules can match
+only L1 table accesses and not other metadata or guest data requests.
+
+The core events are:
+
+  read_aio - guest data read
+
+  write_aio - guest data write
+
+  flush_to_os - write out unwritten block driver state (e.g. cached metadata)
+
+  flush_to_disk - flush the host block device's disk cache
+
+See block/blkdebug.c:event_names[] for the list of available events.  You may
+need to grep block driver source code to understand the meaning of specific
+events.
+
+State transitions
+-----------------
+There are cases where more power is needed to match a particular I/O request in
+a longer sequence of requests.  For example:
+
+  write_aio
+  flush_to_disk
+  write_aio
+
+How do we match the 2nd write_aio but not the first?  This is where state
+transitions come in.
+
+The error injection engine has an integer called the "state" that always starts
+initialized to 1.  Rules can be conditional on the state and they can
+transition to a new state.
+
+For example, to match the 2nd write_aio:
+
+  [set-state]
+  event = "write_aio"
+  state = "1"
+  new_state = "2"
+
+  [inject-error]
+  event = "write_aio"
+  state = "2"
+  errno = "5"
+
+The first write_aio request matches the set-state rule and transitions from
+state 0 to state 1.  Once state 1 has been entered, the set-state rule no
+longer matches since it required state 0.  But the inject-error rule now
+matches the next write_aio request and injects EIO (5).
+
+State transition rules support the following attributes:
+
+  event - which type of operation to match (e.g. read_aio, write_aio,
+	  flush_to_os, flush_to_disk).  See the "Events" section for
+          information on events.
+
+  state - (optional) the engine must be in this state number in order for this
+	  rule to match
+
+  new_state - transition to this state number
+
+Suspend and resume
+------------------
+Exercising code paths in block drivers may require specific ordering amongst
+concurrent requests.  The "breakpoint" feature allows requests to be halted on
+a blkdebug event and resumed later.  This makes it possible to achieve
+deterministic ordering when multiple requests are in flight.
+
+Breakpoints on blkdebug events are associated with a user-defined "tag" string.
+This tag serves as an identifier by which the request can be resumed at a later
+point.
+
+See the qemu-io(1) break, resume, remove_break, and wait_break commands for
+details.
-- 
1.9.3

Re: [Qemu-devel] [PATCH v2] docs: add blkdebug block driver documentation

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 5188 bytes --]

On 09/23/2014 04:09 AM, Stefan Hajnoczi wrote:
> The blkdebug block driver is undocumented.  Documenting it is worthwhile
> since it offers powerful error injection features that are used by
> qemu-iotests test cases.
> 
> This document will make it easier for people to learn about and use
> blkdebug.
> 
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> v2:
>  * Added GPL v2 or later license and Red Hat copyright [Eric]
>  * Expanded ini rules file explanation [Paolo]
>  * Added note that errno values depend on the host [Eric]
> 
>  docs/blkdebug.txt | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 155 insertions(+)
>  create mode 100644 docs/blkdebug.txt
> 


> +Rules support the following attributes:
> +
> +  event - which type of operation to match (e.g. read_aio, write_aio,
> +	  flush_to_os, flush_to_disk).  See the "Events" section for
> +          information on events.

TAB vs space damage?

> +
> +  state - (optional) the engine must be in this state number in order for this
> +	  rule to match.  See the "State transitions" section for information
> +          on states.

and again?

> +
> +  errno - the numeric errno value to return when a request matches this rule.
> +          The errno values depend on the host since the numeric values are not
> +          standarized in the POSIX specification.
> +
> +  sector - (optional) a sector number that the request must overlap in order to
> +           match this rule
> +
> +  once - (optional, default "off") only execute this action on the first
> +         matching request
> +
> +  immediately - (optional, default "off") return a NULL BlockDriverAIOCB
> +		pointer and fail without an errno instead.  This exercises the
> +		code path where BlockDriverAIOCB fails and the caller's
> +                BlockDriverCompletionFunc is not invoked.

again?

> +
> +Events
> +------
> +Block drivers provide information about the type of I/O request they are about
> +to make so rules can match specific types of requests.  For example, the qcow2
> +block driver tells blkdebug when it accesses the L1 table so rules can match
> +only L1 table accesses and not other metadata or guest data requests.
> +
> +The core events are:
> +
> +  read_aio - guest data read
> +
> +  write_aio - guest data write
> +
> +  flush_to_os - write out unwritten block driver state (e.g. cached metadata)
> +
> +  flush_to_disk - flush the host block device's disk cache
> +
> +See block/blkdebug.c:event_names[] for the list of available events.  You may

s/available events/additional available events/ ?

> +need to grep block driver source code to understand the meaning of specific
> +events.
> +
> +State transitions
> +-----------------
> +There are cases where more power is needed to match a particular I/O request in
> +a longer sequence of requests.  For example:
> +
> +  write_aio
> +  flush_to_disk
> +  write_aio
> +
> +How do we match the 2nd write_aio but not the first?  This is where state
> +transitions come in.
> +
> +The error injection engine has an integer called the "state" that always starts
> +initialized to 1.  Rules can be conditional on the state and they can
> +transition to a new state.

Is the current state of the engine in a running guest introspectible,
such as through 'query-block'?

> +
> +For example, to match the 2nd write_aio:
> +
> +  [set-state]
> +  event = "write_aio"
> +  state = "1"
> +  new_state = "2"
> +
> +  [inject-error]
> +  event = "write_aio"
> +  state = "2"
> +  errno = "5"
> +
> +The first write_aio request matches the set-state rule and transitions from
> +state 0 to state 1.  Once state 1 has been entered, the set-state rule no
> +longer matches since it required state 0.  But the inject-error rule now

state 0/1 or state 1/2 ?

> +matches the next write_aio request and injects EIO (5).
> +
> +State transition rules support the following attributes:
> +
> +  event - which type of operation to match (e.g. read_aio, write_aio,
> +	  flush_to_os, flush_to_disk).  See the "Events" section for
> +          information on events.

More tab damage?

> +
> +  state - (optional) the engine must be in this state number in order for this
> +	  rule to match
> +
> +  new_state - transition to this state number
> +
> +Suspend and resume
> +------------------
> +Exercising code paths in block drivers may require specific ordering amongst
> +concurrent requests.  The "breakpoint" feature allows requests to be halted on
> +a blkdebug event and resumed later.  This makes it possible to achieve
> +deterministic ordering when multiple requests are in flight.
> +
> +Breakpoints on blkdebug events are associated with a user-defined "tag" string.
> +This tag serves as an identifier by which the request can be resumed at a later
> +point.
> +
> +See the qemu-io(1) break, resume, remove_break, and wait_break commands for
> +details.
> 

Looking closer.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 539 bytes --]

Re: [Qemu-devel] [PATCH v2] docs: add blkdebug block driver documentation

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2679 bytes --]

On Tue, Sep 23, 2014 at 11:12:52AM -0600, Eric Blake wrote:
> On 09/23/2014 04:09 AM, Stefan Hajnoczi wrote:
> > +Rules support the following attributes:
> > +
> > +  event - which type of operation to match (e.g. read_aio, write_aio,
> > +	  flush_to_os, flush_to_disk).  See the "Events" section for
> > +          information on events.
> 
> TAB vs space damage?

Yes, thanks for spotting it.  Fixed in v3.

> > +Events
> > +------
> > +Block drivers provide information about the type of I/O request they are about
> > +to make so rules can match specific types of requests.  For example, the qcow2
> > +block driver tells blkdebug when it accesses the L1 table so rules can match
> > +only L1 table accesses and not other metadata or guest data requests.
> > +
> > +The core events are:
> > +
> > +  read_aio - guest data read
> > +
> > +  write_aio - guest data write
> > +
> > +  flush_to_os - write out unwritten block driver state (e.g. cached metadata)
> > +
> > +  flush_to_disk - flush the host block device's disk cache
> > +
> > +See block/blkdebug.c:event_names[] for the list of available events.  You may
> 
> s/available events/additional available events/ ?

I'll say "for the full list of events" since event_names[] includes
everything.

> > +State transitions
> > +-----------------
> > +There are cases where more power is needed to match a particular I/O request in
> > +a longer sequence of requests.  For example:
> > +
> > +  write_aio
> > +  flush_to_disk
> > +  write_aio
> > +
> > +How do we match the 2nd write_aio but not the first?  This is where state
> > +transitions come in.
> > +
> > +The error injection engine has an integer called the "state" that always starts
> > +initialized to 1.  Rules can be conditional on the state and they can
> > +transition to a new state.
> 
> Is the current state of the engine in a running guest introspectible,
> such as through 'query-block'?

No.

> > +
> > +For example, to match the 2nd write_aio:
> > +
> > +  [set-state]
> > +  event = "write_aio"
> > +  state = "1"
> > +  new_state = "2"
> > +
> > +  [inject-error]
> > +  event = "write_aio"
> > +  state = "2"
> > +  errno = "5"
> > +
> > +The first write_aio request matches the set-state rule and transitions from
> > +state 0 to state 1.  Once state 1 has been entered, the set-state rule no
> > +longer matches since it required state 0.  But the inject-error rule now
> 
> state 0/1 or state 1/2 ?

Thanks for catching this, I should explain the difference between state
0 and 1.

This was a mistake because I original used "0" but it actually has to be
"1".

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]