[Qemu-devel] Block I/O Tracing

[Qemu-devel] Block I/O Tracing

[Luis Pabón]

Hi all,
   I am really interested in the following feature: 
http://wiki.qemu.org/Features/Block/Todo#Trace_guest_block_I.2FO.2C_replay_with_qemu-io 
.  Is there any more information about this feature?

Thanks,

- Luis

Re: [Qemu-devel] Block I/O Tracing

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2996 bytes --]

On Wed, Dec 16, 2015 at 12:37:39PM -0500, Luis Pabón wrote:
>   I am really interested in the following feature: http://wiki.qemu.org/Features/Block/Todo#Trace_guest_block_I.2FO.2C_replay_with_qemu-io
> .  Is there any more information about this feature?

Hi Luis,
It's a useful idea but non-trivial to implement, I think.

Often times when an I/O performance problem is reported:

1. Reproducing the poor performance is difficult due to the requirements
   of setting up the same VM and guest application on comparable storage
   hardware.  Usually developers ask the bug reporter questions until
   the issue is narrowed down to essentially a microbenchmark that can
   easily be reproduced.

   This is really a funnel: only a fraction of cases makes it from the
   original bug report to a solution because the bug reporter or
   developer might give up during the back-and-forth stage of asking
   questions to narrow down the problem.

2. File system, volume manager, and operating system block layer
   developers need straightforward reproducible test cases that do not
   involve setting up a VM.  It's hard to get a file system developer's
   attention if you ask them to set up a VM in the hopes of spotting a
   file system bug.

3. QEMU developers may implement performance optimizations which need to
   be evaluated against real-world I/O patterns.  Again, if it takes too
   much effort setting up various VMs to benchmark against then less
   performance evaluation will end up being done.

So the idea was to record I/O patterns from a real workload and then
share them so others can replay them with qemu-io or fio (without
running a VM).

This is appealing because qemu-io and fio are small userspace programs
that do not run a VM.  They are much easier to run and analyze than
QEMU.

Unfortunately there are a few problems with the approach of replaying
recorded I/O patterns:

1. Eliminating the VM also removes the storage controller emulation
   (e.g. virtio-blk) and its impact on performance.  Some bottlenecks
   are related to the storage controller but they wouldn't be
   reproduced.

2. Eliminating the guest application that generates I/O makes it hard to
   replay the "think time" and dependencies between I/O requests.  For
   example, a database application may submit a read request and a write
   request at the same time.  When the write request completes it does
   some additional processing (the "think time" or delay before the next
   I/O request) and then submits a flush request.

   If you have a recorded I/O pattern it does not include the
   information that the flush request must happen n microseconds after
   the write request completes.  So when you replay the I/O pattern the
   behavior will not be the same as the real application.

Anyway, it's an interesting problem and I'd start with blktrace and fio.
Then look at how to integrate that sort of trace with QEMU.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] Block I/O Tracing

[Luis Pabon]

Great summary Stefan.  I think you are correct, and it is definitely
non-trivial to get the accuracy of the trace as close as possible to
real-world timings.  Sometimes, like in the block driver IO responses
or caching algorithm studies, having the IO trace of an application
be 'close' enough could be still beneficial.  

I definitely agree with your statements, and I will start by checking
first how blktrace does it.

- Luis


----- Original Message -----
From: "Stefan Hajnoczi" <stefanha@gmail.com>
To: "Luis Pabón" <lpabon@redhat.com>
Cc: qemu-devel@nongnu.org
Sent: Tuesday, December 22, 2015 10:21:52 PM
Subject: Re: [Qemu-devel] Block I/O Tracing

On Wed, Dec 16, 2015 at 12:37:39PM -0500, Luis Pabón wrote:
>   I am really interested in the following feature: http://wiki.qemu.org/Features/Block/Todo#Trace_guest_block_I.2FO.2C_replay_with_qemu-io
> .  Is there any more information about this feature?

Hi Luis,
It's a useful idea but non-trivial to implement, I think.

Often times when an I/O performance problem is reported:

1. Reproducing the poor performance is difficult due to the requirements
   of setting up the same VM and guest application on comparable storage
   hardware.  Usually developers ask the bug reporter questions until
   the issue is narrowed down to essentially a microbenchmark that can
   easily be reproduced.

   This is really a funnel: only a fraction of cases makes it from the
   original bug report to a solution because the bug reporter or
   developer might give up during the back-and-forth stage of asking
   questions to narrow down the problem.

2. File system, volume manager, and operating system block layer
   developers need straightforward reproducible test cases that do not
   involve setting up a VM.  It's hard to get a file system developer's
   attention if you ask them to set up a VM in the hopes of spotting a
   file system bug.

3. QEMU developers may implement performance optimizations which need to
   be evaluated against real-world I/O patterns.  Again, if it takes too
   much effort setting up various VMs to benchmark against then less
   performance evaluation will end up being done.

So the idea was to record I/O patterns from a real workload and then
share them so others can replay them with qemu-io or fio (without
running a VM).

This is appealing because qemu-io and fio are small userspace programs
that do not run a VM.  They are much easier to run and analyze than
QEMU.

Unfortunately there are a few problems with the approach of replaying
recorded I/O patterns:

1. Eliminating the VM also removes the storage controller emulation
   (e.g. virtio-blk) and its impact on performance.  Some bottlenecks
   are related to the storage controller but they wouldn't be
   reproduced.

2. Eliminating the guest application that generates I/O makes it hard to
   replay the "think time" and dependencies between I/O requests.  For
   example, a database application may submit a read request and a write
   request at the same time.  When the write request completes it does
   some additional processing (the "think time" or delay before the next
   I/O request) and then submits a flush request.

   If you have a recorded I/O pattern it does not include the
   information that the flush request must happen n microseconds after
   the write request completes.  So when you replay the I/O pattern the
   behavior will not be the same as the real application.

Anyway, it's an interesting problem and I'd start with blktrace and fio.
Then look at how to integrate that sort of trace with QEMU.

Stefan