[Qemu-devel] Docs for and debugging of Asynchronous I/O

[Qemu-devel] Docs for and debugging of Asynchronous I/O

[Ot ten Thije]

[-- Attachment #1: Type: text/plain, Size: 1383 bytes --]

Hello,

I am working on fixing the savevm/loadvm functionality in the Android
emulator, and the two issues I've encountered so far both appear to stem
from the asynchronous I/O (AIO) code. In both cases, the emulator busy-waits
indefinitely for an operation that never signals completion.

Unfortunately I am not really familiar with AIO, so I was hoping one of the
emulator devs could point me some resources (design docs, general
introduction, etc.). I've done some searching myself and found some docs for
the Linux kernel AIO implementation (http://lse.sourceforge.net/io/aio.html),
but I'm not sure to what extent it applies to the QEMU code.

Tips for debugging AIO would also be greatly appreciated. I can trace the
execution until I am within the (emulated) device driver (i.e.
block/qcow2.c:qcow_aio_writev()), but haven't been able to pinpoint the
exact location where the actual async call is made. This makes it difficult
to identify the code that should signal completion back to the main process
(and apparently fails to do so). I know this code is called though, because
some asynchronous calls *do* signal completion.

I realize that the Android emulator is a rather heavy fork of QEMU, so
giving specific advice will probably be difficult. However, the overall
approach is still the same, so I hope you can help me get a better
understanding of that.


Ot ten Thije

[-- Attachment #2: Type: text/html, Size: 1644 bytes --]

Re: [Qemu-devel] Docs for and debugging of Asynchronous I/O

[Stefan Hajnoczi]

On Tue, Jul 20, 2010 at 7:34 PM, Ot ten Thije <ottenthije@google.com> wrote:
> Hello,
> I am working on fixing the savevm/loadvm functionality in the Android
> emulator, and the two issues I've encountered so far both appear to stem
> from the asynchronous I/O (AIO) code. In both cases, the emulator busy-waits
> indefinitely for an operation that never signals completion.
> Unfortunately I am not really familiar with AIO, so I was hoping one of the
> emulator devs could point me some resources (design docs, general
> introduction, etc.). I've done some searching myself and found some docs for
> the Linux kernel AIO implementation
> (http://lse.sourceforge.net/io/aio.html), but I'm not sure to what extent it
> applies to the QEMU code.

qemu.git has support for Linux AIO, which you linked to.  The code for
that is in linux-aio.c.  But by default it uses a thread pool to
perform the normal blocking syscalls out-of-line with respect to the
VM, see posix-aio-compat.c.

Both of these aio backends are driven from the QEMU block layer, which
provides a block device interface that several drivers implement
(qcow2, raw, ...).  bdrv_aio_readv() and bdrv_aio_writev() are the
main entry points.

I wonder if you can track down the sequence of block interface calls
that causes the hang.  Then it should be possible to construct a
simple test case and even try it again qemu.git/master.

> Tips for debugging AIO would also be greatly appreciated. I can trace the
> execution until I am within the (emulated) device driver (i.e.
> block/qcow2.c:qcow_aio_writev()), but haven't been able to pinpoint the
> exact location where the actual async call is made. This makes it difficult
> to identify the code that should signal completion back to the main process
> (and apparently fails to do so). I know this code is called though, because
> some asynchronous calls *do* signal completion.
> I realize that the Android emulator is a rather heavy fork of QEMU, so
> giving specific advice will probably be difficult. However, the overall
> approach is still the same, so I hope you can help me get a better
> understanding of that.

Have you checked QEMU's git log to see if there are fixes in qemu.git
that could fix this for the Android emulator?

Stefan

>
> Ot ten Thije

Re: [Qemu-devel] Docs for and debugging of Asynchronous I/O

[Anthony Liguori]

On 07/20/2010 01:34 PM, Ot ten Thije wrote:
> Hello,
>
> I am working on fixing the savevm/loadvm functionality in the Android 
> emulator, and the two issues I've encountered so far both appear to 
> stem from the asynchronous I/O (AIO) code. In both cases, the emulator 
> busy-waits indefinitely for an operation that never signals completion.
>
> Unfortunately I am not really familiar with AIO, so I was hoping one 
> of the emulator devs could point me some resources (design docs, 
> general introduction, etc.). I've done some searching myself and found 
> some docs for the Linux kernel AIO implementation 
> (http://lse.sourceforge.net/io/aio.html), but I'm not sure to what 
> extent it applies to the QEMU code.
>
> Tips for debugging AIO would also be greatly appreciated. I can trace 
> the execution until I am within the (emulated) device driver (i.e. 
> block/qcow2.c:qcow_aio_writev()), but haven't been able to pinpoint 
> the exact location where the actual async call is made. This makes it 
> difficult to identify the code that should signal completion back to 
> the main process (and apparently fails to do so). I know this code is 
> called though, because some asynchronous calls *do* signal completion.

TCG translates guest code into small sequences of host code (basic 
blocks).  These basic blocks can be chained together such that one block 
directly jmps to the next block.  The effect is that a guest can run a 
tight loop whereas guest code continuously runs without a chance for 
QEMU to do any work.

To allow qemu to make forward progress in such a scenario, we program 
signals to fire.  Currently, the signals fire in a number of 
circumstances including when AIO operations complete, or when a periodic 
timer needs to fire.

When dealing with multiple threads, it's very easy to screw things up by 
not masking signals properly.  Often times, this is hidden because the 
periodic timer runs often enough that it doesn't matter if you miss a 
signal.  An exception, however, would be emulation of synchronous code.  
This tends to happen in qcow2 metadata operations since they are still 
synchronous.  To complete this emulation, we have to block the current 
thread until the I/O operation completes.  But since qemu isn't 
re-entrant, we can't run the full main loop as that could trigger 
re-entrancy in qcow2.  To work around this, we implement "idle bottom 
halves" which are special bottom halves that are run by the normal io 
loop but also by a special I/O used exclusive for emulating synchronous 
writes.

To further complicate matters, non-x86 platforms (like ARM) are more 
likely to not use a periodic timer which makes these bugs much more obvious.

>
> I realize that the Android emulator is a rather heavy fork of QEMU, so 
> giving specific advice will probably be difficult. However, the 
> overall approach is still the same, so I hope you can help me get a 
> better understanding of that.

This is the problem with forking.  This is very hairy code that requires 
careful attention to detail.  If you're introducing any type of 
threading, disk emulation, or changes to the block subsystem, chances 
are you've done it wrong.

Regards,

Anthony Liguori

>
> Ot ten Thije

Re: [Qemu-devel] Docs for and debugging of Asynchronous I/O

[Ot ten Thije]

On Tue, Jul 20, 2010 at 9:57 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> On Tue, Jul 20, 2010 at 7:34 PM, Ot ten Thije <ottenthije@google.com> wrote:
> > Hello,
> > I am working on fixing the savevm/loadvm functionality in the Android
> > emulator, and the two issues I've encountered so far both appear to stem
> > from the asynchronous I/O (AIO) code. In both cases, the emulator busy-waits
> > indefinitely for an operation that never signals completion.
> > Unfortunately I am not really familiar with AIO, so I was hoping one of the
> > emulator devs could point me some resources (design docs, general
> > introduction, etc.). I've done some searching myself and found some docs for
> > the Linux kernel AIO implementation
> > (http://lse.sourceforge.net/io/aio.html), but I'm not sure to what extent it
> > applies to the QEMU code.
>
> qemu.git has support for Linux AIO, which you linked to.  The code for
> that is in linux-aio.c.  But by default it uses a thread pool to
> perform the normal blocking syscalls out-of-line with respect to the
> VM, see posix-aio-compat.c.
>
> Both of these aio backends are driven from the QEMU block layer, which
> provides a block device interface that several drivers implement
> (qcow2, raw, ...).  bdrv_aio_readv() and bdrv_aio_writev() are the
> main entry points.
>
> I wonder if you can track down the sequence of block interface calls
> that causes the hang.  Then it should be possible to construct a
> simple test case and even try it again qemu.git/master.
>

I've been trying to track this down, and it seems that the program
blocks/busy-waits the first time data is actually being flushed to
file. Unfortunately I haven't been able to identify the exact point at
which the background thread is started that takes care of the actual
I/O.

I would like to put a breakpoint on that code to see what exactly is
going on there. Could you tell me which function does this in mainline
QEMU? I've seen some suggestive names (qcow_aio_setup and
qemu_iovec_init_external), but their implementations do not hint in
any way at initiating a background process/thread. Or am I
misunderstanding the nature of AIO calls entirely?


> > Tips for debugging AIO would also be greatly appreciated. I can trace the
> > execution until I am within the (emulated) device driver (i.e.
> > block/qcow2.c:qcow_aio_writev()), but haven't been able to pinpoint the
> > exact location where the actual async call is made. This makes it difficult
> > to identify the code that should signal completion back to the main process
> > (and apparently fails to do so). I know this code is called though, because
> > some asynchronous calls *do* signal completion.
> > I realize that the Android emulator is a rather heavy fork of QEMU, so
> > giving specific advice will probably be difficult. However, the overall
> > approach is still the same, so I hope you can help me get a better
> > understanding of that.
>
> Have you checked QEMU's git log to see if there are fixes in qemu.git
> that could fix this for the Android emulator?

I've gone through the logs as you suggested (grepping for "aio" and
"qcow"). Unfortunately the number of changes in qcow/aio related code
since the last merge from mainline to Android is quite large, making
it very difficult to pinpoint any specific change as a breaking point.
A great deal of the API has changed as well, making it difficult to
replace the current implementations with those from mainline.

Ot

>
> Stefan
>
> >
> > Ot ten Thije

Re: [Qemu-devel] Docs for and debugging of Asynchronous I/O

[Ot ten Thije]

On Tue, Jul 20, 2010 at 10:47 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> On 07/20/2010 01:34 PM, Ot ten Thije wrote:
>>
>> Hello,
>>
>> I am working on fixing the savevm/loadvm functionality in the Android
>> emulator, and the two issues I've encountered so far both appear to stem
>> from the asynchronous I/O (AIO) code. In both cases, the emulator busy-waits
>> indefinitely for an operation that never signals completion.
>>
>> Unfortunately I am not really familiar with AIO, so I was hoping one of
>> the emulator devs could point me some resources (design docs, general
>> introduction, etc.). I've done some searching myself and found some docs for
>> the Linux kernel AIO implementation
>> (http://lse.sourceforge.net/io/aio.html), but I'm not sure to what extent it
>> applies to the QEMU code.
>>
>> Tips for debugging AIO would also be greatly appreciated. I can trace the
>> execution until I am within the (emulated) device driver (i.e.
>> block/qcow2.c:qcow_aio_writev()), but haven't been able to pinpoint the
>> exact location where the actual async call is made. This makes it difficult
>> to identify the code that should signal completion back to the main process
>> (and apparently fails to do so). I know this code is called though, because
>> some asynchronous calls *do* signal completion.
>
> TCG translates guest code into small sequences of host code (basic blocks).
>  These basic blocks can be chained together such that one block directly
> jmps to the next block.  The effect is that a guest can run a tight loop
> whereas guest code continuously runs without a chance for QEMU to do any
> work.
>
> To allow qemu to make forward progress in such a scenario, we program
> signals to fire.  Currently, the signals fire in a number of circumstances
> including when AIO operations complete, or when a periodic timer needs to
> fire.
>
> When dealing with multiple threads, it's very easy to screw things up by not
> masking signals properly.  Often times, this is hidden because the periodic
> timer runs often enough that it doesn't matter if you miss a signal.  An
> exception, however, would be emulation of synchronous code.  This tends to
> happen in qcow2 metadata operations since they are still synchronous.  To
> complete this emulation, we have to block the current thread until the I/O
> operation completes.  But since qemu isn't re-entrant, we can't run the full
> main loop as that could trigger re-entrancy in qcow2.  To work around this,
> we implement "idle bottom halves" which are special bottom halves that are
> run by the normal io loop but also by a special I/O used exclusive for
> emulating synchronous writes.
>
> To further complicate matters, non-x86 platforms (like ARM) are more likely
> to not use a periodic timer which makes these bugs much more obvious.
>

Thank you for the explanation, this makes the high-level structure of
the system more understandable. However, as I replied to Stefan just
now, I still have some difficulty finding the exact code that is
executed while the main loop busy-waits. That is, the code executed
asynchronously that should raise the signal indicating that the
operation completed. Could you point me to the code doing this in
mainline QEMU? Hopefully I can use that to find the corresponding code
in the Android emulator.

Regards,

Ot ten Thije


>>
>> I realize that the Android emulator is a rather heavy fork of QEMU, so
>> giving specific advice will probably be difficult. However, the overall
>> approach is still the same, so I hope you can help me get a better
>> understanding of that.
>
> This is the problem with forking.  This is very hairy code that requires
> careful attention to detail.  If you're introducing any type of threading,
> disk emulation, or changes to the block subsystem, chances are you've done
> it wrong.
>
> Regards,
>
> Anthony Liguori
>
>>
>> Ot ten Thije
>
>
>