Re: [Qemu-devel] Docs for and debugging of Asynchronous I/O

[Ot ten Thije <ottenthije@google.com>]

On Tue, Jul 20, 2010 at 10:47 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> On 07/20/2010 01:34 PM, Ot ten Thije wrote:
>>
>> Hello,
>>
>> I am working on fixing the savevm/loadvm functionality in the Android
>> emulator, and the two issues I've encountered so far both appear to stem
>> from the asynchronous I/O (AIO) code. In both cases, the emulator busy-waits
>> indefinitely for an operation that never signals completion.
>>
>> Unfortunately I am not really familiar with AIO, so I was hoping one of
>> the emulator devs could point me some resources (design docs, general
>> introduction, etc.). I've done some searching myself and found some docs for
>> the Linux kernel AIO implementation
>> (http://lse.sourceforge.net/io/aio.html), but I'm not sure to what extent it
>> applies to the QEMU code.
>>
>> Tips for debugging AIO would also be greatly appreciated. I can trace the
>> execution until I am within the (emulated) device driver (i.e.
>> block/qcow2.c:qcow_aio_writev()), but haven't been able to pinpoint the
>> exact location where the actual async call is made. This makes it difficult
>> to identify the code that should signal completion back to the main process
>> (and apparently fails to do so). I know this code is called though, because
>> some asynchronous calls *do* signal completion.
>
> TCG translates guest code into small sequences of host code (basic blocks).
>  These basic blocks can be chained together such that one block directly
> jmps to the next block.  The effect is that a guest can run a tight loop
> whereas guest code continuously runs without a chance for QEMU to do any
> work.
>
> To allow qemu to make forward progress in such a scenario, we program
> signals to fire.  Currently, the signals fire in a number of circumstances
> including when AIO operations complete, or when a periodic timer needs to
> fire.
>
> When dealing with multiple threads, it's very easy to screw things up by not
> masking signals properly.  Often times, this is hidden because the periodic
> timer runs often enough that it doesn't matter if you miss a signal.  An
> exception, however, would be emulation of synchronous code.  This tends to
> happen in qcow2 metadata operations since they are still synchronous.  To
> complete this emulation, we have to block the current thread until the I/O
> operation completes.  But since qemu isn't re-entrant, we can't run the full
> main loop as that could trigger re-entrancy in qcow2.  To work around this,
> we implement "idle bottom halves" which are special bottom halves that are
> run by the normal io loop but also by a special I/O used exclusive for
> emulating synchronous writes.
>
> To further complicate matters, non-x86 platforms (like ARM) are more likely
> to not use a periodic timer which makes these bugs much more obvious.
>

Thank you for the explanation, this makes the high-level structure of
the system more understandable. However, as I replied to Stefan just
now, I still have some difficulty finding the exact code that is
executed while the main loop busy-waits. That is, the code executed
asynchronously that should raise the signal indicating that the
operation completed. Could you point me to the code doing this in
mainline QEMU? Hopefully I can use that to find the corresponding code
in the Android emulator.

Regards,

Ot ten Thije


>>
>> I realize that the Android emulator is a rather heavy fork of QEMU, so
>> giving specific advice will probably be difficult. However, the overall
>> approach is still the same, so I hope you can help me get a better
>> understanding of that.
>
> This is the problem with forking.  This is very hairy code that requires
> careful attention to detail.  If you're introducing any type of threading,
> disk emulation, or changes to the block subsystem, chances are you've done
> it wrong.
>
> Regards,
>
> Anthony Liguori
>
>>
>> Ot ten Thije
>
>
>