Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2

[Blue Swirl <blauwirbel@gmail.com>]

On Sat, Nov 27, 2010 at 8:53 AM, Yoshiaki Tamura
<tamura.yoshiaki@lab.ntt.co.jp> wrote:
> 2010/11/27 Stefan Hajnoczi <stefanha@gmail.com>:
>> On Sat, Nov 27, 2010 at 4:29 AM, Yoshiaki Tamura
>> <tamura.yoshiaki@lab.ntt.co.jp> wrote:
>>> 2010/11/27 Blue Swirl <blauwirbel@gmail.com>:
>>>> On Thu, Nov 25, 2010 at 6:06 AM, Yoshiaki Tamura
>>>> <tamura.yoshiaki@lab.ntt.co.jp> wrote:
>>>>> Hi,
>>>>>
>>>>> This patch series is a revised version of Kemari for KVM, which
>>>>> applied comments for the previous post and KVM Forum 2010.  The
>>>>> current code is based on qemu.git
>>>>> f711df67d611e4762966a249742a5f7499e19f99.
>>>>>
>>>>> For general information about Kemari, I've made a wiki page at
>>>>> qemu.org.
>>>>>
>>>>> http://wiki.qemu.org/Features/FaultTolerance
>>>>>
>>>>> The changes from v0.1.1 -> v0.2 are:
>>>>>
>>>>> - Introduce a queue in event-tap to make VM sync live.
>>>>> - Change transaction receiver to a state machine for async receiving.
>>>>> - Replace net/block layer functions with event-tap proxy functions.
>>>>> - Remove dirty bitmap optimization for now.
>>>>> - convert DPRINTF() in ft_trans_file to trace functions.
>>>>> - convert fprintf() in ft_trans_file to error_report().
>>>>> - improved error handling in ft_trans_file.
>>>>> - add a tmp pointer to qemu_del_vm_change_state_handler.
>>>>>
>>>>> The changes from v0.1 -> v0.1.1 are:
>>>>>
>>>>> - events are tapped in net/block layer instead of device emulation layer.
>>>>> - Introduce a new option for -incoming to accept FT transaction.
>>>>> - Removed writev() support to QEMUFile and FdMigrationState for now.  I would
>>>>>  post this work in a different series.
>>>>> - Modified virtio-blk save/load handler to send inuse variable to
>>>>>  correctly replay.
>>>>> - Removed configure --enable-ft-mode.
>>>>> - Removed unnecessary check for qemu_realloc().
>>>>>
>>>>> The first 6 patches modify several functions of qemu to prepare
>>>>> introducing Kemari specific components.
>>>>>
>>>>> The next 6 patches are the components of Kemari.  They introduce
>>>>> event-tap and the FT transaction protocol file based on buffered file.
>>>>> The design document of FT transaction protocol can be found at,
>>>>> http://wiki.qemu.org/images/b/b1/Kemari_sender_receiver_0.5a.pdf
>>>>>
>>>>> Then the following 4 patches modifies dma-helpers, virtio-blk
>>>>> virtio-net and e1000 to replace net/block layer functions with
>>>>> event-tap proxy functions.  Please note that if Kemari is off,
>>>>> event-tap will just passthrough, and there is most no intrusion to
>>>>> exisiting functions including normal live migration.
>>>>
>>>> Would it be possible to make the changes only in the block/net layer,
>>>> so that the devices are not modified at all? That is, the proxy
>>>> function would always replaces the unproxied version.
>>>
>>> I understand the benefit of your suggestion.  However it seems a bit
>>> tricky.  It's because event-tap uses functions of emulators and net,
>>> but block.c is also linked for utilities like qemu-img that doesn't
>>> need emulators or net.  In the previous version, I added function
>>> pointers to get around.
>>>
>>> http://lists.nongnu.org/archive/html/qemu-devel/2010-05/msg02378.html
>>>
>>> I wasn't confident of this approach and discussed it at KVM Forum, and
>>> decided to give a try to replace emulator functions with proxies.
>>> Suggestions are welcomed of course.
>>>
>>>> Somehow I find some similarities to instrumentation patches. Perhaps
>>>> the instrumentation framework could be used (maybe with some changes)
>>>> for Kemari as well? That could be beneficial to both.
>>>
>>> Yes.  I had the same idea but I'm not sure how tracing works.  I think
>>> Stefan Hajnoczi knows it better.
>>>
>>> Stefan, is it possible to call arbitrary functions from the trace
>>> points?
>>
>> Yes, if you add code to ./tracetool.  I'm not sure I see the
>> connection between Kemari and tracing though.
>
> The connection is that it may be possible to remove Kemari
> specific hook point like in ioport.c and exec.c, and let tracing
> notify Kemari instead.

This all depends on how generic we want the trace points become.

One possible extension to the event injection or instrumentation could
be fault injection: based on some rule, make the instrumented function
return error. That would be interesting for testing how guest handles
failure cases.

Maybe it should be also possible to handle event injection in a
generic way. Split the instrumented function to two, before and after
the tracepoint. The tracepoint registers the tail function in addition
to the parameters. This may require a lot of refactoring though.

>> One question I have about Kemari is whether it adds new constraints to
>> the QEMU codebase?  Fault tolerance seems like a cross-cutting concern
>> - everyone writing device emulation or core QEMU code may need to be
>> aware of new constraints.  For example, "you are not allowed to
>> release I/O operations to the outside world directly, instead you need
>> to go through Kemari code which makes I/O transactional and
>> communicates with the passive host".  You have converted e1000,
>> virtio-net, and virtio-blk.  How do we make sure new devices that are
>> merged into qemu.git don't break Kemari?  How do we go about
>> supporting the existing hw/* devices?
>
> Whether Kemari adds constraints such as you mentioned, yes.  If
> the devices (including existing ones) don't call Kemari code,
> they would certainly break Kemari.  Altough using proxies looks
> explicit, to make it unaware from people writing device
> emulation, it's possible to remove proxies and put changes only
> into the block/net layer as Blue suggested.

I'd prefer that approach if possible.