Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2

[Yoshiaki Tamura <tamura.yoshiaki@lab.ntt.co.jp>]

2010/11/27 Blue Swirl <blauwirbel@gmail.com>:
> On Sat, Nov 27, 2010 at 8:53 AM, Yoshiaki Tamura
> <tamura.yoshiaki@lab.ntt.co.jp> wrote:
>> 2010/11/27 Stefan Hajnoczi <stefanha@gmail.com>:
>>> On Sat, Nov 27, 2010 at 4:29 AM, Yoshiaki Tamura
>>> <tamura.yoshiaki@lab.ntt.co.jp> wrote:
>>>> 2010/11/27 Blue Swirl <blauwirbel@gmail.com>:
>>>>> On Thu, Nov 25, 2010 at 6:06 AM, Yoshiaki Tamura
>>>>> <tamura.yoshiaki@lab.ntt.co.jp> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> This patch series is a revised version of Kemari for KVM, which
>>>>>> applied comments for the previous post and KVM Forum 2010.  The
>>>>>> current code is based on qemu.git
>>>>>> f711df67d611e4762966a249742a5f7499e19f99.
>>>>>>
>>>>>> For general information about Kemari, I've made a wiki page at
>>>>>> qemu.org.
>>>>>>
>>>>>> http://wiki.qemu.org/Features/FaultTolerance
>>>>>>
>>>>>> The changes from v0.1.1 -> v0.2 are:
>>>>>>
>>>>>> - Introduce a queue in event-tap to make VM sync live.
>>>>>> - Change transaction receiver to a state machine for async receiving.
>>>>>> - Replace net/block layer functions with event-tap proxy functions.
>>>>>> - Remove dirty bitmap optimization for now.
>>>>>> - convert DPRINTF() in ft_trans_file to trace functions.
>>>>>> - convert fprintf() in ft_trans_file to error_report().
>>>>>> - improved error handling in ft_trans_file.
>>>>>> - add a tmp pointer to qemu_del_vm_change_state_handler.
>>>>>>
>>>>>> The changes from v0.1 -> v0.1.1 are:
>>>>>>
>>>>>> - events are tapped in net/block layer instead of device emulation layer.
>>>>>> - Introduce a new option for -incoming to accept FT transaction.
>>>>>> - Removed writev() support to QEMUFile and FdMigrationState for now.  I would
>>>>>>  post this work in a different series.
>>>>>> - Modified virtio-blk save/load handler to send inuse variable to
>>>>>>  correctly replay.
>>>>>> - Removed configure --enable-ft-mode.
>>>>>> - Removed unnecessary check for qemu_realloc().
>>>>>>
>>>>>> The first 6 patches modify several functions of qemu to prepare
>>>>>> introducing Kemari specific components.
>>>>>>
>>>>>> The next 6 patches are the components of Kemari.  They introduce
>>>>>> event-tap and the FT transaction protocol file based on buffered file.
>>>>>> The design document of FT transaction protocol can be found at,
>>>>>> http://wiki.qemu.org/images/b/b1/Kemari_sender_receiver_0.5a.pdf
>>>>>>
>>>>>> Then the following 4 patches modifies dma-helpers, virtio-blk
>>>>>> virtio-net and e1000 to replace net/block layer functions with
>>>>>> event-tap proxy functions.  Please note that if Kemari is off,
>>>>>> event-tap will just passthrough, and there is most no intrusion to
>>>>>> exisiting functions including normal live migration.
>>>>>
>>>>> Would it be possible to make the changes only in the block/net layer,
>>>>> so that the devices are not modified at all? That is, the proxy
>>>>> function would always replaces the unproxied version.
>>>>
>>>> I understand the benefit of your suggestion.  However it seems a bit
>>>> tricky.  It's because event-tap uses functions of emulators and net,
>>>> but block.c is also linked for utilities like qemu-img that doesn't
>>>> need emulators or net.  In the previous version, I added function
>>>> pointers to get around.
>>>>
>>>> http://lists.nongnu.org/archive/html/qemu-devel/2010-05/msg02378.html
>>>>
>>>> I wasn't confident of this approach and discussed it at KVM Forum, and
>>>> decided to give a try to replace emulator functions with proxies.
>>>> Suggestions are welcomed of course.
>>>>
>>>>> Somehow I find some similarities to instrumentation patches. Perhaps
>>>>> the instrumentation framework could be used (maybe with some changes)
>>>>> for Kemari as well? That could be beneficial to both.
>>>>
>>>> Yes.  I had the same idea but I'm not sure how tracing works.  I think
>>>> Stefan Hajnoczi knows it better.
>>>>
>>>> Stefan, is it possible to call arbitrary functions from the trace
>>>> points?
>>>
>>> Yes, if you add code to ./tracetool.  I'm not sure I see the
>>> connection between Kemari and tracing though.
>>
>> The connection is that it may be possible to remove Kemari
>> specific hook point like in ioport.c and exec.c, and let tracing
>> notify Kemari instead.
>
> This all depends on how generic we want the trace points become.
>
> One possible extension to the event injection or instrumentation could
> be fault injection: based on some rule, make the instrumented function
> return error. That would be interesting for testing how guest handles
> failure cases.
>
> Maybe it should be also possible to handle event injection in a
> generic way. Split the instrumented function to two, before and after
> the tracepoint. The tracepoint registers the tail function in addition
> to the parameters. This may require a lot of refactoring though.

The idea looks cool but it's a bit out of the range I can handle
now:-)  Let's keep the idea of binding with trace points for now,
and focus on how to insert net/block tap points.

>>> One question I have about Kemari is whether it adds new constraints to
>>> the QEMU codebase?  Fault tolerance seems like a cross-cutting concern
>>> - everyone writing device emulation or core QEMU code may need to be
>>> aware of new constraints.  For example, "you are not allowed to
>>> release I/O operations to the outside world directly, instead you need
>>> to go through Kemari code which makes I/O transactional and
>>> communicates with the passive host".  You have converted e1000,
>>> virtio-net, and virtio-blk.  How do we make sure new devices that are
>>> merged into qemu.git don't break Kemari?  How do we go about
>>> supporting the existing hw/* devices?
>>
>> Whether Kemari adds constraints such as you mentioned, yes.  If
>> the devices (including existing ones) don't call Kemari code,
>> they would certainly break Kemari.  Altough using proxies looks
>> explicit, to make it unaware from people writing device
>> emulation, it's possible to remove proxies and put changes only
>> into the block/net layer as Blue suggested.
>
> I'd prefer that approach if possible.

Thanks.  Let me see how others think too.

Yoshi

> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>