Re: [Qemu-devel] Microcheckpointing: Memory-VCPU / Disk State consistency

[Hongyang Yang <yanghy@cn.fujitsu.com>]

在 09/11/2014 09:50 AM, Michael R. Hines 写道:
> On 09/10/2014 11:43 PM, Walid Nouri wrote:
>> Hello Michael, Hello Paolo
>> i have „studied“ the available documentation/Information and tried to get an
>> idea of the QEMU live block operation possibilities.
>>
>> I think the MC protocol doesn’t need synchronous block device replication
>> because primary and secondary VM are not synchronous. The state of the primary
>> is allays ahead of the state of the secondary. When the primary is in epoch(n)
>> the secondary is in epoch(n-1).
>>
>> What MC needs is a block device agnostic, controlled and asynchronous approach
>> for replicating the contents of block devices and its state changes to the
>> secondary VM while the primary VM is running. Asynchronous block transfer is
>> important to allow maximum performance for the primary VM, while keeping the
>> secondary VM updated with state changes.
>>
>> The block device replication should be possible in two stages or modes.
>>
>> The first stage is the live copy of all block devices of the primary to the
>> secondary. This is necessary if the secondary doesn’t have an existing image
>> which is in sync with the primary at the time MC has started. This is not very
>> convenient but as far as I know actually there is no mechanism for persistent
>> dirty bitmap in QEMU.
>>
>> The second stage (mode) is the replication of block device state changes
>> (modified blocks) to keep the image on the secondary in sync with the primary.
>> The mirrored blocks must be buffered in ram (block buffer) until the complete
>> Checkpoint (RAM, vCPU, device state) can be committed.
>>
>> For keeping the complete system state consistent on the secondary system there
>> must be a possibility for MC to commit/discard block device state changes. In
>> normal operation the mirrored block device state changes (block buffer) are
>> committed to disk when the complete checkpoint is committed. In case of a
>> crash of the primary system while transferring a checkpoint the data in the
>> block buffer corresponding to the failed Checkpoint must be discarded.
>>
>> The storage architecture should be “shared nothing” so that no shared storage
>> is required and primary/secondary can have separate block device images.
>>
>> I think this can be achieved by drive-mirror and a filter block driver.
>> Another approach could be to exploit the block migration functionality of live
>> migration with a filter block driver.
>>
>> The drive-mirror (and live migration) does not rely on shared storage and
>> allow live block device copy and incremental syncing.
>>
>> A block buffer can be implemented with a QEMU filter block driver. It should
>> sit at the same position as the Quorum driver in the block driver hierarchy.
>> When using block filter approach MC will be transparent and block device
>> agnostic.
>>
>> The block buffer filter must have an Interface which allows MC control the
>> commits or discards of block device state changes. I have no idea where to put
>> such an interface to stay conform with QEMU coding style.
>>
>>
>> I’m sure there are alternative and better approaches and I’m open for any ideas
>>
>>
>> Walid
>>
>> Am 17.08.2014 11:52, schrieb Paolo Bonzini:
>>> Il 11/08/2014 22:15, Michael R. Hines ha scritto:
>>>> Excellent question: QEMU does have a feature called "drive-mirror"
>>>> in block/mirror.c that was introduced a couple of years ago. I'm not
>>>> sure what the
>>>> adoption rate of the feature is, but I would start with that one.
>>>
>>> block/mirror.c is asynchronous, and there's no support for communicating
>>> checkpoints back to the master. However, the quorum disk driver could
>>> be what you need.
>>>
>>> There's also a series on the mailing list that lets quorum read only
>>> from the primary, so that quorum can still do replication and fault
>>> tolerance, but skip fault detection.
>>>
>>> Paolo
>>>
>>>> There is also a second fault tolerance implementation that works a
>>>> little differently called
>>>> "COLO" - you may have seen those emails on the list too, but their
>>>> method does not require a disk replication solution, if I recall correctly.
>>>
>>
>
> Nice description of the problem - would you like to put this information on the
> MC wiki page? (Just send an email to the list that says "request for wiki
> account, please" in the subject - and they will make an account for you.
>
> A drive-mirror + filter driver solution sounds like a good plan overall,
> of course the devil is in the details =)

If I understand correctly, this is similar to our approach. Disk replication
is definitely on our plan and we will post RFC patches that include Disk
replication, you can keep an eye on COLO patches:)

>
> I don't know how much time you have to spend on actual code, but even a
> description of what a "theoretical" interface between MC and drive-mirror would
> look like would go a long way even without code.
>
> Your investigations would also help "drive" a solution to this problem for the
> COLO team as well - I believe they need the same thing....
>
> - Michael
>
> .
>

-- 
Thanks,
Yang.