Re: [Qemu-devel] Microcheckpointing: Memory-VCPU / Disk State consistency

["Michael R. Hines" <mrhines@linux.vnet.ibm.com>]

Excellent question: QEMU does have a feature called "drive-mirror"
in block/mirror.c that was introduced a couple of years ago. I'm not 
sure what the
adoption rate of the feature is, but I would start with that one.

There is also a second fault tolerance implementation that works a 
little differently called
"COLO" - you may have seen those emails on the list too, but their
method does not require a disk replication solution, if I recall correctly.

I know the time pressure that comes during a thesis, though =), so
there's no pressure to work on it - but that is the most pressing issue
in the implementation today. (Lack of disk replication in 
micro-checkpointing.)

The MC implementation also needs to be re-based against the latest
master - I just haven't had a chance to do it yet because some of my
hardware has been taken away from me the last few months - will
see if I can find some reasonable hardware soon.

- Michael

On 08/12/2014 01:22 AM, Walid Nouri wrote:
> Hi,
> I will do my best to make a contribution :-)
>
> Are there alternative ways of replicating local storage other than 
> DRBD that are possibly feasible?
> Some that are directly build into Qemu?
>
> Walid
>
> Am 09.08.2014 14:25, schrieb Michael R. Hines:
>> On Sat, 2014-08-09 at 14:08 +0200, Walid Nouri wrote:
>>> Hi Michael,
>>> how is the weather in Bejing? :-)
>> It's terrible. Lots of pollution =(
>>
>>> May I ask you some questions to your MC implementation?
>>>
>>> Currently i'm trying  to understand the general working of the MC
>>> protokoll and possible problems that can occur so that I can discuss it
>>> in my thesis.
>>>
>>> As far as i have understand MC relies on a shared disk. Output of the
>>> primary vm are directly written, network output is buffered until the
>>> corresponding checkpoint is acknowledged.
>>>
>>> One problem that comes into my mind is: What happens when the 
>>> primary vm
>>> writes to the disk and crashes before sending a corresponding 
>>> checkpoint?
>>>
>> The MC implementation itself is incomplete, today. (I need help).
>>
>> The Xen Remus implementation uses the DRBD system to "mirror" all disk
>> writes to the source and destination before completing each checkpoint.
>>
>> The KVM (mc) implementation needs exactly the same support, but it is
>> missing today.
>>
>> Until that happens, we are *required* to use root-over-iSCSI or
>> root-over-NFS (meaning that the guest filesystem is mounted directly
>> inside the virtual machine without the host knowing about it.
>>
>> This has the effect of translating all disk I/O into network I/O,
>> and since network I/O is already buffered, then we are safe.
>>
>>
>>> Here an example: The Primary state is in the actual epoch epoch (n),
>>> secondary state is in epoch (n-1). The primary writes to disk and
>>> crashes before or while sending the checkpoint n. In this case the
>>> secondary memory state is still at epoch (n-1) and the state of the
>>> shared Disk corresponds to the primary state of epoch (n).
>>>
>>> How does MC guaranty that the Disk state of the backup vm is consistent
>>> with its Memory state?
>> As I mentioned above, we need the equivalent of the Xen solution, but I
>> just haven't had the time to write it (or incorporate someone else's
>> implementation). Patch is welcome =)
>>
>>> Is Memory-VCPU / Disk State consistency necessary under all 
>>> circumstances?
>>> Or can this be neglected because the secondary will (after a fail over)
>>> repeat the same instructions and finally write to disk the same (as the
>>> primary before) data for a second time?
>>> Could this lead to fatal inconsistencies?
>>>
>>> Walid
>>>
>>
>>
>>
>> - Michael
>>
>>
>>
>
>