Re: [Qemu-devel] Microcheckpointing: Memory-VCPU / Disk State consistency

[Walid Nouri <walid.nouri@gmail.com>]

Yes...
Time is a problem, and it‘s currently running out... ;-)

I think the first step is to reason about possible approaches and how 
they can be implemented in QEMU. The implementation can follow later :-)

Thank you for the hint with the drive-mirror feature.
I will take a look at it and surely come back with new questions :-)

I also think that disc replication ist the most pressing issue for MC.
While looking to find some ideas for approaches to replicating block 
devices I have read the paper about the Remus implementation. I think MC 
can take a similar approach for local disk.

Here are the main facts that I have understood:

Local disk contents is viewed as internal state the primary and secondary.
In the explanation they describe that for keeping disc semantics of the 
primary and to allow the primary to run speculatively all disc state 
changes are directly written to the disk. In parrallel and 
asynchronously send to the secondary. The secondary keeps the pending 
writing requests in two disk buffers. A speculation-disk-buffer and a 
write-out-buffer.

After the reception of the next checkpoint the secondary copies the 
speculation buffer to the write out buffer, commits the checkpoint and 
applies the write out buffer to its local disk.

When the primary fails the secondary must wait until write-out-buffer 
has been completely written to disk before before changing the execution 
mode to run as primary. In this case (failure of primary) the secondary 
discards pending disk writes in its speculation buffer. This protocol 
keeps the disc state consistent with the last checkpoint.

Remus uses the XEN specific blktap driver. As far as I know this can’t 
be used with QEMU (KVM).

I must see how drive-mirror can be used for this kind of protocol.


Am 11.08.2014 22:15, schrieb Michael R. Hines:
> Excellent question: QEMU does have a feature called "drive-mirror"
> in block/mirror.c that was introduced a couple of years ago. I'm not
> sure what the
> adoption rate of the feature is, but I would start with that one.
>
> There is also a second fault tolerance implementation that works a
> little differently called
> "COLO" - you may have seen those emails on the list too, but their
> method does not require a disk replication solution, if I recall correctly.

I have taken a look at COLO. They have also published a good paper about 
their approach. The paper is about the XEN implementation of COLO.  It’s 
an interesting approach to use a coarse grained lockspepping combined 
with checkpointing. From what i have understood the possible show 
stopper for general application is the depency of COLO to custom changes 
of the tcp stack to make it more deterministic.

IMHO there are two points. Custom changes of the TCP-Stack are a no-go 
for proprietary operating systems like Windows. It makes COLO 
application agnostic but not operating system agnostic. The other point 
is that with I/O intensive workloads COLO will tend to behave like MC. 
This is my point of view but i didn’t invest much time to understand 
everything in detail.



>
> I know the time pressure that comes during a thesis, though =), so
> there's no pressure to work on it - but that is the most pressing issue
> in the implementation today. (Lack of disk replication in
> micro-checkpointing.)
>
> The MC implementation also needs to be re-based against the latest
> master - I just haven't had a chance to do it yet because some of my
> hardware has been taken away from me the last few months - will
> see if I can find some reasonable hardware soon.
>
> - Michael
>
> On 08/12/2014 01:22 AM, Walid Nouri wrote:
>> Hi,
>> I will do my best to make a contribution :-)
>>
>> Are there alternative ways of replicating local storage other than
>> DRBD that are possibly feasible?
>> Some that are directly build into Qemu?
>>
>> Walid
>>
>> Am 09.08.2014 14:25, schrieb Michael R. Hines:
>>> On Sat, 2014-08-09 at 14:08 +0200, Walid Nouri wrote:
>>>> Hi Michael,
>>>> how is the weather in Bejing? :-)
>>> It's terrible. Lots of pollution =(
>>>
>>>> May I ask you some questions to your MC implementation?
>>>>
>>>> Currently i'm trying  to understand the general working of the MC
>>>> protokoll and possible problems that can occur so that I can discuss it
>>>> in my thesis.
>>>>
>>>> As far as i have understand MC relies on a shared disk. Output of the
>>>> primary vm are directly written, network output is buffered until the
>>>> corresponding checkpoint is acknowledged.
>>>>
>>>> One problem that comes into my mind is: What happens when the
>>>> primary vm
>>>> writes to the disk and crashes before sending a corresponding
>>>> checkpoint?
>>>>
>>> The MC implementation itself is incomplete, today. (I need help).
>>>
>>> The Xen Remus implementation uses the DRBD system to "mirror" all disk
>>> writes to the source and destination before completing each checkpoint.
>>>
>>> The KVM (mc) implementation needs exactly the same support, but it is
>>> missing today.
>>>
>>> Until that happens, we are *required* to use root-over-iSCSI or
>>> root-over-NFS (meaning that the guest filesystem is mounted directly
>>> inside the virtual machine without the host knowing about it.
>>>
>>> This has the effect of translating all disk I/O into network I/O,
>>> and since network I/O is already buffered, then we are safe.
>>>
>>>
>>>> Here an example: The Primary state is in the actual epoch epoch (n),
>>>> secondary state is in epoch (n-1). The primary writes to disk and
>>>> crashes before or while sending the checkpoint n. In this case the
>>>> secondary memory state is still at epoch (n-1) and the state of the
>>>> shared Disk corresponds to the primary state of epoch (n).
>>>>
>>>> How does MC guaranty that the Disk state of the backup vm is consistent
>>>> with its Memory state?
>>> As I mentioned above, we need the equivalent of the Xen solution, but I
>>> just haven't had the time to write it (or incorporate someone else's
>>> implementation). Patch is welcome =)
>>>
>>>> Is Memory-VCPU / Disk State consistency necessary under all
>>>> circumstances?
>>>> Or can this be neglected because the secondary will (after a fail over)
>>>> repeat the same instructions and finally write to disk the same (as the
>>>> primary before) data for a second time?
>>>> Could this lead to fatal inconsistencies?
>>>>
>>>> Walid
>>>>
>>>
>>>
>>>
>>> - Michael
>>>
>>>
>>>
>>
>>
>