Re: Internal Qemu snapshots with RBD and libvirt

[Wido den Hollander <wido@42on.com>]

On 07/20/2013 02:48 AM, Josh Durgin wrote:
> On 07/19/2013 03:47 PM, Marcus Sorensen wrote:
>> Does RBD not honor barriers and do proper sync flushes? Or does this
>> have to do with RBD caching? Just wondering why online snapshots
>> aren't safe.
>
> They're safe at the filesystem level, but I think Wido's after
> more application level consistency. If the fs journaled the metadata
> for a file but didn't save the data yet, it'd be nice to be able to
> restore the complete file.
>

Indeed, I'm about application level consistency. I'm now implementing a 
PoC where I simply snapshot the RBD image while the Instance is running.

Since CloudStack uses cache=none and RBD cache isn't enabled either it 
shouldn't hurt that much.

>> Qcow2 can keep snapshots internally, but qemu is also capable of doing
>> external dumps for other backing stores. I was thinking about this,
>> and it seems like you'd put the memory dump on secondary storage, like
>> a rados gateway or nfs share, so it can be read wherever the VM is
>> restored to. It would require some work in tracking that location,
>> however.
>
> This sounds like a good idea to me.
>

I haven't looked into that, but that seems like a libvirt thing other 
then CloudStack since libvirt is the one talking to Qemu.

I however think it should be generic, somebody wanting to snapshot a 
running RBD guest via libvirt shouldn't have to go through all kinds of 
trouble to get the Instance snapshotted.

But the question then indeed, where to store the memory contents? Create 
a new RBD image?

<orig rbd img name>.memory.<snapshot name>

Like that?

Wido

> Josh
>
>> On Fri, Jul 19, 2013 at 4:41 PM, Sage Weil <sage@inktank.com> wrote:
>>> On Fri, 19 Jul 2013, Josh Durgin wrote:
>>>> On 07/18/2013 08:21 AM, Wido den Hollander wrote:
>>>>> Hi,
>>>>>
>>>>> I'm working on the RBD integration for CloudStack 4.2 and now I got to
>>>>> the point snapshotting.
>>>>>
>>>>> The "problem" is that CloudStack uses libvirt for snapshotting
>>>>> Instances, but Qemu/libvirt also tries to store the memory contents of
>>>>> the domain to assure the snapshot is consistent.
>>>>>
>>>>> So the way libvirt tries to do it is not possible with RBD right now,
>>>>> since there is no way to store the internal memory.
>>>
>>> It seems like the way to view this is that to snapshot a VM, we need to
>>> snapshot all N block devices attached to it, plus the internal memory.
>>> It's not that there is something missing from the RBD block device
>>> snapshot function, but that it is not clear where to put the memory at
>>> all.
>>>
>>> Maybe the libvirt or qemu VM metadata should specify a separate image
>>> target for the RAM?  How is this normally done when you're using, say,
>>> qcow2?  It is assumed that it can be somehow stored with the first block
>>> device or something?
>>>
>>> sage
>>>
>>>>>
>>>>> I was thinking about using the Java librbd bindings to create the
>>>>> snapshot, but that will not be consistent thus not 100% safe, so I'd
>>>>> rather avoid that.
>>>>>
>>>>> How is this done in OpenStack? Or are you facing similar issues?
>>>>
>>>> OpenStack doesn't store the memory contents of a domain. For volume
>>>> snapshots, it requires that the volume is detached, so there can be
>>>> no inconsistency, and the actual snapshot handling is done by the
>>>> volume
>>>> driver in cinder, so libvirt is not involved at all. It just uses the
>>>> rbd command (or now the python bindings).
>>>>
>>>>> P.S.: I'm testing with libvirt 1.0.6 from the Ubuntu Cloud Team
>>>>> archive
>>>>> with packages for OpenStack Havana.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on