* Internal Qemu snapshots with RBD and libvirt @ 2013-07-18 15:21 Wido den Hollander 2013-07-19 22:30 ` Josh Durgin 0 siblings, 1 reply; 6+ messages in thread From: Wido den Hollander @ 2013-07-18 15:21 UTC (permalink / raw) To: ceph-devel Hi, I'm working on the RBD integration for CloudStack 4.2 and now I got to the point snapshotting. The "problem" is that CloudStack uses libvirt for snapshotting Instances, but Qemu/libvirt also tries to store the memory contents of the domain to assure the snapshot is consistent. So the way libvirt tries to do it is not possible with RBD right now, since there is no way to store the internal memory. I was thinking about using the Java librbd bindings to create the snapshot, but that will not be consistent thus not 100% safe, so I'd rather avoid that. How is this done in OpenStack? Or are you facing similar issues? P.S.: I'm testing with libvirt 1.0.6 from the Ubuntu Cloud Team archive with packages for OpenStack Havana. -- Wido den Hollander 42on B.V. Phone: +31 (0)20 700 9902 Skype: contact42on ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Internal Qemu snapshots with RBD and libvirt 2013-07-18 15:21 Internal Qemu snapshots with RBD and libvirt Wido den Hollander @ 2013-07-19 22:30 ` Josh Durgin 2013-07-19 22:41 ` Sage Weil 0 siblings, 1 reply; 6+ messages in thread From: Josh Durgin @ 2013-07-19 22:30 UTC (permalink / raw) To: Wido den Hollander; +Cc: ceph-devel On 07/18/2013 08:21 AM, Wido den Hollander wrote: > Hi, > > I'm working on the RBD integration for CloudStack 4.2 and now I got to > the point snapshotting. > > The "problem" is that CloudStack uses libvirt for snapshotting > Instances, but Qemu/libvirt also tries to store the memory contents of > the domain to assure the snapshot is consistent. > > So the way libvirt tries to do it is not possible with RBD right now, > since there is no way to store the internal memory. > > I was thinking about using the Java librbd bindings to create the > snapshot, but that will not be consistent thus not 100% safe, so I'd > rather avoid that. > > How is this done in OpenStack? Or are you facing similar issues? OpenStack doesn't store the memory contents of a domain. For volume snapshots, it requires that the volume is detached, so there can be no inconsistency, and the actual snapshot handling is done by the volume driver in cinder, so libvirt is not involved at all. It just uses the rbd command (or now the python bindings). > P.S.: I'm testing with libvirt 1.0.6 from the Ubuntu Cloud Team archive > with packages for OpenStack Havana. > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Internal Qemu snapshots with RBD and libvirt 2013-07-19 22:30 ` Josh Durgin @ 2013-07-19 22:41 ` Sage Weil 2013-07-19 22:47 ` Marcus Sorensen 0 siblings, 1 reply; 6+ messages in thread From: Sage Weil @ 2013-07-19 22:41 UTC (permalink / raw) To: Josh Durgin; +Cc: Wido den Hollander, ceph-devel On Fri, 19 Jul 2013, Josh Durgin wrote: > On 07/18/2013 08:21 AM, Wido den Hollander wrote: > > Hi, > > > > I'm working on the RBD integration for CloudStack 4.2 and now I got to > > the point snapshotting. > > > > The "problem" is that CloudStack uses libvirt for snapshotting > > Instances, but Qemu/libvirt also tries to store the memory contents of > > the domain to assure the snapshot is consistent. > > > > So the way libvirt tries to do it is not possible with RBD right now, > > since there is no way to store the internal memory. It seems like the way to view this is that to snapshot a VM, we need to snapshot all N block devices attached to it, plus the internal memory. It's not that there is something missing from the RBD block device snapshot function, but that it is not clear where to put the memory at all. Maybe the libvirt or qemu VM metadata should specify a separate image target for the RAM? How is this normally done when you're using, say, qcow2? It is assumed that it can be somehow stored with the first block device or something? sage > > > > I was thinking about using the Java librbd bindings to create the > > snapshot, but that will not be consistent thus not 100% safe, so I'd > > rather avoid that. > > > > How is this done in OpenStack? Or are you facing similar issues? > > OpenStack doesn't store the memory contents of a domain. For volume > snapshots, it requires that the volume is detached, so there can be > no inconsistency, and the actual snapshot handling is done by the volume > driver in cinder, so libvirt is not involved at all. It just uses the > rbd command (or now the python bindings). > > > P.S.: I'm testing with libvirt 1.0.6 from the Ubuntu Cloud Team archive > > with packages for OpenStack Havana. > > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Internal Qemu snapshots with RBD and libvirt 2013-07-19 22:41 ` Sage Weil @ 2013-07-19 22:47 ` Marcus Sorensen 2013-07-20 0:48 ` Josh Durgin 0 siblings, 1 reply; 6+ messages in thread From: Marcus Sorensen @ 2013-07-19 22:47 UTC (permalink / raw) To: Sage Weil; +Cc: Josh Durgin, Wido den Hollander, ceph-devel Does RBD not honor barriers and do proper sync flushes? Or does this have to do with RBD caching? Just wondering why online snapshots aren't safe. Qcow2 can keep snapshots internally, but qemu is also capable of doing external dumps for other backing stores. I was thinking about this, and it seems like you'd put the memory dump on secondary storage, like a rados gateway or nfs share, so it can be read wherever the VM is restored to. It would require some work in tracking that location, however. On Fri, Jul 19, 2013 at 4:41 PM, Sage Weil <sage@inktank.com> wrote: > On Fri, 19 Jul 2013, Josh Durgin wrote: >> On 07/18/2013 08:21 AM, Wido den Hollander wrote: >> > Hi, >> > >> > I'm working on the RBD integration for CloudStack 4.2 and now I got to >> > the point snapshotting. >> > >> > The "problem" is that CloudStack uses libvirt for snapshotting >> > Instances, but Qemu/libvirt also tries to store the memory contents of >> > the domain to assure the snapshot is consistent. >> > >> > So the way libvirt tries to do it is not possible with RBD right now, >> > since there is no way to store the internal memory. > > It seems like the way to view this is that to snapshot a VM, we need to > snapshot all N block devices attached to it, plus the internal memory. > It's not that there is something missing from the RBD block device > snapshot function, but that it is not clear where to put the memory at > all. > > Maybe the libvirt or qemu VM metadata should specify a separate image > target for the RAM? How is this normally done when you're using, say, > qcow2? It is assumed that it can be somehow stored with the first block > device or something? > > sage > >> > >> > I was thinking about using the Java librbd bindings to create the >> > snapshot, but that will not be consistent thus not 100% safe, so I'd >> > rather avoid that. >> > >> > How is this done in OpenStack? Or are you facing similar issues? >> >> OpenStack doesn't store the memory contents of a domain. For volume >> snapshots, it requires that the volume is detached, so there can be >> no inconsistency, and the actual snapshot handling is done by the volume >> driver in cinder, so libvirt is not involved at all. It just uses the >> rbd command (or now the python bindings). >> >> > P.S.: I'm testing with libvirt 1.0.6 from the Ubuntu Cloud Team archive >> > with packages for OpenStack Havana. >> > >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Internal Qemu snapshots with RBD and libvirt 2013-07-19 22:47 ` Marcus Sorensen @ 2013-07-20 0:48 ` Josh Durgin 2013-07-20 8:42 ` Wido den Hollander 0 siblings, 1 reply; 6+ messages in thread From: Josh Durgin @ 2013-07-20 0:48 UTC (permalink / raw) To: Marcus Sorensen; +Cc: Sage Weil, Wido den Hollander, ceph-devel On 07/19/2013 03:47 PM, Marcus Sorensen wrote: > Does RBD not honor barriers and do proper sync flushes? Or does this > have to do with RBD caching? Just wondering why online snapshots > aren't safe. They're safe at the filesystem level, but I think Wido's after more application level consistency. If the fs journaled the metadata for a file but didn't save the data yet, it'd be nice to be able to restore the complete file. > Qcow2 can keep snapshots internally, but qemu is also capable of doing > external dumps for other backing stores. I was thinking about this, > and it seems like you'd put the memory dump on secondary storage, like > a rados gateway or nfs share, so it can be read wherever the VM is > restored to. It would require some work in tracking that location, > however. This sounds like a good idea to me. Josh > On Fri, Jul 19, 2013 at 4:41 PM, Sage Weil <sage@inktank.com> wrote: >> On Fri, 19 Jul 2013, Josh Durgin wrote: >>> On 07/18/2013 08:21 AM, Wido den Hollander wrote: >>>> Hi, >>>> >>>> I'm working on the RBD integration for CloudStack 4.2 and now I got to >>>> the point snapshotting. >>>> >>>> The "problem" is that CloudStack uses libvirt for snapshotting >>>> Instances, but Qemu/libvirt also tries to store the memory contents of >>>> the domain to assure the snapshot is consistent. >>>> >>>> So the way libvirt tries to do it is not possible with RBD right now, >>>> since there is no way to store the internal memory. >> >> It seems like the way to view this is that to snapshot a VM, we need to >> snapshot all N block devices attached to it, plus the internal memory. >> It's not that there is something missing from the RBD block device >> snapshot function, but that it is not clear where to put the memory at >> all. >> >> Maybe the libvirt or qemu VM metadata should specify a separate image >> target for the RAM? How is this normally done when you're using, say, >> qcow2? It is assumed that it can be somehow stored with the first block >> device or something? >> >> sage >> >>>> >>>> I was thinking about using the Java librbd bindings to create the >>>> snapshot, but that will not be consistent thus not 100% safe, so I'd >>>> rather avoid that. >>>> >>>> How is this done in OpenStack? Or are you facing similar issues? >>> >>> OpenStack doesn't store the memory contents of a domain. For volume >>> snapshots, it requires that the volume is detached, so there can be >>> no inconsistency, and the actual snapshot handling is done by the volume >>> driver in cinder, so libvirt is not involved at all. It just uses the >>> rbd command (or now the python bindings). >>> >>>> P.S.: I'm testing with libvirt 1.0.6 from the Ubuntu Cloud Team archive >>>> with packages for OpenStack Havana. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Internal Qemu snapshots with RBD and libvirt 2013-07-20 0:48 ` Josh Durgin @ 2013-07-20 8:42 ` Wido den Hollander 0 siblings, 0 replies; 6+ messages in thread From: Wido den Hollander @ 2013-07-20 8:42 UTC (permalink / raw) To: Josh Durgin; +Cc: Marcus Sorensen, Sage Weil, ceph-devel On 07/20/2013 02:48 AM, Josh Durgin wrote: > On 07/19/2013 03:47 PM, Marcus Sorensen wrote: >> Does RBD not honor barriers and do proper sync flushes? Or does this >> have to do with RBD caching? Just wondering why online snapshots >> aren't safe. > > They're safe at the filesystem level, but I think Wido's after > more application level consistency. If the fs journaled the metadata > for a file but didn't save the data yet, it'd be nice to be able to > restore the complete file. > Indeed, I'm about application level consistency. I'm now implementing a PoC where I simply snapshot the RBD image while the Instance is running. Since CloudStack uses cache=none and RBD cache isn't enabled either it shouldn't hurt that much. >> Qcow2 can keep snapshots internally, but qemu is also capable of doing >> external dumps for other backing stores. I was thinking about this, >> and it seems like you'd put the memory dump on secondary storage, like >> a rados gateway or nfs share, so it can be read wherever the VM is >> restored to. It would require some work in tracking that location, >> however. > > This sounds like a good idea to me. > I haven't looked into that, but that seems like a libvirt thing other then CloudStack since libvirt is the one talking to Qemu. I however think it should be generic, somebody wanting to snapshot a running RBD guest via libvirt shouldn't have to go through all kinds of trouble to get the Instance snapshotted. But the question then indeed, where to store the memory contents? Create a new RBD image? <orig rbd img name>.memory.<snapshot name> Like that? Wido > Josh > >> On Fri, Jul 19, 2013 at 4:41 PM, Sage Weil <sage@inktank.com> wrote: >>> On Fri, 19 Jul 2013, Josh Durgin wrote: >>>> On 07/18/2013 08:21 AM, Wido den Hollander wrote: >>>>> Hi, >>>>> >>>>> I'm working on the RBD integration for CloudStack 4.2 and now I got to >>>>> the point snapshotting. >>>>> >>>>> The "problem" is that CloudStack uses libvirt for snapshotting >>>>> Instances, but Qemu/libvirt also tries to store the memory contents of >>>>> the domain to assure the snapshot is consistent. >>>>> >>>>> So the way libvirt tries to do it is not possible with RBD right now, >>>>> since there is no way to store the internal memory. >>> >>> It seems like the way to view this is that to snapshot a VM, we need to >>> snapshot all N block devices attached to it, plus the internal memory. >>> It's not that there is something missing from the RBD block device >>> snapshot function, but that it is not clear where to put the memory at >>> all. >>> >>> Maybe the libvirt or qemu VM metadata should specify a separate image >>> target for the RAM? How is this normally done when you're using, say, >>> qcow2? It is assumed that it can be somehow stored with the first block >>> device or something? >>> >>> sage >>> >>>>> >>>>> I was thinking about using the Java librbd bindings to create the >>>>> snapshot, but that will not be consistent thus not 100% safe, so I'd >>>>> rather avoid that. >>>>> >>>>> How is this done in OpenStack? Or are you facing similar issues? >>>> >>>> OpenStack doesn't store the memory contents of a domain. For volume >>>> snapshots, it requires that the volume is detached, so there can be >>>> no inconsistency, and the actual snapshot handling is done by the >>>> volume >>>> driver in cinder, so libvirt is not involved at all. It just uses the >>>> rbd command (or now the python bindings). >>>> >>>>> P.S.: I'm testing with libvirt 1.0.6 from the Ubuntu Cloud Team >>>>> archive >>>>> with packages for OpenStack Havana. > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Wido den Hollander 42on B.V. Phone: +31 (0)20 700 9902 Skype: contact42on ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-07-20 8:42 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-07-18 15:21 Internal Qemu snapshots with RBD and libvirt Wido den Hollander 2013-07-19 22:30 ` Josh Durgin 2013-07-19 22:41 ` Sage Weil 2013-07-19 22:47 ` Marcus Sorensen 2013-07-20 0:48 ` Josh Durgin 2013-07-20 8:42 ` Wido den Hollander
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.