[Qemu-devel] Some question about savem/qcow2 incremental snapshot

[Qemu-devel] Some question about savem/qcow2 incremental snapshot

[He Junyan]

hi all:

I am now focusing on snapshot optimization for Intel NVDimm kind
memory. Different from the normal memory, the NVDimm may be 128G, 256G
or even more for just one guest, and its speed is slower than the
normal memory. So sometimes it may take several minutes to complete
just one snapshot saving. Even with compression enabled, the snapshot
point may consume more than 30G disk space. 
We decide to add incremental kind snapshot saving to resolve this. Just
store difference between snapshot points to save time and disk space.
But the current snapshot/save_vm framework seems not to support this.
We need to add snapshot dependency and extra operations when we LOAD
and DELETE the snapshot point.
Is that possible to modify the savevm framework and add some
incremental snapshot support to QCOW2 format?

Thanks

[Qemu-devel] Some question about savem/qcow2 incremental snapshot

[He, Junyan]

hi all:

I am now focusing on snapshot optimization for Intel NVDimm kind memory. Different from the normal memory, the NVDimm may be 128G, 256G or even more for just one guest, and its speed is slower than the normal memory. So sometimes it may take several minutes to complete just one snapshot saving. Even with compression enabled, the snapshot point may consume more than 30G disk space. We decide to add incremental kind snapshot saving to resolve this. Just store difference between snapshot points to save time and disk space.
But the current snapshot/save_vm framework seems not to support this.
We need to add snapshot dependency and extra operations when we LOAD and DELETE the snapshot point.
Is that possible to modify the savevm framework and add some incremental snapshot support to QCOW2 format?

Thanks

Re: [Qemu-devel] Some question about savem/qcow2 incremental snapshot

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1748 bytes --]

On Mon, Dec 25, 2017 at 07:54:00AM +0000, He, Junyan wrote:
> I am now focusing on snapshot optimization for Intel NVDimm kind memory. Different from the normal memory, the NVDimm may be 128G, 256G or even more for just one guest, and its speed is slower than the normal memory. So sometimes it may take several minutes to complete just one snapshot saving. Even with compression enabled, the snapshot point may consume more than 30G disk space. We decide to add incremental kind snapshot saving to resolve this. Just store difference between snapshot points to save time and disk space.
> But the current snapshot/save_vm framework seems not to support this.
> We need to add snapshot dependency and extra operations when we LOAD and DELETE the snapshot point.
> Is that possible to modify the savevm framework and add some incremental snapshot support to QCOW2 format?

It sounds like you'd like to save incremental guest RAM snapshots based
on the contents of earlier snapshots.  QEMU has no way of doing this at
the moment.

In order to do this QEMU needs to keep dirty memory logging enabled at
all times so it knows which pages have been written since the last
snapshot.  This will affect guest performance.

Certain guest operations like rebooting or zeroing memory will defeat
the incremental guest RAM snapshot feature.  It's worth thinking about
these cases to make sure this feature would be worth it in real use
cases.

What you're proposing isn't specific to NVDIMM.  Regular RAM use cases
also benefit from incremental snapshots because less disk space is
required for the savevm data.

Some questions about your use case:

1. Does the guest have -device nvdimm?

2. Is the qcow2 file on NVDIMM?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Some question about savem/qcow2 incremental snapshot

[Dr. David Alan Gilbert]

* Stefan Hajnoczi (stefanha@gmail.com) wrote:
> On Mon, Dec 25, 2017 at 07:54:00AM +0000, He, Junyan wrote:
> > I am now focusing on snapshot optimization for Intel NVDimm kind memory. Different from the normal memory, the NVDimm may be 128G, 256G or even more for just one guest, and its speed is slower than the normal memory. So sometimes it may take several minutes to complete just one snapshot saving. Even with compression enabled, the snapshot point may consume more than 30G disk space. We decide to add incremental kind snapshot saving to resolve this. Just store difference between snapshot points to save time and disk space.
> > But the current snapshot/save_vm framework seems not to support this.
> > We need to add snapshot dependency and extra operations when we LOAD and DELETE the snapshot point.
> > Is that possible to modify the savevm framework and add some incremental snapshot support to QCOW2 format?
> 
> It sounds like you'd like to save incremental guest RAM snapshots based
> on the contents of earlier snapshots.  QEMU has no way of doing this at
> the moment.
> 
> In order to do this QEMU needs to keep dirty memory logging enabled at
> all times so it knows which pages have been written since the last
> snapshot.  This will affect guest performance.

Keeping the dirty logging going is something that COLO does, to send
incremental snapshots to the secondary.   As you say, it does slow
things down and you have to be able to keep the state between the
migration runs.

> Certain guest operations like rebooting or zeroing memory will defeat
> the incremental guest RAM snapshot feature.  It's worth thinking about
> these cases to make sure this feature would be worth it in real use
> cases.

But those probably wouldn't upset an NVDimm?

> What you're proposing isn't specific to NVDIMM.  Regular RAM use cases
> also benefit from incremental snapshots because less disk space is
> required for the savevm data.
> 
> Some questions about your use case:
> 
> 1. Does the guest have -device nvdimm?
> 
> 2. Is the qcow2 file on NVDIMM?
> 
> Stefan

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] Some question about savem/qcow2 incremental snapshot

[Stefan Hajnoczi]

On Tue, Jan 9, 2018 at 7:55 PM, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
>> Certain guest operations like rebooting or zeroing memory will defeat
>> the incremental guest RAM snapshot feature.  It's worth thinking about
>> these cases to make sure this feature would be worth it in real use
>> cases.
>
> But those probably wouldn't upset an NVDimm?

If the guest dirties all RAM then the incremental snapshot feature
degrades to a full snapshot.  I'm asking if there are common
operations where that happens.

I seem to remember Windows guests zero all pages on cold boot.  Maybe
that's not the case anymore.

Worth checking before embarking on this feature because it could be a
waste of effort if it turns out real-world guests dirty all memory in
common cases.

Re: [Qemu-devel] Some question about savem/qcow2 incremental snapshot

[Dr. David Alan Gilbert]

* Stefan Hajnoczi (stefanha@gmail.com) wrote:
> On Tue, Jan 9, 2018 at 7:55 PM, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> >> Certain guest operations like rebooting or zeroing memory will defeat
> >> the incremental guest RAM snapshot feature.  It's worth thinking about
> >> these cases to make sure this feature would be worth it in real use
> >> cases.
> >
> > But those probably wouldn't upset an NVDimm?
> 
> If the guest dirties all RAM then the incremental snapshot feature
> degrades to a full snapshot.  I'm asking if there are common
> operations where that happens.
> 
> I seem to remember Windows guests zero all pages on cold boot.  Maybe
> that's not the case anymore.
> 
> Worth checking before embarking on this feature because it could be a
> waste of effort if it turns out real-world guests dirty all memory in
> common cases.

Right, but I'm hoping that there's some magic somewhere where an NVDimm doesn't
get zero'd because of a cold boot since that would seem to make it
volatile.

Dave

--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] Some question about savem/qcow2 incremental snapshot

[Stefan Hajnoczi]

On Wed, Jan 10, 2018 at 8:15 PM, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
> * Stefan Hajnoczi (stefanha@gmail.com) wrote:
>> On Tue, Jan 9, 2018 at 7:55 PM, Dr. David Alan Gilbert
>> <dgilbert@redhat.com> wrote:
>> >> Certain guest operations like rebooting or zeroing memory will defeat
>> >> the incremental guest RAM snapshot feature.  It's worth thinking about
>> >> these cases to make sure this feature would be worth it in real use
>> >> cases.
>> >
>> > But those probably wouldn't upset an NVDimm?
>>
>> If the guest dirties all RAM then the incremental snapshot feature
>> degrades to a full snapshot.  I'm asking if there are common
>> operations where that happens.
>>
>> I seem to remember Windows guests zero all pages on cold boot.  Maybe
>> that's not the case anymore.
>>
>> Worth checking before embarking on this feature because it could be a
>> waste of effort if it turns out real-world guests dirty all memory in
>> common cases.
>
> Right, but I'm hoping that there's some magic somewhere where an NVDimm doesn't
> get zero'd because of a cold boot since that would seem to make it
> volatile.

This feature isn't specific to NVDIMM though.  It would be equally
useful for regular RAM.

Stefan

Re: [Qemu-devel] Some question about savem/qcow2 incremental snapshot

[He Junyan]

On 三, 2018-01-10 at 20:17 +0000, Stefan Hajnoczi wrote:
> On Wed, Jan 10, 2018 at 8:15 PM, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> > 
> > * Stefan Hajnoczi (stefanha@gmail.com) wrote:
> > > 
> > > On Tue, Jan 9, 2018 at 7:55 PM, Dr. David Alan Gilbert
> > > <dgilbert@redhat.com> wrote:
> > > > 
> > > > > 
> > > > > Certain guest operations like rebooting or zeroing memory
> > > > > will defeat
> > > > > the incremental guest RAM snapshot feature.  It's worth
> > > > > thinking about
> > > > > these cases to make sure this feature would be worth it in
> > > > > real use
> > > > > cases.
> > > > But those probably wouldn't upset an NVDimm?
> > > If the guest dirties all RAM then the incremental snapshot
> > > feature
> > > degrades to a full snapshot.  I'm asking if there are common
> > > operations where that happens.
> > > 
> > > I seem to remember Windows guests zero all pages on cold
> > > boot.  Maybe
> > > that's not the case anymore.
> > > 
> > > Worth checking before embarking on this feature because it could
> > > be a
> > > waste of effort if it turns out real-world guests dirty all
> > > memory in
> > > common cases.
> > Right, but I'm hoping that there's some magic somewhere where an
> > NVDimm doesn't
> > get zero'd because of a cold boot since that would seem to make it
> > volatile.
> This feature isn't specific to NVDIMM though.  It would be equally
> useful for regular RAM.
> 
> Stefan
> 

Thanks for all your advices.
I already did a lot of investigation and write some code. My
consideration is as following:
1. As the first step, I will use is_active() function to make it just
work for nvdimm kind memory region and just for snapshot saving, not
for live migration. I understand that this can work for all kinds of
memory, but it may low guest's performance if we always enable dirty
log tracking for memory. For the live migration, all the data need to
be copied and it seems can not get benefit by this manner.
2. Saving and Loading is relatively easy to do, while deleting some
snapshot point needs a lot of work to do. The current framework just
deletes all the data of one snapshot point by one shot. I want to add
some reference to L1/L2 table of QCOW2 when the cluster's data is
depended by other snapshot point, so we can keep the data when
deleting.

I will also check whether the cold boot will zero all pages later.
 
Thanks,
Junyan

Re: [Qemu-devel] Some question about savem/qcow2 incremental snapshot

[Eric Blake]

On 12/25/2017 01:33 AM, He Junyan wrote:
> hi all:
> 
> I am now focusing on snapshot optimization for Intel NVDimm kind
> memory. Different from the normal memory, the NVDimm may be 128G, 256G
> or even more for just one guest, and its speed is slower than the
> normal memory. So sometimes it may take several minutes to complete
> just one snapshot saving. Even with compression enabled, the snapshot
> point may consume more than 30G disk space.
> We decide to add incremental kind snapshot saving to resolve this. Just
> store difference between snapshot points to save time and disk space.
> But the current snapshot/save_vm framework seems not to support this.
> We need to add snapshot dependency and extra operations when we LOAD
> and DELETE the snapshot point.
> Is that possible to modify the savevm framework and add some
> incremental snapshot support to QCOW2 format?

In general, the list has tended to focus on external snapshots rather 
than internal; where persistent bitmaps have been the proposed mechanism 
for tracking incremental differences between snapshots.  But yes, it is 
certainly feasible that patches to improve internal snapshots to take 
advantage of incremental relationships may prove useful.  You will need 
to document all enhancements to the qcow2 file format and get that 
approved first, as interoperability demands that others reading the same 
spec would be able to interpret the image you create that is utilizing 
an internal snapshot with an incremental diff.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org