[RFC] provide an API to userspace doing memory snapshot

[RFC] provide an API to userspace doing memory snapshot

[Wenchao Xia]

Hi,
  I'd like to add/export an function which allow userspace program
to take snapshot for a region of memory. Since it is not implemented yet
I will describe it as C APIs, it is quite simple now and if it is worthy
I'll improve the interface later:

Simple prototype:
C API in userspace:
/*
 *   This function will mark a section of memory as COW, and return
 * a new virtual address of it. User space program can dump out the
 * content as a snapshot while other thread continue modify the content
 * in the region.
 *   @addr: the virtual address to be snapshotted.
 *   @length: the length of it.
 *   This function returns a new virtual address which can be used as
 * snapshot. Return NULL on fail.
 */
void *memory_snapshot_create(void *addr, uint64_t length);

/*
 *   This function will free the memory snapshot.
 *   @addr: the virtual snapshot addr to be freed, it should be the
 * returned one in memory_snapshot_create().
 */
void memory_snapshot_delete(void *addr);

In kernel space:
  The pages in those virtual address will be marked as COW. Take a
page with physical addr P0 as example, it will have two virtual addr:
old A0 and new A1. When modified, kernel should create a new page P1
with same contents, and mapping A1 to P1. When NUMA is used, P1 can
be a slower page.
  It is quite like fork(), but only COW part of pages. Maybe add it
as an ioctl() in kvm.ko, and change the input/output as a structure
describe guest memory state.

Why bring it to kernel space:
  Compared with fork():
  1 take less RAM, less halt time by avoid marking unnecessary pages
COW.
  2 take less RAM if API can return a bitmap let qemu consume dirty
page first.
  3 much nicer userspace program model, no need to pipe or memcpy(),
brings better performance.
  4 optimization space in kernel space, since snapshoted pages
can be set into slower memory in NUMA when change comes.

Re: [RFC] provide an API to userspace doing memory snapshot

[Stefan Hajnoczi]

On Mon, Apr 15, 2013 at 09:03:36PM +0800, Wenchao Xia wrote:
>   I'd like to add/export an function which allow userspace program
> to take snapshot for a region of memory. Since it is not implemented yet
> I will describe it as C APIs, it is quite simple now and if it is worthy
> I'll improve the interface later:

We talked about a simple approach using fork(2) on IRC yesterday.

Is this email outdated?

Stefan

Re: [RFC] provide an API to userspace doing memory snapshot

[Wenchao Xia]

于 2013-4-16 13:51, Stefan Hajnoczi 写道:
> On Mon, Apr 15, 2013 at 09:03:36PM +0800, Wenchao Xia wrote:
>>    I'd like to add/export an function which allow userspace program
>> to take snapshot for a region of memory. Since it is not implemented yet
>> I will describe it as C APIs, it is quite simple now and if it is worthy
>> I'll improve the interface later:
>
> We talked about a simple approach using fork(2) on IRC yesterday.
>
> Is this email outdated?
>
> Stefan
>
   No, after the discuss on IRC, I agree that fork() is a simpler
method to do it, which can comes to qemu fast, since user wants it.
   With a more consideration, still I think a KVM's mem snapshot would
be an long term solution for it:
   The source of the problem comes from acceleration module, kvm.ko, when
qemu does not use it, no troubles. This means an acceleration module
missed a function while caller requires. My instinct idea is: when
acceleration module replace a pure software one, it should try provide
all parts or not stop software filling the gap, and doing so brings
benefits, so hope to add it.
   My API description is old, the core is COW pages, maybe redesign if
reasonable.

Re: [RFC] provide an API to userspace doing memory snapshot

[Stefan Hajnoczi]

On Tue, Apr 16, 2013 at 03:54:15PM +0800, Wenchao Xia wrote:
> 于 2013-4-16 13:51, Stefan Hajnoczi 写道:
> >On Mon, Apr 15, 2013 at 09:03:36PM +0800, Wenchao Xia wrote:
> >>   I'd like to add/export an function which allow userspace program
> >>to take snapshot for a region of memory. Since it is not implemented yet
> >>I will describe it as C APIs, it is quite simple now and if it is worthy
> >>I'll improve the interface later:
> >
> >We talked about a simple approach using fork(2) on IRC yesterday.
> >
> >Is this email outdated?
> >
> >Stefan
> >
>   No, after the discuss on IRC, I agree that fork() is a simpler
> method to do it, which can comes to qemu fast, since user wants it.
>   With a more consideration, still I think a KVM's mem snapshot would
> be an long term solution for it:
>   The source of the problem comes from acceleration module, kvm.ko, when
> qemu does not use it, no troubles. This means an acceleration module
> missed a function while caller requires. My instinct idea is: when
> acceleration module replace a pure software one, it should try provide
> all parts or not stop software filling the gap, and doing so brings
> benefits, so hope to add it.
>   My API description is old, the core is COW pages, maybe redesign if
> reasonable.

QEMU is a userspace process that has guest RAM mmapped.  You want to
snapshot that mmap region but there is no Linux system call to do that.
Maybe a new mremap(2) flag is what you want.

But I don't see the connection to kvm.ko which you mention.  The feature
you're wishing for has nothing to do with kvm.ko.

Stefan