[Qemu-devel] feature proposal: checkpoint-assisted migration

[Qemu-devel] feature proposal: checkpoint-assisted migration

[Thomas Knauth]

Dear list,

my research revolves around cloud computing, virtual machines and
migration. In this context I came across the following: a recent study
by IBM indicates that a typical VM only migrates between a small set
of physical servers; often just two.

The potential for optimization is clear. By storing a snapshot of the
VM's memory on the migration source, we can reuse (some) of this
information on a subsequent incoming migration.

In the course of our research we implemented a prototype of this
feature within kvm/qemu. We would like to contribute it to mainline,
but it needs cleanup and proper testing. As is the nature with
research prototypes, the code is ugly and not well integrated with the
existing kvm/qemu codebase. To avoid confusion and irritation, I want
to mention that I have little experience in contributing to large
open-source projects. So if I violate some unwritten protocol or best
practises, please be patient.

Initially, I'm hoping to get some feedback on the current state of the
implementation. It would be immensely helpful if someone more
intimately familiar with the migration code/framework could comment on
the prototyp's current state. The code very likely needs restructuring
to make it fit better into the overall codebase. Getting information
on what needs to change and how to change it would be my goal.

The prototype also touches the migration protocol. Changes in this
part probably need discussion. The basic idea is that if a block of
memory (e.g., a 4 KiB page) already exists at the migration
destination, than the source only sends a checksum of the block
(currently MD5). The destination uses the checksum to find the
corresponding block, e.g., by reading it from local storage (instead
of transferring it over the network). This definitely reduces the
migration traffic and usually also the overall migration time.

We currently use MD5 checksums to identify (un)modified blocks. For
strict ping-pong migration, where a VM only migrates between two
servers, there is also the possibility to use dirty page tracking to
identify modified pages. This has not been implemented so far. We are
also unclear about the potential performance tradeoffs this might
entail and how it would interact with the dirty page tracking code
during a live migration.

Our research also includes a look at real world data to motivate that
this optimization actually does make sense in practise. If you are
interested, you can find a draft of the relevant paper at:

https://www.dropbox.com/s/v7qzim8exmji6j5/paper.pdf?dl=0

Keep in mind that the paper is not published yet and, hence, work in progress.

As you can see, there are many open/unanswered questions, but I'm
hopeful that this feature will eventually become part of kvm/qemu such
that everyone can benefit from it.

Please find the current code at
https://bitbucket.org/tknauth/vecycle-qemu/branch/checkpoint-assisted-migration

Looking forward to your feedback,
Thomas.

Re: [Qemu-devel] feature proposal: checkpoint-assisted migration

[Dr. David Alan Gilbert]

* Thomas Knauth (thomas.knauth@googlemail.com) wrote:
> Dear list,
> 
> my research revolves around cloud computing, virtual machines and
> migration. In this context I came across the following: a recent study
> by IBM indicates that a typical VM only migrates between a small set
> of physical servers; often just two.
> 
> The potential for optimization is clear. By storing a snapshot of the
> VM's memory on the migration source, we can reuse (some) of this
> information on a subsequent incoming migration.
> 
> In the course of our research we implemented a prototype of this
> feature within kvm/qemu. We would like to contribute it to mainline,
> but it needs cleanup and proper testing. As is the nature with
> research prototypes, the code is ugly and not well integrated with the
> existing kvm/qemu codebase. To avoid confusion and irritation, I want
> to mention that I have little experience in contributing to large
> open-source projects. So if I violate some unwritten protocol or best
> practises, please be patient.
> 
> Initially, I'm hoping to get some feedback on the current state of the
> implementation. It would be immensely helpful if someone more
> intimately familiar with the migration code/framework could comment on
> the prototyp's current state. The code very likely needs restructuring
> to make it fit better into the overall codebase. Getting information
> on what needs to change and how to change it would be my goal.
> 
> The prototype also touches the migration protocol. Changes in this
> part probably need discussion. The basic idea is that if a block of
> memory (e.g., a 4 KiB page) already exists at the migration
> destination, than the source only sends a checksum of the block
> (currently MD5). The destination uses the checksum to find the
> corresponding block, e.g., by reading it from local storage (instead
> of transferring it over the network). This definitely reduces the
> migration traffic and usually also the overall migration time.
> 
> We currently use MD5 checksums to identify (un)modified blocks. For
> strict ping-pong migration, where a VM only migrates between two
> servers, there is also the possibility to use dirty page tracking to
> identify modified pages. This has not been implemented so far. We are
> also unclear about the potential performance tradeoffs this might
> entail and how it would interact with the dirty page tracking code
> during a live migration.

I like your basic idea, and I kind of agreed with your argument that
if it's good enough for rsync then it's good enough; however, then I
found:
  https://github.com/therealmik/rsync-collision

which complicates the argument!  Those are 700byte blocks, so I guess
the chance of a collision on a 4kB page must be less likely; but I'd
want a crypto guy to say what was actually safeish.

> Our research also includes a look at real world data to motivate that
> this optimization actually does make sense in practise. If you are
> interested, you can find a draft of the relevant paper at:
> 
> https://www.dropbox.com/s/v7qzim8exmji6j5/paper.pdf?dl=0
> 
> Keep in mind that the paper is not published yet and, hence, work in progress.
> 
> As you can see, there are many open/unanswered questions, but I'm
> hopeful that this feature will eventually become part of kvm/qemu such
> that everyone can benefit from it.
> 
> Please find the current code at
> https://bitbucket.org/tknauth/vecycle-qemu/branch/checkpoint-assisted-migration

That asks me to login just to read it; which is a very odd thing
to have.    I suggest you post your code to the list, with basically
this message at the start of the series, saying it's very new and you
expect it needs lots of changes, that way people can more easily
look at it.

Dave

> 
> Looking forward to your feedback,
> Thomas.
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] feature proposal: checkpoint-assisted migration

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]

On Tue, Apr 14, 2015 at 01:20:33PM +0200, Thomas Knauth wrote:
> In the course of our research we implemented a prototype of this
> feature within kvm/qemu. We would like to contribute it to mainline,
> but it needs cleanup and proper testing. As is the nature with
> research prototypes, the code is ugly and not well integrated with the
> existing kvm/qemu codebase. To avoid confusion and irritation, I want
> to mention that I have little experience in contributing to large
> open-source projects. So if I violate some unwritten protocol or best
> practises, please be patient.

Guidelines for contributing patches are here:
http://qemu-project.org/Contribute/SubmitAPatch

Regarding the idea, it sounds like it could be beneficial for some
migration use cases.  I've thought about an rsync approach for disk
images, and that is similar to your idea for RAM.  There are image file
formats that keep generation counts for regions of the disk image,
making it quick to find out which regions have changed between two
images.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]