virtio assisted migration

virtio assisted migration

[Dave Young]

Hi,

I worked on virtio assisted kvm guest migration five years ago in school.
Later I moved to other areas so I have no time to continue on it, I would
like to bring up the idea here to see if anyone is interested in it.
If you think it is worth feel free to continue on it, if not please just ignore
the email.

The code is based on qemu-kvm 0.13.0, kernel version should be 2.6.37 or 2.6.38
The code is very hacky and ugly, but the basic idea is clear I will clarify it.
http://people.redhat.com/ruyang/kvm/

The patches is trying to improve migration performance in both memory and block
layer. The obvious bad side is it will cause problem when there's large memory
or block usage while migration on-going. But other than that it will speedup
the migration significantly.

* memory
For memory the problem is original migration copys all the memory chunks to
new target, it does not consider the working set, caches, free memory etc.
Actually we can avoid caches and free memory by reserve them while migrating
begin, inform host side about reserved pages bitmap, host will only copy the
pages which is not reserved in guest. for memory we have reserved, release them
after migration being done on target machine immediately.

create a virtio driver based on virtio-balloon to reserve free memory, because
there could be more memory allocations during migration progress so in the
virtio driver I keep some free memory to avoid oom.

* block device
For block storage migration, the problem is similar as memory. The original
migration does not consider storage usage ratio it just copy all the sectors.

I handled it in two places:
- before migration starting I run some routines to punk hole and free
unused block storage
- introduce some virtio functionality in filesystem, I did some hack in ext4
filesystem. Just like the virtio mem driver, while migrating start filesystem
driver reserve most of the free blocks as being used, then pass the reserved
bitmap to host side, the migration process will skip those reserved blocks.
After migration finished filesystem driver will free the reserved blocks. 

Thanks
Dave

Re: virtio assisted migration

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

On Sat, Oct 24, 2015 at 05:21:01PM +0800, Dave Young wrote:
> * block device
> For block storage migration, the problem is similar as memory. The original
> migration does not consider storage usage ratio it just copy all the sectors.

This is equivalent to issuing discard requests.  File systems can
already do that.

The block/mirror.c code checks if sectors are allocated so this should
be possible to achieve this today (with guest cooperation).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: virtio assisted migration

[Dave Young]

On 10/28/15 at 10:27am, Stefan Hajnoczi wrote:
> On Sat, Oct 24, 2015 at 05:21:01PM +0800, Dave Young wrote:
> > * block device
> > For block storage migration, the problem is similar as memory. The original
> > migration does not consider storage usage ratio it just copy all the sectors.
> 
> This is equivalent to issuing discard requests.  File systems can
> already do that.
> 
> The block/mirror.c code checks if sectors are allocated so this should
> be possible to achieve this today (with guest cooperation).

Yes, I did using hole punching in my code, I remember I did it on top of some
old thread from Christoph Hellwig about puncing hole, not sure if they have been
merged in qemu upstream.

I lost my original patch thus I just put tarballs on the website, but seems I
successfully generated the diff with official qemu-kvm-0.13.0.

I have just put the hack patch of qemu to the url:
http://people.redhat.com/ruyang/kvm/qemu-kvm-patches/qemu-kvm-0.13.0-block-migration-sparse.patch
Another two patches for virtio qemu host side to work stay same directory as
 well.

Thanks
Dave