[Qemu-devel] Migration ToDo list

[Qemu-devel] Migration ToDo list

[Juan Quintela]

Hi

If you have anything else to put, please add.

Migration Thread
* Plan is integrate it as one of first thing in December (me)
* Remove copies with buffered file (me)

Bitmap Optimization
* Finish moving to individual bitmaps for migration/vga/code
* Make sure we don't copy things around
* Shared memory bitmap with kvm?
* Move to 2MB pages bitmap and then fine grain?

QIDL
* Review the patches (me)

PostCopy
* Review patches?
* See what we can already integrate?
  I remember for last year that we could integrate the 1st third or so

RDMA
* Send RDMA/tcp/.... library they already have (Benoit)
* This is required for postcopy
* This can be used for precopy

General
* Change protocol to:
  a) being always 16byte aligned (paolo said that is faster)
  b) do scatter/gather of the pages?

Fault Tolerance
* That is built on top of migration code, but I have nothing to add.

Any more ideas?

Later, Juan.

Re: [Qemu-devel] Migration ToDo list

[Paolo Bonzini]

Il 13/11/2012 17:18, Juan Quintela ha scritto:
> Migration Thread
> * Plan is integrate it as one of first thing in December (me)

Please make sure to take a look at the latest reviews I sent.

> * Remove copies with buffered file (me)

I also have some prototype of this.

> RDMA
> * Send RDMA/tcp/.... library they already have (Benoit)
> * This is required for postcopy
> * This can be used for precopy

* Investigate RDS (Reliable Datagram Socket, which work on top of both
TCP and InfiniBand/RDMA.

> General
> * Change protocol to:
>   a) being always 16byte aligned (paolo said that is faster)

Well, it's faster with the buffers.  Hopefully they go away and we do
not have the problem.

>   b) do scatter/gather of the pages?

c) Remove compression of non-zero repetitive pages.

All of the above, I'd say.

Paolo

Re: [Qemu-devel] Migration ToDo list

[Orit Wasserman]

On 11/13/2012 06:28 PM, Paolo Bonzini wrote:
> Il 13/11/2012 17:18, Juan Quintela ha scritto:
>> Migration Thread
>> * Plan is integrate it as one of first thing in December (me)
> 
> Please make sure to take a look at the latest reviews I sent.
> 
>> * Remove copies with buffered file (me)
> 
> I also have some prototype of this.
> 
>> RDMA
>> * Send RDMA/tcp/.... library they already have (Benoit)
>> * This is required for postcopy
>> * This can be used for precopy
> 
> * Investigate RDS (Reliable Datagram Socket, which work on top of both
> TCP and InfiniBand/RDMA.
> 
>> General
>> * Change protocol to:
>>   a) being always 16byte aligned (paolo said that is faster)
> 
> Well, it's faster with the buffers.  Hopefully they go away and we do
> not have the problem.
> 
>>   b) do scatter/gather of the pages?
I would prefer to postpone changing the protocol and start with using iov (writev)
for sending the pages (still sending header and than the page). Later we can 
move to scatter/gather I'm not sure of how large the performance gain will be.
> 
> c) Remove compression of non-zero repetitive pages.
+1
we can look of identify the zero pages without calling is_dup_page which looks
expensive.

Orit
> 
> All of the above, I'd say.
> 
> Paolo
> 

Re: [Qemu-devel] Migration ToDo list

[Paolo Bonzini]

Il 13/11/2012 18:09, Orit Wasserman ha scritto:
>> > 
>>> >>   b) do scatter/gather of the pages?
> I would prefer to postpone changing the protocol and start with using iov (writev)
> for sending the pages (still sending header and than the page). Later we can 
> move to scatter/gather I'm not sure of how large the performance gain will be.
>> > 
>> > c) Remove compression of non-zero repetitive pages.
> +1
> we can look of identify the zero pages without calling is_dup_page which looks
> expensive.

Identifying ballooned zero pages is useful, because those cause the
clear_page calls in the kernel even in a guest that has been running for
a while.

But a generic solution doesn't really matter, because is_dup_page and
clear_page shouldn't really be in the profile in practice, except in
microbenchmarks.

Paolo

Re: [Qemu-devel] Migration ToDo list

[Isaku Yamahata]

On Tue, Nov 13, 2012 at 06:16:30PM +0100, Paolo Bonzini wrote:
> Il 13/11/2012 18:09, Orit Wasserman ha scritto:
> >> > 
> >>> >>   b) do scatter/gather of the pages?
> > I would prefer to postpone changing the protocol and start with using iov (writev)
> > for sending the pages (still sending header and than the page). Later we can 
> > move to scatter/gather I'm not sure of how large the performance gain will be.
> >> > 
> >> > c) Remove compression of non-zero repetitive pages.
> > +1
> > we can look of identify the zero pages without calling is_dup_page which looks
> > expensive.
> 
> Identifying ballooned zero pages is useful, because those cause the
> clear_page calls in the kernel even in a guest that has been running for
> a while.
> 
> But a generic solution doesn't really matter, because is_dup_page and
> clear_page shouldn't really be in the profile in practice, except in
> microbenchmarks.

I guess mincore(2) can be used as easy way to detect non-mapped page.
This is just implementation detail anyway.
-- 
yamahata

Re: [Qemu-devel] Migration ToDo list

[Paolo Bonzini]

Il 14/11/2012 03:14, Isaku Yamahata ha scritto:
>> > Identifying ballooned zero pages is useful, because those cause the
>> > clear_page calls in the kernel even in a guest that has been running for
>> > a while.
>> > 
>> > But a generic solution doesn't really matter, because is_dup_page and
>> > clear_page shouldn't really be in the profile in practice, except in
>> > microbenchmarks.
> I guess mincore(2) can be used as easy way to detect non-mapped page.
> This is just implementation detail anyway.

Doesn't work if the page is swapped, doesn't it?

But I wonder if the clear_page occurrences are because of the problem
described recently on LWN (http://lwn.net/Articles/517465/), and so
really more of a kernel bug than anything else.

Paolo

Re: [Qemu-devel] Migration ToDo list

[Isaku Yamahata]

On Wed, Nov 14, 2012 at 03:20:08AM +0100, Paolo Bonzini wrote:
> Il 14/11/2012 03:14, Isaku Yamahata ha scritto:
> >> > Identifying ballooned zero pages is useful, because those cause the
> >> > clear_page calls in the kernel even in a guest that has been running for
> >> > a while.
> >> > 
> >> > But a generic solution doesn't really matter, because is_dup_page and
> >> > clear_page shouldn't really be in the profile in practice, except in
> >> > microbenchmarks.
> > I guess mincore(2) can be used as easy way to detect non-mapped page.
> > This is just implementation detail anyway.
> 
> Doesn't work if the page is swapped, doesn't it?

Ah, I meant if page is in memory, we can skip is_dup_page for that page.
Only check pages that is not-mapped or swapped.
If more detailed info is needed, /proc/<pid>/pagemap would be used.


> But I wonder if the clear_page occurrences are because of the problem
> described recently on LWN (http://lwn.net/Articles/517465/), and so
> really more of a kernel bug than anything else.
> 
> Paolo
> 

-- 
yamahata

Re: [Qemu-devel] Migration ToDo list

[Orit Wasserman]

On 11/13/2012 06:18 PM, Juan Quintela wrote:
> 
> Hi
> 
> If you have anything else to put, please add.
> 
> Migration Thread
> * Plan is integrate it as one of first thing in December (me)
> * Remove copies with buffered file (me)
> 
> Bitmap Optimization
> * Finish moving to individual bitmaps for migration/vga/code
> * Make sure we don't copy things around
> * Shared memory bitmap with kvm?
> * Move to 2MB pages bitmap and then fine grain?
> 
> QIDL
> * Review the patches (me)
> 
> PostCopy
> * Review patches?
> * See what we can already integrate?
>   I remember for last year that we could integrate the 1st third or so
> 
> RDMA
> * Send RDMA/tcp/.... library they already have (Benoit)
Use RDS (Reliable Datagram Sockets), which allows us to use the same API
when using tcp or RDMA (me)
> * This is required for postcopy
> * This can be used for precopy
> 
> General
> * Change protocol to:
>   a) being always 16byte aligned (paolo said that is faster)
>   b) do scatter/gather of the pages?
> 
> Fault Tolerance
> * That is built on top of migration code, but I have nothing to add.
> 
> Any more ideas?
copyless networking - maybe use virtio zero copy mechanism? (me)
EPT/NPT dirty bits (will make the sync more expensive but will improve guest performance) 

Regards,
Orit
> 
> Later, Juan.
> 

Re: [Qemu-devel] Migration ToDo list

[Chegu Vinod]

On 11/13/2012 8:18 AM, Juan Quintela wrote:
> Hi
>
> If you have anything else to put, please add.
>
> Migration Thread
> * Plan is integrate it as one of first thing in December (me)
> * Remove copies with buffered file (me)
>
> Bitmap Optimization
> * Finish moving to individual bitmaps for migration/vga/code
> * Make sure we don't copy things around
> * Shared memory bitmap with kvm?
> * Move to 2MB pages bitmap and then fine grain?

If its not already implied in the above ...  the long freezes observed 
at the start of the migration needs to be addressed (its most likely 
related to BQL ?).

>
> QIDL
> * Review the patches (me)
>
> PostCopy
> * Review patches?
> * See what we can already integrate?
>    I remember for last year that we could integrate the 1st third or so
>
> RDMA
> * Send RDMA/tcp/.... library they already have (Benoit)
> * This is required for postcopy
> * This can be used for precopy

Not sure if what Benoit has can be directly used for pre-copy also.

As Paolo said... we need to look at RDS API's for pre-copy.  ('have just 
started looking at the same). Would like to know if SDP can be used...

> General
> * Change protocol to:
>    a) being always 16byte aligned (paolo said that is faster)
>    b) do scatter/gather of the pages?

Control of where the migration thread(s) run...

--

BTW, has anyone tried doing multiple guest migration from a host ? Are 
there limitations  (enforced via higher level management tools) as to 
how many guests can be migrated at once (in an attempt to quickly 
evacuate a flaky host) ?

Vinod

> Fault Tolerance
> * That is built on top of migration code, but I have nothing to add.
>
> Any more ideas?
>
> Later, Juan.
> .
>

Re: [Qemu-devel] Migration ToDo list

[Orit Wasserman]

On 11/13/2012 06:48 PM, Chegu Vinod wrote:
> On 11/13/2012 8:18 AM, Juan Quintela wrote:
>> Hi
>>
>> If you have anything else to put, please add.
>>
>> Migration Thread
>> * Plan is integrate it as one of first thing in December (me)
>> * Remove copies with buffered file (me)
>>
>> Bitmap Optimization
>> * Finish moving to individual bitmaps for migration/vga/code
>> * Make sure we don't copy things around
>> * Shared memory bitmap with kvm?
>> * Move to 2MB pages bitmap and then fine grain?
> 
> If its not already implied in the above ...  the long freezes observed at the start of the migration needs to be addressed (its most likely related to BQL ?).
> 
>>
>> QIDL
>> * Review the patches (me)
>>
>> PostCopy
>> * Review patches?
>> * See what we can already integrate?
>>    I remember for last year that we could integrate the 1st third or so
>>
>> RDMA
>> * Send RDMA/tcp/.... library they already have (Benoit)
>> * This is required for postcopy
>> * This can be used for precopy
> 
> Not sure if what Benoit has can be directly used for pre-copy also.
> 
> As Paolo said... we need to look at RDS API's for pre-copy.  ('have just started looking at the same). Would like to know if SDP can be used...
> 
>> General
>> * Change protocol to:
>>    a) being always 16byte aligned (paolo said that is faster)
>>    b) do scatter/gather of the pages?
> 
> Control of where the migration thread(s) run...
> 
> -- 
> 
> BTW, has anyone tried doing multiple guest migration from a host ? Are there limitations  (enforced via higher level management tools) as to how many guests can be migrated at once (in an attempt to quickly evacuate a flaky host) ?
libvirt has support to concurrent migrations but I didn't try it.
> 
> Vinod
> 
>> Fault Tolerance
>> * That is built on top of migration code, but I have nothing to add.
>>
>> Any more ideas?
>>
>> Later, Juan.
>> .
>>
> 
>