Re: [Qemu-devel] Migration To-do list

[Orit Wasserman <owasserm@redhat.com>]

On 11/14/2012 04:23 AM, Isaku Yamahata wrote:
> On Tue, Nov 13, 2012 at 05:46:13PM +0000, Hudzia, Benoit wrote:
>> Hi,
>>
>> One concept we have been playing around in the context of  and hybrid and post copy and might make sense if you are orienting your effort toward RDMA / Post copy is to move most of the logic in the destination side. 
>>
>> This is one thing you might want to consider as it  can solve some of the issue you currently have and allow you to maintain almost a single API / Protocol once integrating with post copy approach. 
>>
>> The idea is to drive the migration from the destination side. I.e. The page are pulled from the destination and not pushed from the source side. 
>>
>> Ex: current pre-copy :
>>
>> 	*extract dirty bitmap ( dirty bitmap extraction can be scheduled or triggered by destination) 
>> 	* send it to the destination side
>> 	* have the destination iterating over the bitmap ( can do page prioritization here)  
> 
> IIRC last year, you mentioned page prioritization, but didn't this year.
> Is it still supported?
> Where is it implemented? in qemu or kernel?
I did a prototype and couldn't find workload that benefits from it so 
it was moved down in priority. Maybe I will return to it in the future.

Regards,
Orit
> 
> 
>> 	* depending of protocol :
>> 		_  with standard socket ( or RDS) : 
>> 			. Destination : request page(s)<- can be batched
>> 			.  source receive request send back the page
>> 			. destination process  
>> 		_ with RDMA : 
>> 			. Destination Read Page from source to local page ( the page have been mapped to RDMA at the bitmap extraction) ( RDMA support scatter gather) 
> 
> Although I'm not familiar with RDMA, RDMA requires the exchange of DMA-address between
> sender and receiver in advance and pinning down pages.
> It it correct?
> 
> 
>> 		_ with post copy 
>> 			. pretty much the same but the dirty bitmap reset is done in kernel during the post copy operation ( provide a better dirty bit tracking granularity)
>>
>>
>> Disadvantage: 
>> 	* add a round trip that can be compensate with batch operation ( only with standard socket)  
>>
>> Advantage :
>> 	* most of the heavy lifting is done at the destination side leaving the source to respond to request in an event based format  
>> 	* resolve a lot of issue you have with your threading form the sender side ( accounting etc.. )
>> 	* extremely friendly to optimised solution 
>> 	* if the bitmap generation is expensive we can overlap their generation creating a semi continuous delivery of them guaranteeing an uninterrupted and optimised  flow. => we decouple the bitmap generation from the send/ receive operation. 
>>
>>
>>
>> Anyway , I will notify you as soon as I have the patch / library available for RDMA / postcopy. 
>>
>> Note On the fault tolerance part: this require a lot more heavy code optimisation and poking around to guarantee efficient checkpointing. Most of the solution we tested so far ( Remus and an old version of kemari) scale poorly . Again, an RDMA / post copy solution is kind of necessary when you talk about check pointing enterprise class applications. 
> 
> IIRC Kemari guys evaluated IB case. I'm not sure that it was with RDMA or IPoIB.
> 
> thanks,
>>
>>
>> Regards
>> Benoit
>>
>>
>>
>>
>>
>>> -----Original Message-----
>>> From: Juan Quintela [mailto:quintela@redhat.com]
>>> Sent: 13 November 2012 16:19
>>> To: qemu-devel qemu-devel; Orit Wasserman; chegu_vinod@hp.com;
>>> Hudzia, Benoit; Isaku Yamahata; Michael Roth
>>> Subject: Migration ToDo list
>>>
>>>
>>> Hi
>>>
>>> If you have anything else to put, please add.
>>>
>>> Migration Thread
>>> * Plan is integrate it as one of first thing in December (me)
>>> * Remove copies with buffered file (me)
>>>
>>> Bitmap Optimization
>>> * Finish moving to individual bitmaps for migration/vga/code
>>> * Make sure we don't copy things around
>>> * Shared memory bitmap with kvm?
>>> * Move to 2MB pages bitmap and then fine grain?
>>>
>>> QIDL
>>> * Review the patches (me)
>>>
>>> PostCopy
>>> * Review patches?
>>> * See what we can already integrate?
>>>   I remember for last year that we could integrate the 1st third or so
>>>
>>> RDMA
>>> * Send RDMA/tcp/.... library they already have (Benoit)
>>> * This is required for postcopy
>>> * This can be used for precopy
>>>
>>> General
>>> * Change protocol to:
>>>   a) being always 16byte aligned (paolo said that is faster)
>>>   b) do scatter/gather of the pages?
>>>
>>> Fault Tolerance
>>> * That is built on top of migration code, but I have nothing to add.
>>>
>>> Any more ideas?
>>>
>>> Later, Juan.
>>
>