Re: [Qemu-devel] Migration To-do list

[Isaku Yamahata <yamahata@valinux.co.jp>]

On Tue, Nov 13, 2012 at 05:46:13PM +0000, Hudzia, Benoit wrote:
> Hi,
> 
> One concept we have been playing around in the context of  and hybrid and post copy and might make sense if you are orienting your effort toward RDMA / Post copy is to move most of the logic in the destination side. 
> 
> This is one thing you might want to consider as it  can solve some of the issue you currently have and allow you to maintain almost a single API / Protocol once integrating with post copy approach. 
> 
> The idea is to drive the migration from the destination side. I.e. The page are pulled from the destination and not pushed from the source side. 
> 
> Ex: current pre-copy :
> 
> 	*extract dirty bitmap ( dirty bitmap extraction can be scheduled or triggered by destination) 
> 	* send it to the destination side
> 	* have the destination iterating over the bitmap ( can do page prioritization here)  

IIRC last year, you mentioned page prioritization, but didn't this year.
Is it still supported?
Where is it implemented? in qemu or kernel?


> 	* depending of protocol :
> 		_  with standard socket ( or RDS) : 
> 			. Destination : request page(s)<- can be batched
> 			.  source receive request send back the page
> 			. destination process  
> 		_ with RDMA : 
> 			. Destination Read Page from source to local page ( the page have been mapped to RDMA at the bitmap extraction) ( RDMA support scatter gather) 

Although I'm not familiar with RDMA, RDMA requires the exchange of DMA-address between
sender and receiver in advance and pinning down pages.
It it correct?


> 		_ with post copy 
> 			. pretty much the same but the dirty bitmap reset is done in kernel during the post copy operation ( provide a better dirty bit tracking granularity)
> 
> 
> Disadvantage: 
> 	* add a round trip that can be compensate with batch operation ( only with standard socket)  
> 
> Advantage :
> 	* most of the heavy lifting is done at the destination side leaving the source to respond to request in an event based format  
> 	* resolve a lot of issue you have with your threading form the sender side ( accounting etc.. )
> 	* extremely friendly to optimised solution 
> 	* if the bitmap generation is expensive we can overlap their generation creating a semi continuous delivery of them guaranteeing an uninterrupted and optimised  flow. => we decouple the bitmap generation from the send/ receive operation. 
> 
> 
> 
> Anyway , I will notify you as soon as I have the patch / library available for RDMA / postcopy. 
> 
> Note On the fault tolerance part: this require a lot more heavy code optimisation and poking around to guarantee efficient checkpointing. Most of the solution we tested so far ( Remus and an old version of kemari) scale poorly . Again, an RDMA / post copy solution is kind of necessary when you talk about check pointing enterprise class applications. 

IIRC Kemari guys evaluated IB case. I'm not sure that it was with RDMA or IPoIB.

thanks,
> 
> 
> Regards
> Benoit
> 
> 
> 
> 
> 
> > -----Original Message-----
> > From: Juan Quintela [mailto:quintela@redhat.com]
> > Sent: 13 November 2012 16:19
> > To: qemu-devel qemu-devel; Orit Wasserman; chegu_vinod@hp.com;
> > Hudzia, Benoit; Isaku Yamahata; Michael Roth
> > Subject: Migration ToDo list
> > 
> > 
> > Hi
> > 
> > If you have anything else to put, please add.
> > 
> > Migration Thread
> > * Plan is integrate it as one of first thing in December (me)
> > * Remove copies with buffered file (me)
> > 
> > Bitmap Optimization
> > * Finish moving to individual bitmaps for migration/vga/code
> > * Make sure we don't copy things around
> > * Shared memory bitmap with kvm?
> > * Move to 2MB pages bitmap and then fine grain?
> > 
> > QIDL
> > * Review the patches (me)
> > 
> > PostCopy
> > * Review patches?
> > * See what we can already integrate?
> >   I remember for last year that we could integrate the 1st third or so
> > 
> > RDMA
> > * Send RDMA/tcp/.... library they already have (Benoit)
> > * This is required for postcopy
> > * This can be used for precopy
> > 
> > General
> > * Change protocol to:
> >   a) being always 16byte aligned (paolo said that is faster)
> >   b) do scatter/gather of the pages?
> > 
> > Fault Tolerance
> > * That is built on top of migration code, but I have nothing to add.
> > 
> > Any more ideas?
> > 
> > Later, Juan.
> 

-- 
yamahata