Re: [Qemu-devel] [RFC 00/13] Multiple fd migration support

[Juan Quintela <quintela@redhat.com>]

"Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
> * Juan Quintela (quintela@redhat.com) wrote:
>> Hi
>> 
>> This patch series is "an" initial implementation of multiple fd migration.
>> This is to get something out for others to comment, it is not finished at all.
>
> I've had a quick skim:
>   a) I think mst is right about the risk of getting stale pages out of order.

I have been thinking about this.  We just need to send a "we have
finish" this round packet.  And reception has to wait for all threads to
finish before continue.  It is easier and not expensive.  We never
resend the same page during the same round. 

>   b) Since you don't change the URI at all, it's a bit restricted; for example,
>      it means I can't run separate sessions over different NICs unless I've
>      done something clever at the routing/or bonded them.
>      One thing I liked the sound of multi-fd for is NUMA; get a BIG box
>      and give each numa node a separate NIC and run a separate thread on each
>      node.

If we want this _how_ we want to configure it.  This was part of the
reason to post the patch.  It works only for tcp, I don't even try the
others, just to see what people want.


>   c) Hmm we do still have a single thread doing all the bitmap syncing and scanning,
>      we'll have to watch out if that is the bottleneck at all.

Yeap.  My idea here was to still maintain the bitmap scanning on the
main thread, but send work to the "worker threads" in batches, not in
single pages.  But I haven't really profiled how long we spend there.

>   d) All the zero testing is still done in the main thread which we know is
>      expensive.

Not trivial if we don't want to send control information over the
"other" channels.  One solution would be split the main memory in
different "main" threads.  No performance profiles.

>   e) Do we need to do something for security with having multiple ports? How
>      do we check that nothing snuck in on one of our extra ports, have we got
>      sanity checks to make sure it's actually the right stream.


We only have a single port.  We opened it several times.  It shouldn't
require changes in either libvirt/firewall.  (Famous last words)

>   f) You're handing out pages to the sending threads on the basis of which one
>      is free (in the same way as the multi threaded compression); but I think
>      it needs some sanity adding to only hand out whole host pages - it feels
>      like receiving all the chunks of one host page down separate FDs would
>      be horrible.

Trivial optimization would be to send _whole_ huge pages in one go.  I
wanted comments about what people wanted here.  My idea was really to
add multipage or several pages in one go.  Would reduce synchronization
a lot.   I do to the 1st that becomes free because ...... I don't know
how long a specific transmission is going to take.  TCP for you :-(


>   g) I think you might be able to combine the compression into the same threads;
>      so that if multi-fd + multi-threaded-compresison is set you don't end
>      up with 2 sets of threads and it might be the simplest way to make them
>      work together.

Yeap, I thought that.  But I didn't want to merge them in a first
stage.  It makes much more sense to _not_ send the compressed data
through the main channel.  But that would be v2 (or 3, or 4 ...)


>   h) You've used the last free RAM_SAVE_FLAG!  And the person who takes the last
>      slice^Wbit has to get some more.
>      Since arm, ppc, and 68k have variants that have TARGET_PAGE_BITS 10  that
>      means we're full; I suggest what you do is use that flag to mean that we
>      send another 64bit word; and in that word you use the bottom 7 bits for
>      the fd index and bit 7 is set to indicate it's fd.  The other bits are sent
>      as zero and available for the next use.
>      Either that or start combining with some other flags.
>      (I may have a use for some more bits in mind!)

Ok.  I can looke at that.

>   i) Is this safe for xbzrle - what happens to the cache (or is it all
>      still the main thread?)

Nope.  Only way to use xbzrle is:

if (zero(page) {
   ...
} else if (xbzrle(page)) {

} else {
    multifd(page)
}

Otherwise we would have to make xbzrle multithread, or split memory
between fd's.  Problem to split memory between fd's is that we need to
know where the hot spots are.


>   j) For postcopy I could do with a separate fd for the requested pages
>      (but again that comes back to needing an easy solution to the ordering)

The ordering was easy, as said.  You can just use that command with each
postcopy requested page.  Or something similar, no?

I think that just forgetting about that pages, and each time that we
receive a requested page, we first wait for the main thread to finish
its pages should be enough, no?

> Dave

Thanks very much, JUan.