Re: [PATCH 09/10] Exit loop if we have been there too long

[Avi Kivity <avi@redhat.com>]

On 11/30/2010 04:17 PM, Anthony Liguori wrote:
>> What's the problem with burning that cpu?  per guest page, 
>> compressing takes less than sending.  Is it just an issue of qemu 
>> mutex hold time?
>
>
> If you have a 512GB guest, then you have a 16MB dirty bitmap which 
> ends up being an 128MB dirty bitmap in QEMU because we represent dirty 
> bits with 8 bits.

Was there not a patchset to split each bit into its own bitmap?  And 
then copy the kvm or qemu master bitmap into each client bitmap as it 
became needed?

> Walking 16mb (or 128mb) of memory just fine find a few pages to send 
> over the wire is a big waste of CPU time.  If kvm.ko used a 
> multi-level table to represent dirty info, we could walk the memory 
> mapping at 2MB chunks allowing us to skip a large amount of the 
> comparisons.

There's no reason to assume dirty pages would be clustered.  If 0.2% of 
memory were dirty, but scattered uniformly, there would be no win from 
the two-level bitmap.  A loss, in fact: 2MB can be represented as 512 
bits or 64 bytes, just one cache line.  Any two-level thing will need more.

We might have a more compact encoding for sparse bitmaps, like 
run-length encoding.

>
>>> In the short term, fixing (2) by accounting zero pages as full sized 
>>> pages should "fix" the problem.
>>>
>>> In the long term, we need a new dirty bit interface from kvm.ko that 
>>> uses a multi-level table.  That should dramatically improve scan 
>>> performance. 
>>
>> Why would a multi-level table help?  (or rather, please explain what 
>> you mean by a multi-level table).
>>
>> Something we could do is divide memory into more slots, and polling 
>> each slot when we start to scan its page range.  That reduces the 
>> time between sampling a page's dirtiness and sending it off, and 
>> reduces the latency incurred by the sampling.  There are also 
>> non-interface-changing ways to reduce this latency, like O(1) write 
>> protection, or using dirty bits instead of write protection when 
>> available.
>
> BTW, we should also refactor qemu to use the kvm dirty bitmap directly 
> instead of mapping it to the main dirty bitmap.

That's what the patch set I was alluding to did.  Or maybe I imagined 
the whole thing.

>>> We also need to implement live migration in a separate thread that 
>>> doesn't carry qemu_mutex while it runs.
>>
>> IMO that's the biggest hit currently.
>
> Yup.  That's the Correct solution to the problem.

Then let's just Do it.

-- 
error compiling committee.c: too many arguments to function