Re: slow live magration / xc_restore on xen4 pvops

[Keir Fraser <keir.fraser@eu.citrix.com>]

On 03/06/2010 07:55, "Brendan Cully" <Brendan@cs.ubc.ca> wrote:

>> kernel, min call time, max call time
>> 2.6.18, 4 us, 72 us
>> pvops, 202 us, 10696 us (!)
>> 
>> It looks like pvops is dramatically slower to perform the
>> xc_domain_memory_populate_physmap call!
> 
> Looking at changeset 20841:
> 
>   Allow certain performance-critical hypercall wrappers to register data
>   buffers via a new interface which allows them to be 'bounced' into a
>   pre-mlock'ed page-sized per-thread data area. This saves the cost of
>   mlock/munlock on every such hypercall, which can be very expensive on
>   modern kernels.
> 
> ...maybe the lock_pages call in xc_memory_op (called from
> xc_domain_memory_populate_physmap) has gotten very expensive?
> Especially considering this hypercall is now issued once per page.

Maybe there are two issues here then. I mean, there's slow, and there's 10ms
for a presumably in-core kernel operation, which is rather mad.

Getting our batching back for 4k allocations is the most critical thing
though, of course.

 -- Keir