[Qemu-devel] Re: [PATCH] qemu-kvm: response to SIGUSR1 to start/stop a VCPU (v2)

[Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>]

On Wed, Nov 24, 2010 at 04:23:15PM +0200, Avi Kivity wrote:
> >>I'm more concerned about lock holder preemption, and interaction
> >>of this mechanism with any kernel solution for LHP.
> >
> >Can you suggest some scenarios and I'll create some test cases?
> >I'm trying figure out the best way to evaluate this.
> 
> Booting 64-vcpu Windows on a 64-cpu host with PLE but without
> directed yield takes longer than forever because PLE detects
> contention within the guest, which under our current PLE
> implementation (usleep(100)) converts guest contention into delays.

Is there any way of optimizing PLE at runtime in such special case? For ex: 
either turn off PLE feature or gradually increase (spin-)timeout when PLE should
kick in ..

> (a directed yield implementation would find that all vcpus are
> runnable, yielding optimal results under this test case).

I would think a plain yield() (rather than usleep/directed yield) would suffice
here (yield would realize that there is nobody else to yield to and continue
running the same vcpu thread). As regards to any concern of leaking cpu 
bandwidth because of a plain yield, I think it can be fixed by a more
simpler modification to yield that allows a thread to reclaim whatever timeslice
it gave up previously [1].

Regarding directed yield, do we have any reliable mechanism to find target of 
directed yield in this (unmodified/non-paravirtualized guest) case? IOW how do 
we determine the vcpu thread to which cycles need to be yielded upon contention?

> So if you were to test something similar running with a 20% vcpu
> cap, I'm sure you'd run into similar issues.  It may show with fewer
> vcpus (I've only tested 64).
> 
> >Are you assuming the existence of a directed yield and the
> >specific concern is what happens when a directed yield happens
> >after a PLE and the target of the yield has been capped?
> 
> Yes.  My concern is that we will see the same kind of problems
> directed yield was designed to fix, but without allowing directed
> yield to fix them.  Directed yield was designed to fix lock holder
> preemption under contention,

For modified guests, something like [2] seems to be the best approach to fix
lock-holder preemption (LHP) problem, which does not require any sort of 
directed yield support. Essentially upon contention, a vcpu registers its lock 
of interest and goes to sleep (via hypercall) waiting for lock-owner to wake it 
up (again via another hypercall).

For unmodified guests, IMHO a plain yield (or slightly enhanced yield [1])
should fix the LHP problem.

Fyi, Xen folks also seem to be avoiding a directed yield for some of the same
reasons [3].

Given this line of thinking, hard-limiting guests (either in user-space or
kernel-space, latter being what I prefer) should not have adverse interactions 
with LHP-related solutions.

> now you're inducing contention but not
> allowing directed yield to work, even when we will have it.

- vatsa

References:

1. http://lkml.org/lkml/2010/8/3/38
2. https://lkml.org/lkml/2010/11/16/479
3. http://www.gossamer-threads.com/lists/xen/devel/182179#182179