Re: [RFC PATCH 3/3] kvm: use yield_to instead of sleep in kvm_vcpu_on_spin

Avi Kivity
2010-12-05T12:56:46Z

On 12/02/2010 09:45 PM, Rik van Riel wrote:
> Instead of sleeping in kvm_vcpu_on_spin, which can cause gigantic
> slowdowns of certain workloads, we instead use yield_to to hand
> the rest of our timeslice to another vcpu in the same KVM guest.
>
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 80f17db..a6eeafc 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -1880,18 +1880,53 @@ void kvm_resched(struct kvm_vcpu *vcpu)
> }
> EXPORT_SYMBOL_GPL(kvm_resched);
>
> -void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu)
> +void kvm_vcpu_on_spin(struct kvm_vcpu *me)
> {
> - ktime_t expires;
> - DEFINE_WAIT(wait);
> + struct kvm *kvm = me->kvm;
> + struct kvm_vcpu *vcpu;
> + int last_boosted_vcpu = me->kvm->last_boosted_vcpu;
> + int first_round = 1;
> + int i;
>
> - prepare_to_wait(&vcpu->wq,&wait, TASK_INTERRUPTIBLE);
> + me->spinning = 1;
> +
> + /*
> + * We boost the priority of a VCPU that is runnable but not
> + * currently running, because it got preempted by something
> + * else and called schedule in __vcpu_run. Hopefully that
> + * VCPU is holding the lock that we need and will release it.
> + * We approximate round-robin by starting at the last boosted VCPU.
> + */
> + again:
> + kvm_for_each_vcpu(i, vcpu, kvm) {
> + struct task_struct *task = vcpu->task;
> + if (first_round&& i< last_boosted_vcpu) {
> + i = last_boosted_vcpu;
> + continue;
> + } else if (!first_round&& i> last_boosted_vcpu)
> + break;
> + if (vcpu == me)
> + continue;
> + if (vcpu->spinning)
> + continue;

You may well want to wake up a spinner. Suppose

A takes a lock
B preempts A
B grabs a ticket, starts spinning, yields to A
A releases lock
A grabs ticket, starts spinning

at this point, we want A to yield to B, but it won't because of this check.

> + if (!task)
> + continue;
> + if (waitqueue_active(&vcpu->wq))
> + continue;
> + if (task->flags& PF_VCPU)
> + continue;
> + kvm->last_boosted_vcpu = i;
> + yield_to(task);
> + break;
> + }

I think a random selection algorithm will be a better fit against
special guest behaviour.

>
> - /* Sleep for 100 us, and hope lock-holder got scheduled */
> - expires = ktime_add_ns(ktime_get(), 100000UL);
> - schedule_hrtimeout(&expires, HRTIMER_MODE_ABS);
> + if (first_round&& last_boosted_vcpu == kvm->last_boosted_vcpu) {
> + /* We have not found anyone yet. */
> + first_round = 0;
> + goto again;

Need to guarantee termination.

--
error compiling committee.c: too many arguments to function