Re: [PATCH] sched,kvm: Fix KVM preempt_notifier usage

Peter Zijlstra
2015-07-03T13:17:27Z

On Fri, Jul 03, 2015 at 02:31:25PM +0200, Paolo Bonzini wrote:
> On 03/07/2015 14:19, Peter Zijlstra wrote:
> > On Fri, Jul 03, 2015 at 01:12:11PM +0200, Paolo Bonzini wrote:
> >> In fact you shouldn't have just tested the patch on a case _without_
> >> preemption notifiers, you should have also benchmarked the impact that
> >> static keys have _with_ preemption notifiers. In a
> >> not-really-artificial case (one single-processor guest running on the
> >> host), the static key patch adds a static_key_slow_inc on a relatively
> >> hot path for KVM, which is not acceptable.
> >
> > Spawning the first vcpu is a hot path?
>
> This is not *spawning* the first VCPU. Basically any critical section
> for vcpu->mutex includes a preempt_notifier_register/unregister pair:
>
> /*
> * Switches to specified vcpu, until a matching vcpu_put()
> */
> int vcpu_load(struct kvm_vcpu *vcpu)
> {
> int cpu;
>
> if (mutex_lock_killable(&vcpu->mutex))
> return -EINTR;
> cpu = get_cpu();
> preempt_notifier_register(&vcpu->preempt_notifier);
> kvm_arch_vcpu_load(vcpu, cpu);
> put_cpu();
> return 0;
> }
>
> void vcpu_put(struct kvm_vcpu *vcpu)
> {
> preempt_disable();
> kvm_arch_vcpu_put(vcpu);
> preempt_notifier_unregister(&vcpu->preempt_notifier);
> preempt_enable();
> mutex_unlock(&vcpu->mutex);
> }
>
> So basically you're adding at least one static_key_slow_inc/dec pair to
> every userspace exit.

Ugh, ok that is not what I was expecting to happen. I'll ask Ingo to
queue a revert until we can fix this better.

I thought these were vcpu create/destroy functions.

That said, the slow_inc/dec are really only slow on the 0<->!0
transitions.

But, could we rework the code so that you register the preempt notifier
when creating the vcpu thread and leave it installed forevermore?