Re: [patch 2/3] KVM: x86: KVM_HC_RT_PRIO hypercall (host-side)

Marcelo Tosatti
2017-09-22T12:25:10Z

On Fri, Sep 22, 2017 at 09:23:47AM +0200, Paolo Bonzini wrote:
> On 22/09/2017 03:08, Marcelo Tosatti wrote:
> > On Thu, Sep 21, 2017 at 03:49:33PM +0200, Paolo Bonzini wrote:
> >> On 21/09/2017 15:32, Konrad Rzeszutek Wilk wrote:
> >>> So the guest can change the scheduling decisions at the host level?
> >>> And the host HAS to follow it? There is no policy override for the
> >>> host to say - nah, not going to do it?
> >
> > In that case the host should not even configure the guest with this
> > option (this is QEMU's 'enable-rt-fifo-hc' option).
> >
> >>> Also wouldn't the guest want to always be at SCHED_FIFO? [I am thinking
> >>> of a guest admin who wants all the CPU resources he can get]
> >
> > No. Because in the following code, executed by the housekeeping vCPU
> > running at constant SCHED_FIFO priority:
> >
> > 1. Start disk I/O.
> > 2. busy spin
> >
> > With the emulator thread sharing the same pCPU with the housekeeping
> > vCPU, the emulator thread (which runs at SCHED_NORMAL), will never
> > be scheduled in in place of the vcpu thread at SCHED_FIFO.
> >
> > This causes a hang.
>
> But if the emulator thread can interrupt the housekeeping thread, the
> emulator thread should also be SCHED_FIFO at higher priority; IIRC this
> was in Jan's talk from a few years ago.

The point is we do not want the emulator thread to interrupt the
housekeeping thread at all times: we only want it to interrupt the
housekeeping thread when it is not in a spinlock protected section (because
that has an effect on realtime vcpu's attempting to grab
that particular spinlock).

Otherwise, it can interrupt the housekeeping thread.

> QEMU would also have to use PI mutexes (which is the main reason why
> it's using QemuMutex instead of e.g. GMutex).
>
> >> Yeah, I do not understand why there should be a housekeeping VCPU that
> >> is running at SCHED_NORMAL. If it hurts, don't do it...
> >
> > Hope explanation above makes sense (in fact, it was you who pointed
> > out SCHED_FIFO should not be constant on the housekeeping vCPU,
> > when sharing pCPU with emulator thread at SCHED_NORMAL).
>
> The two are not exclusive... As you point out, it depends on the
> workload. For DPDK you can put both of them at SCHED_NORMAL. For
> kernel-intensive uses you must use SCHED_FIFO.
>
> Perhaps we could consider running these threads at SCHED_RR instead.
> Unlike SCHED_NORMAL, I am not against a hypercall that bumps temporarily
> SCHED_RR to SCHED_FIFO, but perhaps that's not even necessary.

Sorry Paolo, i don't see how SCHED_RR is going to help here:

" SCHED_RR: Round-robin scheduling
SCHED_RR is a simple enhancement of SCHED_FIFO. Everything
described
above for SCHED_FIFO also applies to SCHED_RR, except that each
thread is allowed to run only for a maximum time quantum."

What must happen is that vcpu0 should run _until its finished with
spinlock protected section_ (that is, any job the emulator thread
has, in that period where vcpu0 has work to do, is of less priority
and must not execute). Otherwise vcpu1, running a realtime workload,
will attempt to grab the spinlock vcpu0 has grabbed, and busy
spin waiting on the emulator thread to finish.

If you have the emulator thread at a higher priority than vcpu0, as you
suggested above, the same problem will happen. So that option is not
viable.

We tried to have vcpu0 with SCHED_FIFO at all times, to avoid this
hypercall, but unfortunately that'll cause the hang as described in the
trace.

So i fail to see how SCHED_RR should help here?

Thanks