Re: [PATCH 3/3] KVM: arm64: nv: Punt stage-2 recycling to a vCPU request

[Oliver Upton <oliver.upton@linux.dev>]

On Thu, Oct 03, 2024 at 09:45:40AM -0700, Sean Christopherson wrote:

[...[

> > > OTOH, our global TLBs don't model hardware exactly since a vCPU doing
> > > rapid context switches trash the TLBs of *all* vCPUs in the system.
> > > The cost of reusing an MMU is quite noticeable, since our unmap
> > > implementation is slightly crap at the moment, the cost of which shows
> > > up both on sides of the reclaim (victim and user).
> > 
> > Oh, and why unmap is crap:
> 
> Heh, isn't unmap by definition crap?  If KVM needs to unmap and rebuild an S2 MMU,
> then KVM is already in a slow, sub-optimal situation.

Not really, the unmap plumbing is used for applying the intent of a
guest TLBI too. Sub-optimal or not, it is exactly what the VM asked for,
and it'd be in our interest to handle the unmap as expeditiously as
possible.

> > > Still should drop the reference in most other cases, as I do *not* want
> > > to entertain vCPUs holding a reference when they've gone out to
> > > userspace.
> 
> Why not?  The vCPU is still running, keeping its S2 MMU resident is desirable, no?

How could we possibly know what the intent of userspace is? The VMM
could just as well throw that vCPU fd on ice for an eternity.

For example, you could have a PSCI implementation that lives in
userspace. Guest does CPU_OFF and the VMM decides to terminate the
backing thread and keep the FD around for the next CPU_ON.

Since KVM still views that fd as 'runnable', it'd sit on the reference
that vCPU holds indefinitely. On top of that, it adds complexity to the
implementation since we would need more refcount cleanup flows to handle
these straggler references.

> Essentially all I'm suggesting is that instead of having a common pool of 2*vCPUs
> TLBs per L1 VMM, have 2 (or however many) TLBs per L1 vCPU, plus maybe N extra
> TLBs per L1 VMM.  I.e. mimic the hierarchical design of hardware caches and TLBs
> to some extent.

Making TLBs private to the L1 vCPU is almost guaranteed to be a net loss
in performance. The common case for an L2 VM is that all L2 vCPUs share the
*exact* same translation tables and MMU configuration. So even if you're
running an N vCPU L2, you've only allocated 1 nested MMU context to it.

The ARM architecture has gone as far as making this an explicit contract
between hardware + software. That is, FEAT_TTCNP allows translations to
be shared across the Inner Shareable domain. There's hardware out there
that takes advantage of this, and there is a performance advantage to
enabling the feature.

The guest hypervisor is free to disregard this part of the architecture
and keep MMUs private between CPUs. At the same time, I see zero
incentive for optimizing this use case and am absolutely fine with there
being a performance penalty associated with this configuration.

-- 
Thanks,
Oliver