Re: RMRR device on non-Intel platform

[Jason Gunthorpe <jgg@nvidia.com>]

On Wed, Apr 26, 2023 at 01:24:21PM +0100, Robin Murphy wrote:

> A real *virtual* ITS page (IPA/GPA, not PA) as above, because the Arm system
> architecture does not define a fixed address map thus that is free to be
> virtualised as well, but otherwise, yes, indeed it should, and it could, on
> both GICv3 and AMD/Intel IRQ remapping. Why isn't Linux letting VMMs do
> that?

At the root, we don't have an interface out of VFIO for setting up
interrupts in such a sophisticated way.

> Fair enough for VFIO, since that existed long before any architectural MSI
> support on Arm, and has crusty PCI/X legacy on x86 to deal with too, but
> IOMMUFD is a new thing for modern use-cases on modern hardware. 

Ah but iommufd isn't touching interrupts :)

I'm scared we need a irqfd kind of idea to expose all this
acclerated IRQ hardware to the VMM as well.

> > > ...I believe the remaining missing part is a UAPI for the VMM to ask the
> > > host kernel to configure a "physical" vLPI for a given device and EventID,
> > > at the point when its vITS emulation is handling the guest's configuration
> > > command. With that we would no longer have to rewrite the MSI payload
> > > either, so can avoid trapping the device's MSI-X capability at all, and the
> > > VM could actually have non-terrible interrupt performance.
> > 
> > Yes.. More broadly I think we'd need to allow the vGIC code to
> > understand that it has complete control over a SID, and like we are
> > talking about for SMMU a vSID mapping as well.
> 
> Marc, any thoughts on how much of this is actually missing from the MSI
> domain infrastructure today? I'm guessing the main thing is needing some
> sort of msi_domain_alloc_virq() API so that the caller can provide the
> predetermined message data to replace the normal compose/write flow

We don't want the VMM to write the MSI-X data. This isn't going to get
us far enough. Talk to ARM's rep on the OCP SIOVr2 workgroup to get
some sense what is being proposed for next-generation interrupt
handling.

There will likely be no standard MSI-X table or equivalent and no
place to put a hypertrap.

If we are fixing things we need to fix it fully - the VM programs the
MSI-X registers directly. The VM's irq_chip does any hypertraps using
the GIC programming model, NOT PCI, to get the VMM to setup the IRQ
remapping HW.

This suits ARM better anyhow since the VM is in control of the IOVA
for the ITS page.

Jason