Re: [RFC PATCH] xen/amd-iommu: Add interrupt remapping quirk for ath11k

["Roger Pau Monné" <roger.pau@citrix.com>]

On Tue, Mar 04, 2025 at 10:15:21AM -0500, Jason Andryuk wrote:
> On 2025-03-04 05:23, Roger Pau Monné wrote:
> > On Fri, Feb 28, 2025 at 03:25:52PM -0500, Jason Andryuk wrote:
> > > On 2025-02-28 04:36, Roger Pau Monné wrote:
> > > > On Thu, Feb 27, 2025 at 01:28:11PM -0500, Jason Andryuk wrote:
> > > > > On 2025-02-27 05:23, Roger Pau Monné wrote:
> > > > > > On Wed, Feb 26, 2025 at 04:11:25PM -0500, Jason Andryuk wrote:
> > > > > > > To work around this, we can, for per-device IRTs, program the hardware
> > > > > > > to use the guest data & associated IRTE.  The address doesn't matter
> > > > > > > since the IRTE handles that, and the Xen address & vector can be used as
> > > > > > > expected.
> > > > > > 
> > > > > > All this work on AMD because when interrupt remapping is enabled all
> > > > > > MSIs are handled by the remapping table, while on Intel there's still
> > > > > > a bit in the MSI address field to signal whether the MSI is using a
> > > > > > remapping entry, or is using the "compatibility" format (iow: no
> > > > > > remapping).
> > > > > 
> > > > > So, on Intel, if the guest hands the device the MSI address, it can decided
> > > > > to bypass remapping?
> > > > > 
> > > > > Thanks for providing insight into the Intel inner workings.  That's why I am
> > > > > asking.
> > > > 
> > > > Yes, sorry, I'm afraid I don't have any good solution for Intel, at
> > > > least not anything similar to what you propose to do on AMD-Vi.  I
> > > > guess we could take a partial solution for AMD-Vi only, but it's
> > > > sub-optimal from Xen perspective to have a piece of hardware working
> > > > fine on AMD and not on Intel.
> > > 
> > > I only need AMD to work ;)
> > > 
> > > But yeah, I thought I should make an effort to get both working.
> > 
> > Kind of tangential to this approach.  Do you know which register(s)
> > are used to store the non-architectural MSI address and data fields?
> > 
> > I'm wondering if it simply would be easier to introduce a quirk for
> > this device in vPCI (and possibly QEMU?) that intercepts writes to the
> > out of band MSI registers.  That should work for both Intel and AMD,
> > but would have the side effect that Xen would need to intercept
> > accesses to at least a full page, and possibly forward accesses to
> > adjacent registers.
> 
> From the QEMU part of the vfio hack:
> * We therefore come up with a really crude quirk that looks for values
> * written to the ATH11K_PCI_WINDOW (defined in Linux driver as starting
> * at 0x80000 with an 18-bit mask, ie. 256k) that match the guest MSI
> * address.  When found we replace the data with the host physical
> * address and set a cookie to match the MSI data write, again replacing
> * with the host value and clearing the cookie.
> 
> https://lore.kernel.org/ath11k/20240812170045.1584000-1-alex.williamson@redhat.com/
> 
> This is inside BAR0, AIUI.  I'm guessing, but I think the driver puts them
> into a command ring, so it's not a fixed location.  The large area, and
> since we won't normally intercept BAR access, made me not want to pursue
> this.

Oh, I see, it's not a fixed register, but something like a command
queue.  Great, that makes it way worse to deal with.  It would also
imply that Xen would need to possibly map the whole 256k ring in order
to forward requests to the hardware.

I would like for a solution that covers both Intel and AMD, but I
don't have the card myself to test or develop any of those solutions,
so I think having a solution that works on AMD is likely better than
having no solution at all.  Whoever wants to use the card on Intel
will have to come up with a solution there.

Thanks, Roger.