Re: [PATCH v2 8/9] vfio: Check compatibility of CPU and IOMMU address space width

[Alex Williamson <alex.williamson@redhat.com>]

On Fri, 31 Jan 2025 14:23:58 +0100
Gerd Hoffmann <kraxel@redhat.com> wrote:

> On Fri, Jan 31, 2025 at 01:42:31PM +0100, Cédric Le Goater wrote:
> > + Gerd for insights regarding vIOMMU support in edk2.
> >   
> > > > +static bool vfio_device_check_address_space(VFIODevice *vbasedev, Error **errp)
> > > > +{
> > > > +    uint32_t cpu_aw_bits = cpu_get_phys_bits(first_cpu);
> > > > +    uint32_t iommu_aw_bits = vfio_device_get_aw_bits(vbasedev);
> > > > +
> > > > +    if (cpu_aw_bits && cpu_aw_bits > iommu_aw_bits) {  
> > > 
> > > Should we be testing the 64-bit MMIO window and maximum RAM GPA rather
> > > than the vCPU physical address width?  
> 
> Placing the 64-bit  mmio window is done by the guest firmware,
> so this isn't something qemu can check before boot.
> 
> > > However, we've decided to do exactly that for the 64-bit MMIO window.
> > > It's not that the vCPU width being greater than the IOMMU width is a
> > > fundamental problem, it's that we've chosen a 64-bit MMIO policy that
> > > makes this become a problem and QEMU only has a convenient mechanism to
> > > check the host IOMMU width when a vfio device is present.  Arguably,
> > > when a vIOMMU is present guest firmware should be enlightened to
> > > understand the address width of that vIOMMU when placing the 64-bit
> > > MMIO window.  I'd say the failure to do so is a current firmware bug.  
> 
> edk2 has no iommu driver ...
> 
> Is there some simple way to figure what the iommu width is (inside the
> guest)?

If the guest firmware is exposing a DMAR table (VT-d), there's a host
address width field in that table.  Otherwise there are capability
registers on the DRHD units that could be queried.  AMD-Vi seems to
have similar information in the IVinfo table linked from the IVRS
table.  Maybe an iterative solution is ok, starting with the most
common IOMMU types and assuming others are 64-bits wide until proven
otherwise.

> 
> Note that there is a 'guest-phys-bits' property for x86 CPUs, which is a
> hint for the guest what the usable address width is.  It was added
> because there are cases where the guest simply can't figure that it is
> not possible to use the full physical address space of the cpu.  There
> are some non-obvious limitations around 5-level paging.  Intel has some
> CPUs with phys-bits > 48 but only 4-level EPT for example.
> 
> So one option to handle this is to make sure guest-phys-bits is not
> larger than the iommu width.

Right, as Cédric notes that's sort of what happens here, but my concern
was that the we're kind of incorrectly linking the cpu address width to
the platform address width, which isn't specifically the requirement.
It's valid to have CPU physical address width great than IOMMU address
width, that exists on bare metal, but guest firmware needs to take
IOMMU address width into account in placing the 64-bit MMIO window if
we want PCI BARs to be DMA addressable.  Thanks,

Alex