Re: virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI bridge: VFIO devices are not assigned any iommu group

[Eric Auger <eauger@redhat.com>]

Hi,

On 1/9/23 14:24, Eric Auger wrote:
> Hi,
> 
> we have a trouble with virtio-iommu and protected assigned devices
> downstream to a pcie-to-pci bridge. In that use case we observe the
> assigned devices are not put to any group. This is true on both x86 and
> aarch64. This use case works with intel-iommu.
> 
> *** Guest PCI topology is:
> lspci -tv
> -[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM
> Controller
>            +-01.0  Device 1234:1111
>            +-02.0-[01-02]----00.0-[02]----01.0  Broadcom Inc. and
> subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
>            +-02.1-[03]--
>            +-02.2-[04]----00.0  Red Hat, Inc. Virtio block device
>            +-0a.0  Red Hat, Inc. Device 1057
>            +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
>            +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port
> SATA Controller [AHCI mode]
>            \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
> 
> 
> All the assigned devices are aliased and they get devfn=0x0.
> see qemu pci_device_iommu_address_space in hw/pci.c
> 
> Initially I see the following traces
> pci_device_iommu_address_space name=vfio-pci BDF=0x8 bus=0 devfn=0x8
> pci_device_iommu_address_space name=vfio-pci BDF=0x8 bus=0 devfn=0x8
> call iommu_fn with bus=0x55f556dde180 and devfn=0
> virtio_iommu_init_iommu_mr init virtio-iommu-memory-region-0-0
> 
> Note the bus is 0 at this time and devfn that is used in the
> virtio-iommu is 0. So an associated IOMMU MR is created with this bus at
> devfn=0 slot. This is before bus actual numbering.
> 
> However later on, I see virtio_iommu_probe() and virtio_iommu_attach()
> getting called with ep_id=520
> because in the qemu virtio-iommu device, virtio_iommu_mr(pe_id) fails to
> find the iommu_mr and returns -ENOENT
> 
> On guest side I see that
> acpi_iommu_configure_id/iommu_probe_device() fails
> (__iommu_probe_device) and also __iommu_attach_device would also fail
> anyway.
> 
> I guess those get called before actual bus number recomputation?
> 
> on aarch64 I eventually see the "good" MR beeing created, ie. featuring
> the right bus number:
> qemu-system-aarch64: pci_device_iommu_address_space name=vfio-pci
> BDF=0x208 bus=2 devfn=0x8
> qemu-system-aarch64: pci_device_iommu_address_space name=vfio-pci
> BDF=0x208 bus=2 devfn=0x8 call iommu_fn with bus=0xaaaaef12c450 and devfn=0
> 
> But this does not happen on x86.
> 
> Jean, do you have any idea about how to fix that? Do you think we have a
> trouble in the acpi/viot setup or virtio-iommu probe sequence. It looks
> like virtio probe and attach commands are called too early, before the
> bus is actually correctly numbered.

So after further investigations looks this is not a problem of bus
number, which is good at the time of the virtio cmd calls but rather a
problem related to the devfn (0 was used when creating the IOMMU MR)
whereas the virtio-iommu cmds looks for the non aliased devfn. With that
fixed, the probe and attach at least succeeds. The device still does not
work for me but I will continue my investigations and send a tentative fix.

Thanks

Eric
> 
> Thanks
> 
> Eric
> 
> 
> 
> 
> 
> 
>