Re: [Intel-gfx] [PATCH v5 00/19] Add vfio_device cdev for iommufd support

[Nicolin Chen <nicolinc@nvidia.com>]

On Tue, Mar 14, 2023 at 11:38:11AM +0000, Shameerali Kolothum Thodi wrote:

> Hi Nicolin,
> 
> I rebased your latest Qemu branch[1] on top of v7.2.0 and not observed
> the above issue so far. However noticed couple of other issues when
> we try to hot add/remove devices.
> 
> (qemu) device_del net1
> qemu-system-aarch64-iommufd: Failed to free id: 4 Inappropriate ioctl for device
> qemu-system-aarch64-iommufd: IOMMU_IOAS_UNMAP failed: No such file or directory
> qemu-system-aarch64-iommufd: vfio_dma_unmap(0xaaaaf587a3d0, 0x8000101000, 0xf000) = -2 (No such file or directory)
> qemu-system-aarch64-iommufd: IOMMU_IOAS_UNMAP failed: No such file or directory
> qemu-system-aarch64-iommufd: vfio_dma_unmap(0xaaaaf587a3d0, 0x8000000000, 0x100000) = -2 (No such file or directory)
> qemu-system-aarch64-iommufd: Failed to free id:1 Device or resource busy
> 
> Ignoring the MMIO UNMAP errors, it looks like the object free is
> not proper on dev removal path. I have few quick fixes here
> for this,
> https://github.com/hisilicon/qemu/tree/private-v7.2.0-iommufd-nesting

The smmuv3 change looks good to me. I will let Yi check the
iommufd change.

Yi, I wonder if this is the hot reset case that you asked me
for, a couple of weeks ago.

> With the above, it seems the HWPT/IOAS objects are destroyed properly
> on dev detach path. But when the dev is added back, gets a Qemu seg fault
> and so far I have no clue why that happens.
>
> (qemu) device_add vfio-pci,host=0000:7d:02.1,iommufd=iommufd0,bus=rp1,id=net1
> ./qemu_run-iommufd-nested: line 13:  7041 Segmentation fault
> (core dumped) ./qemu-system-aarch64-iommufd
> -machine virt,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0
> -enable-kvm -cpu host -m 1G -smp cpus=8,maxcpus=8 -object
> iommufd,id=iommufd0 -bios QEMU_EFI_Dec2018.fd -kernel
> Image-iommufd -initrd rootfs-iperf.cpio -device
> ioh3420,id=rp1 -device
> vfio-pci,host=0000:7d:02.1,iommufd=iommufd0,bus=rp1,id=net1 -append
> "rdinit=init console=ttyAMA0 root=/dev/vda rw
> earlycon=pl011,0x9000000" -net none -nographic -trace events=events -D
> trace_iommufd
> 
> There are no kernel log/crash and not much useful traces while this happens.
> Understand these are early days and it is not robust in anyway, but please
> let me know if you suspect anything. I will continue debugging and will update
> if anything.

Thanks! That'd be very helpful.

Nicolin