Re: EBUSY when using NVMe Block Driver with multiple devices in the same IOMMU group

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 1557 bytes --]

On Tue, Aug 23, 2022 at 10:36:00PM +0000, Martin Oliveira wrote:
> Hello,
> 
> I'm trying to use the QEMU NVMe userspace driver and I'm hitting an error when trying to use more than one device from an IOMMU group:
> 
>     Failed to open VFIO group file: /dev/vfio/39: Device or resource busy
> 
> If devices belong to different IOMMU groups, then it works as expected.
> 
> For each device, I bind it to vfio-pci and then use something like this:
> 
>     -drive file=nvme://0000:26:00.0,if=none,id=drive0,format=raw
>     -device virtio-blk,drive=drive0,id=virtio0,serial=nvme0
> 
> Using the file-based protocol (file=/dev/nvme0n1) works with multiple devices from the same group.
> 
> My host is running a 5.19 kernel and QEMU is the latest upstream (a8cc5842b5cb).

First, multiple QEMU instances cannot access nvme:// devices sharing the
same IOMMU group. I don't think this will ever be possible because it
opens a backdoor around process memory isolation.

However, a single QEMU (or qemu-storage-daemon) instance should be able
to access multiple nvme:// devices in the same IOMMU group.
Unfortunately the code currently doesn't support that.
util/vfio-helpers.c:qemu_vfio_init_pci() has no logic for sharing
groups/containers. Opening the group fails with EBUSY because the kernel
only allows the file to be opened once at any given time.

It's possible to extend the util/vfio-helpers.c code to reuse VFIO
groups (and share VFIO containers), but I'm not aware of anyone who is
currently working on that.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]