EBUSY when using NVMe Block Driver with multiple devices in the same IOMMU group

EBUSY when using NVMe Block Driver with multiple devices in the same IOMMU group

[Martin Oliveira]

Hello,

I'm trying to use the QEMU NVMe userspace driver and I'm hitting an error when trying to use more than one device from an IOMMU group:

    Failed to open VFIO group file: /dev/vfio/39: Device or resource busy

If devices belong to different IOMMU groups, then it works as expected.

For each device, I bind it to vfio-pci and then use something like this:

    -drive file=nvme://0000:26:00.0,if=none,id=drive0,format=raw
    -device virtio-blk,drive=drive0,id=virtio0,serial=nvme0

Using the file-based protocol (file=/dev/nvme0n1) works with multiple devices from the same group.

My host is running a 5.19 kernel and QEMU is the latest upstream (a8cc5842b5cb).

Thanks,
Martin

Re: EBUSY when using NVMe Block Driver with multiple devices in the same IOMMU group

[Martin Oliveira]

> On Aug 24, 2022, at 4:36 PM, Martin Oliveira <martin.oliveira@eideticom.com> wrote:
>    -drive file=nvme://0000:26:00.0,if=none,id=drive0,format=raw
>    -device virtio-blk,drive=drive0,id=virtio0,serial=nvme0

Small typo above, I missed the /1, it should read:

    -drive file=nvme://0000:26:00.0/1,if=none,id=drive0,format=raw
    -device virtio-blk,drive=drive0,id=virtio0,serial=nvme0

Martin

Re: EBUSY when using NVMe Block Driver with multiple devices in the same IOMMU group

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1557 bytes --]

On Tue, Aug 23, 2022 at 10:36:00PM +0000, Martin Oliveira wrote:
> Hello,
> 
> I'm trying to use the QEMU NVMe userspace driver and I'm hitting an error when trying to use more than one device from an IOMMU group:
> 
>     Failed to open VFIO group file: /dev/vfio/39: Device or resource busy
> 
> If devices belong to different IOMMU groups, then it works as expected.
> 
> For each device, I bind it to vfio-pci and then use something like this:
> 
>     -drive file=nvme://0000:26:00.0,if=none,id=drive0,format=raw
>     -device virtio-blk,drive=drive0,id=virtio0,serial=nvme0
> 
> Using the file-based protocol (file=/dev/nvme0n1) works with multiple devices from the same group.
> 
> My host is running a 5.19 kernel and QEMU is the latest upstream (a8cc5842b5cb).

First, multiple QEMU instances cannot access nvme:// devices sharing the
same IOMMU group. I don't think this will ever be possible because it
opens a backdoor around process memory isolation.

However, a single QEMU (or qemu-storage-daemon) instance should be able
to access multiple nvme:// devices in the same IOMMU group.
Unfortunately the code currently doesn't support that.
util/vfio-helpers.c:qemu_vfio_init_pci() has no logic for sharing
groups/containers. Opening the group fails with EBUSY because the kernel
only allows the file to be opened once at any given time.

It's possible to extend the util/vfio-helpers.c code to reuse VFIO
groups (and share VFIO containers), but I'm not aware of anyone who is
currently working on that.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: EBUSY when using NVMe Block Driver with multiple devices in the same IOMMU group

[Chaitanya Kulkarni]

> First, multiple QEMU instances cannot access nvme:// devices sharing the
> same IOMMU group. I don't think this will ever be possible because it
> opens a backdoor around process memory isolation.
> 
> However, a single QEMU (or qemu-storage-daemon) instance should be able
> to access multiple nvme:// devices in the same IOMMU group.
> Unfortunately the code currently doesn't support that.
> util/vfio-helpers.c:qemu_vfio_init_pci() has no logic for sharing
> groups/containers. Opening the group fails with EBUSY because the kernel
> only allows the file to be opened once at any given time.
> 
> It's possible to extend the util/vfio-helpers.c code to reuse VFIO
> groups (and share VFIO containers), but I'm not aware of anyone who is
> currently working on that.
> 
> Stefan

Thanks a lot for detailed explanation, it is really helpful.

-ck