Re: [RFC PATCH 5/5] VFIO based device assignment

Avi Kivity
2010-07-11T18:27:39Z

On 07/11/2010 09:09 PM, Alex Williamson wrote:
> This patch adds qemu device assignment support using the proposed
> VFIO/UIOMMU kernel interfaces. The existing KVM-only device assignment
> code makes use of various pci sysfs files for config space, MMIO BAR
> mapping, and misc other config items. It then jumps over to KVM-specific
> ioctls for enabling interrupts and assigning devices to IOMMU domains.
> Finally, IO-port support uses in/out directly. This is a messy model
> to support and causes numerous issues when we try to allow unprivileged
> users to access PCI devices.
>
> VFIO/UIOMMU reduces this to two interfaces, /dev/vfioX and /dev/uiommu.
> The VFIO device file provides all the necessary support for accessing
> PCI config space, read/write/mmap BARs (including IO-port space),
> configuring INTx/MSI/MSI-X interupts and setting up DMA mapping. The
> UIOMMU interface allows iommu domains to be created, and via vfio,
> devices can be bound to a domain. This provides an easier model to
> support (IMHO) and removes the bindings that make current device
> assignment only useable for KVM enabled guests.
>
> Usage is similar to KVM device assignment. Rather than binding the
> device to the pci-stub driver, vfio devices need to be bound to the
> vfio driver. From there, it's a simple matter of specifying the
> device as:
>
> -device vfio,host=01:00.0
>
> This example requires either root privileges or proper permissions on
> /dev/uiommu and /dev/vfioX. To support unprivileged operation, the
> options vfiofd= and uiommufd= are available. Depending on the usage
> of uiommufd, each guest device can be assigned to the same iommu
> domain, or to independent iommu domains. In the example above, each
> device is assigned to a separate iommu domain.
>
> As VFIO has no KVM dependencies, this patch works with or without
> -enable-kvm. I have successfully used a couple assigned devices in a
> guest without KVM support, however Michael Tsirkin warns that tcg
> may not provide atomic operations to memory visible to the passthrough
> device, which could result in failures for devices depending on such
> for synchronization.
>
> This patch is functional, but hasn't seen a lot of testing. I've
> tested 82576 PFs and VFs, an Intel HDA audio device, and UHCI and EHCI
> USB devices (this actually includes INTx/MSI/MSI-X, 4k aligned MMIO
> BARs, non-4k aligned MMIO BARs, and IO-Port BARs).
>
>

Good stuff.

I presume the iommu interface is responsible for page pinning. What
about page attributes?

There are two cases:

- snoop capable iommu - can use write-backed RAM, but need to enable
snoop. BARs still need to respect page attributes.
- older mmu - need to respect guest memory type; probably cannot be done
without kvm.

If the guest maps a BAR or RAM using write-combine memory type, can we
reflect that? This may provide a considerable performance benefit.

--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.