[Qemu-devel] Re: [RFC PATCH 5/5] VFIO based device assignment

[Avi Kivity <avi@redhat.com>]

On 07/11/2010 09:09 PM, Alex Williamson wrote:
> This patch adds qemu device assignment support using the proposed
> VFIO/UIOMMU kernel interfaces.  The existing KVM-only device assignment
> code makes use of various pci sysfs files for config space, MMIO BAR
> mapping, and misc other config items.  It then jumps over to KVM-specific
> ioctls for enabling interrupts and assigning devices to IOMMU domains.
> Finally, IO-port support uses in/out directly.  This is a messy model
> to support and causes numerous issues when we try to allow unprivileged
> users to access PCI devices.
>
> VFIO/UIOMMU reduces this to two interfaces, /dev/vfioX and /dev/uiommu.
> The VFIO device file provides all the necessary support for accessing
> PCI config space, read/write/mmap BARs (including IO-port space),
> configuring INTx/MSI/MSI-X interupts and setting up DMA mapping.  The
> UIOMMU interface allows iommu domains to be created, and via vfio,
> devices can be bound to a domain.  This provides an easier model to
> support (IMHO) and removes the bindings that make current device
> assignment only useable for KVM enabled guests.
>
> Usage is similar to KVM device assignment.  Rather than binding the
> device to the pci-stub driver, vfio devices need to be bound to the
> vfio driver.  From there, it's a simple matter of specifying the
> device as:
>
> -device vfio,host=01:00.0
>
> This example requires either root privileges or proper permissions on
> /dev/uiommu and /dev/vfioX.  To support unprivileged operation, the
> options vfiofd= and uiommufd= are available.  Depending on the usage
> of uiommufd, each guest device can be assigned to the same iommu
> domain, or to independent iommu domains.  In the example above, each
> device is assigned to a separate iommu domain.
>
> As VFIO has no KVM dependencies, this patch works with or without
> -enable-kvm.  I have successfully used a couple assigned devices in a
> guest without KVM support, however Michael Tsirkin warns that tcg
> may not provide atomic operations to memory visible to the passthrough
> device, which could result in failures for devices depending on such
> for synchronization.
>
> This patch is functional, but hasn't seen a lot of testing.  I've
> tested 82576 PFs and VFs, an Intel HDA audio device, and UHCI and EHCI
> USB devices (this actually includes INTx/MSI/MSI-X, 4k aligned MMIO
> BARs, non-4k aligned MMIO BARs, and IO-Port BARs).
>
>    

Good stuff.

I presume the iommu interface is responsible for page pinning.  What 
about page attributes?

There are two cases:

- snoop capable iommu - can use write-backed RAM, but need to enable 
snoop.  BARs still need to respect page attributes.
- older mmu - need to respect guest memory type; probably cannot be done 
without kvm.

If the guest maps a BAR or RAM using write-combine memory type, can we 
reflect that?  This may provide a considerable performance benefit.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.