Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Alex Williamson <alex.williamson@redhat.com>]

On Wed, 2013-09-11 at 16:42 +0000, Yoder Stuart-B08248 wrote:
> 
> > -----Original Message-----
> > From: Yoder Stuart-B08248
> > Sent: Thursday, September 05, 2013 12:51 PM
> > To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> > Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> > 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> > Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
> > devel@nongnu.org'
> > Subject: vfio for platform devices - 9/5/2012 - minutes
> > 
> > We had a call with those interested and/or working on vfio
> > for platform devices.
> > 
> > Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
> >               Santosh Shukla, Alex Williamson, Alexander Graf,
> >               Antonios Motakis, Christoffer Dall, Kim Phillips,
> >               Stuart Yoder
> > 
> > Several aspects to vfio for platform devices:
> > 
> > 1. IOMMU groups
> > 
> >  -iommu driver needs to register a bus notifier for the platform bus
> >   and create groups for relevant platform devices
> >  -Antonios is looking at this for several ARM IOMMUs
> >  -PAMU (Freescale) driver already does this
> > 
> > 2. unbinding device from host
> > 
> >  PCI:
> >    echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
> >  Platform:
> >    echo ffe101300.dma >
> > /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> > 
> >  -don't believe there are issues or work to do here
> > 
> > 3. binding device to vfio-platform driver
> > 
> >  PCI:
> >    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> > 
> >  -this is probably the least understood issue-- platform drivers
> >   register themselves with the bus for a specific "name"
> >   string.  That is matched with device tree "compatible" strings
> >   later to bind a device to a driver
> >  -we want is to have the vfio-platform driver to dynamically bind
> >   to a variety of platform devices previously unknown to
> >   vfio-platform
> >  -ideally unbinding and binding could be an atomic operation
> >  -Alex W pointed out that x86 could leverage this work so
> >   keep that in mind in what we design
> >  -Kim Phillips (Linaro) will start working on this
> 
> One thing we didn't discuss needs to be considered (probably by
> Kim who is looking at the 'binding device' issue) is around
> returning a passthru device back to the host.
> 
> After a platform device has been bound to vfio and is in use by
> user space or a virtual machine, we also need to be able
> to unwind all that and return the device back to the host
> in a sane state.
> 
> What happens when user space exits and the vfio file
> descriptors are closed?

For reference, expectations of how vfio-pci handles these situations:

For vfio-pci, when the reference count on the device fd drops to zero we
call a device disable function that includes disabling the bus master
bit in config space stop ongoing DMA.

> What if the device is still active and doing bus 
> mastering?   (e.g. a VM crashed causing a QEMU
> exit)

If the VM crashes the vfio fds get released resulting in the above
opportunity for the vfio device driver to quiesce the device.

> How can the vfio-platform layer in the host kernel
> get a specific device in a sane state?

It's not easy on pci either.  We save config space prior to exposing the
device and restore config space later, but it's not complete.  We mostly
rely on device (function) resets, to put things in a sane state, but
those aren't always supported.  I just introduced patches for v3.12 that
enable a PCI bus reset interface, but it's mostly useful for userspace,
since on PCI it's often the case that a bus contains multiple devices
which don't necessarily align to iommu group boundaries.

> When a plaform device is 'unbound' from vfio, what
> specifically happens to the device?

The driver remove function is called, which needs to call
vfio_del_group_dev(), which will block if the device is in use.  If it's
unused, the release function should have already been called and the
device should be in a quiesced state.  On x86/pci, there is no auto
probe of devices when unbound, so the device sits without a driver until
manually bound or until something causes an auto probe.  Thanks,

Alex

> Platform devices don't have generic mechanisms like on PCI
> to disable bus mastering or even disable or reset a
> device.
> 
> Haven't thought through all this yet, but just raising
> some issues I see.
> 
> Regards,
> Stuart
> 
> 
> 
>