[Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Yoder Stuart-B08248]

We had a call with those interested and/or working on vfio
for platform devices.

Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
              Santosh Shukla, Alex Williamson, Alexander Graf,
              Antonios Motakis, Christoffer Dall, Kim Phillips,
              Stuart Yoder

Several aspects to vfio for platform devices:

1. IOMMU groups

 -iommu driver needs to register a bus notifier for the platform bus
  and create groups for relevant platform devices
 -Antonios is looking at this for several ARM IOMMUs
 -PAMU (Freescale) driver already does this

2. unbinding device from host

 PCI:
   echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
 Platform:
   echo ffe101300.dma > /sys/bus/platform/devices/ffe101300.dma/driver/unbind

 -don't believe there are issues or work to do here

3. binding device to vfio-platform driver

 PCI:
   echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id

 -this is probably the least understood issue-- platform drivers
  register themselves with the bus for a specific "name"
  string.  That is matched with device tree "compatible" strings
  later to bind a device to a driver
 -we want is to have the vfio-platform driver to dynamically bind
  to a variety of platform devices previously unknown to
  vfio-platform
 -ideally unbinding and binding could be an atomic operation
 -Alex W pointed out that x86 could leverage this work so
  keep that in mind in what we design
 -Kim Phillips (Linaro) will start working on this

4. vfio kernel interface

 -exposes regions and interrupts to user space via FDs
 -there are 'info' ioctls that allow getting info about
  regions/interrupts such as size and type of interrupt
 -there is a proposed extension to the 'info' ioctls that
  provides device tree paths, allowing user space to coorelate
  resources with the device tree
  (https://lists.cs.columbia.edu/pipermail/kvmarm/2013-July/006237.html)

5. QEMU
 -some key tasks
   -interacts with vfio kernel interface
   -registers memslots
   -needs to dynamically get device hooked up to VM's
    platform bus, including IRQs
   -needs to generate device tree node
 -a key point: we don't believe that platform device passthru
  in QEMU can be solved in a completely generic way.  There will
  need to be device specific code for each device type being passed
  through...to do things like generate device tree nodes
 -in general we expect a relatively small number of device types
  to be passed through to VMs
 -Alex Graf is working on dynamic creation of platform devices
  for the PPC e500 paravirt machine
 -see: http://lists.nongnu.org/archive/html/qemu-devel/2013-07/msg03614.html
 -first step is dynamically generating a virtual UART
 -that sets the stage to create and create vfio devices backed
  by real hardware

There is a session at Linux Plumbers in a couple of weeks
and further discussions will happen there.

Regards,
Stuart

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Sethi Varun-B16395]

I have a query about the ARM SMMU driver. In the ARM smmu driver I see, that bus notifiers are registered for both amba and platform bus. Amba is the I/O interconnect, right? Why is bus notifier required for the amba bus?
 
-Varun

> -----Original Message-----
> From: Yoder Stuart-B08248
> Sent: Thursday, September 05, 2013 11:21 PM
> To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> Cc: kvmarm@lists.cs.columbia.edu; kvm-ppc@vger.kernel.org; qemu-
> devel@nongnu.org
> Subject: vfio for platform devices - 9/5/2012 - minutes
> 
> We had a call with those interested and/or working on vfio for platform
> devices.
> 
> Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
>               Santosh Shukla, Alex Williamson, Alexander Graf,
>               Antonios Motakis, Christoffer Dall, Kim Phillips,
>               Stuart Yoder
> 
> Several aspects to vfio for platform devices:
> 
> 1. IOMMU groups
> 
>  -iommu driver needs to register a bus notifier for the platform bus
>   and create groups for relevant platform devices  -Antonios is looking
> at this for several ARM IOMMUs  -PAMU (Freescale) driver already does
> this
> 
> 2. unbinding device from host
> 
>  PCI:
>    echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
>  Platform:
>    echo ffe101300.dma >
> /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> 
>  -don't believe there are issues or work to do here
> 
> 3. binding device to vfio-platform driver
> 
>  PCI:
>    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> 
>  -this is probably the least understood issue-- platform drivers
>   register themselves with the bus for a specific "name"
>   string.  That is matched with device tree "compatible" strings
>   later to bind a device to a driver
>  -we want is to have the vfio-platform driver to dynamically bind
>   to a variety of platform devices previously unknown to
>   vfio-platform
>  -ideally unbinding and binding could be an atomic operation  -Alex W
> pointed out that x86 could leverage this work so
>   keep that in mind in what we design
>  -Kim Phillips (Linaro) will start working on this
> 
> 4. vfio kernel interface
> 
>  -exposes regions and interrupts to user space via FDs  -there are 'info'
> ioctls that allow getting info about
>   regions/interrupts such as size and type of interrupt  -there is a
> proposed extension to the 'info' ioctls that
>   provides device tree paths, allowing user space to coorelate
>   resources with the device tree
>   (https://lists.cs.columbia.edu/pipermail/kvmarm/2013-July/006237.html)
> 
> 5. QEMU
>  -some key tasks
>    -interacts with vfio kernel interface
>    -registers memslots
>    -needs to dynamically get device hooked up to VM's
>     platform bus, including IRQs
>    -needs to generate device tree node
>  -a key point: we don't believe that platform device passthru
>   in QEMU can be solved in a completely generic way.  There will
>   need to be device specific code for each device type being passed
>   through...to do things like generate device tree nodes  -in general we
> expect a relatively small number of device types
>   to be passed through to VMs
>  -Alex Graf is working on dynamic creation of platform devices
>   for the PPC e500 paravirt machine
>  -see: http://lists.nongnu.org/archive/html/qemu-devel/2013-
> 07/msg03614.html
>  -first step is dynamically generating a virtual UART  -that sets the
> stage to create and create vfio devices backed
>   by real hardware
> 
> There is a session at Linux Plumbers in a couple of weeks and further
> discussions will happen there.
> 
> Regards,
> Stuart

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Yoder Stuart-B08248]

Adding Will...

> -----Original Message-----
> From: Sethi Varun-B16395
> Sent: Friday, September 06, 2013 11:56 AM
> To: Yoder Stuart-B08248; Wood Scott-B07421; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> Cc: kvmarm@lists.cs.columbia.edu; kvm-ppc@vger.kernel.org; qemu-
> devel@nongnu.org
> Subject: RE: vfio for platform devices - 9/5/2012 - minutes
> 
> I have a query about the ARM SMMU driver. In the ARM smmu driver I see,
> that bus notifiers are registered for both amba and platform bus. Amba is
> the I/O interconnect, right? Why is bus notifier required for the amba
> bus?
> 
> -Varun
> 
> > -----Original Message-----
> > From: Yoder Stuart-B08248
> > Sent: Thursday, September 05, 2013 11:21 PM
> > To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777;
> 'Peter
> > Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> > 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> > Cc: kvmarm@lists.cs.columbia.edu; kvm-ppc@vger.kernel.org; qemu-
> > devel@nongnu.org
> > Subject: vfio for platform devices - 9/5/2012 - minutes
> >
> > We had a call with those interested and/or working on vfio for platform
> > devices.
> >
> > Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
> >               Santosh Shukla, Alex Williamson, Alexander Graf,
> >               Antonios Motakis, Christoffer Dall, Kim Phillips,
> >               Stuart Yoder
> >
> > Several aspects to vfio for platform devices:
> >
> > 1. IOMMU groups
> >
> >  -iommu driver needs to register a bus notifier for the platform bus
> >   and create groups for relevant platform devices  -Antonios is looking
> > at this for several ARM IOMMUs  -PAMU (Freescale) driver already does
> > this
> >
> > 2. unbinding device from host
> >
> >  PCI:
> >    echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
> >  Platform:
> >    echo ffe101300.dma >
> > /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> >
> >  -don't believe there are issues or work to do here
> >
> > 3. binding device to vfio-platform driver
> >
> >  PCI:
> >    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> >
> >  -this is probably the least understood issue-- platform drivers
> >   register themselves with the bus for a specific "name"
> >   string.  That is matched with device tree "compatible" strings
> >   later to bind a device to a driver
> >  -we want is to have the vfio-platform driver to dynamically bind
> >   to a variety of platform devices previously unknown to
> >   vfio-platform
> >  -ideally unbinding and binding could be an atomic operation  -Alex W
> > pointed out that x86 could leverage this work so
> >   keep that in mind in what we design
> >  -Kim Phillips (Linaro) will start working on this
> >
> > 4. vfio kernel interface
> >
> >  -exposes regions and interrupts to user space via FDs  -there are
> 'info'
> > ioctls that allow getting info about
> >   regions/interrupts such as size and type of interrupt  -there is a
> > proposed extension to the 'info' ioctls that
> >   provides device tree paths, allowing user space to coorelate
> >   resources with the device tree
> >   (https://lists.cs.columbia.edu/pipermail/kvmarm/2013-
> July/006237.html)
> >
> > 5. QEMU
> >  -some key tasks
> >    -interacts with vfio kernel interface
> >    -registers memslots
> >    -needs to dynamically get device hooked up to VM's
> >     platform bus, including IRQs
> >    -needs to generate device tree node
> >  -a key point: we don't believe that platform device passthru
> >   in QEMU can be solved in a completely generic way.  There will
> >   need to be device specific code for each device type being passed
> >   through...to do things like generate device tree nodes  -in general
> we
> > expect a relatively small number of device types
> >   to be passed through to VMs
> >  -Alex Graf is working on dynamic creation of platform devices
> >   for the PPC e500 paravirt machine
> >  -see: http://lists.nongnu.org/archive/html/qemu-devel/2013-
> > 07/msg03614.html
> >  -first step is dynamically generating a virtual UART  -that sets the
> > stage to create and create vfio devices backed
> >   by real hardware
> >
> > There is a session at Linux Plumbers in a couple of weeks and further
> > discussions will happen there.
> >
> > Regards,
> > Stuart

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Will Deacon]

On Fri, Sep 06, 2013 at 07:20:19PM +0100, Yoder Stuart-B08248 wrote:
> Adding Will...

[...]

> > I have a query about the ARM SMMU driver. In the ARM smmu driver I see,
> > that bus notifiers are registered for both amba and platform bus. Amba is
> > the I/O interconnect, right? Why is bus notifier required for the amba
> > bus?

Not sure I follow the question, really. If you have a DMA master registered
as an AMBA device (e.g. PL330) and it wants to use an SMMU, then you need to
be registered on that bus.

What would you prefer instead?

Will

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Sethi Varun-B16395]

Hi Will,
Just trying to understand the scope of platform device assignment to guest on ARM. So, are the AMBA devices also represented in the device tree?

Regards
Varun

> -----Original Message-----
> From: kvm-ppc-owner@vger.kernel.org [mailto:kvm-ppc-
> owner@vger.kernel.org] On Behalf Of Will Deacon
> Sent: Monday, September 09, 2013 2:23 PM
> To: Yoder Stuart-B08248
> Cc: Sethi Varun-B16395; Wood Scott-B07421; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org';
> kvmarm@lists.cs.columbia.edu; kvm-ppc@vger.kernel.org; qemu-
> devel@nongnu.org
> Subject: Re: vfio for platform devices - 9/5/2012 - minutes
> 
> On Fri, Sep 06, 2013 at 07:20:19PM +0100, Yoder Stuart-B08248 wrote:
> > Adding Will...
> 
> [...]
> 
> > > I have a query about the ARM SMMU driver. In the ARM smmu driver I
> > > see, that bus notifiers are registered for both amba and platform
> > > bus. Amba is the I/O interconnect, right? Why is bus notifier
> > > required for the amba bus?
> 
> Not sure I follow the question, really. If you have a DMA master
> registered as an AMBA device (e.g. PL330) and it wants to use an SMMU,
> then you need to be registered on that bus.
> 
> What would you prefer instead?
> 
> Will
> --
> To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Yoder Stuart-B08248]

> -----Original Message-----
> From: Yoder Stuart-B08248
> Sent: Thursday, September 05, 2013 12:51 PM
> To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
> devel@nongnu.org'
> Subject: vfio for platform devices - 9/5/2012 - minutes
> 
> We had a call with those interested and/or working on vfio
> for platform devices.
> 
> Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
>               Santosh Shukla, Alex Williamson, Alexander Graf,
>               Antonios Motakis, Christoffer Dall, Kim Phillips,
>               Stuart Yoder
> 
> Several aspects to vfio for platform devices:
> 
> 1. IOMMU groups
> 
>  -iommu driver needs to register a bus notifier for the platform bus
>   and create groups for relevant platform devices
>  -Antonios is looking at this for several ARM IOMMUs
>  -PAMU (Freescale) driver already does this
> 
> 2. unbinding device from host
> 
>  PCI:
>    echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
>  Platform:
>    echo ffe101300.dma >
> /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> 
>  -don't believe there are issues or work to do here
> 
> 3. binding device to vfio-platform driver
> 
>  PCI:
>    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> 
>  -this is probably the least understood issue-- platform drivers
>   register themselves with the bus for a specific "name"
>   string.  That is matched with device tree "compatible" strings
>   later to bind a device to a driver
>  -we want is to have the vfio-platform driver to dynamically bind
>   to a variety of platform devices previously unknown to
>   vfio-platform
>  -ideally unbinding and binding could be an atomic operation
>  -Alex W pointed out that x86 could leverage this work so
>   keep that in mind in what we design
>  -Kim Phillips (Linaro) will start working on this

One thing we didn't discuss needs to be considered (probably by
Kim who is looking at the 'binding device' issue) is around
returning a passthru device back to the host.

After a platform device has been bound to vfio and is in use by
user space or a virtual machine, we also need to be able
to unwind all that and return the device back to the host
in a sane state.

What happens when user space exits and the vfio file
descriptors are closed?

What if the device is still active and doing bus 
mastering?   (e.g. a VM crashed causing a QEMU
exit)

How can the vfio-platform layer in the host kernel
get a specific device in a sane state?

When a plaform device is 'unbound' from vfio, what
specifically happens to the device?

Platform devices don't have generic mechanisms like on PCI
to disable bus mastering or even disable or reset a
device.

Haven't thought through all this yet, but just raising
some issues I see.

Regards,
Stuart

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Scott Wood]

On Wed, 2013-09-11 at 11:42 -0500, Yoder Stuart-B08248 wrote:
> One thing we didn't discuss needs to be considered (probably by
> Kim who is looking at the 'binding device' issue) is around
> returning a passthru device back to the host.
> 
> After a platform device has been bound to vfio and is in use by
> user space or a virtual machine, we also need to be able
> to unwind all that and return the device back to the host
> in a sane state.
> 
> What happens when user space exits and the vfio file
> descriptors are closed?
> 
> What if the device is still active and doing bus 
> mastering?   (e.g. a VM crashed causing a QEMU
> exit)
> 
> How can the vfio-platform layer in the host kernel
> get a specific device in a sane state?
> 
> When a plaform device is 'unbound' from vfio, what
> specifically happens to the device?
> 
> Platform devices don't have generic mechanisms like on PCI
> to disable bus mastering or even disable or reset a
> device.
> 
> Haven't thought through all this yet, but just raising
> some issues I see.

In the absence of some code in the host kernel that knows how to reset
the device, I think the model needs to be that devices start out with
the IOMMU blocking all transactions, and the user must ensure that the
device has been quiesced prior to enabling the device at the IOMMU.
This may require a hypercall or similar to let QEMU know when it's safe
to enable the device's IOMMU mappings (assuming that there's no virtual
IOMMU visible to the VM).

-Scott

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Alex Williamson]

On Wed, 2013-09-11 at 16:42 +0000, Yoder Stuart-B08248 wrote:
> 
> > -----Original Message-----
> > From: Yoder Stuart-B08248
> > Sent: Thursday, September 05, 2013 12:51 PM
> > To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> > Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> > 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> > Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
> > devel@nongnu.org'
> > Subject: vfio for platform devices - 9/5/2012 - minutes
> > 
> > We had a call with those interested and/or working on vfio
> > for platform devices.
> > 
> > Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
> >               Santosh Shukla, Alex Williamson, Alexander Graf,
> >               Antonios Motakis, Christoffer Dall, Kim Phillips,
> >               Stuart Yoder
> > 
> > Several aspects to vfio for platform devices:
> > 
> > 1. IOMMU groups
> > 
> >  -iommu driver needs to register a bus notifier for the platform bus
> >   and create groups for relevant platform devices
> >  -Antonios is looking at this for several ARM IOMMUs
> >  -PAMU (Freescale) driver already does this
> > 
> > 2. unbinding device from host
> > 
> >  PCI:
> >    echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
> >  Platform:
> >    echo ffe101300.dma >
> > /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> > 
> >  -don't believe there are issues or work to do here
> > 
> > 3. binding device to vfio-platform driver
> > 
> >  PCI:
> >    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> > 
> >  -this is probably the least understood issue-- platform drivers
> >   register themselves with the bus for a specific "name"
> >   string.  That is matched with device tree "compatible" strings
> >   later to bind a device to a driver
> >  -we want is to have the vfio-platform driver to dynamically bind
> >   to a variety of platform devices previously unknown to
> >   vfio-platform
> >  -ideally unbinding and binding could be an atomic operation
> >  -Alex W pointed out that x86 could leverage this work so
> >   keep that in mind in what we design
> >  -Kim Phillips (Linaro) will start working on this
> 
> One thing we didn't discuss needs to be considered (probably by
> Kim who is looking at the 'binding device' issue) is around
> returning a passthru device back to the host.
> 
> After a platform device has been bound to vfio and is in use by
> user space or a virtual machine, we also need to be able
> to unwind all that and return the device back to the host
> in a sane state.
> 
> What happens when user space exits and the vfio file
> descriptors are closed?

For reference, expectations of how vfio-pci handles these situations:

For vfio-pci, when the reference count on the device fd drops to zero we
call a device disable function that includes disabling the bus master
bit in config space stop ongoing DMA.

> What if the device is still active and doing bus 
> mastering?   (e.g. a VM crashed causing a QEMU
> exit)

If the VM crashes the vfio fds get released resulting in the above
opportunity for the vfio device driver to quiesce the device.

> How can the vfio-platform layer in the host kernel
> get a specific device in a sane state?

It's not easy on pci either.  We save config space prior to exposing the
device and restore config space later, but it's not complete.  We mostly
rely on device (function) resets, to put things in a sane state, but
those aren't always supported.  I just introduced patches for v3.12 that
enable a PCI bus reset interface, but it's mostly useful for userspace,
since on PCI it's often the case that a bus contains multiple devices
which don't necessarily align to iommu group boundaries.

> When a plaform device is 'unbound' from vfio, what
> specifically happens to the device?

The driver remove function is called, which needs to call
vfio_del_group_dev(), which will block if the device is in use.  If it's
unused, the release function should have already been called and the
device should be in a quiesced state.  On x86/pci, there is no auto
probe of devices when unbound, so the device sits without a driver until
manually bound or until something causes an auto probe.  Thanks,

Alex

> Platform devices don't have generic mechanisms like on PCI
> to disable bus mastering or even disable or reset a
> device.
> 
> Haven't thought through all this yet, but just raising
> some issues I see.
> 
> Regards,
> Stuart
> 
> 
> 
> 

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Bhushan Bharat-R65777]

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Wednesday, September 11, 2013 10:45 PM
> To: Yoder Stuart-B08248
> Cc: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alexander Graf'; 'Antonios Motakis'; 'Christoffer
> Dall'; 'kim.phillips@linaro.org'; kvmarm@lists.cs.columbia.edu; kvm-
> ppc@vger.kernel.org; qemu-devel@nongnu.org
> Subject: Re: vfio for platform devices - 9/5/2012 - minutes
> 
> On Wed, 2013-09-11 at 16:42 +0000, Yoder Stuart-B08248 wrote:
> >
> > > -----Original Message-----
> > > From: Yoder Stuart-B08248
> > > Sent: Thursday, September 05, 2013 12:51 PM
> > > To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777;
> > > 'Peter Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander
> > > Graf'; 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> > > Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
> > > devel@nongnu.org'
> > > Subject: vfio for platform devices - 9/5/2012 - minutes
> > >
> > > We had a call with those interested and/or working on vfio for
> > > platform devices.
> > >
> > > Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
> > >               Santosh Shukla, Alex Williamson, Alexander Graf,
> > >               Antonios Motakis, Christoffer Dall, Kim Phillips,
> > >               Stuart Yoder
> > >
> > > Several aspects to vfio for platform devices:
> > >
> > > 1. IOMMU groups
> > >
> > >  -iommu driver needs to register a bus notifier for the platform bus
> > >   and create groups for relevant platform devices  -Antonios is
> > > looking at this for several ARM IOMMUs  -PAMU (Freescale) driver
> > > already does this
> > >
> > > 2. unbinding device from host
> > >
> > >  PCI:
> > >    echo 0000:06:0d.0 >
> > > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
> > >  Platform:
> > >    echo ffe101300.dma >
> > > /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> > >
> > >  -don't believe there are issues or work to do here
> > >
> > > 3. binding device to vfio-platform driver
> > >
> > >  PCI:
> > >    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> > >
> > >  -this is probably the least understood issue-- platform drivers
> > >   register themselves with the bus for a specific "name"
> > >   string.  That is matched with device tree "compatible" strings
> > >   later to bind a device to a driver  -we want is to have the
> > > vfio-platform driver to dynamically bind
> > >   to a variety of platform devices previously unknown to
> > >   vfio-platform
> > >  -ideally unbinding and binding could be an atomic operation  -Alex
> > > W pointed out that x86 could leverage this work so
> > >   keep that in mind in what we design  -Kim Phillips (Linaro) will
> > > start working on this
> >
> > One thing we didn't discuss needs to be considered (probably by Kim
> > who is looking at the 'binding device' issue) is around returning a
> > passthru device back to the host.
> >
> > After a platform device has been bound to vfio and is in use by user
> > space or a virtual machine, we also need to be able to unwind all that
> > and return the device back to the host in a sane state.
> >
> > What happens when user space exits and the vfio file descriptors are
> > closed?
> 
> For reference, expectations of how vfio-pci handles these situations:
> 
> For vfio-pci, when the reference count on the device fd drops to zero we call a
> device disable function that includes disabling the bus master bit in config
> space stop ongoing DMA.

There is no bus mastering for platform devices and device disabling is not a standard like PCI. Do you think that we might need to do some device specific handling. For example for DMA controller, we need to atleast disable the DMA controller and mask its interrupts (may be ensure that there is no pending/running dma transaction, release irqs etc). As we are not yet having any linkage to respective kernel driver then I am not sure how we will do that specific handling?

> 
> > What if the device is still active and doing bus
> > mastering?   (e.g. a VM crashed causing a QEMU
> > exit)
> 
> If the VM crashes the vfio fds get released resulting in the above opportunity
> for the vfio device driver to quiesce the device.

I think the quiesing of devices with we device specific, so the generic vfio=platform driver may not be able to handle that, right?

> 
> > How can the vfio-platform layer in the host kernel get a specific
> > device in a sane state?
> 
> It's not easy on pci either.  We save config space prior to exposing the device
> and restore config space later, but it's not complete.  We mostly rely on device
> (function) resets, to put things in a sane state, but those aren't always
> supported.

All platform devices also may not have reset capability (and if any then not generic way for all devices).

>  I just introduced patches for v3.12 that enable a PCI bus reset
> interface, but it's mostly useful for userspace, since on PCI it's often the
> case that a bus contains multiple devices which don't necessarily align to iommu
> group boundaries.
> 
> > When a plaform device is 'unbound' from vfio, what specifically
> > happens to the device?
> 
> The driver remove function is called, which needs to call vfio_del_group_dev(),
> which will block if the device is in use.  If it's unused, the release function
> should have already been called and the device should be in a quiesced state.

What does it mean by quiesced state here? Does this mean:
 - Device does not take more work from software; This will done if device is not attached to any driver.
 - Device does not create work for software (no interrupt etc). This can achieved if interrupts masked at device/PIC and device in disabled.

Thanks
-Bharat

> On x86/pci, there is no auto probe of devices when unbound, so the device sits
> without a driver until manually bound or until something causes an auto probe.
> Thanks,
> 
> Alex
> 
> > Platform devices don't have generic mechanisms like on PCI to disable
> > bus mastering or even disable or reset a device.
> >
> > Haven't thought through all this yet, but just raising some issues I
> > see.
> >
> > Regards,
> > Stuart
> >
> >
> >
> >
> 
> 
> 

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Scott Wood]

On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
> 
> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: Wednesday, September 11, 2013 10:45 PM
> > To: Yoder Stuart-B08248
> > Cc: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> > Maydell'; 'Santosh Shukla'; 'Alexander Graf'; 'Antonios Motakis'; 'Christoffer
> > Dall'; 'kim.phillips@linaro.org'; kvmarm@lists.cs.columbia.edu; kvm-
> > ppc@vger.kernel.org; qemu-devel@nongnu.org
> > Subject: Re: vfio for platform devices - 9/5/2012 - minutes
> > 
> > On Wed, 2013-09-11 at 16:42 +0000, Yoder Stuart-B08248 wrote:
> > >
> > > > -----Original Message-----
> > > > From: Yoder Stuart-B08248
> > > > Sent: Thursday, September 05, 2013 12:51 PM
> > > > To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777;
> > > > 'Peter Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander
> > > > Graf'; 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> > > > Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
> > > > devel@nongnu.org'
> > > > Subject: vfio for platform devices - 9/5/2012 - minutes
> > > >
> > > > We had a call with those interested and/or working on vfio for
> > > > platform devices.
> > > >
> > > > Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
> > > >               Santosh Shukla, Alex Williamson, Alexander Graf,
> > > >               Antonios Motakis, Christoffer Dall, Kim Phillips,
> > > >               Stuart Yoder
> > > >
> > > > Several aspects to vfio for platform devices:
> > > >
> > > > 1. IOMMU groups
> > > >
> > > >  -iommu driver needs to register a bus notifier for the platform bus
> > > >   and create groups for relevant platform devices  -Antonios is
> > > > looking at this for several ARM IOMMUs  -PAMU (Freescale) driver
> > > > already does this
> > > >
> > > > 2. unbinding device from host
> > > >
> > > >  PCI:
> > > >    echo 0000:06:0d.0 >
> > > > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
> > > >  Platform:
> > > >    echo ffe101300.dma >
> > > > /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> > > >
> > > >  -don't believe there are issues or work to do here
> > > >
> > > > 3. binding device to vfio-platform driver
> > > >
> > > >  PCI:
> > > >    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > >
> > > >  -this is probably the least understood issue-- platform drivers
> > > >   register themselves with the bus for a specific "name"
> > > >   string.  That is matched with device tree "compatible" strings
> > > >   later to bind a device to a driver  -we want is to have the
> > > > vfio-platform driver to dynamically bind
> > > >   to a variety of platform devices previously unknown to
> > > >   vfio-platform
> > > >  -ideally unbinding and binding could be an atomic operation  -Alex
> > > > W pointed out that x86 could leverage this work so
> > > >   keep that in mind in what we design  -Kim Phillips (Linaro) will
> > > > start working on this
> > >
> > > One thing we didn't discuss needs to be considered (probably by Kim
> > > who is looking at the 'binding device' issue) is around returning a
> > > passthru device back to the host.
> > >
> > > After a platform device has been bound to vfio and is in use by user
> > > space or a virtual machine, we also need to be able to unwind all that
> > > and return the device back to the host in a sane state.
> > >
> > > What happens when user space exits and the vfio file descriptors are
> > > closed?
> > 
> > For reference, expectations of how vfio-pci handles these situations:
> > 
> > For vfio-pci, when the reference count on the device fd drops to zero we call a
> > device disable function that includes disabling the bus master bit in config
> > space stop ongoing DMA.
> 
> There is no bus mastering for platform devices

Sure there is -- that's just PCI's name for DMA.

> and device disabling is not a standard like PCI. Do you think that we might need to do some
> device specific handling.

It would be better if we could avoid the need for device specific
handling.  DMA can be disabled via the IOMMU.

The tricky part is how to enable the device in the next user; as I wrote
elsewhere in the thread I think this needs to be something the user is
aware of, so that the user quiesces the device prior to enabling IOMMU
mappings.

> >  I just introduced patches for v3.12 that enable a PCI bus reset
> > interface, but it's mostly useful for userspace, since on PCI it's often the
> > case that a bus contains multiple devices which don't necessarily align to iommu
> > group boundaries.
> > 
> > > When a plaform device is 'unbound' from vfio, what specifically
> > > happens to the device?
> > 
> > The driver remove function is called, which needs to call vfio_del_group_dev(),
> > which will block if the device is in use.  If it's unused, the release function
> > should have already been called and the device should be in a quiesced state.
> 
> What does it mean by quiesced state here? Does this mean:
>  - Device does not take more work from software; This will done if device is not attached to any driver.
>  - Device does not create work for software (no interrupt etc). This can achieved if interrupts masked at device/PIC and device in disabled.

It at least means that the device will not do any DMA or generate any
interrupts.

-Scott

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Alexander Graf]

On 12.09.2013, at 13:10, Scott Wood wrote:

> On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
>> 
>>> -----Original Message-----
>>> From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>> Sent: Wednesday, September 11, 2013 10:45 PM
>>> To: Yoder Stuart-B08248
>>> Cc: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
>>> Maydell'; 'Santosh Shukla'; 'Alexander Graf'; 'Antonios Motakis'; 'Christoffer
>>> Dall'; 'kim.phillips@linaro.org'; kvmarm@lists.cs.columbia.edu; kvm-
>>> ppc@vger.kernel.org; qemu-devel@nongnu.org
>>> Subject: Re: vfio for platform devices - 9/5/2012 - minutes
>>> 
>>> On Wed, 2013-09-11 at 16:42 +0000, Yoder Stuart-B08248 wrote:
>>>> 
>>>>> -----Original Message-----
>>>>> From: Yoder Stuart-B08248
>>>>> Sent: Thursday, September 05, 2013 12:51 PM
>>>>> To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777;
>>>>> 'Peter Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander
>>>>> Graf'; 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
>>>>> Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
>>>>> devel@nongnu.org'
>>>>> Subject: vfio for platform devices - 9/5/2012 - minutes
>>>>> 
>>>>> We had a call with those interested and/or working on vfio for
>>>>> platform devices.
>>>>> 
>>>>> Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
>>>>>              Santosh Shukla, Alex Williamson, Alexander Graf,
>>>>>              Antonios Motakis, Christoffer Dall, Kim Phillips,
>>>>>              Stuart Yoder
>>>>> 
>>>>> Several aspects to vfio for platform devices:
>>>>> 
>>>>> 1. IOMMU groups
>>>>> 
>>>>> -iommu driver needs to register a bus notifier for the platform bus
>>>>>  and create groups for relevant platform devices  -Antonios is
>>>>> looking at this for several ARM IOMMUs  -PAMU (Freescale) driver
>>>>> already does this
>>>>> 
>>>>> 2. unbinding device from host
>>>>> 
>>>>> PCI:
>>>>>   echo 0000:06:0d.0 >
>>>>> /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
>>>>> Platform:
>>>>>   echo ffe101300.dma >
>>>>> /sys/bus/platform/devices/ffe101300.dma/driver/unbind
>>>>> 
>>>>> -don't believe there are issues or work to do here
>>>>> 
>>>>> 3. binding device to vfio-platform driver
>>>>> 
>>>>> PCI:
>>>>>   echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
>>>>> 
>>>>> -this is probably the least understood issue-- platform drivers
>>>>>  register themselves with the bus for a specific "name"
>>>>>  string.  That is matched with device tree "compatible" strings
>>>>>  later to bind a device to a driver  -we want is to have the
>>>>> vfio-platform driver to dynamically bind
>>>>>  to a variety of platform devices previously unknown to
>>>>>  vfio-platform
>>>>> -ideally unbinding and binding could be an atomic operation  -Alex
>>>>> W pointed out that x86 could leverage this work so
>>>>>  keep that in mind in what we design  -Kim Phillips (Linaro) will
>>>>> start working on this
>>>> 
>>>> One thing we didn't discuss needs to be considered (probably by Kim
>>>> who is looking at the 'binding device' issue) is around returning a
>>>> passthru device back to the host.
>>>> 
>>>> After a platform device has been bound to vfio and is in use by user
>>>> space or a virtual machine, we also need to be able to unwind all that
>>>> and return the device back to the host in a sane state.
>>>> 
>>>> What happens when user space exits and the vfio file descriptors are
>>>> closed?
>>> 
>>> For reference, expectations of how vfio-pci handles these situations:
>>> 
>>> For vfio-pci, when the reference count on the device fd drops to zero we call a
>>> device disable function that includes disabling the bus master bit in config
>>> space stop ongoing DMA.
>> 
>> There is no bus mastering for platform devices
> 
> Sure there is -- that's just PCI's name for DMA.
> 
>> and device disabling is not a standard like PCI. Do you think that we might need to do some
>> device specific handling.
> 
> It would be better if we could avoid the need for device specific
> handling.  DMA can be disabled via the IOMMU.
> 
> The tricky part is how to enable the device in the next user; as I wrote
> elsewhere in the thread I think this needs to be something the user is
> aware of, so that the user quiesces the device prior to enabling IOMMU
> mappings.

Or resets it. I think we can make it mandatory to VFIO users (and Linux drivers) to reset the device prior to mapping it. QEMU needs to have device specific code to assemble its device tree fragment anyways, so it can just as well reset the device before it enables it to DMA.

Alex

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Scott Wood]

On Thu, 2013-09-12 at 16:23 -0500, Alexander Graf wrote:
> On 12.09.2013, at 13:10, Scott Wood wrote:
> 
> > On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
> >> and device disabling is not a standard like PCI. Do you think that we might need to do some
> >> device specific handling.
> > 
> > It would be better if we could avoid the need for device specific
> > handling.  DMA can be disabled via the IOMMU.
> > 
> > The tricky part is how to enable the device in the next user; as I wrote
> > elsewhere in the thread I think this needs to be something the user is
> > aware of, so that the user quiesces the device prior to enabling IOMMU
> > mappings.
> 
> Or resets it. I think we can make it mandatory to VFIO users (and Linux
> drivers) to reset the device prior to mapping it.

Reset is one way to quiesce it, but as Bharat pointed out not all
devices make that possible (even with device specific knowledge).  In
particular, Freescale datapath stuff has problems with this.

> QEMU needs to have device specific code to assemble its device tree fragment anyways, so
> it can just as well reset the device before it enables it to DMA.

Now you're adding to the depth of knowledge QEMU needs to have about a
device.  It's not that big of a deal if it's just "set this bit and wait
for it to clear", but it could be a pain with a device that can't be
reset and has a complicated sequence needed to quiesce -- and as you
noted above, Linux drivers would need to take care of this anyway.
Doing it in QEMU would still help with non-Linux guests and avoid the
need for a hypercall to indicate when the IOMMU can be opened, though.

-Scott

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Alexander Graf]

On 12.09.2013, at 16:45, Scott Wood wrote:

> On Thu, 2013-09-12 at 16:23 -0500, Alexander Graf wrote:
>> On 12.09.2013, at 13:10, Scott Wood wrote:
>> 
>>> On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
>>>> and device disabling is not a standard like PCI. Do you think that we might need to do some
>>>> device specific handling.
>>> 
>>> It would be better if we could avoid the need for device specific
>>> handling.  DMA can be disabled via the IOMMU.
>>> 
>>> The tricky part is how to enable the device in the next user; as I wrote
>>> elsewhere in the thread I think this needs to be something the user is
>>> aware of, so that the user quiesces the device prior to enabling IOMMU
>>> mappings.
>> 
>> Or resets it. I think we can make it mandatory to VFIO users (and Linux
>> drivers) to reset the device prior to mapping it.
> 
> Reset is one way to quiesce it, but as Bharat pointed out not all
> devices make that possible (even with device specific knowledge).  In
> particular, Freescale datapath stuff has problems with this.

I'm sure once Freescale datapath drivers are upstream in Linux we can also get that code upstream in QEMU :).

>> QEMU needs to have device specific code to assemble its device tree fragment anyways, so
>> it can just as well reset the device before it enables it to DMA.
> 
> Now you're adding to the depth of knowledge QEMU needs to have about a
> device.  It's not that big of a deal if it's just "set this bit and wait
> for it to clear", but it could be a pain with a device that can't be
> reset and has a complicated sequence needed to quiesce -- and as you
> noted above, Linux drivers would need to take care of this anyway.
> Doing it in QEMU would still help with non-Linux guests and avoid the
> need for a hypercall to indicate when the IOMMU can be opened, though.

Yes, and you don't want to burden the Linux driver with a situation it doesn't face in non-virtualized environments :).


Alex

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Scott Wood]

On Thu, 2013-09-12 at 16:48 -0500, Alexander Graf wrote:
> On 12.09.2013, at 16:45, Scott Wood wrote:
> 
> > On Thu, 2013-09-12 at 16:23 -0500, Alexander Graf wrote:
> >> On 12.09.2013, at 13:10, Scott Wood wrote:
> >> 
> >>> On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
> >>>> and device disabling is not a standard like PCI. Do you think that we might need to do some
> >>>> device specific handling.
> >>> 
> >>> It would be better if we could avoid the need for device specific
> >>> handling.  DMA can be disabled via the IOMMU.
> >>> 
> >>> The tricky part is how to enable the device in the next user; as I wrote
> >>> elsewhere in the thread I think this needs to be something the user is
> >>> aware of, so that the user quiesces the device prior to enabling IOMMU
> >>> mappings.
> >> 
> >> Or resets it. I think we can make it mandatory to VFIO users (and Linux
> >> drivers) to reset the device prior to mapping it.
> > 
> > Reset is one way to quiesce it, but as Bharat pointed out not all
> > devices make that possible (even with device specific knowledge).  In
> > particular, Freescale datapath stuff has problems with this.
> 
> I'm sure once Freescale datapath drivers are upstream in Linux we can also get that code upstream in QEMU :).

Yeah, but the fewer places it lives, the better. :-)

> >> QEMU needs to have device specific code to assemble its device tree fragment anyways, so
> >> it can just as well reset the device before it enables it to DMA.
> > 
> > Now you're adding to the depth of knowledge QEMU needs to have about a
> > device.  It's not that big of a deal if it's just "set this bit and wait
> > for it to clear", but it could be a pain with a device that can't be
> > reset and has a complicated sequence needed to quiesce -- and as you
> > noted above, Linux drivers would need to take care of this anyway.
> > Doing it in QEMU would still help with non-Linux guests and avoid the
> > need for a hypercall to indicate when the IOMMU can be opened, though.
> 
> Yes, and you don't want to burden the Linux driver with a situation it doesn't face in non-virtualized environments :).

The driver does need to handle it in non-virtualized environments, if
you want to be able to return a device to the host driver after a VFIO
user has had it.  It will also be needed for kexec.

-Scott