Re: KVM vs. PCI-E Function Level Reset (FLR) ...

[Casey Leedom <leedom@chelsio.com>]

| From: Sheng Yang <sheng@linux.intel.com>
| Date: Tuesday, July 13, 2010 05:53 pm
| 
| On Wednesday 14 July 2010 04:41:01 Casey Leedom wrote:
| > 
| >   It looks like the Linux KVM kernel support code issues an FLR against
| > an Assigned Device when the device is assigned and when it's freed
| > but it's not clear when those actions are taken.  For instance,
| > if a device is assigned to a VM and then the VM reboots itself, is
| > that counted as another assignment point? I.e. will KVM issue a
| > new pci_reset_function() on the device so it shows up reset in
| > the newly rebooted VM?
| 
| The assignment/deassignment happened when guest was created/destroyed.
| Currently it wouldn't issue a FLR when guest reset.

  Hhrrmmm, this seems like a semantic error.  "Resetting" the Vm should be 
morally equivalent to resetting a real physical machine.  And when a real 
physical machine is reset, all of its buses, etc. get reset which propagates to 
Device Resets on those buses ...  I think that Assigned Devices should be reset 
for reboots and power off/on ...

| >   And what happens if the VM OS/Driver attempts to write the PCI Pass
| > Through Device's PCI-E FLR bit?  I assume that that write (and the
| > following polling reads) are trapped by the KVM code but I can't find the
| > code which implements the PCI Configuration Space emulation to see if the
| > FLR is implemented there.  For instance, if I run Linux 2.6.30 in the VM
| > and my Device Driver calls pci_reset_function() in its "probe()" function
| > will that result in a Device FLR? it doesn't appear to be the case ...
| 
| The PCI configuration space emulated is in QEmu rather than KVM. You can
| check qemu-kvm/hw/device-assignment.c. We didn't emulate FLR capability
| now. (OK, some other device specific reset method may involved, you can
| check pci_dev_reset())

  Okay, I think that this is also going to be an issue for supporting Assigned 
Devices.  For PCI-E SR-IOV Virtual Functions which are assigned to a VM, they 
need to be reset at reboot/power off/power on and the Configuration Space 
emulation needs to support the Guest OS/VM Device Driver issuing an FLR ...

| >       Note that it's impossible for a Device Driver to call 
| >       pci_reset_function() under Linux 2.6.31 and later
| >       because a call to device_lock() was added to
| >       pci_dev_reset() in chageset 8e9394ce on Feb
| >       17, 2010 by Greg Kroah-Hartman.  This means
| >       that a call to pci_reset_function() in a device
| >       driver's "probe()" routine will result in an
| >       immediate deadlock.
| 
| What I saw the code is like this:
| 
| static int pci_dev_reset(struct pci_dev *dev, int probe)
| {
|         int rc;
| 
|         might_sleep();
| 
|         if (!probe) {
|                 pci_block_user_cfg_access(dev);
|                 /* block PM suspend, driver probe, etc. */
|                 device_lock(&dev->dev);
|         }
| [...]
| 
| So seems it's fine with _probe_ set, to use with probe() routine.

  You're looking at a local routine in drivers/pci/pci.c.  That routine is 
called twice in pci_reset_function().  The "probe" parameter is used to indicate 
whether the caller wants to "probe" for the ability to perform a PCI Function 
Reset or to actually _do_ the reset.  pci_reset_function() first calls 
pci_dev_reset() is probe=1 and, if that returns an error code, it returns 
immediately with the error.  Otherwise it saves the PCI State of the device, 
makes another call to pci_dev_reset() with probe=0, and then restores the 
device's PCI State.  Thus, this "probe" in pci_dev_reset() doesn't have anything 
to do with the possibility that a device's own (pci_dev *)->driver->probe() 
routine happens to be calling pci_reset_function().  Since, apparently, the 
device's own ...->probe() routine is called with the device's (pci_dev *)->lock 
held, a call to pci_reset_function() on itself will result in an immediate 
deadlock from Linux 2.6.31 on ...

Casey