Re: KVM vs. PCI-E Function Level Reset (FLR) ...

[Sheng Yang <sheng@linux.intel.com>]

On Thursday 15 July 2010 02:01:29 Casey Leedom wrote:
> | From: Sheng Yang <sheng@linux.intel.com>
> | Date: Tuesday, July 13, 2010 05:53 pm

(Please use reply to all next time.)
> | 
> | On Wednesday 14 July 2010 04:41:01 Casey Leedom wrote:
> | >   It looks like the Linux KVM kernel support code issues an FLR against
> | > 
> | > an Assigned Device when the device is assigned and when it's freed
> | > but it's not clear when those actions are taken.  For instance,
> | > if a device is assigned to a VM and then the VM reboots itself, is
> | > that counted as another assignment point? I.e. will KVM issue a
> | > new pci_reset_function() on the device so it shows up reset in
> | > the newly rebooted VM?
> | 
> | The assignment/deassignment happened when guest was created/destroyed.
> | Currently it wouldn't issue a FLR when guest reset.
>   Hhrrmmm, this seems like a semantic error.  "Resetting" the Vm should be
> morally equivalent to resetting a real physical machine.  And when a real
> physical machine is reset, all of its buses, etc. get reset which
> propagates to Device Resets on those buses ...  I think that Assigned
> Devices should be reset for reboots and power off/on ...

Yes, it should be done when reset. And power on/off should be there, because that's 
means the create and destroy the guest.
> 
> | >   And what happens if the VM OS/Driver attempts to write the PCI Pass
> | > 
> | > Through Device's PCI-E FLR bit?  I assume that that write (and the
> | > following polling reads) are trapped by the KVM code but I can't find
> | > the code which implements the PCI Configuration Space emulation to see
> | > if the FLR is implemented there.  For instance, if I run Linux 2.6.30
> | > in the VM and my Device Driver calls pci_reset_function() in its
> | > "probe()" function will that result in a Device FLR? it doesn't appear
> | > to be the case ...
> | 
> | The PCI configuration space emulated is in QEmu rather than KVM. You can
> | check qemu-kvm/hw/device-assignment.c. We didn't emulate FLR capability
> | now. (OK, some other device specific reset method may involved, you can
> | check pci_dev_reset())
> 
>   Okay, I think that this is also going to be an issue for supporting
> Assigned Devices.  For PCI-E SR-IOV Virtual Functions which are assigned
> to a VM, they need to be reset at reboot/power off/power on and the
> Configuration Space emulation needs to support the Guest OS/VM Device
> Driver issuing an FLR ...

You can add the FLR support for your device if you need to call it in the guest. 
But I don't quite understand why you need to call it in the guest. KVM should has 
already done that, and it's not necessary for native device.

> 
> | >       Note that it's impossible for a Device Driver to call
> | >       pci_reset_function() under Linux 2.6.31 and later
> | >       because a call to device_lock() was added to
> | >       pci_dev_reset() in chageset 8e9394ce on Feb
> | >       17, 2010 by Greg Kroah-Hartman.  This means
> | >       that a call to pci_reset_function() in a device
> | >       driver's "probe()" routine will result in an
> | >       immediate deadlock.
> | 
> | What I saw the code is like this:
> | 
> | static int pci_dev_reset(struct pci_dev *dev, int probe)
> | {
> | 
> |         int rc;
> |         
> |         might_sleep();
> |         
> |         if (!probe) {
> |         
> |                 pci_block_user_cfg_access(dev);
> |                 /* block PM suspend, driver probe, etc. */
> |                 device_lock(&dev->dev);
> |         
> |         }
> | 
> | [...]
> | 
> | So seems it's fine with _probe_ set, to use with probe() routine.
> 
>   You're looking at a local routine in drivers/pci/pci.c.  That routine is
> called twice in pci_reset_function().  The "probe" parameter is used to
> indicate whether the caller wants to "probe" for the ability to perform a
> PCI Function Reset or to actually _do_ the reset.  pci_reset_function()
> first calls pci_dev_reset() is probe=1 and, if that returns an error code,
> it returns immediately with the error.  Otherwise it saves the PCI State
> of the device, makes another call to pci_dev_reset() with probe=0, and
> then restores the device's PCI State.  Thus, this "probe" in
> pci_dev_reset() doesn't have anything to do with the possibility that a
> device's own (pci_dev *)->driver->probe() routine happens to be calling
> pci_reset_function().  Since, apparently, the device's own ...->probe()
> routine is called with the device's (pci_dev *)->lock held, a call to
> pci_reset_function() on itself will result in an immediate deadlock from
> Linux 2.6.31 on ...

So you want to issue FLR(rather than probing the feature) when driver->probe() 
called? Current seems KVM is the only user of pci_reset_function(), so I think we 
can update it after your driver posted to the mailing list. Also I am not sure why 
you want to reset function when probing. KVM should cover them all I think.

--
regards
Yang, Sheng

> 
> Casey
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html