Re: [PATCH v4 2/5] xen/pciback: Don't deadlock when unbinding.

[Boris Ostrovsky <boris.ostrovsky@oracle.com>]

On 07/14/2014 10:13 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Jul 11, 2014 at 05:02:01PM -0400, Konrad Rzeszutek Wilk wrote:
>>>> --- a/drivers/xen/xen-pciback/pci_stub.c
>>>> +++ b/drivers/xen/xen-pciback/pci_stub.c
>>>> @@ -250,6 +250,8 @@ struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev,
>>>>    *  - 'echo BDF > unbind' with a guest still using it. See pcistub_remove
>>>>    *
>>>>    *  As such we have to be careful.
>>>> + *
>>>> + *  To make this easier, the caller has to hold the device lock.
>>> Should we assert that the lock is being held?
>> Yes of course we should. Thank you!
> How about this:
>
>  From 388a03c598218dac8bfeb6c5bf3992e0d1e37d1e Mon Sep 17 00:00:00 2001
> From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> Date: Tue, 8 Jul 2014 11:12:02 -0400
> Subject: [PATCH] xen/pciback: Don't deadlock when unbinding.
>
> As commit 0a9fd0152929db372ff61b0d6c280fdd34ae8bdb
> 'xen/pciback: Document the entry points for 'pcistub_put_pci_dev''
> explained there are four entry points in this function.
> Two of them are when the user fiddles in the SysFS to
> unbind a device which might be in use by a guest or not.
>
> Both 'unbind' states will cause a deadlock as the the PCI lock has
> already been taken, which then pci_device_reset tries to take.
>
> We can simplify this by requiring that all callers of
> pcistub_put_pci_dev MUST hold the device lock. And then
> we can just call the lockless version of pci_device_reset.
>
> To make it even simpler we will modify xen_pcibk_release_pci_dev
> to quality whether it should take a lock or not - as it ends
> up calling xen_pcibk_release_pci_dev and needs to hold the lock.
>
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
> [v2: Per David Vrabel's suggestion - use lockless version of reset]
> [v3: Per Boris suggestion add assertion mechanism]
> ---
>   drivers/xen/xen-pciback/passthrough.c |  9 +++++++--
>   drivers/xen/xen-pciback/pci_stub.c    | 12 ++++++------
>   drivers/xen/xen-pciback/pciback.h     |  7 ++++---
>   drivers/xen/xen-pciback/vpci.c        |  9 +++++++--
>   drivers/xen/xen-pciback/xenbus.c      |  2 +-
>   5 files changed, 25 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/xen/xen-pciback/passthrough.c b/drivers/xen/xen-pciback/passthrough.c
> index 828dddc..d0c3fb4 100644
> --- a/drivers/xen/xen-pciback/passthrough.c
> +++ b/drivers/xen/xen-pciback/passthrough.c
> @@ -69,7 +69,7 @@ static int __xen_pcibk_add_pci_dev(struct xen_pcibk_device *pdev,
>   }
>   
>   static void __xen_pcibk_release_pci_dev(struct xen_pcibk_device *pdev,
> -					struct pci_dev *dev)
> +					struct pci_dev *dev, bool lock)
>   {
>   	struct passthrough_dev_data *dev_data = pdev->pci_dev_data;
>   	struct pci_dev_entry *dev_entry, *t;
> @@ -87,8 +87,13 @@ static void __xen_pcibk_release_pci_dev(struct xen_pcibk_device *pdev,
>   
>   	mutex_unlock(&dev_data->lock);
>   
> -	if (found_dev)
> +	if (found_dev) {
> +		if (lock)
> +			device_lock(&found_dev->dev);
>   		pcistub_put_pci_dev(found_dev);
> +		if (lock)
> +			device_unlock(&found_dev->dev);
> +	}
>   }
>   
>   static int __xen_pcibk_init_devices(struct xen_pcibk_device *pdev)
> diff --git a/drivers/xen/xen-pciback/pci_stub.c b/drivers/xen/xen-pciback/pci_stub.c
> index d57a173..8293fbb 100644
> --- a/drivers/xen/xen-pciback/pci_stub.c
> +++ b/drivers/xen/xen-pciback/pci_stub.c
> @@ -250,6 +250,8 @@ struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev,
>    *  - 'echo BDF > unbind' with a guest still using it. See pcistub_remove
>    *
>    *  As such we have to be careful.
> + *
> + *  To make this easier, the caller has to hold the device lock.
>    */
>   void pcistub_put_pci_dev(struct pci_dev *dev)
>   {
> @@ -276,11 +278,8 @@ void pcistub_put_pci_dev(struct pci_dev *dev)
>   	/* Cleanup our device
>   	 * (so it's ready for the next domain)
>   	 */
> -
> -	/* This is OK - we are running from workqueue context
> -	 * and want to inhibit the user from fiddling with 'reset'
> -	 */
> -	pci_reset_function(dev);
> +	lockdep_assert_held(&dev->dev.mutex);
> +	__pci_reset_function_locked(dev);
>   	pci_restore_state(dev);

Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>

(Although I wonder about the fact that we are exposing the mutex which 
is typically hidden by device_lock()/unlock() inlines. Have you 
considered adding something like is_device_locked() to device.h?)

-boris