[Qemu-devel] [PATCH] vfio: fix duplicate function call

[Qemu-devel] [PATCH] vfio: fix duplicate function call

[Cao jin]

When vfio device is reset(encounter FLR, or bus reset), if need to do
bus reset(vfio_pci_hot_reset_one is called), vfio_pci_pre_reset &
vfio_pci_post_reset will be called twice.

Signed-off-by: Cao jin <caoj.fnst@cn.fujitsu.com>
---
Also has a little question on vfio_pci_reset. it will be called when encounter
bus reset, or FLR. The reset method's priority in this function now is:

    1. If has "device specific reset function", then do it
    2. If has FLR, then do it.
    3. If it can do bus reset(only 1 affected device), then do it
    4. If has pm_reset, then do it

The question is: why pm reset has low priority than bus reset(if it does
can do a bus reset)? why bus reset is not the last choice? In PCI driver
of kernel, pls see __pci_dev_reset, we can see, if device support pm reset,
it won't do bus reset.

 hw/vfio/pci.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index cce3024..ca4d1c1 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -1930,7 +1930,9 @@ static int vfio_pci_hot_reset(VFIOPCIDevice *vdev, bool single)
 
     trace_vfio_pci_hot_reset(vdev->vbasedev.name, single ? "one" : "multi");
 
-    vfio_pci_pre_reset(vdev);
+    if (!single) {
+        vfio_pci_pre_reset(vdev);
+    }
     vdev->vbasedev.needs_reset = false;
 
     info = g_malloc0(sizeof(*info));
@@ -2088,7 +2090,9 @@ out:
         }
     }
 out_single:
-    vfio_pci_post_reset(vdev);
+    if (!single) {
+        vfio_pci_post_reset(vdev);
+    }
     g_free(info);
 
     return ret;
-- 
2.1.0

Re: [Qemu-devel] [PATCH] vfio: fix duplicate function call

[Alex Williamson]

On Fri, 14 Oct 2016 19:16:59 +0800
Cao jin <caoj.fnst@cn.fujitsu.com> wrote:

> When vfio device is reset(encounter FLR, or bus reset), if need to do
> bus reset(vfio_pci_hot_reset_one is called), vfio_pci_pre_reset &
> vfio_pci_post_reset will be called twice.
> 
> Signed-off-by: Cao jin <caoj.fnst@cn.fujitsu.com>
> ---
> Also has a little question on vfio_pci_reset. it will be called when encounter
> bus reset, or FLR. The reset method's priority in this function now is:
> 
>     1. If has "device specific reset function", then do it
>     2. If has FLR, then do it.
>     3. If it can do bus reset(only 1 affected device), then do it
>     4. If has pm_reset, then do it
> 
> The question is: why pm reset has low priority than bus reset(if it does
> can do a bus reset)? why bus reset is not the last choice? In PCI driver
> of kernel, pls see __pci_dev_reset, we can see, if device support pm reset,
> it won't do bus reset.

The PCI spec doesn't really define what sort of reset is done with a PM
reset.  My thinking was that if a device advertises an FLR capability
then the hardware has made a concerted effort to have a per function
reset mechanism available.  NoSoftRst- is not terribly common and it's
not entirely clear to me that the hardware has made a conscious effort
to provide this for the purposes of per function reset mechanism.
Therefore I've opt'd to prioritize a bus reset over a PM reset.
 
>  hw/vfio/pci.c | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> index cce3024..ca4d1c1 100644
> --- a/hw/vfio/pci.c
> +++ b/hw/vfio/pci.c
> @@ -1930,7 +1930,9 @@ static int vfio_pci_hot_reset(VFIOPCIDevice *vdev, bool single)
>  
>      trace_vfio_pci_hot_reset(vdev->vbasedev.name, single ? "one" : "multi");
>  
> -    vfio_pci_pre_reset(vdev);
> +    if (!single) {
> +        vfio_pci_pre_reset(vdev);
> +    }
>      vdev->vbasedev.needs_reset = false;
>  
>      info = g_malloc0(sizeof(*info));
> @@ -2088,7 +2090,9 @@ out:
>          }
>      }
>  out_single:
> -    vfio_pci_post_reset(vdev);
> +    if (!single) {
> +        vfio_pci_post_reset(vdev);
> +    }
>      g_free(info);
>  
>      return ret;

Looks ok to me, I'll queue it.  Thanks,

Alex

Re: [Qemu-devel] [PATCH] vfio: fix duplicate function call

[Cao jin]

On 10/14/2016 11:50 PM, Alex Williamson wrote:
> On Fri, 14 Oct 2016 19:16:59 +0800
> Cao jin <caoj.fnst@cn.fujitsu.com> wrote:
>
>> When vfio device is reset(encounter FLR, or bus reset), if need to do
>> bus reset(vfio_pci_hot_reset_one is called), vfio_pci_pre_reset &
>> vfio_pci_post_reset will be called twice.
>>
>> Signed-off-by: Cao jin <caoj.fnst@cn.fujitsu.com>
>> ---
>> Also has a little question on vfio_pci_reset. it will be called when encounter
>> bus reset, or FLR. The reset method's priority in this function now is:
>>
>>      1. If has "device specific reset function", then do it
>>      2. If has FLR, then do it.
>>      3. If it can do bus reset(only 1 affected device), then do it
>>      4. If has pm_reset, then do it
>>
>> The question is: why pm reset has low priority than bus reset(if it does
>> can do a bus reset)? why bus reset is not the last choice? In PCI driver
>> of kernel, pls see __pci_dev_reset, we can see, if device support pm reset,
>> it won't do bus reset.
>
> The PCI spec doesn't really define what sort of reset is done with a PM
> reset.  My thinking was that if a device advertises an FLR capability
> then the hardware has made a concerted effort to have a per function
> reset mechanism available.  NoSoftRst- is not terribly common and it's
> not entirely clear to me that the hardware has made a conscious effort
> to provide this for the purposes of per function reset mechanism.
> Therefore I've opt'd to prioritize a bus reset over a PM reset.
>

I see, thanks for your infomation.

-- 
Yours Sincerely,

Cao jin

Re: [Qemu-devel] [PATCH] vfio: fix duplicate function call

[Cao jin]

Hi,

On 10/14/2016 11:50 PM, Alex Williamson wrote:
> On Fri, 14 Oct 2016 19:16:59 +0800
> Cao jin <caoj.fnst@cn.fujitsu.com> wrote:
>
>> When vfio device is reset(encounter FLR, or bus reset), if need to do
>> bus reset(vfio_pci_hot_reset_one is called), vfio_pci_pre_reset &
>> vfio_pci_post_reset will be called twice.
>>
>> Signed-off-by: Cao jin <caoj.fnst@cn.fujitsu.com>
>> ---
>> Also has a little question on vfio_pci_reset. it will be called when encounter
>> bus reset, or FLR. The reset method's priority in this function now is:
>>
>>      1. If has "device specific reset function", then do it
>>      2. If has FLR, then do it.
>>      3. If it can do bus reset(only 1 affected device), then do it
>>      4. If has pm_reset, then do it
>>
>> The question is: why pm reset has low priority than bus reset(if it does
>> can do a bus reset)? why bus reset is not the last choice? In PCI driver
>> of kernel, pls see __pci_dev_reset, we can see, if device support pm reset,
>> it won't do bus reset.
>
> The PCI spec doesn't really define what sort of reset is done with a PM
> reset.  My thinking was that if a device advertises an FLR capability
> then the hardware has made a concerted effort to have a per function
> reset mechanism available.  NoSoftRst- is not terribly common and it's
> not entirely clear to me that the hardware has made a conscious effort
> to provide this for the purposes of per function reset mechanism.
> Therefore I've opt'd to prioritize a bus reset over a PM reset.
>

I still have a question about vfio_pci_reset. I checked commit message 
in f16f39c3, if I understand right, couldn't we put

     /* See if we can do our own bus reset */
     if (!vfio_pci_hot_reset_one(vdev)) {
         goto post_reset;
     }

in the 1st priority? Because if there is 1 affected device, then it will 
do bus reset which is the best reset we can do; if there are more than 1 
affected devices, after this patch, vfio_pci_hot_reset_one will do 
nothing, and then try other reset methods.

-- 
Yours Sincerely,

Cao jin

Re: [Qemu-devel] [PATCH] vfio: fix duplicate function call

[Alex Williamson]

On Mon, 17 Oct 2016 16:57:08 +0800
Cao jin <caoj.fnst@cn.fujitsu.com> wrote:

> Hi,
> 
> On 10/14/2016 11:50 PM, Alex Williamson wrote:
> > On Fri, 14 Oct 2016 19:16:59 +0800
> > Cao jin <caoj.fnst@cn.fujitsu.com> wrote:
> >  
> >> When vfio device is reset(encounter FLR, or bus reset), if need to do
> >> bus reset(vfio_pci_hot_reset_one is called), vfio_pci_pre_reset &
> >> vfio_pci_post_reset will be called twice.
> >>
> >> Signed-off-by: Cao jin <caoj.fnst@cn.fujitsu.com>
> >> ---
> >> Also has a little question on vfio_pci_reset. it will be called when encounter
> >> bus reset, or FLR. The reset method's priority in this function now is:
> >>
> >>      1. If has "device specific reset function", then do it
> >>      2. If has FLR, then do it.
> >>      3. If it can do bus reset(only 1 affected device), then do it
> >>      4. If has pm_reset, then do it
> >>
> >> The question is: why pm reset has low priority than bus reset(if it does
> >> can do a bus reset)? why bus reset is not the last choice? In PCI driver
> >> of kernel, pls see __pci_dev_reset, we can see, if device support pm reset,
> >> it won't do bus reset.  
> >
> > The PCI spec doesn't really define what sort of reset is done with a PM
> > reset.  My thinking was that if a device advertises an FLR capability
> > then the hardware has made a concerted effort to have a per function
> > reset mechanism available.  NoSoftRst- is not terribly common and it's
> > not entirely clear to me that the hardware has made a conscious effort
> > to provide this for the purposes of per function reset mechanism.
> > Therefore I've opt'd to prioritize a bus reset over a PM reset.
> >  
> 
> I still have a question about vfio_pci_reset. I checked commit message 
> in f16f39c3, if I understand right, couldn't we put
> 
>      /* See if we can do our own bus reset */
>      if (!vfio_pci_hot_reset_one(vdev)) {
>          goto post_reset;
>      }
> 
> in the 1st priority? Because if there is 1 affected device, then it will 
> do bus reset which is the best reset we can do; if there are more than 1 
> affected devices, after this patch, vfio_pci_hot_reset_one will do 
> nothing, and then try other reset methods.

It's possible, yes, but that disregards that the hardware has gone to
the trouble to implement a proper function level reset.  As I
explained, I de-prioritize PM reset, specifically because I'm not sure
if hardware designers are necessarily intending it for the purpose of a
device reset.  For FLR this is the entire purpose of the interface.  We
also have a fair bit of experience with the current priority scheme and
I would not take it lightly to change without some compelling evidence
to prove that a new priority scheme is better than the existing.  There
do also exist devices which do not behave properly with a secondary bus
reset, see drivers/pci/quirks.c:quirk_no_bus_reset() in the kernel
tree.  It's possible more devices like this exist, but we don't see
them because they implement FLR.  A bus reset may result in a more
complete device reset, but it's also more disruptive to the system.
Thanks,

Alex

Re: [Qemu-devel] [PATCH] vfio: fix duplicate function call

[Cao jin]

On 10/17/2016 11:01 PM, Alex Williamson wrote:
> On Mon, 17 Oct 2016 16:57:08 +0800
> Cao jin <caoj.fnst@cn.fujitsu.com> wrote:
>
>> Hi,
>>
>> On 10/14/2016 11:50 PM, Alex Williamson wrote:
>>> On Fri, 14 Oct 2016 19:16:59 +0800
>>> Cao jin <caoj.fnst@cn.fujitsu.com> wrote:
>>>
>>>> When vfio device is reset(encounter FLR, or bus reset), if need to do
>>>> bus reset(vfio_pci_hot_reset_one is called), vfio_pci_pre_reset &
>>>> vfio_pci_post_reset will be called twice.
>>>>
>>>> Signed-off-by: Cao jin <caoj.fnst@cn.fujitsu.com>
>>>> ---
>>>> Also has a little question on vfio_pci_reset. it will be called when encounter
>>>> bus reset, or FLR. The reset method's priority in this function now is:
>>>>
>>>>       1. If has "device specific reset function", then do it
>>>>       2. If has FLR, then do it.
>>>>       3. If it can do bus reset(only 1 affected device), then do it
>>>>       4. If has pm_reset, then do it
>>>>
>>>> The question is: why pm reset has low priority than bus reset(if it does
>>>> can do a bus reset)? why bus reset is not the last choice? In PCI driver
>>>> of kernel, pls see __pci_dev_reset, we can see, if device support pm reset,
>>>> it won't do bus reset.
>>>
>>> The PCI spec doesn't really define what sort of reset is done with a PM
>>> reset.  My thinking was that if a device advertises an FLR capability
>>> then the hardware has made a concerted effort to have a per function
>>> reset mechanism available.  NoSoftRst- is not terribly common and it's
>>> not entirely clear to me that the hardware has made a conscious effort
>>> to provide this for the purposes of per function reset mechanism.
>>> Therefore I've opt'd to prioritize a bus reset over a PM reset.
>>>
>>
>> I still have a question about vfio_pci_reset. I checked commit message
>> in f16f39c3, if I understand right, couldn't we put
>>
>>       /* See if we can do our own bus reset */
>>       if (!vfio_pci_hot_reset_one(vdev)) {
>>           goto post_reset;
>>       }
>>
>> in the 1st priority? Because if there is 1 affected device, then it will
>> do bus reset which is the best reset we can do; if there are more than 1
>> affected devices, after this patch, vfio_pci_hot_reset_one will do
>> nothing, and then try other reset methods.
>
> It's possible, yes, but that disregards that the hardware has gone to
> the trouble to implement a proper function level reset.  As I
> explained, I de-prioritize PM reset, specifically because I'm not sure
> if hardware designers are necessarily intending it for the purpose of a
> device reset.  For FLR this is the entire purpose of the interface.  We
> also have a fair bit of experience with the current priority scheme and
> I would not take it lightly to change without some compelling evidence
> to prove that a new priority scheme is better than the existing.  There
> do also exist devices which do not behave properly with a secondary bus
> reset, see drivers/pci/quirks.c:quirk_no_bus_reset() in the kernel
> tree.  It's possible more devices like this exist, but we don't see
> them because they implement FLR.  A bus reset may result in a more
> complete device reset, but it's also more disruptive to the system.
> Thanks,
>
> Alex
>

I see. Thanks Alex, I think these are valuable info to me, although 
maybe I still need more time in the future to understand these totally.

-- 
Yours Sincerely,

Cao jin