Re: [PATCH v2 net-next 2/2] ionic: support sr-iov operations

[Shannon Nelson <snelson@pensando.io>]

On 12/16/19 12:46 AM, Parav Pandit wrote:
> On 12/16/2019 11:32 AM, Shannon Nelson wrote:
>> On 12/15/19 8:47 PM, Parav Pandit wrote:
>>> On 12/13/2019 4:10 AM, Shannon Nelson wrote:
>>>> On 12/12/19 2:24 PM, Parav Pandit wrote:
>>>>> On 12/12/2019 3:35 PM, Jakub Kicinski wrote:
>>>>>> On Thu, 12 Dec 2019 11:59:50 -0800, Shannon Nelson wrote:
>>>>>>> On 12/12/19 11:52 AM, Jakub Kicinski wrote:
>>>>>>>> On Thu, 12 Dec 2019 06:53:42 +0000, Parav Pandit wrote:
>>>>>>>>>>      static void ionic_remove(struct pci_dev *pdev)
>>>>>>>>>>      {
>>>>>>>>>>          struct ionic *ionic = pci_get_drvdata(pdev);
>>>>>>>>>> @@ -257,6 +338,9 @@ static void ionic_remove(struct pci_dev *pdev)
>>>>>>>>>>          if (!ionic)
>>>>>>>>>>              return;
>>>>>>>>>>      +    if (pci_num_vf(pdev))
>>>>>>>>>> +        ionic_sriov_configure(pdev, 0);
>>>>>>>>>> +
>>>>>>>>> Usually sriov is left enabled while removing PF.
>>>>>>>>> It is not the role of the pci PF removal to disable it sriov.
>>>>>>>> I don't think that's true. I consider igb and ixgbe to set the
>>>>>>>> standard
>>>>>>>> for legacy SR-IOV handling since they were one of the first (the
>>>>>>>> first?)
>>>>>>>> and Alex Duyck wrote them.
>>>>>>>>
>>>>>>>> mlx4, bnxt and nfp all disable SR-IOV on remove.
>>>>>>> This was my understanding as well, but now I can see that ixgbe and
>>>>>>> i40e
>>>>>>> are both checking for existing VFs in probe and setting up to use
>>>>>>> them,
>>>>>>> as well as the newer ice driver.  I found this today by looking for
>>>>>>> where they use pci_num_vf().
>>>>>> Right, if the VFs very already enabled on probe they are set up.
>>>>>>
>>>>>> It's a bit of a asymmetric design, in case some other driver left
>>>>>> SR-IOV on, I guess.
>>>>>>
>>>>> I remember on one email thread on netdev list from someone that in one
>>>>> use case, they upgrade the PF driver while VFs are still bound and
>>>>> SR-IOV kept enabled.
>>>>> I am not sure how much it is used in practice/or practical.
>>>>> Such use case may be the reason to keep SR-IOV enabled.
>>>> This brings up a potential corner case where it would be better for the
>>>> driver to use its own num_vfs value rather than relying on the
>>>> pci_num_vf() when answering the ndo_get_vf_*() callbacks, and at least
>>>> the igb may be susceptible.
>>> Please do not cache num_vfs in driver. Use the pci core's pci_num_vf()
>>> in the new code that you are adding.
>> I disagree.  The pci_num_vf() tells us what the kernel has set up for
>> VFs running, while the driver's num_vfs tracks how many resources the
>> driver has set up for handling VFs: these are two different numbers, and
>> there are times in the life of the driver when these numbers are
>> different.  Yes, these are small windows of time, but they are different
>> and need to be treated differently.
>>
> They shouldn't be different. Why are they different?

One simple case where they are different is when .sriov_config is first 
called and the driver hasn't set anything up for handling the VFs.  Once 
the driver has set up whatever resources it needs, then they definitely 
should be the same.  This works in reverse when tearing down the VFs: 
pci_num_vf() says 0, but the driver thinks it has N VFs configured until 
it has torn down its resources and decremented its own counter.  I'm 
sure there are some drivers that don't need to allocate any resources, 
but if they do, these need to be tracked for tearing down later.

In this ionic driver, after working through this and fixing all the 
places where ionic->num_vfs can/should be replaced by pci_num_vf(), I'm 
still left with this issue: since pci_num_vf() returns 0 when 
.sriov_configure() is called when disabling sr-iov, the driver doesn't 
know how many ionic->vf[] had been set up earlier and can't properly 
unmap the DMA stats memory set up for each VF.

I think the driver needs to keep its own resource count.

sln