Re: [PATCH v3 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

[Yishai Hadas <yishaih@nvidia.com>]

On 29/02/2024 16:10, Jason Gunthorpe wrote:
> On Wed, Feb 28, 2024 at 12:07:06PM -0700, Alex Williamson wrote:
>> On Mon, 26 Feb 2024 15:24:58 -0700
>> Alex Williamson <alex.williamson@redhat.com> wrote:
>>
>>> On Mon, 26 Feb 2024 15:49:52 -0400
>>> Jason Gunthorpe <jgg@nvidia.com> wrote:
>>>
>>>> On Mon, Feb 26, 2024 at 12:41:07PM -0700, Alex Williamson wrote:
>>>>> On Mon, 26 Feb 2024 15:12:20 -0400
>>>>> libvirt recently implemented support for managed="yes" with variant
>>>>> drivers where it will find the best "vfio_pci" driver for a device
>>>>> using an algorithm like Max suggested, but in practice it's not clear
>>>>> how useful that will be considering devices likes CX7 require
>>>>> configuration before binding to the variant driver.  libvirt has no
>>>>> hooks to specify or perform configuration at that point.
>>>>
>>>> I don't think this is fully accurate (or at least not what was
>>>> intended), the VFIO device can be configured any time up until the VM
>>>> mlx5 driver reaches the device startup.
>>>>
>>>> Is something preventing this? Did we accidentally cache the migratable
>>>> flag in vfio or something??
>>>
>>> I don't think so, I think this was just the policy we had decided
>>> relative to profiling VFs when they're created rather than providing a
>>> means to do it though a common vfio variant driver interface[1].
>>
>> Turns out that yes, migration support needs to be established at probe
>> time.  vfio_pci_core_register_device() expects migration_flags,
>> mig_ops, and log_ops to all be established by this point, which for
>> mlx5-vfio-pci occurs when the .init function calls
>> mlx5vf_cmd_set_migratable().
> 
> This is unfortunate, we should look at trying to accomodate this,
> IMHO. Yishai?

I'm not sure what is the alternative here.

Moving to the open() might be too late if the guest driver will be 
active already, this might prevent changing/setting some caps/profiling, 
by that time.

In addition, as Alex wrote, today upon probe() each driver calls 
vfio_pci_core_register_device() and by that time the 'mig/log' ops are 
checked.

Making a change here, I believe, requires a good justification / use 
case and a working alternative and design.

Note:
The above is true for other migration drivers around which upon probe 
should supply their 'mig/log' ops.

> 
>> That also makes me wonder what happens if migration support is disabled
>> via devlink after binding the VF to mlx5-vfio-pci.  Arguably this could
>> be considered user error, but what's the failure mode and support
>> implication?  Thanks,
> 
> I think tThe FW will start failing the migration commands.
> 

Right, please see my previous answer here, I supplied some further details.

Yishai