Re: [PATCH v2 0/4] vfio-pci support pasid attach/detach

[Yi Liu <yi.l.liu@intel.com>]

Hi Alex, Jason,

We didn't get any conclusion on the vPASID cap last time. Shall we
have more alignment on it? +Vasant as well.

On 2024/4/30 01:44, Jason Gunthorpe wrote:
> On Fri, Apr 26, 2024 at 02:13:54PM -0600, Alex Williamson wrote:
>> On Fri, 26 Apr 2024 11:11:17 -0300
>> Jason Gunthorpe <jgg@nvidia.com> wrote:
>>
>>> On Wed, Apr 24, 2024 at 02:13:49PM -0600, Alex Williamson wrote:
>>>
>>>> This is kind of an absurd example to portray as a ubiquitous problem.
>>>> Typically the config space layout is a reflection of hardware whether
>>>> the device supports migration or not.
>>>
>>> Er, all our HW has FW constructed config space. It changes with FW
>>> upgrades. We change it during the life of the product. This has to be
>>> considered..
>>
>> So as I understand it, the concern is that you have firmware that
>> supports migration, but it also openly hostile to the fundamental
>> aspects of exposing a stable device ABI in support of migration.
> 
> Well, that makes it sound rude, but yes that is part of it.
> 
> mlx5 is tremendously FW defined. The FW can only cope with migration
> in some limited cases today. Making that compatability bigger is
> ongoing work.
> 
> Config space is one of the areas that has not been addressed.
> Currently things are such that the FW won't support migration in
> combinations that have different physical config space - so it is not
> a problem.
> 
> But, in principle, it is an issue. AFAIK, the only complete solution
> is for the hypervisor to fully synthesize a stable config space.
> 
> So, if we keep this in the kernel then I'd imagine the kernel will
> need to grow some shared infrastructure to fully synthezise the config
> space - not text file based, but basically the same as what I
> described for the VMM.
> 
>>> But that won't be true if the kernel is making decisions. The config
>>> space layout depends now on the kernel driver version too.
>>
>> But in the cases where we support migration there's a device specific
>> variant driver that supports that migration.  It's the job of that
>> variant driver to not only export and import the device state, but also
>> to provide a consistent ABI to the user, which includes the config
>> space layout.
> 
> Yes, we could do that, but I'm not sure how it will work in all cases.
> 
>> I don't understand why we'd say the device programming ABI itself
>> falls within the purview of the device/variant driver, but PCI
>> config space is defined by device specific code at a higher level.
> 
> The "device programming ABI" doesn't contain any *policy*. The layout
> of the config space is like 50% policy. Especially when we start to
> talk about standards defined migration. The standards will set the
> "device programming ABI" and maybe even specify the migration
> stream. They will not, and arguably can not, specify the config space.
> 
> Config space layout is substantially policy of the instance type. Even
> little things like the vendor IDs can be meaningfully replaced in VMs.
> 
>> Regarding "if we accept that text file configuration should be
>> something the VMM supports", I'm not on board with this yet, so
>> applying it to PASID discussion seems premature.
> 
> Sure, I'm just explaining a way this could all fit together.
> 
>> We've developed variant drivers specifically to host the device specific
>> aspects of migration support.  The requirement of a consistent config
>> space layout is a problem that only exists relative to migration.
> 
> Well, I wouldn't go quite so far. Arguably even non-migritable
> instance types may want to adjust thier config space. Eg if I'm using
> a DPU and I get a NVMe/Virtio PCI function I may want to scrub out
> details from the config space to make it more general. Even without
> migration.
> 
> This already happens today in places like VDPA which completely
> replace the underlying config space in some cases.
> 
> I see it as a difference from a world of highly constrained "instance
> types" and a more ad hoc world.
> 
>> is an issue that I would have considered the responsibility of the
>> variant driver, which would likely expect a consistent interface from
>> the hardware/firmware.  Why does hostile firmware suddenly make it the
>> VMM's problem to provide a consistent ABI to the config space of the
>> device rather than the variant driver?
> 
> It is not "hostile firmware"! It accepting that a significant aspect
> of the config layout is actually policy.
> 
> Plus the standards limitations that mean we can't change the config
> space on the fly make it pretty much impossible for the device to
> acutally do anything to help here. Software must fix the config space.
> 
>> Obviously config maps are something that a VMM could do, but it also
>> seems to impose a non-trivial burden that every VMM requires an
>> implementation of a config space map and integration for each device
>> rather than simply expecting the exposed config space of the device to
>> be part of the migration ABI.
> 
> Well, the flip is true to, it is alot of burden on every variant
> device driver implement and on the kernel in general to manage config
> space policy on behalf of the VMM.
> 
> My point is if the VMM is already going to be forced to manage config
> space policy for other good reasons, are we sure we want to put a
> bunch of stuff in the kernel that sometimes won't be used?
> 
>> Also this solution specifically only addresses config space
>> compatibility without considering the more generic issue that a
>> variant driver can expose different device personas.  A versioned
>> persona and config space virtualization in the variant driver is a
>> much more flexible solution.
> 
> It is addressed, the different personas would have their own text file
> maps. The target VMM would have to load the right map. Shared common
> code across all the variant drivers.
> 
> Jason

-- 
Regards,
Yi Liu