Re: [PATCH v2 0/4] vfio-pci support pasid attach/detach

[Jason Gunthorpe <jgg@nvidia.com>]

On Mon, Aug 05, 2024 at 05:35:17AM +0000, Tian, Kevin wrote:

> Okay. With that I edited my earlier reply a bit by removing the note
> of cmdline option, adding DVSEC possibility, and making it clear that
> the PASID option is in vIOMMU:
> 
> "
> Overall this sounds a feasible path to move forward - starting with
> the VMM to find the gap automatically if PASID is opted in vIOMMU. 
> Devices with hidden registers may fail. Devices with volatile
> config space due to FW upgrade or cross vendors may fail to migrate.
> Then evolving it to the file-based scheme, and there is time to discuss
> any intermediate improvement (fixed quirks, DVSEC, etc.) in between.
> "
> 
> Jason, your thoughts?

This thread is big and I've read it quickly, but I could support the
above summary.

For migration, where we are today, it is completely up to the device
and it's FW to present a new config space that is stable across
migration. In practice this means the device needs to have pre-set the
PF to whatever config state it needs during FLR, and refuse to migrate
if things are not correct.

Moving away from that limitation toward a VMM created stable config
space is definitely a long term interest. I understand other VMM's are
already doing things like this.

It seems we need more time to bake on this long term direction. I
suggested a text file as something very general, but we can do other
things too.

What I see as the general industry direction is toward a very
perspective vPCI function, where you might have v1, v2, etc of a
device that present different config space/behavior/etc. I expect
standards bodies are going to define reference vPCI functions for
their standards along these lines. This would specify all the MMIO
memory layout and every bit of config space right down to each offset
and bit.

This is the sort of restriction that will be needed to allow more
generic live migration between different devices and different
FW. There is a pretty clear role here for the VMM to synthesize a
highly perscribed config space. How we get there and how the libvirt
stack should support this, I don't know.

Jason