Re: [PATCH rfc] vfio-pci: Allow write combining

[Jason Gunthorpe <jgg@ziepe.ca>]

On Fri, Aug 02, 2024 at 11:05:06AM -0600, Alex Williamson wrote:

> > Well, again, it is not a region, it is just a record that this mmap
> > cookie uses X region with Y mapping flags. The number of regions don't
> > change. Critically from a driver perspective the number of regions and
> > region indexes wouldn't change.
> 
> Why is this critical?

So we don't leak this too much into the drivers? Why should all the
VFIO drivers have to be changed to alter how their region indexes work
just to add a single flag?? 

> > Well, that is just the current implementation. What we did in RDMA
> > when we switched from hard coded mmap cookies to dynamic ones is
> > use an xarray (today this should be a maple tree) to dynamically
> > allocate mmap cookies whenever the driver returns something to
> > userspace. During the mmap fop the pgoff is fed back through the maple
> > tree to get the description of what the cookie represents.
> 
> Sure, we could do that too, the current implementation (not uAPI) just
> uses some upper bits to create fixed region address spaces.  The only
> thing we should need to keep consistent is the mapping of indexes to
> device resources up through VFIO_PCI_NUM_REGIONS.

I fear we might need to do this as there may not be room in the pgoff
space (at least for 32 bit) to duplicate everything....

> > My point is to not confuse the pgoff encoding with the driver concept
> > of a region. The region is a single peice of memory, the "mmap cookie"s
> > are just handles to it. Adding more data to the handle is not the same
> > as adding more regions.
> 
> I don't get it.  Take for instance PCI config space.  Given the right
> GPU, I can get to config space through an I/O port region, an MMIO
> region (possibly multiple ways), and the config space region itself.
> Therefore based on this hardware implementation there is no unique
> mapping that says that config space is uniquely accessible via a single
> region.  

That doesn't seem like this sitation. Those are multiple different HW
paths with different HW addresses, sure they can have different
regions.

Here we are talking about the same HW path with the same HW
addresses. It shouldn't be duplicated.

> BAR can only be accessed via a signle region and we need to play games
> with terminology to call it an mmap cookie rather than officially
> creating a region with WC mmap semantics?

Because if you keep adding more regions for what are attributes of a
mapping we may end up with a combinatoral explosion of regions.

I already know there is interest in doing non-cache/cache mapping
attributes too.

Approaching this as a fixed number of regions reflecting the HW
addresses and a variable number of flags requested by the user is alot
more reasonable than trying to have a list of every permutation of
every address for every combination of flags.

Jason