Re: Locking between vfio hot-remove and pci sysfs sriov_numvfs

[Jason Gunthorpe <jgg@nvidia.com>]

On Fri, Dec 08, 2023 at 11:12:15AM -0700, Alex Williamson wrote:

> > > > avoided. But in that case RDMA knows the BAR memory is used only for
> > > > doorbell write so this is a reasonable thing to do.  
> > > 
> > > Yeah, this is exactly what SPDK (and DPDK) does today.  
> > 
> > To be clear, I mean we did it in the kernel.
> > 
> > When the device driver is removed we zap all the VMAs and install a
> > fault handler that installs the dummy page instead of SIGBUS
> > 
> > The application doesn't do anything, and this is how SPDK already will
> > be supporting device hot unplug of the RDMA drivers.
> 
> But I think you can only do that in the kernel because you understand
> the device uses those pages for doorbells and it's not a general
> purpose solution, right?
> 
> Perhaps a variant driver could do something similar for NVMe devices
> doorbell pages, but a device agnostic driver like vfio-pci would need
> to SIGBUS on access or else we risk significant data integrity issues.
> Thanks,

Yes, basically.

Might be interesting to consider having a VFIO FEATURE flag to opt
into SIGBUS or dummy page, perhaps even on a VMA by VMA basis.

Jason