Re: Locking between vfio hot-remove and pci sysfs sriov_numvfs

[Jason Gunthorpe <jgg@nvidia.com>]

On Fri, Dec 08, 2023 at 05:07:22PM +0000, Jim Harris wrote:
> On Thu, Dec 07, 2023 at 07:48:10PM -0400, Jason Gunthorpe wrote:
> > On Thu, Dec 07, 2023 at 04:21:48PM -0700, Alex Williamson wrote:
> > > On Thu, 7 Dec 2023 22:38:23 +0000
> > > Jim Harris <jim.harris@samsung.com> wrote:
> > > 
> > > device_lock() has been a recurring problem.  We don't have a lot of
> > > leeway in how we support the driver remove callback, the device needs
> > > to be released.  We can't return -EBUSY and I don't think we can drop
> > > the mutex while we're waiting on userspace.
> > 
> > The mechanism of waiting in remove for userspace is inherently flawed,
> > it can never work fully correctly. :( I've hit this many times.
> > 
> > Upon remove VFIO should immediately remove itself and leave behind a
> > non-functional file descriptor. Userspace should catch up eventually
> > and see it is toast.
> > 
> > The kernel locking model just cannot support userspace delaying this
> > process.
> > 
> > Jason
> 
> Maybe for now we just whack this specific mole with a separate mutex
> for synchronizing access to sriov->num_VFs in the sysfs paths?
> Something like this (tested on my system):

TBH, I don't have the time right now to unpack this locking
mystery. Maybe Leon remembers?

device_lock() gets everywhere and does a lot of different stuff, so I
would be surprised if it was so easy..

Jason