How to disable Linux kernel nvme driver for a particular PCI address ?

How to disable Linux kernel nvme driver for a particular PCI address ?

[Ashutosh Sharma]

Hi,

I have multiple NVMe drives of same type (same vendor and same model)
attached to my system running Ubuntu 22.04.2 LTS with Linux kernel
version 5.19.0-35-generic.I have unbound one drive from 'nvme' driver
and bound to the 'vfio-pci' driver using "driverctl
set-override"command.

But when I perform the hot plugging on that particular drive, then
after plugged in, the drive by default binds with 'nvme' driver. So, I
want to permanently bypass/disable the 'nvme' driver only for a
particular pci address/slot. I cannot blacklist the 'nvme' driver
entirely, as other drives still need to be bound with 'nvme' driver.

So, Is there any way to disable the 'nvme' driver for a particular PCI
address/slot ?

Regards,
Ashutosh

Re: How to disable Linux kernel nvme driver for a particular PCI address ?

[Bjorn Helgaas]

[+cc Greg, Rafael, Timothy]

On Mon, May 01, 2023 at 12:19:39PM +0530, Ashutosh Sharma wrote:
> Hi,
> 
> I have multiple NVMe drives of same type (same vendor and same model)
> attached to my system running Ubuntu 22.04.2 LTS with Linux kernel
> version 5.19.0-35-generic.I have unbound one drive from 'nvme' driver
> and bound to the 'vfio-pci' driver using "driverctl
> set-override"command.
> 
> But when I perform the hot plugging on that particular drive, then
> after plugged in, the drive by default binds with 'nvme' driver. So, I
> want to permanently bypass/disable the 'nvme' driver only for a
> particular pci address/slot. I cannot blacklist the 'nvme' driver
> entirely, as other drives still need to be bound with 'nvme' driver.
> 
> So, Is there any way to disable the 'nvme' driver for a particular PCI
> address/slot ?

I think this is more of a device model or udev question than a PCI
subsystem question, so I cc'd some of those folks.

Bjorn

Re: How to disable Linux kernel nvme driver for a particular PCI address ?

[Greg Kroah-Hartman]

On Mon, May 01, 2023 at 11:31:28AM -0500, Bjorn Helgaas wrote:
> [+cc Greg, Rafael, Timothy]
> 
> On Mon, May 01, 2023 at 12:19:39PM +0530, Ashutosh Sharma wrote:
> > Hi,
> > 
> > I have multiple NVMe drives of same type (same vendor and same model)
> > attached to my system running Ubuntu 22.04.2 LTS with Linux kernel
> > version 5.19.0-35-generic.I have unbound one drive from 'nvme' driver
> > and bound to the 'vfio-pci' driver using "driverctl
> > set-override"command.

First off, I just "love" how the vfio people have taken a debugging tool
and made it part of a "enterprise configuration" process.  That's a
horrible hack and the vfio developers really should not be doing this as
people have found out.

> > But when I perform the hot plugging on that particular drive, then
> > after plugged in, the drive by default binds with 'nvme' driver. So, I
> > want to permanently bypass/disable the 'nvme' driver only for a
> > particular pci address/slot. I cannot blacklist the 'nvme' driver
> > entirely, as other drives still need to be bound with 'nvme' driver.
> > 
> > So, Is there any way to disable the 'nvme' driver for a particular PCI
> > address/slot ?
> 
> I think this is more of a device model or udev question than a PCI
> subsystem question, so I cc'd some of those folks.

It's up to userspace to write tools to do this if they want to continue
to force userspace to be the one that does this binding/unbinding for
the vfio drivers.  Otherwise, the vfio driver itself should be the one
doing the binding to the device automatically, not the nvme driver, IF
that driver is supposed to be the one actually controlling it.

sorry, and good luck!

greg k-h