Re: vfio-pci rejects binding to devices having same pcie vendor id and device id

[Alex Williamson <alex.williamson@redhat.com>]

On Thu, 8 Dec 2022 10:50:08 -0600
Bjorn Helgaas <helgaas@kernel.org> wrote:

> [+cc VFIO folks and Zhenzhong (author of the commit you mention)]
> 
> On Thu, Dec 08, 2022 at 09:24:31PM +0530, Major Saheb wrote:
> > I have a linux system running in kvm, with 6 qemu emulated NVMe
> > drives, as expected all of them have the same PCIe Vendor ID and
> > Device ID(VID: 0x1b36 DID: 0x0010).
> >
> > When I try to unbind them from the kernel NVMe driver and bind it to
> > vfio-pci one by one, I am getting "write error: File exists" when I
> > try to bind the 2nd(and other) drive to vfio-pci.
> > 
> > Kernel version
> > 
> > 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64
> > x86_64 x86_64 GNU/Linux
> > 
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme1n1 -> ../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme2n1 -> ../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme3n1 -> ../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme4n1 -> ../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1
> > lrwxrwxrwx 1 root root 0 Dec  8 11:32 /sys/block/nvme5n1 -> ../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1
> > 
> > Steps for repro
> > ubind nvme2 from kernel NVMe driver and bind it to vfio
> > $ ls -l /sys/bus/pci/drivers/vfio-pci/
> > lrwxrwxrwx 1 root root    0 Dec  8 13:04 0000:00:05.0 -> ../../../../devices/pci0000:00/0000:00:05.0
> > --w------- 1 root root 4096 Dec  8 13:07 bind
> > lrwxrwxrwx 1 root root    0 Dec  8 13:07 module -> ../../../../module/vfio_pci
> > --w------- 1 root root 4096 Dec  8 13:04 new_id
> > --w------- 1 root root 4096 Dec  8 13:07 remove_id
> > --w------- 1 root root 4096 Dec  8 11:32 uevent
> > --w------- 1 root root 4096 Dec  8 13:07 unbind
> > 
> > Unbind nvme3 from  kernel NVMe driver
> > Try binding to vfio-pci
> > # echo "0x1b36  0x0010" >  /sys/bus/pci/drivers/vfio-pci/new_id
> > -bash: echo: write error: File exists

Presumably you already wrote this same ID to the dynamic ID table from
the first device, so yes, it's going to rightfully complain that this
ID already exists.  The new_id interface has numerous problems, which
is why we added the driver_override interface, which is used by tools
like libvirt and driverctl in place of this old interface.

I'd recommend something like:

# driverctl --nosave set-override 0000:00:03.0 vfio-pci
# driverctl --nosave set-override 0000:00:04.0 vfio-pci
# driverctl --nosave set-override 0000:00:05.0 vfio-pci
...

Or if vfio-pci is generally the preferred driver for these devices, you
could remove the --nosave option to have them automatically bound at
boot.  You could also make use of pre-filling the vfio device table
using vfio-pci.ids=1b36:0010 on the kernel command line and making sure
the vfio-pci driver is loaded before the nvme driver.  In general, for
dynamic binding of devices, driver_override is the recommended solution.
Thanks,

Alex