* vfio-pci rejects binding to devices having same pcie vendor id and device id @ 2022-12-08 15:54 Major Saheb 2022-12-08 16:50 ` Bjorn Helgaas 0 siblings, 1 reply; 8+ messages in thread From: Major Saheb @ 2022-12-08 15:54 UTC (permalink / raw) To: linux-pci I have a linux system running in kvm, with 6 qemu emulated NVMe drives, as expected all of them have the same PCIe Vendor ID and Device ID(VID: 0x1b36 DID: 0x0010). When I try to unbind them from the kernel NVMe driver and bind it to vfio-pci one by one, I am getting "write error: File exists" when I try to bind the 2nd(and other) drive to vfio-pci. Kernel version 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1 lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme1n1 -> ../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1 lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme2n1 -> ../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1 lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme3n1 -> ../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1 lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme4n1 -> ../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1 lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme5n1 -> ../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1 Steps for repro ubind nvme2 from kernel NVMe driver and bind it to vfio $ ls -l /sys/bus/pci/drivers/vfio-pci/ total 0 lrwxrwxrwx 1 root root 0 Dec 8 13:04 0000:00:05.0 -> ../../../../devices/pci0000:00/0000:00:05.0 --w------- 1 root root 4096 Dec 8 13:07 bind lrwxrwxrwx 1 root root 0 Dec 8 13:07 module -> ../../../../module/vfio_pci --w------- 1 root root 4096 Dec 8 13:04 new_id --w------- 1 root root 4096 Dec 8 13:07 remove_id --w------- 1 root root 4096 Dec 8 11:32 uevent --w------- 1 root root 4096 Dec 8 13:07 unbind Unbind nvme3 from kernel NVMe driver Try binding to vfio-pci # echo "0x1b36 0x0010" > /sys/bus/pci/drivers/vfio-pci/new_id -bash: echo: write error: File exists Not sure but this seems interesting https://github.com/torvalds/linux/commit/3853f9123c185eb4018f5ccd3cdda5968efb5e10#diff-625d2827bff96bb3a019fa705d99f0b89ec32f281c38a844457b3413d9172007 Can some help ? ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: vfio-pci rejects binding to devices having same pcie vendor id and device id 2022-12-08 15:54 vfio-pci rejects binding to devices having same pcie vendor id and device id Major Saheb @ 2022-12-08 16:50 ` Bjorn Helgaas 2022-12-08 17:25 ` Alex Williamson 2022-12-12 4:44 ` Zhenzhong Duan 0 siblings, 2 replies; 8+ messages in thread From: Bjorn Helgaas @ 2022-12-08 16:50 UTC (permalink / raw) To: Major Saheb Cc: Alex Williamson, Cornelia Huck, Zhenzhong Duan, kvm, linux-pci [+cc VFIO folks and Zhenzhong (author of the commit you mention)] On Thu, Dec 08, 2022 at 09:24:31PM +0530, Major Saheb wrote: > I have a linux system running in kvm, with 6 qemu emulated NVMe > drives, as expected all of them have the same PCIe Vendor ID and > Device ID(VID: 0x1b36 DID: 0x0010). > > When I try to unbind them from the kernel NVMe driver and bind it to > vfio-pci one by one, I am getting "write error: File exists" when I > try to bind the 2nd(and other) drive to vfio-pci. > > Kernel version > > 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 > x86_64 x86_64 GNU/Linux > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1 > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme1n1 -> ../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1 > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme2n1 -> ../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1 > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme3n1 -> ../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1 > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme4n1 -> ../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1 > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme5n1 -> ../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1 > > Steps for repro > ubind nvme2 from kernel NVMe driver and bind it to vfio > $ ls -l /sys/bus/pci/drivers/vfio-pci/ > lrwxrwxrwx 1 root root 0 Dec 8 13:04 0000:00:05.0 -> ../../../../devices/pci0000:00/0000:00:05.0 > --w------- 1 root root 4096 Dec 8 13:07 bind > lrwxrwxrwx 1 root root 0 Dec 8 13:07 module -> ../../../../module/vfio_pci > --w------- 1 root root 4096 Dec 8 13:04 new_id > --w------- 1 root root 4096 Dec 8 13:07 remove_id > --w------- 1 root root 4096 Dec 8 11:32 uevent > --w------- 1 root root 4096 Dec 8 13:07 unbind > > Unbind nvme3 from kernel NVMe driver > Try binding to vfio-pci > # echo "0x1b36 0x0010" > /sys/bus/pci/drivers/vfio-pci/new_id > -bash: echo: write error: File exists > > Not sure but this seems interesting > https://github.com/torvalds/linux/commit/3853f9123c185eb4018f5ccd3cdda5968efb5e10#diff-625d2827bff96bb3a019fa705d99f0b89ec32f281c38a844457b3413d9172007 > > Can some help ? ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: vfio-pci rejects binding to devices having same pcie vendor id and device id 2022-12-08 16:50 ` Bjorn Helgaas @ 2022-12-08 17:25 ` Alex Williamson 2022-12-08 17:55 ` Major Saheb 2022-12-12 4:44 ` Zhenzhong Duan 1 sibling, 1 reply; 8+ messages in thread From: Alex Williamson @ 2022-12-08 17:25 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: Major Saheb, Cornelia Huck, Zhenzhong Duan, kvm, linux-pci On Thu, 8 Dec 2022 10:50:08 -0600 Bjorn Helgaas <helgaas@kernel.org> wrote: > [+cc VFIO folks and Zhenzhong (author of the commit you mention)] > > On Thu, Dec 08, 2022 at 09:24:31PM +0530, Major Saheb wrote: > > I have a linux system running in kvm, with 6 qemu emulated NVMe > > drives, as expected all of them have the same PCIe Vendor ID and > > Device ID(VID: 0x1b36 DID: 0x0010). > > > > When I try to unbind them from the kernel NVMe driver and bind it to > > vfio-pci one by one, I am getting "write error: File exists" when I > > try to bind the 2nd(and other) drive to vfio-pci. > > > > Kernel version > > > > 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 > > x86_64 x86_64 GNU/Linux > > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme1n1 -> ../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme2n1 -> ../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme3n1 -> ../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme4n1 -> ../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme5n1 -> ../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1 > > > > Steps for repro > > ubind nvme2 from kernel NVMe driver and bind it to vfio > > $ ls -l /sys/bus/pci/drivers/vfio-pci/ > > lrwxrwxrwx 1 root root 0 Dec 8 13:04 0000:00:05.0 -> ../../../../devices/pci0000:00/0000:00:05.0 > > --w------- 1 root root 4096 Dec 8 13:07 bind > > lrwxrwxrwx 1 root root 0 Dec 8 13:07 module -> ../../../../module/vfio_pci > > --w------- 1 root root 4096 Dec 8 13:04 new_id > > --w------- 1 root root 4096 Dec 8 13:07 remove_id > > --w------- 1 root root 4096 Dec 8 11:32 uevent > > --w------- 1 root root 4096 Dec 8 13:07 unbind > > > > Unbind nvme3 from kernel NVMe driver > > Try binding to vfio-pci > > # echo "0x1b36 0x0010" > /sys/bus/pci/drivers/vfio-pci/new_id > > -bash: echo: write error: File exists Presumably you already wrote this same ID to the dynamic ID table from the first device, so yes, it's going to rightfully complain that this ID already exists. The new_id interface has numerous problems, which is why we added the driver_override interface, which is used by tools like libvirt and driverctl in place of this old interface. I'd recommend something like: # driverctl --nosave set-override 0000:00:03.0 vfio-pci # driverctl --nosave set-override 0000:00:04.0 vfio-pci # driverctl --nosave set-override 0000:00:05.0 vfio-pci ... Or if vfio-pci is generally the preferred driver for these devices, you could remove the --nosave option to have them automatically bound at boot. You could also make use of pre-filling the vfio device table using vfio-pci.ids=1b36:0010 on the kernel command line and making sure the vfio-pci driver is loaded before the nvme driver. In general, for dynamic binding of devices, driver_override is the recommended solution. Thanks, Alex ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: vfio-pci rejects binding to devices having same pcie vendor id and device id 2022-12-08 17:25 ` Alex Williamson @ 2022-12-08 17:55 ` Major Saheb 2022-12-08 18:21 ` Alex Williamson 0 siblings, 1 reply; 8+ messages in thread From: Major Saheb @ 2022-12-08 17:55 UTC (permalink / raw) To: Alex Williamson Cc: Bjorn Helgaas, Cornelia Huck, Zhenzhong Duan, kvm, linux-pci Thanks Alex , That works for me ~$ sudo driverctl --nosave set-override 0000:00:05.0 vfio-pci ~$ sudo driverctl --nosave set-override 0000:00:06.0 vfio-pci ~$ sudo driverctl --nosave set-override 0000:00:07.0 vfio-pci admin@node-1:~$ sudo nvme list Node SN Model Namespace Usage Format FW Rev --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme10n1 akqvf2-0_10 QEMU NVMe Ctrl 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 /dev/nvme11n1 akqvf2-0_11 QEMU NVMe Ctrl 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 /dev/nvme5n1 akqvf2-0_5 QEMU NVMe Ctrl 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 /dev/nvme6n1 akqvf2-0_6 QEMU NVMe Ctrl 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 /dev/nvme7n1 akqvf2-0_7 QEMU NVMe Ctrl 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 /dev/nvme8n1 akqvf2-0_8 QEMU NVMe Ctrl 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 /dev/nvme9n1 akqvf2-0_9 QEMU NVMe Ctrl 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 I came across you blogspot after I fired the mail http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-3-host.html Some should update https://docs.kernel.org/driver-api/vfio.html in public interest , If I knew how to do that I would do it, Thanks again. On Thu, Dec 8, 2022 at 10:55 PM Alex Williamson <alex.williamson@redhat.com> wrote: > > On Thu, 8 Dec 2022 10:50:08 -0600 > Bjorn Helgaas <helgaas@kernel.org> wrote: > > > [+cc VFIO folks and Zhenzhong (author of the commit you mention)] > > > > On Thu, Dec 08, 2022 at 09:24:31PM +0530, Major Saheb wrote: > > > I have a linux system running in kvm, with 6 qemu emulated NVMe > > > drives, as expected all of them have the same PCIe Vendor ID and > > > Device ID(VID: 0x1b36 DID: 0x0010). > > > > > > When I try to unbind them from the kernel NVMe driver and bind it to > > > vfio-pci one by one, I am getting "write error: File exists" when I > > > try to bind the 2nd(and other) drive to vfio-pci. > > > > > > Kernel version > > > > > > 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 > > > x86_64 x86_64 GNU/Linux > > > > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1 > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme1n1 -> ../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1 > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme2n1 -> ../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1 > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme3n1 -> ../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1 > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme4n1 -> ../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1 > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme5n1 -> ../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1 > > > > > > Steps for repro > > > ubind nvme2 from kernel NVMe driver and bind it to vfio > > > $ ls -l /sys/bus/pci/drivers/vfio-pci/ > > > lrwxrwxrwx 1 root root 0 Dec 8 13:04 0000:00:05.0 -> ../../../../devices/pci0000:00/0000:00:05.0 > > > --w------- 1 root root 4096 Dec 8 13:07 bind > > > lrwxrwxrwx 1 root root 0 Dec 8 13:07 module -> ../../../../module/vfio_pci > > > --w------- 1 root root 4096 Dec 8 13:04 new_id > > > --w------- 1 root root 4096 Dec 8 13:07 remove_id > > > --w------- 1 root root 4096 Dec 8 11:32 uevent > > > --w------- 1 root root 4096 Dec 8 13:07 unbind > > > > > > Unbind nvme3 from kernel NVMe driver > > > Try binding to vfio-pci > > > # echo "0x1b36 0x0010" > /sys/bus/pci/drivers/vfio-pci/new_id > > > -bash: echo: write error: File exists > > Presumably you already wrote this same ID to the dynamic ID table from > the first device, so yes, it's going to rightfully complain that this > ID already exists. The new_id interface has numerous problems, which > is why we added the driver_override interface, which is used by tools > like libvirt and driverctl in place of this old interface. > > I'd recommend something like: > > # driverctl --nosave set-override 0000:00:03.0 vfio-pci > # driverctl --nosave set-override 0000:00:04.0 vfio-pci > # driverctl --nosave set-override 0000:00:05.0 vfio-pci > ... > > Or if vfio-pci is generally the preferred driver for these devices, you > could remove the --nosave option to have them automatically bound at > boot. You could also make use of pre-filling the vfio device table > using vfio-pci.ids=1b36:0010 on the kernel command line and making sure > the vfio-pci driver is loaded before the nvme driver. In general, for > dynamic binding of devices, driver_override is the recommended solution. > Thanks, > > Alex > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: vfio-pci rejects binding to devices having same pcie vendor id and device id 2022-12-08 17:55 ` Major Saheb @ 2022-12-08 18:21 ` Alex Williamson 2022-12-08 18:48 ` Major Saheb 0 siblings, 1 reply; 8+ messages in thread From: Alex Williamson @ 2022-12-08 18:21 UTC (permalink / raw) To: Major Saheb; +Cc: Bjorn Helgaas, Cornelia Huck, Zhenzhong Duan, kvm, linux-pci On Thu, 8 Dec 2022 23:25:07 +0530 Major Saheb <majosaheb@gmail.com> wrote: > Thanks Alex , > That works for me > > ~$ sudo driverctl --nosave set-override 0000:00:05.0 vfio-pci > ~$ sudo driverctl --nosave set-override 0000:00:06.0 vfio-pci > ~$ sudo driverctl --nosave set-override 0000:00:07.0 vfio-pci > admin@node-1:~$ sudo nvme list > Node SN Model > Namespace Usage Format FW > Rev > --------------------- -------------------- > ---------------------------------------- --------- > -------------------------- ---------------- -------- > /dev/nvme10n1 akqvf2-0_10 QEMU NVMe Ctrl > 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 > /dev/nvme11n1 akqvf2-0_11 QEMU NVMe Ctrl > 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 > /dev/nvme5n1 akqvf2-0_5 QEMU NVMe Ctrl > 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 > /dev/nvme6n1 akqvf2-0_6 QEMU NVMe Ctrl > 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 > /dev/nvme7n1 akqvf2-0_7 QEMU NVMe Ctrl > 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 > /dev/nvme8n1 akqvf2-0_8 QEMU NVMe Ctrl > 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 > /dev/nvme9n1 akqvf2-0_9 QEMU NVMe Ctrl > 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 > > I came across you blogspot after I fired the mail > http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-3-host.html > Some should update https://docs.kernel.org/driver-api/vfio.html in > public interest , If I knew how to do that I would do it, Yes, that documentation is from before the driver_override method was introduced. There's some non vfio specific documentation of driver_override here: https://docs.kernel.org/admin-guide/abi-testing.html?highlight=driver_override#abi-sys-bus-pci-devices-driver-override Otherwise, documentation updates gladly accepted. This documentation is part of the kernel source tree and follows the same process as submitting code changes, outlined here: https://docs.kernel.org/process/submitting-patches.html The kvm@vger.kernel.org mailing list handles patches for vfio, but please keep me in Cc if you submit something. Thanks, Alex ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: vfio-pci rejects binding to devices having same pcie vendor id and device id 2022-12-08 18:21 ` Alex Williamson @ 2022-12-08 18:48 ` Major Saheb 0 siblings, 0 replies; 8+ messages in thread From: Major Saheb @ 2022-12-08 18:48 UTC (permalink / raw) To: Alex Williamson Cc: Bjorn Helgaas, Cornelia Huck, Zhenzhong Duan, kvm, linux-pci Sure , I will read the process and try to submit the patch. Thanks. On Thu, Dec 8, 2022 at 11:51 PM Alex Williamson <alex.williamson@redhat.com> wrote: > > On Thu, 8 Dec 2022 23:25:07 +0530 > Major Saheb <majosaheb@gmail.com> wrote: > > > Thanks Alex , > > That works for me > > > > ~$ sudo driverctl --nosave set-override 0000:00:05.0 vfio-pci > > ~$ sudo driverctl --nosave set-override 0000:00:06.0 vfio-pci > > ~$ sudo driverctl --nosave set-override 0000:00:07.0 vfio-pci > > admin@node-1:~$ sudo nvme list > > Node SN Model > > Namespace Usage Format FW > > Rev > > --------------------- -------------------- > > ---------------------------------------- --------- > > -------------------------- ---------------- -------- > > /dev/nvme10n1 akqvf2-0_10 QEMU NVMe Ctrl > > 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 > > /dev/nvme11n1 akqvf2-0_11 QEMU NVMe Ctrl > > 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 > > /dev/nvme5n1 akqvf2-0_5 QEMU NVMe Ctrl > > 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 > > /dev/nvme6n1 akqvf2-0_6 QEMU NVMe Ctrl > > 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 > > /dev/nvme7n1 akqvf2-0_7 QEMU NVMe Ctrl > > 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 > > /dev/nvme8n1 akqvf2-0_8 QEMU NVMe Ctrl > > 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 > > /dev/nvme9n1 akqvf2-0_9 QEMU NVMe Ctrl > > 1 274.88 GB / 274.88 GB 512 B + 0 B 1.0 > > > > I came across you blogspot after I fired the mail > > http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-3-host.html > > Some should update https://docs.kernel.org/driver-api/vfio.html in > > public interest , If I knew how to do that I would do it, > > Yes, that documentation is from before the driver_override method was > introduced. There's some non vfio specific documentation of > driver_override here: > > https://docs.kernel.org/admin-guide/abi-testing.html?highlight=driver_override#abi-sys-bus-pci-devices-driver-override > > Otherwise, documentation updates gladly accepted. This documentation > is part of the kernel source tree and follows the same process as > submitting code changes, outlined here: > > https://docs.kernel.org/process/submitting-patches.html > > The kvm@vger.kernel.org mailing list handles patches for vfio, but > please keep me in Cc if you submit something. Thanks, > > Alex > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: vfio-pci rejects binding to devices having same pcie vendor id and device id 2022-12-08 16:50 ` Bjorn Helgaas 2022-12-08 17:25 ` Alex Williamson @ 2022-12-12 4:44 ` Zhenzhong Duan 2022-12-12 9:26 ` Major Saheb 1 sibling, 1 reply; 8+ messages in thread From: Zhenzhong Duan @ 2022-12-12 4:44 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: Major Saheb, Alex Williamson, Cornelia Huck, kvm, linux-pci On Fri, Dec 9, 2022 at 12:50 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > [+cc VFIO folks and Zhenzhong (author of the commit you mention)] > > On Thu, Dec 08, 2022 at 09:24:31PM +0530, Major Saheb wrote: > > I have a linux system running in kvm, with 6 qemu emulated NVMe > > drives, as expected all of them have the same PCIe Vendor ID and > > Device ID(VID: 0x1b36 DID: 0x0010). > > > > When I try to unbind them from the kernel NVMe driver and bind it to > > vfio-pci one by one, I am getting "write error: File exists" when I > > try to bind the 2nd(and other) drive to vfio-pci. > > > > Kernel version > > > > 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 > > x86_64 x86_64 GNU/Linux > > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme1n1 -> ../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme2n1 -> ../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme3n1 -> ../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme4n1 -> ../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme5n1 -> ../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1 > > > > Steps for repro > > ubind nvme2 from kernel NVMe driver and bind it to vfio > > $ ls -l /sys/bus/pci/drivers/vfio-pci/ > > lrwxrwxrwx 1 root root 0 Dec 8 13:04 0000:00:05.0 -> ../../../../devices/pci0000:00/0000:00:05.0 > > --w------- 1 root root 4096 Dec 8 13:07 bind > > lrwxrwxrwx 1 root root 0 Dec 8 13:07 module -> ../../../../module/vfio_pci > > --w------- 1 root root 4096 Dec 8 13:04 new_id > > --w------- 1 root root 4096 Dec 8 13:07 remove_id > > --w------- 1 root root 4096 Dec 8 11:32 uevent > > --w------- 1 root root 4096 Dec 8 13:07 unbind > > > > Unbind nvme3 from kernel NVMe driver > > Try binding to vfio-pci > > # echo "0x1b36 0x0010" > /sys/bus/pci/drivers/vfio-pci/new_id > > -bash: echo: write error: File exists Above operation added new id to vfio-pci, it will trigger scanning of unbind devices once by default. To bind other devices of same id, besides what Alex suggested, I think you can also use /sys/bus/pci/drivers/vfio-pci/bind interface. Thanks Zhenzhong > > > > Not sure but this seems interesting > > https://github.com/torvalds/linux/commit/3853f9123c185eb4018f5ccd3cdda5968efb5e10#diff-625d2827bff96bb3a019fa705d99f0b89ec32f281c38a844457b3413d9172007 > > > > Can some help ? ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: vfio-pci rejects binding to devices having same pcie vendor id and device id 2022-12-12 4:44 ` Zhenzhong Duan @ 2022-12-12 9:26 ` Major Saheb 0 siblings, 0 replies; 8+ messages in thread From: Major Saheb @ 2022-12-12 9:26 UTC (permalink / raw) To: Zhenzhong Duan Cc: Bjorn Helgaas, Alex Williamson, Cornelia Huck, kvm, linux-pci Ok, thanks. On Mon, Dec 12, 2022 at 10:14 AM Zhenzhong Duan <zhenzhong.duan@gmail.com> wrote: > > On Fri, Dec 9, 2022 at 12:50 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > > > [+cc VFIO folks and Zhenzhong (author of the commit you mention)] > > > > On Thu, Dec 08, 2022 at 09:24:31PM +0530, Major Saheb wrote: > > > I have a linux system running in kvm, with 6 qemu emulated NVMe > > > drives, as expected all of them have the same PCIe Vendor ID and > > > Device ID(VID: 0x1b36 DID: 0x0010). > > > > > > When I try to unbind them from the kernel NVMe driver and bind it to > > > vfio-pci one by one, I am getting "write error: File exists" when I > > > try to bind the 2nd(and other) drive to vfio-pci. > > > > > > Kernel version > > > > > > 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 > > > x86_64 x86_64 GNU/Linux > > > > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1 > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme1n1 -> ../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1 > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme2n1 -> ../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1 > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme3n1 -> ../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1 > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme4n1 -> ../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1 > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme5n1 -> ../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1 > > > > > > Steps for repro > > > ubind nvme2 from kernel NVMe driver and bind it to vfio > > > $ ls -l /sys/bus/pci/drivers/vfio-pci/ > > > lrwxrwxrwx 1 root root 0 Dec 8 13:04 0000:00:05.0 -> ../../../../devices/pci0000:00/0000:00:05.0 > > > --w------- 1 root root 4096 Dec 8 13:07 bind > > > lrwxrwxrwx 1 root root 0 Dec 8 13:07 module -> ../../../../module/vfio_pci > > > --w------- 1 root root 4096 Dec 8 13:04 new_id > > > --w------- 1 root root 4096 Dec 8 13:07 remove_id > > > --w------- 1 root root 4096 Dec 8 11:32 uevent > > > --w------- 1 root root 4096 Dec 8 13:07 unbind > > > > > > Unbind nvme3 from kernel NVMe driver > > > Try binding to vfio-pci > > > # echo "0x1b36 0x0010" > /sys/bus/pci/drivers/vfio-pci/new_id > > > -bash: echo: write error: File exists > > Above operation added new id to vfio-pci, it will trigger scanning of > unbind devices once by default. > To bind other devices of same id, besides what Alex suggested, I think you can > also use /sys/bus/pci/drivers/vfio-pci/bind interface. > > Thanks > Zhenzhong > > > > > > > Not sure but this seems interesting > > > https://github.com/torvalds/linux/commit/3853f9123c185eb4018f5ccd3cdda5968efb5e10#diff-625d2827bff96bb3a019fa705d99f0b89ec32f281c38a844457b3413d9172007 > > > > > > Can some help ? ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2022-12-12 9:27 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-12-08 15:54 vfio-pci rejects binding to devices having same pcie vendor id and device id Major Saheb 2022-12-08 16:50 ` Bjorn Helgaas 2022-12-08 17:25 ` Alex Williamson 2022-12-08 17:55 ` Major Saheb 2022-12-08 18:21 ` Alex Williamson 2022-12-08 18:48 ` Major Saheb 2022-12-12 4:44 ` Zhenzhong Duan 2022-12-12 9:26 ` Major Saheb
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox