[Qemu-devel] PCI Passthrough KVM with vfio-pci giving EINVAL on PCI Probe

[Qemu-devel] PCI Passthrough KVM with vfio-pci giving EINVAL on PCI Probe

[Nicholas Andre]

[-- Attachment #1: Type: text/plain, Size: 2394 bytes --]

I am attempting to attach PCIe devices in passthrough to my KVM domain. I
have two devices in one IOMMU group (#28):

03:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge
04:00.0 Multimedia audio controller: Texas Instruments TMS320C6414
TMS320C6415 TMS320C6416

and lspci -n values:

03:00.0 0604: 104c:8240
04:00.0 0401: 104c:a106

Both of which are attached to my KVM domain with managed configuration.

I am running Ubuntu 14.04. I have:

   - Installed KVM, set up domains, ensured KVM and vfio-pci modules are
   loaded
   - Set intel_iommu=on on the kernel command line and verified it is
   enabled with dmesg
   - kvm-ok prints acceleration is all set, VT-d is supported on the chipset
   - Verified that the 03:00.0 device does not have a driver currently
   listed under ls -l /sys/bus/pci/devices/0000\:03\:00.0/ though it does
   have a firmware listed.

When attempting to start the VM I get:

virsh # start rivendell
error: Failed to start domain rivendell
error: Failed to bind PCI device '0000:03:00.0' to vfio-pci: No such device

and under syslog I get what I believe to be the PCIe EINVAL code:

Feb 23 19:46:46 tupac kernel: [  214.159625] vfio-pci: probe of
0000:03:00.0 failed with error -22

Trying to do this manually: I do this to bind the devices to vfio-pci:

# echo "104c a106" > /sys/bus/pci/drivers/vfio-pci/new_id
# echo "104c 8240" > /sys/bus/pci/drivers/vfio-pci/new_id

Then:

root@tupac:~# echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
bash: echo: write error: No such device

However, the multimedia controller appears AOK:

root@tupac:~# ls -l /sys/bus/pci/drivers/vfio-pci/
total 0
lrwxrwxrwx 1 root root    0 Feb 24 13:07 0000:04:00.0 ->
../../../../devices/pci0000:00/0000:00:02.2/0000:03:00.0/0000:04:00.0
--w------- 1 root root 4096 Feb 24 13:05 bind
lrwxrwxrwx 1 root root    0 Feb 24 13:07 module -> ../../../../module/vfio_pci
--w------- 1 root root 4096 Feb 24 13:15 new_id
--w------- 1 root root 4096 Feb 23 19:47 remove_id
--w------- 1 root root 4096 Feb 23 19:43 uevent
--w------- 1 root root 4096 Feb 24 13:07 unbind

03:00.0 (which is a PCIe to PCI adapter) is not showing up under vfio-pci's
devices. They're in the same IOMMU group so I don't think I can attach them
individually.

What is causing this? Is this some issue with the presence of the regular
old PCI bus in the system behind the adapter?

--Nick

[-- Attachment #2: Type: text/html, Size: 10185 bytes --]

Re: [Qemu-devel] PCI Passthrough KVM with vfio-pci giving EINVAL on PCI Probe

[Alex Williamson]

On Wed, 24 Feb 2016 20:04:08 +0000
Nicholas Andre <n@axfp.org> wrote:

> I am attempting to attach PCIe devices in passthrough to my KVM domain. I
> have two devices in one IOMMU group (#28):
> 
> 03:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge
> 04:00.0 Multimedia audio controller: Texas Instruments TMS320C6414
> TMS320C6415 TMS320C6416
> 
> and lspci -n values:
> 
> 03:00.0 0604: 104c:8240
> 04:00.0 0401: 104c:a106
> 
> Both of which are attached to my KVM domain with managed configuration.
> 
> I am running Ubuntu 14.04. I have:
> 
>    - Installed KVM, set up domains, ensured KVM and vfio-pci modules are
>    loaded
>    - Set intel_iommu=on on the kernel command line and verified it is
>    enabled with dmesg
>    - kvm-ok prints acceleration is all set, VT-d is supported on the chipset
>    - Verified that the 03:00.0 device does not have a driver currently
>    listed under ls -l /sys/bus/pci/devices/0000\:03\:00.0/ though it does
>    have a firmware listed.
> 
> When attempting to start the VM I get:
> 
> virsh # start rivendell
> error: Failed to start domain rivendell
> error: Failed to bind PCI device '0000:03:00.0' to vfio-pci: No such device
> 
> and under syslog I get what I believe to be the PCIe EINVAL code:
> 
> Feb 23 19:46:46 tupac kernel: [  214.159625] vfio-pci: probe of
> 0000:03:00.0 failed with error -22
> 
> Trying to do this manually: I do this to bind the devices to vfio-pci:
> 
> # echo "104c a106" > /sys/bus/pci/drivers/vfio-pci/new_id
> # echo "104c 8240" > /sys/bus/pci/drivers/vfio-pci/new_id
> 
> Then:
> 
> root@tupac:~# echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
> bash: echo: write error: No such device
> 
> However, the multimedia controller appears AOK:
> 
> root@tupac:~# ls -l /sys/bus/pci/drivers/vfio-pci/
> total 0
> lrwxrwxrwx 1 root root    0 Feb 24 13:07 0000:04:00.0 ->
> ../../../../devices/pci0000:00/0000:00:02.2/0000:03:00.0/0000:04:00.0
> --w------- 1 root root 4096 Feb 24 13:05 bind
> lrwxrwxrwx 1 root root    0 Feb 24 13:07 module -> ../../../../module/vfio_pci
> --w------- 1 root root 4096 Feb 24 13:15 new_id
> --w------- 1 root root 4096 Feb 23 19:47 remove_id
> --w------- 1 root root 4096 Feb 23 19:43 uevent
> --w------- 1 root root 4096 Feb 24 13:07 unbind
> 
> 03:00.0 (which is a PCIe to PCI adapter) is not showing up under vfio-pci's
> devices. They're in the same IOMMU group so I don't think I can attach them
> individually.
> 
> What is causing this? Is this some issue with the presence of the regular
> old PCI bus in the system behind the adapter?

vfio-pci does not support bridges nor is there any reason to bind the
bridge to vfio-pci.  The IOMMU group requirement is satisfied by the
bridge not being bound to any drivers.  We used to have some bridge
drivers whitelisted to allow the group to be viable with them attached,
newer kernels skip testing the driver of bridge devices.  Is there some
specific reason you want to attach this bridge to the VM?  Thanks,

Alex

Re: [Qemu-devel] PCI Passthrough KVM with vfio-pci giving EINVAL on PCI Probe

[Nicholas Andre]

[-- Attachment #1: Type: text/plain, Size: 3516 bytes --]

No, I was under the impression that since they were in the IOMMU group that
they must be attached together. I will try it without (not sure if the
driver for some reason requires both to be present on the guest system; I'm
not particularly savvy with PCI bus programming).

On Wed, Feb 24, 2016 at 5:12 PM Alex Williamson <alex.williamson@redhat.com>
wrote:

> On Wed, 24 Feb 2016 20:04:08 +0000
> Nicholas Andre <n@axfp.org> wrote:
>
> > I am attempting to attach PCIe devices in passthrough to my KVM domain. I
> > have two devices in one IOMMU group (#28):
> >
> > 03:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge
> > 04:00.0 Multimedia audio controller: Texas Instruments TMS320C6414
> > TMS320C6415 TMS320C6416
> >
> > and lspci -n values:
> >
> > 03:00.0 0604: 104c:8240
> > 04:00.0 0401: 104c:a106
> >
> > Both of which are attached to my KVM domain with managed configuration.
> >
> > I am running Ubuntu 14.04. I have:
> >
> >    - Installed KVM, set up domains, ensured KVM and vfio-pci modules are
> >    loaded
> >    - Set intel_iommu=on on the kernel command line and verified it is
> >    enabled with dmesg
> >    - kvm-ok prints acceleration is all set, VT-d is supported on the
> chipset
> >    - Verified that the 03:00.0 device does not have a driver currently
> >    listed under ls -l /sys/bus/pci/devices/0000\:03\:00.0/ though it does
> >    have a firmware listed.
> >
> > When attempting to start the VM I get:
> >
> > virsh # start rivendell
> > error: Failed to start domain rivendell
> > error: Failed to bind PCI device '0000:03:00.0' to vfio-pci: No such
> device
> >
> > and under syslog I get what I believe to be the PCIe EINVAL code:
> >
> > Feb 23 19:46:46 tupac kernel: [  214.159625] vfio-pci: probe of
> > 0000:03:00.0 failed with error -22
> >
> > Trying to do this manually: I do this to bind the devices to vfio-pci:
> >
> > # echo "104c a106" > /sys/bus/pci/drivers/vfio-pci/new_id
> > # echo "104c 8240" > /sys/bus/pci/drivers/vfio-pci/new_id
> >
> > Then:
> >
> > root@tupac:~# echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
> > bash: echo: write error: No such device
> >
> > However, the multimedia controller appears AOK:
> >
> > root@tupac:~# ls -l /sys/bus/pci/drivers/vfio-pci/
> > total 0
> > lrwxrwxrwx 1 root root    0 Feb 24 13:07 0000:04:00.0 ->
> > ../../../../devices/pci0000:00/0000:00:02.2/0000:03:00.0/0000:04:00.0
> > --w------- 1 root root 4096 Feb 24 13:05 bind
> > lrwxrwxrwx 1 root root    0 Feb 24 13:07 module ->
> ../../../../module/vfio_pci
> > --w------- 1 root root 4096 Feb 24 13:15 new_id
> > --w------- 1 root root 4096 Feb 23 19:47 remove_id
> > --w------- 1 root root 4096 Feb 23 19:43 uevent
> > --w------- 1 root root 4096 Feb 24 13:07 unbind
> >
> > 03:00.0 (which is a PCIe to PCI adapter) is not showing up under
> vfio-pci's
> > devices. They're in the same IOMMU group so I don't think I can attach
> them
> > individually.
> >
> > What is causing this? Is this some issue with the presence of the regular
> > old PCI bus in the system behind the adapter?
>
> vfio-pci does not support bridges nor is there any reason to bind the
> bridge to vfio-pci.  The IOMMU group requirement is satisfied by the
> bridge not being bound to any drivers.  We used to have some bridge
> drivers whitelisted to allow the group to be viable with them attached,
> newer kernels skip testing the driver of bridge devices.  Is there some
> specific reason you want to attach this bridge to the VM?  Thanks,
>
> Alex
>

[-- Attachment #2: Type: text/html, Size: 4329 bytes --]