pci device passthrough: Failed to assign irq

pci device passthrough: Failed to assign irq

[Gerd v. Egidy]

Hi,

I'm trying to pass a pcie-device into a kvm guest. qemu-kvm is started
by libvirt with this command:

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root LOGNAME=root QEMU_AUDIO_DRV=none 
/usr/bin/qemu-kvm -S -M pc-0.12 -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -name test
-uuid cefc7515-c0c2-27fc-8e7e-e0a65f9cb95b -nodefaults 
-chardev socket,id=monitor,path=/var/lib/libvirt/qemu/test.monitor,server,nowait 
-mon chardev=monitor,mode=readline -rtc base=utc -boot c 
-drive file=/dev/vg_main/test,if=none,id=drive-virtio-disk0,boot=on,format=raw 
-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 
-drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on 
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 
-device virtio-net-pci,vlan=0,id=net0,mac=52:54:00:d0:84:c7,bus=pci.0,addr=0x7 
-net tap,fd=59,vlan=0,name=hostnet0 
-chardev pty,id=serial0 
-device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k de 
-vga cirrus -device pci-assign,host=05:00.0,id=hostdev0,bus=pci.0,addr=0x8 
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 

It failes with:

Failed to assign irq for "hostdev0": Operation not permitted
Perhaps you are assigning a device that shares an IRQ with another device?

The irq of the device does not seem to be shared:

05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
        Subsystem: ASUSTeK Computer Inc. Device 8369
        Flags: bus master, fast devsel, latency 0, IRQ 18
        Memory at faee0000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at cc00 [size=32]
        Memory at faedc000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [e0] Express Endpoint, MSI 00
        Capabilities: [a0] MSI-X: Enable- Count=5 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 90-e6-ba-ff-ff-7c-c6-ef
        Kernel driver in use: pci-stub
        Kernel modules: e1000e

when kvm is not running and the device used as eth0 on host:

# cat /proc/interrupts 
[...]
 17:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   sata_sil24
 23:         69          0          0          0          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2
[...]
 37:          2          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-rx-0
 38:          4          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-tx-0
 39:          1          0          0          0          0          0          0          0   PCI-MSI-edge      eth0
[...]

qemu-kvm is running as root, selinux is running in permissive mode. So the rights shouldn't be
an issue too.

The processor is an Intel Core i7-860 on an Asus P7F-E board, chipset is an Intel 3420.
Bios is ver 0607, vt-d is enabled in bios setup.

This is logged in dmesg, so to me it seems like the system is indeed vt-d capable:

[...]
ACPI: Core revision 20091214
DMAR: Host address width 36
DMAR: DRHD base: 0x000000fed90000 flags: 0x1
IOMMU fed90000: ver 1:0 cap c90780106f0462 ecap f020e3
DMAR: RMRR base: 0x000000000e4000 end: 0x000000000e7fff
DMAR: RMRR base: 0x000000bf7ec000 end: 0x000000bf7fffff
DMAR: No ATSR found
Setting APIC routing to physical flat
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
CPU0: Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz stepping 05
[...]
IOMMU 0xfed90000: using Queued invalidation
IOMMU: Setting RMRR:
IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf7ec000 - 0xbf800000]
IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf7ec000 - 0xbf800000]
IOMMU: Setting identity map for device 0000:00:1d.0 [0xe4000 - 0xe8000]
IOMMU: Setting identity map for device 0000:00:1a.0 [0xe4000 - 0xe8000]
IOMMU: Prepare 0-16MiB unity mapping for LPC
IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0x1000000]
  alloc irq_desc for 29 on node 0
  alloc kstat_irqs on node 0
PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
Intel PCLMULQDQ-NI instructions are not detected.
[...]

Kernel is 2.6.33.3-85.fc13.x86_64. I tried qemu-0.12.3-8.fc13 as it came with
Fedora 13 beta and qemu-kvm-0.12.4, same results.

Any idea whats going wrong or what else I could try?

Kind regards,

Gerd

Re: pci device passthrough: Failed to assign irq

[Gerd v. Egidy]

> I'm trying to pass a pcie-device into a kvm guest. qemu-kvm is started
> by libvirt with this command:
[...]
> Failed to assign irq for "hostdev0": Operation not permitted
> Perhaps you are assigning a device that shares an IRQ with another device?

Found out what was causing the issue:

Current libvirt seems to drop CAP_SYS_RAWIO before forking qemu-kvm.

When I patch libvirt to not drop the capabilities, everything works as 
expected. 

So no problem in kvm, sorry for the noise.

Kind regards,

Gerd