KVM PCI passthrough IRQ limitations

KVM PCI passthrough IRQ limitations

[Jason Kwon]

Hello,

I am attempting to make use of '-pcidevice' on my system, but have run 
into some problems.  First, my setup:

Fedora 10 x86_64 host system
2.6.28.1-19.fc10 kernel from Koji
KVM-84
Debian Lenny guest VM

I am attempting to passthrough a Xilinx FPGA device to the Lenny guest. 
  No driver for this device exists on the host.

% /usr/local/kvm/bin/qemu-system-x86_64 ./vdisk.img -m 384 -pcidevice 
host=02:00.0
init_assigned_device: Registering real physical device 02:00.0 (bus=2 
dev=0 func=0)
get_real_device: region 0 size 1024 start 0xe1200000 type 512 resource_fd 12
Failed to assign irq for "02:00.0": Input/output error
Perhaps you are assigning a device that shares an IRQ with another device?

When the Fedora host was first booted, the Xilinx was assigned IRQ 10, 
which appeared to be unshared:

02:00.0 Memory controller: Xilinx Corporation Device 0007
         Subsystem: Xilinx Corporation Device 0007
         Flags: bus master, fast devsel, latency 0, IRQ 10
         Memory at e1200000 (64-bit, non-prefetchable) [size=1K]
         Capabilities: <access denied>

However, after executing qemu-system-x86_64, lspci showed the Xilinx 
assigned to IRQ 16, which is shared with a SATA controller:

02:00.0 Memory controller: Xilinx Corporation Device 0007
         Subsystem: Xilinx Corporation Device 0007
         Flags: fast devsel, IRQ 16
         Memory at e1200000 (64-bit, non-prefetchable) [size=1K]
         Capabilities: [40] Power Management version 3
         Capabilities: [48] Message Signalled Interrupts: Mask+ 64bit+ 
Count=1/1 Enable-
         Capabilities: [60] Express Endpoint, MSI 00
         Capabilities: [100] Device Serial Number XX-XX-XX-XX-XX-XX-XX-XX

Moving the card to a different slot gave the same results.

My hope is to eventually have two Xilinx FPGA cards in the same system, 
and to passthrough each card to its own VM instance.  Would such a 
scheme be possible?  Is there a way around the IRQ assignment failure?

Thanks,

Jason

Re: KVM PCI passthrough IRQ limitations

[Chris Wright]

* Jason Kwon (jkwon@redback.com) wrote:
> I am attempting to make use of '-pcidevice' on my system, but have run  
> into some problems.  First, my setup:
>
> Fedora 10 x86_64 host system
> 2.6.28.1-19.fc10 kernel from Koji
> KVM-84

You mean both userspace and kernel module built externally?

> % /usr/local/kvm/bin/qemu-system-x86_64 ./vdisk.img -m 384 -pcidevice  
> host=02:00.0
> init_assigned_device: Registering real physical device 02:00.0 (bus=2  
> dev=0 func=0)
> get_real_device: region 0 size 1024 start 0xe1200000 type 512 resource_fd 12
> Failed to assign irq for "02:00.0": Input/output error
> Perhaps you are assigning a device that shares an IRQ with another device?

any info in dmesg?  what's lspci -t show?

Re: KVM PCI passthrough IRQ limitations

[Jason Kwon]

Chris Wright wrote:
> * Jason Kwon (jkwon@redback.com) wrote:
>> I am attempting to make use of '-pcidevice' on my system, but have run  
>> into some problems.  First, my setup:
>>
>> Fedora 10 x86_64 host system
>> 2.6.28.1-19.fc10 kernel from Koji
>> KVM-84
> 
> You mean both userspace and kernel module built externally?

Yes, I built both from the kvm-84.tar.gz package.

>> % /usr/local/kvm/bin/qemu-system-x86_64 ./vdisk.img -m 384 -pcidevice  
>> host=02:00.0
>> init_assigned_device: Registering real physical device 02:00.0 (bus=2  
>> dev=0 func=0)
>> get_real_device: region 0 size 1024 start 0xe1200000 type 512 resource_fd 12
>> Failed to assign irq for "02:00.0": Input/output error
>> Perhaps you are assigning a device that shares an IRQ with another device?
> 
> any info in dmesg?  what's lspci -t show?

dmesg gives:

pci 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
IRQ handler type mismatch for IRQ 16
current handler: uhci_hcd:usb3
Pid: 3597, comm: qemu-system-x86 Not tainted 2.6.28.1-19.fc10.x86_64 #1
Call Trace:
  [<ffffffff810891f6>] __setup_irq+0x1f4/0x227
  [<ffffffffa01bc917>] ? kvm_assigned_dev_intr+0x0/0x2c [kvm]
  [<ffffffff810892fc>] request_irq+0xd3/0x15a
  [<ffffffffa01bc9d2>] assigned_device_update_intx+0x8f/0xa9 [kvm]
  [<ffffffffa01be711>] kvm_vm_ioctl+0x6af/0x79c [kvm]
  [<ffffffff8114d9bf>] ? inode_has_perm+0x5b/0x61
  [<ffffffff81051241>] ? __sigqueue_free+0x38/0x3c
  [<ffffffff81051f39>] ? __dequeue_signal+0x103/0x12f
  [<ffffffff810d82ec>] vfs_ioctl+0x2a/0x78
  [<ffffffff810d86cc>] do_vfs_ioctl+0x392/0x3c0
  [<ffffffff810d874f>] sys_ioctl+0x55/0x77
  [<ffffffff8101106a>] system_call_fastpath+0x16/0x1b
kvm: failed to enable INTx device!
pci 0000:02:00.0: PCI INT A disabled

lspci -t shows:

-[0000:00]-+-00.0
            +-01.0-[0000:01]--
            +-03.0-[0000:02]----00.0
            +-07.0-[0000:03]--
            +-10.0
            +-10.1
            +-14.0
            +-14.1
            +-14.2
            +-14.3
            +-19.0
            +-1a.0
            +-1a.1
            +-1a.2
            +-1a.7
            +-1b.0
            +-1c.0-[0000:04]--
            +-1c.1-[0000:05]--
            +-1c.4-[0000:06]----00.0
            +-1d.0
            +-1d.1
            +-1d.2
            +-1d.7
            +-1e.0-[0000:07]--+-02.0
            |                 \-03.0
            +-1f.0
            +-1f.2
            +-1f.3
            \-1f.5

Thanks,

Jason

Re: KVM PCI passthrough IRQ limitations

[Chris Wright]

* Jason Kwon (jkwon@redback.com) wrote:
> pci 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> IRQ handler type mismatch for IRQ 16
> current handler: uhci_hcd:usb3

This is a shared interrupt (and pci device assignment will request a
non-shared interrupt).  Are you using that usb host controller?  Can you
unbind it?

> Pid: 3597, comm: qemu-system-x86 Not tainted 2.6.28.1-19.fc10.x86_64 #1
> Call Trace:
>  [<ffffffff810891f6>] __setup_irq+0x1f4/0x227
>  [<ffffffffa01bc917>] ? kvm_assigned_dev_intr+0x0/0x2c [kvm]
>  [<ffffffff810892fc>] request_irq+0xd3/0x15a
>  [<ffffffffa01bc9d2>] assigned_device_update_intx+0x8f/0xa9 [kvm]
>  [<ffffffffa01be711>] kvm_vm_ioctl+0x6af/0x79c [kvm]
>  [<ffffffff8114d9bf>] ? inode_has_perm+0x5b/0x61
>  [<ffffffff81051241>] ? __sigqueue_free+0x38/0x3c
>  [<ffffffff81051f39>] ? __dequeue_signal+0x103/0x12f
>  [<ffffffff810d82ec>] vfs_ioctl+0x2a/0x78
>  [<ffffffff810d86cc>] do_vfs_ioctl+0x392/0x3c0
>  [<ffffffff810d874f>] sys_ioctl+0x55/0x77
>  [<ffffffff8101106a>] system_call_fastpath+0x16/0x1b
> kvm: failed to enable INTx device!
> pci 0000:02:00.0: PCI INT A disabled
>
> lspci -t shows:
>
> -[0000:00]-+-00.0
>            +-01.0-[0000:01]--
>            +-03.0-[0000:02]----00.0

OK, only the one device behind this bridge

Re: KVM PCI passthrough IRQ limitations

[Jason Kwon]

Chris Wright wrote:
> * Jason Kwon (jkwon@redback.com) wrote:
>> pci 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
>> IRQ handler type mismatch for IRQ 16
>> current handler: uhci_hcd:usb3
> 
> This is a shared interrupt (and pci device assignment will request a
> non-shared interrupt).  Are you using that usb host controller?  Can you
> unbind it?

I attempted unbinding the usb host driver, but the IRQ was/is also 
shared by the SATA drive controller, so I'm stuck.  I tried moving the 
card to a different slot but the IRQ assignments were the same.

Does it make any difference whether the device is MSI-capable or not?

Thanks,

Jason

Re: KVM PCI passthrough IRQ limitations

[Chris Wright]

* Jason Kwon (jkwon@redback.com) wrote:
> Does it make any difference whether the device is MSI-capable or not?

Well, does it work with msi2intx=0 kvm module parameter?

Re: KVM PCI passthrough IRQ limitations

[Sheng Yang]

On Wednesday 04 March 2009 09:15:57 Jason Kwon wrote:
> Chris Wright wrote:
> > * Jason Kwon (jkwon@redback.com) wrote:
> >> pci 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> >> IRQ handler type mismatch for IRQ 16
> >> current handler: uhci_hcd:usb3
> >
> > This is a shared interrupt (and pci device assignment will request a
> > non-shared interrupt).  Are you using that usb host controller?  Can you
> > unbind it?
>
> I attempted unbinding the usb host driver, but the IRQ was/is also
> shared by the SATA drive controller, so I'm stuck.  I tried moving the
> card to a different slot but the IRQ assignments were the same.
>
> Does it make any difference whether the device is MSI-capable or not?

If the device support MSI, msi2intx=1(default value) would resolve the sharing 
interrupt issue. So seems the device didn't support MSI...

-- 
regards
Yang, Sheng