Re: PCI device assignment over KVM

[Sheng Yang <sheng@linux.intel.com>]

On Wednesday 06 May 2009 08:42:37 Tong Liu wrote:
> Hi Sheng,
>
> I have installed latest KVM-85 release and the "Failed to assign irq"
> error goes away.
> However my device kernel driver on guest OS (RHEL5u2) can't query my
> card successfully. Our developer said there is memory mapping error
> during KVM device assignment.

Could your developer elaborate it? 

> I am providing all the steps I have done and captures the output as
> pictures here.
> Can you help take a look and see if really my device is not supported by
> KVM yet?

Every other things looks fine. I think the key point should be that "memory 
mapping error".

-- 
regards
Yang, Sheng

>
> Step 1: Before device assignment, /proc/interrupts has my device which
> is not sharing IRQ with others due to MSI-x enabled
> Picture 1.jpg. Even though dmesg shows it is using IRQ 18 during boot
> but /proc shows IRQ 58, etc, I guess it is changed to that number by
> MSI-X after boot.
>
> Step 2: Unbinding my device 0000:01:00.0 from host and start kvm guest
> Picture 2.jpg, some phy mem error reported but I am assuming it is not
> critical
>
> At this moment, /proc/interrupts on local machine is changed. My device
> is disppeared and another entry is created: "kvm_assigned_intx_device"
> as picture 3.jpg shown
>
> And dmesg didn't show obvious errors too, it shows pci device assigned.
> Picture 4.jpg
>
> Log into guest OS(RHEL5u2), run dmesg, it shows errors about my device
> which means it is not passed successfully.
> Picture 5.jpg
>
>
> I tried other device 0000:06:00.1 which is OK.
>
> Regards
> Tong
>
>
> -----Original Message-----
> From: Sheng Yang [mailto:sheng@linux.intel.com]
> Sent: Monday, May 04, 2009 9:35 PM
> To: Tong Liu
> Subject: Re: PCI device assignment over KVM
>
> On Tuesday 05 May 2009 12:06:09 Tong Liu wrote:
> > BTW, as you mentioned, after boot up, 06:00.0 got IRQ 56 because it's
> > MSI enabled.
> > And 01:00.0 is using IRQ18, actually they are not sharing IRQ with any
> >
> > others.
> > (Even though 06:00.0 was using IRQ 18 with 01:00.0 during boot but it
> > is converted to 56 after boot up, so nothing shared)
> >
> > Why is KVM still trying to enable 01:00.0 INTx (dmesg error I put in
> > the first email) if 01:00.0 is not sharing IRQ with any other after
>
> boot?
>
> You can use "cat /proc/interrupts" to know if there are other IRQ
> handler for IRQ 18. Also lspci -v.
>
> If you are sure that there is no sharing interrupt for IRQ 18, you may
> need to look into INTx enabling part of KVM
> (virt/kvm/kvm_main.c:kvm_vm_ioctl_assign_irq()) to know what's happening
> exactly.
>
> And please try latest KVM and qemu-kvm as well(or kvm-85 release). The
> your dmesg show that the version of your KVM is old - no thing like
> "failed to enable INTx device!" in current code now. We rework the
> framework two monthes ago.
>
> --
> regards
> Yang, Sheng
>
> > Thanks
> > Tong
> >
> >
> > -----Original Message-----
> > From: Sheng Yang [mailto:sheng@linux.intel.com]
> > Sent: Monday, May 04, 2009 6:26 PM
> > To: Tong Liu
> > Subject: Re: PCI device assignment over KVM
> >
> > On Tuesday 05 May 2009 05:32:32 Tong Liu wrote:
> > > Hi Sheng,
> > >
> > > My system has VT-d support and I want to assign one PCI-E card to
> > > guest OS.
> > > I have an issue with PCI device assignment over KVM.
> > > I am using latest kernerl 2.6.30-rc4.
> > >
> > > Here are the steps I have done:
> > >
> > > 1. Unbind PCI device 01:00.0 from host using steps documented on KVM
> > >
> > > webiste.
> > > http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM
> > > 2. Then run the following command: qemu-system-x86_64 -m 4096 -boot
> > > c -net none -hda vdisk.img -pcidevice host=01:00.0 It reports
>
> errors:
> > > Assign_irq: deassign: Invalid argument Failed to assign irq for
> > > "01:00.0": Input/output error Perhaps you are assigning a device
> > > that shares an IRQ with another device?
> > > Failed to deassign device "01:00.0" : Invalid argument
> > >
> > > In dmesg it shows the following error:
> > > pci-stub 0000:01:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
> > > assign device: host bdf = 1:0:0
> > > kvm: failed to enable INTx device!
> > > pci-stub 0000:01:00.0: PCI INT A disabled
> > > kvm_vm_ioctl_deassign_device: device hasn't been assigned before, so
> > >
> > > cannot be deassigned
> > >
> > > Info for my device 0000:01:00.0:
> > > lspci -t:
> > >
> > > -[0000:00]-+-00.0
> > >            +-05.0-[0000:01]----00.0
> > >
> > > +-09.0-[0000:02-07]--+-00.0-[0000:03-06]--+-00.0-[0000:04]--
> >
> > +-01.0-[0000:05]--
> >
> > > \-02.0-[0000:06]--+-00.0
> > >
> > > \-00.1
> > >
> > >            |                    \-00.3-[0000:07]--
> > >
> > > lspci -v:
> > >
> > > 01:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX IB QDR,
> > > PCIe 2.0 5GT/s] (rev a0)
> > >         Subsystem: Mellanox Technologies Unknown device 0005
> > >         Flags: fast devsel, IRQ 18
> > >         Memory at 99300000 (64-bit, non-prefetchable) [size=1M]
> > >         Memory at 98000000 (64-bit, prefetchable) [size=8M]
> > >         Capabilities: [40] Power Management version 3
> > >         Capabilities: [48] Vital Product Data
> > >         Capabilities: [9c] MSI-X: Enable+ Mask- TabSize=256
> > >         Capabilities: [60] Express Endpoint IRQ 0
> > >         Capabilities: [100] Unknown (14)
> > >
> > > After running unbind and trying device assignment, MSI-X of it
> > > changed
> > >
> > > to Enable-. I don't know why.
> > >
> > >
> > > If I run dmesg | grep IRQ after server boot, i can see the following
> > >
> > > (my device is sharing IRQ with another device):
> > > mlx4_core 0000:01:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
> > > e1000e 0000:06:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
> > >
> > > However lspci -v shows different thing, 01:00.0 and 06:00.0 are
> > > using different IRQ, I am not sure which one is correct?
> > >
> > > 01:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX IB QDR,
> > > PCIe 2.0 5GT/s] (rev a0)
> > >         Subsystem: Mellanox Technologies Unknown device 0005
> > >         Flags: bus master, fast devsel, latency 0, IRQ 18
> > >         Memory at 99300000 (64-bit, non-prefetchable) [size=1M]
> > >         Memory at 98000000 (64-bit, prefetchable) [size=8M]
> > >         Capabilities: [40] Power Management version 3
> > >         Capabilities: [48] Vital Product Data
> > >         Capabilities: [9c] MSI-X: Enable+ Mask- TabSize=256
> > >         Capabilities: [60] Express Endpoint IRQ 0
> > >         Capabilities: [100] Unknown (14)
> >
> > Hi Tong
> >
> > I found your card don't support MSI, so you got interrupt sharing
> > trouble.
> >
> > The way we avoid to share interrupt with host is convert MSI to INTx,
> > not MSI- X(which is mostly impossible to convert to INTx). So for you
> > card, you need to avoid sharing interrupt in the host. Try different
> > slot should be work. For the same reason, 06:00.0 can work because it
> > support MSI, then no need to avoid interrupt sharing.
> >
> > After you boot up, 06:00.0 got IRQ 56 because it's MSI enabled, so 56
> > is a MSI vector. If you unload e1000e driver, you would see the
> > default interrupt setting, and I think it would be 18. You can also
> > tell the one sharing interrupt from your dmesg - kernel would complain
> >
> > the sharing with some other interrupt handler(not sure if need some
>
> debug option).
>
> > You can also find the conflict one from lspci to check the IRQ number.
> >
> > Another thing is, it's better to post the thread to mailing list
> > rather than try to contact the developer directly. In this way most
> > people see you problem can try to help, which is more efficient.
> >
> > --
> > regards
> > Yang, Sheng
> >
> > > 06:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit
> > > Ethernet Controller (Copper) (rev 01)
> > >         Subsystem: Intel Corporation Unknown device 34d4
> > >         Flags: bus master, fast devsel, latency 0, IRQ 56
> > >         Memory at 99020000 (32-bit, non-prefetchable) [size=128K]
> > >         Memory at 98c00000 (32-bit, non-prefetchable) [size=4M]
> > >         I/O ports at 2020 [size=32]
> > >         Capabilities: [c8] Power Management version 2
> > >         Capabilities: [d0] Message Signalled Interrupts: 64bit+
> > > Queue=0/0 Enable                           +
> > >         Capabilities: [e0] Express Endpoint IRQ 0
> > >         Capabilities: [100] Advanced Error Reporting
> > >         Capabilities: [140] Device Serial Number
> > > 2a-99-6b-ff-ff-17-15-00
> > >
> > > BTW, I can assign 06:00.0 to guest OS but only having issue with my
> > > PCI device 01:00.0 which I need to do.
> > > My device has MSI-X enabled but why KVM still reports IRQ error?
> > >
> > > Regards
> > > Tong