From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hannes Reinecke Subject: Re: MSI-X not enabled for ixgbe device-passthrough Date: Mon, 29 Mar 2010 08:28:17 +0200 Message-ID: <4BB04881.7030708@suse.de> References: <4BAA35A7.2090104@suse.de> <20100325175446.GI1744@sequoia.sous-sol.org> <4BACDB56.4020508@suse.de> <20100326194023.GC29241@sequoia.sous-sol.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: kvm@vger.kernel.org, Sheng Yang , Alexander Graf To: Chris Wright Return-path: Received: from cantor2.suse.de ([195.135.220.15]:45125 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753697Ab0C2G2V (ORCPT ); Mon, 29 Mar 2010 02:28:21 -0400 In-Reply-To: <20100326194023.GC29241@sequoia.sous-sol.org> Sender: kvm-owner@vger.kernel.org List-ID: Chris Wright wrote: > * Hannes Reinecke (hare@suse.de) wrote: >> Chris Wright wrote: >>> * Hannes Reinecke (hare@suse.de) wrote: >>>> Hi all, >>>> >>>> I'm trying to setup a system with device-passthrough for >>>> an ixgbe NIC. >>>> The device itself seems to work, but it isn't using MSI-X. >>>> So some more advanced features like DCB offloading etc >>>> won't work. >>> Please send the relevant dmesg from the guest when it initializes t= he >>> device. BTW, more typical case for that NIC is to assign the VF to= the >>> guest, not the whole PF. >>> >> Yes, I know. But my kernel I'm testing with doesn't have one for ixg= be. >=20 > Ah, you mean it's an older (heh, ok, older actually means not brand n= ew > upstream) kernel, w/out the recent sr-iov additions, fair enough. >=20 >> So I tested that one. And KVM really should enable MSI-X here, >> VFs notwithstanding. >=20 > Yeah, although it's not just KVM involved, it's very much driven by > the guest too. The guest will see (or at least should see) the MSI-X > capability, and decide based on the number of queues whether to enabl= e > MSI-X (completely driver dependent here). >=20 > Did you have a chance to boot the guest again, and send the lspci -vv= v from > the guest POV? You should see two PCI capabilities (MSI and 0x40 and > MSI-X at 0x50). >=20 > Actually, I have one of these devices, let me give it a quick test... > working fine here. Here's some relevant information: >=20 > Host: >=20 > $ sudo lspci -v -s 04:00.0 > 04:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Net= work Connection (rev 01) > Subsystem: Intel Corporation Ethernet Server Adapter X520-2 > Flags: bus master, fast devsel, latency 0, IRQ 16 > Memory at f8080000 (64-bit, prefetchable) [size=3D512K] > I/O ports at d020 [size=3D32] > Memory at f8104000 (64-bit, prefetchable) [size=3D16K] > Capabilities: [40] Power Management version 3 > Capabilities: [50] MSI: Enable- Count=3D1/1 Maskable+ 64bit+ > Capabilities: [70] MSI-X: Enable+ Count=3D64 Masked- > Capabilities: [a0] Express Endpoint, MSI 00 > Capabilities: [100] Advanced Error Reporting > Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-3f-a1-b0 > Capabilities: [150] Alternative Routing-ID Interpretation (ARI) > Capabilities: [160] Single Root I/O Virtualization (SR-IOV) >=20 > $ grep kvm /proc/interrupts=20 > 95: 289 0 0 0 0 = 0 0 0 0 0 0 0 = 0 0 0 0 PCI-MSI-edge kvm_assig= ned_msix_device > 96: 32 0 0 0 292 = 0 0 0 0 0 0 0 = 0 0 0 0 PCI-MSI-edge kvm_assig= ned_msix_device > 97: 2 0 0 0 0 = 0 0 0 0 0 0 0 = 0 0 0 0 PCI-MSI-edge kvm_assig= ned_msix_device >=20 > Guest side: > $ sudo lspci -vvv -s 00:04.0 > 00:04.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Net= work Connection (rev 01) > Subsystem: Intel Corporation Ethernet Server Adapter X520-2 > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- = Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=3Dfast >TAbort- SERR- Latency: 0, Cache Line Size: 64 bytes > Interrupt: pin A routed to IRQ 11 > Region 0: Memory at f2080000 (32-bit, non-prefetchable) > Region 2: I/O ports at c200 > Region 4: Memory at f2100000 (32-bit, non-prefetchable) > Capabilities: [40] MSI: Enable- Count=3D1/1 Maskable- 64bit- > Address: 00000000 Data: 0000 > Capabilities: [50] MSI-X: Enable+ Count=3D64 Masked- > Vector table: BAR=3D4 offset=3D00000000 > PBA: BAR=3D4 offset=3D00002000 >=20 > $ grep eth1 /proc/interrupts > 177: 387 PCI-MSI-X eth1-rx-0 > 185: 421 PCI-MSI-X eth1-tx-0 > 193: 2 PCI-MSI-X eth1:lsc >=20 > $ dmesg | grep -e 00:04.0 -e ixgbe > ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 2.0.8= -k2 > ixgbe: Copyright (c) 1999-2009 Intel Corporation. > ACPI: PCI Interrupt 0000:00:04.0[A] -> Link [LNKD] -> GSI 10 (level, = high) -> IRQ 10 > PCI: Setting latency timer of device 0000:00:04.0 to 64 > ixgbe: 0000:00:04.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled= : Rx Queue count =3D 1, Tx Queue count =3D 1 > ixgbe 0000:00:04.0: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:3f:a1:b0 > ixgbe 0000:00:04.0: MAC: 2, PHY: 11, SFP+: 5, PBA No: e66562-003 > ixgbe 0000:00:04.0: Intel(R) 10 Gigabit Network Connection > ixgbe: eth1 NIC Link is Up 10 Gbps, Flow Control: None >=20 Ah. So I'll have to shout at Alex Graf. No problems there :-) Thanks for your help. Cheers, Hannes --=20 Dr. Hannes Reinecke zSeries & Storage hare@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 N=FCrnberg GF: Markus Rex, HRB 16746 (AG N=FCrnberg)