From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chris Wright Subject: Re: MSI-X not enabled for ixgbe device-passthrough Date: Fri, 26 Mar 2010 12:40:23 -0700 Message-ID: <20100326194023.GC29241@sequoia.sous-sol.org> References: <4BAA35A7.2090104@suse.de> <20100325175446.GI1744@sequoia.sous-sol.org> <4BACDB56.4020508@suse.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Chris Wright , kvm@vger.kernel.org, Sheng Yang To: Hannes Reinecke Return-path: Received: from sous-sol.org ([216.99.217.87]:36548 "EHLO sequoia.sous-sol.org" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751752Ab0CZTkb (ORCPT ); Fri, 26 Mar 2010 15:40:31 -0400 Content-Disposition: inline In-Reply-To: <4BACDB56.4020508@suse.de> Sender: kvm-owner@vger.kernel.org List-ID: * Hannes Reinecke (hare@suse.de) wrote: > Chris Wright wrote: > > * Hannes Reinecke (hare@suse.de) wrote: > >> Hi all, > >> > >> I'm trying to setup a system with device-passthrough for > >> an ixgbe NIC. > >> The device itself seems to work, but it isn't using MSI-X. > >> So some more advanced features like DCB offloading etc > >> won't work. > > > > Please send the relevant dmesg from the guest when it initializes the > > device. BTW, more typical case for that NIC is to assign the VF to the > > guest, not the whole PF. > > > Yes, I know. But my kernel I'm testing with doesn't have one for ixgbe. Ah, you mean it's an older (heh, ok, older actually means not brand new upstream) kernel, w/out the recent sr-iov additions, fair enough. > So I tested that one. And KVM really should enable MSI-X here, > VFs notwithstanding. Yeah, although it's not just KVM involved, it's very much driven by the guest too. The guest will see (or at least should see) the MSI-X capability, and decide based on the number of queues whether to enable MSI-X (completely driver dependent here). Did you have a chance to boot the guest again, and send the lspci -vvv from the guest POV? You should see two PCI capabilities (MSI and 0x40 and MSI-X at 0x50). Actually, I have one of these devices, let me give it a quick test... working fine here. Here's some relevant information: Host: $ sudo lspci -v -s 04:00.0 04:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter X520-2 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at f8080000 (64-bit, prefetchable) [size=512K] I/O ports at d020 [size=32] Memory at f8104000 (64-bit, prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=64 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-3f-a1-b0 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) $ grep kvm /proc/interrupts 95: 289 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge kvm_assigned_msix_device 96: 32 0 0 0 292 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge kvm_assigned_msix_device 97: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge kvm_assigned_msix_device Guest side: $ sudo lspci -vvv -s 00:04.0 00:04.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter X520-2 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Link [LNKD] -> GSI 10 (level, high) -> IRQ 10 PCI: Setting latency timer of device 0000:00:04.0 to 64 ixgbe: 0000:00:04.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 ixgbe 0000:00:04.0: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:3f:a1:b0 ixgbe 0000:00:04.0: MAC: 2, PHY: 11, SFP+: 5, PBA No: e66562-003 ixgbe 0000:00:04.0: Intel(R) 10 Gigabit Network Connection ixgbe: eth1 NIC Link is Up 10 Gbps, Flow Control: None