From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hannes Reinecke Subject: Re: MSI-X not enabled for ixgbe device-passthrough Date: Thu, 25 Mar 2010 16:29:02 +0100 Message-ID: <4BAB813E.1080303@suse.de> References: <4BAA35A7.2090104@suse.de> <201003250939.56947.sheng@linux.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: kvm@vger.kernel.org To: Sheng Yang Return-path: Received: from cantor.suse.de ([195.135.220.2]:43859 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754926Ab0CYP3G (ORCPT ); Thu, 25 Mar 2010 11:29:06 -0400 In-Reply-To: <201003250939.56947.sheng@linux.intel.com> Sender: kvm-owner@vger.kernel.org List-ID: Sheng Yang wrote: > On Wednesday 24 March 2010 23:54:15 Hannes Reinecke wrote: >> Hi all, >> >> I'm trying to setup a system with device-passthrough for >> an ixgbe NIC. >> The device itself seems to work, but it isn't using MSI-X. >> So some more advanced features like DCB offloading etc >> won't work. >=20 > How about lspci result in the guest? >=20 I have to boot a different system; will provide it shortly. > And some guest dmesg would also help. >=20 Ok: device: 07:00.0: driver=3D"pci-assign" host=3D"07:00.0" device: 07:00.1: driver=3D"pci-assign" host=3D"07:00.1" [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 2.6.32.9-0.5-default (geeko@buildhost) (gc= c version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP = 2010-03-15 12:22:00 +0100 [ 0.000000] Command line: console=3DttyS0 [ 0.000000] KERNEL supported cpus: [ 0.000000] Intel GenuineIntel [ 0.000000] AMD AuthenticAMD [ 0.000000] Centaur CentaurHauls [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009f400 (usable) [ 0.000000] BIOS-e820: 000000000009f400 - 00000000000a0000 (reserve= d) [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserve= d) [ 0.000000] BIOS-e820: 0000000000100000 - 0000000007ffd000 (usable) [ 0.000000] BIOS-e820: 0000000007ffd000 - 0000000008000000 (reserve= d) [ 0.000000] BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserve= d) [ 0.000000] DMI 2.4 present. [ 0.000000] last_pfn =3D 0x7ffd max_arch_pfn =3D 0x400000000 [ 0.000000] PAT not supported by CPU. [ 0.000000] Scanning 1 areas for low memory corruption [ 0.000000] modified physical RAM map: [ 0.000000] modified: 0000000000000000 - 0000000000001000 (usable) [ 0.000000] modified: 0000000000001000 - 0000000000006000 (reserved= ) [ 0.000000] modified: 0000000000006000 - 000000000009f400 (usable) [ 0.000000] modified: 000000000009f400 - 00000000000a0000 (reserved= ) [ 0.000000] modified: 00000000000f0000 - 0000000000100000 (reserved= ) [ 0.000000] modified: 0000000000100000 - 0000000007ffd000 (usable) [ 0.000000] modified: 0000000007ffd000 - 0000000008000000 (reserved= ) [ 0.000000] modified: 00000000fffbc000 - 0000000100000000 (reserved= ) [ 0.000000] init_memory_mapping: 0000000000000000-0000000007ffd000 [ 0.000000] RAMDISK: 07849000 - 07feff7e [ 0.000000] ACPI: RSDP 00000000000f8950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0000000007ffde30 00034 (v01 BOCHS BXPCRSDT 0= 0000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0000000007fffe70 00074 (v01 BOCHS BXPCFACP 0= 0000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0000000007ffdfd0 01E22 (v01 BXPC BXDSDT 0= 0000001 INTL 20090123) [ 0.000000] ACPI: FACS 0000000007fffe00 00040 [ 0.000000] ACPI: SSDT 0000000007ffdf90 00037 (v01 BOCHS BXPCSSDT 0= 0000001 BXPC 00000001) [ 0.000000] ACPI: APIC 0000000007ffdeb0 00072 (v01 BOCHS BXPCAPIC 0= 0000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0000000007ffde70 00038 (v01 BOCHS BXPCHPET 0= 0000001 BXPC 00000001) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at 0000000000000000-0000000007ffd000 [ 0.000000] Bootmem setup node 0 0000000000000000-0000000007ffd000 [ 0.000000] NODE_DATA [0000000000009000 - 000000000003cfff] [ 0.000000] bootmap [000000000003d000 - 000000000003dfff] pages 1 [ 0.000000] (7 early reservations) =3D=3D> bootmem [0000000000 - 000= 7ffd000] [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page =3D=3D> = [0000000000 - 0000001000] [ 0.000000] #1 [0000006000 - 0000008000] TRAMPOLINE =3D=3D> = [0000006000 - 0000008000] [ 0.000000] #2 [0001000000 - 0001cd6778] TEXT DATA BSS =3D=3D> = [0001000000 - 0001cd6778] [ 0.000000] #3 [0007849000 - 0007feff7e] RAMDISK =3D=3D> = [0007849000 - 0007feff7e] [ 0.000000] #4 [000009f400 - 0000100000] BIOS reserved =3D=3D> = [000009f400 - 0000100000] [ 0.000000] #5 [0001cd7000 - 0001cd7049] BRK =3D=3D> = [0001cd7000 - 0001cd7049] [ 0.000000] #6 [0000008000 - 0000009000] PGTABLE =3D=3D> = [0000008000 - 0000009000] [ 0.000000] found SMP MP-table at [ffff8800000f89a0] f89a0 [ 0.000000] kvm-clock: cpu 0, msr 0:1940501, boot clock [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000000 -> 0x00001000 [ 0.000000] DMA32 0x00001000 -> 0x00100000 [ 0.000000] Normal 0x00100000 -> 0x00100000 [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[3] active PFN ranges [ 0.000000] 0: 0x00000000 -> 0x00000001 [ 0.000000] 0: 0x00000006 -> 0x0000009f [ 0.000000] 0: 0x00000100 -> 0x00007ffd [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GS= I 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high lev= el) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high lev= el) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high l= evel) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high l= evel) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: 0000000000001000 - 0000000= 000006000 [ 0.000000] PM: Registered nosave memory: 000000000009f000 - 0000000= 0000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000= 0000f0000 [ 0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000= 000100000 [ 0.000000] Allocating PCI resources starting at 8000000 (gap: 80000= 00:f7fbc000) [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] NR_CPUS:4096 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:= 1 [ 0.000000] PERCPU: Embedded 28 pages/cpu @ffff880001e00000 s81944 r= 8192 d24552 u2097152 [ 0.000000] pcpu-alloc: s81944 r8192 d24552 u2097152 alloc=3D1*20971= 52 [ 0.000000] pcpu-alloc: [0] 0=20 [ 0.000000] kvm-clock: cpu 0, msr 0:1e13501, primary cpu clock [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. = Total pages: 32115 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: console=3DttyS0 [ 0.000000] PID hash table entries: 512 (order: 0, 4096 bytes) [ 0.000000] Checking aperture... [ 0.000000] No AGP bridge found [ 0.000000] Memory: 107028k/131060k available (3694k kernel code, 40= 8k absent, 23624k reserved, 5696k data, 964k init) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] NR_IRQS:33024 nr_irqs:256 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 1310720 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=3Dmemory' option if you don't= want memory cgroups [ 0.000000] HPET: 3 timers in total, 0 timers will be used for per-c= pu timer [ 0.000000] Detected 2533.412 MHz processor. [ 0.028001] Calibrating delay loop (skipped) preset value.. 5066.82 = BogoMIPS (lpj=3D10133648) [ 0.032258] kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright= SGI, All Rights Reserved kdb_cmd[0]: defcmd archkdb "" "First line arch debugging" kdb_cmd[8]: defcmd archkdbcpu "" "archkdb with only tasks on cpus" kdb_cmd[15]: defcmd archkdbshort "" "archkdb with less detailed backtra= ce" kdb_cmd[22]: defcmd archkdbcommon "" "Common arch debugging" [ 0.052279] Security Framework initialized [ 0.055281] AppArmor: AppArmor initialized [ 0.056063] Dentry cache hash table entries: 16384 (order: 5, 131072= bytes) [ 0.060094] Inode-cache hash table entries: 8192 (order: 4, 65536 by= tes) [ 0.064086] Mount-cache hash table entries: 256 [ 0.068195] Initializing cgroup subsys ns [ 0.069959] Initializing cgroup subsys cpuacct [ 0.072006] Initializing cgroup subsys memory [ 0.076023] Initializing cgroup subsys devices [ 0.080005] Initializing cgroup subsys freezer [ 0.082913] Initializing cgroup subsys net_cls [ 0.084100] mce: CPU supports 10 MCE banks [ 0.088041] Performance Events: unsupported p6 CPU model 2 no PMU dr= iver, software events only. [ 0.096021] SMP alternatives: switching to UP code [ 0.107845] Freeing SMP alternatives: 29k freed [ 0.108022] ACPI: Core revision 20090903 [ 0.110723] Setting APIC routing to flat [ 0.112173] ..TIMER: vector=3D0x30 apic1=3D0 pin1=3D2 apic2=3D-1 pin= 2=3D-1 [ 0.113513] CPU0: Intel QEMU Virtual CPU version 0.12.3 stepping 03 [ 0.120001] Brought up 1 CPUs [ 0.120001] Total of 1 processors activated (5066.82 BogoMIPS). [ 0.120001] devtmpfs: initialized [ 0.120319] regulator: core version 0.5 [ 0.120752] Time: 14:21:27 Date: 03/24/10 [ 0.121295] NET: Registered protocol family 16 [ 0.121753] ACPI: bus type pci registered [ 0.122149] PCI: Using configuration type 1 for base access [ 0.123262] bio: create slab at 0 [ 0.126128] ACPI: Interpreter enabled [ 0.126432] ACPI: (supports S0 S3 S4 S5) [ 0.126818] ACPI: Using IOAPIC for interrupt routing [ 0.129754] ACPI: No dock devices found. [ 0.130081] PCI: Ignoring host bridge windows from ACPI; if necessar= y, use "pci=3Duse_crs" and report a bug [ 0.130868] ACPI: PCI Root Bridge [PCI0] (0000:00) [ 0.132037] pci 0000:00:01.3: quirk: region b000-b03f claimed by PII= X4 ACPI [ 0.132705] pci 0000:00:01.3: quirk: region b100-b10f claimed by PII= X4 SMB [ 0.143297] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.143939] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.144345] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.144996] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.145797] vgaarb: device added: PCI:0000:00:02.0,decodes=3Dio+mem,= owns=3Dio+mem,locks=3Dnone [ 0.146455] vgaarb: loaded [ 0.146832] PCI: Using ACPI for IRQ routing [ 0.147408] NetLabel: Initializing [ 0.148006] NetLabel: domain hash size =3D 128 [ 0.148447] NetLabel: protocols =3D UNLABELED CIPSOv4 [ 0.148919] NetLabel: unlabeled traffic allowed by default [ 0.149406] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.149851] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.160054] Switching to clocksource kvm-clock [ 0.160870] AppArmor: AppArmor Filesystem Enabled [ 0.161315] pnp: PnP ACPI init [ 0.161587] ACPI: bus type pnp registered [ 0.162625] pnp: PnP ACPI: found 8 devices [ 0.162962] ACPI: ACPI bus type pnp unregistered [ 0.167632] NET: Registered protocol family 2 [ 0.168223] IP route cache hash table entries: 1024 (order: 1, 8192 = bytes) [ 0.169194] TCP established hash table entries: 4096 (order: 4, 6553= 6 bytes) [ 0.169792] TCP bind hash table entries: 4096 (order: 4, 65536 bytes= ) [ 0.170330] TCP: Hash tables configured (established 4096 bind 4096) [ 0.170850] TCP reno registered [ 0.171236] NET: Registered protocol family 1 [ 0.171607] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.172309] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.173126] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.177945] Unpacking initramfs... [ 0.398942] Freeing initrd memory: 7835k freed [ 0.401774] Scanning for low memory corruption every 60 seconds [ 0.402994] audit: initializing netlink socket (disabled) [ 0.403727] type=3D2000 audit(1269440488.400:1): initialized [ 0.414717] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.416207] VFS: Disk quotas dquot_6.5.2 [ 0.416870] Dquot-cache hash table entries: 512 (order 0, 4096 bytes= ) [ 0.417910] msgmni has been set to 56 [ 0.418619] alg: No test for stdrng (krng) [ 0.419263] Block layer SCSI generic (bsg) driver version 0.4 loaded= (major 254) [ 0.420182] io scheduler noop registered [ 0.420743] io scheduler anticipatory registered [ 0.421381] io scheduler deadline registered [ 0.422062] io scheduler cfq registered (default) [ 0.422872] pci-stub: invalid id string "" [ 0.425427] Non-volatile memory driver v1.3 [ 0.425972] Linux agpgart interface v0.103 [ 0.426316] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled [ 0.426998] serial8250: ttyS0 at I/O 0x3f8 (irq =3D 4) is a 16550A [ 0.427904] 00:06: ttyS0 at I/O 0x3f8 (irq =3D 4) is a 16550A [ 0.428520] Fixed MDIO Bus: probed [ 0.428848] PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0= x64 irq 1,12 [ 0.429942] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.430350] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.430794] mice: PS/2 mouse device common for all mice [ 0.431474] input: AT Translated Set 2 keyboard as /devices/platform= /i8042/serio0/input/input0 [ 0.432604] cpuidle: using governor ladder [ 0.432953] cpuidle: using governor menu [ 0.433485] TCP cubic registered [ 0.433849] registered taskstats version 1 [ 0.434411] Magic number: 2:93:385 [ 0.434767] Freeing unused kernel memory: 964k freed [ 0.435578] Write protecting the kernel read-only data: 8192k [ 0.436196] Freeing unused kernel memory: 384k freed [ 0.436944] Freeing unused kernel memory: 1380k freed [ 0.485966] device-mapper: uevent: version 1.0.3 [ 0.486728] device-mapper: ioctl: 4.15.0-ioctl (2009-04-01) initiali= sed: dm-devel@redhat.com [ 0.511928] udevd version 128 started [ 0.542626] dca service started, version 1.12.1 [ 0.548078] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - = version 2.0.44-k2 [ 0.549251] ixgbe: Copyright (c) 1999-2009 Intel Corporation. [ 0.649034] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.649867] ixgbe 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (l= evel, high) -> IRQ 11 [ 0.873218] ixgbe: 0000:00:04.0: ixgbe_init_interrupt_scheme: Multiq= ueue Disabled: Rx Queue count =3D 1, Tx Queue count =3D 1 [ 0.874664] ixgbe 0000:00:04.0: (PCI Express:5.0Gb/s:Width x8) 00:1b= :21:3c:9e:40 [ 0.875789] ixgbe 0000:00:04.0: MAC: 2, PHY: 9, SFP+: 5, PBA No: e66= 562-000 [ 0.877496] ixgbe 0000:00:04.0: Intel(R) 10 Gigabit Network Connecti= on [ 0.878673] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 0.879476] ixgbe 0000:00:05.0: PCI INT B -> Link[LNKB] -> GSI 10 (l= evel, high) -> IRQ 10 [ 1.001120] ixgbe: 0000:00:05.0: ixgbe_init_interrupt_scheme: Multiq= ueue Disabled: Rx Queue count =3D 1, Tx Queue count =3D 1 [ 1.002593] ixgbe 0000:00:05.0: (PCI Express:5.0Gb/s:Width x8) 00:1b= :21:3c:9e:41 [ 1.003652] ixgbe 0000:00:05.0: MAC: 2, PHY: 7, SFP+: 3, PBA No: e66= 562-000 [ 1.005237] ixgbe 0000:00:05.0: Intel(R) 10 Gigabit Network Connecti= on Cheers, Hannes --=20 Dr. Hannes Reinecke zSeries & Storage hare@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 N=FCrnberg GF: Markus Rex, HRB 16746 (AG N=FCrnberg)