Re: MSI-X not enabled for ixgbe device-passthrough

[Hannes Reinecke <hare@suse.de>]

Sheng Yang wrote:
> On Wednesday 24 March 2010 23:54:15 Hannes Reinecke wrote:
>> Hi all,
>>
>> I'm trying to setup a system with device-passthrough for
>> an ixgbe NIC.
>> The device itself seems to work, but it isn't using MSI-X.
>> So some more advanced features like DCB offloading etc
>> won't work.
> 
> How about lspci result in the guest?
> 
I have to boot a different system; will provide it shortly.

> And some guest dmesg would also help.
> 
Ok:

device: 07:00.0: driver="pci-assign" host="07:00.0"

device: 07:00.1: driver="pci-assign" host="07:00.1"

[    0.000000] Initializing cgroup subsys cpuset


[    0.000000] Initializing cgroup subsys cpu


[    0.000000] Linux version 2.6.32.9-0.5-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP 2010-03-15 12:22:00 +0100


[    0.000000] Command line: console=ttyS0


[    0.000000] KERNEL supported cpus:


[    0.000000]   Intel GenuineIntel


[    0.000000]   AMD AuthenticAMD


[    0.000000]   Centaur CentaurHauls


[    0.000000] BIOS-provided physical RAM map:


[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009f400 (usable)


[    0.000000]  BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)


[    0.000000]  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)


[    0.000000]  BIOS-e820: 0000000000100000 - 0000000007ffd000 (usable)


[    0.000000]  BIOS-e820: 0000000007ffd000 - 0000000008000000 (reserved)


[    0.000000]  BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved)


[    0.000000] DMI 2.4 present.


[    0.000000] last_pfn = 0x7ffd max_arch_pfn = 0x400000000


[    0.000000] PAT not supported by CPU.


[    0.000000] Scanning 1 areas for low memory corruption


[    0.000000] modified physical RAM map:


[    0.000000]  modified: 0000000000000000 - 0000000000001000 (usable)


[    0.000000]  modified: 0000000000001000 - 0000000000006000 (reserved)


[    0.000000]  modified: 0000000000006000 - 000000000009f400 (usable)


[    0.000000]  modified: 000000000009f400 - 00000000000a0000 (reserved)


[    0.000000]  modified: 00000000000f0000 - 0000000000100000 (reserved)


[    0.000000]  modified: 0000000000100000 - 0000000007ffd000 (usable)


[    0.000000]  modified: 0000000007ffd000 - 0000000008000000 (reserved)


[    0.000000]  modified: 00000000fffbc000 - 0000000100000000 (reserved)


[    0.000000] init_memory_mapping: 0000000000000000-0000000007ffd000


[    0.000000] RAMDISK: 07849000 - 07feff7e


[    0.000000] ACPI: RSDP 00000000000f8950 00014 (v00 BOCHS )


[    0.000000] ACPI: RSDT 0000000007ffde30 00034 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)


[    0.000000] ACPI: FACP 0000000007fffe70 00074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)


[    0.000000] ACPI: DSDT 0000000007ffdfd0 01E22 (v01   BXPC   BXDSDT 00000001 INTL 20090123)


[    0.000000] ACPI: FACS 0000000007fffe00 00040


[    0.000000] ACPI: SSDT 0000000007ffdf90 00037 (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)


[    0.000000] ACPI: APIC 0000000007ffdeb0 00072 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)


[    0.000000] ACPI: HPET 0000000007ffde70 00038 (v01 BOCHS  BXPCHPET 00000001 BXPC 00000001)


[    0.000000] No NUMA configuration found


[    0.000000] Faking a node at 0000000000000000-0000000007ffd000


[    0.000000] Bootmem setup node 0 0000000000000000-0000000007ffd000


[    0.000000]   NODE_DATA [0000000000009000 - 000000000003cfff]


[    0.000000]   bootmap [000000000003d000 -  000000000003dfff] pages 1


[    0.000000] (7 early reservations) ==> bootmem [0000000000 - 0007ffd000]


[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]


[    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]


[    0.000000]   #2 [0001000000 - 0001cd6778]    TEXT DATA BSS ==> [0001000000 - 0001cd6778]


[    0.000000]   #3 [0007849000 - 0007feff7e]          RAMDISK ==> [0007849000 - 0007feff7e]


[    0.000000]   #4 [000009f400 - 0000100000]    BIOS reserved ==> [000009f400 - 0000100000]


[    0.000000]   #5 [0001cd7000 - 0001cd7049]              BRK ==> [0001cd7000 - 0001cd7049]


[    0.000000]   #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]


[    0.000000] found SMP MP-table at [ffff8800000f89a0] f89a0


[    0.000000] kvm-clock: cpu 0, msr 0:1940501, boot clock


[    0.000000] Zone PFN ranges:


[    0.000000]   DMA      0x00000000 -> 0x00001000


[    0.000000]   DMA32    0x00001000 -> 0x00100000


[    0.000000]   Normal   0x00100000 -> 0x00100000


[    0.000000] Movable zone start PFN for each node


[    0.000000] early_node_map[3] active PFN ranges


[    0.000000]     0: 0x00000000 -> 0x00000001


[    0.000000]     0: 0x00000006 -> 0x0000009f


[    0.000000]     0: 0x00000100 -> 0x00007ffd


[    0.000000] ACPI: PM-Timer IO Port: 0xb008


[    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)


[    0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])


[    0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23


[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)


[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)


[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)


[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)


[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)


[    0.000000] Using ACPI (MADT) for SMP configuration information


[    0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000


[    0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs


[    0.000000] PM: Registered nosave memory: 0000000000001000 - 0000000000006000


[    0.000000] PM: Registered nosave memory: 000000000009f000 - 00000000000a0000


[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000


[    0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000


[    0.000000] Allocating PCI resources starting at 8000000 (gap: 8000000:f7fbc000)


[    0.000000] Booting paravirtualized kernel on KVM


[    0.000000] NR_CPUS:4096 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1


[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff880001e00000 s81944 r8192 d24552 u2097152


[    0.000000] pcpu-alloc: s81944 r8192 d24552 u2097152 alloc=1*2097152


[    0.000000] pcpu-alloc: [0] 0 


[    0.000000] kvm-clock: cpu 0, msr 0:1e13501, primary cpu clock


[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 32115


[    0.000000] Policy zone: DMA32


[    0.000000] Kernel command line: console=ttyS0


[    0.000000] PID hash table entries: 512 (order: 0, 4096 bytes)


[    0.000000] Checking aperture...


[    0.000000] No AGP bridge found


[    0.000000] Memory: 107028k/131060k available (3694k kernel code, 408k absent, 23624k reserved, 5696k data, 964k init)


[    0.000000] Hierarchical RCU implementation.


[    0.000000] NR_IRQS:33024 nr_irqs:256


[    0.000000] Console: colour VGA+ 80x25


[    0.000000] console [ttyS0] enabled


[    0.000000] allocated 1310720 bytes of page_cgroup


[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups


[    0.000000] HPET: 3 timers in total, 0 timers will be used for per-cpu timer


[    0.000000] Detected 2533.412 MHz processor.


[    0.028001] Calibrating delay loop (skipped) preset value.. 5066.82 BogoMIPS (lpj=10133648)


[    0.032258] kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright SGI, All Rights Reserved


kdb_cmd[0]: defcmd archkdb "" "First line arch debugging"


kdb_cmd[8]: defcmd archkdbcpu "" "archkdb with only tasks on cpus"


kdb_cmd[15]: defcmd archkdbshort "" "archkdb with less detailed backtrace"


kdb_cmd[22]: defcmd archkdbcommon "" "Common arch debugging"


[    0.052279] Security Framework initialized


[    0.055281] AppArmor: AppArmor initialized


[    0.056063] Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)


[    0.060094] Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)


[    0.064086] Mount-cache hash table entries: 256


[    0.068195] Initializing cgroup subsys ns


[    0.069959] Initializing cgroup subsys cpuacct


[    0.072006] Initializing cgroup subsys memory


[    0.076023] Initializing cgroup subsys devices


[    0.080005] Initializing cgroup subsys freezer


[    0.082913] Initializing cgroup subsys net_cls


[    0.084100] mce: CPU supports 10 MCE banks


[    0.088041] Performance Events: unsupported p6 CPU model 2 no PMU driver, software events only.


[    0.096021] SMP alternatives: switching to UP code


[    0.107845] Freeing SMP alternatives: 29k freed


[    0.108022] ACPI: Core revision 20090903


[    0.110723] Setting APIC routing to flat


[    0.112173] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1


[    0.113513] CPU0: Intel QEMU Virtual CPU version 0.12.3 stepping 03


[    0.120001] Brought up 1 CPUs


[    0.120001] Total of 1 processors activated (5066.82 BogoMIPS).


[    0.120001] devtmpfs: initialized


[    0.120319] regulator: core version 0.5


[    0.120752] Time: 14:21:27  Date: 03/24/10


[    0.121295] NET: Registered protocol family 16


[    0.121753] ACPI: bus type pci registered


[    0.122149] PCI: Using configuration type 1 for base access


[    0.123262] bio: create slab <bio-0> at 0


[    0.126128] ACPI: Interpreter enabled


[    0.126432] ACPI: (supports S0 S3 S4 S5)


[    0.126818] ACPI: Using IOAPIC for interrupt routing


[    0.129754] ACPI: No dock devices found.


[    0.130081] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug


[    0.130868] ACPI: PCI Root Bridge [PCI0] (0000:00)


[    0.132037] pci 0000:00:01.3: quirk: region b000-b03f claimed by PIIX4 ACPI


[    0.132705] pci 0000:00:01.3: quirk: region b100-b10f claimed by PIIX4 SMB


[    0.143297] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)


[    0.143939] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)


[    0.144345] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)


[    0.144996] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)


[    0.145797] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none


[    0.146455] vgaarb: loaded


[    0.146832] PCI: Using ACPI for IRQ routing


[    0.147408] NetLabel: Initializing


[    0.148006] NetLabel:  domain hash size = 128


[    0.148447] NetLabel:  protocols = UNLABELED CIPSOv4


[    0.148919] NetLabel:  unlabeled traffic allowed by default


[    0.149406] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0


[    0.149851] hpet0: 3 comparators, 64-bit 100.000000 MHz counter


[    0.160054] Switching to clocksource kvm-clock


[    0.160870] AppArmor: AppArmor Filesystem Enabled


[    0.161315] pnp: PnP ACPI init


[    0.161587] ACPI: bus type pnp registered


[    0.162625] pnp: PnP ACPI: found 8 devices


[    0.162962] ACPI: ACPI bus type pnp unregistered


[    0.167632] NET: Registered protocol family 2


[    0.168223] IP route cache hash table entries: 1024 (order: 1, 8192 bytes)


[    0.169194] TCP established hash table entries: 4096 (order: 4, 65536 bytes)


[    0.169792] TCP bind hash table entries: 4096 (order: 4, 65536 bytes)


[    0.170330] TCP: Hash tables configured (established 4096 bind 4096)


[    0.170850] TCP reno registered


[    0.171236] NET: Registered protocol family 1


[    0.171607] pci 0000:00:00.0: Limiting direct PCI/PCI transfers


[    0.172309] pci 0000:00:01.0: PIIX3: Enabling Passive Release


[    0.173126] pci 0000:00:01.0: Activating ISA DMA hang workarounds


[    0.177945] Unpacking initramfs...


[    0.398942] Freeing initrd memory: 7835k freed


[    0.401774] Scanning for low memory corruption every 60 seconds


[    0.402994] audit: initializing netlink socket (disabled)


[    0.403727] type=2000 audit(1269440488.400:1): initialized


[    0.414717] HugeTLB registered 2 MB page size, pre-allocated 0 pages


[    0.416207] VFS: Disk quotas dquot_6.5.2


[    0.416870] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)


[    0.417910] msgmni has been set to 56


[    0.418619] alg: No test for stdrng (krng)


[    0.419263] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)


[    0.420182] io scheduler noop registered


[    0.420743] io scheduler anticipatory registered


[    0.421381] io scheduler deadline registered


[    0.422062] io scheduler cfq registered (default)


[    0.422872] pci-stub: invalid id string ""


[    0.425427] Non-volatile memory driver v1.3


[    0.425972] Linux agpgart interface v0.103


[    0.426316] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled


[    0.426998] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A


[    0.427904] 00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A


[    0.428520] Fixed MDIO Bus: probed


[    0.428848] PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12


[    0.429942] serio: i8042 KBD port at 0x60,0x64 irq 1


[    0.430350] serio: i8042 AUX port at 0x60,0x64 irq 12


[    0.430794] mice: PS/2 mouse device common for all mice


[    0.431474] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0


[    0.432604] cpuidle: using governor ladder


[    0.432953] cpuidle: using governor menu


[    0.433485] TCP cubic registered


[    0.433849] registered taskstats version 1


[    0.434411]   Magic number: 2:93:385


[    0.434767] Freeing unused kernel memory: 964k freed


[    0.435578] Write protecting the kernel read-only data: 8192k


[    0.436196] Freeing unused kernel memory: 384k freed


[    0.436944] Freeing unused kernel memory: 1380k freed


[    0.485966] device-mapper: uevent: version 1.0.3


[    0.486728] device-mapper: ioctl: 4.15.0-ioctl (2009-04-01) initialised: dm-devel@redhat.com


[    0.511928] udevd version 128 started


[    0.542626] dca service started, version 1.12.1


[    0.548078] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 2.0.44-k2


[    0.549251] ixgbe: Copyright (c) 1999-2009 Intel Corporation.


[    0.649034] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11


[    0.649867] ixgbe 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11


[    0.873218] ixgbe: 0000:00:04.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1


[    0.874664] ixgbe 0000:00:04.0: (PCI Express:5.0Gb/s:Width x8) 00:1b:21:3c:9e:40


[    0.875789] ixgbe 0000:00:04.0: MAC: 2, PHY: 9, SFP+: 5, PBA No: e66562-000


[    0.877496] ixgbe 0000:00:04.0: Intel(R) 10 Gigabit Network Connection


[    0.878673] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10


[    0.879476] ixgbe 0000:00:05.0: PCI INT B -> Link[LNKB] -> GSI 10 (level, high) -> IRQ 10


[    1.001120] ixgbe: 0000:00:05.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1


[    1.002593] ixgbe 0000:00:05.0: (PCI Express:5.0Gb/s:Width x8) 00:1b:21:3c:9e:41


[    1.003652] ixgbe 0000:00:05.0: MAC: 2, PHY: 7, SFP+: 3, PBA No: e66562-000


[    1.005237] ixgbe 0000:00:05.0: Intel(R) 10 Gigabit Network Connection



Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)