MSI-X not enabled for ixgbe device-passthrough

MSI-X not enabled for ixgbe device-passthrough

[Hannes Reinecke]

Hi all,

I'm trying to setup a system with device-passthrough for
an ixgbe NIC.
The device itself seems to work, but it isn't using MSI-X.
So some more advanced features like DCB offloading etc
won't work.

lspci output of the device:
07:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter X520-2
        Flags: bus master, fast devsel, latency 0, IRQ 24
        Memory at f5c80000 (64-bit, prefetchable) [size=512K]
        I/O ports at 5000 [size=32]
        Memory at f5c70000 (64-bit, prefetchable) [size=16K]
        [virtual] Expansion ROM at e7100000 [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+ Count=1/1 Enable-
        Capabilities: [70] MSI-X: Enable+ Mask- TabSize=64
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSVoil-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSVoil-
                UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSVoil-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [140] Device Serial Number 40-9e-3c-ff-ff-21-1b-00
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
                IOVSta: Migration-
                Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
                VF offset: 128, stride: 2, Device ID: 10ed
                Supported Page Size: 00000553, System Page Size: 00000001
                VF Migration: offset: 00000000, BIR: 1
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

please let me know if you need more information.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

Re: MSI-X not enabled for ixgbe device-passthrough

[Sheng Yang]

On Wednesday 24 March 2010 23:54:15 Hannes Reinecke wrote:
> Hi all,
> 
> I'm trying to setup a system with device-passthrough for
> an ixgbe NIC.
> The device itself seems to work, but it isn't using MSI-X.
> So some more advanced features like DCB offloading etc
> won't work.

How about lspci result in the guest?

And some guest dmesg would also help.

-- 
regards
Yang, Sheng

> 
> lspci output of the device:
> 07:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network
>  Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter
>  X520-2 Flags: bus master, fast devsel, latency 0, IRQ 24
>         Memory at f5c80000 (64-bit, prefetchable) [size=512K]
>         I/O ports at 5000 [size=32]
>         Memory at f5c70000 (64-bit, prefetchable) [size=16K]
>         [virtual] Expansion ROM at e7100000 [disabled] [size=512K]
>         Capabilities: [40] Power Management version 3
>         Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+
>  Count=1/1 Enable- Capabilities: [70] MSI-X: Enable+ Mask- TabSize=64
>         Capabilities: [a0] Express Endpoint, MSI 00
>         Capabilities: [100] Advanced Error Reporting
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
>  RxOF- MalfTLP- ECRC- UnsupReq+ ACSVoil- UEMsk:  DLP- SDES- TLP- FCP-
>  CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSVoil-
>  UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
>  ECRC- UnsupReq- ACSVoil- CESta:  RxErr- BadTLP- BadDLLP- Rollover-
>  Timeout- NonFatalErr+ CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
>  NonFatalErr- AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
>  ChkEn- Capabilities: [140] Device Serial Number 40-9e-3c-ff-ff-21-1b-00
>  Capabilities: [150] Alternative Routing-ID Interpretation (ARI) ARICap:
>  MFVC- ACS-, Next Function: 1
>                 ARICtl: MFVC- ACS-, Function Group: 0
>         Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
>                 IOVCap: Migration-, Interrupt Message Number: 000
>                 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
>                 IOVSta: Migration-
>                 Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function
>  Dependency Link: 00 VF offset: 128, stride: 2, Device ID: 10ed
>                 Supported Page Size: 00000553, System Page Size: 00000001
>                 VF Migration: offset: 00000000, BIR: 1
>         Kernel driver in use: ixgbe
>         Kernel modules: ixgbe
> 
> please let me know if you need more information.
> 
> Cheers,
> 
> Hannes
> 

Re: MSI-X not enabled for ixgbe device-passthrough

[Hannes Reinecke]

Sheng Yang wrote:
> On Wednesday 24 March 2010 23:54:15 Hannes Reinecke wrote:
>> Hi all,
>>
>> I'm trying to setup a system with device-passthrough for
>> an ixgbe NIC.
>> The device itself seems to work, but it isn't using MSI-X.
>> So some more advanced features like DCB offloading etc
>> won't work.
> 
> How about lspci result in the guest?
> 
I have to boot a different system; will provide it shortly.

> And some guest dmesg would also help.
> 
Ok:

device: 07:00.0: driver="pci-assign" host="07:00.0"

device: 07:00.1: driver="pci-assign" host="07:00.1"

[    0.000000] Initializing cgroup subsys cpuset


[    0.000000] Initializing cgroup subsys cpu


[    0.000000] Linux version 2.6.32.9-0.5-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP 2010-03-15 12:22:00 +0100


[    0.000000] Command line: console=ttyS0


[    0.000000] KERNEL supported cpus:


[    0.000000]   Intel GenuineIntel


[    0.000000]   AMD AuthenticAMD


[    0.000000]   Centaur CentaurHauls


[    0.000000] BIOS-provided physical RAM map:


[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009f400 (usable)


[    0.000000]  BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)


[    0.000000]  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)


[    0.000000]  BIOS-e820: 0000000000100000 - 0000000007ffd000 (usable)


[    0.000000]  BIOS-e820: 0000000007ffd000 - 0000000008000000 (reserved)


[    0.000000]  BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved)


[    0.000000] DMI 2.4 present.


[    0.000000] last_pfn = 0x7ffd max_arch_pfn = 0x400000000


[    0.000000] PAT not supported by CPU.


[    0.000000] Scanning 1 areas for low memory corruption


[    0.000000] modified physical RAM map:


[    0.000000]  modified: 0000000000000000 - 0000000000001000 (usable)


[    0.000000]  modified: 0000000000001000 - 0000000000006000 (reserved)


[    0.000000]  modified: 0000000000006000 - 000000000009f400 (usable)


[    0.000000]  modified: 000000000009f400 - 00000000000a0000 (reserved)


[    0.000000]  modified: 00000000000f0000 - 0000000000100000 (reserved)


[    0.000000]  modified: 0000000000100000 - 0000000007ffd000 (usable)


[    0.000000]  modified: 0000000007ffd000 - 0000000008000000 (reserved)


[    0.000000]  modified: 00000000fffbc000 - 0000000100000000 (reserved)


[    0.000000] init_memory_mapping: 0000000000000000-0000000007ffd000


[    0.000000] RAMDISK: 07849000 - 07feff7e


[    0.000000] ACPI: RSDP 00000000000f8950 00014 (v00 BOCHS )


[    0.000000] ACPI: RSDT 0000000007ffde30 00034 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)


[    0.000000] ACPI: FACP 0000000007fffe70 00074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)


[    0.000000] ACPI: DSDT 0000000007ffdfd0 01E22 (v01   BXPC   BXDSDT 00000001 INTL 20090123)


[    0.000000] ACPI: FACS 0000000007fffe00 00040


[    0.000000] ACPI: SSDT 0000000007ffdf90 00037 (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)


[    0.000000] ACPI: APIC 0000000007ffdeb0 00072 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)


[    0.000000] ACPI: HPET 0000000007ffde70 00038 (v01 BOCHS  BXPCHPET 00000001 BXPC 00000001)


[    0.000000] No NUMA configuration found


[    0.000000] Faking a node at 0000000000000000-0000000007ffd000


[    0.000000] Bootmem setup node 0 0000000000000000-0000000007ffd000


[    0.000000]   NODE_DATA [0000000000009000 - 000000000003cfff]


[    0.000000]   bootmap [000000000003d000 -  000000000003dfff] pages 1


[    0.000000] (7 early reservations) ==> bootmem [0000000000 - 0007ffd000]


[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]


[    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]


[    0.000000]   #2 [0001000000 - 0001cd6778]    TEXT DATA BSS ==> [0001000000 - 0001cd6778]


[    0.000000]   #3 [0007849000 - 0007feff7e]          RAMDISK ==> [0007849000 - 0007feff7e]


[    0.000000]   #4 [000009f400 - 0000100000]    BIOS reserved ==> [000009f400 - 0000100000]


[    0.000000]   #5 [0001cd7000 - 0001cd7049]              BRK ==> [0001cd7000 - 0001cd7049]


[    0.000000]   #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]


[    0.000000] found SMP MP-table at [ffff8800000f89a0] f89a0


[    0.000000] kvm-clock: cpu 0, msr 0:1940501, boot clock


[    0.000000] Zone PFN ranges:


[    0.000000]   DMA      0x00000000 -> 0x00001000


[    0.000000]   DMA32    0x00001000 -> 0x00100000


[    0.000000]   Normal   0x00100000 -> 0x00100000


[    0.000000] Movable zone start PFN for each node


[    0.000000] early_node_map[3] active PFN ranges


[    0.000000]     0: 0x00000000 -> 0x00000001


[    0.000000]     0: 0x00000006 -> 0x0000009f


[    0.000000]     0: 0x00000100 -> 0x00007ffd


[    0.000000] ACPI: PM-Timer IO Port: 0xb008


[    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)


[    0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])


[    0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23


[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)


[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)


[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)


[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)


[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)


[    0.000000] Using ACPI (MADT) for SMP configuration information


[    0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000


[    0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs


[    0.000000] PM: Registered nosave memory: 0000000000001000 - 0000000000006000


[    0.000000] PM: Registered nosave memory: 000000000009f000 - 00000000000a0000


[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000


[    0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000


[    0.000000] Allocating PCI resources starting at 8000000 (gap: 8000000:f7fbc000)


[    0.000000] Booting paravirtualized kernel on KVM


[    0.000000] NR_CPUS:4096 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1


[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff880001e00000 s81944 r8192 d24552 u2097152


[    0.000000] pcpu-alloc: s81944 r8192 d24552 u2097152 alloc=1*2097152


[    0.000000] pcpu-alloc: [0] 0 


[    0.000000] kvm-clock: cpu 0, msr 0:1e13501, primary cpu clock


[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 32115


[    0.000000] Policy zone: DMA32


[    0.000000] Kernel command line: console=ttyS0


[    0.000000] PID hash table entries: 512 (order: 0, 4096 bytes)


[    0.000000] Checking aperture...


[    0.000000] No AGP bridge found


[    0.000000] Memory: 107028k/131060k available (3694k kernel code, 408k absent, 23624k reserved, 5696k data, 964k init)


[    0.000000] Hierarchical RCU implementation.


[    0.000000] NR_IRQS:33024 nr_irqs:256


[    0.000000] Console: colour VGA+ 80x25


[    0.000000] console [ttyS0] enabled


[    0.000000] allocated 1310720 bytes of page_cgroup


[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups


[    0.000000] HPET: 3 timers in total, 0 timers will be used for per-cpu timer


[    0.000000] Detected 2533.412 MHz processor.


[    0.028001] Calibrating delay loop (skipped) preset value.. 5066.82 BogoMIPS (lpj=10133648)


[    0.032258] kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright SGI, All Rights Reserved


kdb_cmd[0]: defcmd archkdb "" "First line arch debugging"


kdb_cmd[8]: defcmd archkdbcpu "" "archkdb with only tasks on cpus"


kdb_cmd[15]: defcmd archkdbshort "" "archkdb with less detailed backtrace"


kdb_cmd[22]: defcmd archkdbcommon "" "Common arch debugging"


[    0.052279] Security Framework initialized


[    0.055281] AppArmor: AppArmor initialized


[    0.056063] Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)


[    0.060094] Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)


[    0.064086] Mount-cache hash table entries: 256


[    0.068195] Initializing cgroup subsys ns


[    0.069959] Initializing cgroup subsys cpuacct


[    0.072006] Initializing cgroup subsys memory


[    0.076023] Initializing cgroup subsys devices


[    0.080005] Initializing cgroup subsys freezer


[    0.082913] Initializing cgroup subsys net_cls


[    0.084100] mce: CPU supports 10 MCE banks


[    0.088041] Performance Events: unsupported p6 CPU model 2 no PMU driver, software events only.


[    0.096021] SMP alternatives: switching to UP code


[    0.107845] Freeing SMP alternatives: 29k freed


[    0.108022] ACPI: Core revision 20090903


[    0.110723] Setting APIC routing to flat


[    0.112173] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1


[    0.113513] CPU0: Intel QEMU Virtual CPU version 0.12.3 stepping 03


[    0.120001] Brought up 1 CPUs


[    0.120001] Total of 1 processors activated (5066.82 BogoMIPS).


[    0.120001] devtmpfs: initialized


[    0.120319] regulator: core version 0.5


[    0.120752] Time: 14:21:27  Date: 03/24/10


[    0.121295] NET: Registered protocol family 16


[    0.121753] ACPI: bus type pci registered


[    0.122149] PCI: Using configuration type 1 for base access


[    0.123262] bio: create slab <bio-0> at 0


[    0.126128] ACPI: Interpreter enabled


[    0.126432] ACPI: (supports S0 S3 S4 S5)


[    0.126818] ACPI: Using IOAPIC for interrupt routing


[    0.129754] ACPI: No dock devices found.


[    0.130081] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug


[    0.130868] ACPI: PCI Root Bridge [PCI0] (0000:00)


[    0.132037] pci 0000:00:01.3: quirk: region b000-b03f claimed by PIIX4 ACPI


[    0.132705] pci 0000:00:01.3: quirk: region b100-b10f claimed by PIIX4 SMB


[    0.143297] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)


[    0.143939] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)


[    0.144345] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)


[    0.144996] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)


[    0.145797] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none


[    0.146455] vgaarb: loaded


[    0.146832] PCI: Using ACPI for IRQ routing


[    0.147408] NetLabel: Initializing


[    0.148006] NetLabel:  domain hash size = 128


[    0.148447] NetLabel:  protocols = UNLABELED CIPSOv4


[    0.148919] NetLabel:  unlabeled traffic allowed by default


[    0.149406] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0


[    0.149851] hpet0: 3 comparators, 64-bit 100.000000 MHz counter


[    0.160054] Switching to clocksource kvm-clock


[    0.160870] AppArmor: AppArmor Filesystem Enabled


[    0.161315] pnp: PnP ACPI init


[    0.161587] ACPI: bus type pnp registered


[    0.162625] pnp: PnP ACPI: found 8 devices


[    0.162962] ACPI: ACPI bus type pnp unregistered


[    0.167632] NET: Registered protocol family 2


[    0.168223] IP route cache hash table entries: 1024 (order: 1, 8192 bytes)


[    0.169194] TCP established hash table entries: 4096 (order: 4, 65536 bytes)


[    0.169792] TCP bind hash table entries: 4096 (order: 4, 65536 bytes)


[    0.170330] TCP: Hash tables configured (established 4096 bind 4096)


[    0.170850] TCP reno registered


[    0.171236] NET: Registered protocol family 1


[    0.171607] pci 0000:00:00.0: Limiting direct PCI/PCI transfers


[    0.172309] pci 0000:00:01.0: PIIX3: Enabling Passive Release


[    0.173126] pci 0000:00:01.0: Activating ISA DMA hang workarounds


[    0.177945] Unpacking initramfs...


[    0.398942] Freeing initrd memory: 7835k freed


[    0.401774] Scanning for low memory corruption every 60 seconds


[    0.402994] audit: initializing netlink socket (disabled)


[    0.403727] type=2000 audit(1269440488.400:1): initialized


[    0.414717] HugeTLB registered 2 MB page size, pre-allocated 0 pages


[    0.416207] VFS: Disk quotas dquot_6.5.2


[    0.416870] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)


[    0.417910] msgmni has been set to 56


[    0.418619] alg: No test for stdrng (krng)


[    0.419263] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)


[    0.420182] io scheduler noop registered


[    0.420743] io scheduler anticipatory registered


[    0.421381] io scheduler deadline registered


[    0.422062] io scheduler cfq registered (default)


[    0.422872] pci-stub: invalid id string ""


[    0.425427] Non-volatile memory driver v1.3


[    0.425972] Linux agpgart interface v0.103


[    0.426316] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled


[    0.426998] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A


[    0.427904] 00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A


[    0.428520] Fixed MDIO Bus: probed


[    0.428848] PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12


[    0.429942] serio: i8042 KBD port at 0x60,0x64 irq 1


[    0.430350] serio: i8042 AUX port at 0x60,0x64 irq 12


[    0.430794] mice: PS/2 mouse device common for all mice


[    0.431474] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0


[    0.432604] cpuidle: using governor ladder


[    0.432953] cpuidle: using governor menu


[    0.433485] TCP cubic registered


[    0.433849] registered taskstats version 1


[    0.434411]   Magic number: 2:93:385


[    0.434767] Freeing unused kernel memory: 964k freed


[    0.435578] Write protecting the kernel read-only data: 8192k


[    0.436196] Freeing unused kernel memory: 384k freed


[    0.436944] Freeing unused kernel memory: 1380k freed


[    0.485966] device-mapper: uevent: version 1.0.3


[    0.486728] device-mapper: ioctl: 4.15.0-ioctl (2009-04-01) initialised: dm-devel@redhat.com


[    0.511928] udevd version 128 started


[    0.542626] dca service started, version 1.12.1


[    0.548078] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 2.0.44-k2


[    0.549251] ixgbe: Copyright (c) 1999-2009 Intel Corporation.


[    0.649034] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11


[    0.649867] ixgbe 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11


[    0.873218] ixgbe: 0000:00:04.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1


[    0.874664] ixgbe 0000:00:04.0: (PCI Express:5.0Gb/s:Width x8) 00:1b:21:3c:9e:40


[    0.875789] ixgbe 0000:00:04.0: MAC: 2, PHY: 9, SFP+: 5, PBA No: e66562-000


[    0.877496] ixgbe 0000:00:04.0: Intel(R) 10 Gigabit Network Connection


[    0.878673] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10


[    0.879476] ixgbe 0000:00:05.0: PCI INT B -> Link[LNKB] -> GSI 10 (level, high) -> IRQ 10


[    1.001120] ixgbe: 0000:00:05.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1


[    1.002593] ixgbe 0000:00:05.0: (PCI Express:5.0Gb/s:Width x8) 00:1b:21:3c:9e:41


[    1.003652] ixgbe 0000:00:05.0: MAC: 2, PHY: 7, SFP+: 3, PBA No: e66562-000


[    1.005237] ixgbe 0000:00:05.0: Intel(R) 10 Gigabit Network Connection



Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

Re: MSI-X not enabled for ixgbe device-passthrough

[Chris Wright]

* Hannes Reinecke (hare@suse.de) wrote:
> Hi all,
> 
> I'm trying to setup a system with device-passthrough for
> an ixgbe NIC.
> The device itself seems to work, but it isn't using MSI-X.
> So some more advanced features like DCB offloading etc
> won't work.

Please send the relevant dmesg from the guest when it initializes the
device.  BTW, more typical case for that NIC is to assign the VF to the
guest, not the whole PF.

thanks,
-chris

Re: MSI-X not enabled for ixgbe device-passthrough

[Hannes Reinecke]

Chris Wright wrote:
> * Hannes Reinecke (hare@suse.de) wrote:
>> Hi all,
>>
>> I'm trying to setup a system with device-passthrough for
>> an ixgbe NIC.
>> The device itself seems to work, but it isn't using MSI-X.
>> So some more advanced features like DCB offloading etc
>> won't work.
> 
> Please send the relevant dmesg from the guest when it initializes the
> device.  BTW, more typical case for that NIC is to assign the VF to the
> guest, not the whole PF.
> 
Yes, I know. But my kernel I'm testing with doesn't have one for ixgbe.
So I tested that one. And KVM really should enable MSI-X here,
VFs notwithstanding.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

Re: MSI-X not enabled for ixgbe device-passthrough

[Chris Wright]

* Hannes Reinecke (hare@suse.de) wrote:
> Chris Wright wrote:
> > * Hannes Reinecke (hare@suse.de) wrote:
> >> Hi all,
> >>
> >> I'm trying to setup a system with device-passthrough for
> >> an ixgbe NIC.
> >> The device itself seems to work, but it isn't using MSI-X.
> >> So some more advanced features like DCB offloading etc
> >> won't work.
> > 
> > Please send the relevant dmesg from the guest when it initializes the
> > device.  BTW, more typical case for that NIC is to assign the VF to the
> > guest, not the whole PF.
> > 
> Yes, I know. But my kernel I'm testing with doesn't have one for ixgbe.

Ah, you mean it's an older (heh, ok, older actually means not brand new
upstream) kernel, w/out the recent sr-iov additions, fair enough.

> So I tested that one. And KVM really should enable MSI-X here,
> VFs notwithstanding.

Yeah, although it's not just KVM involved, it's very much driven by
the guest too.  The guest will see (or at least should see) the MSI-X
capability, and decide based on the number of queues whether to enable
MSI-X (completely driver dependent here).

Did you have a chance to boot the guest again, and send the lspci -vvv from
the guest POV?  You should see two PCI capabilities (MSI and 0x40 and
MSI-X at 0x50).

Actually, I have one of these devices, let me give it a quick test...
working fine here.  Here's some relevant information:

Host:

$ sudo lspci -v -s 04:00.0
04:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Ethernet Server Adapter X520-2
	Flags: bus master, fast devsel, latency 0, IRQ 16
	Memory at f8080000 (64-bit, prefetchable) [size=512K]
	I/O ports at d020 [size=32]
	Memory at f8104000 (64-bit, prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
	Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
	Capabilities: [a0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-3f-a1-b0
	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [160] Single Root I/O Virtualization (SR-IOV)

$ grep kvm /proc/interrupts 
  95:        289          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      kvm_assigned_msix_device
  96:         32          0          0          0        292          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      kvm_assigned_msix_device
  97:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      kvm_assigned_msix_device

Guest side:
$ sudo lspci -vvv -s 00:04.0
00:04.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Ethernet Server Adapter X520-2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at f2080000 (32-bit, non-prefetchable)
	Region 2: I/O ports at c200
	Region 4: Memory at f2100000 (32-bit, non-prefetchable)
	Capabilities: [40] MSI: Enable- Count=1/1 Maskable- 64bit-
		Address: 00000000  Data: 0000
	Capabilities: [50] MSI-X: Enable+ Count=64 Masked-
		Vector table: BAR=4 offset=00000000
		PBA: BAR=4 offset=00002000

$ grep eth1 /proc/interrupts
177:        387       PCI-MSI-X  eth1-rx-0
185:        421       PCI-MSI-X  eth1-tx-0
193:          2       PCI-MSI-X  eth1:lsc

$ dmesg | grep -e 00:04.0 -e ixgbe
ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 2.0.8-k2
ixgbe: Copyright (c) 1999-2009 Intel Corporation.
ACPI: PCI Interrupt 0000:00:04.0[A] -> Link [LNKD] -> GSI 10 (level, high) -> IRQ 10
PCI: Setting latency timer of device 0000:00:04.0 to 64
ixgbe: 0000:00:04.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
ixgbe 0000:00:04.0: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:3f:a1:b0
ixgbe 0000:00:04.0: MAC: 2, PHY: 11, SFP+: 5, PBA No: e66562-003
ixgbe 0000:00:04.0: Intel(R) 10 Gigabit Network Connection
ixgbe: eth1 NIC Link is Up 10 Gbps, Flow Control: None

Re: MSI-X not enabled for ixgbe device-passthrough

[Hannes Reinecke]

Chris Wright wrote:
> * Hannes Reinecke (hare@suse.de) wrote:
>> Chris Wright wrote:
>>> * Hannes Reinecke (hare@suse.de) wrote:
>>>> Hi all,
>>>>
>>>> I'm trying to setup a system with device-passthrough for
>>>> an ixgbe NIC.
>>>> The device itself seems to work, but it isn't using MSI-X.
>>>> So some more advanced features like DCB offloading etc
>>>> won't work.
>>> Please send the relevant dmesg from the guest when it initializes the
>>> device.  BTW, more typical case for that NIC is to assign the VF to the
>>> guest, not the whole PF.
>>>
>> Yes, I know. But my kernel I'm testing with doesn't have one for ixgbe.
> 
> Ah, you mean it's an older (heh, ok, older actually means not brand new
> upstream) kernel, w/out the recent sr-iov additions, fair enough.
> 
>> So I tested that one. And KVM really should enable MSI-X here,
>> VFs notwithstanding.
> 
> Yeah, although it's not just KVM involved, it's very much driven by
> the guest too.  The guest will see (or at least should see) the MSI-X
> capability, and decide based on the number of queues whether to enable
> MSI-X (completely driver dependent here).
> 
> Did you have a chance to boot the guest again, and send the lspci -vvv from
> the guest POV?  You should see two PCI capabilities (MSI and 0x40 and
> MSI-X at 0x50).
> 
> Actually, I have one of these devices, let me give it a quick test...
> working fine here.  Here's some relevant information:
> 
> Host:
> 
> $ sudo lspci -v -s 04:00.0
> 04:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)
> 	Subsystem: Intel Corporation Ethernet Server Adapter X520-2
> 	Flags: bus master, fast devsel, latency 0, IRQ 16
> 	Memory at f8080000 (64-bit, prefetchable) [size=512K]
> 	I/O ports at d020 [size=32]
> 	Memory at f8104000 (64-bit, prefetchable) [size=16K]
> 	Capabilities: [40] Power Management version 3
> 	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> 	Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
> 	Capabilities: [a0] Express Endpoint, MSI 00
> 	Capabilities: [100] Advanced Error Reporting
> 	Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-3f-a1-b0
> 	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
> 	Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
> 
> $ grep kvm /proc/interrupts 
>   95:        289          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      kvm_assigned_msix_device
>   96:         32          0          0          0        292          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      kvm_assigned_msix_device
>   97:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      kvm_assigned_msix_device
> 
> Guest side:
> $ sudo lspci -vvv -s 00:04.0
> 00:04.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)
> 	Subsystem: Intel Corporation Ethernet Server Adapter X520-2
> 	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 	Latency: 0, Cache Line Size: 64 bytes
> 	Interrupt: pin A routed to IRQ 11
> 	Region 0: Memory at f2080000 (32-bit, non-prefetchable)
> 	Region 2: I/O ports at c200
> 	Region 4: Memory at f2100000 (32-bit, non-prefetchable)
> 	Capabilities: [40] MSI: Enable- Count=1/1 Maskable- 64bit-
> 		Address: 00000000  Data: 0000
> 	Capabilities: [50] MSI-X: Enable+ Count=64 Masked-
> 		Vector table: BAR=4 offset=00000000
> 		PBA: BAR=4 offset=00002000
> 
> $ grep eth1 /proc/interrupts
> 177:        387       PCI-MSI-X  eth1-rx-0
> 185:        421       PCI-MSI-X  eth1-tx-0
> 193:          2       PCI-MSI-X  eth1:lsc
> 
> $ dmesg | grep -e 00:04.0 -e ixgbe
> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 2.0.8-k2
> ixgbe: Copyright (c) 1999-2009 Intel Corporation.
> ACPI: PCI Interrupt 0000:00:04.0[A] -> Link [LNKD] -> GSI 10 (level, high) -> IRQ 10
> PCI: Setting latency timer of device 0000:00:04.0 to 64
> ixgbe: 0000:00:04.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
> ixgbe 0000:00:04.0: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:3f:a1:b0
> ixgbe 0000:00:04.0: MAC: 2, PHY: 11, SFP+: 5, PBA No: e66562-003
> ixgbe 0000:00:04.0: Intel(R) 10 Gigabit Network Connection
> ixgbe: eth1 NIC Link is Up 10 Gbps, Flow Control: None
> 
Ah. So I'll have to shout at Alex Graf.

No problems there :-)

Thanks for your help.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

Re: MSI-X not enabled for ixgbe device-passthrough

[Chris Wright]

* Hannes Reinecke (hare@suse.de) wrote:
> Ah. So I'll have to shout at Alex Graf.
> 
> No problems there :-)

I like that, when in doubt, shout at Alex ;-)

thanks,
-chris

Re: MSI-X not enabled for ixgbe device-passthrough

[Alexander Graf]

Am 29.03.2010 um 18:46 schrieb Chris Wright <chrisw@sous-sol.org>:

> * Hannes Reinecke (hare@suse.de) wrote:
>> Ah. So I'll have to shout at Alex Graf.
>>
>> No problems there :-)
>
> I like that, when in doubt, shout at Alex ;-)

Yep, whenever in doubt, just shout at me :).


Alex

>