* Error message when trying to use Infiniband virtual functions in virtual machine
@ 2013-01-23 15:29 Mathis GAVILLON
[not found] ` <CAOoYFkGrsWKGaFcNAQQnkks2Ku6O8xkPk=8BGQfRR74QhRU=VA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 16+ messages in thread
From: Mathis GAVILLON @ 2013-01-23 15:29 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
Hi,
I'm working on a project in which I try to virtualize Infiniband. I
use ConnectX3 Infiniband cards with FW 2.9.1200.
I've installed a Fedora 18 OS with 3.8.0rc2 recompiled kernel. I use
KVM 1.2 to run virtual machines.
For my tests, I've created one virtual machine on which I've installed
RHEL6.3 with 3.8.0rc2 recompiled kernel.
On the host, I've loaded mlx4_core with num_vfs=2. Two virtual
functions are normaly available.
I've linked one virtual function with the VM. The return of lspci on
the VM is normal :
[root@welling0 ~]# lspci -s 00:08.0
00:08.0 Network controller: Mellanox Technologies MT27500 Family
[ConnectX-3 Virtual Function]
All sould work fine. But when I try to load mlx4_core driver on my VM,
this error message appears :
mlx4_core 0000:00:08.0: failed execution of VHCR_POST commandopcode 0x31
Here is the result of dmesg command :
[root@welling0 ~]# dmesg
mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011)
mlx4_core: Initializing 0000:00:08.0
mlx4_core 0000:00:08.0: Detected virtual function - running in slave mode
mlx4_core 0000:00:08.0: Sending reset
mlx4_core 0000:00:08.0: Sending vhcr0
mlx4_core 0000:00:08.0: 64B EQEs/CQEs supported by the device but not enabled
mlx4_core 0000:00:08.0: HCA minimum page size:512
mlx4_core 0000:00:08.0: irq 48 for MSI/MSI-X
mlx4_core 0000:00:08.0: irq 49 for MSI/MSI-X
mlx4_core 0000:00:08.0: failed execution of VHCR_POST commandopcode 0x31
mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 49).
mlx4_core 0000:00:08.0: Trying again without MSI-X.
mlx4_core: probe of 0000:00:08.0 failed with error -16
Have you got any idea about this error ?
Thank's for your help !!
Mathis
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread[parent not found: <CAOoYFkGrsWKGaFcNAQQnkks2Ku6O8xkPk=8BGQfRR74QhRU=VA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <CAOoYFkGrsWKGaFcNAQQnkks2Ku6O8xkPk=8BGQfRR74QhRU=VA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-01-24 14:41 ` Or Gerlitz [not found] ` <CAOoYFkFQOyWbrZtufpk=82ButMxPjsue1Psih40h=F1=U7ng6g@mail.gmail.com> 0 siblings, 1 reply; 16+ messages in thread From: Or Gerlitz @ 2013-01-24 14:41 UTC (permalink / raw) To: Mathis GAVILLON; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Jack Morgenstein On Wed, Jan 23, 2013 at 5:29 PM, Mathis GAVILLON <jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > I'm working on a project in which I try to virtualize Infiniband. I use ConnectX3 Infiniband cards with FW 2.9.1200. > I've installed a Fedora 18 OS with 3.8.0rc2 recompiled kernel. I use KVM 1.2 to run virtual machines. Your firmware version is a bit old... if you can upgrade to the latest GA on the Mellanox site (2.11.0500) would be good, So what version of qemu is installed on the distro you use? Are you able to work over a ConnectX VF if probed to the host? e.g use num_vfs=3 probe_vf=1 and see how the related mlx4 device (e.g mlx4_1) is functioning. Also would be of help if you probe mlx4_core on both VM and host with debug_level=1 Or. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <CAOoYFkFQOyWbrZtufpk=82ButMxPjsue1Psih40h=F1=U7ng6g@mail.gmail.com>]
[parent not found: <CAOoYFkGe=kKDDKvhOdRVCXQbGfV52q38dz6cYTpQXmFGOsBwsg@mail.gmail.com>]
[parent not found: <CAOoYFkGe=kKDDKvhOdRVCXQbGfV52q38dz6cYTpQXmFGOsBwsg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Fwd: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <CAOoYFkGe=kKDDKvhOdRVCXQbGfV52q38dz6cYTpQXmFGOsBwsg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-01-28 8:29 ` Mathis GAVILLON [not found] ` <CAOoYFkFVvK+1YNJ_N6r0N_3ZMF6P9Q+0dbsDg4v8=3JfDPVOOA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 16+ messages in thread From: Mathis GAVILLON @ 2013-01-28 8:29 UTC (permalink / raw) To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb The version of firmware I gave you was wrong (I've two cards (ConnectX2 and ConnectX3, I only use ConnectX 3) in my computer and I took the bad one). The firmware installed on my ConnectX3 Infiniband Card is 2.11.500. KVM is 1.2.2-1 version. I tried to load the driver with options you gave me but the error message is the same. Nothing else happen. Here is the dmesg command return from host (mlx4_core loaded with options debug_level=1, num_vfs=3, probe_vf=1) : [root@barb0 ~]# dmesg [65182.838953] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) [65182.838955] mlx4_core: Initializing 0000:05:00.0 [65182.839144] mlx4_core 0000:05:00.0: Enabling SR-IOV with 3 VFs [65182.940316] pci 0000:05:00.1: [15b3:1002] type 00 class 0x0c0600 [65182.941105] pci 0000:05:00.2: [15b3:1002] type 00 class 0x0c0600 [65182.941876] pci 0000:05:00.3: [15b3:1002] type 00 class 0x0c0600 [65182.942585] mlx4_core 0000:05:00.0: Running in master mode [65183.945306] mlx4_core 0000:05:00.0: FW version 2.9.1200 (cmd intf rev 3), max commands 16 [65183.945309] mlx4_core 0000:05:00.0: Catastrophic error buffer at 0x1f020, size 0x10, BAR 0 [65183.945311] mlx4_core 0000:05:00.0: Communication vector bar:2 offset:0x800 [65183.945312] mlx4_core 0000:05:00.0: FW size 385 KB [65183.945313] mlx4_core 0000:05:00.0: Clear int @ f0058, BAR 0 [65183.947177] mlx4_core 0000:05:00.0: Mapped 26 chunks/6168 KB for FW. [65184.614576] mlx4_core 0000:05:00.0: BlueFlame available (reg size 512, regs/page 8) [65184.614742] mlx4_core 0000:05:00.0: Base MM extensions: flags 00000cc0, rsvd L_Key 00000500 [65184.614743] mlx4_core 0000:05:00.0: Max ICM size 4294967296 MB [65184.614745] mlx4_core 0000:05:00.0: Max QPs: 16777216, reserved QPs: 64, entry size: 256 [65184.614747] mlx4_core 0000:05:00.0: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 [65184.614748] mlx4_core 0000:05:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 128 [65184.614750] mlx4_core 0000:05:00.0: Max EQs: 32, reserved EQs: 28, entry size: 128 [65184.614751] mlx4_core 0000:05:00.0: reserved MPTs: 16, reserved MTTs: 16 [65184.614753] mlx4_core 0000:05:00.0: Max PDs: 131072, reserved PDs: 4, reserved UARs: 7 [65184.614754] mlx4_core 0000:05:00.0: Max QP/MCG: 131072, reserved MGMs: 0 [65184.614756] mlx4_core 0000:05:00.0: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 [65184.614758] mlx4_core 0000:05:00.0: Local CA ACK delay: 15, max MTU: 4096, port width cap: 3 [65184.614759] mlx4_core 0000:05:00.0: Max SQ desc size: 1008, max SQ S/G: 62 [65184.614760] mlx4_core 0000:05:00.0: Max RQ desc size: 512, max RQ S/G: 32 [65184.614762] mlx4_core 0000:05:00.0: Max GSO size: 131072 [65184.614763] mlx4_core 0000:05:00.0: Max counters: 256 [65184.614764] mlx4_core 0000:05:00.0: Max RSS Table size: 256 [65184.614766] mlx4_core 0000:05:00.0: DEV_CAP flags: [65184.614767] mlx4_core 0000:05:00.0: RC transport [65184.614768] mlx4_core 0000:05:00.0: UC transport [65184.614769] mlx4_core 0000:05:00.0: UD transport [65184.614771] mlx4_core 0000:05:00.0: XRC transport [65184.614772] mlx4_core 0000:05:00.0: FCoIB support [65184.614773] mlx4_core 0000:05:00.0: SRQ support [65184.614774] mlx4_core 0000:05:00.0: IPoIB checksum offload [65184.614775] mlx4_core 0000:05:00.0: P_Key violation counter [65184.614777] mlx4_core 0000:05:00.0: Q_Key violation counter [65184.614778] mlx4_core 0000:05:00.0: DPDP [65184.614779] mlx4_core 0000:05:00.0: Big LSO headers [65184.614780] mlx4_core 0000:05:00.0: APM support [65184.614782] mlx4_core 0000:05:00.0: Atomic ops support [65184.614783] mlx4_core 0000:05:00.0: Address vector port checking support [65184.614784] mlx4_core 0000:05:00.0: UD multicast support [65184.614785] mlx4_core 0000:05:00.0: Router support [65184.614787] mlx4_core 0000:05:00.0: IBoE support [65184.614788] mlx4_core 0000:05:00.0: Unicast loopback support [65184.614789] mlx4_core 0000:05:00.0: UDP RSS support [65184.614790] mlx4_core 0000:05:00.0: Unicast VEP steering support [65184.614792] mlx4_core 0000:05:00.0: Multicast VEP steering support [65184.614793] mlx4_core 0000:05:00.0: Counters support [65184.614795] mlx4_core 0000:05:00.0: RSS support [65184.614796] mlx4_core 0000:05:00.0: RSS Toeplitz Hash Function support [65184.614797] mlx4_core 0000:05:00.0: RSS XOR Hash Function support [65184.614800] mlx4_core 0000:05:00.0: Steering mode is: B0 steering, oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10 [65184.614807] mlx4_core 0000:05:00.0: profile[ 0] ( CMPT): 2^26 entries @ 0x 0, size 0x 100000000 [65184.614809] mlx4_core 0000:05:00.0: profile[ 1] (RDMARC): 2^22 entries @ 0x 100000000, size 0x 8000000 [65184.614811] mlx4_core 0000:05:00.0: profile[ 2] ( MTT): 2^24 entries @ 0x 108000000, size 0x 8000000 [65184.614813] mlx4_core 0000:05:00.0: profile[ 3] ( QP): 2^18 entries @ 0x 110000000, size 0x 4000000 [65184.614815] mlx4_core 0000:05:00.0: profile[ 4] ( DMPT): 2^19 entries @ 0x 114000000, size 0x 2000000 [65184.614816] mlx4_core 0000:05:00.0: profile[ 5] ( ALTC): 2^18 entries @ 0x 116000000, size 0x 1000000 [65184.614818] mlx4_core 0000:05:00.0: profile[ 6] ( SRQ): 2^16 entries @ 0x 117000000, size 0x 800000 [65184.614820] mlx4_core 0000:05:00.0: profile[ 7] ( CQ): 2^16 entries @ 0x 117800000, size 0x 800000 [65184.614822] mlx4_core 0000:05:00.0: profile[ 8] ( MCG): 2^13 entries @ 0x 118000000, size 0x 800000 [65184.614824] mlx4_core 0000:05:00.0: profile[ 9] ( AUXC): 2^18 entries @ 0x 118800000, size 0x 40000 [65184.614826] mlx4_core 0000:05:00.0: profile[10] ( EQ): 2^10 entries @ 0x 118840000, size 0x 20000 [65184.614827] mlx4_core 0000:05:00.0: HCA context memory: reserving 4596096 KB [65184.614844] mlx4_core 0000:05:00.0: 4596096 KB of HCA context requires 9068 KB aux memory. [65184.644227] mlx4_core 0000:05:00.0: Mapped 39 chunks/9068 KB for ICM aux. [65184.645547] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 0 for ICM. [65184.646846] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 40000000 for ICM. [65184.648146] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 80000000 for ICM. [65184.648483] mlx4_core 0000:05:00.0: Mapped 1 chunks/64 KB at c0000000 for ICM. [65184.649133] mlx4_core 0000:05:00.0: Mapped 1 chunks/128 KB at 118840000 for ICM. [65184.650413] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 108000000 for ICM. [65184.651737] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 114000000 for ICM. [65184.653016] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 110000000 for ICM. [65184.654304] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118800000 for ICM. [65184.655585] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 116000000 for ICM. [65184.656866] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 100000000 for ICM. [65184.658146] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 117800000 for ICM. [65184.659426] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 117000000 for ICM. [65184.660705] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118000000 for ICM. [65184.661986] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118040000 for ICM. [65184.663265] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118080000 for ICM. [65184.664550] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1180c0000 for ICM. [65184.665839] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118100000 for ICM. [65184.667117] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118140000 for ICM. [65184.668395] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118180000 for ICM. [65184.669675] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1181c0000 for ICM. [65184.670955] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118200000 for ICM. [65184.672234] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118240000 for ICM. [65184.673511] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118280000 for ICM. [65184.674794] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1182c0000 for ICM. [65184.676070] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118300000 for ICM. [65184.677351] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118340000 for ICM. [65184.678629] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118380000 for ICM. [65184.679906] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1183c0000 for ICM. [65184.681185] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118400000 for ICM. [65184.682463] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118440000 for ICM. [65184.683742] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118480000 for ICM. [65184.685028] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1184c0000 for ICM. [65184.686306] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118500000 for ICM. [65184.687583] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118540000 for ICM. [65184.688869] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118580000 for ICM. [65184.690148] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1185c0000 for ICM. [65184.691428] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118600000 for ICM. [65184.692706] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118640000 for ICM. [65184.693984] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118680000 for ICM. [65184.695268] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1186c0000 for ICM. [65184.696546] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118700000 for ICM. [65184.697825] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118740000 for ICM. [65184.699103] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118780000 for ICM. [65184.700380] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1187c0000 for ICM. [65185.202376] mlx4_core 0000:05:00.0: Started init_resource_tracker: 80 slaves [65185.202819] mlx4_core 0000:05:00.0: irq 176 for MSI/MSI-X [65185.202826] mlx4_core 0000:05:00.0: irq 177 for MSI/MSI-X [65185.207838] mlx4_core 0000:05:00.0: NOP command IRQ test passed [65185.208638] mlx4_core: Initializing 0000:82:00.0 [65185.208742] mlx4_core 0000:82:00.0: Enabling SR-IOV with 3 VFs [65185.309000] pci 0000:82:00.1: [15b3:1004] type 00 class 0x028000 [65185.309781] pci 0000:82:00.2: [15b3:1004] type 00 class 0x028000 [65185.310507] pci 0000:82:00.3: [15b3:1004] type 00 class 0x028000 [65185.311176] mlx4_core 0000:82:00.0: Running in master mode [65186.312479] mlx4_core 0000:82:00.0: FW version 2.11.500 (cmd intf rev 3), max commands 16 [65186.312482] mlx4_core 0000:82:00.0: Catastrophic error buffer at 0x1f020, size 0x10, BAR 0 [65186.312484] mlx4_core 0000:82:00.0: Communication vector bar:2 offset:0x800 [65186.312485] mlx4_core 0000:82:00.0: FW size 385 KB [65186.312486] mlx4_core 0000:82:00.0: Clear int @ f0058, BAR 0 [65186.313785] mlx4_core 0000:82:00.0: Mapped 26 chunks/6168 KB for FW. [65188.980410] mlx4_core 0000:82:00.0: BlueFlame available (reg size 512, regs/page 8) [65188.980535] mlx4_core 0000:82:00.0: Base MM extensions: flags 00104cc0, rsvd L_Key 00008000 [65188.980536] mlx4_core 0000:82:00.0: Max ICM size 4294967296 MB [65188.980538] mlx4_core 0000:82:00.0: Max QPs: 16777216, reserved QPs: 64, entry size: 256 [65188.980540] mlx4_core 0000:82:00.0: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 [65188.980541] mlx4_core 0000:82:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 128 [65188.980543] mlx4_core 0000:82:00.0: Max EQs: 32, reserved EQs: 28, entry size: 128 [65188.980544] mlx4_core 0000:82:00.0: reserved MPTs: 256, reserved MTTs: 32 [65188.980546] mlx4_core 0000:82:00.0: Max PDs: 131072, reserved PDs: 4, reserved UARs: 7 [65188.980548] mlx4_core 0000:82:00.0: Max QP/MCG: 131072, reserved MGMs: 0 [65188.980549] mlx4_core 0000:82:00.0: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 [65188.980551] mlx4_core 0000:82:00.0: Local CA ACK delay: 15, max MTU: 4096, port width cap: 2 [65188.980552] mlx4_core 0000:82:00.0: Max SQ desc size: 1008, max SQ S/G: 62 [65188.980554] mlx4_core 0000:82:00.0: Max RQ desc size: 512, max RQ S/G: 32 [65188.980555] mlx4_core 0000:82:00.0: Max GSO size: 131072 [65188.980556] mlx4_core 0000:82:00.0: Max counters: 256 [65188.980558] mlx4_core 0000:82:00.0: Max RSS Table size: 256 [65188.980559] mlx4_core 0000:82:00.0: DEV_CAP flags: [65188.980560] mlx4_core 0000:82:00.0: RC transport [65188.980561] mlx4_core 0000:82:00.0: UC transport [65188.980563] mlx4_core 0000:82:00.0: UD transport [65188.980564] mlx4_core 0000:82:00.0: XRC transport [65188.980565] mlx4_core 0000:82:00.0: FCoIB support [65188.980566] mlx4_core 0000:82:00.0: SRQ support [65188.980568] mlx4_core 0000:82:00.0: IPoIB checksum offload [65188.980569] mlx4_core 0000:82:00.0: P_Key violation counter [65188.980570] mlx4_core 0000:82:00.0: Q_Key violation counter [65188.980571] mlx4_core 0000:82:00.0: DPDP [65188.980573] mlx4_core 0000:82:00.0: Big LSO headers [65188.980574] mlx4_core 0000:82:00.0: APM support [65188.980575] mlx4_core 0000:82:00.0: Atomic ops support [65188.980576] mlx4_core 0000:82:00.0: Address vector port checking support [65188.980578] mlx4_core 0000:82:00.0: UD multicast support [65188.980579] mlx4_core 0000:82:00.0: Router support [65188.980580] mlx4_core 0000:82:00.0: IBoE support [65188.980582] mlx4_core 0000:82:00.0: Unicast loopback support [65188.980583] mlx4_core 0000:82:00.0: FCS header control [65188.980584] mlx4_core 0000:82:00.0: UDP RSS support [65188.980586] mlx4_core 0000:82:00.0: Unicast VEP steering support [65188.980587] mlx4_core 0000:82:00.0: Multicast VEP steering support [65188.980588] mlx4_core 0000:82:00.0: Counters support [65188.980589] mlx4_core 0000:82:00.0: Port management change event support [65188.980591] mlx4_core 0000:82:00.0: 64 byte EQE support [65188.980592] mlx4_core 0000:82:00.0: 64 byte CQE support [65188.980593] mlx4_core 0000:82:00.0: RSS support [65188.980595] mlx4_core 0000:82:00.0: RSS Toeplitz Hash Function support [65188.980596] mlx4_core 0000:82:00.0: RSS XOR Hash Function support [65188.980597] mlx4_core 0000:82:00.0: Device manage flow steering support [65188.980599] mlx4_core 0000:82:00.0: 64B EQEs/CQEs supported by the device but not enabled [65188.980602] mlx4_core 0000:82:00.0: Steering mode is: B0 steering, oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10 [65188.980609] mlx4_core 0000:82:00.0: profile[ 0] ( CMPT): 2^26 entries @ 0x 0, size 0x 100000000 [65188.980611] mlx4_core 0000:82:00.0: profile[ 1] (RDMARC): 2^22 entries @ 0x 100000000, size 0x 8000000 [65188.980613] mlx4_core 0000:82:00.0: profile[ 2] ( MTT): 2^24 entries @ 0x 108000000, size 0x 8000000 [65188.980615] mlx4_core 0000:82:00.0: profile[ 3] ( QP): 2^18 entries @ 0x 110000000, size 0x 4000000 [65188.980616] mlx4_core 0000:82:00.0: profile[ 4] ( DMPT): 2^19 entries @ 0x 114000000, size 0x 2000000 [65188.980618] mlx4_core 0000:82:00.0: profile[ 5] ( ALTC): 2^18 entries @ 0x 116000000, size 0x 1000000 [65188.980620] mlx4_core 0000:82:00.0: profile[ 6] ( SRQ): 2^16 entries @ 0x 117000000, size 0x 800000 [65188.980622] mlx4_core 0000:82:00.0: profile[ 7] ( CQ): 2^16 entries @ 0x 117800000, size 0x 800000 [65188.980624] mlx4_core 0000:82:00.0: profile[ 8] ( MCG): 2^13 entries @ 0x 118000000, size 0x 800000 [65188.980625] mlx4_core 0000:82:00.0: profile[ 9] ( AUXC): 2^18 entries @ 0x 118800000, size 0x 40000 [65188.980627] mlx4_core 0000:82:00.0: profile[10] ( EQ): 2^10 entries @ 0x 118840000, size 0x 20000 [65188.980629] mlx4_core 0000:82:00.0: HCA context memory: reserving 4596096 KB [65188.980645] mlx4_core 0000:82:00.0: 4596096 KB of HCA context requires 9068 KB aux memory. [65189.017150] mlx4_core 0000:82:00.0: Mapped 39 chunks/9068 KB for ICM aux. [65189.018533] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 0 for ICM. [65189.019895] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 40000000 for ICM. [65189.021266] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 80000000 for ICM. [65189.021621] mlx4_core 0000:82:00.0: Mapped 1 chunks/64 KB at c0000000 for ICM. [65189.022303] mlx4_core 0000:82:00.0: Mapped 1 chunks/128 KB at 118840000 for ICM. [65189.023671] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 108000000 for ICM. [65189.025060] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 114000000 for ICM. [65189.026419] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 110000000 for ICM. [65189.027778] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118800000 for ICM. [65189.029123] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 116000000 for ICM. [65189.030477] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 100000000 for ICM. [65189.031834] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 117800000 for ICM. [65189.033197] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 117000000 for ICM. [65189.034559] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118000000 for ICM. [65189.035906] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118040000 for ICM. [65189.037254] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118080000 for ICM. [65189.038633] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1180c0000 for ICM. [65189.039991] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118100000 for ICM. [65189.041349] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118140000 for ICM. [65189.042700] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118180000 for ICM. [65189.044050] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1181c0000 for ICM. [65189.045409] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118200000 for ICM. [65189.046758] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118240000 for ICM. [65189.048098] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118280000 for ICM. [65189.049466] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1182c0000 for ICM. [65189.050813] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118300000 for ICM. [65189.052169] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118340000 for ICM. [65189.053521] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118380000 for ICM. [65189.054862] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1183c0000 for ICM. [65189.056218] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118400000 for ICM. [65189.057563] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118440000 for ICM. [65189.058913] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118480000 for ICM. [65189.060272] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1184c0000 for ICM. [65189.061616] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118500000 for ICM. [65189.062971] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118540000 for ICM. [65189.064314] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118580000 for ICM. [65189.065669] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1185c0000 for ICM. [65189.067011] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118600000 for ICM. [65189.068364] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118640000 for ICM. [65189.069728] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118680000 for ICM. [65189.071086] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1186c0000 for ICM. [65189.072438] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118700000 for ICM. [65189.073788] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118740000 for ICM. [65189.075145] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118780000 for ICM. [65189.076503] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1187c0000 for ICM. [65189.643406] mlx4_core 0000:82:00.0: Started init_resource_tracker: 80 slaves [65189.643891] mlx4_core 0000:82:00.0: irq 178 for MSI/MSI-X [65189.643899] mlx4_core 0000:82:00.0: irq 179 for MSI/MSI-X [65189.650122] mlx4_core 0000:82:00.0: NOP command IRQ test passed [65189.650652] mlx4_core: Initializing 0000:05:00.1 [65189.650667] mlx4_core 0000:05:00.1: enabling device (0000 -> 0002) [65189.650709] mlx4_core 0000:05:00.1: Detected virtual function - running in slave mode [65189.650792] mlx4_core 0000:05:00.1: Sending reset [65189.650847] mlx4_core 0000:05:00.0: Received reset from slave:1 [65189.650859] mlx4_core 0000:05:00.1: Sending vhcr0 [65189.651166] mlx4_core 0000:05:00.1: BlueFlame not available [65189.651412] mlx4_core 0000:05:00.1: Base MM extensions: flags 00000cc0, rsvd L_Key 00000500 [65189.651414] mlx4_core 0000:05:00.1: Max ICM size 4294967296 MB [65189.651417] mlx4_core 0000:05:00.1: Max QPs: 16777216, reserved QPs: 64, entry size: 256 [65189.651419] mlx4_core 0000:05:00.1: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 [65189.651421] mlx4_core 0000:05:00.1: Max CQs: 16777216, reserved CQs: 128, entry size: 128 [65189.651422] mlx4_core 0000:05:00.1: Max EQs: 32, reserved EQs: 28, entry size: 128 [65189.651424] mlx4_core 0000:05:00.1: reserved MPTs: 16, reserved MTTs: 16 [65189.651425] mlx4_core 0000:05:00.1: Max PDs: 131072, reserved PDs: 4, reserved UARs: 7 [65189.651427] mlx4_core 0000:05:00.1: Max QP/MCG: 131072, reserved MGMs: 0 [65189.651428] mlx4_core 0000:05:00.1: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 [65189.651430] mlx4_core 0000:05:00.1: Local CA ACK delay: 15, max MTU: 4096, port width cap: 3 [65189.651431] mlx4_core 0000:05:00.1: Max SQ desc size: 1008, max SQ S/G: 62 [65189.651433] mlx4_core 0000:05:00.1: Max RQ desc size: 512, max RQ S/G: 32 [65189.651434] mlx4_core 0000:05:00.1: Max GSO size: 131072 [65189.651436] mlx4_core 0000:05:00.1: Max counters: 256 [65189.651437] mlx4_core 0000:05:00.1: Max RSS Table size: 256 [65189.651438] mlx4_core 0000:05:00.1: DEV_CAP flags: [65189.651440] mlx4_core 0000:05:00.1: RC transport [65189.651441] mlx4_core 0000:05:00.1: UC transport [65189.651442] mlx4_core 0000:05:00.1: UD transport [65189.651443] mlx4_core 0000:05:00.1: XRC transport [65189.651444] mlx4_core 0000:05:00.1: FCoIB support [65189.651446] mlx4_core 0000:05:00.1: SRQ support [65189.651447] mlx4_core 0000:05:00.1: IPoIB checksum offload [65189.651448] mlx4_core 0000:05:00.1: P_Key violation counter [65189.651449] mlx4_core 0000:05:00.1: Q_Key violation counter [65189.651451] mlx4_core 0000:05:00.1: DPDP [65189.651452] mlx4_core 0000:05:00.1: Big LSO headers [65189.651453] mlx4_core 0000:05:00.1: APM support [65189.651454] mlx4_core 0000:05:00.1: Atomic ops support [65189.651456] mlx4_core 0000:05:00.1: Address vector port checking support [65189.651457] mlx4_core 0000:05:00.1: UD multicast support [65189.651458] mlx4_core 0000:05:00.1: Router support [65189.651459] mlx4_core 0000:05:00.1: IBoE support [65189.651461] mlx4_core 0000:05:00.1: Unicast loopback support [65189.651462] mlx4_core 0000:05:00.1: UDP RSS support [65189.651463] mlx4_core 0000:05:00.1: Unicast VEP steering support [65189.651465] mlx4_core 0000:05:00.1: Multicast VEP steering support [65189.651466] mlx4_core 0000:05:00.1: Counters support [65189.651467] mlx4_core 0000:05:00.1: Port management change event support [65189.651469] mlx4_core 0000:05:00.1: RSS support [65189.651470] mlx4_core 0000:05:00.1: RSS Toeplitz Hash Function support [65189.651471] mlx4_core 0000:05:00.1: RSS XOR Hash Function support [65189.651567] mlx4_core 0000:05:00.1: HCA minimum page size:512 [65189.652182] mlx4_core 0000:05:00.1: Steering mode is: B0 steering [65189.652184] mlx4_core 0000:05:00.1: Failed to map blue flame area [65189.652438] mlx4_core 0000:05:00.1: irq 180 for MSI/MSI-X [65189.652446] mlx4_core 0000:05:00.1: irq 181 for MSI/MSI-X [65189.657770] mlx4_core 0000:05:00.1: NOP command IRQ test passed [65189.657791] mlx4_core: Initializing 0000:05:00.2 [65189.657801] mlx4_core 0000:05:00.2: enabling device (0000 -> 0002) [65189.657836] mlx4_core 0000:05:00.2: Skipping virtual function:2 [65189.657856] mlx4_core: Initializing 0000:05:00.3 [65189.657865] mlx4_core 0000:05:00.3: enabling device (0000 -> 0002) [65189.657887] mlx4_core 0000:05:00.3: Skipping virtual function:3 [65189.657904] mlx4_core: Initializing 0000:82:00.1 [65189.657914] mlx4_core 0000:82:00.1: enabling device (0000 -> 0002) [65189.657940] mlx4_core 0000:82:00.1: Detected virtual function - running in slave mode [65189.658047] mlx4_core 0000:82:00.1: Sending reset [65189.658128] mlx4_core 0000:82:00.0: Received reset from slave:1 [65189.658137] mlx4_core 0000:82:00.1: Sending vhcr0 [65189.658463] mlx4_core 0000:82:00.1: BlueFlame not available [65189.658646] mlx4_core 0000:82:00.1: Base MM extensions: flags 00104cc0, rsvd L_Key 00008000 [65189.658650] mlx4_core 0000:82:00.1: Max ICM size 4294967296 MB [65189.658653] mlx4_core 0000:82:00.1: Max QPs: 16777216, reserved QPs: 64, entry size: 256 [65189.658655] mlx4_core 0000:82:00.1: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 [65189.658657] mlx4_core 0000:82:00.1: Max CQs: 16777216, reserved CQs: 128, entry size: 128 [65189.658658] mlx4_core 0000:82:00.1: Max EQs: 32, reserved EQs: 28, entry size: 128 [65189.658660] mlx4_core 0000:82:00.1: reserved MPTs: 256, reserved MTTs: 32 [65189.658661] mlx4_core 0000:82:00.1: Max PDs: 131072, reserved PDs: 4, reserved UARs: 7 [65189.658663] mlx4_core 0000:82:00.1: Max QP/MCG: 131072, reserved MGMs: 0 [65189.658664] mlx4_core 0000:82:00.1: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 [65189.658666] mlx4_core 0000:82:00.1: Local CA ACK delay: 15, max MTU: 4096, port width cap: 2 [65189.658667] mlx4_core 0000:82:00.1: Max SQ desc size: 1008, max SQ S/G: 62 [65189.658669] mlx4_core 0000:82:00.1: Max RQ desc size: 512, max RQ S/G: 32 [65189.658670] mlx4_core 0000:82:00.1: Max GSO size: 131072 [65189.658671] mlx4_core 0000:82:00.1: Max counters: 256 [65189.658673] mlx4_core 0000:82:00.1: Max RSS Table size: 256 [65189.658674] mlx4_core 0000:82:00.1: DEV_CAP flags: [65189.658675] mlx4_core 0000:82:00.1: RC transport [65189.658676] mlx4_core 0000:82:00.1: UC transport [65189.658678] mlx4_core 0000:82:00.1: UD transport [65189.658679] mlx4_core 0000:82:00.1: XRC transport [65189.658680] mlx4_core 0000:82:00.1: FCoIB support [65189.658681] mlx4_core 0000:82:00.1: SRQ support [65189.658682] mlx4_core 0000:82:00.1: IPoIB checksum offload [65189.658684] mlx4_core 0000:82:00.1: P_Key violation counter [65189.658685] mlx4_core 0000:82:00.1: Q_Key violation counter [65189.658688] mlx4_core 0000:82:00.1: DPDP [65189.658691] mlx4_core 0000:82:00.1: Big LSO headers [65189.658695] mlx4_core 0000:82:00.1: APM support [65189.658699] mlx4_core 0000:82:00.1: Atomic ops support [65189.658702] mlx4_core 0000:82:00.1: Address vector port checking support [65189.658705] mlx4_core 0000:82:00.1: UD multicast support [65189.658707] mlx4_core 0000:82:00.1: Router support [65189.658708] mlx4_core 0000:82:00.1: IBoE support [65189.658709] mlx4_core 0000:82:00.1: Unicast loopback support [65189.658710] mlx4_core 0000:82:00.1: FCS header control [65189.658712] mlx4_core 0000:82:00.1: UDP RSS support [65189.658713] mlx4_core 0000:82:00.1: Unicast VEP steering support [65189.658714] mlx4_core 0000:82:00.1: Multicast VEP steering support [65189.658715] mlx4_core 0000:82:00.1: Counters support [65189.658717] mlx4_core 0000:82:00.1: Port management change event support [65189.658718] mlx4_core 0000:82:00.1: 64 byte EQE support [65189.658719] mlx4_core 0000:82:00.1: 64 byte CQE support [65189.658721] mlx4_core 0000:82:00.1: RSS support [65189.658722] mlx4_core 0000:82:00.1: RSS Toeplitz Hash Function support [65189.658723] mlx4_core 0000:82:00.1: RSS XOR Hash Function support [65189.658725] mlx4_core 0000:82:00.1: Device manage flow steering support [65189.658727] mlx4_core 0000:82:00.1: 64B EQEs/CQEs supported by the device but not enabled [65189.658863] mlx4_core 0000:82:00.1: HCA minimum page size:512 [65189.659129] mlx4_core 0000:82:00.1: Steering mode is: B0 steering [65189.659130] mlx4_core 0000:82:00.1: Failed to map blue flame area [65189.659417] mlx4_core 0000:82:00.1: irq 182 for MSI/MSI-X [65189.659428] mlx4_core 0000:82:00.1: irq 183 for MSI/MSI-X [65189.666007] mlx4_core 0000:82:00.1: NOP command IRQ test passed [65189.666033] mlx4_core: Initializing 0000:82:00.2 [65189.666045] mlx4_core 0000:82:00.2: enabling device (0000 -> 0002) [65189.666083] mlx4_core 0000:82:00.2: Skipping virtual function:2 [65189.666106] mlx4_core: Initializing 0000:82:00.3 [65189.666116] mlx4_core 0000:82:00.3: enabling device (0000 -> 0002) [65189.666142] mlx4_core 0000:82:00.3: Skipping virtual function:3 [65189.690937] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008) [65189.779306] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 110040000 for ICM. [65189.780627] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 100080000 for ICM. [65189.786286] mlx4_core 0000:05:00.0: mlx4_ib: multi-function enabled [65189.788526] mlx4_core 0000:05:00.0: mlx4_ib: initializing demux service for 80 qp1 clients [65189.792146] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 100040000 for ICM. [65189.793859] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [65189.803736] IPv6: ADDRCONF(NETDEV_UP): ib1: link is not ready [65189.862554] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 110040000 for ICM. [65189.863888] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 100080000 for ICM. [65189.864185] mlx4_core 0000:82:00.0: mlx4_ib: multi-function enabled [65189.865217] mlx4_core 0000:82:00.0: mlx4_ib: initializing demux service for 80 qp1 clients [65189.868388] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 100040000 for ICM. [65189.875240] IPv6: ADDRCONF(NETDEV_UP): ib2: link is not ready [65189.991130] mlx4_core 0000:05:00.1: mlx4_ib: multi-function enabled [65189.991132] mlx4_core 0000:05:00.1: mlx4_ib: operating in qp1 tunnel mode [65189.994633] IPv6: ADDRCONF(NETDEV_UP): ib3: link is not ready [65190.005505] IPv6: ADDRCONF(NETDEV_UP): ib4: link is not ready [65190.061269] mlx4_core 0000:82:00.1: mlx4_ib: multi-function enabled [65190.061271] mlx4_core 0000:82:00.1: mlx4_ib: operating in qp1 tunnel mode [65190.071641] IPv6: ADDRCONF(NETDEV_UP): ib5: link is not ready Here is dmesg command return of virtual machine (mlx4_core loaded with options debug_level=1) : [root@welling0 ~]# dmesg mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) mlx4_core: Initializing 0000:00:08.0 mlx4_core 0000:00:08.0: Detected virtual function - running in slave mode mlx4_core 0000:00:08.0: Sending reset mlx4_core 0000:00:08.0: Sending vhcr0 mlx4_core 0000:00:08.0: BlueFlame not available mlx4_core 0000:00:08.0: Base MM extensions: flags 00104cc0, rsvd L_Key 00008000 mlx4_core 0000:00:08.0: Max ICM size 4294967296 MB mlx4_core 0000:00:08.0: Max QPs: 16777216, reserved QPs: 64, entry size: 256 mlx4_core 0000:00:08.0: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 mlx4_core 0000:00:08.0: Max CQs: 16777216, reserved CQs: 128, entry size: 128 mlx4_core 0000:00:08.0: Max EQs: 32, reserved EQs: 28, entry size: 128 mlx4_core 0000:00:08.0: reserved MPTs: 256, reserved MTTs: 32 mlx4_core 0000:00:08.0: Max PDs: 131072, reserved PDs: 4, reserved UARs: 7 mlx4_core 0000:00:08.0: Max QP/MCG: 131072, reserved MGMs: 0 mlx4_core 0000:00:08.0: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 mlx4_core 0000:00:08.0: Local CA ACK delay: 15, max MTU: 4096, port width cap: 2 mlx4_core 0000:00:08.0: Max SQ desc size: 1008, max SQ S/G: 62 mlx4_core 0000:00:08.0: Max RQ desc size: 512, max RQ S/G: 32 mlx4_core 0000:00:08.0: Max GSO size: 131072 mlx4_core 0000:00:08.0: Max counters: 256 mlx4_core 0000:00:08.0: Max RSS Table size: 256 mlx4_core 0000:00:08.0: DEV_CAP flags: mlx4_core 0000:00:08.0: RC transport mlx4_core 0000:00:08.0: UC transport mlx4_core 0000:00:08.0: UD transport mlx4_core 0000:00:08.0: XRC transport mlx4_core 0000:00:08.0: FCoIB support mlx4_core 0000:00:08.0: SRQ support mlx4_core 0000:00:08.0: IPoIB checksum offload mlx4_core 0000:00:08.0: P_Key violation counter mlx4_core 0000:00:08.0: Q_Key violation counter mlx4_core 0000:00:08.0: DPDP mlx4_core 0000:00:08.0: Big LSO headers mlx4_core 0000:00:08.0: APM support mlx4_core 0000:00:08.0: Atomic ops support mlx4_core 0000:00:08.0: Address vector port checking support mlx4_core 0000:00:08.0: UD multicast support mlx4_core 0000:00:08.0: Router support mlx4_core 0000:00:08.0: IBoE support mlx4_core 0000:00:08.0: Unicast loopback support mlx4_core 0000:00:08.0: FCS header control mlx4_core 0000:00:08.0: UDP RSS support mlx4_core 0000:00:08.0: Unicast VEP steering support mlx4_core 0000:00:08.0: Multicast VEP steering support mlx4_core 0000:00:08.0: Counters support mlx4_core 0000:00:08.0: Port management change event support mlx4_core 0000:00:08.0: 64 byte EQE support mlx4_core 0000:00:08.0: 64 byte CQE support mlx4_core 0000:00:08.0: RSS support mlx4_core 0000:00:08.0: RSS Toeplitz Hash Function support mlx4_core 0000:00:08.0: RSS XOR Hash Function support mlx4_core 0000:00:08.0: Device manage flow steering support mlx4_core 0000:00:08.0: 64B EQEs/CQEs supported by the device but not enabled mlx4_core 0000:00:08.0: HCA minimum page size:512 mlx4_core 0000:00:08.0: Steering mode is: B0 steering mlx4_core 0000:00:08.0: Failed to map blue flame area mlx4_core 0000:00:08.0: irq 48 for MSI/MSI-X mlx4_core 0000:00:08.0: irq 49 for MSI/MSI-X mlx4_core 0000:00:08.0: failed execution of VHCR_POST commandopcode 0x31 mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 49). mlx4_core 0000:00:08.0: Trying again without MSI-X. mlx4_core: probe of 0000:00:08.0 failed with error -16 Mathis 2013/1/24 Mathis GAVILLON <jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>: > ---------- Forwarded message ---------- > From: Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> > Date: 2013/1/24 > Subject: Re: Error message when trying to use Infiniband virtual > functions in virtual machine > To: Mathis GAVILLON <jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> > Cc : linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> > > > On Wed, Jan 23, 2013 at 5:29 PM, Mathis GAVILLON <jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > >> I'm working on a project in which I try to virtualize Infiniband. I use ConnectX3 Infiniband cards with FW 2.9.1200. >> I've installed a Fedora 18 OS with 3.8.0rc2 recompiled kernel. I use KVM 1.2 to run virtual machines. > > Your firmware version is a bit old... if you can upgrade to the latest > GA on the Mellanox site (2.11.0500) would be good, > > So what version of qemu is installed on the distro you use? Are you > able to work over a ConnectX VF if probed > to the host? e.g use num_vfs=3 probe_vf=1 and see how the related mlx4 > device (e.g mlx4_1) is functioning. > > Also would be of help if you probe mlx4_core on both VM and host with > debug_level=1 > > > Or. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <CAOoYFkFVvK+1YNJ_N6r0N_3ZMF6P9Q+0dbsDg4v8=3JfDPVOOA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <CAOoYFkFVvK+1YNJ_N6r0N_3ZMF6P9Q+0dbsDg4v8=3JfDPVOOA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-01-28 16:35 ` Jack Morgenstein [not found] ` <201301281835.18533.jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 0 siblings, 1 reply; 16+ messages in thread From: Jack Morgenstein @ 2013-01-28 16:35 UTC (permalink / raw) To: Mathis GAVILLON; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz Which VF did you map to the guest O/S? In your configuration, both HCAs will have the PF and the first VF running on the hypervisor, and the remaining VFs available for binding to guests. You have: mlx4_core 0000:05:00.0: Enabling SR-IOV with 3 VFs <== PF pci 0000:05:00.1: [15b3:1002] type 00 class 0x0c0600 <== VFs pci 0000:05:00.2: [15b3:1002] type 00 class 0x0c0600 pci 0000:05:00.3: [15b3:1002] type 00 class 0x0c0600 05:00.0 and 05:00.1 run on the hypervisor, and 05:00.2/3 are available for binding to guests. Similarly, mlx4_core 0000:82:00.0: Enabling SR-IOV with 3 VFs pci 0000:82:00.1: [15b3:1004] type 00 class 0x028000 pci 0000:82:00.2: [15b3:1004] type 00 class 0x028000 pci 0000:82:00.3: [15b3:1004] type 00 class 0x028000 82:00.0 and 82:00.1 run on the hypervisor, and 82:00.2/3 are available for binding to guests. The "skipping" messages you see for 2/3 indicate that these are not yet bound, and the hypervisor is not claiming them (this arises from setting probe_vf=1 --> only ONE guest is claimed/bound by the Hypervisor). I don't see any "received reset from xxx" lines in the dumps below, which would indicate that the hypervisor received a reset request from a guest O/S bound to one of the available VFs. This looks to me like a kernel configuration problem. You did not mention what CPU you are using. Did you do the following when building your kernel? make menuconfig (stuff below is shown for kernel 3.7. I don't know if it changed in 3.8) Then, under "Device Drivers", put an '*' by "IOMMU hardware support" and go into that submenu. Make sure to check ALL the boxes which apply for your CPU in the IOMMU submenu. For Intel, at least, you should also add "intel_iommu=on" to the line for your kernel in file /boot/grub/menu.lst -Jack On Monday 28 January 2013 10:29, Mathis GAVILLON wrote: > The version of firmware I gave you was wrong (I've two cards > (ConnectX2 and ConnectX3, I only use ConnectX 3) in my computer and I > took the bad one). The firmware installed on my ConnectX3 Infiniband > Card is 2.11.500. > KVM is 1.2.2-1 version. > I tried to load the driver with options you gave me but the error > message is the same. Nothing else happen. > > Here is the dmesg command return from host (mlx4_core loaded with > options debug_level=1, num_vfs=3, probe_vf=1) : > > [root@barb0 ~]# dmesg > [65182.838953] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) > [65182.838955] mlx4_core: Initializing 0000:05:00.0 > [65182.839144] mlx4_core 0000:05:00.0: Enabling SR-IOV with 3 VFs > [65182.940316] pci 0000:05:00.1: [15b3:1002] type 00 class 0x0c0600 > [65182.941105] pci 0000:05:00.2: [15b3:1002] type 00 class 0x0c0600 > [65182.941876] pci 0000:05:00.3: [15b3:1002] type 00 class 0x0c0600 > [65182.942585] mlx4_core 0000:05:00.0: Running in master mode > [65183.945306] mlx4_core 0000:05:00.0: FW version 2.9.1200 (cmd intf > rev 3), max commands 16 > [65183.945309] mlx4_core 0000:05:00.0: Catastrophic error buffer at > 0x1f020, size 0x10, BAR 0 > [65183.945311] mlx4_core 0000:05:00.0: Communication vector bar:2 offset:0x800 > [65183.945312] mlx4_core 0000:05:00.0: FW size 385 KB > [65183.945313] mlx4_core 0000:05:00.0: Clear int @ f0058, BAR 0 > [65183.947177] mlx4_core 0000:05:00.0: Mapped 26 chunks/6168 KB for FW. > [65184.614576] mlx4_core 0000:05:00.0: BlueFlame available (reg size > 512, regs/page 8) > [65184.614742] mlx4_core 0000:05:00.0: Base MM extensions: flags > 00000cc0, rsvd L_Key 00000500 > [65184.614743] mlx4_core 0000:05:00.0: Max ICM size 4294967296 MB > [65184.614745] mlx4_core 0000:05:00.0: Max QPs: 16777216, reserved > QPs: 64, entry size: 256 > [65184.614747] mlx4_core 0000:05:00.0: Max SRQs: 16777216, reserved > SRQs: 64, entry size: 128 > [65184.614748] mlx4_core 0000:05:00.0: Max CQs: 16777216, reserved > CQs: 128, entry size: 128 > [65184.614750] mlx4_core 0000:05:00.0: Max EQs: 32, reserved EQs: 28, > entry size: 128 > [65184.614751] mlx4_core 0000:05:00.0: reserved MPTs: 16, reserved MTTs: 16 > [65184.614753] mlx4_core 0000:05:00.0: Max PDs: 131072, reserved PDs: > 4, reserved UARs: 7 > [65184.614754] mlx4_core 0000:05:00.0: Max QP/MCG: 131072, reserved MGMs: 0 > [65184.614756] mlx4_core 0000:05:00.0: Max CQEs: 4194304, max WQEs: > 16384, max SRQ WQEs: 16384 > [65184.614758] mlx4_core 0000:05:00.0: Local CA ACK delay: 15, max > MTU: 4096, port width cap: 3 > [65184.614759] mlx4_core 0000:05:00.0: Max SQ desc size: 1008, max SQ S/G: 62 > [65184.614760] mlx4_core 0000:05:00.0: Max RQ desc size: 512, max RQ S/G: 32 > [65184.614762] mlx4_core 0000:05:00.0: Max GSO size: 131072 > [65184.614763] mlx4_core 0000:05:00.0: Max counters: 256 > [65184.614764] mlx4_core 0000:05:00.0: Max RSS Table size: 256 > [65184.614766] mlx4_core 0000:05:00.0: DEV_CAP flags: > [65184.614767] mlx4_core 0000:05:00.0: RC transport > [65184.614768] mlx4_core 0000:05:00.0: UC transport > [65184.614769] mlx4_core 0000:05:00.0: UD transport > [65184.614771] mlx4_core 0000:05:00.0: XRC transport > [65184.614772] mlx4_core 0000:05:00.0: FCoIB support > [65184.614773] mlx4_core 0000:05:00.0: SRQ support > [65184.614774] mlx4_core 0000:05:00.0: IPoIB checksum offload > [65184.614775] mlx4_core 0000:05:00.0: P_Key violation counter > [65184.614777] mlx4_core 0000:05:00.0: Q_Key violation counter > [65184.614778] mlx4_core 0000:05:00.0: DPDP > [65184.614779] mlx4_core 0000:05:00.0: Big LSO headers > [65184.614780] mlx4_core 0000:05:00.0: APM support > [65184.614782] mlx4_core 0000:05:00.0: Atomic ops support > [65184.614783] mlx4_core 0000:05:00.0: Address vector port checking support > [65184.614784] mlx4_core 0000:05:00.0: UD multicast support > [65184.614785] mlx4_core 0000:05:00.0: Router support > [65184.614787] mlx4_core 0000:05:00.0: IBoE support > [65184.614788] mlx4_core 0000:05:00.0: Unicast loopback support > [65184.614789] mlx4_core 0000:05:00.0: UDP RSS support > [65184.614790] mlx4_core 0000:05:00.0: Unicast VEP steering support > [65184.614792] mlx4_core 0000:05:00.0: Multicast VEP steering support > [65184.614793] mlx4_core 0000:05:00.0: Counters support > [65184.614795] mlx4_core 0000:05:00.0: RSS support > [65184.614796] mlx4_core 0000:05:00.0: RSS Toeplitz Hash Function support > [65184.614797] mlx4_core 0000:05:00.0: RSS XOR Hash Function support > [65184.614800] mlx4_core 0000:05:00.0: Steering mode is: B0 steering, > oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10 > [65184.614807] mlx4_core 0000:05:00.0: profile[ 0] ( CMPT): 2^26 > entries @ 0x 0, size 0x 100000000 > [65184.614809] mlx4_core 0000:05:00.0: profile[ 1] (RDMARC): 2^22 > entries @ 0x 100000000, size 0x 8000000 > [65184.614811] mlx4_core 0000:05:00.0: profile[ 2] ( MTT): 2^24 > entries @ 0x 108000000, size 0x 8000000 > [65184.614813] mlx4_core 0000:05:00.0: profile[ 3] ( QP): 2^18 > entries @ 0x 110000000, size 0x 4000000 > [65184.614815] mlx4_core 0000:05:00.0: profile[ 4] ( DMPT): 2^19 > entries @ 0x 114000000, size 0x 2000000 > [65184.614816] mlx4_core 0000:05:00.0: profile[ 5] ( ALTC): 2^18 > entries @ 0x 116000000, size 0x 1000000 > [65184.614818] mlx4_core 0000:05:00.0: profile[ 6] ( SRQ): 2^16 > entries @ 0x 117000000, size 0x 800000 > [65184.614820] mlx4_core 0000:05:00.0: profile[ 7] ( CQ): 2^16 > entries @ 0x 117800000, size 0x 800000 > [65184.614822] mlx4_core 0000:05:00.0: profile[ 8] ( MCG): 2^13 > entries @ 0x 118000000, size 0x 800000 > [65184.614824] mlx4_core 0000:05:00.0: profile[ 9] ( AUXC): 2^18 > entries @ 0x 118800000, size 0x 40000 > [65184.614826] mlx4_core 0000:05:00.0: profile[10] ( EQ): 2^10 > entries @ 0x 118840000, size 0x 20000 > [65184.614827] mlx4_core 0000:05:00.0: HCA context memory: reserving 4596096 KB > [65184.614844] mlx4_core 0000:05:00.0: 4596096 KB of HCA context > requires 9068 KB aux memory. > [65184.644227] mlx4_core 0000:05:00.0: Mapped 39 chunks/9068 KB for ICM aux. > [65184.645547] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 0 for ICM. > [65184.646846] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 40000000 for ICM. > [65184.648146] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 80000000 for ICM. > [65184.648483] mlx4_core 0000:05:00.0: Mapped 1 chunks/64 KB at > c0000000 for ICM. > [65184.649133] mlx4_core 0000:05:00.0: Mapped 1 chunks/128 KB at > 118840000 for ICM. > [65184.650413] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 108000000 for ICM. > [65184.651737] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 114000000 for ICM. > [65184.653016] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 110000000 for ICM. > [65184.654304] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118800000 for ICM. > [65184.655585] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 116000000 for ICM. > [65184.656866] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 100000000 for ICM. > [65184.658146] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 117800000 for ICM. > [65184.659426] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 117000000 for ICM. > [65184.660705] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118000000 for ICM. > [65184.661986] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118040000 for ICM. > [65184.663265] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118080000 for ICM. > [65184.664550] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1180c0000 for ICM. > [65184.665839] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118100000 for ICM. > [65184.667117] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118140000 for ICM. > [65184.668395] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118180000 for ICM. > [65184.669675] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1181c0000 for ICM. > [65184.670955] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118200000 for ICM. > [65184.672234] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118240000 for ICM. > [65184.673511] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118280000 for ICM. > [65184.674794] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1182c0000 for ICM. > [65184.676070] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118300000 for ICM. > [65184.677351] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118340000 for ICM. > [65184.678629] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118380000 for ICM. > [65184.679906] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1183c0000 for ICM. > [65184.681185] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118400000 for ICM. > [65184.682463] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118440000 for ICM. > [65184.683742] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118480000 for ICM. > [65184.685028] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1184c0000 for ICM. > [65184.686306] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118500000 for ICM. > [65184.687583] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118540000 for ICM. > [65184.688869] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118580000 for ICM. > [65184.690148] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1185c0000 for ICM. > [65184.691428] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118600000 for ICM. > [65184.692706] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118640000 for ICM. > [65184.693984] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118680000 for ICM. > [65184.695268] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1186c0000 for ICM. > [65184.696546] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118700000 for ICM. > [65184.697825] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118740000 for ICM. > [65184.699103] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118780000 for ICM. > [65184.700380] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1187c0000 for ICM. > [65185.202376] mlx4_core 0000:05:00.0: Started init_resource_tracker: 80 slaves > [65185.202819] mlx4_core 0000:05:00.0: irq 176 for MSI/MSI-X > [65185.202826] mlx4_core 0000:05:00.0: irq 177 for MSI/MSI-X > [65185.207838] mlx4_core 0000:05:00.0: NOP command IRQ test passed > [65185.208638] mlx4_core: Initializing 0000:82:00.0 > [65185.208742] mlx4_core 0000:82:00.0: Enabling SR-IOV with 3 VFs > [65185.309000] pci 0000:82:00.1: [15b3:1004] type 00 class 0x028000 > [65185.309781] pci 0000:82:00.2: [15b3:1004] type 00 class 0x028000 > [65185.310507] pci 0000:82:00.3: [15b3:1004] type 00 class 0x028000 > [65185.311176] mlx4_core 0000:82:00.0: Running in master mode > [65186.312479] mlx4_core 0000:82:00.0: FW version 2.11.500 (cmd intf > rev 3), max commands 16 > [65186.312482] mlx4_core 0000:82:00.0: Catastrophic error buffer at > 0x1f020, size 0x10, BAR 0 > [65186.312484] mlx4_core 0000:82:00.0: Communication vector bar:2 offset:0x800 > [65186.312485] mlx4_core 0000:82:00.0: FW size 385 KB > [65186.312486] mlx4_core 0000:82:00.0: Clear int @ f0058, BAR 0 > [65186.313785] mlx4_core 0000:82:00.0: Mapped 26 chunks/6168 KB for FW. > [65188.980410] mlx4_core 0000:82:00.0: BlueFlame available (reg size > 512, regs/page 8) > [65188.980535] mlx4_core 0000:82:00.0: Base MM extensions: flags > 00104cc0, rsvd L_Key 00008000 > [65188.980536] mlx4_core 0000:82:00.0: Max ICM size 4294967296 MB > [65188.980538] mlx4_core 0000:82:00.0: Max QPs: 16777216, reserved > QPs: 64, entry size: 256 > [65188.980540] mlx4_core 0000:82:00.0: Max SRQs: 16777216, reserved > SRQs: 64, entry size: 128 > [65188.980541] mlx4_core 0000:82:00.0: Max CQs: 16777216, reserved > CQs: 128, entry size: 128 > [65188.980543] mlx4_core 0000:82:00.0: Max EQs: 32, reserved EQs: 28, > entry size: 128 > [65188.980544] mlx4_core 0000:82:00.0: reserved MPTs: 256, reserved MTTs: 32 > [65188.980546] mlx4_core 0000:82:00.0: Max PDs: 131072, reserved PDs: > 4, reserved UARs: 7 > [65188.980548] mlx4_core 0000:82:00.0: Max QP/MCG: 131072, reserved MGMs: 0 > [65188.980549] mlx4_core 0000:82:00.0: Max CQEs: 4194304, max WQEs: > 16384, max SRQ WQEs: 16384 > [65188.980551] mlx4_core 0000:82:00.0: Local CA ACK delay: 15, max > MTU: 4096, port width cap: 2 > [65188.980552] mlx4_core 0000:82:00.0: Max SQ desc size: 1008, max SQ S/G: 62 > [65188.980554] mlx4_core 0000:82:00.0: Max RQ desc size: 512, max RQ S/G: 32 > [65188.980555] mlx4_core 0000:82:00.0: Max GSO size: 131072 > [65188.980556] mlx4_core 0000:82:00.0: Max counters: 256 > [65188.980558] mlx4_core 0000:82:00.0: Max RSS Table size: 256 > [65188.980559] mlx4_core 0000:82:00.0: DEV_CAP flags: > [65188.980560] mlx4_core 0000:82:00.0: RC transport > [65188.980561] mlx4_core 0000:82:00.0: UC transport > [65188.980563] mlx4_core 0000:82:00.0: UD transport > [65188.980564] mlx4_core 0000:82:00.0: XRC transport > [65188.980565] mlx4_core 0000:82:00.0: FCoIB support > [65188.980566] mlx4_core 0000:82:00.0: SRQ support > [65188.980568] mlx4_core 0000:82:00.0: IPoIB checksum offload > [65188.980569] mlx4_core 0000:82:00.0: P_Key violation counter > [65188.980570] mlx4_core 0000:82:00.0: Q_Key violation counter > [65188.980571] mlx4_core 0000:82:00.0: DPDP > [65188.980573] mlx4_core 0000:82:00.0: Big LSO headers > [65188.980574] mlx4_core 0000:82:00.0: APM support > [65188.980575] mlx4_core 0000:82:00.0: Atomic ops support > [65188.980576] mlx4_core 0000:82:00.0: Address vector port checking support > [65188.980578] mlx4_core 0000:82:00.0: UD multicast support > [65188.980579] mlx4_core 0000:82:00.0: Router support > [65188.980580] mlx4_core 0000:82:00.0: IBoE support > [65188.980582] mlx4_core 0000:82:00.0: Unicast loopback support > [65188.980583] mlx4_core 0000:82:00.0: FCS header control > [65188.980584] mlx4_core 0000:82:00.0: UDP RSS support > [65188.980586] mlx4_core 0000:82:00.0: Unicast VEP steering support > [65188.980587] mlx4_core 0000:82:00.0: Multicast VEP steering support > [65188.980588] mlx4_core 0000:82:00.0: Counters support > [65188.980589] mlx4_core 0000:82:00.0: Port management change event support > [65188.980591] mlx4_core 0000:82:00.0: 64 byte EQE support > [65188.980592] mlx4_core 0000:82:00.0: 64 byte CQE support > [65188.980593] mlx4_core 0000:82:00.0: RSS support > [65188.980595] mlx4_core 0000:82:00.0: RSS Toeplitz Hash Function support > [65188.980596] mlx4_core 0000:82:00.0: RSS XOR Hash Function support > [65188.980597] mlx4_core 0000:82:00.0: Device manage flow steering support > [65188.980599] mlx4_core 0000:82:00.0: 64B EQEs/CQEs supported by the > device but not enabled > [65188.980602] mlx4_core 0000:82:00.0: Steering mode is: B0 steering, > oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10 > [65188.980609] mlx4_core 0000:82:00.0: profile[ 0] ( CMPT): 2^26 > entries @ 0x 0, size 0x 100000000 > [65188.980611] mlx4_core 0000:82:00.0: profile[ 1] (RDMARC): 2^22 > entries @ 0x 100000000, size 0x 8000000 > [65188.980613] mlx4_core 0000:82:00.0: profile[ 2] ( MTT): 2^24 > entries @ 0x 108000000, size 0x 8000000 > [65188.980615] mlx4_core 0000:82:00.0: profile[ 3] ( QP): 2^18 > entries @ 0x 110000000, size 0x 4000000 > [65188.980616] mlx4_core 0000:82:00.0: profile[ 4] ( DMPT): 2^19 > entries @ 0x 114000000, size 0x 2000000 > [65188.980618] mlx4_core 0000:82:00.0: profile[ 5] ( ALTC): 2^18 > entries @ 0x 116000000, size 0x 1000000 > [65188.980620] mlx4_core 0000:82:00.0: profile[ 6] ( SRQ): 2^16 > entries @ 0x 117000000, size 0x 800000 > [65188.980622] mlx4_core 0000:82:00.0: profile[ 7] ( CQ): 2^16 > entries @ 0x 117800000, size 0x 800000 > [65188.980624] mlx4_core 0000:82:00.0: profile[ 8] ( MCG): 2^13 > entries @ 0x 118000000, size 0x 800000 > [65188.980625] mlx4_core 0000:82:00.0: profile[ 9] ( AUXC): 2^18 > entries @ 0x 118800000, size 0x 40000 > [65188.980627] mlx4_core 0000:82:00.0: profile[10] ( EQ): 2^10 > entries @ 0x 118840000, size 0x 20000 > [65188.980629] mlx4_core 0000:82:00.0: HCA context memory: reserving 4596096 KB > [65188.980645] mlx4_core 0000:82:00.0: 4596096 KB of HCA context > requires 9068 KB aux memory. > [65189.017150] mlx4_core 0000:82:00.0: Mapped 39 chunks/9068 KB for ICM aux. > [65189.018533] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 0 for ICM. > [65189.019895] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 40000000 for ICM. > [65189.021266] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 80000000 for ICM. > [65189.021621] mlx4_core 0000:82:00.0: Mapped 1 chunks/64 KB at > c0000000 for ICM. > [65189.022303] mlx4_core 0000:82:00.0: Mapped 1 chunks/128 KB at > 118840000 for ICM. > [65189.023671] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 108000000 for ICM. > [65189.025060] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 114000000 for ICM. > [65189.026419] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 110000000 for ICM. > [65189.027778] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118800000 for ICM. > [65189.029123] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 116000000 for ICM. > [65189.030477] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 100000000 for ICM. > [65189.031834] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 117800000 for ICM. > [65189.033197] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 117000000 for ICM. > [65189.034559] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118000000 for ICM. > [65189.035906] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118040000 for ICM. > [65189.037254] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118080000 for ICM. > [65189.038633] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1180c0000 for ICM. > [65189.039991] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118100000 for ICM. > [65189.041349] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118140000 for ICM. > [65189.042700] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118180000 for ICM. > [65189.044050] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1181c0000 for ICM. > [65189.045409] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118200000 for ICM. > [65189.046758] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118240000 for ICM. > [65189.048098] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118280000 for ICM. > [65189.049466] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1182c0000 for ICM. > [65189.050813] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118300000 for ICM. > [65189.052169] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118340000 for ICM. > [65189.053521] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118380000 for ICM. > [65189.054862] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1183c0000 for ICM. > [65189.056218] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118400000 for ICM. > [65189.057563] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118440000 for ICM. > [65189.058913] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118480000 for ICM. > [65189.060272] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1184c0000 for ICM. > [65189.061616] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118500000 for ICM. > [65189.062971] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118540000 for ICM. > [65189.064314] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118580000 for ICM. > [65189.065669] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1185c0000 for ICM. > [65189.067011] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118600000 for ICM. > [65189.068364] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118640000 for ICM. > [65189.069728] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118680000 for ICM. > [65189.071086] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1186c0000 for ICM. > [65189.072438] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118700000 for ICM. > [65189.073788] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118740000 for ICM. > [65189.075145] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118780000 for ICM. > [65189.076503] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1187c0000 for ICM. > [65189.643406] mlx4_core 0000:82:00.0: Started init_resource_tracker: 80 slaves > [65189.643891] mlx4_core 0000:82:00.0: irq 178 for MSI/MSI-X > [65189.643899] mlx4_core 0000:82:00.0: irq 179 for MSI/MSI-X > [65189.650122] mlx4_core 0000:82:00.0: NOP command IRQ test passed > [65189.650652] mlx4_core: Initializing 0000:05:00.1 > [65189.650667] mlx4_core 0000:05:00.1: enabling device (0000 -> 0002) > [65189.650709] mlx4_core 0000:05:00.1: Detected virtual function - > running in slave mode > [65189.650792] mlx4_core 0000:05:00.1: Sending reset > [65189.650847] mlx4_core 0000:05:00.0: Received reset from slave:1 > [65189.650859] mlx4_core 0000:05:00.1: Sending vhcr0 > [65189.651166] mlx4_core 0000:05:00.1: BlueFlame not available > [65189.651412] mlx4_core 0000:05:00.1: Base MM extensions: flags > 00000cc0, rsvd L_Key 00000500 > [65189.651414] mlx4_core 0000:05:00.1: Max ICM size 4294967296 MB > [65189.651417] mlx4_core 0000:05:00.1: Max QPs: 16777216, reserved > QPs: 64, entry size: 256 > [65189.651419] mlx4_core 0000:05:00.1: Max SRQs: 16777216, reserved > SRQs: 64, entry size: 128 > [65189.651421] mlx4_core 0000:05:00.1: Max CQs: 16777216, reserved > CQs: 128, entry size: 128 > [65189.651422] mlx4_core 0000:05:00.1: Max EQs: 32, reserved EQs: 28, > entry size: 128 > [65189.651424] mlx4_core 0000:05:00.1: reserved MPTs: 16, reserved MTTs: 16 > [65189.651425] mlx4_core 0000:05:00.1: Max PDs: 131072, reserved PDs: > 4, reserved UARs: 7 > [65189.651427] mlx4_core 0000:05:00.1: Max QP/MCG: 131072, reserved MGMs: 0 > [65189.651428] mlx4_core 0000:05:00.1: Max CQEs: 4194304, max WQEs: > 16384, max SRQ WQEs: 16384 > [65189.651430] mlx4_core 0000:05:00.1: Local CA ACK delay: 15, max > MTU: 4096, port width cap: 3 > [65189.651431] mlx4_core 0000:05:00.1: Max SQ desc size: 1008, max SQ S/G: 62 > [65189.651433] mlx4_core 0000:05:00.1: Max RQ desc size: 512, max RQ S/G: 32 > [65189.651434] mlx4_core 0000:05:00.1: Max GSO size: 131072 > [65189.651436] mlx4_core 0000:05:00.1: Max counters: 256 > [65189.651437] mlx4_core 0000:05:00.1: Max RSS Table size: 256 > [65189.651438] mlx4_core 0000:05:00.1: DEV_CAP flags: > [65189.651440] mlx4_core 0000:05:00.1: RC transport > [65189.651441] mlx4_core 0000:05:00.1: UC transport > [65189.651442] mlx4_core 0000:05:00.1: UD transport > [65189.651443] mlx4_core 0000:05:00.1: XRC transport > [65189.651444] mlx4_core 0000:05:00.1: FCoIB support > [65189.651446] mlx4_core 0000:05:00.1: SRQ support > [65189.651447] mlx4_core 0000:05:00.1: IPoIB checksum offload > [65189.651448] mlx4_core 0000:05:00.1: P_Key violation counter > [65189.651449] mlx4_core 0000:05:00.1: Q_Key violation counter > [65189.651451] mlx4_core 0000:05:00.1: DPDP > [65189.651452] mlx4_core 0000:05:00.1: Big LSO headers > [65189.651453] mlx4_core 0000:05:00.1: APM support > [65189.651454] mlx4_core 0000:05:00.1: Atomic ops support > [65189.651456] mlx4_core 0000:05:00.1: Address vector port checking support > [65189.651457] mlx4_core 0000:05:00.1: UD multicast support > [65189.651458] mlx4_core 0000:05:00.1: Router support > [65189.651459] mlx4_core 0000:05:00.1: IBoE support > [65189.651461] mlx4_core 0000:05:00.1: Unicast loopback support > [65189.651462] mlx4_core 0000:05:00.1: UDP RSS support > [65189.651463] mlx4_core 0000:05:00.1: Unicast VEP steering support > [65189.651465] mlx4_core 0000:05:00.1: Multicast VEP steering support > [65189.651466] mlx4_core 0000:05:00.1: Counters support > [65189.651467] mlx4_core 0000:05:00.1: Port management change event support > [65189.651469] mlx4_core 0000:05:00.1: RSS support > [65189.651470] mlx4_core 0000:05:00.1: RSS Toeplitz Hash Function support > [65189.651471] mlx4_core 0000:05:00.1: RSS XOR Hash Function support > [65189.651567] mlx4_core 0000:05:00.1: HCA minimum page size:512 > [65189.652182] mlx4_core 0000:05:00.1: Steering mode is: B0 steering > [65189.652184] mlx4_core 0000:05:00.1: Failed to map blue flame area > [65189.652438] mlx4_core 0000:05:00.1: irq 180 for MSI/MSI-X > [65189.652446] mlx4_core 0000:05:00.1: irq 181 for MSI/MSI-X > [65189.657770] mlx4_core 0000:05:00.1: NOP command IRQ test passed > [65189.657791] mlx4_core: Initializing 0000:05:00.2 > [65189.657801] mlx4_core 0000:05:00.2: enabling device (0000 -> 0002) > [65189.657836] mlx4_core 0000:05:00.2: Skipping virtual function:2 > [65189.657856] mlx4_core: Initializing 0000:05:00.3 > [65189.657865] mlx4_core 0000:05:00.3: enabling device (0000 -> 0002) > [65189.657887] mlx4_core 0000:05:00.3: Skipping virtual function:3 > [65189.657904] mlx4_core: Initializing 0000:82:00.1 > [65189.657914] mlx4_core 0000:82:00.1: enabling device (0000 -> 0002) > [65189.657940] mlx4_core 0000:82:00.1: Detected virtual function - > running in slave mode > [65189.658047] mlx4_core 0000:82:00.1: Sending reset > [65189.658128] mlx4_core 0000:82:00.0: Received reset from slave:1 > [65189.658137] mlx4_core 0000:82:00.1: Sending vhcr0 > [65189.658463] mlx4_core 0000:82:00.1: BlueFlame not available > [65189.658646] mlx4_core 0000:82:00.1: Base MM extensions: flags > 00104cc0, rsvd L_Key 00008000 > [65189.658650] mlx4_core 0000:82:00.1: Max ICM size 4294967296 MB > [65189.658653] mlx4_core 0000:82:00.1: Max QPs: 16777216, reserved > QPs: 64, entry size: 256 > [65189.658655] mlx4_core 0000:82:00.1: Max SRQs: 16777216, reserved > SRQs: 64, entry size: 128 > [65189.658657] mlx4_core 0000:82:00.1: Max CQs: 16777216, reserved > CQs: 128, entry size: 128 > [65189.658658] mlx4_core 0000:82:00.1: Max EQs: 32, reserved EQs: 28, > entry size: 128 > [65189.658660] mlx4_core 0000:82:00.1: reserved MPTs: 256, reserved MTTs: 32 > [65189.658661] mlx4_core 0000:82:00.1: Max PDs: 131072, reserved PDs: > 4, reserved UARs: 7 > [65189.658663] mlx4_core 0000:82:00.1: Max QP/MCG: 131072, reserved MGMs: 0 > [65189.658664] mlx4_core 0000:82:00.1: Max CQEs: 4194304, max WQEs: > 16384, max SRQ WQEs: 16384 > [65189.658666] mlx4_core 0000:82:00.1: Local CA ACK delay: 15, max > MTU: 4096, port width cap: 2 > [65189.658667] mlx4_core 0000:82:00.1: Max SQ desc size: 1008, max SQ S/G: 62 > [65189.658669] mlx4_core 0000:82:00.1: Max RQ desc size: 512, max RQ S/G: 32 > [65189.658670] mlx4_core 0000:82:00.1: Max GSO size: 131072 > [65189.658671] mlx4_core 0000:82:00.1: Max counters: 256 > [65189.658673] mlx4_core 0000:82:00.1: Max RSS Table size: 256 > [65189.658674] mlx4_core 0000:82:00.1: DEV_CAP flags: > [65189.658675] mlx4_core 0000:82:00.1: RC transport > [65189.658676] mlx4_core 0000:82:00.1: UC transport > [65189.658678] mlx4_core 0000:82:00.1: UD transport > [65189.658679] mlx4_core 0000:82:00.1: XRC transport > [65189.658680] mlx4_core 0000:82:00.1: FCoIB support > [65189.658681] mlx4_core 0000:82:00.1: SRQ support > [65189.658682] mlx4_core 0000:82:00.1: IPoIB checksum offload > [65189.658684] mlx4_core 0000:82:00.1: P_Key violation counter > [65189.658685] mlx4_core 0000:82:00.1: Q_Key violation counter > [65189.658688] mlx4_core 0000:82:00.1: DPDP > [65189.658691] mlx4_core 0000:82:00.1: Big LSO headers > [65189.658695] mlx4_core 0000:82:00.1: APM support > [65189.658699] mlx4_core 0000:82:00.1: Atomic ops support > [65189.658702] mlx4_core 0000:82:00.1: Address vector port checking support > [65189.658705] mlx4_core 0000:82:00.1: UD multicast support > [65189.658707] mlx4_core 0000:82:00.1: Router support > [65189.658708] mlx4_core 0000:82:00.1: IBoE support > [65189.658709] mlx4_core 0000:82:00.1: Unicast loopback support > [65189.658710] mlx4_core 0000:82:00.1: FCS header control > [65189.658712] mlx4_core 0000:82:00.1: UDP RSS support > [65189.658713] mlx4_core 0000:82:00.1: Unicast VEP steering support > [65189.658714] mlx4_core 0000:82:00.1: Multicast VEP steering support > [65189.658715] mlx4_core 0000:82:00.1: Counters support > [65189.658717] mlx4_core 0000:82:00.1: Port management change event support > [65189.658718] mlx4_core 0000:82:00.1: 64 byte EQE support > [65189.658719] mlx4_core 0000:82:00.1: 64 byte CQE support > [65189.658721] mlx4_core 0000:82:00.1: RSS support > [65189.658722] mlx4_core 0000:82:00.1: RSS Toeplitz Hash Function support > [65189.658723] mlx4_core 0000:82:00.1: RSS XOR Hash Function support > [65189.658725] mlx4_core 0000:82:00.1: Device manage flow steering support > [65189.658727] mlx4_core 0000:82:00.1: 64B EQEs/CQEs supported by the > device but not enabled > [65189.658863] mlx4_core 0000:82:00.1: HCA minimum page size:512 > [65189.659129] mlx4_core 0000:82:00.1: Steering mode is: B0 steering > [65189.659130] mlx4_core 0000:82:00.1: Failed to map blue flame area > [65189.659417] mlx4_core 0000:82:00.1: irq 182 for MSI/MSI-X > [65189.659428] mlx4_core 0000:82:00.1: irq 183 for MSI/MSI-X > [65189.666007] mlx4_core 0000:82:00.1: NOP command IRQ test passed > [65189.666033] mlx4_core: Initializing 0000:82:00.2 > [65189.666045] mlx4_core 0000:82:00.2: enabling device (0000 -> 0002) > [65189.666083] mlx4_core 0000:82:00.2: Skipping virtual function:2 > [65189.666106] mlx4_core: Initializing 0000:82:00.3 > [65189.666116] mlx4_core 0000:82:00.3: enabling device (0000 -> 0002) > [65189.666142] mlx4_core 0000:82:00.3: Skipping virtual function:3 > [65189.690937] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX > InfiniBand driver v1.0 (April 4, 2008) > [65189.779306] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 110040000 for ICM. > [65189.780627] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 100080000 for ICM. > [65189.786286] mlx4_core 0000:05:00.0: mlx4_ib: multi-function enabled > [65189.788526] mlx4_core 0000:05:00.0: mlx4_ib: initializing demux > service for 80 qp1 clients > [65189.792146] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 100040000 for ICM. > [65189.793859] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready > [65189.803736] IPv6: ADDRCONF(NETDEV_UP): ib1: link is not ready > [65189.862554] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 110040000 for ICM. > [65189.863888] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 100080000 for ICM. > [65189.864185] mlx4_core 0000:82:00.0: mlx4_ib: multi-function enabled > [65189.865217] mlx4_core 0000:82:00.0: mlx4_ib: initializing demux > service for 80 qp1 clients > [65189.868388] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 100040000 for ICM. > [65189.875240] IPv6: ADDRCONF(NETDEV_UP): ib2: link is not ready > [65189.991130] mlx4_core 0000:05:00.1: mlx4_ib: multi-function enabled > [65189.991132] mlx4_core 0000:05:00.1: mlx4_ib: operating in qp1 tunnel mode > [65189.994633] IPv6: ADDRCONF(NETDEV_UP): ib3: link is not ready > [65190.005505] IPv6: ADDRCONF(NETDEV_UP): ib4: link is not ready > [65190.061269] mlx4_core 0000:82:00.1: mlx4_ib: multi-function enabled > [65190.061271] mlx4_core 0000:82:00.1: mlx4_ib: operating in qp1 tunnel mode > [65190.071641] IPv6: ADDRCONF(NETDEV_UP): ib5: link is not ready > > Here is dmesg command return of virtual machine (mlx4_core loaded with > options debug_level=1) : > > [root@welling0 ~]# dmesg > mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) > mlx4_core: Initializing 0000:00:08.0 > mlx4_core 0000:00:08.0: Detected virtual function - running in slave mode > mlx4_core 0000:00:08.0: Sending reset > mlx4_core 0000:00:08.0: Sending vhcr0 > mlx4_core 0000:00:08.0: BlueFlame not available > mlx4_core 0000:00:08.0: Base MM extensions: flags 00104cc0, rsvd L_Key 00008000 > mlx4_core 0000:00:08.0: Max ICM size 4294967296 MB > mlx4_core 0000:00:08.0: Max QPs: 16777216, reserved QPs: 64, entry size: 256 > mlx4_core 0000:00:08.0: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 > mlx4_core 0000:00:08.0: Max CQs: 16777216, reserved CQs: 128, entry size: 128 > mlx4_core 0000:00:08.0: Max EQs: 32, reserved EQs: 28, entry size: 128 > mlx4_core 0000:00:08.0: reserved MPTs: 256, reserved MTTs: 32 > mlx4_core 0000:00:08.0: Max PDs: 131072, reserved PDs: 4, reserved UARs: 7 > mlx4_core 0000:00:08.0: Max QP/MCG: 131072, reserved MGMs: 0 > mlx4_core 0000:00:08.0: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 > mlx4_core 0000:00:08.0: Local CA ACK delay: 15, max MTU: 4096, port width cap: 2 > mlx4_core 0000:00:08.0: Max SQ desc size: 1008, max SQ S/G: 62 > mlx4_core 0000:00:08.0: Max RQ desc size: 512, max RQ S/G: 32 > mlx4_core 0000:00:08.0: Max GSO size: 131072 > mlx4_core 0000:00:08.0: Max counters: 256 > mlx4_core 0000:00:08.0: Max RSS Table size: 256 > mlx4_core 0000:00:08.0: DEV_CAP flags: > mlx4_core 0000:00:08.0: RC transport > mlx4_core 0000:00:08.0: UC transport > mlx4_core 0000:00:08.0: UD transport > mlx4_core 0000:00:08.0: XRC transport > mlx4_core 0000:00:08.0: FCoIB support > mlx4_core 0000:00:08.0: SRQ support > mlx4_core 0000:00:08.0: IPoIB checksum offload > mlx4_core 0000:00:08.0: P_Key violation counter > mlx4_core 0000:00:08.0: Q_Key violation counter > mlx4_core 0000:00:08.0: DPDP > mlx4_core 0000:00:08.0: Big LSO headers > mlx4_core 0000:00:08.0: APM support > mlx4_core 0000:00:08.0: Atomic ops support > mlx4_core 0000:00:08.0: Address vector port checking support > mlx4_core 0000:00:08.0: UD multicast support > mlx4_core 0000:00:08.0: Router support > mlx4_core 0000:00:08.0: IBoE support > mlx4_core 0000:00:08.0: Unicast loopback support > mlx4_core 0000:00:08.0: FCS header control > mlx4_core 0000:00:08.0: UDP RSS support > mlx4_core 0000:00:08.0: Unicast VEP steering support > mlx4_core 0000:00:08.0: Multicast VEP steering support > mlx4_core 0000:00:08.0: Counters support > mlx4_core 0000:00:08.0: Port management change event support > mlx4_core 0000:00:08.0: 64 byte EQE support > mlx4_core 0000:00:08.0: 64 byte CQE support > mlx4_core 0000:00:08.0: RSS support > mlx4_core 0000:00:08.0: RSS Toeplitz Hash Function support > mlx4_core 0000:00:08.0: RSS XOR Hash Function support > mlx4_core 0000:00:08.0: Device manage flow steering support > mlx4_core 0000:00:08.0: 64B EQEs/CQEs supported by the device but not enabled > mlx4_core 0000:00:08.0: HCA minimum page size:512 > mlx4_core 0000:00:08.0: Steering mode is: B0 steering > mlx4_core 0000:00:08.0: Failed to map blue flame area > mlx4_core 0000:00:08.0: irq 48 for MSI/MSI-X > mlx4_core 0000:00:08.0: irq 49 for MSI/MSI-X > mlx4_core 0000:00:08.0: failed execution of VHCR_POST commandopcode 0x31 > mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 49). > mlx4_core 0000:00:08.0: Trying again without MSI-X. > mlx4_core: probe of 0000:00:08.0 failed with error -16 > > Mathis > > > 2013/1/24 Mathis GAVILLON <jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>: > > ---------- Forwarded message ---------- > > From: Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> > > Date: 2013/1/24 > > Subject: Re: Error message when trying to use Infiniband virtual > > functions in virtual machine > > To: Mathis GAVILLON <jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> > > Cc : linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> > > > > > > On Wed, Jan 23, 2013 at 5:29 PM, Mathis GAVILLON <jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > >> I'm working on a project in which I try to virtualize Infiniband. I use ConnectX3 Infiniband cards with FW 2.9.1200. > >> I've installed a Fedora 18 OS with 3.8.0rc2 recompiled kernel. I use KVM 1.2 to run virtual machines. > > > > Your firmware version is a bit old... if you can upgrade to the latest > > GA on the Mellanox site (2.11.0500) would be good, > > > > So what version of qemu is installed on the distro you use? Are you > > able to work over a ConnectX VF if probed > > to the host? e.g use num_vfs=3 probe_vf=1 and see how the related mlx4 > > device (e.g mlx4_1) is functioning. > > > > Also would be of help if you probe mlx4_core on both VM and host with > > debug_level=1 > > > > > > Or. > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <201301281835.18533.jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>]
* Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <201301281835.18533.jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> @ 2013-01-29 13:15 ` Mathis GAVILLON [not found] ` <CAOoYFkE6YNfEJGJqGmpQTdSxR2Y_9pzkdcnHf_7jjqq57B_Nsw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 16+ messages in thread From: Mathis GAVILLON @ 2013-01-29 13:15 UTC (permalink / raw) To: Jack Morgenstein; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz I try to map the first virtual function with guest O/S. I understand (in your mail) that all virtual functions (which are maped to a guest O/S) must be probed. I didn't know it, that's why I tried with both situations. In fact, one boxe wasn't check in IOMMU submenu (Support for Interrupt Remapping (EXPERIMENTAL)). I've recompiled the kernel of the host and guest (is it necessary for this last one ?). But the error message is already present. Just below, new dmeg log from host and guest O/S (mlx4_core loaded with options num_vfs=1 probe_vf=1 debug_level=1 on host and debug_level=1 on guest) : [ 7061.889083] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) [ 7061.889085] mlx4_core: Initializing 0000:05:00.0 [ 7061.889317] mlx4_core 0000:05:00.0: Enabling SR-IOV with 1 VFs [ 7061.989979] pci 0000:05:00.1: [15b3:1002] type 00 class 0x0c0600 [ 7061.990728] mlx4_core 0000:05:00.0: Running in master mode [ 7062.993002] mlx4_core 0000:05:00.0: FW version 2.9.1200 (cmd intf rev 3), max commands 16 [ 7062.993005] mlx4_core 0000:05:00.0: Catastrophic error buffer at 0x1f020, size 0x10, BAR 0 [ 7062.993006] mlx4_core 0000:05:00.0: Communication vector bar:2 offset:0x800 [ 7062.993008] mlx4_core 0000:05:00.0: FW size 385 KB [ 7062.993009] mlx4_core 0000:05:00.0: Clear int @ f0058, BAR 0 [ 7062.994914] mlx4_core 0000:05:00.0: Mapped 26 chunks/6168 KB for FW. [ 7063.662759] mlx4_core 0000:05:00.0: BlueFlame available (reg size 512, regs/page 8) [ 7063.662941] mlx4_core 0000:05:00.0: Base MM extensions: flags 00000cc0, rsvd L_Key 00000500 [ 7063.662942] mlx4_core 0000:05:00.0: Max ICM size 4294967296 MB [ 7063.662944] mlx4_core 0000:05:00.0: Max QPs: 16777216, reserved QPs: 64, entry size: 256 [ 7063.662946] mlx4_core 0000:05:00.0: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 [ 7063.662947] mlx4_core 0000:05:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 128 [ 7063.662949] mlx4_core 0000:05:00.0: Max EQs: 32, reserved EQs: 28, entry size: 128 [ 7063.662950] mlx4_core 0000:05:00.0: reserved MPTs: 16, reserved MTTs: 16 [ 7063.662952] mlx4_core 0000:05:00.0: Max PDs: 131072, reserved PDs: 4, reserved UARs: 7 [ 7063.662953] mlx4_core 0000:05:00.0: Max QP/MCG: 131072, reserved MGMs: 0 [ 7063.662955] mlx4_core 0000:05:00.0: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 [ 7063.662957] mlx4_core 0000:05:00.0: Local CA ACK delay: 15, max MTU: 4096, port width cap: 3 [ 7063.662958] mlx4_core 0000:05:00.0: Max SQ desc size: 1008, max SQ S/G: 62 [ 7063.662959] mlx4_core 0000:05:00.0: Max RQ desc size: 512, max RQ S/G: 32 [ 7063.662961] mlx4_core 0000:05:00.0: Max GSO size: 131072 [ 7063.662962] mlx4_core 0000:05:00.0: Max counters: 256 [ 7063.662963] mlx4_core 0000:05:00.0: Max RSS Table size: 256 [ 7063.662965] mlx4_core 0000:05:00.0: DEV_CAP flags: [ 7063.662966] mlx4_core 0000:05:00.0: RC transport [ 7063.662967] mlx4_core 0000:05:00.0: UC transport [ 7063.662968] mlx4_core 0000:05:00.0: UD transport [ 7063.662970] mlx4_core 0000:05:00.0: XRC transport [ 7063.662971] mlx4_core 0000:05:00.0: FCoIB support [ 7063.662973] mlx4_core 0000:05:00.0: SRQ support [ 7063.662974] mlx4_core 0000:05:00.0: IPoIB checksum offload [ 7063.662975] mlx4_core 0000:05:00.0: P_Key violation counter [ 7063.662977] mlx4_core 0000:05:00.0: Q_Key violation counter [ 7063.662978] mlx4_core 0000:05:00.0: DPDP [ 7063.662979] mlx4_core 0000:05:00.0: Big LSO headers [ 7063.662980] mlx4_core 0000:05:00.0: APM support [ 7063.662982] mlx4_core 0000:05:00.0: Atomic ops support [ 7063.662983] mlx4_core 0000:05:00.0: Address vector port checking support [ 7063.662984] mlx4_core 0000:05:00.0: UD multicast support [ 7063.662986] mlx4_core 0000:05:00.0: Router support [ 7063.662987] mlx4_core 0000:05:00.0: IBoE support [ 7063.662988] mlx4_core 0000:05:00.0: Unicast loopback support [ 7063.662989] mlx4_core 0000:05:00.0: UDP RSS support [ 7063.662991] mlx4_core 0000:05:00.0: Unicast VEP steering support [ 7063.662992] mlx4_core 0000:05:00.0: Multicast VEP steering support [ 7063.662993] mlx4_core 0000:05:00.0: Counters support [ 7063.662995] mlx4_core 0000:05:00.0: RSS support [ 7063.662996] mlx4_core 0000:05:00.0: RSS Toeplitz Hash Function support [ 7063.662997] mlx4_core 0000:05:00.0: RSS XOR Hash Function support [ 7063.663000] mlx4_core 0000:05:00.0: Steering mode is: B0 steering, oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10 [ 7063.663006] mlx4_core 0000:05:00.0: profile[ 0] ( CMPT): 2^26 entries @ 0x 0, size 0x 100000000 [ 7063.663008] mlx4_core 0000:05:00.0: profile[ 1] (RDMARC): 2^22 entries @ 0x 100000000, size 0x 8000000 [ 7063.663010] mlx4_core 0000:05:00.0: profile[ 2] ( MTT): 2^24 entries @ 0x 108000000, size 0x 8000000 [ 7063.663012] mlx4_core 0000:05:00.0: profile[ 3] ( QP): 2^18 entries @ 0x 110000000, size 0x 4000000 [ 7063.663014] mlx4_core 0000:05:00.0: profile[ 4] ( DMPT): 2^19 entries @ 0x 114000000, size 0x 2000000 [ 7063.663015] mlx4_core 0000:05:00.0: profile[ 5] ( ALTC): 2^18 entries @ 0x 116000000, size 0x 1000000 [ 7063.663017] mlx4_core 0000:05:00.0: profile[ 6] ( SRQ): 2^16 entries @ 0x 117000000, size 0x 800000 [ 7063.663019] mlx4_core 0000:05:00.0: profile[ 7] ( CQ): 2^16 entries @ 0x 117800000, size 0x 800000 [ 7063.663021] mlx4_core 0000:05:00.0: profile[ 8] ( MCG): 2^13 entries @ 0x 118000000, size 0x 800000 [ 7063.663023] mlx4_core 0000:05:00.0: profile[ 9] ( AUXC): 2^18 entries @ 0x 118800000, size 0x 40000 [ 7063.663025] mlx4_core 0000:05:00.0: profile[10] ( EQ): 2^10 entries @ 0x 118840000, size 0x 20000 [ 7063.663026] mlx4_core 0000:05:00.0: HCA context memory: reserving 4596096 KB [ 7063.663044] mlx4_core 0000:05:00.0: 4596096 KB of HCA context requires 9068 KB aux memory. [ 7063.692455] mlx4_core 0000:05:00.0: Mapped 39 chunks/9068 KB for ICM aux. [ 7063.693778] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 0 for ICM. [ 7063.695078] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 40000000 for ICM. [ 7063.696378] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 80000000 for ICM. [ 7063.696715] mlx4_core 0000:05:00.0: Mapped 1 chunks/64 KB at c0000000 for ICM. [ 7063.697362] mlx4_core 0000:05:00.0: Mapped 1 chunks/128 KB at 118840000 for ICM. [ 7063.698641] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 108000000 for ICM. [ 7063.699963] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 114000000 for ICM. [ 7063.701248] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 110000000 for ICM. [ 7063.702536] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118800000 for ICM. [ 7063.703817] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 116000000 for ICM. [ 7063.705099] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 100000000 for ICM. [ 7063.706378] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 117800000 for ICM. [ 7063.707656] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 117000000 for ICM. [ 7063.708941] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118000000 for ICM. [ 7063.710220] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118040000 for ICM. [ 7063.711507] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118080000 for ICM. [ 7063.712796] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1180c0000 for ICM. [ 7063.714073] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118100000 for ICM. [ 7063.715350] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118140000 for ICM. [ 7063.716628] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118180000 for ICM. [ 7063.717907] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1181c0000 for ICM. [ 7063.719194] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118200000 for ICM. [ 7063.720473] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118240000 for ICM. [ 7063.721751] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118280000 for ICM. [ 7063.723041] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1182c0000 for ICM. [ 7063.724321] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118300000 for ICM. [ 7063.725599] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118340000 for ICM. [ 7063.726883] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118380000 for ICM. [ 7063.728168] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1183c0000 for ICM. [ 7063.729456] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118400000 for ICM. [ 7063.730741] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118440000 for ICM. [ 7063.732025] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118480000 for ICM. [ 7063.733322] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1184c0000 for ICM. [ 7063.734601] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118500000 for ICM. [ 7063.735879] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118540000 for ICM. [ 7063.737165] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118580000 for ICM. [ 7063.738441] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1185c0000 for ICM. [ 7063.739721] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118600000 for ICM. [ 7063.741005] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118640000 for ICM. [ 7063.742290] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118680000 for ICM. [ 7063.743579] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1186c0000 for ICM. [ 7063.744863] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118700000 for ICM. [ 7063.746147] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118740000 for ICM. [ 7063.747424] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118780000 for ICM. [ 7063.748702] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1187c0000 for ICM. [ 7064.308524] mlx4_core 0000:05:00.0: Started init_resource_tracker: 80 slaves [ 7064.309007] mlx4_core 0000:05:00.0: irq 176 for MSI/MSI-X [ 7064.309019] mlx4_core 0000:05:00.0: irq 177 for MSI/MSI-X [ 7064.314858] mlx4_core 0000:05:00.0: NOP command IRQ test passed [ 7064.315674] mlx4_core: Initializing 0000:82:00.0 [ 7064.315774] mlx4_core 0000:82:00.0: Enabling SR-IOV with 1 VFs [ 7064.416589] pci 0000:82:00.1: [15b3:1004] type 00 class 0x028000 [ 7064.417304] mlx4_core 0000:82:00.0: Running in master mode [ 7065.419026] mlx4_core 0000:82:00.0: FW version 2.11.500 (cmd intf rev 3), max commands 16 [ 7065.419028] mlx4_core 0000:82:00.0: Catastrophic error buffer at 0x1f020, size 0x10, BAR 0 [ 7065.419030] mlx4_core 0000:82:00.0: Communication vector bar:2 offset:0x800 [ 7065.419032] mlx4_core 0000:82:00.0: FW size 385 KB [ 7065.419033] mlx4_core 0000:82:00.0: Clear int @ f0058, BAR 0 [ 7065.420332] mlx4_core 0000:82:00.0: Mapped 26 chunks/6168 KB for FW. [ 7068.086064] mlx4_core 0000:82:00.0: BlueFlame available (reg size 512, regs/page 8) [ 7068.086196] mlx4_core 0000:82:00.0: Base MM extensions: flags 00104cc0, rsvd L_Key 00008000 [ 7068.086198] mlx4_core 0000:82:00.0: Max ICM size 4294967296 MB [ 7068.086199] mlx4_core 0000:82:00.0: Max QPs: 16777216, reserved QPs: 64, entry size: 256 [ 7068.086201] mlx4_core 0000:82:00.0: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 [ 7068.086203] mlx4_core 0000:82:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 128 [ 7068.086204] mlx4_core 0000:82:00.0: Max EQs: 32, reserved EQs: 28, entry size: 128 [ 7068.086206] mlx4_core 0000:82:00.0: reserved MPTs: 256, reserved MTTs: 32 [ 7068.086208] mlx4_core 0000:82:00.0: Max PDs: 131072, reserved PDs: 4, reserved UARs: 7 [ 7068.086209] mlx4_core 0000:82:00.0: Max QP/MCG: 131072, reserved MGMs: 0 [ 7068.086211] mlx4_core 0000:82:00.0: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 [ 7068.086213] mlx4_core 0000:82:00.0: Local CA ACK delay: 15, max MTU: 4096, port width cap: 2 [ 7068.086214] mlx4_core 0000:82:00.0: Max SQ desc size: 1008, max SQ S/G: 62 [ 7068.086215] mlx4_core 0000:82:00.0: Max RQ desc size: 512, max RQ S/G: 32 [ 7068.086217] mlx4_core 0000:82:00.0: Max GSO size: 131072 [ 7068.086218] mlx4_core 0000:82:00.0: Max counters: 256 [ 7068.086219] mlx4_core 0000:82:00.0: Max RSS Table size: 256 [ 7068.086221] mlx4_core 0000:82:00.0: DEV_CAP flags: [ 7068.086222] mlx4_core 0000:82:00.0: RC transport [ 7068.086223] mlx4_core 0000:82:00.0: UC transport [ 7068.086224] mlx4_core 0000:82:00.0: UD transport [ 7068.086226] mlx4_core 0000:82:00.0: XRC transport [ 7068.086227] mlx4_core 0000:82:00.0: FCoIB support [ 7068.086228] mlx4_core 0000:82:00.0: SRQ support [ 7068.086229] mlx4_core 0000:82:00.0: IPoIB checksum offload [ 7068.086231] mlx4_core 0000:82:00.0: P_Key violation counter [ 7068.086232] mlx4_core 0000:82:00.0: Q_Key violation counter [ 7068.086233] mlx4_core 0000:82:00.0: DPDP [ 7068.086235] mlx4_core 0000:82:00.0: Big LSO headers [ 7068.086236] mlx4_core 0000:82:00.0: APM support [ 7068.086237] mlx4_core 0000:82:00.0: Atomic ops support [ 7068.086238] mlx4_core 0000:82:00.0: Address vector port checking support [ 7068.086240] mlx4_core 0000:82:00.0: UD multicast support [ 7068.086241] mlx4_core 0000:82:00.0: Router support [ 7068.086242] mlx4_core 0000:82:00.0: IBoE support [ 7068.086244] mlx4_core 0000:82:00.0: Unicast loopback support [ 7068.086245] mlx4_core 0000:82:00.0: FCS header control [ 7068.086246] mlx4_core 0000:82:00.0: UDP RSS support [ 7068.086247] mlx4_core 0000:82:00.0: Unicast VEP steering support [ 7068.086249] mlx4_core 0000:82:00.0: Multicast VEP steering support [ 7068.086250] mlx4_core 0000:82:00.0: Counters support [ 7068.086252] mlx4_core 0000:82:00.0: Port management change event support [ 7068.086253] mlx4_core 0000:82:00.0: 64 byte EQE support [ 7068.086254] mlx4_core 0000:82:00.0: 64 byte CQE support [ 7068.086256] mlx4_core 0000:82:00.0: RSS support [ 7068.086257] mlx4_core 0000:82:00.0: RSS Toeplitz Hash Function support [ 7068.086258] mlx4_core 0000:82:00.0: RSS XOR Hash Function support [ 7068.086260] mlx4_core 0000:82:00.0: Device manage flow steering support [ 7068.086263] mlx4_core 0000:82:00.0: 64B EQEs/CQEs supported by the device but not enabled [ 7068.086266] mlx4_core 0000:82:00.0: Steering mode is: B0 steering, oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10 [ 7068.086271] mlx4_core 0000:82:00.0: profile[ 0] ( CMPT): 2^26 entries @ 0x 0, size 0x 100000000 [ 7068.086273] mlx4_core 0000:82:00.0: profile[ 1] (RDMARC): 2^22 entries @ 0x 100000000, size 0x 8000000 [ 7068.086275] mlx4_core 0000:82:00.0: profile[ 2] ( MTT): 2^24 entries @ 0x 108000000, size 0x 8000000 [ 7068.086277] mlx4_core 0000:82:00.0: profile[ 3] ( QP): 2^18 entries @ 0x 110000000, size 0x 4000000 [ 7068.086279] mlx4_core 0000:82:00.0: profile[ 4] ( DMPT): 2^19 entries @ 0x 114000000, size 0x 2000000 [ 7068.086281] mlx4_core 0000:82:00.0: profile[ 5] ( ALTC): 2^18 entries @ 0x 116000000, size 0x 1000000 [ 7068.086283] mlx4_core 0000:82:00.0: profile[ 6] ( SRQ): 2^16 entries @ 0x 117000000, size 0x 800000 [ 7068.086285] mlx4_core 0000:82:00.0: profile[ 7] ( CQ): 2^16 entries @ 0x 117800000, size 0x 800000 [ 7068.086286] mlx4_core 0000:82:00.0: profile[ 8] ( MCG): 2^13 entries @ 0x 118000000, size 0x 800000 [ 7068.086288] mlx4_core 0000:82:00.0: profile[ 9] ( AUXC): 2^18 entries @ 0x 118800000, size 0x 40000 [ 7068.086290] mlx4_core 0000:82:00.0: profile[10] ( EQ): 2^10 entries @ 0x 118840000, size 0x 20000 [ 7068.086292] mlx4_core 0000:82:00.0: HCA context memory: reserving 4596096 KB [ 7068.086309] mlx4_core 0000:82:00.0: 4596096 KB of HCA context requires 9068 KB aux memory. [ 7068.122796] mlx4_core 0000:82:00.0: Mapped 39 chunks/9068 KB for ICM aux. [ 7068.124193] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 0 for ICM. [ 7068.125561] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 40000000 for ICM. [ 7068.126936] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 80000000 for ICM. [ 7068.127289] mlx4_core 0000:82:00.0: Mapped 1 chunks/64 KB at c0000000 for ICM. [ 7068.127981] mlx4_core 0000:82:00.0: Mapped 1 chunks/128 KB at 118840000 for ICM. [ 7068.129333] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 108000000 for ICM. [ 7068.130740] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 114000000 for ICM. [ 7068.132097] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 110000000 for ICM. [ 7068.133457] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118800000 for ICM. [ 7068.134819] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 116000000 for ICM. [ 7068.136195] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 100000000 for ICM. [ 7068.137547] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 117800000 for ICM. [ 7068.138927] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 117000000 for ICM. [ 7068.140292] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118000000 for ICM. [ 7068.141659] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118040000 for ICM. [ 7068.143009] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118080000 for ICM. [ 7068.144369] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1180c0000 for ICM. [ 7068.145725] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118100000 for ICM. [ 7068.147083] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118140000 for ICM. [ 7068.148431] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118180000 for ICM. [ 7068.149801] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1181c0000 for ICM. [ 7068.151179] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118200000 for ICM. [ 7068.152538] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118240000 for ICM. [ 7068.153901] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118280000 for ICM. [ 7068.155277] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1182c0000 for ICM. [ 7068.156626] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118300000 for ICM. [ 7068.157985] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118340000 for ICM. [ 7068.159323] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118380000 for ICM. [ 7068.160691] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1183c0000 for ICM. [ 7068.162058] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118400000 for ICM. [ 7068.163416] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118440000 for ICM. [ 7068.164775] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118480000 for ICM. [ 7068.166152] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1184c0000 for ICM. [ 7068.167510] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118500000 for ICM. [ 7068.168873] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118540000 for ICM. [ 7068.170236] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118580000 for ICM. [ 7068.171594] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1185c0000 for ICM. [ 7068.172959] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118600000 for ICM. [ 7068.174297] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118640000 for ICM. [ 7068.175657] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118680000 for ICM. [ 7068.177028] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1186c0000 for ICM. [ 7068.178382] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118700000 for ICM. [ 7068.179745] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118740000 for ICM. [ 7068.181102] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118780000 for ICM. [ 7068.182455] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1187c0000 for ICM. [ 7068.762473] mlx4_core 0000:82:00.0: Started init_resource_tracker: 80 slaves [ 7068.762941] mlx4_core 0000:82:00.0: irq 178 for MSI/MSI-X [ 7068.762951] mlx4_core 0000:82:00.0: irq 179 for MSI/MSI-X [ 7068.769218] mlx4_core 0000:82:00.0: NOP command IRQ test passed [ 7068.769774] mlx4_core: Initializing 0000:05:00.1 [ 7068.769786] mlx4_core 0000:05:00.1: enabling device (0000 -> 0002) [ 7068.769823] mlx4_core 0000:05:00.1: Detected virtual function - running in slave mode [ 7068.769913] mlx4_core 0000:05:00.1: Sending reset [ 7068.770021] mlx4_core 0000:05:00.0: Received reset from slave:1 [ 7068.770033] mlx4_core 0000:05:00.1: Sending vhcr0 [ 7068.770391] mlx4_core 0000:05:00.1: BlueFlame not available [ 7068.770681] mlx4_core 0000:05:00.1: Base MM extensions: flags 00000cc0, rsvd L_Key 00000500 [ 7068.770685] mlx4_core 0000:05:00.1: Max ICM size 4294967296 MB [ 7068.770686] mlx4_core 0000:05:00.1: Max QPs: 16777216, reserved QPs: 64, entry size: 256 [ 7068.770688] mlx4_core 0000:05:00.1: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 [ 7068.770690] mlx4_core 0000:05:00.1: Max CQs: 16777216, reserved CQs: 128, entry size: 128 [ 7068.770691] mlx4_core 0000:05:00.1: Max EQs: 32, reserved EQs: 28, entry size: 128 [ 7068.770693] mlx4_core 0000:05:00.1: reserved MPTs: 16, reserved MTTs: 16 [ 7068.770694] mlx4_core 0000:05:00.1: Max PDs: 131072, reserved PDs: 4, reserved UARs: 7 [ 7068.770696] mlx4_core 0000:05:00.1: Max QP/MCG: 131072, reserved MGMs: 0 [ 7068.770697] mlx4_core 0000:05:00.1: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 [ 7068.770699] mlx4_core 0000:05:00.1: Local CA ACK delay: 15, max MTU: 4096, port width cap: 3 [ 7068.770700] mlx4_core 0000:05:00.1: Max SQ desc size: 1008, max SQ S/G: 62 [ 7068.770702] mlx4_core 0000:05:00.1: Max RQ desc size: 512, max RQ S/G: 32 [ 7068.770703] mlx4_core 0000:05:00.1: Max GSO size: 131072 [ 7068.770704] mlx4_core 0000:05:00.1: Max counters: 256 [ 7068.770706] mlx4_core 0000:05:00.1: Max RSS Table size: 256 [ 7068.770707] mlx4_core 0000:05:00.1: DEV_CAP flags: [ 7068.770708] mlx4_core 0000:05:00.1: RC transport [ 7068.770709] mlx4_core 0000:05:00.1: UC transport [ 7068.770711] mlx4_core 0000:05:00.1: UD transport [ 7068.770712] mlx4_core 0000:05:00.1: XRC transport [ 7068.770713] mlx4_core 0000:05:00.1: FCoIB support [ 7068.770714] mlx4_core 0000:05:00.1: SRQ support [ 7068.770716] mlx4_core 0000:05:00.1: IPoIB checksum offload [ 7068.770717] mlx4_core 0000:05:00.1: P_Key violation counter [ 7068.770718] mlx4_core 0000:05:00.1: Q_Key violation counter [ 7068.770719] mlx4_core 0000:05:00.1: DPDP [ 7068.770721] mlx4_core 0000:05:00.1: Big LSO headers [ 7068.770722] mlx4_core 0000:05:00.1: APM support [ 7068.770723] mlx4_core 0000:05:00.1: Atomic ops support [ 7068.770724] mlx4_core 0000:05:00.1: Address vector port checking support [ 7068.770726] mlx4_core 0000:05:00.1: UD multicast support [ 7068.770727] mlx4_core 0000:05:00.1: Router support [ 7068.770728] mlx4_core 0000:05:00.1: IBoE support [ 7068.770729] mlx4_core 0000:05:00.1: Unicast loopback support [ 7068.770731] mlx4_core 0000:05:00.1: UDP RSS support [ 7068.770732] mlx4_core 0000:05:00.1: Unicast VEP steering support [ 7068.770733] mlx4_core 0000:05:00.1: Multicast VEP steering support [ 7068.770735] mlx4_core 0000:05:00.1: Counters support [ 7068.770736] mlx4_core 0000:05:00.1: Port management change event support [ 7068.770737] mlx4_core 0000:05:00.1: RSS support [ 7068.770739] mlx4_core 0000:05:00.1: RSS Toeplitz Hash Function support [ 7068.770740] mlx4_core 0000:05:00.1: RSS XOR Hash Function support [ 7068.770856] mlx4_core 0000:05:00.1: HCA minimum page size:512 [ 7068.771367] mlx4_core 0000:05:00.1: Steering mode is: B0 steering [ 7068.771369] mlx4_core 0000:05:00.1: Failed to map blue flame area [ 7068.771638] mlx4_core 0000:05:00.1: irq 180 for MSI/MSI-X [ 7068.771650] mlx4_core 0000:05:00.1: irq 181 for MSI/MSI-X [ 7068.777099] mlx4_core 0000:05:00.1: NOP command IRQ test passed [ 7068.777122] mlx4_core: Initializing 0000:82:00.1 [ 7068.777133] mlx4_core 0000:82:00.1: enabling device (0000 -> 0002) [ 7068.777174] mlx4_core 0000:82:00.1: Detected virtual function - running in slave mode [ 7068.777297] mlx4_core 0000:82:00.1: Sending reset [ 7068.777362] mlx4_core 0000:82:00.0: Received reset from slave:1 [ 7068.777372] mlx4_core 0000:82:00.1: Sending vhcr0 [ 7068.777707] mlx4_core 0000:82:00.1: BlueFlame not available [ 7068.777869] mlx4_core 0000:82:00.1: Base MM extensions: flags 00104cc0, rsvd L_Key 00008000 [ 7068.777870] mlx4_core 0000:82:00.1: Max ICM size 4294967296 MB [ 7068.777872] mlx4_core 0000:82:00.1: Max QPs: 16777216, reserved QPs: 64, entry size: 256 [ 7068.777874] mlx4_core 0000:82:00.1: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 [ 7068.777875] mlx4_core 0000:82:00.1: Max CQs: 16777216, reserved CQs: 128, entry size: 128 [ 7068.777877] mlx4_core 0000:82:00.1: Max EQs: 32, reserved EQs: 28, entry size: 128 [ 7068.777879] mlx4_core 0000:82:00.1: reserved MPTs: 256, reserved MTTs: 32 [ 7068.777880] mlx4_core 0000:82:00.1: Max PDs: 131072, reserved PDs: 4, reserved UARs: 7 [ 7068.777882] mlx4_core 0000:82:00.1: Max QP/MCG: 131072, reserved MGMs: 0 [ 7068.777883] mlx4_core 0000:82:00.1: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 [ 7068.777885] mlx4_core 0000:82:00.1: Local CA ACK delay: 15, max MTU: 4096, port width cap: 2 [ 7068.777887] mlx4_core 0000:82:00.1: Max SQ desc size: 1008, max SQ S/G: 62 [ 7068.777888] mlx4_core 0000:82:00.1: Max RQ desc size: 512, max RQ S/G: 32 [ 7068.777889] mlx4_core 0000:82:00.1: Max GSO size: 131072 [ 7068.777891] mlx4_core 0000:82:00.1: Max counters: 256 [ 7068.777892] mlx4_core 0000:82:00.1: Max RSS Table size: 256 [ 7068.777893] mlx4_core 0000:82:00.1: DEV_CAP flags: [ 7068.777895] mlx4_core 0000:82:00.1: RC transport [ 7068.777896] mlx4_core 0000:82:00.1: UC transport [ 7068.777897] mlx4_core 0000:82:00.1: UD transport [ 7068.777898] mlx4_core 0000:82:00.1: XRC transport [ 7068.777899] mlx4_core 0000:82:00.1: FCoIB support [ 7068.777901] mlx4_core 0000:82:00.1: SRQ support [ 7068.777902] mlx4_core 0000:82:00.1: IPoIB checksum offload [ 7068.777903] mlx4_core 0000:82:00.1: P_Key violation counter [ 7068.777904] mlx4_core 0000:82:00.1: Q_Key violation counter [ 7068.777906] mlx4_core 0000:82:00.1: DPDP [ 7068.777907] mlx4_core 0000:82:00.1: Big LSO headers [ 7068.777908] mlx4_core 0000:82:00.1: APM support [ 7068.777909] mlx4_core 0000:82:00.1: Atomic ops support [ 7068.777911] mlx4_core 0000:82:00.1: Address vector port checking support [ 7068.777912] mlx4_core 0000:82:00.1: UD multicast support [ 7068.777913] mlx4_core 0000:82:00.1: Router support [ 7068.777915] mlx4_core 0000:82:00.1: IBoE support [ 7068.777916] mlx4_core 0000:82:00.1: Unicast loopback support [ 7068.777917] mlx4_core 0000:82:00.1: FCS header control [ 7068.777918] mlx4_core 0000:82:00.1: UDP RSS support [ 7068.777920] mlx4_core 0000:82:00.1: Unicast VEP steering support [ 7068.777921] mlx4_core 0000:82:00.1: Multicast VEP steering support [ 7068.777922] mlx4_core 0000:82:00.1: Counters support [ 7068.777924] mlx4_core 0000:82:00.1: Port management change event support [ 7068.777925] mlx4_core 0000:82:00.1: 64 byte EQE support [ 7068.777926] mlx4_core 0000:82:00.1: 64 byte CQE support [ 7068.777928] mlx4_core 0000:82:00.1: RSS support [ 7068.777929] mlx4_core 0000:82:00.1: RSS Toeplitz Hash Function support [ 7068.777930] mlx4_core 0000:82:00.1: RSS XOR Hash Function support [ 7068.777932] mlx4_core 0000:82:00.1: Device manage flow steering support [ 7068.777933] mlx4_core 0000:82:00.1: 64B EQEs/CQEs supported by the device but not enabled [ 7068.778041] mlx4_core 0000:82:00.1: HCA minimum page size:512 [ 7068.778342] mlx4_core 0000:82:00.1: Steering mode is: B0 steering [ 7068.778343] mlx4_core 0000:82:00.1: Failed to map blue flame area [ 7068.778631] mlx4_core 0000:82:00.1: irq 182 for MSI/MSI-X [ 7068.778642] mlx4_core 0000:82:00.1: irq 183 for MSI/MSI-X [ 7068.784770] mlx4_core 0000:82:00.1: NOP command IRQ test passed [ 7267.277498] RPC: Registered named UNIX socket transport module. [ 7267.277500] RPC: Registered udp transport module. [ 7267.277501] RPC: Registered tcp transport module. [ 7267.277502] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 7267.521859] Installing knfsd (copyright (C) 1996 okir-pn4DOG8n3UYbFoVRYvo4fw@public.gmane.org). [ 7267.579717] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory [ 7267.580043] NFSD: starting 90-second grace period (net ffffffff81ab1c40) [ 7560.813405] mlx4_core 0000:82:00.0: Received reset from slave:1 [ 7560.813663] pci-stub 0000:82:00.1: claimed by stub [ 7560.819403] pci-stub 0000:82:00.1: enabling device (0000 -> 0002) [ 7560.821843] mlx4_core 0000:82:00.0: FLR event for slave: 1 [ 7560.821890] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr [ 7560.821893] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 [ 7560.923233] assign device 0:82:0.1 [ 7560.925757] mlx4_core 0000:82:00.0: FLR event for slave: 1 [ 7560.925768] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr [ 7560.925770] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 [ 7561.046117] mlx4_core 0000:82:00.0: Received reset from slave:1 [ 7561.060850] pci-stub 0000:82:00.1: irq 182 for MSI/MSI-X [ 7561.083880] pci-stub 0000:82:00.1: irq 182 for MSI/MSI-X [ 7561.083895] pci-stub 0000:82:00.1: irq 183 for MSI/MSI-X [ 7571.189144] mlx4_core 0000:82:00.0: Received reset from slave:1 [ 7618.874766] mlx4_core 0000:82:00.0: Received reset from slave:1 [ 7618.888390] pci-stub 0000:82:00.1: irq 182 for MSI/MSI-X [ 7618.909284] pci-stub 0000:82:00.1: irq 182 for MSI/MSI-X [ 7618.909308] pci-stub 0000:82:00.1: irq 183 for MSI/MSI-X [ 7629.014979] mlx4_core 0000:82:00.0: Received reset from slave:1 mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) mlx4_core: Initializing 0000:00:08.0 mlx4_core 0000:00:08.0: Detected virtual function - running in slave mode mlx4_core 0000:00:08.0: Sending reset mlx4_core 0000:00:08.0: Sending vhcr0 mlx4_core 0000:00:08.0: BlueFlame not available mlx4_core 0000:00:08.0: Base MM extensions: flags 00104cc0, rsvd L_Key 00008000 mlx4_core 0000:00:08.0: Max ICM size 4294967296 MB mlx4_core 0000:00:08.0: Max QPs: 16777216, reserved QPs: 64, entry size: 256 mlx4_core 0000:00:08.0: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 mlx4_core 0000:00:08.0: Max CQs: 16777216, reserved CQs: 128, entry size: 128 mlx4_core 0000:00:08.0: Max EQs: 32, reserved EQs: 28, entry size: 128 mlx4_core 0000:00:08.0: reserved MPTs: 256, reserved MTTs: 32 mlx4_core 0000:00:08.0: Max PDs: 131072, reserved PDs: 4, reserved UARs: 7 mlx4_core 0000:00:08.0: Max QP/MCG: 131072, reserved MGMs: 0 mlx4_core 0000:00:08.0: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 mlx4_core 0000:00:08.0: Local CA ACK delay: 15, max MTU: 4096, port width cap: 2 mlx4_core 0000:00:08.0: Max SQ desc size: 1008, max SQ S/G: 62 mlx4_core 0000:00:08.0: Max RQ desc size: 512, max RQ S/G: 32 mlx4_core 0000:00:08.0: Max GSO size: 131072 mlx4_core 0000:00:08.0: Max counters: 256 mlx4_core 0000:00:08.0: Max RSS Table size: 256 mlx4_core 0000:00:08.0: DEV_CAP flags: mlx4_core 0000:00:08.0: RC transport mlx4_core 0000:00:08.0: UC transport mlx4_core 0000:00:08.0: UD transport mlx4_core 0000:00:08.0: XRC transport mlx4_core 0000:00:08.0: FCoIB support mlx4_core 0000:00:08.0: SRQ support mlx4_core 0000:00:08.0: IPoIB checksum offload mlx4_core 0000:00:08.0: P_Key violation counter mlx4_core 0000:00:08.0: Q_Key violation counter mlx4_core 0000:00:08.0: DPDP mlx4_core 0000:00:08.0: Big LSO headers mlx4_core 0000:00:08.0: APM support mlx4_core 0000:00:08.0: Atomic ops support mlx4_core 0000:00:08.0: Address vector port checking support mlx4_core 0000:00:08.0: UD multicast support mlx4_core 0000:00:08.0: Router support mlx4_core 0000:00:08.0: IBoE support mlx4_core 0000:00:08.0: Unicast loopback support mlx4_core 0000:00:08.0: FCS header control mlx4_core 0000:00:08.0: UDP RSS support mlx4_core 0000:00:08.0: Unicast VEP steering support mlx4_core 0000:00:08.0: Multicast VEP steering support mlx4_core 0000:00:08.0: Counters support mlx4_core 0000:00:08.0: Port management change event support mlx4_core 0000:00:08.0: 64 byte EQE support mlx4_core 0000:00:08.0: 64 byte CQE support mlx4_core 0000:00:08.0: RSS support mlx4_core 0000:00:08.0: RSS Toeplitz Hash Function support mlx4_core 0000:00:08.0: RSS XOR Hash Function support mlx4_core 0000:00:08.0: Device manage flow steering support mlx4_core 0000:00:08.0: 64B EQEs/CQEs supported by the device but not enabled mlx4_core 0000:00:08.0: HCA minimum page size:512 mlx4_core 0000:00:08.0: Steering mode is: B0 steering mlx4_core 0000:00:08.0: Failed to map blue flame area mlx4_core 0000:00:08.0: irq 48 for MSI/MSI-X mlx4_core 0000:00:08.0: irq 49 for MSI/MSI-X mlx4_core 0000:00:08.0: failed execution of VHCR_POST commandopcode 0x31 mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 49). mlx4_core 0000:00:08.0: Trying again without MSI-X. mlx4_core: probe of 0000:00:08.0 failed with error -16 Mathis 2013/1/28 Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>: > Which VF did you map to the guest O/S? > In your configuration, both HCAs will have the PF and the > first VF running on the hypervisor, > and the remaining VFs available for binding to guests. > > You have: > mlx4_core 0000:05:00.0: Enabling SR-IOV with 3 VFs <== PF > pci 0000:05:00.1: [15b3:1002] type 00 class 0x0c0600 <== VFs > pci 0000:05:00.2: [15b3:1002] type 00 class 0x0c0600 > pci 0000:05:00.3: [15b3:1002] type 00 class 0x0c0600 > > 05:00.0 and 05:00.1 run on the hypervisor, and 05:00.2/3 are > available for binding to guests. > > Similarly, > mlx4_core 0000:82:00.0: Enabling SR-IOV with 3 VFs > pci 0000:82:00.1: [15b3:1004] type 00 class 0x028000 > pci 0000:82:00.2: [15b3:1004] type 00 class 0x028000 > pci 0000:82:00.3: [15b3:1004] type 00 class 0x028000 > > 82:00.0 and 82:00.1 run on the hypervisor, and 82:00.2/3 are > available for binding to guests. > > The "skipping" messages you see for 2/3 indicate that these > are not yet bound, and the hypervisor is not claiming them > (this arises from setting probe_vf=1 --> only ONE guest is > claimed/bound by the Hypervisor). > > I don't see any "received reset from xxx" lines in the dumps > below, which would indicate that the hypervisor received a > reset request from a guest O/S bound to one of the available VFs. > > This looks to me like a kernel configuration problem. > > You did not mention what CPU you are using. > Did you do the following when building your kernel? > make menuconfig (stuff below is shown for kernel 3.7. I don't know if it > changed in 3.8) > Then, under "Device Drivers", put an '*' by "IOMMU hardware support" and > go into that submenu. > Make sure to check ALL the boxes which apply for your CPU in the IOMMU submenu. > > For Intel, at least, you should also add "intel_iommu=on" to the line for your kernel > in file /boot/grub/menu.lst > > -Jack > > On Monday 28 January 2013 10:29, Mathis GAVILLON wrote: >> The version of firmware I gave you was wrong (I've two cards >> (ConnectX2 and ConnectX3, I only use ConnectX 3) in my computer and I >> took the bad one). The firmware installed on my ConnectX3 Infiniband >> Card is 2.11.500. >> KVM is 1.2.2-1 version. >> I tried to load the driver with options you gave me but the error >> message is the same. Nothing else happen. >> >> Here is the dmesg command return from host (mlx4_core loaded with >> options debug_level=1, num_vfs=3, probe_vf=1) : >> >> [root@barb0 ~]# dmesg >> [65182.838953] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) >> [65182.838955] mlx4_core: Initializing 0000:05:00.0 >> [65182.839144] mlx4_core 0000:05:00.0: Enabling SR-IOV with 3 VFs >> [65182.940316] pci 0000:05:00.1: [15b3:1002] type 00 class 0x0c0600 >> [65182.941105] pci 0000:05:00.2: [15b3:1002] type 00 class 0x0c0600 >> [65182.941876] pci 0000:05:00.3: [15b3:1002] type 00 class 0x0c0600 >> [65182.942585] mlx4_core 0000:05:00.0: Running in master mode >> [65183.945306] mlx4_core 0000:05:00.0: FW version 2.9.1200 (cmd intf >> rev 3), max commands 16 >> [65183.945309] mlx4_core 0000:05:00.0: Catastrophic error buffer at >> 0x1f020, size 0x10, BAR 0 >> [65183.945311] mlx4_core 0000:05:00.0: Communication vector bar:2 offset:0x800 >> [65183.945312] mlx4_core 0000:05:00.0: FW size 385 KB >> [65183.945313] mlx4_core 0000:05:00.0: Clear int @ f0058, BAR 0 >> [65183.947177] mlx4_core 0000:05:00.0: Mapped 26 chunks/6168 KB for FW. >> [65184.614576] mlx4_core 0000:05:00.0: BlueFlame available (reg size >> 512, regs/page 8) >> [65184.614742] mlx4_core 0000:05:00.0: Base MM extensions: flags >> 00000cc0, rsvd L_Key 00000500 >> [65184.614743] mlx4_core 0000:05:00.0: Max ICM size 4294967296 MB >> [65184.614745] mlx4_core 0000:05:00.0: Max QPs: 16777216, reserved >> QPs: 64, entry size: 256 >> [65184.614747] mlx4_core 0000:05:00.0: Max SRQs: 16777216, reserved >> SRQs: 64, entry size: 128 >> [65184.614748] mlx4_core 0000:05:00.0: Max CQs: 16777216, reserved >> CQs: 128, entry size: 128 >> [65184.614750] mlx4_core 0000:05:00.0: Max EQs: 32, reserved EQs: 28, >> entry size: 128 >> [65184.614751] mlx4_core 0000:05:00.0: reserved MPTs: 16, reserved MTTs: 16 >> [65184.614753] mlx4_core 0000:05:00.0: Max PDs: 131072, reserved PDs: >> 4, reserved UARs: 7 >> [65184.614754] mlx4_core 0000:05:00.0: Max QP/MCG: 131072, reserved MGMs: 0 >> [65184.614756] mlx4_core 0000:05:00.0: Max CQEs: 4194304, max WQEs: >> 16384, max SRQ WQEs: 16384 >> [65184.614758] mlx4_core 0000:05:00.0: Local CA ACK delay: 15, max >> MTU: 4096, port width cap: 3 >> [65184.614759] mlx4_core 0000:05:00.0: Max SQ desc size: 1008, max SQ S/G: 62 >> [65184.614760] mlx4_core 0000:05:00.0: Max RQ desc size: 512, max RQ S/G: 32 >> [65184.614762] mlx4_core 0000:05:00.0: Max GSO size: 131072 >> [65184.614763] mlx4_core 0000:05:00.0: Max counters: 256 >> [65184.614764] mlx4_core 0000:05:00.0: Max RSS Table size: 256 >> [65184.614766] mlx4_core 0000:05:00.0: DEV_CAP flags: >> [65184.614767] mlx4_core 0000:05:00.0: RC transport >> [65184.614768] mlx4_core 0000:05:00.0: UC transport >> [65184.614769] mlx4_core 0000:05:00.0: UD transport >> [65184.614771] mlx4_core 0000:05:00.0: XRC transport >> [65184.614772] mlx4_core 0000:05:00.0: FCoIB support >> [65184.614773] mlx4_core 0000:05:00.0: SRQ support >> [65184.614774] mlx4_core 0000:05:00.0: IPoIB checksum offload >> [65184.614775] mlx4_core 0000:05:00.0: P_Key violation counter >> [65184.614777] mlx4_core 0000:05:00.0: Q_Key violation counter >> [65184.614778] mlx4_core 0000:05:00.0: DPDP >> [65184.614779] mlx4_core 0000:05:00.0: Big LSO headers >> [65184.614780] mlx4_core 0000:05:00.0: APM support >> [65184.614782] mlx4_core 0000:05:00.0: Atomic ops support >> [65184.614783] mlx4_core 0000:05:00.0: Address vector port checking support >> [65184.614784] mlx4_core 0000:05:00.0: UD multicast support >> [65184.614785] mlx4_core 0000:05:00.0: Router support >> [65184.614787] mlx4_core 0000:05:00.0: IBoE support >> [65184.614788] mlx4_core 0000:05:00.0: Unicast loopback support >> [65184.614789] mlx4_core 0000:05:00.0: UDP RSS support >> [65184.614790] mlx4_core 0000:05:00.0: Unicast VEP steering support >> [65184.614792] mlx4_core 0000:05:00.0: Multicast VEP steering support >> [65184.614793] mlx4_core 0000:05:00.0: Counters support >> [65184.614795] mlx4_core 0000:05:00.0: RSS support >> [65184.614796] mlx4_core 0000:05:00.0: RSS Toeplitz Hash Function support >> [65184.614797] mlx4_core 0000:05:00.0: RSS XOR Hash Function support >> [65184.614800] mlx4_core 0000:05:00.0: Steering mode is: B0 steering, >> oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10 >> [65184.614807] mlx4_core 0000:05:00.0: profile[ 0] ( CMPT): 2^26 >> entries @ 0x 0, size 0x 100000000 >> [65184.614809] mlx4_core 0000:05:00.0: profile[ 1] (RDMARC): 2^22 >> entries @ 0x 100000000, size 0x 8000000 >> [65184.614811] mlx4_core 0000:05:00.0: profile[ 2] ( MTT): 2^24 >> entries @ 0x 108000000, size 0x 8000000 >> [65184.614813] mlx4_core 0000:05:00.0: profile[ 3] ( QP): 2^18 >> entries @ 0x 110000000, size 0x 4000000 >> [65184.614815] mlx4_core 0000:05:00.0: profile[ 4] ( DMPT): 2^19 >> entries @ 0x 114000000, size 0x 2000000 >> [65184.614816] mlx4_core 0000:05:00.0: profile[ 5] ( ALTC): 2^18 >> entries @ 0x 116000000, size 0x 1000000 >> [65184.614818] mlx4_core 0000:05:00.0: profile[ 6] ( SRQ): 2^16 >> entries @ 0x 117000000, size 0x 800000 >> [65184.614820] mlx4_core 0000:05:00.0: profile[ 7] ( CQ): 2^16 >> entries @ 0x 117800000, size 0x 800000 >> [65184.614822] mlx4_core 0000:05:00.0: profile[ 8] ( MCG): 2^13 >> entries @ 0x 118000000, size 0x 800000 >> [65184.614824] mlx4_core 0000:05:00.0: profile[ 9] ( AUXC): 2^18 >> entries @ 0x 118800000, size 0x 40000 >> [65184.614826] mlx4_core 0000:05:00.0: profile[10] ( EQ): 2^10 >> entries @ 0x 118840000, size 0x 20000 >> [65184.614827] mlx4_core 0000:05:00.0: HCA context memory: reserving 4596096 KB >> [65184.614844] mlx4_core 0000:05:00.0: 4596096 KB of HCA context >> requires 9068 KB aux memory. >> [65184.644227] mlx4_core 0000:05:00.0: Mapped 39 chunks/9068 KB for ICM aux. >> [65184.645547] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 0 for ICM. >> [65184.646846] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 40000000 for ICM. >> [65184.648146] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 80000000 for ICM. >> [65184.648483] mlx4_core 0000:05:00.0: Mapped 1 chunks/64 KB at >> c0000000 for ICM. >> [65184.649133] mlx4_core 0000:05:00.0: Mapped 1 chunks/128 KB at >> 118840000 for ICM. >> [65184.650413] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 108000000 for ICM. >> [65184.651737] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 114000000 for ICM. >> [65184.653016] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 110000000 for ICM. >> [65184.654304] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118800000 for ICM. >> [65184.655585] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 116000000 for ICM. >> [65184.656866] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 100000000 for ICM. >> [65184.658146] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 117800000 for ICM. >> [65184.659426] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 117000000 for ICM. >> [65184.660705] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118000000 for ICM. >> [65184.661986] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118040000 for ICM. >> [65184.663265] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118080000 for ICM. >> [65184.664550] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 1180c0000 for ICM. >> [65184.665839] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118100000 for ICM. >> [65184.667117] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118140000 for ICM. >> [65184.668395] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118180000 for ICM. >> [65184.669675] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 1181c0000 for ICM. >> [65184.670955] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118200000 for ICM. >> [65184.672234] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118240000 for ICM. >> [65184.673511] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118280000 for ICM. >> [65184.674794] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 1182c0000 for ICM. >> [65184.676070] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118300000 for ICM. >> [65184.677351] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118340000 for ICM. >> [65184.678629] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118380000 for ICM. >> [65184.679906] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 1183c0000 for ICM. >> [65184.681185] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118400000 for ICM. >> [65184.682463] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118440000 for ICM. >> [65184.683742] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118480000 for ICM. >> [65184.685028] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 1184c0000 for ICM. >> [65184.686306] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118500000 for ICM. >> [65184.687583] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118540000 for ICM. >> [65184.688869] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118580000 for ICM. >> [65184.690148] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 1185c0000 for ICM. >> [65184.691428] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118600000 for ICM. >> [65184.692706] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118640000 for ICM. >> [65184.693984] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118680000 for ICM. >> [65184.695268] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 1186c0000 for ICM. >> [65184.696546] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118700000 for ICM. >> [65184.697825] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118740000 for ICM. >> [65184.699103] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 118780000 for ICM. >> [65184.700380] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 1187c0000 for ICM. >> [65185.202376] mlx4_core 0000:05:00.0: Started init_resource_tracker: 80 slaves >> [65185.202819] mlx4_core 0000:05:00.0: irq 176 for MSI/MSI-X >> [65185.202826] mlx4_core 0000:05:00.0: irq 177 for MSI/MSI-X >> [65185.207838] mlx4_core 0000:05:00.0: NOP command IRQ test passed >> [65185.208638] mlx4_core: Initializing 0000:82:00.0 >> [65185.208742] mlx4_core 0000:82:00.0: Enabling SR-IOV with 3 VFs >> [65185.309000] pci 0000:82:00.1: [15b3:1004] type 00 class 0x028000 >> [65185.309781] pci 0000:82:00.2: [15b3:1004] type 00 class 0x028000 >> [65185.310507] pci 0000:82:00.3: [15b3:1004] type 00 class 0x028000 >> [65185.311176] mlx4_core 0000:82:00.0: Running in master mode >> [65186.312479] mlx4_core 0000:82:00.0: FW version 2.11.500 (cmd intf >> rev 3), max commands 16 >> [65186.312482] mlx4_core 0000:82:00.0: Catastrophic error buffer at >> 0x1f020, size 0x10, BAR 0 >> [65186.312484] mlx4_core 0000:82:00.0: Communication vector bar:2 offset:0x800 >> [65186.312485] mlx4_core 0000:82:00.0: FW size 385 KB >> [65186.312486] mlx4_core 0000:82:00.0: Clear int @ f0058, BAR 0 >> [65186.313785] mlx4_core 0000:82:00.0: Mapped 26 chunks/6168 KB for FW. >> [65188.980410] mlx4_core 0000:82:00.0: BlueFlame available (reg size >> 512, regs/page 8) >> [65188.980535] mlx4_core 0000:82:00.0: Base MM extensions: flags >> 00104cc0, rsvd L_Key 00008000 >> [65188.980536] mlx4_core 0000:82:00.0: Max ICM size 4294967296 MB >> [65188.980538] mlx4_core 0000:82:00.0: Max QPs: 16777216, reserved >> QPs: 64, entry size: 256 >> [65188.980540] mlx4_core 0000:82:00.0: Max SRQs: 16777216, reserved >> SRQs: 64, entry size: 128 >> [65188.980541] mlx4_core 0000:82:00.0: Max CQs: 16777216, reserved >> CQs: 128, entry size: 128 >> [65188.980543] mlx4_core 0000:82:00.0: Max EQs: 32, reserved EQs: 28, >> entry size: 128 >> [65188.980544] mlx4_core 0000:82:00.0: reserved MPTs: 256, reserved MTTs: 32 >> [65188.980546] mlx4_core 0000:82:00.0: Max PDs: 131072, reserved PDs: >> 4, reserved UARs: 7 >> [65188.980548] mlx4_core 0000:82:00.0: Max QP/MCG: 131072, reserved MGMs: 0 >> [65188.980549] mlx4_core 0000:82:00.0: Max CQEs: 4194304, max WQEs: >> 16384, max SRQ WQEs: 16384 >> [65188.980551] mlx4_core 0000:82:00.0: Local CA ACK delay: 15, max >> MTU: 4096, port width cap: 2 >> [65188.980552] mlx4_core 0000:82:00.0: Max SQ desc size: 1008, max SQ S/G: 62 >> [65188.980554] mlx4_core 0000:82:00.0: Max RQ desc size: 512, max RQ S/G: 32 >> [65188.980555] mlx4_core 0000:82:00.0: Max GSO size: 131072 >> [65188.980556] mlx4_core 0000:82:00.0: Max counters: 256 >> [65188.980558] mlx4_core 0000:82:00.0: Max RSS Table size: 256 >> [65188.980559] mlx4_core 0000:82:00.0: DEV_CAP flags: >> [65188.980560] mlx4_core 0000:82:00.0: RC transport >> [65188.980561] mlx4_core 0000:82:00.0: UC transport >> [65188.980563] mlx4_core 0000:82:00.0: UD transport >> [65188.980564] mlx4_core 0000:82:00.0: XRC transport >> [65188.980565] mlx4_core 0000:82:00.0: FCoIB support >> [65188.980566] mlx4_core 0000:82:00.0: SRQ support >> [65188.980568] mlx4_core 0000:82:00.0: IPoIB checksum offload >> [65188.980569] mlx4_core 0000:82:00.0: P_Key violation counter >> [65188.980570] mlx4_core 0000:82:00.0: Q_Key violation counter >> [65188.980571] mlx4_core 0000:82:00.0: DPDP >> [65188.980573] mlx4_core 0000:82:00.0: Big LSO headers >> [65188.980574] mlx4_core 0000:82:00.0: APM support >> [65188.980575] mlx4_core 0000:82:00.0: Atomic ops support >> [65188.980576] mlx4_core 0000:82:00.0: Address vector port checking support >> [65188.980578] mlx4_core 0000:82:00.0: UD multicast support >> [65188.980579] mlx4_core 0000:82:00.0: Router support >> [65188.980580] mlx4_core 0000:82:00.0: IBoE support >> [65188.980582] mlx4_core 0000:82:00.0: Unicast loopback support >> [65188.980583] mlx4_core 0000:82:00.0: FCS header control >> [65188.980584] mlx4_core 0000:82:00.0: UDP RSS support >> [65188.980586] mlx4_core 0000:82:00.0: Unicast VEP steering support >> [65188.980587] mlx4_core 0000:82:00.0: Multicast VEP steering support >> [65188.980588] mlx4_core 0000:82:00.0: Counters support >> [65188.980589] mlx4_core 0000:82:00.0: Port management change event support >> [65188.980591] mlx4_core 0000:82:00.0: 64 byte EQE support >> [65188.980592] mlx4_core 0000:82:00.0: 64 byte CQE support >> [65188.980593] mlx4_core 0000:82:00.0: RSS support >> [65188.980595] mlx4_core 0000:82:00.0: RSS Toeplitz Hash Function support >> [65188.980596] mlx4_core 0000:82:00.0: RSS XOR Hash Function support >> [65188.980597] mlx4_core 0000:82:00.0: Device manage flow steering support >> [65188.980599] mlx4_core 0000:82:00.0: 64B EQEs/CQEs supported by the >> device but not enabled >> [65188.980602] mlx4_core 0000:82:00.0: Steering mode is: B0 steering, >> oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10 >> [65188.980609] mlx4_core 0000:82:00.0: profile[ 0] ( CMPT): 2^26 >> entries @ 0x 0, size 0x 100000000 >> [65188.980611] mlx4_core 0000:82:00.0: profile[ 1] (RDMARC): 2^22 >> entries @ 0x 100000000, size 0x 8000000 >> [65188.980613] mlx4_core 0000:82:00.0: profile[ 2] ( MTT): 2^24 >> entries @ 0x 108000000, size 0x 8000000 >> [65188.980615] mlx4_core 0000:82:00.0: profile[ 3] ( QP): 2^18 >> entries @ 0x 110000000, size 0x 4000000 >> [65188.980616] mlx4_core 0000:82:00.0: profile[ 4] ( DMPT): 2^19 >> entries @ 0x 114000000, size 0x 2000000 >> [65188.980618] mlx4_core 0000:82:00.0: profile[ 5] ( ALTC): 2^18 >> entries @ 0x 116000000, size 0x 1000000 >> [65188.980620] mlx4_core 0000:82:00.0: profile[ 6] ( SRQ): 2^16 >> entries @ 0x 117000000, size 0x 800000 >> [65188.980622] mlx4_core 0000:82:00.0: profile[ 7] ( CQ): 2^16 >> entries @ 0x 117800000, size 0x 800000 >> [65188.980624] mlx4_core 0000:82:00.0: profile[ 8] ( MCG): 2^13 >> entries @ 0x 118000000, size 0x 800000 >> [65188.980625] mlx4_core 0000:82:00.0: profile[ 9] ( AUXC): 2^18 >> entries @ 0x 118800000, size 0x 40000 >> [65188.980627] mlx4_core 0000:82:00.0: profile[10] ( EQ): 2^10 >> entries @ 0x 118840000, size 0x 20000 >> [65188.980629] mlx4_core 0000:82:00.0: HCA context memory: reserving 4596096 KB >> [65188.980645] mlx4_core 0000:82:00.0: 4596096 KB of HCA context >> requires 9068 KB aux memory. >> [65189.017150] mlx4_core 0000:82:00.0: Mapped 39 chunks/9068 KB for ICM aux. >> [65189.018533] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 0 for ICM. >> [65189.019895] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 40000000 for ICM. >> [65189.021266] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 80000000 for ICM. >> [65189.021621] mlx4_core 0000:82:00.0: Mapped 1 chunks/64 KB at >> c0000000 for ICM. >> [65189.022303] mlx4_core 0000:82:00.0: Mapped 1 chunks/128 KB at >> 118840000 for ICM. >> [65189.023671] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 108000000 for ICM. >> [65189.025060] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 114000000 for ICM. >> [65189.026419] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 110000000 for ICM. >> [65189.027778] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118800000 for ICM. >> [65189.029123] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 116000000 for ICM. >> [65189.030477] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 100000000 for ICM. >> [65189.031834] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 117800000 for ICM. >> [65189.033197] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 117000000 for ICM. >> [65189.034559] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118000000 for ICM. >> [65189.035906] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118040000 for ICM. >> [65189.037254] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118080000 for ICM. >> [65189.038633] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 1180c0000 for ICM. >> [65189.039991] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118100000 for ICM. >> [65189.041349] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118140000 for ICM. >> [65189.042700] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118180000 for ICM. >> [65189.044050] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 1181c0000 for ICM. >> [65189.045409] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118200000 for ICM. >> [65189.046758] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118240000 for ICM. >> [65189.048098] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118280000 for ICM. >> [65189.049466] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 1182c0000 for ICM. >> [65189.050813] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118300000 for ICM. >> [65189.052169] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118340000 for ICM. >> [65189.053521] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118380000 for ICM. >> [65189.054862] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 1183c0000 for ICM. >> [65189.056218] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118400000 for ICM. >> [65189.057563] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118440000 for ICM. >> [65189.058913] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118480000 for ICM. >> [65189.060272] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 1184c0000 for ICM. >> [65189.061616] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118500000 for ICM. >> [65189.062971] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118540000 for ICM. >> [65189.064314] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118580000 for ICM. >> [65189.065669] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 1185c0000 for ICM. >> [65189.067011] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118600000 for ICM. >> [65189.068364] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118640000 for ICM. >> [65189.069728] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118680000 for ICM. >> [65189.071086] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 1186c0000 for ICM. >> [65189.072438] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118700000 for ICM. >> [65189.073788] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118740000 for ICM. >> [65189.075145] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 118780000 for ICM. >> [65189.076503] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 1187c0000 for ICM. >> [65189.643406] mlx4_core 0000:82:00.0: Started init_resource_tracker: 80 slaves >> [65189.643891] mlx4_core 0000:82:00.0: irq 178 for MSI/MSI-X >> [65189.643899] mlx4_core 0000:82:00.0: irq 179 for MSI/MSI-X >> [65189.650122] mlx4_core 0000:82:00.0: NOP command IRQ test passed >> [65189.650652] mlx4_core: Initializing 0000:05:00.1 >> [65189.650667] mlx4_core 0000:05:00.1: enabling device (0000 -> 0002) >> [65189.650709] mlx4_core 0000:05:00.1: Detected virtual function - >> running in slave mode >> [65189.650792] mlx4_core 0000:05:00.1: Sending reset >> [65189.650847] mlx4_core 0000:05:00.0: Received reset from slave:1 >> [65189.650859] mlx4_core 0000:05:00.1: Sending vhcr0 >> [65189.651166] mlx4_core 0000:05:00.1: BlueFlame not available >> [65189.651412] mlx4_core 0000:05:00.1: Base MM extensions: flags >> 00000cc0, rsvd L_Key 00000500 >> [65189.651414] mlx4_core 0000:05:00.1: Max ICM size 4294967296 MB >> [65189.651417] mlx4_core 0000:05:00.1: Max QPs: 16777216, reserved >> QPs: 64, entry size: 256 >> [65189.651419] mlx4_core 0000:05:00.1: Max SRQs: 16777216, reserved >> SRQs: 64, entry size: 128 >> [65189.651421] mlx4_core 0000:05:00.1: Max CQs: 16777216, reserved >> CQs: 128, entry size: 128 >> [65189.651422] mlx4_core 0000:05:00.1: Max EQs: 32, reserved EQs: 28, >> entry size: 128 >> [65189.651424] mlx4_core 0000:05:00.1: reserved MPTs: 16, reserved MTTs: 16 >> [65189.651425] mlx4_core 0000:05:00.1: Max PDs: 131072, reserved PDs: >> 4, reserved UARs: 7 >> [65189.651427] mlx4_core 0000:05:00.1: Max QP/MCG: 131072, reserved MGMs: 0 >> [65189.651428] mlx4_core 0000:05:00.1: Max CQEs: 4194304, max WQEs: >> 16384, max SRQ WQEs: 16384 >> [65189.651430] mlx4_core 0000:05:00.1: Local CA ACK delay: 15, max >> MTU: 4096, port width cap: 3 >> [65189.651431] mlx4_core 0000:05:00.1: Max SQ desc size: 1008, max SQ S/G: 62 >> [65189.651433] mlx4_core 0000:05:00.1: Max RQ desc size: 512, max RQ S/G: 32 >> [65189.651434] mlx4_core 0000:05:00.1: Max GSO size: 131072 >> [65189.651436] mlx4_core 0000:05:00.1: Max counters: 256 >> [65189.651437] mlx4_core 0000:05:00.1: Max RSS Table size: 256 >> [65189.651438] mlx4_core 0000:05:00.1: DEV_CAP flags: >> [65189.651440] mlx4_core 0000:05:00.1: RC transport >> [65189.651441] mlx4_core 0000:05:00.1: UC transport >> [65189.651442] mlx4_core 0000:05:00.1: UD transport >> [65189.651443] mlx4_core 0000:05:00.1: XRC transport >> [65189.651444] mlx4_core 0000:05:00.1: FCoIB support >> [65189.651446] mlx4_core 0000:05:00.1: SRQ support >> [65189.651447] mlx4_core 0000:05:00.1: IPoIB checksum offload >> [65189.651448] mlx4_core 0000:05:00.1: P_Key violation counter >> [65189.651449] mlx4_core 0000:05:00.1: Q_Key violation counter >> [65189.651451] mlx4_core 0000:05:00.1: DPDP >> [65189.651452] mlx4_core 0000:05:00.1: Big LSO headers >> [65189.651453] mlx4_core 0000:05:00.1: APM support >> [65189.651454] mlx4_core 0000:05:00.1: Atomic ops support >> [65189.651456] mlx4_core 0000:05:00.1: Address vector port checking support >> [65189.651457] mlx4_core 0000:05:00.1: UD multicast support >> [65189.651458] mlx4_core 0000:05:00.1: Router support >> [65189.651459] mlx4_core 0000:05:00.1: IBoE support >> [65189.651461] mlx4_core 0000:05:00.1: Unicast loopback support >> [65189.651462] mlx4_core 0000:05:00.1: UDP RSS support >> [65189.651463] mlx4_core 0000:05:00.1: Unicast VEP steering support >> [65189.651465] mlx4_core 0000:05:00.1: Multicast VEP steering support >> [65189.651466] mlx4_core 0000:05:00.1: Counters support >> [65189.651467] mlx4_core 0000:05:00.1: Port management change event support >> [65189.651469] mlx4_core 0000:05:00.1: RSS support >> [65189.651470] mlx4_core 0000:05:00.1: RSS Toeplitz Hash Function support >> [65189.651471] mlx4_core 0000:05:00.1: RSS XOR Hash Function support >> [65189.651567] mlx4_core 0000:05:00.1: HCA minimum page size:512 >> [65189.652182] mlx4_core 0000:05:00.1: Steering mode is: B0 steering >> [65189.652184] mlx4_core 0000:05:00.1: Failed to map blue flame area >> [65189.652438] mlx4_core 0000:05:00.1: irq 180 for MSI/MSI-X >> [65189.652446] mlx4_core 0000:05:00.1: irq 181 for MSI/MSI-X >> [65189.657770] mlx4_core 0000:05:00.1: NOP command IRQ test passed >> [65189.657791] mlx4_core: Initializing 0000:05:00.2 >> [65189.657801] mlx4_core 0000:05:00.2: enabling device (0000 -> 0002) >> [65189.657836] mlx4_core 0000:05:00.2: Skipping virtual function:2 >> [65189.657856] mlx4_core: Initializing 0000:05:00.3 >> [65189.657865] mlx4_core 0000:05:00.3: enabling device (0000 -> 0002) >> [65189.657887] mlx4_core 0000:05:00.3: Skipping virtual function:3 >> [65189.657904] mlx4_core: Initializing 0000:82:00.1 >> [65189.657914] mlx4_core 0000:82:00.1: enabling device (0000 -> 0002) >> [65189.657940] mlx4_core 0000:82:00.1: Detected virtual function - >> running in slave mode >> [65189.658047] mlx4_core 0000:82:00.1: Sending reset >> [65189.658128] mlx4_core 0000:82:00.0: Received reset from slave:1 >> [65189.658137] mlx4_core 0000:82:00.1: Sending vhcr0 >> [65189.658463] mlx4_core 0000:82:00.1: BlueFlame not available >> [65189.658646] mlx4_core 0000:82:00.1: Base MM extensions: flags >> 00104cc0, rsvd L_Key 00008000 >> [65189.658650] mlx4_core 0000:82:00.1: Max ICM size 4294967296 MB >> [65189.658653] mlx4_core 0000:82:00.1: Max QPs: 16777216, reserved >> QPs: 64, entry size: 256 >> [65189.658655] mlx4_core 0000:82:00.1: Max SRQs: 16777216, reserved >> SRQs: 64, entry size: 128 >> [65189.658657] mlx4_core 0000:82:00.1: Max CQs: 16777216, reserved >> CQs: 128, entry size: 128 >> [65189.658658] mlx4_core 0000:82:00.1: Max EQs: 32, reserved EQs: 28, >> entry size: 128 >> [65189.658660] mlx4_core 0000:82:00.1: reserved MPTs: 256, reserved MTTs: 32 >> [65189.658661] mlx4_core 0000:82:00.1: Max PDs: 131072, reserved PDs: >> 4, reserved UARs: 7 >> [65189.658663] mlx4_core 0000:82:00.1: Max QP/MCG: 131072, reserved MGMs: 0 >> [65189.658664] mlx4_core 0000:82:00.1: Max CQEs: 4194304, max WQEs: >> 16384, max SRQ WQEs: 16384 >> [65189.658666] mlx4_core 0000:82:00.1: Local CA ACK delay: 15, max >> MTU: 4096, port width cap: 2 >> [65189.658667] mlx4_core 0000:82:00.1: Max SQ desc size: 1008, max SQ S/G: 62 >> [65189.658669] mlx4_core 0000:82:00.1: Max RQ desc size: 512, max RQ S/G: 32 >> [65189.658670] mlx4_core 0000:82:00.1: Max GSO size: 131072 >> [65189.658671] mlx4_core 0000:82:00.1: Max counters: 256 >> [65189.658673] mlx4_core 0000:82:00.1: Max RSS Table size: 256 >> [65189.658674] mlx4_core 0000:82:00.1: DEV_CAP flags: >> [65189.658675] mlx4_core 0000:82:00.1: RC transport >> [65189.658676] mlx4_core 0000:82:00.1: UC transport >> [65189.658678] mlx4_core 0000:82:00.1: UD transport >> [65189.658679] mlx4_core 0000:82:00.1: XRC transport >> [65189.658680] mlx4_core 0000:82:00.1: FCoIB support >> [65189.658681] mlx4_core 0000:82:00.1: SRQ support >> [65189.658682] mlx4_core 0000:82:00.1: IPoIB checksum offload >> [65189.658684] mlx4_core 0000:82:00.1: P_Key violation counter >> [65189.658685] mlx4_core 0000:82:00.1: Q_Key violation counter >> [65189.658688] mlx4_core 0000:82:00.1: DPDP >> [65189.658691] mlx4_core 0000:82:00.1: Big LSO headers >> [65189.658695] mlx4_core 0000:82:00.1: APM support >> [65189.658699] mlx4_core 0000:82:00.1: Atomic ops support >> [65189.658702] mlx4_core 0000:82:00.1: Address vector port checking support >> [65189.658705] mlx4_core 0000:82:00.1: UD multicast support >> [65189.658707] mlx4_core 0000:82:00.1: Router support >> [65189.658708] mlx4_core 0000:82:00.1: IBoE support >> [65189.658709] mlx4_core 0000:82:00.1: Unicast loopback support >> [65189.658710] mlx4_core 0000:82:00.1: FCS header control >> [65189.658712] mlx4_core 0000:82:00.1: UDP RSS support >> [65189.658713] mlx4_core 0000:82:00.1: Unicast VEP steering support >> [65189.658714] mlx4_core 0000:82:00.1: Multicast VEP steering support >> [65189.658715] mlx4_core 0000:82:00.1: Counters support >> [65189.658717] mlx4_core 0000:82:00.1: Port management change event support >> [65189.658718] mlx4_core 0000:82:00.1: 64 byte EQE support >> [65189.658719] mlx4_core 0000:82:00.1: 64 byte CQE support >> [65189.658721] mlx4_core 0000:82:00.1: RSS support >> [65189.658722] mlx4_core 0000:82:00.1: RSS Toeplitz Hash Function support >> [65189.658723] mlx4_core 0000:82:00.1: RSS XOR Hash Function support >> [65189.658725] mlx4_core 0000:82:00.1: Device manage flow steering support >> [65189.658727] mlx4_core 0000:82:00.1: 64B EQEs/CQEs supported by the >> device but not enabled >> [65189.658863] mlx4_core 0000:82:00.1: HCA minimum page size:512 >> [65189.659129] mlx4_core 0000:82:00.1: Steering mode is: B0 steering >> [65189.659130] mlx4_core 0000:82:00.1: Failed to map blue flame area >> [65189.659417] mlx4_core 0000:82:00.1: irq 182 for MSI/MSI-X >> [65189.659428] mlx4_core 0000:82:00.1: irq 183 for MSI/MSI-X >> [65189.666007] mlx4_core 0000:82:00.1: NOP command IRQ test passed >> [65189.666033] mlx4_core: Initializing 0000:82:00.2 >> [65189.666045] mlx4_core 0000:82:00.2: enabling device (0000 -> 0002) >> [65189.666083] mlx4_core 0000:82:00.2: Skipping virtual function:2 >> [65189.666106] mlx4_core: Initializing 0000:82:00.3 >> [65189.666116] mlx4_core 0000:82:00.3: enabling device (0000 -> 0002) >> [65189.666142] mlx4_core 0000:82:00.3: Skipping virtual function:3 >> [65189.690937] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX >> InfiniBand driver v1.0 (April 4, 2008) >> [65189.779306] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 110040000 for ICM. >> [65189.780627] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 100080000 for ICM. >> [65189.786286] mlx4_core 0000:05:00.0: mlx4_ib: multi-function enabled >> [65189.788526] mlx4_core 0000:05:00.0: mlx4_ib: initializing demux >> service for 80 qp1 clients >> [65189.792146] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at >> 100040000 for ICM. >> [65189.793859] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready >> [65189.803736] IPv6: ADDRCONF(NETDEV_UP): ib1: link is not ready >> [65189.862554] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 110040000 for ICM. >> [65189.863888] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 100080000 for ICM. >> [65189.864185] mlx4_core 0000:82:00.0: mlx4_ib: multi-function enabled >> [65189.865217] mlx4_core 0000:82:00.0: mlx4_ib: initializing demux >> service for 80 qp1 clients >> [65189.868388] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at >> 100040000 for ICM. >> [65189.875240] IPv6: ADDRCONF(NETDEV_UP): ib2: link is not ready >> [65189.991130] mlx4_core 0000:05:00.1: mlx4_ib: multi-function enabled >> [65189.991132] mlx4_core 0000:05:00.1: mlx4_ib: operating in qp1 tunnel mode >> [65189.994633] IPv6: ADDRCONF(NETDEV_UP): ib3: link is not ready >> [65190.005505] IPv6: ADDRCONF(NETDEV_UP): ib4: link is not ready >> [65190.061269] mlx4_core 0000:82:00.1: mlx4_ib: multi-function enabled >> [65190.061271] mlx4_core 0000:82:00.1: mlx4_ib: operating in qp1 tunnel mode >> [65190.071641] IPv6: ADDRCONF(NETDEV_UP): ib5: link is not ready >> >> Here is dmesg command return of virtual machine (mlx4_core loaded with >> options debug_level=1) : >> >> [root@welling0 ~]# dmesg >> mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) >> mlx4_core: Initializing 0000:00:08.0 >> mlx4_core 0000:00:08.0: Detected virtual function - running in slave mode >> mlx4_core 0000:00:08.0: Sending reset >> mlx4_core 0000:00:08.0: Sending vhcr0 >> mlx4_core 0000:00:08.0: BlueFlame not available >> mlx4_core 0000:00:08.0: Base MM extensions: flags 00104cc0, rsvd L_Key 00008000 >> mlx4_core 0000:00:08.0: Max ICM size 4294967296 MB >> mlx4_core 0000:00:08.0: Max QPs: 16777216, reserved QPs: 64, entry size: 256 >> mlx4_core 0000:00:08.0: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 >> mlx4_core 0000:00:08.0: Max CQs: 16777216, reserved CQs: 128, entry size: 128 >> mlx4_core 0000:00:08.0: Max EQs: 32, reserved EQs: 28, entry size: 128 >> mlx4_core 0000:00:08.0: reserved MPTs: 256, reserved MTTs: 32 >> mlx4_core 0000:00:08.0: Max PDs: 131072, reserved PDs: 4, reserved UARs: 7 >> mlx4_core 0000:00:08.0: Max QP/MCG: 131072, reserved MGMs: 0 >> mlx4_core 0000:00:08.0: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 >> mlx4_core 0000:00:08.0: Local CA ACK delay: 15, max MTU: 4096, port width cap: 2 >> mlx4_core 0000:00:08.0: Max SQ desc size: 1008, max SQ S/G: 62 >> mlx4_core 0000:00:08.0: Max RQ desc size: 512, max RQ S/G: 32 >> mlx4_core 0000:00:08.0: Max GSO size: 131072 >> mlx4_core 0000:00:08.0: Max counters: 256 >> mlx4_core 0000:00:08.0: Max RSS Table size: 256 >> mlx4_core 0000:00:08.0: DEV_CAP flags: >> mlx4_core 0000:00:08.0: RC transport >> mlx4_core 0000:00:08.0: UC transport >> mlx4_core 0000:00:08.0: UD transport >> mlx4_core 0000:00:08.0: XRC transport >> mlx4_core 0000:00:08.0: FCoIB support >> mlx4_core 0000:00:08.0: SRQ support >> mlx4_core 0000:00:08.0: IPoIB checksum offload >> mlx4_core 0000:00:08.0: P_Key violation counter >> mlx4_core 0000:00:08.0: Q_Key violation counter >> mlx4_core 0000:00:08.0: DPDP >> mlx4_core 0000:00:08.0: Big LSO headers >> mlx4_core 0000:00:08.0: APM support >> mlx4_core 0000:00:08.0: Atomic ops support >> mlx4_core 0000:00:08.0: Address vector port checking support >> mlx4_core 0000:00:08.0: UD multicast support >> mlx4_core 0000:00:08.0: Router support >> mlx4_core 0000:00:08.0: IBoE support >> mlx4_core 0000:00:08.0: Unicast loopback support >> mlx4_core 0000:00:08.0: FCS header control >> mlx4_core 0000:00:08.0: UDP RSS support >> mlx4_core 0000:00:08.0: Unicast VEP steering support >> mlx4_core 0000:00:08.0: Multicast VEP steering support >> mlx4_core 0000:00:08.0: Counters support >> mlx4_core 0000:00:08.0: Port management change event support >> mlx4_core 0000:00:08.0: 64 byte EQE support >> mlx4_core 0000:00:08.0: 64 byte CQE support >> mlx4_core 0000:00:08.0: RSS support >> mlx4_core 0000:00:08.0: RSS Toeplitz Hash Function support >> mlx4_core 0000:00:08.0: RSS XOR Hash Function support >> mlx4_core 0000:00:08.0: Device manage flow steering support >> mlx4_core 0000:00:08.0: 64B EQEs/CQEs supported by the device but not enabled >> mlx4_core 0000:00:08.0: HCA minimum page size:512 >> mlx4_core 0000:00:08.0: Steering mode is: B0 steering >> mlx4_core 0000:00:08.0: Failed to map blue flame area >> mlx4_core 0000:00:08.0: irq 48 for MSI/MSI-X >> mlx4_core 0000:00:08.0: irq 49 for MSI/MSI-X >> mlx4_core 0000:00:08.0: failed execution of VHCR_POST commandopcode 0x31 >> mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 49). >> mlx4_core 0000:00:08.0: Trying again without MSI-X. >> mlx4_core: probe of 0000:00:08.0 failed with error -16 >> >> Mathis >> >> >> 2013/1/24 Mathis GAVILLON <jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>: >> > ---------- Forwarded message ---------- >> > From: Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> >> > Date: 2013/1/24 >> > Subject: Re: Error message when trying to use Infiniband virtual >> > functions in virtual machine >> > To: Mathis GAVILLON <jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> >> > Cc : linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> >> > >> > >> > On Wed, Jan 23, 2013 at 5:29 PM, Mathis GAVILLON <jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >> > >> >> I'm working on a project in which I try to virtualize Infiniband. I use ConnectX3 Infiniband cards with FW 2.9.1200. >> >> I've installed a Fedora 18 OS with 3.8.0rc2 recompiled kernel. I use KVM 1.2 to run virtual machines. >> > >> > Your firmware version is a bit old... if you can upgrade to the latest >> > GA on the Mellanox site (2.11.0500) would be good, >> > >> > So what version of qemu is installed on the distro you use? Are you >> > able to work over a ConnectX VF if probed >> > to the host? e.g use num_vfs=3 probe_vf=1 and see how the related mlx4 >> > device (e.g mlx4_1) is functioning. >> > >> > Also would be of help if you probe mlx4_core on both VM and host with >> > debug_level=1 >> > >> > >> > Or. >> -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <CAOoYFkE6YNfEJGJqGmpQTdSxR2Y_9pzkdcnHf_7jjqq57B_Nsw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <CAOoYFkE6YNfEJGJqGmpQTdSxR2Y_9pzkdcnHf_7jjqq57B_Nsw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-01-29 13:20 ` Mathis GAVILLON 2013-01-29 14:33 ` Jack Morgenstein 2013-01-29 14:44 ` Jack Morgenstein 2 siblings, 0 replies; 16+ messages in thread From: Mathis GAVILLON @ 2013-01-29 13:20 UTC (permalink / raw) To: Jack Morgenstein; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz I forgot : my CPU is an Intel(R) Xeon(R) CPU E5-2660 and "intel_iommu=on" was already added in kernel command line . Thanks Mathis -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <CAOoYFkE6YNfEJGJqGmpQTdSxR2Y_9pzkdcnHf_7jjqq57B_Nsw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-01-29 13:20 ` Mathis GAVILLON @ 2013-01-29 14:33 ` Jack Morgenstein 2013-01-29 14:44 ` Jack Morgenstein 2 siblings, 0 replies; 16+ messages in thread From: Jack Morgenstein @ 2013-01-29 14:33 UTC (permalink / raw) To: Mathis GAVILLON; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz On Tuesday 29 January 2013 15:15, Mathis GAVILLON wrote: > In fact, one boxe wasn't check in IOMMU submenu (Support for Interrupt > Remapping (EXPERIMENTAL)). I've recompiled the kernel of the host and > guest (is it necessary for this last one ?) Yes, it is necessary. Without that checked, I noticed that SRIOV did not work properly. -Jack > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <CAOoYFkE6YNfEJGJqGmpQTdSxR2Y_9pzkdcnHf_7jjqq57B_Nsw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-01-29 13:20 ` Mathis GAVILLON 2013-01-29 14:33 ` Jack Morgenstein @ 2013-01-29 14:44 ` Jack Morgenstein [not found] ` <201301291644.35771.jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 2 siblings, 1 reply; 16+ messages in thread From: Jack Morgenstein @ 2013-01-29 14:44 UTC (permalink / raw) To: Mathis GAVILLON; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz On Tuesday 29 January 2013 15:15, Mathis GAVILLON wrote: > In fact, one boxe wasn't check in IOMMU submenu (Support for Interrupt > Remapping (EXPERIMENTAL)). I've recompiled the kernel of the host and > guest (is it necessary for this last one ?). But the error message is > already present. Just below, new dmeg log from host and guest O/S > (mlx4_core loaded with options num_vfs=1 probe_vf=1 debug_level=1 on > host and debug_level=1 on guest) : > Please do the following: 1. Check the Support for Interrupt Remapping (EXPERIMENTAL) box and recompile/reinstall. 2. set probe_vf=0, so that no vfs will run on the Hypervisor. That way you will see if any VF comes up without the "noise" in /var/log/messages in the Hypervisor generated by having one of the VFs run on the Hypervisor. 3. I noticed that you are enabling only a single VF, and that you are having the VF come up first on the Hypervisor, then moving it to run on a guest. It is better not to do things that way. Best to have it come up cleanly once on the guest. (the FLR event in your log results from this move. The VF gets reset). If you must have a VF running on the Hypervisor, use "num_vfs=2 probe_vf=1". You will then see Devices ...xx.0, ...xx.1 and ...xx.2 in lspci. The first VF (...xx.1) will be initialized on the Hypervisor. Attach ...xx.2 to the VM. -Jack -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <201301291644.35771.jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>]
* Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <201301291644.35771.jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> @ 2013-01-29 15:18 ` Mathis GAVILLON [not found] ` <CAOoYFkErXNxSBrQC_spMwcEiVmOpkzqQwuTQPOwO70bigsgYkQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> [not found] ` <201302031040.20906.jackm@dev.mellanox.co.il> 0 siblings, 2 replies; 16+ messages in thread From: Mathis GAVILLON @ 2013-01-29 15:18 UTC (permalink / raw) To: Jack Morgenstein; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz Support for interrupt remapping should be now available (kernel has been recompiled with but is there any way to check if it's available ?). I tried to do what you say in step 2 (num_vfs=1 and probe_vf=0) but it doesn't change anything. I also tried num_vfs=2 and probe_vf=1 but nothing is better. Nevertheless, a strange information message appear in shell when I boot my guest : Message from syslogd@localhost at Jan 29 15:51:49 ... kernel:Disabling IRQ #16 I put dmesg log from step 2 (host and guest) again : [ 1419.382452] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) [ 1419.382454] mlx4_core: Initializing 0000:05:00.0 [ 1419.382653] mlx4_core 0000:05:00.0: Enabling SR-IOV with 1 VFs [ 1419.483687] pci 0000:05:00.1: [15b3:1002] type 00 class 0x0c0600 [ 1419.484427] mlx4_core 0000:05:00.0: Running in master mode [ 1420.486608] mlx4_core 0000:05:00.0: FW version 2.9.1200 (cmd intf rev 3), max commands 16 [ 1420.486610] mlx4_core 0000:05:00.0: Catastrophic error buffer at 0x1f020, size 0x10, BAR 0 [ 1420.486612] mlx4_core 0000:05:00.0: Communication vector bar:2 offset:0x800 [ 1420.486613] mlx4_core 0000:05:00.0: FW size 385 KB [ 1420.486615] mlx4_core 0000:05:00.0: Clear int @ f0058, BAR 0 [ 1420.488474] mlx4_core 0000:05:00.0: Mapped 26 chunks/6168 KB for FW. [ 1421.155871] mlx4_core 0000:05:00.0: BlueFlame available (reg size 512, regs/page 8) [ 1421.156035] mlx4_core 0000:05:00.0: Base MM extensions: flags 00000cc0, rsvd L_Key 00000500 [ 1421.156036] mlx4_core 0000:05:00.0: Max ICM size 4294967296 MB [ 1421.156038] mlx4_core 0000:05:00.0: Max QPs: 16777216, reserved QPs: 64, entry size: 256 [ 1421.156040] mlx4_core 0000:05:00.0: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 [ 1421.156041] mlx4_core 0000:05:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 128 [ 1421.156043] mlx4_core 0000:05:00.0: Max EQs: 32, reserved EQs: 28, entry size: 128 [ 1421.156045] mlx4_core 0000:05:00.0: reserved MPTs: 16, reserved MTTs: 16 [ 1421.156046] mlx4_core 0000:05:00.0: Max PDs: 131072, reserved PDs: 4, reserved UARs: 7 [ 1421.156048] mlx4_core 0000:05:00.0: Max QP/MCG: 131072, reserved MGMs: 0 [ 1421.156049] mlx4_core 0000:05:00.0: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 [ 1421.156051] mlx4_core 0000:05:00.0: Local CA ACK delay: 15, max MTU: 4096, port width cap: 3 [ 1421.156052] mlx4_core 0000:05:00.0: Max SQ desc size: 1008, max SQ S/G: 62 [ 1421.156054] mlx4_core 0000:05:00.0: Max RQ desc size: 512, max RQ S/G: 32 [ 1421.156055] mlx4_core 0000:05:00.0: Max GSO size: 131072 [ 1421.156056] mlx4_core 0000:05:00.0: Max counters: 256 [ 1421.156058] mlx4_core 0000:05:00.0: Max RSS Table size: 256 [ 1421.156059] mlx4_core 0000:05:00.0: DEV_CAP flags: [ 1421.156060] mlx4_core 0000:05:00.0: RC transport [ 1421.156062] mlx4_core 0000:05:00.0: UC transport [ 1421.156063] mlx4_core 0000:05:00.0: UD transport [ 1421.156064] mlx4_core 0000:05:00.0: XRC transport [ 1421.156065] mlx4_core 0000:05:00.0: FCoIB support [ 1421.156066] mlx4_core 0000:05:00.0: SRQ support [ 1421.156068] mlx4_core 0000:05:00.0: IPoIB checksum offload [ 1421.156069] mlx4_core 0000:05:00.0: P_Key violation counter [ 1421.156070] mlx4_core 0000:05:00.0: Q_Key violation counter [ 1421.156071] mlx4_core 0000:05:00.0: DPDP [ 1421.156073] mlx4_core 0000:05:00.0: Big LSO headers [ 1421.156074] mlx4_core 0000:05:00.0: APM support [ 1421.156075] mlx4_core 0000:05:00.0: Atomic ops support [ 1421.156077] mlx4_core 0000:05:00.0: Address vector port checking support [ 1421.156078] mlx4_core 0000:05:00.0: UD multicast support [ 1421.156079] mlx4_core 0000:05:00.0: Router support [ 1421.156080] mlx4_core 0000:05:00.0: IBoE support [ 1421.156082] mlx4_core 0000:05:00.0: Unicast loopback support [ 1421.156083] mlx4_core 0000:05:00.0: UDP RSS support [ 1421.156084] mlx4_core 0000:05:00.0: Unicast VEP steering support [ 1421.156085] mlx4_core 0000:05:00.0: Multicast VEP steering support [ 1421.156087] mlx4_core 0000:05:00.0: Counters support [ 1421.156088] mlx4_core 0000:05:00.0: RSS support [ 1421.156090] mlx4_core 0000:05:00.0: RSS Toeplitz Hash Function support [ 1421.156091] mlx4_core 0000:05:00.0: RSS XOR Hash Function support [ 1421.156093] mlx4_core 0000:05:00.0: Steering mode is: B0 steering, oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10 [ 1421.156099] mlx4_core 0000:05:00.0: profile[ 0] ( CMPT): 2^26 entries @ 0x 0, size 0x 100000000 [ 1421.156101] mlx4_core 0000:05:00.0: profile[ 1] (RDMARC): 2^22 entries @ 0x 100000000, size 0x 8000000 [ 1421.156103] mlx4_core 0000:05:00.0: profile[ 2] ( MTT): 2^24 entries @ 0x 108000000, size 0x 8000000 [ 1421.156105] mlx4_core 0000:05:00.0: profile[ 3] ( QP): 2^18 entries @ 0x 110000000, size 0x 4000000 [ 1421.156107] mlx4_core 0000:05:00.0: profile[ 4] ( DMPT): 2^19 entries @ 0x 114000000, size 0x 2000000 [ 1421.156109] mlx4_core 0000:05:00.0: profile[ 5] ( ALTC): 2^18 entries @ 0x 116000000, size 0x 1000000 [ 1421.156111] mlx4_core 0000:05:00.0: profile[ 6] ( SRQ): 2^16 entries @ 0x 117000000, size 0x 800000 [ 1421.156113] mlx4_core 0000:05:00.0: profile[ 7] ( CQ): 2^16 entries @ 0x 117800000, size 0x 800000 [ 1421.156115] mlx4_core 0000:05:00.0: profile[ 8] ( MCG): 2^13 entries @ 0x 118000000, size 0x 800000 [ 1421.156117] mlx4_core 0000:05:00.0: profile[ 9] ( AUXC): 2^18 entries @ 0x 118800000, size 0x 40000 [ 1421.156119] mlx4_core 0000:05:00.0: profile[10] ( EQ): 2^10 entries @ 0x 118840000, size 0x 20000 [ 1421.156120] mlx4_core 0000:05:00.0: HCA context memory: reserving 4596096 KB [ 1421.156136] mlx4_core 0000:05:00.0: 4596096 KB of HCA context requires 9068 KB aux memory. [ 1421.185505] mlx4_core 0000:05:00.0: Mapped 39 chunks/9068 KB for ICM aux. [ 1421.186824] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 0 for ICM. [ 1421.188122] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 40000000 for ICM. [ 1421.189418] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 80000000 for ICM. [ 1421.189754] mlx4_core 0000:05:00.0: Mapped 1 chunks/64 KB at c0000000 for ICM. [ 1421.190400] mlx4_core 0000:05:00.0: Mapped 1 chunks/128 KB at 118840000 for ICM. [ 1421.191677] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 108000000 for ICM. [ 1421.193001] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 114000000 for ICM. [ 1421.194279] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 110000000 for ICM. [ 1421.195563] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118800000 for ICM. [ 1421.196841] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 116000000 for ICM. [ 1421.198127] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 100000000 for ICM. [ 1421.199405] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 117800000 for ICM. [ 1421.200684] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 117000000 for ICM. [ 1421.201964] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118000000 for ICM. [ 1421.203240] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118040000 for ICM. [ 1421.204517] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118080000 for ICM. [ 1421.205802] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1180c0000 for ICM. [ 1421.207079] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118100000 for ICM. [ 1421.208356] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118140000 for ICM. [ 1421.209633] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118180000 for ICM. [ 1421.210912] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1181c0000 for ICM. [ 1421.212191] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118200000 for ICM. [ 1421.213465] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118240000 for ICM. [ 1421.214742] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118280000 for ICM. [ 1421.216024] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1182c0000 for ICM. [ 1421.217303] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118300000 for ICM. [ 1421.218580] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118340000 for ICM. [ 1421.219855] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118380000 for ICM. [ 1421.221132] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1183c0000 for ICM. [ 1421.222407] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118400000 for ICM. [ 1421.223685] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118440000 for ICM. [ 1421.224963] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118480000 for ICM. [ 1421.226245] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1184c0000 for ICM. [ 1421.227522] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118500000 for ICM. [ 1421.228801] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118540000 for ICM. [ 1421.230079] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118580000 for ICM. [ 1421.231355] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1185c0000 for ICM. [ 1421.232633] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118600000 for ICM. [ 1421.233909] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118640000 for ICM. [ 1421.235186] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118680000 for ICM. [ 1421.236470] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1186c0000 for ICM. [ 1421.237748] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118700000 for ICM. [ 1421.239026] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118740000 for ICM. [ 1421.240303] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 118780000 for ICM. [ 1421.241581] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 1187c0000 for ICM. [ 1421.744993] mlx4_core 0000:05:00.0: Started init_resource_tracker: 80 slaves [ 1421.745454] mlx4_core 0000:05:00.0: irq 176 for MSI/MSI-X [ 1421.745466] mlx4_core 0000:05:00.0: irq 177 for MSI/MSI-X [ 1421.750286] mlx4_core 0000:05:00.0: NOP command IRQ test passed [ 1421.750911] mlx4_core: Initializing 0000:82:00.0 [ 1421.751006] mlx4_core 0000:82:00.0: Enabling SR-IOV with 1 VFs [ 1421.851346] pci 0000:82:00.1: [15b3:1004] type 00 class 0x028000 [ 1421.852051] mlx4_core 0000:82:00.0: Running in master mode [ 1422.853834] mlx4_core 0000:82:00.0: FW version 2.11.500 (cmd intf rev 3), max commands 16 [ 1422.853837] mlx4_core 0000:82:00.0: Catastrophic error buffer at 0x1f020, size 0x10, BAR 0 [ 1422.853839] mlx4_core 0000:82:00.0: Communication vector bar:2 offset:0x800 [ 1422.853840] mlx4_core 0000:82:00.0: FW size 385 KB [ 1422.853842] mlx4_core 0000:82:00.0: Clear int @ f0058, BAR 0 [ 1422.855140] mlx4_core 0000:82:00.0: Mapped 26 chunks/6168 KB for FW. [ 1425.518005] mlx4_core 0000:82:00.0: BlueFlame available (reg size 512, regs/page 8) [ 1425.518117] mlx4_core 0000:82:00.0: Base MM extensions: flags 00104cc0, rsvd L_Key 00008000 [ 1425.518119] mlx4_core 0000:82:00.0: Max ICM size 4294967296 MB [ 1425.518120] mlx4_core 0000:82:00.0: Max QPs: 16777216, reserved QPs: 64, entry size: 256 [ 1425.518122] mlx4_core 0000:82:00.0: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128 [ 1425.518124] mlx4_core 0000:82:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 128 [ 1425.518125] mlx4_core 0000:82:00.0: Max EQs: 32, reserved EQs: 28, entry size: 128 [ 1425.518127] mlx4_core 0000:82:00.0: reserved MPTs: 256, reserved MTTs: 32 [ 1425.518128] mlx4_core 0000:82:00.0: Max PDs: 131072, reserved PDs: 4, reserved UARs: 7 [ 1425.518130] mlx4_core 0000:82:00.0: Max QP/MCG: 131072, reserved MGMs: 0 [ 1425.518131] mlx4_core 0000:82:00.0: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384 [ 1425.518133] mlx4_core 0000:82:00.0: Local CA ACK delay: 15, max MTU: 4096, port width cap: 2 [ 1425.518134] mlx4_core 0000:82:00.0: Max SQ desc size: 1008, max SQ S/G: 62 [ 1425.518136] mlx4_core 0000:82:00.0: Max RQ desc size: 512, max RQ S/G: 32 [ 1425.518137] mlx4_core 0000:82:00.0: Max GSO size: 131072 [ 1425.518138] mlx4_core 0000:82:00.0: Max counters: 256 [ 1425.518140] mlx4_core 0000:82:00.0: Max RSS Table size: 256 [ 1425.518141] mlx4_core 0000:82:00.0: DEV_CAP flags: [ 1425.518142] mlx4_core 0000:82:00.0: RC transport [ 1425.518144] mlx4_core 0000:82:00.0: UC transport [ 1425.518145] mlx4_core 0000:82:00.0: UD transport [ 1425.518146] mlx4_core 0000:82:00.0: XRC transport [ 1425.518147] mlx4_core 0000:82:00.0: FCoIB support [ 1425.518148] mlx4_core 0000:82:00.0: SRQ support [ 1425.518150] mlx4_core 0000:82:00.0: IPoIB checksum offload [ 1425.518151] mlx4_core 0000:82:00.0: P_Key violation counter [ 1425.518152] mlx4_core 0000:82:00.0: Q_Key violation counter [ 1425.518154] mlx4_core 0000:82:00.0: DPDP [ 1425.518155] mlx4_core 0000:82:00.0: Big LSO headers [ 1425.518156] mlx4_core 0000:82:00.0: APM support [ 1425.518158] mlx4_core 0000:82:00.0: Atomic ops support [ 1425.518159] mlx4_core 0000:82:00.0: Address vector port checking support [ 1425.518160] mlx4_core 0000:82:00.0: UD multicast support [ 1425.518161] mlx4_core 0000:82:00.0: Router support [ 1425.518163] mlx4_core 0000:82:00.0: IBoE support [ 1425.518164] mlx4_core 0000:82:00.0: Unicast loopback support [ 1425.518165] mlx4_core 0000:82:00.0: FCS header control [ 1425.518167] mlx4_core 0000:82:00.0: UDP RSS support [ 1425.518168] mlx4_core 0000:82:00.0: Unicast VEP steering support [ 1425.518169] mlx4_core 0000:82:00.0: Multicast VEP steering support [ 1425.518170] mlx4_core 0000:82:00.0: Counters support [ 1425.518172] mlx4_core 0000:82:00.0: Port management change event support [ 1425.518173] mlx4_core 0000:82:00.0: 64 byte EQE support [ 1425.518175] mlx4_core 0000:82:00.0: 64 byte CQE support [ 1425.518176] mlx4_core 0000:82:00.0: RSS support [ 1425.518177] mlx4_core 0000:82:00.0: RSS Toeplitz Hash Function support [ 1425.518178] mlx4_core 0000:82:00.0: RSS XOR Hash Function support [ 1425.518180] mlx4_core 0000:82:00.0: Device manage flow steering support [ 1425.518182] mlx4_core 0000:82:00.0: 64B EQEs/CQEs supported by the device but not enabled [ 1425.518184] mlx4_core 0000:82:00.0: Steering mode is: B0 steering, oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10 [ 1425.518190] mlx4_core 0000:82:00.0: profile[ 0] ( CMPT): 2^26 entries @ 0x 0, size 0x 100000000 [ 1425.518193] mlx4_core 0000:82:00.0: profile[ 1] (RDMARC): 2^22 entries @ 0x 100000000, size 0x 8000000 [ 1425.518195] mlx4_core 0000:82:00.0: profile[ 2] ( MTT): 2^24 entries @ 0x 108000000, size 0x 8000000 [ 1425.518197] mlx4_core 0000:82:00.0: profile[ 3] ( QP): 2^18 entries @ 0x 110000000, size 0x 4000000 [ 1425.518199] mlx4_core 0000:82:00.0: profile[ 4] ( DMPT): 2^19 entries @ 0x 114000000, size 0x 2000000 [ 1425.518200] mlx4_core 0000:82:00.0: profile[ 5] ( ALTC): 2^18 entries @ 0x 116000000, size 0x 1000000 [ 1425.518202] mlx4_core 0000:82:00.0: profile[ 6] ( SRQ): 2^16 entries @ 0x 117000000, size 0x 800000 [ 1425.518204] mlx4_core 0000:82:00.0: profile[ 7] ( CQ): 2^16 entries @ 0x 117800000, size 0x 800000 [ 1425.518206] mlx4_core 0000:82:00.0: profile[ 8] ( MCG): 2^13 entries @ 0x 118000000, size 0x 800000 [ 1425.518208] mlx4_core 0000:82:00.0: profile[ 9] ( AUXC): 2^18 entries @ 0x 118800000, size 0x 40000 [ 1425.518210] mlx4_core 0000:82:00.0: profile[10] ( EQ): 2^10 entries @ 0x 118840000, size 0x 20000 [ 1425.518212] mlx4_core 0000:82:00.0: HCA context memory: reserving 4596096 KB [ 1425.518228] mlx4_core 0000:82:00.0: 4596096 KB of HCA context requires 9068 KB aux memory. [ 1425.554670] mlx4_core 0000:82:00.0: Mapped 39 chunks/9068 KB for ICM aux. [ 1425.556059] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 0 for ICM. [ 1425.557429] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 40000000 for ICM. [ 1425.558795] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 80000000 for ICM. [ 1425.559144] mlx4_core 0000:82:00.0: Mapped 1 chunks/64 KB at c0000000 for ICM. [ 1425.559827] mlx4_core 0000:82:00.0: Mapped 1 chunks/128 KB at 118840000 for ICM. [ 1425.561182] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 108000000 for ICM. [ 1425.562580] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 114000000 for ICM. [ 1425.563932] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 110000000 for ICM. [ 1425.565306] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118800000 for ICM. [ 1425.566666] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 116000000 for ICM. [ 1425.568018] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 100000000 for ICM. [ 1425.569390] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 117800000 for ICM. [ 1425.570754] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 117000000 for ICM. [ 1425.572109] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118000000 for ICM. [ 1425.573454] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118040000 for ICM. [ 1425.574822] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118080000 for ICM. [ 1425.576175] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1180c0000 for ICM. [ 1425.577533] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118100000 for ICM. [ 1425.578899] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118140000 for ICM. [ 1425.580246] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118180000 for ICM. [ 1425.581593] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1181c0000 for ICM. [ 1425.582929] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118200000 for ICM. [ 1425.584289] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118240000 for ICM. [ 1425.585645] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118280000 for ICM. [ 1425.587011] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1182c0000 for ICM. [ 1425.588376] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118300000 for ICM. [ 1425.589740] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118340000 for ICM. [ 1425.591098] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118380000 for ICM. [ 1425.592460] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1183c0000 for ICM. [ 1425.593820] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118400000 for ICM. [ 1425.595170] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118440000 for ICM. [ 1425.596532] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118480000 for ICM. [ 1425.597912] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1184c0000 for ICM. [ 1425.599257] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118500000 for ICM. [ 1425.600610] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118540000 for ICM. [ 1425.601967] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118580000 for ICM. [ 1425.603314] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1185c0000 for ICM. [ 1425.604674] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118600000 for ICM. [ 1425.606018] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118640000 for ICM. [ 1425.607374] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118680000 for ICM. [ 1425.608741] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1186c0000 for ICM. [ 1425.610095] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118700000 for ICM. [ 1425.611456] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118740000 for ICM. [ 1425.612815] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 118780000 for ICM. [ 1425.614165] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 1187c0000 for ICM. [ 1426.193222] mlx4_core 0000:82:00.0: Started init_resource_tracker: 80 slaves [ 1426.193692] mlx4_core 0000:82:00.0: irq 178 for MSI/MSI-X [ 1426.193705] mlx4_core 0000:82:00.0: irq 179 for MSI/MSI-X [ 1426.200029] mlx4_core 0000:82:00.0: NOP command IRQ test passed [ 1426.200605] mlx4_core: Initializing 0000:05:00.1 [ 1426.200618] mlx4_core 0000:05:00.1: enabling device (0000 -> 0002) [ 1426.200654] mlx4_core 0000:05:00.1: Skipping virtual function:1 [ 1426.200674] mlx4_core: Initializing 0000:82:00.1 [ 1426.200685] mlx4_core 0000:82:00.1: enabling device (0000 -> 0002) [ 1426.200711] mlx4_core 0000:82:00.1: Skipping virtual function:1 [ 1438.961741] pci-stub 0000:82:00.1: claimed by stub [ 1438.967250] pci-stub 0000:82:00.1: enabling device (0000 -> 0002) [ 1438.969739] mlx4_core 0000:82:00.0: FLR event for slave: 1 [ 1438.969775] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr [ 1438.969776] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 [ 1439.070931] assign device 0:82:0.1 [ 1439.073469] mlx4_core 0000:82:00.0: FLR event for slave: 1 [ 1439.073526] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr [ 1439.073529] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 [ 1439.226723] mlx4_core 0000:82:00.0: Received reset from slave:1 [ 1439.239542] pci-stub 0000:82:00.1: irq 180 for MSI/MSI-X [ 1439.272548] pci-stub 0000:82:00.1: irq 180 for MSI/MSI-X [ 1439.272559] pci-stub 0000:82:00.1: irq 181 for MSI/MSI-X [ 1449.381120] mlx4_core 0000:82:00.0: Received reset from slave:1 mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) mlx4_core: Initializing 0000:00:0b.0 mlx4_core 0000:00:0b.0: enabling device (0000 -> 0002) mlx4_core 0000:00:0b.0: Detected virtual function - running in slave mode mlx4_core 0000:00:0b.0: Sending reset mlx4_core 0000:00:0b.0: Sending vhcr0 mlx4_core 0000:00:0b.0: 64B EQEs/CQEs supported by the device but not enabled mlx4_core 0000:00:0b.0: HCA minimum page size:512 mlx4_core 0000:00:0b.0: irq 48 for MSI/MSI-X mlx4_core 0000:00:0b.0: irq 49 for MSI/MSI-X mlx4_core 0000:00:0b.0: failed execution of VHCR_POST commandopcode 0x31 mlx4_core 0000:00:0b.0: NOP command failed to generate MSI-X interrupt IRQ 49). mlx4_core 0000:00:0b.0: Trying again without MSI-X. mlx4_core: probe of 0000:00:0b.0 failed with error -16 Mathis 2013/1/29 Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>: > On Tuesday 29 January 2013 15:15, Mathis GAVILLON wrote: >> In fact, one boxe wasn't check in IOMMU submenu (Support for Interrupt >> Remapping (EXPERIMENTAL)). I've recompiled the kernel of the host and >> guest (is it necessary for this last one ?). But the error message is >> already present. Just below, new dmeg log from host and guest O/S >> (mlx4_core loaded with options num_vfs=1 probe_vf=1 debug_level=1 on >> host and debug_level=1 on guest) : >> > Please do the following: > 1. Check the Support for Interrupt Remapping (EXPERIMENTAL) box and recompile/reinstall. > 2. set probe_vf=0, so that no vfs will run on the Hypervisor. That way you will > see if any VF comes up without the "noise" in /var/log/messages in the Hypervisor generated > by having one of the VFs run on the Hypervisor. > > 3. I noticed that you are enabling only a single VF, and that you are having the > VF come up first on the Hypervisor, then moving it to run on a guest. > It is better not to do things that way. Best to have it come up cleanly once on > the guest. (the FLR event in your log results from this move. The VF gets reset). > > If you must have a VF running on the Hypervisor, use "num_vfs=2 probe_vf=1". > You will then see Devices ...xx.0, ...xx.1 and ...xx.2 in lspci. The first VF > (...xx.1) will be initialized on the Hypervisor. Attach ...xx.2 to the VM. > > -Jack -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <CAOoYFkErXNxSBrQC_spMwcEiVmOpkzqQwuTQPOwO70bigsgYkQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <CAOoYFkErXNxSBrQC_spMwcEiVmOpkzqQwuTQPOwO70bigsgYkQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-01-29 18:36 ` Jack Morgenstein 0 siblings, 0 replies; 16+ messages in thread From: Jack Morgenstein @ 2013-01-29 18:36 UTC (permalink / raw) To: Mathis GAVILLON, Or Gerlitz; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA On Tuesday 29 January 2013 17:18, Mathis GAVILLON wrote: > Support for interrupt remapping should be now available (kernel has > been recompiled with but is there any way to check if it's available > ?). > I tried to do what you say in step 2 (num_vfs=1 and probe_vf=0) but it > doesn't change anything. I also tried num_vfs=2 and probe_vf=1 but > nothing is better. > Nevertheless, a strange information message appear in shell when I > boot my guest : > > Message from syslogd@localhost at Jan 29 15:51:49 ... > kernel:Disabling IRQ #16 > Or, Mathis, This now looks like the problem that we had with a qemu change. qemu-kvm-0.12.1.2-2.348.el6 (or possibly earlier) was changed from the previous qemu version so that the pci header for the guest was not updated to indicate MSI_X enabled when the guest called pci_enable_msix() -- rather, a "lazy update" policy was introduced which did not update the flag in the Hypervisor until the guest called request_irq(). This caused problems for some vendor's network sriov drivers, and RH implemented instead a hybrid policy, compatible with their previous policy, which did not break any sriov drivers. The updated qemu module for RH is: qemu-kvm-0.12.1.2-2.349.el6 To see which version of qemu you have installed, just enter the following command: rpm -qvia | grep qemu I found a web site which has the rpm's of interest (for fedora core 18): http://rpmfind.net/linux/rpm2html/search.php?query=qemu-kvm You might try using the following rpm (built Jan 16): ftp://rpmfind.net/linux/fedora/linux/updates/18/x86_64/qemu-kvm-1.2.2-2.fc18.x86_64.rpm (which probably has the fix) If that does not work, you might try: ftp://rpmfind.net/linux/fedora/linux/development/rawhide/x86_64/os/Packages/q/qemu-kvm-1.3.0-5.fc19.x86_64.rpm -Jack > I put dmesg log from step 2 (host and guest) again : > > [ 1419.382452] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) > [ 1419.382454] mlx4_core: Initializing 0000:05:00.0 > [ 1419.382653] mlx4_core 0000:05:00.0: Enabling SR-IOV with 1 VFs > [ 1419.483687] pci 0000:05:00.1: [15b3:1002] type 00 class 0x0c0600 > [ 1419.484427] mlx4_core 0000:05:00.0: Running in master mode > [ 1420.486608] mlx4_core 0000:05:00.0: FW version 2.9.1200 (cmd intf > rev 3), max commands 16 > [ 1420.486610] mlx4_core 0000:05:00.0: Catastrophic error buffer at > 0x1f020, size 0x10, BAR 0 > [ 1420.486612] mlx4_core 0000:05:00.0: Communication vector bar:2 offset:0x800 > [ 1420.486613] mlx4_core 0000:05:00.0: FW size 385 KB > [ 1420.486615] mlx4_core 0000:05:00.0: Clear int @ f0058, BAR 0 > [ 1420.488474] mlx4_core 0000:05:00.0: Mapped 26 chunks/6168 KB for FW. > [ 1421.155871] mlx4_core 0000:05:00.0: BlueFlame available (reg size > 512, regs/page 8) > [ 1421.156035] mlx4_core 0000:05:00.0: Base MM extensions: flags > 00000cc0, rsvd L_Key 00000500 > [ 1421.156036] mlx4_core 0000:05:00.0: Max ICM size 4294967296 MB > [ 1421.156038] mlx4_core 0000:05:00.0: Max QPs: 16777216, reserved > QPs: 64, entry size: 256 > [ 1421.156040] mlx4_core 0000:05:00.0: Max SRQs: 16777216, reserved > SRQs: 64, entry size: 128 > [ 1421.156041] mlx4_core 0000:05:00.0: Max CQs: 16777216, reserved > CQs: 128, entry size: 128 > [ 1421.156043] mlx4_core 0000:05:00.0: Max EQs: 32, reserved EQs: 28, > entry size: 128 > [ 1421.156045] mlx4_core 0000:05:00.0: reserved MPTs: 16, reserved MTTs: 16 > [ 1421.156046] mlx4_core 0000:05:00.0: Max PDs: 131072, reserved PDs: > 4, reserved UARs: 7 > [ 1421.156048] mlx4_core 0000:05:00.0: Max QP/MCG: 131072, reserved MGMs: 0 > [ 1421.156049] mlx4_core 0000:05:00.0: Max CQEs: 4194304, max WQEs: > 16384, max SRQ WQEs: 16384 > [ 1421.156051] mlx4_core 0000:05:00.0: Local CA ACK delay: 15, max > MTU: 4096, port width cap: 3 > [ 1421.156052] mlx4_core 0000:05:00.0: Max SQ desc size: 1008, max SQ S/G: 62 > [ 1421.156054] mlx4_core 0000:05:00.0: Max RQ desc size: 512, max RQ S/G: 32 > [ 1421.156055] mlx4_core 0000:05:00.0: Max GSO size: 131072 > [ 1421.156056] mlx4_core 0000:05:00.0: Max counters: 256 > [ 1421.156058] mlx4_core 0000:05:00.0: Max RSS Table size: 256 > [ 1421.156059] mlx4_core 0000:05:00.0: DEV_CAP flags: > [ 1421.156060] mlx4_core 0000:05:00.0: RC transport > [ 1421.156062] mlx4_core 0000:05:00.0: UC transport > [ 1421.156063] mlx4_core 0000:05:00.0: UD transport > [ 1421.156064] mlx4_core 0000:05:00.0: XRC transport > [ 1421.156065] mlx4_core 0000:05:00.0: FCoIB support > [ 1421.156066] mlx4_core 0000:05:00.0: SRQ support > [ 1421.156068] mlx4_core 0000:05:00.0: IPoIB checksum offload > [ 1421.156069] mlx4_core 0000:05:00.0: P_Key violation counter > [ 1421.156070] mlx4_core 0000:05:00.0: Q_Key violation counter > [ 1421.156071] mlx4_core 0000:05:00.0: DPDP > [ 1421.156073] mlx4_core 0000:05:00.0: Big LSO headers > [ 1421.156074] mlx4_core 0000:05:00.0: APM support > [ 1421.156075] mlx4_core 0000:05:00.0: Atomic ops support > [ 1421.156077] mlx4_core 0000:05:00.0: Address vector port checking support > [ 1421.156078] mlx4_core 0000:05:00.0: UD multicast support > [ 1421.156079] mlx4_core 0000:05:00.0: Router support > [ 1421.156080] mlx4_core 0000:05:00.0: IBoE support > [ 1421.156082] mlx4_core 0000:05:00.0: Unicast loopback support > [ 1421.156083] mlx4_core 0000:05:00.0: UDP RSS support > [ 1421.156084] mlx4_core 0000:05:00.0: Unicast VEP steering support > [ 1421.156085] mlx4_core 0000:05:00.0: Multicast VEP steering support > [ 1421.156087] mlx4_core 0000:05:00.0: Counters support > [ 1421.156088] mlx4_core 0000:05:00.0: RSS support > [ 1421.156090] mlx4_core 0000:05:00.0: RSS Toeplitz Hash Function support > [ 1421.156091] mlx4_core 0000:05:00.0: RSS XOR Hash Function support > [ 1421.156093] mlx4_core 0000:05:00.0: Steering mode is: B0 steering, > oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10 > [ 1421.156099] mlx4_core 0000:05:00.0: profile[ 0] ( CMPT): 2^26 > entries @ 0x 0, size 0x 100000000 > [ 1421.156101] mlx4_core 0000:05:00.0: profile[ 1] (RDMARC): 2^22 > entries @ 0x 100000000, size 0x 8000000 > [ 1421.156103] mlx4_core 0000:05:00.0: profile[ 2] ( MTT): 2^24 > entries @ 0x 108000000, size 0x 8000000 > [ 1421.156105] mlx4_core 0000:05:00.0: profile[ 3] ( QP): 2^18 > entries @ 0x 110000000, size 0x 4000000 > [ 1421.156107] mlx4_core 0000:05:00.0: profile[ 4] ( DMPT): 2^19 > entries @ 0x 114000000, size 0x 2000000 > [ 1421.156109] mlx4_core 0000:05:00.0: profile[ 5] ( ALTC): 2^18 > entries @ 0x 116000000, size 0x 1000000 > [ 1421.156111] mlx4_core 0000:05:00.0: profile[ 6] ( SRQ): 2^16 > entries @ 0x 117000000, size 0x 800000 > [ 1421.156113] mlx4_core 0000:05:00.0: profile[ 7] ( CQ): 2^16 > entries @ 0x 117800000, size 0x 800000 > [ 1421.156115] mlx4_core 0000:05:00.0: profile[ 8] ( MCG): 2^13 > entries @ 0x 118000000, size 0x 800000 > [ 1421.156117] mlx4_core 0000:05:00.0: profile[ 9] ( AUXC): 2^18 > entries @ 0x 118800000, size 0x 40000 > [ 1421.156119] mlx4_core 0000:05:00.0: profile[10] ( EQ): 2^10 > entries @ 0x 118840000, size 0x 20000 > [ 1421.156120] mlx4_core 0000:05:00.0: HCA context memory: reserving 4596096 KB > [ 1421.156136] mlx4_core 0000:05:00.0: 4596096 KB of HCA context > requires 9068 KB aux memory. > [ 1421.185505] mlx4_core 0000:05:00.0: Mapped 39 chunks/9068 KB for ICM aux. > [ 1421.186824] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 0 for ICM. > [ 1421.188122] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 40000000 for ICM. > [ 1421.189418] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 80000000 for ICM. > [ 1421.189754] mlx4_core 0000:05:00.0: Mapped 1 chunks/64 KB at > c0000000 for ICM. > [ 1421.190400] mlx4_core 0000:05:00.0: Mapped 1 chunks/128 KB at > 118840000 for ICM. > [ 1421.191677] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 108000000 for ICM. > [ 1421.193001] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 114000000 for ICM. > [ 1421.194279] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 110000000 for ICM. > [ 1421.195563] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118800000 for ICM. > [ 1421.196841] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 116000000 for ICM. > [ 1421.198127] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 100000000 for ICM. > [ 1421.199405] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 117800000 for ICM. > [ 1421.200684] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 117000000 for ICM. > [ 1421.201964] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118000000 for ICM. > [ 1421.203240] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118040000 for ICM. > [ 1421.204517] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118080000 for ICM. > [ 1421.205802] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1180c0000 for ICM. > [ 1421.207079] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118100000 for ICM. > [ 1421.208356] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118140000 for ICM. > [ 1421.209633] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118180000 for ICM. > [ 1421.210912] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1181c0000 for ICM. > [ 1421.212191] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118200000 for ICM. > [ 1421.213465] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118240000 for ICM. > [ 1421.214742] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118280000 for ICM. > [ 1421.216024] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1182c0000 for ICM. > [ 1421.217303] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118300000 for ICM. > [ 1421.218580] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118340000 for ICM. > [ 1421.219855] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118380000 for ICM. > [ 1421.221132] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1183c0000 for ICM. > [ 1421.222407] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118400000 for ICM. > [ 1421.223685] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118440000 for ICM. > [ 1421.224963] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118480000 for ICM. > [ 1421.226245] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1184c0000 for ICM. > [ 1421.227522] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118500000 for ICM. > [ 1421.228801] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118540000 for ICM. > [ 1421.230079] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118580000 for ICM. > [ 1421.231355] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1185c0000 for ICM. > [ 1421.232633] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118600000 for ICM. > [ 1421.233909] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118640000 for ICM. > [ 1421.235186] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118680000 for ICM. > [ 1421.236470] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1186c0000 for ICM. > [ 1421.237748] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118700000 for ICM. > [ 1421.239026] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118740000 for ICM. > [ 1421.240303] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 118780000 for ICM. > [ 1421.241581] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at > 1187c0000 for ICM. > [ 1421.744993] mlx4_core 0000:05:00.0: Started init_resource_tracker: 80 slaves > [ 1421.745454] mlx4_core 0000:05:00.0: irq 176 for MSI/MSI-X > [ 1421.745466] mlx4_core 0000:05:00.0: irq 177 for MSI/MSI-X > [ 1421.750286] mlx4_core 0000:05:00.0: NOP command IRQ test passed > [ 1421.750911] mlx4_core: Initializing 0000:82:00.0 > [ 1421.751006] mlx4_core 0000:82:00.0: Enabling SR-IOV with 1 VFs > [ 1421.851346] pci 0000:82:00.1: [15b3:1004] type 00 class 0x028000 > [ 1421.852051] mlx4_core 0000:82:00.0: Running in master mode > [ 1422.853834] mlx4_core 0000:82:00.0: FW version 2.11.500 (cmd intf > rev 3), max commands 16 > [ 1422.853837] mlx4_core 0000:82:00.0: Catastrophic error buffer at > 0x1f020, size 0x10, BAR 0 > [ 1422.853839] mlx4_core 0000:82:00.0: Communication vector bar:2 offset:0x800 > [ 1422.853840] mlx4_core 0000:82:00.0: FW size 385 KB > [ 1422.853842] mlx4_core 0000:82:00.0: Clear int @ f0058, BAR 0 > [ 1422.855140] mlx4_core 0000:82:00.0: Mapped 26 chunks/6168 KB for FW. > [ 1425.518005] mlx4_core 0000:82:00.0: BlueFlame available (reg size > 512, regs/page 8) > [ 1425.518117] mlx4_core 0000:82:00.0: Base MM extensions: flags > 00104cc0, rsvd L_Key 00008000 > [ 1425.518119] mlx4_core 0000:82:00.0: Max ICM size 4294967296 MB > [ 1425.518120] mlx4_core 0000:82:00.0: Max QPs: 16777216, reserved > QPs: 64, entry size: 256 > [ 1425.518122] mlx4_core 0000:82:00.0: Max SRQs: 16777216, reserved > SRQs: 64, entry size: 128 > [ 1425.518124] mlx4_core 0000:82:00.0: Max CQs: 16777216, reserved > CQs: 128, entry size: 128 > [ 1425.518125] mlx4_core 0000:82:00.0: Max EQs: 32, reserved EQs: 28, > entry size: 128 > [ 1425.518127] mlx4_core 0000:82:00.0: reserved MPTs: 256, reserved MTTs: 32 > [ 1425.518128] mlx4_core 0000:82:00.0: Max PDs: 131072, reserved PDs: > 4, reserved UARs: 7 > [ 1425.518130] mlx4_core 0000:82:00.0: Max QP/MCG: 131072, reserved MGMs: 0 > [ 1425.518131] mlx4_core 0000:82:00.0: Max CQEs: 4194304, max WQEs: > 16384, max SRQ WQEs: 16384 > [ 1425.518133] mlx4_core 0000:82:00.0: Local CA ACK delay: 15, max > MTU: 4096, port width cap: 2 > [ 1425.518134] mlx4_core 0000:82:00.0: Max SQ desc size: 1008, max SQ S/G: 62 > [ 1425.518136] mlx4_core 0000:82:00.0: Max RQ desc size: 512, max RQ S/G: 32 > [ 1425.518137] mlx4_core 0000:82:00.0: Max GSO size: 131072 > [ 1425.518138] mlx4_core 0000:82:00.0: Max counters: 256 > [ 1425.518140] mlx4_core 0000:82:00.0: Max RSS Table size: 256 > [ 1425.518141] mlx4_core 0000:82:00.0: DEV_CAP flags: > [ 1425.518142] mlx4_core 0000:82:00.0: RC transport > [ 1425.518144] mlx4_core 0000:82:00.0: UC transport > [ 1425.518145] mlx4_core 0000:82:00.0: UD transport > [ 1425.518146] mlx4_core 0000:82:00.0: XRC transport > [ 1425.518147] mlx4_core 0000:82:00.0: FCoIB support > [ 1425.518148] mlx4_core 0000:82:00.0: SRQ support > [ 1425.518150] mlx4_core 0000:82:00.0: IPoIB checksum offload > [ 1425.518151] mlx4_core 0000:82:00.0: P_Key violation counter > [ 1425.518152] mlx4_core 0000:82:00.0: Q_Key violation counter > [ 1425.518154] mlx4_core 0000:82:00.0: DPDP > [ 1425.518155] mlx4_core 0000:82:00.0: Big LSO headers > [ 1425.518156] mlx4_core 0000:82:00.0: APM support > [ 1425.518158] mlx4_core 0000:82:00.0: Atomic ops support > [ 1425.518159] mlx4_core 0000:82:00.0: Address vector port checking support > [ 1425.518160] mlx4_core 0000:82:00.0: UD multicast support > [ 1425.518161] mlx4_core 0000:82:00.0: Router support > [ 1425.518163] mlx4_core 0000:82:00.0: IBoE support > [ 1425.518164] mlx4_core 0000:82:00.0: Unicast loopback support > [ 1425.518165] mlx4_core 0000:82:00.0: FCS header control > [ 1425.518167] mlx4_core 0000:82:00.0: UDP RSS support > [ 1425.518168] mlx4_core 0000:82:00.0: Unicast VEP steering support > [ 1425.518169] mlx4_core 0000:82:00.0: Multicast VEP steering support > [ 1425.518170] mlx4_core 0000:82:00.0: Counters support > [ 1425.518172] mlx4_core 0000:82:00.0: Port management change event support > [ 1425.518173] mlx4_core 0000:82:00.0: 64 byte EQE support > [ 1425.518175] mlx4_core 0000:82:00.0: 64 byte CQE support > [ 1425.518176] mlx4_core 0000:82:00.0: RSS support > [ 1425.518177] mlx4_core 0000:82:00.0: RSS Toeplitz Hash Function support > [ 1425.518178] mlx4_core 0000:82:00.0: RSS XOR Hash Function support > [ 1425.518180] mlx4_core 0000:82:00.0: Device manage flow steering support > [ 1425.518182] mlx4_core 0000:82:00.0: 64B EQEs/CQEs supported by the > device but not enabled > [ 1425.518184] mlx4_core 0000:82:00.0: Steering mode is: B0 steering, > oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10 > [ 1425.518190] mlx4_core 0000:82:00.0: profile[ 0] ( CMPT): 2^26 > entries @ 0x 0, size 0x 100000000 > [ 1425.518193] mlx4_core 0000:82:00.0: profile[ 1] (RDMARC): 2^22 > entries @ 0x 100000000, size 0x 8000000 > [ 1425.518195] mlx4_core 0000:82:00.0: profile[ 2] ( MTT): 2^24 > entries @ 0x 108000000, size 0x 8000000 > [ 1425.518197] mlx4_core 0000:82:00.0: profile[ 3] ( QP): 2^18 > entries @ 0x 110000000, size 0x 4000000 > [ 1425.518199] mlx4_core 0000:82:00.0: profile[ 4] ( DMPT): 2^19 > entries @ 0x 114000000, size 0x 2000000 > [ 1425.518200] mlx4_core 0000:82:00.0: profile[ 5] ( ALTC): 2^18 > entries @ 0x 116000000, size 0x 1000000 > [ 1425.518202] mlx4_core 0000:82:00.0: profile[ 6] ( SRQ): 2^16 > entries @ 0x 117000000, size 0x 800000 > [ 1425.518204] mlx4_core 0000:82:00.0: profile[ 7] ( CQ): 2^16 > entries @ 0x 117800000, size 0x 800000 > [ 1425.518206] mlx4_core 0000:82:00.0: profile[ 8] ( MCG): 2^13 > entries @ 0x 118000000, size 0x 800000 > [ 1425.518208] mlx4_core 0000:82:00.0: profile[ 9] ( AUXC): 2^18 > entries @ 0x 118800000, size 0x 40000 > [ 1425.518210] mlx4_core 0000:82:00.0: profile[10] ( EQ): 2^10 > entries @ 0x 118840000, size 0x 20000 > [ 1425.518212] mlx4_core 0000:82:00.0: HCA context memory: reserving 4596096 KB > [ 1425.518228] mlx4_core 0000:82:00.0: 4596096 KB of HCA context > requires 9068 KB aux memory. > [ 1425.554670] mlx4_core 0000:82:00.0: Mapped 39 chunks/9068 KB for ICM aux. > [ 1425.556059] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 0 for ICM. > [ 1425.557429] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 40000000 for ICM. > [ 1425.558795] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 80000000 for ICM. > [ 1425.559144] mlx4_core 0000:82:00.0: Mapped 1 chunks/64 KB at > c0000000 for ICM. > [ 1425.559827] mlx4_core 0000:82:00.0: Mapped 1 chunks/128 KB at > 118840000 for ICM. > [ 1425.561182] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 108000000 for ICM. > [ 1425.562580] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 114000000 for ICM. > [ 1425.563932] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 110000000 for ICM. > [ 1425.565306] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118800000 for ICM. > [ 1425.566666] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 116000000 for ICM. > [ 1425.568018] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 100000000 for ICM. > [ 1425.569390] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 117800000 for ICM. > [ 1425.570754] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 117000000 for ICM. > [ 1425.572109] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118000000 for ICM. > [ 1425.573454] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118040000 for ICM. > [ 1425.574822] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118080000 for ICM. > [ 1425.576175] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1180c0000 for ICM. > [ 1425.577533] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118100000 for ICM. > [ 1425.578899] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118140000 for ICM. > [ 1425.580246] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118180000 for ICM. > [ 1425.581593] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1181c0000 for ICM. > [ 1425.582929] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118200000 for ICM. > [ 1425.584289] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118240000 for ICM. > [ 1425.585645] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118280000 for ICM. > [ 1425.587011] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1182c0000 for ICM. > [ 1425.588376] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118300000 for ICM. > [ 1425.589740] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118340000 for ICM. > [ 1425.591098] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118380000 for ICM. > [ 1425.592460] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1183c0000 for ICM. > [ 1425.593820] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118400000 for ICM. > [ 1425.595170] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118440000 for ICM. > [ 1425.596532] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118480000 for ICM. > [ 1425.597912] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1184c0000 for ICM. > [ 1425.599257] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118500000 for ICM. > [ 1425.600610] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118540000 for ICM. > [ 1425.601967] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118580000 for ICM. > [ 1425.603314] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1185c0000 for ICM. > [ 1425.604674] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118600000 for ICM. > [ 1425.606018] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118640000 for ICM. > [ 1425.607374] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118680000 for ICM. > [ 1425.608741] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1186c0000 for ICM. > [ 1425.610095] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118700000 for ICM. > [ 1425.611456] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118740000 for ICM. > [ 1425.612815] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 118780000 for ICM. > [ 1425.614165] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at > 1187c0000 for ICM. > [ 1426.193222] mlx4_core 0000:82:00.0: Started init_resource_tracker: 80 slaves > [ 1426.193692] mlx4_core 0000:82:00.0: irq 178 for MSI/MSI-X > [ 1426.193705] mlx4_core 0000:82:00.0: irq 179 for MSI/MSI-X > [ 1426.200029] mlx4_core 0000:82:00.0: NOP command IRQ test passed > [ 1426.200605] mlx4_core: Initializing 0000:05:00.1 > [ 1426.200618] mlx4_core 0000:05:00.1: enabling device (0000 -> 0002) > [ 1426.200654] mlx4_core 0000:05:00.1: Skipping virtual function:1 > [ 1426.200674] mlx4_core: Initializing 0000:82:00.1 > [ 1426.200685] mlx4_core 0000:82:00.1: enabling device (0000 -> 0002) > [ 1426.200711] mlx4_core 0000:82:00.1: Skipping virtual function:1 > [ 1438.961741] pci-stub 0000:82:00.1: claimed by stub > [ 1438.967250] pci-stub 0000:82:00.1: enabling device (0000 -> 0002) > [ 1438.969739] mlx4_core 0000:82:00.0: FLR event for slave: 1 > [ 1438.969775] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr > [ 1438.969776] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 > [ 1439.070931] assign device 0:82:0.1 > [ 1439.073469] mlx4_core 0000:82:00.0: FLR event for slave: 1 > [ 1439.073526] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr > [ 1439.073529] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 > [ 1439.226723] mlx4_core 0000:82:00.0: Received reset from slave:1 > [ 1439.239542] pci-stub 0000:82:00.1: irq 180 for MSI/MSI-X > [ 1439.272548] pci-stub 0000:82:00.1: irq 180 for MSI/MSI-X > [ 1439.272559] pci-stub 0000:82:00.1: irq 181 for MSI/MSI-X > [ 1449.381120] mlx4_core 0000:82:00.0: Received reset from slave:1 > > > mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) > mlx4_core: Initializing 0000:00:0b.0 > mlx4_core 0000:00:0b.0: enabling device (0000 -> 0002) > mlx4_core 0000:00:0b.0: Detected virtual function - running in slave mode > mlx4_core 0000:00:0b.0: Sending reset > mlx4_core 0000:00:0b.0: Sending vhcr0 > mlx4_core 0000:00:0b.0: 64B EQEs/CQEs supported by the device but not enabled > mlx4_core 0000:00:0b.0: HCA minimum page size:512 > mlx4_core 0000:00:0b.0: irq 48 for MSI/MSI-X > mlx4_core 0000:00:0b.0: irq 49 for MSI/MSI-X > mlx4_core 0000:00:0b.0: failed execution of VHCR_POST commandopcode 0x31 > mlx4_core 0000:00:0b.0: NOP command failed to generate MSI-X interrupt IRQ 49). > mlx4_core 0000:00:0b.0: Trying again without MSI-X. > mlx4_core: probe of 0000:00:0b.0 failed with error -16 > > Mathis > > 2013/1/29 Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>: > > On Tuesday 29 January 2013 15:15, Mathis GAVILLON wrote: > >> In fact, one boxe wasn't check in IOMMU submenu (Support for Interrupt > >> Remapping (EXPERIMENTAL)). I've recompiled the kernel of the host and > >> guest (is it necessary for this last one ?). But the error message is > >> already present. Just below, new dmeg log from host and guest O/S > >> (mlx4_core loaded with options num_vfs=1 probe_vf=1 debug_level=1 on > >> host and debug_level=1 on guest) : > >> > > Please do the following: > > 1. Check the Support for Interrupt Remapping (EXPERIMENTAL) box and recompile/reinstall. > > 2. set probe_vf=0, so that no vfs will run on the Hypervisor. That way you will > > see if any VF comes up without the "noise" in /var/log/messages in the Hypervisor generated > > by having one of the VFs run on the Hypervisor. > > > > 3. I noticed that you are enabling only a single VF, and that you are having the > > VF come up first on the Hypervisor, then moving it to run on a guest. > > It is better not to do things that way. Best to have it come up cleanly once on > > the guest. (the FLR event in your log results from this move. The VF gets reset). > > > > If you must have a VF running on the Hypervisor, use "num_vfs=2 probe_vf=1". > > You will then see Devices ...xx.0, ...xx.1 and ...xx.2 in lspci. The first VF > > (...xx.1) will be initialized on the Hypervisor. Attach ...xx.2 to the VM. > > > > -Jack > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <201302031040.20906.jackm@dev.mellanox.co.il>]
[parent not found: <CAOoYFkHnrty9Nh294y94bWx2PEXHxk8_cVbYbPJ+=jLgf4m+0A@mail.gmail.com>]
[parent not found: <CAOoYFkHnrty9Nh294y94bWx2PEXHxk8_cVbYbPJ+=jLgf4m+0A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* RE: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <CAOoYFkHnrty9Nh294y94bWx2PEXHxk8_cVbYbPJ+=jLgf4m+0A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-02-04 11:23 ` Or Gerlitz [not found] ` <B855DA7245737D4B83EDD9C0B7698E655952F564-LSMZvP3E4uyuSA5JZHE7gA@public.gmane.org> 0 siblings, 1 reply; 16+ messages in thread From: Or Gerlitz @ 2013-02-04 11:23 UTC (permalink / raw) To: Mathis GAVILLON, alex.williamson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org Cc: Jack Morgenstein, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > Should I try to launch VM in an older version of qemu-kvm ? > dmesg when using qemu-kvm 1.2.2-6 : You need to use a release/drop of qemu which has these fixes, http://marc.info/?l=qemu-devel&m=135835580814935&w=2 http://marc.info/?l=qemu-devel&m=135836502119460&w=2 Alex, what stable branch/release would you recommend to use? Or. ________________________________________ From: Mathis GAVILLON [jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org] Sent: 04 February 2013 11:42 To: Jack Morgenstein Cc: Or Gerlitz Subject: Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine The solution you proposed hasn't changed anything. The error message in the VM and "Disabling IRQ #16" is already present in log files. Should I try to launch VM in an older version of qemu-kvm ? dmesg when using qemu-kvm 1.2.2-6 : [ 1914.152816] pci 0000:82:00.1: enabling device (0000 -> 0002) [ 1914.156091] mlx4_core 0000:82:00.0: FLR event for slave: 1 [ 1914.156101] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr [ 1914.156103] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 [ 1914.878353] assign device 0:82:0.1 [ 1915.080617] mlx4_core 0000:82:00.0: FLR event for slave: 1 [ 1915.080651] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr [ 1915.080652] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 [ 1932.546300] mlx4_core 0000:82:00.0: Received reset from slave:1 [ 1932.559015] pci 0000:82:00.1: irq 178 for MSI/MSI-X [ 1932.576928] pci 0000:82:00.1: irq 178 for MSI/MSI-X [ 1932.576951] pci 0000:82:00.1: irq 179 for MSI/MSI-X [ 1933.017996] irq 16: nobody cared (try booting with the "irqpoll" option) [ 1933.018000] Pid: 0, comm: swapper/3 Not tainted 3.8.0-rc2 #1 [ 1933.018001] Call Trace: [ 1933.018002] <IRQ> [<ffffffff810dc41d>] __report_bad_irq+0x3d/0xe0 [ 1933.018015] [<ffffffff810dc713>] note_interrupt+0x1a3/0x1f0 [ 1933.018020] [<ffffffff814362d0>] ? userspace_cpufreq_notifier+0x90/0x90 [ 1933.018023] [<ffffffff810d9e69>] handle_irq_event_percpu+0xb9/0x200 [ 1933.018026] [<ffffffff814362d0>] ? userspace_cpufreq_notifier+0x90/0x90 [ 1933.018029] [<ffffffff810d9ff2>] handle_irq_event+0x42/0x70 [ 1933.018031] [<ffffffff810dd259>] handle_fasteoi_irq+0x59/0x100 [ 1933.018036] [<ffffffff8101616f>] handle_irq+0xbf/0x150 [ 1933.018041] [<ffffffff8155b792>] ? __atomic_notifier_call_chain+0x12/0x20 [ 1933.018044] [<ffffffff8155b7b6>] ? atomic_notifier_call_chain+0x16/0x20 [ 1933.018047] [<ffffffff8156188a>] do_IRQ+0x5a/0xe0 [ 1933.018050] [<ffffffff81557aad>] common_interrupt+0x6d/0x6d [ 1933.018051] <EOI> [<ffffffff81436d00>] ? cpuidle_wrap_enter+0x50/0xa0 [ 1933.018057] [<ffffffff81436cf9>] ? cpuidle_wrap_enter+0x49/0xa0 [ 1933.018060] [<ffffffff81436d60>] cpuidle_enter_tk+0x10/0x20 [ 1933.018063] [<ffffffff8143698b>] cpuidle_idle_call+0xbb/0x280 [ 1933.018067] [<ffffffff8101d2ef>] cpu_idle+0xaf/0x120 [ 1933.018071] [<ffffffff81544bb9>] start_secondary+0x255/0x257 [ 1933.018072] handlers: [ 1933.018075] [<ffffffff813cea90>] usb_hcd_irq [ 1933.018077] Disabling IRQ #16 [ 1942.678990] mlx4_core 0000:82:00.0: Received reset from slave:1 dmesg when using qemu-kvm 1.3.0-5 : [ 181.886165] pci 0000:82:00.1: enabling device (0000 -> 0002) [ 181.889480] mlx4_core 0000:82:00.0: FLR event for slave: 1 [ 181.889517] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr [ 181.889519] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 [ 182.622378] assign device 0:82:0.1 [ 182.798342] mlx4_core 0000:82:00.0: FLR event for slave: 1 [ 182.798352] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr [ 182.798354] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 [ 202.515121] mlx4_core 0000:82:00.0: Received reset from slave:1 [ 202.527857] pci 0000:82:00.1: irq 178 for MSI/MSI-X [ 202.549725] pci 0000:82:00.1: irq 178 for MSI/MSI-X [ 202.549740] pci 0000:82:00.1: irq 179 for MSI/MSI-X [ 202.991521] irq 16: nobody cared (try booting with the "irqpoll" option) [ 202.991526] Pid: 0, comm: swapper/3 Not tainted 3.8.0-rc2 #1 [ 202.991527] Call Trace: [ 202.991528] <IRQ> [<ffffffff810dc41d>] __report_bad_irq+0x3d/0xe0 [ 202.991541] [<ffffffff810dc713>] note_interrupt+0x1a3/0x1f0 [ 202.991546] [<ffffffff814362d0>] ? userspace_cpufreq_notifier+0x90/0x90 [ 202.991550] [<ffffffff810d9e69>] handle_irq_event_percpu+0xb9/0x200 [ 202.991553] [<ffffffff814362d0>] ? userspace_cpufreq_notifier+0x90/0x90 [ 202.991555] [<ffffffff810d9ff2>] handle_irq_event+0x42/0x70 [ 202.991558] [<ffffffff810dd259>] handle_fasteoi_irq+0x59/0x100 [ 202.991562] [<ffffffff8101616f>] handle_irq+0xbf/0x150 [ 202.991567] [<ffffffff8155b792>] ? __atomic_notifier_call_chain+0x12/0x20 [ 202.991570] [<ffffffff8155b7b6>] ? atomic_notifier_call_chain+0x16/0x20 [ 202.991574] [<ffffffff8156188a>] do_IRQ+0x5a/0xe0 [ 202.991577] [<ffffffff81557aad>] common_interrupt+0x6d/0x6d [ 202.991577] <EOI> [<ffffffff81436d00>] ? cpuidle_wrap_enter+0x50/0xa0 [ 202.991584] [<ffffffff81436cf9>] ? cpuidle_wrap_enter+0x49/0xa0 [ 202.991587] [<ffffffff81436d60>] cpuidle_enter_tk+0x10/0x20 [ 202.991589] [<ffffffff8143698b>] cpuidle_idle_call+0xbb/0x280 [ 202.991593] [<ffffffff8101d2ef>] cpu_idle+0xaf/0x120 [ 202.991597] [<ffffffff81544bb9>] start_secondary+0x255/0x257 [ 202.991598] handlers: [ 202.991602] [<ffffffff813cea90>] usb_hcd_irq [ 202.991603] Disabling IRQ #16 [ 212.669908] mlx4_core 0000:82:00.0: Received reset from slave:1 Thanks -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <B855DA7245737D4B83EDD9C0B7698E655952F564-LSMZvP3E4uyuSA5JZHE7gA@public.gmane.org>]
* Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <B855DA7245737D4B83EDD9C0B7698E655952F564-LSMZvP3E4uyuSA5JZHE7gA@public.gmane.org> @ 2013-02-04 15:24 ` Alex Williamson [not found] ` <1359991471.11144.396.camel-xdHQ/5r00wBBDLzU/O5InQ@public.gmane.org> 0 siblings, 1 reply; 16+ messages in thread From: Alex Williamson @ 2013-02-04 15:24 UTC (permalink / raw) To: Or Gerlitz Cc: Mathis GAVILLON, Jack Morgenstein, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org On Mon, 2013-02-04 at 11:23 +0000, Or Gerlitz wrote: > > Should I try to launch VM in an older version of qemu-kvm ? > > dmesg when using qemu-kvm 1.2.2-6 : > > You need to use a release/drop of qemu which has these fixes, > > http://marc.info/?l=qemu-devel&m=135835580814935&w=2 > http://marc.info/?l=qemu-devel&m=135836502119460&w=2 > > Alex, what stable branch/release would you recommend to use? These are available in the qemu 1.3.1 stable release http://lists.gnu.org/archive/html/qemu-devel/2013-01/msg05403.html And they're in the current development tree which is stabilizing for the 1.4.0 release. Thanks, Alex > ________________________________________ > From: Mathis GAVILLON [jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org] > Sent: 04 February 2013 11:42 > To: Jack Morgenstein > Cc: Or Gerlitz > Subject: Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine > > The solution you proposed hasn't changed anything. The error message > in the VM and "Disabling IRQ #16" is already present in log files. > Should I try to launch VM in an older version of qemu-kvm ? > > dmesg when using qemu-kvm 1.2.2-6 : > > [ 1914.152816] pci 0000:82:00.1: enabling device (0000 -> 0002) > [ 1914.156091] mlx4_core 0000:82:00.0: FLR event for slave: 1 > [ 1914.156101] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr > [ 1914.156103] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 > [ 1914.878353] assign device 0:82:0.1 > [ 1915.080617] mlx4_core 0000:82:00.0: FLR event for slave: 1 > [ 1915.080651] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr > [ 1915.080652] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 > [ 1932.546300] mlx4_core 0000:82:00.0: Received reset from slave:1 > [ 1932.559015] pci 0000:82:00.1: irq 178 for MSI/MSI-X > [ 1932.576928] pci 0000:82:00.1: irq 178 for MSI/MSI-X > [ 1932.576951] pci 0000:82:00.1: irq 179 for MSI/MSI-X > [ 1933.017996] irq 16: nobody cared (try booting with the "irqpoll" option) > [ 1933.018000] Pid: 0, comm: swapper/3 Not tainted 3.8.0-rc2 #1 > [ 1933.018001] Call Trace: > [ 1933.018002] <IRQ> [<ffffffff810dc41d>] __report_bad_irq+0x3d/0xe0 > [ 1933.018015] [<ffffffff810dc713>] note_interrupt+0x1a3/0x1f0 > [ 1933.018020] [<ffffffff814362d0>] ? userspace_cpufreq_notifier+0x90/0x90 > [ 1933.018023] [<ffffffff810d9e69>] handle_irq_event_percpu+0xb9/0x200 > [ 1933.018026] [<ffffffff814362d0>] ? userspace_cpufreq_notifier+0x90/0x90 > [ 1933.018029] [<ffffffff810d9ff2>] handle_irq_event+0x42/0x70 > [ 1933.018031] [<ffffffff810dd259>] handle_fasteoi_irq+0x59/0x100 > [ 1933.018036] [<ffffffff8101616f>] handle_irq+0xbf/0x150 > [ 1933.018041] [<ffffffff8155b792>] ? __atomic_notifier_call_chain+0x12/0x20 > [ 1933.018044] [<ffffffff8155b7b6>] ? atomic_notifier_call_chain+0x16/0x20 > [ 1933.018047] [<ffffffff8156188a>] do_IRQ+0x5a/0xe0 > [ 1933.018050] [<ffffffff81557aad>] common_interrupt+0x6d/0x6d > [ 1933.018051] <EOI> [<ffffffff81436d00>] ? cpuidle_wrap_enter+0x50/0xa0 > [ 1933.018057] [<ffffffff81436cf9>] ? cpuidle_wrap_enter+0x49/0xa0 > [ 1933.018060] [<ffffffff81436d60>] cpuidle_enter_tk+0x10/0x20 > [ 1933.018063] [<ffffffff8143698b>] cpuidle_idle_call+0xbb/0x280 > [ 1933.018067] [<ffffffff8101d2ef>] cpu_idle+0xaf/0x120 > [ 1933.018071] [<ffffffff81544bb9>] start_secondary+0x255/0x257 > [ 1933.018072] handlers: > [ 1933.018075] [<ffffffff813cea90>] usb_hcd_irq > [ 1933.018077] Disabling IRQ #16 > [ 1942.678990] mlx4_core 0000:82:00.0: Received reset from slave:1 > > dmesg when using qemu-kvm 1.3.0-5 : > > [ 181.886165] pci 0000:82:00.1: enabling device (0000 -> 0002) > [ 181.889480] mlx4_core 0000:82:00.0: FLR event for slave: 1 > [ 181.889517] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr > [ 181.889519] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 > [ 182.622378] assign device 0:82:0.1 > [ 182.798342] mlx4_core 0000:82:00.0: FLR event for slave: 1 > [ 182.798352] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr > [ 182.798354] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 > [ 202.515121] mlx4_core 0000:82:00.0: Received reset from slave:1 > [ 202.527857] pci 0000:82:00.1: irq 178 for MSI/MSI-X > [ 202.549725] pci 0000:82:00.1: irq 178 for MSI/MSI-X > [ 202.549740] pci 0000:82:00.1: irq 179 for MSI/MSI-X > [ 202.991521] irq 16: nobody cared (try booting with the "irqpoll" option) > [ 202.991526] Pid: 0, comm: swapper/3 Not tainted 3.8.0-rc2 #1 > [ 202.991527] Call Trace: > [ 202.991528] <IRQ> [<ffffffff810dc41d>] __report_bad_irq+0x3d/0xe0 > [ 202.991541] [<ffffffff810dc713>] note_interrupt+0x1a3/0x1f0 > [ 202.991546] [<ffffffff814362d0>] ? userspace_cpufreq_notifier+0x90/0x90 > [ 202.991550] [<ffffffff810d9e69>] handle_irq_event_percpu+0xb9/0x200 > [ 202.991553] [<ffffffff814362d0>] ? userspace_cpufreq_notifier+0x90/0x90 > [ 202.991555] [<ffffffff810d9ff2>] handle_irq_event+0x42/0x70 > [ 202.991558] [<ffffffff810dd259>] handle_fasteoi_irq+0x59/0x100 > [ 202.991562] [<ffffffff8101616f>] handle_irq+0xbf/0x150 > [ 202.991567] [<ffffffff8155b792>] ? __atomic_notifier_call_chain+0x12/0x20 > [ 202.991570] [<ffffffff8155b7b6>] ? atomic_notifier_call_chain+0x16/0x20 > [ 202.991574] [<ffffffff8156188a>] do_IRQ+0x5a/0xe0 > [ 202.991577] [<ffffffff81557aad>] common_interrupt+0x6d/0x6d > [ 202.991577] <EOI> [<ffffffff81436d00>] ? cpuidle_wrap_enter+0x50/0xa0 > [ 202.991584] [<ffffffff81436cf9>] ? cpuidle_wrap_enter+0x49/0xa0 > [ 202.991587] [<ffffffff81436d60>] cpuidle_enter_tk+0x10/0x20 > [ 202.991589] [<ffffffff8143698b>] cpuidle_idle_call+0xbb/0x280 > [ 202.991593] [<ffffffff8101d2ef>] cpu_idle+0xaf/0x120 > [ 202.991597] [<ffffffff81544bb9>] start_secondary+0x255/0x257 > [ 202.991598] handlers: > [ 202.991602] [<ffffffff813cea90>] usb_hcd_irq > [ 202.991603] Disabling IRQ #16 > [ 212.669908] mlx4_core 0000:82:00.0: Received reset from slave:1 > > Thanks -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <1359991471.11144.396.camel-xdHQ/5r00wBBDLzU/O5InQ@public.gmane.org>]
* Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <1359991471.11144.396.camel-xdHQ/5r00wBBDLzU/O5InQ@public.gmane.org> @ 2013-02-05 9:51 ` Or Gerlitz 2013-02-05 10:32 ` Mathis GAVILLON 1 sibling, 0 replies; 16+ messages in thread From: Or Gerlitz @ 2013-02-05 9:51 UTC (permalink / raw) To: Alex Williamson, Mathis GAVILLON Cc: Jack Morgenstein, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org On 04/02/2013 17:24, Alex Williamson wrote: > These are available in the qemu 1.3.1 stable release > > http://lists.gnu.org/archive/html/qemu-devel/2013-01/msg05403.html > > And they're in the current development tree which is stabilizing for the > 1.4.0 release thanks Alex! Mathis, does this helps? Or. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <1359991471.11144.396.camel-xdHQ/5r00wBBDLzU/O5InQ@public.gmane.org> 2013-02-05 9:51 ` Or Gerlitz @ 2013-02-05 10:32 ` Mathis GAVILLON [not found] ` <CAOoYFkEnzYwPDjka+GGXeXt2xkGhX7qwAptGUvUJChgDEjRBGw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 1 sibling, 1 reply; 16+ messages in thread From: Mathis GAVILLON @ 2013-02-05 10:32 UTC (permalink / raw) To: Alex Williamson Cc: Or Gerlitz, Jack Morgenstein, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Alex, Or, Jack, I've just finished to install qemu 1.3.1 and all is working well. IRQ error has disappeared and an IB interface is know available in the virtual machine. Thank's so much to all of you for all of your answers Mathis 2013/2/4 Alex Williamson <alex.williamson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>: > On Mon, 2013-02-04 at 11:23 +0000, Or Gerlitz wrote: >> > Should I try to launch VM in an older version of qemu-kvm ? >> > dmesg when using qemu-kvm 1.2.2-6 : >> >> You need to use a release/drop of qemu which has these fixes, >> >> http://marc.info/?l=qemu-devel&m=135835580814935&w=2 >> http://marc.info/?l=qemu-devel&m=135836502119460&w=2 >> >> Alex, what stable branch/release would you recommend to use? > > These are available in the qemu 1.3.1 stable release > > http://lists.gnu.org/archive/html/qemu-devel/2013-01/msg05403.html > > And they're in the current development tree which is stabilizing for the > 1.4.0 release. Thanks, > > Alex > >> ________________________________________ >> From: Mathis GAVILLON [jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org] >> Sent: 04 February 2013 11:42 >> To: Jack Morgenstein >> Cc: Or Gerlitz >> Subject: Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine >> >> The solution you proposed hasn't changed anything. The error message >> in the VM and "Disabling IRQ #16" is already present in log files. >> Should I try to launch VM in an older version of qemu-kvm ? >> >> dmesg when using qemu-kvm 1.2.2-6 : >> >> [ 1914.152816] pci 0000:82:00.1: enabling device (0000 -> 0002) >> [ 1914.156091] mlx4_core 0000:82:00.0: FLR event for slave: 1 >> [ 1914.156101] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr >> [ 1914.156103] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 >> [ 1914.878353] assign device 0:82:0.1 >> [ 1915.080617] mlx4_core 0000:82:00.0: FLR event for slave: 1 >> [ 1915.080651] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr >> [ 1915.080652] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 >> [ 1932.546300] mlx4_core 0000:82:00.0: Received reset from slave:1 >> [ 1932.559015] pci 0000:82:00.1: irq 178 for MSI/MSI-X >> [ 1932.576928] pci 0000:82:00.1: irq 178 for MSI/MSI-X >> [ 1932.576951] pci 0000:82:00.1: irq 179 for MSI/MSI-X >> [ 1933.017996] irq 16: nobody cared (try booting with the "irqpoll" option) >> [ 1933.018000] Pid: 0, comm: swapper/3 Not tainted 3.8.0-rc2 #1 >> [ 1933.018001] Call Trace: >> [ 1933.018002] <IRQ> [<ffffffff810dc41d>] __report_bad_irq+0x3d/0xe0 >> [ 1933.018015] [<ffffffff810dc713>] note_interrupt+0x1a3/0x1f0 >> [ 1933.018020] [<ffffffff814362d0>] ? userspace_cpufreq_notifier+0x90/0x90 >> [ 1933.018023] [<ffffffff810d9e69>] handle_irq_event_percpu+0xb9/0x200 >> [ 1933.018026] [<ffffffff814362d0>] ? userspace_cpufreq_notifier+0x90/0x90 >> [ 1933.018029] [<ffffffff810d9ff2>] handle_irq_event+0x42/0x70 >> [ 1933.018031] [<ffffffff810dd259>] handle_fasteoi_irq+0x59/0x100 >> [ 1933.018036] [<ffffffff8101616f>] handle_irq+0xbf/0x150 >> [ 1933.018041] [<ffffffff8155b792>] ? __atomic_notifier_call_chain+0x12/0x20 >> [ 1933.018044] [<ffffffff8155b7b6>] ? atomic_notifier_call_chain+0x16/0x20 >> [ 1933.018047] [<ffffffff8156188a>] do_IRQ+0x5a/0xe0 >> [ 1933.018050] [<ffffffff81557aad>] common_interrupt+0x6d/0x6d >> [ 1933.018051] <EOI> [<ffffffff81436d00>] ? cpuidle_wrap_enter+0x50/0xa0 >> [ 1933.018057] [<ffffffff81436cf9>] ? cpuidle_wrap_enter+0x49/0xa0 >> [ 1933.018060] [<ffffffff81436d60>] cpuidle_enter_tk+0x10/0x20 >> [ 1933.018063] [<ffffffff8143698b>] cpuidle_idle_call+0xbb/0x280 >> [ 1933.018067] [<ffffffff8101d2ef>] cpu_idle+0xaf/0x120 >> [ 1933.018071] [<ffffffff81544bb9>] start_secondary+0x255/0x257 >> [ 1933.018072] handlers: >> [ 1933.018075] [<ffffffff813cea90>] usb_hcd_irq >> [ 1933.018077] Disabling IRQ #16 >> [ 1942.678990] mlx4_core 0000:82:00.0: Received reset from slave:1 >> >> dmesg when using qemu-kvm 1.3.0-5 : >> >> [ 181.886165] pci 0000:82:00.1: enabling device (0000 -> 0002) >> [ 181.889480] mlx4_core 0000:82:00.0: FLR event for slave: 1 >> [ 181.889517] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr >> [ 181.889519] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 >> [ 182.622378] assign device 0:82:0.1 >> [ 182.798342] mlx4_core 0000:82:00.0: FLR event for slave: 1 >> [ 182.798352] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr >> [ 182.798354] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1 >> [ 202.515121] mlx4_core 0000:82:00.0: Received reset from slave:1 >> [ 202.527857] pci 0000:82:00.1: irq 178 for MSI/MSI-X >> [ 202.549725] pci 0000:82:00.1: irq 178 for MSI/MSI-X >> [ 202.549740] pci 0000:82:00.1: irq 179 for MSI/MSI-X >> [ 202.991521] irq 16: nobody cared (try booting with the "irqpoll" option) >> [ 202.991526] Pid: 0, comm: swapper/3 Not tainted 3.8.0-rc2 #1 >> [ 202.991527] Call Trace: >> [ 202.991528] <IRQ> [<ffffffff810dc41d>] __report_bad_irq+0x3d/0xe0 >> [ 202.991541] [<ffffffff810dc713>] note_interrupt+0x1a3/0x1f0 >> [ 202.991546] [<ffffffff814362d0>] ? userspace_cpufreq_notifier+0x90/0x90 >> [ 202.991550] [<ffffffff810d9e69>] handle_irq_event_percpu+0xb9/0x200 >> [ 202.991553] [<ffffffff814362d0>] ? userspace_cpufreq_notifier+0x90/0x90 >> [ 202.991555] [<ffffffff810d9ff2>] handle_irq_event+0x42/0x70 >> [ 202.991558] [<ffffffff810dd259>] handle_fasteoi_irq+0x59/0x100 >> [ 202.991562] [<ffffffff8101616f>] handle_irq+0xbf/0x150 >> [ 202.991567] [<ffffffff8155b792>] ? __atomic_notifier_call_chain+0x12/0x20 >> [ 202.991570] [<ffffffff8155b7b6>] ? atomic_notifier_call_chain+0x16/0x20 >> [ 202.991574] [<ffffffff8156188a>] do_IRQ+0x5a/0xe0 >> [ 202.991577] [<ffffffff81557aad>] common_interrupt+0x6d/0x6d >> [ 202.991577] <EOI> [<ffffffff81436d00>] ? cpuidle_wrap_enter+0x50/0xa0 >> [ 202.991584] [<ffffffff81436cf9>] ? cpuidle_wrap_enter+0x49/0xa0 >> [ 202.991587] [<ffffffff81436d60>] cpuidle_enter_tk+0x10/0x20 >> [ 202.991589] [<ffffffff8143698b>] cpuidle_idle_call+0xbb/0x280 >> [ 202.991593] [<ffffffff8101d2ef>] cpu_idle+0xaf/0x120 >> [ 202.991597] [<ffffffff81544bb9>] start_secondary+0x255/0x257 >> [ 202.991598] handlers: >> [ 202.991602] [<ffffffff813cea90>] usb_hcd_irq >> [ 202.991603] Disabling IRQ #16 >> [ 212.669908] mlx4_core 0000:82:00.0: Received reset from slave:1 >> >> Thanks > > > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <CAOoYFkEnzYwPDjka+GGXeXt2xkGhX7qwAptGUvUJChgDEjRBGw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <CAOoYFkEnzYwPDjka+GGXeXt2xkGhX7qwAptGUvUJChgDEjRBGw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-02-05 11:00 ` Or Gerlitz [not found] ` <5110E64F.7080106-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 0 siblings, 1 reply; 16+ messages in thread From: Or Gerlitz @ 2013-02-05 11:00 UTC (permalink / raw) To: Mathis GAVILLON Cc: Alex Williamson, Jack Morgenstein, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org On 05/02/2013 12:32, Mathis GAVILLON wrote: > I've just finished to install qemu 1.3.1 and all is working well. IRQ > error has disappeared and an IB interface is know available in the virtual machine. nice! could you say again what distribution provided you with the "problematic" kvm-qemu and what was the exact version of that qemu? > > Thank's so much to all of you for all of your answers sure -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <5110E64F.7080106-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]
* Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine [not found] ` <5110E64F.7080106-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> @ 2013-02-05 11:45 ` Mathis GAVILLON 0 siblings, 0 replies; 16+ messages in thread From: Mathis GAVILLON @ 2013-02-05 11:45 UTC (permalink / raw) To: Or Gerlitz Cc: Alex Williamson, Jack Morgenstein, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org 2013/2/5 Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>: > On 05/02/2013 12:32, Mathis GAVILLON wrote: >> >> I've just finished to install qemu 1.3.1 and all is working well. IRQ >> error has disappeared and an IB interface is know available in the virtual >> machine. > > > nice! could you say again what distribution provided you with the > "problematic" kvm-qemu and what > was the exact version of that qemu? I'm using Fedora 18 and none of 1.2 versions (I really try 1.2.0, 1.2.2-2, 1.2.2-4 and 1.2.2-6 versions) of qemu-kvm support Infiniband virtualization. Mathis -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2013-02-05 11:45 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-01-23 15:29 Error message when trying to use Infiniband virtual functions in virtual machine Mathis GAVILLON
[not found] ` <CAOoYFkGrsWKGaFcNAQQnkks2Ku6O8xkPk=8BGQfRR74QhRU=VA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-01-24 14:41 ` Or Gerlitz
[not found] ` <CAOoYFkFQOyWbrZtufpk=82ButMxPjsue1Psih40h=F1=U7ng6g@mail.gmail.com>
[not found] ` <CAOoYFkGe=kKDDKvhOdRVCXQbGfV52q38dz6cYTpQXmFGOsBwsg@mail.gmail.com>
[not found] ` <CAOoYFkGe=kKDDKvhOdRVCXQbGfV52q38dz6cYTpQXmFGOsBwsg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-01-28 8:29 ` Fwd: " Mathis GAVILLON
[not found] ` <CAOoYFkFVvK+1YNJ_N6r0N_3ZMF6P9Q+0dbsDg4v8=3JfDPVOOA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-01-28 16:35 ` Jack Morgenstein
[not found] ` <201301281835.18533.jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2013-01-29 13:15 ` Mathis GAVILLON
[not found] ` <CAOoYFkE6YNfEJGJqGmpQTdSxR2Y_9pzkdcnHf_7jjqq57B_Nsw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-01-29 13:20 ` Mathis GAVILLON
2013-01-29 14:33 ` Jack Morgenstein
2013-01-29 14:44 ` Jack Morgenstein
[not found] ` <201301291644.35771.jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2013-01-29 15:18 ` Mathis GAVILLON
[not found] ` <CAOoYFkErXNxSBrQC_spMwcEiVmOpkzqQwuTQPOwO70bigsgYkQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-01-29 18:36 ` Jack Morgenstein
[not found] ` <201302031040.20906.jackm@dev.mellanox.co.il>
[not found] ` <CAOoYFkHnrty9Nh294y94bWx2PEXHxk8_cVbYbPJ+=jLgf4m+0A@mail.gmail.com>
[not found] ` <CAOoYFkHnrty9Nh294y94bWx2PEXHxk8_cVbYbPJ+=jLgf4m+0A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-02-04 11:23 ` Or Gerlitz
[not found] ` <B855DA7245737D4B83EDD9C0B7698E655952F564-LSMZvP3E4uyuSA5JZHE7gA@public.gmane.org>
2013-02-04 15:24 ` Alex Williamson
[not found] ` <1359991471.11144.396.camel-xdHQ/5r00wBBDLzU/O5InQ@public.gmane.org>
2013-02-05 9:51 ` Or Gerlitz
2013-02-05 10:32 ` Mathis GAVILLON
[not found] ` <CAOoYFkEnzYwPDjka+GGXeXt2xkGhX7qwAptGUvUJChgDEjRBGw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-02-05 11:00 ` Or Gerlitz
[not found] ` <5110E64F.7080106-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2013-02-05 11:45 ` Mathis GAVILLON
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox