From: Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
To: Mathis GAVILLON
<jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Re: Fwd: Error message when trying to use Infiniband virtual functions in virtual machine
Date: Tue, 29 Jan 2013 20:36:36 +0200 [thread overview]
Message-ID: <201301292036.36837.jackm@dev.mellanox.co.il> (raw)
In-Reply-To: <CAOoYFkErXNxSBrQC_spMwcEiVmOpkzqQwuTQPOwO70bigsgYkQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Tuesday 29 January 2013 17:18, Mathis GAVILLON wrote:
> Support for interrupt remapping should be now available (kernel has
> been recompiled with but is there any way to check if it's available
> ?).
> I tried to do what you say in step 2 (num_vfs=1 and probe_vf=0) but it
> doesn't change anything. I also tried num_vfs=2 and probe_vf=1 but
> nothing is better.
> Nevertheless, a strange information message appear in shell when I
> boot my guest :
>
> Message from syslogd@localhost at Jan 29 15:51:49 ...
> kernel:Disabling IRQ #16
>
Or, Mathis,
This now looks like the problem that we had with a qemu change.
qemu-kvm-0.12.1.2-2.348.el6 (or possibly earlier) was changed from the
previous qemu version so that the pci header for the
guest was not updated to indicate MSI_X enabled when the guest called
pci_enable_msix() -- rather, a "lazy update" policy was introduced which
did not update the flag in the Hypervisor until the guest called request_irq().
This caused problems for some vendor's network sriov drivers, and RH implemented
instead a hybrid policy, compatible with their previous policy, which did not
break any sriov drivers.
The updated qemu module for RH is:
qemu-kvm-0.12.1.2-2.349.el6
To see which version of qemu you have installed, just enter the following command:
rpm -qvia | grep qemu
I found a web site which has the rpm's of interest (for fedora core 18):
http://rpmfind.net/linux/rpm2html/search.php?query=qemu-kvm
You might try using the following rpm (built Jan 16):
ftp://rpmfind.net/linux/fedora/linux/updates/18/x86_64/qemu-kvm-1.2.2-2.fc18.x86_64.rpm
(which probably has the fix)
If that does not work, you might try:
ftp://rpmfind.net/linux/fedora/linux/development/rawhide/x86_64/os/Packages/q/qemu-kvm-1.3.0-5.fc19.x86_64.rpm
-Jack
> I put dmesg log from step 2 (host and guest) again :
>
> [ 1419.382452] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011)
> [ 1419.382454] mlx4_core: Initializing 0000:05:00.0
> [ 1419.382653] mlx4_core 0000:05:00.0: Enabling SR-IOV with 1 VFs
> [ 1419.483687] pci 0000:05:00.1: [15b3:1002] type 00 class 0x0c0600
> [ 1419.484427] mlx4_core 0000:05:00.0: Running in master mode
> [ 1420.486608] mlx4_core 0000:05:00.0: FW version 2.9.1200 (cmd intf
> rev 3), max commands 16
> [ 1420.486610] mlx4_core 0000:05:00.0: Catastrophic error buffer at
> 0x1f020, size 0x10, BAR 0
> [ 1420.486612] mlx4_core 0000:05:00.0: Communication vector bar:2 offset:0x800
> [ 1420.486613] mlx4_core 0000:05:00.0: FW size 385 KB
> [ 1420.486615] mlx4_core 0000:05:00.0: Clear int @ f0058, BAR 0
> [ 1420.488474] mlx4_core 0000:05:00.0: Mapped 26 chunks/6168 KB for FW.
> [ 1421.155871] mlx4_core 0000:05:00.0: BlueFlame available (reg size
> 512, regs/page 8)
> [ 1421.156035] mlx4_core 0000:05:00.0: Base MM extensions: flags
> 00000cc0, rsvd L_Key 00000500
> [ 1421.156036] mlx4_core 0000:05:00.0: Max ICM size 4294967296 MB
> [ 1421.156038] mlx4_core 0000:05:00.0: Max QPs: 16777216, reserved
> QPs: 64, entry size: 256
> [ 1421.156040] mlx4_core 0000:05:00.0: Max SRQs: 16777216, reserved
> SRQs: 64, entry size: 128
> [ 1421.156041] mlx4_core 0000:05:00.0: Max CQs: 16777216, reserved
> CQs: 128, entry size: 128
> [ 1421.156043] mlx4_core 0000:05:00.0: Max EQs: 32, reserved EQs: 28,
> entry size: 128
> [ 1421.156045] mlx4_core 0000:05:00.0: reserved MPTs: 16, reserved MTTs: 16
> [ 1421.156046] mlx4_core 0000:05:00.0: Max PDs: 131072, reserved PDs:
> 4, reserved UARs: 7
> [ 1421.156048] mlx4_core 0000:05:00.0: Max QP/MCG: 131072, reserved MGMs: 0
> [ 1421.156049] mlx4_core 0000:05:00.0: Max CQEs: 4194304, max WQEs:
> 16384, max SRQ WQEs: 16384
> [ 1421.156051] mlx4_core 0000:05:00.0: Local CA ACK delay: 15, max
> MTU: 4096, port width cap: 3
> [ 1421.156052] mlx4_core 0000:05:00.0: Max SQ desc size: 1008, max SQ S/G: 62
> [ 1421.156054] mlx4_core 0000:05:00.0: Max RQ desc size: 512, max RQ S/G: 32
> [ 1421.156055] mlx4_core 0000:05:00.0: Max GSO size: 131072
> [ 1421.156056] mlx4_core 0000:05:00.0: Max counters: 256
> [ 1421.156058] mlx4_core 0000:05:00.0: Max RSS Table size: 256
> [ 1421.156059] mlx4_core 0000:05:00.0: DEV_CAP flags:
> [ 1421.156060] mlx4_core 0000:05:00.0: RC transport
> [ 1421.156062] mlx4_core 0000:05:00.0: UC transport
> [ 1421.156063] mlx4_core 0000:05:00.0: UD transport
> [ 1421.156064] mlx4_core 0000:05:00.0: XRC transport
> [ 1421.156065] mlx4_core 0000:05:00.0: FCoIB support
> [ 1421.156066] mlx4_core 0000:05:00.0: SRQ support
> [ 1421.156068] mlx4_core 0000:05:00.0: IPoIB checksum offload
> [ 1421.156069] mlx4_core 0000:05:00.0: P_Key violation counter
> [ 1421.156070] mlx4_core 0000:05:00.0: Q_Key violation counter
> [ 1421.156071] mlx4_core 0000:05:00.0: DPDP
> [ 1421.156073] mlx4_core 0000:05:00.0: Big LSO headers
> [ 1421.156074] mlx4_core 0000:05:00.0: APM support
> [ 1421.156075] mlx4_core 0000:05:00.0: Atomic ops support
> [ 1421.156077] mlx4_core 0000:05:00.0: Address vector port checking support
> [ 1421.156078] mlx4_core 0000:05:00.0: UD multicast support
> [ 1421.156079] mlx4_core 0000:05:00.0: Router support
> [ 1421.156080] mlx4_core 0000:05:00.0: IBoE support
> [ 1421.156082] mlx4_core 0000:05:00.0: Unicast loopback support
> [ 1421.156083] mlx4_core 0000:05:00.0: UDP RSS support
> [ 1421.156084] mlx4_core 0000:05:00.0: Unicast VEP steering support
> [ 1421.156085] mlx4_core 0000:05:00.0: Multicast VEP steering support
> [ 1421.156087] mlx4_core 0000:05:00.0: Counters support
> [ 1421.156088] mlx4_core 0000:05:00.0: RSS support
> [ 1421.156090] mlx4_core 0000:05:00.0: RSS Toeplitz Hash Function support
> [ 1421.156091] mlx4_core 0000:05:00.0: RSS XOR Hash Function support
> [ 1421.156093] mlx4_core 0000:05:00.0: Steering mode is: B0 steering,
> oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10
> [ 1421.156099] mlx4_core 0000:05:00.0: profile[ 0] ( CMPT): 2^26
> entries @ 0x 0, size 0x 100000000
> [ 1421.156101] mlx4_core 0000:05:00.0: profile[ 1] (RDMARC): 2^22
> entries @ 0x 100000000, size 0x 8000000
> [ 1421.156103] mlx4_core 0000:05:00.0: profile[ 2] ( MTT): 2^24
> entries @ 0x 108000000, size 0x 8000000
> [ 1421.156105] mlx4_core 0000:05:00.0: profile[ 3] ( QP): 2^18
> entries @ 0x 110000000, size 0x 4000000
> [ 1421.156107] mlx4_core 0000:05:00.0: profile[ 4] ( DMPT): 2^19
> entries @ 0x 114000000, size 0x 2000000
> [ 1421.156109] mlx4_core 0000:05:00.0: profile[ 5] ( ALTC): 2^18
> entries @ 0x 116000000, size 0x 1000000
> [ 1421.156111] mlx4_core 0000:05:00.0: profile[ 6] ( SRQ): 2^16
> entries @ 0x 117000000, size 0x 800000
> [ 1421.156113] mlx4_core 0000:05:00.0: profile[ 7] ( CQ): 2^16
> entries @ 0x 117800000, size 0x 800000
> [ 1421.156115] mlx4_core 0000:05:00.0: profile[ 8] ( MCG): 2^13
> entries @ 0x 118000000, size 0x 800000
> [ 1421.156117] mlx4_core 0000:05:00.0: profile[ 9] ( AUXC): 2^18
> entries @ 0x 118800000, size 0x 40000
> [ 1421.156119] mlx4_core 0000:05:00.0: profile[10] ( EQ): 2^10
> entries @ 0x 118840000, size 0x 20000
> [ 1421.156120] mlx4_core 0000:05:00.0: HCA context memory: reserving 4596096 KB
> [ 1421.156136] mlx4_core 0000:05:00.0: 4596096 KB of HCA context
> requires 9068 KB aux memory.
> [ 1421.185505] mlx4_core 0000:05:00.0: Mapped 39 chunks/9068 KB for ICM aux.
> [ 1421.186824] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at 0 for ICM.
> [ 1421.188122] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 40000000 for ICM.
> [ 1421.189418] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 80000000 for ICM.
> [ 1421.189754] mlx4_core 0000:05:00.0: Mapped 1 chunks/64 KB at
> c0000000 for ICM.
> [ 1421.190400] mlx4_core 0000:05:00.0: Mapped 1 chunks/128 KB at
> 118840000 for ICM.
> [ 1421.191677] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 108000000 for ICM.
> [ 1421.193001] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 114000000 for ICM.
> [ 1421.194279] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 110000000 for ICM.
> [ 1421.195563] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118800000 for ICM.
> [ 1421.196841] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 116000000 for ICM.
> [ 1421.198127] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 100000000 for ICM.
> [ 1421.199405] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 117800000 for ICM.
> [ 1421.200684] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 117000000 for ICM.
> [ 1421.201964] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118000000 for ICM.
> [ 1421.203240] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118040000 for ICM.
> [ 1421.204517] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118080000 for ICM.
> [ 1421.205802] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 1180c0000 for ICM.
> [ 1421.207079] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118100000 for ICM.
> [ 1421.208356] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118140000 for ICM.
> [ 1421.209633] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118180000 for ICM.
> [ 1421.210912] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 1181c0000 for ICM.
> [ 1421.212191] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118200000 for ICM.
> [ 1421.213465] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118240000 for ICM.
> [ 1421.214742] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118280000 for ICM.
> [ 1421.216024] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 1182c0000 for ICM.
> [ 1421.217303] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118300000 for ICM.
> [ 1421.218580] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118340000 for ICM.
> [ 1421.219855] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118380000 for ICM.
> [ 1421.221132] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 1183c0000 for ICM.
> [ 1421.222407] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118400000 for ICM.
> [ 1421.223685] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118440000 for ICM.
> [ 1421.224963] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118480000 for ICM.
> [ 1421.226245] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 1184c0000 for ICM.
> [ 1421.227522] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118500000 for ICM.
> [ 1421.228801] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118540000 for ICM.
> [ 1421.230079] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118580000 for ICM.
> [ 1421.231355] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 1185c0000 for ICM.
> [ 1421.232633] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118600000 for ICM.
> [ 1421.233909] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118640000 for ICM.
> [ 1421.235186] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118680000 for ICM.
> [ 1421.236470] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 1186c0000 for ICM.
> [ 1421.237748] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118700000 for ICM.
> [ 1421.239026] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118740000 for ICM.
> [ 1421.240303] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 118780000 for ICM.
> [ 1421.241581] mlx4_core 0000:05:00.0: Mapped 1 chunks/256 KB at
> 1187c0000 for ICM.
> [ 1421.744993] mlx4_core 0000:05:00.0: Started init_resource_tracker: 80 slaves
> [ 1421.745454] mlx4_core 0000:05:00.0: irq 176 for MSI/MSI-X
> [ 1421.745466] mlx4_core 0000:05:00.0: irq 177 for MSI/MSI-X
> [ 1421.750286] mlx4_core 0000:05:00.0: NOP command IRQ test passed
> [ 1421.750911] mlx4_core: Initializing 0000:82:00.0
> [ 1421.751006] mlx4_core 0000:82:00.0: Enabling SR-IOV with 1 VFs
> [ 1421.851346] pci 0000:82:00.1: [15b3:1004] type 00 class 0x028000
> [ 1421.852051] mlx4_core 0000:82:00.0: Running in master mode
> [ 1422.853834] mlx4_core 0000:82:00.0: FW version 2.11.500 (cmd intf
> rev 3), max commands 16
> [ 1422.853837] mlx4_core 0000:82:00.0: Catastrophic error buffer at
> 0x1f020, size 0x10, BAR 0
> [ 1422.853839] mlx4_core 0000:82:00.0: Communication vector bar:2 offset:0x800
> [ 1422.853840] mlx4_core 0000:82:00.0: FW size 385 KB
> [ 1422.853842] mlx4_core 0000:82:00.0: Clear int @ f0058, BAR 0
> [ 1422.855140] mlx4_core 0000:82:00.0: Mapped 26 chunks/6168 KB for FW.
> [ 1425.518005] mlx4_core 0000:82:00.0: BlueFlame available (reg size
> 512, regs/page 8)
> [ 1425.518117] mlx4_core 0000:82:00.0: Base MM extensions: flags
> 00104cc0, rsvd L_Key 00008000
> [ 1425.518119] mlx4_core 0000:82:00.0: Max ICM size 4294967296 MB
> [ 1425.518120] mlx4_core 0000:82:00.0: Max QPs: 16777216, reserved
> QPs: 64, entry size: 256
> [ 1425.518122] mlx4_core 0000:82:00.0: Max SRQs: 16777216, reserved
> SRQs: 64, entry size: 128
> [ 1425.518124] mlx4_core 0000:82:00.0: Max CQs: 16777216, reserved
> CQs: 128, entry size: 128
> [ 1425.518125] mlx4_core 0000:82:00.0: Max EQs: 32, reserved EQs: 28,
> entry size: 128
> [ 1425.518127] mlx4_core 0000:82:00.0: reserved MPTs: 256, reserved MTTs: 32
> [ 1425.518128] mlx4_core 0000:82:00.0: Max PDs: 131072, reserved PDs:
> 4, reserved UARs: 7
> [ 1425.518130] mlx4_core 0000:82:00.0: Max QP/MCG: 131072, reserved MGMs: 0
> [ 1425.518131] mlx4_core 0000:82:00.0: Max CQEs: 4194304, max WQEs:
> 16384, max SRQ WQEs: 16384
> [ 1425.518133] mlx4_core 0000:82:00.0: Local CA ACK delay: 15, max
> MTU: 4096, port width cap: 2
> [ 1425.518134] mlx4_core 0000:82:00.0: Max SQ desc size: 1008, max SQ S/G: 62
> [ 1425.518136] mlx4_core 0000:82:00.0: Max RQ desc size: 512, max RQ S/G: 32
> [ 1425.518137] mlx4_core 0000:82:00.0: Max GSO size: 131072
> [ 1425.518138] mlx4_core 0000:82:00.0: Max counters: 256
> [ 1425.518140] mlx4_core 0000:82:00.0: Max RSS Table size: 256
> [ 1425.518141] mlx4_core 0000:82:00.0: DEV_CAP flags:
> [ 1425.518142] mlx4_core 0000:82:00.0: RC transport
> [ 1425.518144] mlx4_core 0000:82:00.0: UC transport
> [ 1425.518145] mlx4_core 0000:82:00.0: UD transport
> [ 1425.518146] mlx4_core 0000:82:00.0: XRC transport
> [ 1425.518147] mlx4_core 0000:82:00.0: FCoIB support
> [ 1425.518148] mlx4_core 0000:82:00.0: SRQ support
> [ 1425.518150] mlx4_core 0000:82:00.0: IPoIB checksum offload
> [ 1425.518151] mlx4_core 0000:82:00.0: P_Key violation counter
> [ 1425.518152] mlx4_core 0000:82:00.0: Q_Key violation counter
> [ 1425.518154] mlx4_core 0000:82:00.0: DPDP
> [ 1425.518155] mlx4_core 0000:82:00.0: Big LSO headers
> [ 1425.518156] mlx4_core 0000:82:00.0: APM support
> [ 1425.518158] mlx4_core 0000:82:00.0: Atomic ops support
> [ 1425.518159] mlx4_core 0000:82:00.0: Address vector port checking support
> [ 1425.518160] mlx4_core 0000:82:00.0: UD multicast support
> [ 1425.518161] mlx4_core 0000:82:00.0: Router support
> [ 1425.518163] mlx4_core 0000:82:00.0: IBoE support
> [ 1425.518164] mlx4_core 0000:82:00.0: Unicast loopback support
> [ 1425.518165] mlx4_core 0000:82:00.0: FCS header control
> [ 1425.518167] mlx4_core 0000:82:00.0: UDP RSS support
> [ 1425.518168] mlx4_core 0000:82:00.0: Unicast VEP steering support
> [ 1425.518169] mlx4_core 0000:82:00.0: Multicast VEP steering support
> [ 1425.518170] mlx4_core 0000:82:00.0: Counters support
> [ 1425.518172] mlx4_core 0000:82:00.0: Port management change event support
> [ 1425.518173] mlx4_core 0000:82:00.0: 64 byte EQE support
> [ 1425.518175] mlx4_core 0000:82:00.0: 64 byte CQE support
> [ 1425.518176] mlx4_core 0000:82:00.0: RSS support
> [ 1425.518177] mlx4_core 0000:82:00.0: RSS Toeplitz Hash Function support
> [ 1425.518178] mlx4_core 0000:82:00.0: RSS XOR Hash Function support
> [ 1425.518180] mlx4_core 0000:82:00.0: Device manage flow steering support
> [ 1425.518182] mlx4_core 0000:82:00.0: 64B EQEs/CQEs supported by the
> device but not enabled
> [ 1425.518184] mlx4_core 0000:82:00.0: Steering mode is: B0 steering,
> oper_log_mgm_entry_size = 10, modparam log_num_mgm_entry_size = 10
> [ 1425.518190] mlx4_core 0000:82:00.0: profile[ 0] ( CMPT): 2^26
> entries @ 0x 0, size 0x 100000000
> [ 1425.518193] mlx4_core 0000:82:00.0: profile[ 1] (RDMARC): 2^22
> entries @ 0x 100000000, size 0x 8000000
> [ 1425.518195] mlx4_core 0000:82:00.0: profile[ 2] ( MTT): 2^24
> entries @ 0x 108000000, size 0x 8000000
> [ 1425.518197] mlx4_core 0000:82:00.0: profile[ 3] ( QP): 2^18
> entries @ 0x 110000000, size 0x 4000000
> [ 1425.518199] mlx4_core 0000:82:00.0: profile[ 4] ( DMPT): 2^19
> entries @ 0x 114000000, size 0x 2000000
> [ 1425.518200] mlx4_core 0000:82:00.0: profile[ 5] ( ALTC): 2^18
> entries @ 0x 116000000, size 0x 1000000
> [ 1425.518202] mlx4_core 0000:82:00.0: profile[ 6] ( SRQ): 2^16
> entries @ 0x 117000000, size 0x 800000
> [ 1425.518204] mlx4_core 0000:82:00.0: profile[ 7] ( CQ): 2^16
> entries @ 0x 117800000, size 0x 800000
> [ 1425.518206] mlx4_core 0000:82:00.0: profile[ 8] ( MCG): 2^13
> entries @ 0x 118000000, size 0x 800000
> [ 1425.518208] mlx4_core 0000:82:00.0: profile[ 9] ( AUXC): 2^18
> entries @ 0x 118800000, size 0x 40000
> [ 1425.518210] mlx4_core 0000:82:00.0: profile[10] ( EQ): 2^10
> entries @ 0x 118840000, size 0x 20000
> [ 1425.518212] mlx4_core 0000:82:00.0: HCA context memory: reserving 4596096 KB
> [ 1425.518228] mlx4_core 0000:82:00.0: 4596096 KB of HCA context
> requires 9068 KB aux memory.
> [ 1425.554670] mlx4_core 0000:82:00.0: Mapped 39 chunks/9068 KB for ICM aux.
> [ 1425.556059] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at 0 for ICM.
> [ 1425.557429] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 40000000 for ICM.
> [ 1425.558795] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 80000000 for ICM.
> [ 1425.559144] mlx4_core 0000:82:00.0: Mapped 1 chunks/64 KB at
> c0000000 for ICM.
> [ 1425.559827] mlx4_core 0000:82:00.0: Mapped 1 chunks/128 KB at
> 118840000 for ICM.
> [ 1425.561182] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 108000000 for ICM.
> [ 1425.562580] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 114000000 for ICM.
> [ 1425.563932] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 110000000 for ICM.
> [ 1425.565306] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118800000 for ICM.
> [ 1425.566666] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 116000000 for ICM.
> [ 1425.568018] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 100000000 for ICM.
> [ 1425.569390] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 117800000 for ICM.
> [ 1425.570754] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 117000000 for ICM.
> [ 1425.572109] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118000000 for ICM.
> [ 1425.573454] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118040000 for ICM.
> [ 1425.574822] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118080000 for ICM.
> [ 1425.576175] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 1180c0000 for ICM.
> [ 1425.577533] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118100000 for ICM.
> [ 1425.578899] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118140000 for ICM.
> [ 1425.580246] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118180000 for ICM.
> [ 1425.581593] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 1181c0000 for ICM.
> [ 1425.582929] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118200000 for ICM.
> [ 1425.584289] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118240000 for ICM.
> [ 1425.585645] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118280000 for ICM.
> [ 1425.587011] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 1182c0000 for ICM.
> [ 1425.588376] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118300000 for ICM.
> [ 1425.589740] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118340000 for ICM.
> [ 1425.591098] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118380000 for ICM.
> [ 1425.592460] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 1183c0000 for ICM.
> [ 1425.593820] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118400000 for ICM.
> [ 1425.595170] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118440000 for ICM.
> [ 1425.596532] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118480000 for ICM.
> [ 1425.597912] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 1184c0000 for ICM.
> [ 1425.599257] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118500000 for ICM.
> [ 1425.600610] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118540000 for ICM.
> [ 1425.601967] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118580000 for ICM.
> [ 1425.603314] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 1185c0000 for ICM.
> [ 1425.604674] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118600000 for ICM.
> [ 1425.606018] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118640000 for ICM.
> [ 1425.607374] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118680000 for ICM.
> [ 1425.608741] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 1186c0000 for ICM.
> [ 1425.610095] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118700000 for ICM.
> [ 1425.611456] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118740000 for ICM.
> [ 1425.612815] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 118780000 for ICM.
> [ 1425.614165] mlx4_core 0000:82:00.0: Mapped 1 chunks/256 KB at
> 1187c0000 for ICM.
> [ 1426.193222] mlx4_core 0000:82:00.0: Started init_resource_tracker: 80 slaves
> [ 1426.193692] mlx4_core 0000:82:00.0: irq 178 for MSI/MSI-X
> [ 1426.193705] mlx4_core 0000:82:00.0: irq 179 for MSI/MSI-X
> [ 1426.200029] mlx4_core 0000:82:00.0: NOP command IRQ test passed
> [ 1426.200605] mlx4_core: Initializing 0000:05:00.1
> [ 1426.200618] mlx4_core 0000:05:00.1: enabling device (0000 -> 0002)
> [ 1426.200654] mlx4_core 0000:05:00.1: Skipping virtual function:1
> [ 1426.200674] mlx4_core: Initializing 0000:82:00.1
> [ 1426.200685] mlx4_core 0000:82:00.1: enabling device (0000 -> 0002)
> [ 1426.200711] mlx4_core 0000:82:00.1: Skipping virtual function:1
> [ 1438.961741] pci-stub 0000:82:00.1: claimed by stub
> [ 1438.967250] pci-stub 0000:82:00.1: enabling device (0000 -> 0002)
> [ 1438.969739] mlx4_core 0000:82:00.0: FLR event for slave: 1
> [ 1438.969775] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr
> [ 1438.969776] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1
> [ 1439.070931] assign device 0:82:0.1
> [ 1439.073469] mlx4_core 0000:82:00.0: FLR event for slave: 1
> [ 1439.073526] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr
> [ 1439.073529] mlx4_core 0000:82:00.0: mlx4_handle_slave_flr: clean slave: 1
> [ 1439.226723] mlx4_core 0000:82:00.0: Received reset from slave:1
> [ 1439.239542] pci-stub 0000:82:00.1: irq 180 for MSI/MSI-X
> [ 1439.272548] pci-stub 0000:82:00.1: irq 180 for MSI/MSI-X
> [ 1439.272559] pci-stub 0000:82:00.1: irq 181 for MSI/MSI-X
> [ 1449.381120] mlx4_core 0000:82:00.0: Received reset from slave:1
>
>
> mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011)
> mlx4_core: Initializing 0000:00:0b.0
> mlx4_core 0000:00:0b.0: enabling device (0000 -> 0002)
> mlx4_core 0000:00:0b.0: Detected virtual function - running in slave mode
> mlx4_core 0000:00:0b.0: Sending reset
> mlx4_core 0000:00:0b.0: Sending vhcr0
> mlx4_core 0000:00:0b.0: 64B EQEs/CQEs supported by the device but not enabled
> mlx4_core 0000:00:0b.0: HCA minimum page size:512
> mlx4_core 0000:00:0b.0: irq 48 for MSI/MSI-X
> mlx4_core 0000:00:0b.0: irq 49 for MSI/MSI-X
> mlx4_core 0000:00:0b.0: failed execution of VHCR_POST commandopcode 0x31
> mlx4_core 0000:00:0b.0: NOP command failed to generate MSI-X interrupt IRQ 49).
> mlx4_core 0000:00:0b.0: Trying again without MSI-X.
> mlx4_core: probe of 0000:00:0b.0 failed with error -16
>
> Mathis
>
> 2013/1/29 Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>:
> > On Tuesday 29 January 2013 15:15, Mathis GAVILLON wrote:
> >> In fact, one boxe wasn't check in IOMMU submenu (Support for Interrupt
> >> Remapping (EXPERIMENTAL)). I've recompiled the kernel of the host and
> >> guest (is it necessary for this last one ?). But the error message is
> >> already present. Just below, new dmeg log from host and guest O/S
> >> (mlx4_core loaded with options num_vfs=1 probe_vf=1 debug_level=1 on
> >> host and debug_level=1 on guest) :
> >>
> > Please do the following:
> > 1. Check the Support for Interrupt Remapping (EXPERIMENTAL) box and recompile/reinstall.
> > 2. set probe_vf=0, so that no vfs will run on the Hypervisor. That way you will
> > see if any VF comes up without the "noise" in /var/log/messages in the Hypervisor generated
> > by having one of the VFs run on the Hypervisor.
> >
> > 3. I noticed that you are enabling only a single VF, and that you are having the
> > VF come up first on the Hypervisor, then moving it to run on a guest.
> > It is better not to do things that way. Best to have it come up cleanly once on
> > the guest. (the FLR event in your log results from this move. The VF gets reset).
> >
> > If you must have a VF running on the Hypervisor, use "num_vfs=2 probe_vf=1".
> > You will then see Devices ...xx.0, ...xx.1 and ...xx.2 in lspci. The first VF
> > (...xx.1) will be initialized on the Hypervisor. Attach ...xx.2 to the VM.
> >
> > -Jack
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2013-01-29 18:36 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-01-23 15:29 Error message when trying to use Infiniband virtual functions in virtual machine Mathis GAVILLON
[not found] ` <CAOoYFkGrsWKGaFcNAQQnkks2Ku6O8xkPk=8BGQfRR74QhRU=VA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-01-24 14:41 ` Or Gerlitz
[not found] ` <CAOoYFkFQOyWbrZtufpk=82ButMxPjsue1Psih40h=F1=U7ng6g@mail.gmail.com>
[not found] ` <CAOoYFkGe=kKDDKvhOdRVCXQbGfV52q38dz6cYTpQXmFGOsBwsg@mail.gmail.com>
[not found] ` <CAOoYFkGe=kKDDKvhOdRVCXQbGfV52q38dz6cYTpQXmFGOsBwsg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-01-28 8:29 ` Fwd: " Mathis GAVILLON
[not found] ` <CAOoYFkFVvK+1YNJ_N6r0N_3ZMF6P9Q+0dbsDg4v8=3JfDPVOOA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-01-28 16:35 ` Jack Morgenstein
[not found] ` <201301281835.18533.jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2013-01-29 13:15 ` Mathis GAVILLON
[not found] ` <CAOoYFkE6YNfEJGJqGmpQTdSxR2Y_9pzkdcnHf_7jjqq57B_Nsw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-01-29 13:20 ` Mathis GAVILLON
2013-01-29 14:33 ` Jack Morgenstein
2013-01-29 14:44 ` Jack Morgenstein
[not found] ` <201301291644.35771.jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2013-01-29 15:18 ` Mathis GAVILLON
[not found] ` <CAOoYFkErXNxSBrQC_spMwcEiVmOpkzqQwuTQPOwO70bigsgYkQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-01-29 18:36 ` Jack Morgenstein [this message]
[not found] ` <201302031040.20906.jackm@dev.mellanox.co.il>
[not found] ` <CAOoYFkHnrty9Nh294y94bWx2PEXHxk8_cVbYbPJ+=jLgf4m+0A@mail.gmail.com>
[not found] ` <CAOoYFkHnrty9Nh294y94bWx2PEXHxk8_cVbYbPJ+=jLgf4m+0A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-02-04 11:23 ` Or Gerlitz
[not found] ` <B855DA7245737D4B83EDD9C0B7698E655952F564-LSMZvP3E4uyuSA5JZHE7gA@public.gmane.org>
2013-02-04 15:24 ` Alex Williamson
[not found] ` <1359991471.11144.396.camel-xdHQ/5r00wBBDLzU/O5InQ@public.gmane.org>
2013-02-05 9:51 ` Or Gerlitz
2013-02-05 10:32 ` Mathis GAVILLON
[not found] ` <CAOoYFkEnzYwPDjka+GGXeXt2xkGhX7qwAptGUvUJChgDEjRBGw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-02-05 11:00 ` Or Gerlitz
[not found] ` <5110E64F.7080106-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2013-02-05 11:45 ` Mathis GAVILLON
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=201301292036.36837.jackm@dev.mellanox.co.il \
--to=jackm-ldsdmyg8hgv8yrgs2mwiifqbs+8scbdb@public.gmane.org \
--cc=jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \
--cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.