* Re: Xen and Infiniband as dom0 (only)
2010-03-16 15:56 ` Pasi Kärkkäinen
@ 2010-03-26 10:48 ` Xavier Beaudouin
0 siblings, 0 replies; 3+ messages in thread
From: Xavier Beaudouin @ 2010-03-26 10:48 UTC (permalink / raw)
To: Pasi Kärkkäinen; +Cc: xen-devel
Hi there,
Le 16 mars 2010 à 16:56, Pasi Kärkkäinen a écrit :
> On Mon, Mar 15, 2010 at 01:29:54PM +0100, Xavier Beaudouin wrote:
>> Hello there,
>>
>> I am trying to make infiniband working on a dom0. To allow me fast access to storage, and fast migration between hypervisor.
>>
>> I still don't want to do Ib as Hypervisor Accelerator and specialy what is shown on slide 12 of this paper : http://xen.org/files/Xen_Virtualization_InfiniBand_Mellanox_MKagan.pdf, but I need rdma to get fast access to storage :
>> - iSer (eg iscsi over infiniband)
>> - RDMA for NFS over RDMA or stuff like this.
>>
>> On my host, without Xen (F12, with myoung kernel backported, xen 4.0.0-rc6 and OFED 1.5.1-RC4), I can use correctly ipoib (IP over IB), but rdma is still broken.
>>
>> Is there any people that has made dom0 IB working that can me some hints about what I have to fix ?
>>
>
> Well what actually is the problem?
> What kind of errors do you get?
Simple :
- 192.168.5.202 is the server (filer) with IB and OFED 1.5.1-rc5
$ ibv_rc_pingpong
local address: LID 0x0007, QPN 0x590406, PSN 0x1adab0
remote address: LID 0x000c, QPN 0x0a0406, PSN 0x877182
- client is xen-6 (192.168.5.226) :
$ ibv_rc_pingpong 192.168.5.202
local address: LID 0x000c, QPN 0x0a0406, PSN 0x877182
remote address: LID 0x0007, QPN 0x590406, PSN 0x1adab0
Message from syslogd@xen-6 at Mar 26 12:41:56 ...
kernel:Bad pagetable: 000f [#1] SMP
Message from syslogd@xen-6 at Mar 26 12:41:56 ...
kernel:last sysfs file: /sys/devices/pci0000:00/0000:00:0f.0/0000:04:00.0/infiniband/mthca0/node_type
Killed
dmesg on the host :
ibv_rc_pingpong: Corrupted page table at address 7fa3eccc1010
PGD 58f1d067 PUD 58927067 PMD 53e50067 PTE fffffffffffff237
Bad pagetable: 000f [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:0f.0/0000:04:00.0/infiniband/mt
hca0/node_type
CPU 0
Modules linked in: tun bridge stp llc nfs lockd fscache nfs_acl auth_rpcgss sunr
pc rdma_ucm ib_sdp rdma_cm iw_cm ib_addr ib_ipoib ib_cm ib_sa ipv6 ib_uverbs ib_
umad iw_nes libcrc32c iw_cxgb3 cxgb3 mlx4_ib mlx4_en mlx4_core microcode xen_evt
chn xenfs ib_mthca ib_mad forcedeth amd64_edac_mod i2c_nforce2 ib_core edac_core
i2c_core edac_mce_amd xfs exportfs dm_multipath pata_acpi ata_generic pata_amd
[last unloaded: scsi_wait_scan]
Pid: 4402, comm: ibv_rc_pingpong Tainted: G W 2.6.32.10-1.2.92.xendom0.f
c12.x86_64 #1 empty
RIP: e033:[<00007fa3ec4a3f42>] [<00007fa3ec4a3f42>] 0x7fa3ec4a3f42
RSP: e02b:00007fff80f16ca0 EFLAGS: 00010202
RAX: 000000000a000001 RBX: 0000000000000000 RCX: 00007fa3eccc1000
RDX: 0206040a0a000001 RSI: 00007fa3eccbb008 RDI: 0000000000000000
RBP: 0000000001bdc980 R08: 0000000000000000 R09: 0000000000000007
R10: 0000000000590406 R11: 0000000000000246 R12: 00007fa3ec49f010
R13: 0000000001be3170 R14: 0000000000000001 R15: 00007fff80f17000
FS: 00007fa3eccb8700(0000) GS:ffff880003e7d000(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fa3eccc1010 CR3: 0000000002a12000 CR4: 0000000000000660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ibv_rc_pingpong (pid: 4402, threadinfo ffff88004f9ea000, task ffff88005b
a19760)
RIP [<00007fa3ec4a3f42>] 0x7fa3ec4a3f42
RSP <00007fff80f16ca0>
---[ end trace 4eaa2a86a8e2da24 ]---
Note that I have same problems with intel platform as well...
ipoib works, but I really like to get rid of these errors....
Xavier
--
Xavier Beaudouin - xb@soprive.net
GPG Fingerprints : A6B2 D563 F93B A3AF C08A CBAC 6BC6 79EB DCC9 9867
^ permalink raw reply [flat|nested] 3+ messages in thread