Re: Problems using krping

[Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>]

lihaidong wrote:
> Mr.Wise:
>  Sorry to Bother you Another problem.
>  fastreg(Must with local_dma_key to avoid using ib_reg_phys_mr in my 
> case) with read_inv succeed, yet with server_inv failed.
>  
> What does 
> 'krping: cq completion failed with wr_id 0 status 6 opcode -1 vender_err 78'  
> in client dmesg info mean?


Maybe the mlx4 experts can comment on status 6 vender_err 78?


> Why data transferring happened before server waked up from waiting for 
> CONNECTED?


This can happen.  There is a race between getting the CONNECTED event 
and the first incoming data completion (like a recv completion). 


> Client demsg:
> krping: proc write |client,addr=10.10.10.15,port=8888,count=1,verbose,local_dma_lkey,mem_mode=fastreg,server_inv|
> client
> ipaddr (10.10.10.15)
> port 8888
> count 1
> verbose
> using local dma lkey
> created cm_id ffff88013cd11000
> cma_event type 0 cma_id ffff88013cd11000 (parent)
> cma_event type 2 cma_id ffff88013cd11000 (parent)
> Fastreg supported - device_cap_flags 0x7c9c76
> rdma_resolve_addr - rdma_resolve_route successful
> created pd ffff88012a9fff80
> created cq ffff8801305d5400
> created qp ffff8801305d5800
> krping: krping_setup_buffers called on cb ffff88013cd11800
> krping: fastreg rkey 0x88001a00 page_list ffff88012b3cee80 page_list_len 1
> krping: allocated & registered buffers...
> cma_event type 9 cma_id ffff88013cd11000 (parent)
> ESTABLISHED
> rdma_connect successful
> krping: page_list[0] 0x12a40c000
> krping: post_inv = 0, fastreg new rkey 0x88001a01 shift 12 len 64 iova_start 12a40c180 page_list_len 1
> RDMA addr 12a40c180 rkey 88001a01 len 64
> krping: cq completion failed with wr_id 0 status 6 opcode -1 vender_err 78
> krping: cq completion in ERROR state
> krping: krping_format_send failed
> krping_free_buffers called on cb ffff88013cd11800
> destroy cm_id ffff88013cd11000
>  
> Server dmesg:
> krping: proc write |server,addr=10.10.10.15,port=8888,count=1,verbose,local_dma_lkey,mem_mode=fastreg,server_inv|
> server
> ipaddr (10.10.10.15)
> port 8888
> count 1
> verbose
> using local dma lkey
> created cm_id ffff88007e3c5c00
> rdma_bind_addr successful
> rdma_listen
> cma_event type 4 cma_id ffff88003e0dac00 (child)
> child cma ffff88003e0dac00
> Fastreg supported - device_cap_flags 0x7c9c76
> created pd ffff880035d17d20
> created cq ffff88001e0f9e00
> created qp ffff88001e0f9c00
> krping: krping_setup_buffers called on cb ffff88007e2b0000
> krping: fastreg rkey 0x68001d00 page_list ffff8800027ab580 page_list_len 1
> krping: allocated & registered buffers...
> accepting client connection request
> cma_event type 9 cma_id ffff88003e0dac00 (child)
> ESTABLISHED
> cma_event type 10 cma_id ffff88003e0dac00 (child)
> krping: DISCONNECT EVENT...
> krping: wait for CONNECTED state 10
> krping: connect error -1
> krping_free_buffers called on cb ffff88007e2b0000
> destroy cm_id ffff88007e3c5c00
> 2010-01-22
> ------------------------------------------------------------------------
> lihaidong
> ------------------------------------------------------------------------
> *发件人：* Steve Wise
> *发送时间：* 2010-01-21  06:43:30
> *收件人：* lihaidong
> *抄送：* linux-rdma
> *主题：* Re: Problems using krping
> It appears the MLX4 driver does not support kernel mode memory regions.  
> You'll have to use dma mrs or fast_reg mrs with that device.
> Steve.
> lihaidong wrote:
> > Mr.Wise:
> >  
> > When using mr mode as the memory registration method, krping failed to 
> > get memory region using ib_reg_phys_mr().Could you help me, please?
> >  
> > dmesg:
> > krping_init
> > krping: proc write |client,addr=10.10.10.15,mem_mode=mr,port=9999,count=1,verbose|
> > client
> > ipaddr (10.10.10.15)
> > port 9999
> > count 1
> > verbose
> > created cm_id ffff88013c74c800
> > cma_event type 0 cma_id ffff88013c74c800 (parent)
> > cma_event type 2 cma_id ffff88013c74c800 (parent)
> > rdma_resolve_addr - rdma_resolve_route successful
> > created pd ffff880133845280
> > created cq ffff88013c1ff400
> > created qp ffff88013c1ffe00
> > krping: krping_setup_buffers called on cb ffff88013c59f800
> > krping: recv buf dma_addr 13c59f968 size 16
> > krping: recv_buf reg_mr failed
> > krping: krping_setup_buffers failed: -38
> > destroy cm_id ffff88013c74c800
> > krping: proc write |client,addr=10.10.10.15,mem_mode=mr,port=9999,count=1,verbose|
> > client
> > ipaddr (10.10.10.15)
> > port 9999
> > count 1
> > verbose
> > created cm_id ffff88013c59f800
> > cma_event type 0 cma_id ffff88013c59f800 (parent)
> > cma_event type 2 cma_id ffff88013c59f800 (parent)
> > rdma_resolve_addr - rdma_resolve_route successful
> > created pd ffff88012f5964a0
> > created cq ffff88012faee400
> > created qp ffff88012faeec00
> > krping: krping_setup_buffers called on cb ffff88013c71d400
> > krping: recv buf dma_addr 13c71d568 size 16
> > krping: recv_buf reg_mr failed
> > krping: krping_setup_buffers failed: -38
> > destroy cm_id ffff88013c59f800
> >  
> >  
> > echo "client,addr=10.10.10.15,mem_mode=mr,port=9999,count=1" > /proc/krping 
> >
> > echo "server,addr=10.10.10.15,mem_mode=mr,port=9999" > /proc/krping
> >  
> > Using OFED-1.5 ofa_kernel-1.5
> > HardWare:Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] 
> >
> > put krping source files into drivers/infiniband/hw/mlx4
> >  
> > 2010-01-21
> > ------------------------------------------------------------------------
> > lihaidong
> __________ Information from ESET NOD32 Antivirus, version of virus signature database 4788 (20100120) __________
> The message was checked by ESET NOD32 Antivirus.
> http://www.eset.com

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html