Re: back to back RDMA read fail?

[Dotan Barak <dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]

I have 2 questions:
1) if you change the opcode to RDMA Write, do you still experience this 
problem?
    (assuming that the permissions allow RDMA Write; if not, fix this issue)

2) what are the values of the the outstanding RDMA Read/Atomic in both 
QPs (as initiator and as target)?

Dotan

neutron wrote:
> On Wed, Nov 11, 2009 at 4:52 AM, Dotan Barak <dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>   
>> Hi.
>>
>> how do you connect the QPs?
>> via CM/CMA or by sockets (and you actually call the ibv_modify_qp)?
>>
>>     
>
> I exchange the initial QP infortion (lid, qpn, psn) via sockets.  No
> CM is used. I manually take are of everything.
>
> Thanks!
>
>   
>> Dotan
>>
>> neutron wrote:
>>     
>>> Hi Paul, thanks a lot for your quick reply!
>>>
>>> In my test,  client informs the server of its local memory (rkey,
>>> addr, size) by sending 4 back to back messages,  each message elicits
>>> a RDMA read request (RR) from the server.
>>>
>>> In other words, client exposes its memory to the server, and server
>>> RDMA reads it.
>>>
>>> As far as RDMA read is concerned, server is a requester, and client is
>>> a responder, right?
>>>
>>> The error I encountered happens at the initial phase, when client
>>> sends 4 back to back messages to server(using ibv_post_send ),
>>> containing (rkey, addr, size) client's local memory.
>>>
>>> In these 4 ibv_post_send(), client will see one failure.   At server
>>> side, server has already posted enough WQs in the RQ.  The failures
>>> are included in my first email.
>>>
>>> Looking at the program output, it appears that, server gets messages
>>> 1, issues RR 1, gets message 2, issues RR 2.    But somehow client
>>> reports that "send message 2" fails.
>>>
>>> On the contrary, server reports "receive message 3" fails.
>>>
>>> As a result, server gets message 1,2,4, and succeeds with RR 1,2,4.
>>> But clients sees that message 2 fails, and succeed with message 1,3,4.
>>>  This inconsistency is the problem that puzzled me.
>>>
>>> ------------
>>> By the way, how to interpret the parameters for RDMA, and what are
>>> parameters that control RDMA behavior?  Below are something I can
>>> find, there must be more....
>>>
>>>   max_qp_rd_atom:                 4
>>>   max_res_rd_atom:                258048
>>>   max_qp_init_rd_atom:            128
>>>
>>>   qp_attr.max_dest_rd_atomic
>>>   qp_attr.max_rd_atomic
>>>
>>>
>>>
>>> -neutron
>>>
>>>
>>>
>>> On Tue, Nov 10, 2009 at 2:04 AM, Paul Grun <pgrun-klaOcWyJdxkshyMvu7JE4pqQE7yCjDx5@public.gmane.org>
>>> wrote:
>>>
>>>       
>>>> Is it possible that you exceeded the number of available RDMA Read
>>>> Resources
>>>> available on the server?  There is an expectation that the client knows
>>>> how
>>>> many outstanding RDMA Read Requests the responder (server) is capable of
>>>> handling; if the requester (client) exceeds that number, the responder
>>>> will
>>>> indeed return a NAK-Invalid Request.  Sounds like your server is
>>>> configured
>>>> to accept three outstanding RDMA Read Requests.
>>>> This also explains why it works when you pause the program
>>>> periodically...it
>>>> gives the responder time to generate the RDMA Read Responses and
>>>> therefore
>>>> free up some resources to be used in receiving the next incoming RDMA
>>>> Read
>>>> Request.
>>>>
>>>> -Paul
>>>>
>>>> -----Original Message-----
>>>> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>> [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of neutron
>>>> Sent: Monday, November 09, 2009 9:04 PM
>>>> To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>> Subject: back to back RDMA read fail?
>>>>
>>>> Hi all,
>>>>
>>>> I have a simple program that test back to back RDMA read performance.
>>>> However I encountered errors for unknown reasons.
>>>>
>>>> The basic flow of my program is:
>>>>
>>>> client:
>>>> ibv_post_send() to send 4 back to back messages to server (no delay
>>>> inbetween). Each message contains the (rkey, addr, size) of a local
>>>> buffer. The buffer is registered with remote-read/write/ permissions.
>>>> After that, ibv_poll_cq() is called to wait for completion.
>>>>
>>>> server:
>>>> First, enough receive WRs are posted to the RQ.  Upon receipt of each
>>>> message, immediately post a RDMA read request, using the (rkey, addr,
>>>> size) information contained in the originating message.
>>>>
>>>> --------------
>>>> Both client and server use RC QP.  Some errors are observed.
>>>>
>>>> On client side,  ibv_poll_cq() gets 4 CQE, one out of the 4 CQE is an
>>>> error:
>>>> CQ::  wr_id=0x0, wc_opcode=IBV_WC_SEND, wc_status=remote invalid RD
>>>> request, wc_flag=0x3b
>>>>     byte_len=11338758, immdata=1110104528, qp_num=0x0, src_qp=2290530758
>>>>
>>>> The other 3 CQE are success.
>>>>
>>>> On server side,
>>>> 3 of the 4 messages are successfully received. One message produces an
>>>> error CQE:
>>>> CQ::  wr_id=0x8000000000, wc_opcode=Unknow-wc-opcode,
>>>> wc_status=unknown, wc_flag=0x0
>>>>     byte_len=9569287, immdata=0, qp_num=0x0, src_qp=265551872
>>>>
>>>> The 3 RDMA read corresponding to the successful receive all succeed.
>>>>
>>>> But, if I pause the client program for a short while( usleep(100) for
>>>> example ) after calling ibv_post_send(), then no error occurs.
>>>> Anyone can point out the pitfall here? Thanks!
>>>>
>>>>
>>>> -----------
>>>> On both client and server, I'm using  'mthca0' type MT25208.  The QPs
>>>> are initialized with "qp_attr.max_dest_rd_atomic=4,
>>>> qp_attr.max_rd_atomic = 4".  The QP's "devinfo -v" gives the
>>>> information:
>>>>
>>>> hca_id: mthca0
>>>>       fw_ver:                         5.1.400
>>>>       node_guid:                      0002:c902:0023:c04c
>>>>       sys_image_guid:                 0002:c902:0023:c04f
>>>>       vendor_id:                      0x02c9
>>>>       vendor_part_id:                 25218
>>>>       hw_ver:                         0xA0
>>>>       board_id:                       MT_0370130002
>>>>       phys_port_cnt:                  2
>>>>       max_mr_size:                    0xffffffffffffffff
>>>>       page_size_cap:                  0xfffff000
>>>>       max_qp:                         64512
>>>>       max_qp_wr:                      16384
>>>>       device_cap_flags:               0x00001c76
>>>>       max_sge:                        27
>>>>       max_sge_rd:                     0
>>>>       max_cq:                         65408
>>>>       max_cqe:                        131071
>>>>       max_mr:                         131056
>>>>       max_pd:                         32764
>>>>       max_qp_rd_atom:                 4
>>>>       max_ee_rd_atom:                 0
>>>>       max_res_rd_atom:                258048
>>>>       max_qp_init_rd_atom:            128
>>>>       max_ee_init_rd_atom:            0
>>>>       atomic_cap:                     ATOMIC_HCA (1)
>>>>       max_ee:                         0
>>>>       max_rdd:                        0
>>>>       max_mw:                         0
>>>>       max_raw_ipv6_qp:                0
>>>>       max_raw_ethy_qp:                0
>>>>       max_mcast_grp:                  8192
>>>>       max_mcast_qp_attach:            56
>>>>       max_total_mcast_qp_attach:      458752
>>>>       max_ah:                         0
>>>>       max_fmr:                        0
>>>>       max_srq:                        960
>>>>       max_srq_wr:                     16384
>>>>       max_srq_sge:                    27
>>>>       max_pkeys:                      64
>>>>       local_ca_ack_delay:             15
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>>>
>>>>
>>>>         
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>       
>>     

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html