Re: Fw: About FRMR vec and SRQ post recv

Re: Fw: About FRMR vec and SRQ post recv

[Steve Wise]

lihaidong wrote:
> Sorry.
> sge[1]= {vec1_addr, vec1_length, dma_lkey};
> should be
> sge[1]= {vec2_addr, vec2_length, dma_lkey};
> >Hi, Steve.
> >I have the following questions.
> >1)FRMR
> >I want transfer the data in vec1 and vec2 in node A to the two vecs
> with the same size in node B. Vecs' size is lower than a page.
> >A B
> >+--------+ +--------+
> >| vec1 | | vec1 |
> >+--------+ +--------+
> >
> >+--------+ +--------+
> >| vec2 | | vec2 |
> >+--------+ +--------+
> >
> >Could I use FRMR to register vec1 and vec2 as one Memory region?


yes. There are certain contstraints though. For example, vec1 has to be
physically contiguous and end on a page boundary, and vec2 would have to
start on a page boundary and be physically contiguous. IE fastreg mrs
are typically used to register an array of pages (not necessarily all
contiguous pages), not arbitrary regions.


> >If so, then they have the same rkey, and I could send them to remote
> peer for one time,right?



Yes, if you have the two vecs fast registered as one mr, then you could
send it using a single work request and even a single SGE in that work
request.



> >Or I have to register and send vec1 and vec2 seperately?
> >
> >If I use Send/Recv not RDMA Read/Write to send them. ie. I will use
> dma_mr's lkey/rkey, and let
> >sge[0]= {vec1_addr, vec1_length, dma_lkey};
> >sge[1]= {vec1_addr, vec1_length, dma_lkey};
> >ib_send_wr.sgl_list= sge;
> >

Not exactly. The result of fast registering those two regions would be a
single iova_base (aka the addr field in an SGE) and [lr]key. Those would
be used to describe the data in a send wr, probably as a single SGE.

It sounds like what you really need is either to use a dma mr or to
register the regions separately. Unless those vecs meet the criteria I
mentioned above..


> >and post send this wr. Is it allowed?
> >2) SRQ
> >SRQ's size is 1024, and I use ib_post_srq_recv 1024 times after
> getting SRQ created.
> >When cq_handler get a IB_WC_RECV event, do I need post another recv
> wr?(since in the case without SRQ, it would be yes)


I don't know much about SRQs, maybe someone else knows how these are
implemented.



Steve
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fw: About FRMR vec and SRQ post recv

[Steve Wise]

> I try getting a summary. Give some comments on the following
> sentences,please.
> For local side(A), where data to be transferred is placed.
> send or RDMA Read/Write operations.
> the sge array in one send wr pointed by sgl_list should have the same
> [lr]key.
> a) one send wr could send two arbitrary regions when using dma mr's
> [lr]ley or local_dma_lkey
> b) one send wr could 'not' send two arbitrary regions registered by
> two FRWRs seperately.


b) is not true. If you register these 2 regions separately, then you can
certainly send them in one send wr, each being an sge entry in the wr.


>
> For remote side(B), where data transferred will be placed.
> 2) RDMA Read/Write operations could put data to
> a) a single physical continous region registerd
> b) two or more regions with contstraints that they could be registerd
> by one FRWR.
> c) could not put data to two regions registerd by two FRWR sperately
> since in side A, ib_send_wr.wr.rdma should specify one group
> {rkey,addr} for remote destination region.


This is all correct: The sink (aka the target) of a write or read is a
single rkey/addr/len. So to have it place into two separate regions, you
would need them to have been registered via fastreg-mr and thus they
would have to adhere to the constraints of a fastreg-mr.


>
> and send/recv operatons can also put data to
> c) two or more arbitrary regions using dma mr. since send operation in
> side A do not need remote {addr,rkey} and recv operation in side B
> could post two arbitrary regions using dma mr's lkey or local_dma_lkey
> in a singel recv_wr.
>

This is correct also. incoming sends will be placed into recv buffers as
specified in the sg list for the recv wr.


>
> I want to transfer data from two arbitrary regions of A to two
> arbitrary regions of B for a single RDMA Read/Write operation
> originally.And it seems impossible now.
>

you'll have to issue multiple reads or writes...


Steve
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fw: About FRMR vec and SRQ post recv

[Steve Wise]

lihaidong wrote:
> If I use send/recv operations to send the two arbitrary regions form A
> to B.How about the data order?
> I mean
> for side A
> sge[0] = {A_vec1_addr,A_vec1_len, A_lkey}
> sge[1] = {A_vec2_addr,A_vec2_len, A_lkey}
> ib_send_wr.sg_list = sge;
> for side B
> sge[0] = {B_vec1_addr,B_vec1_len, B_lkey}
> sge[1] = {B_vec2_addr,B_vec2_len, B_lkey}
> ib_recv_wr.sg_list = sge;
> Does lower level layers ensure that data in vec1 in A would be placed
> to vec1 in B and vec2(A) to vec2(B)?


Yes. as long as the lengths match. IE on the recv side, the data is
placed into the receive buffer in sge order based on lengths. So if A's
sge[0] was 100 bytes and sge[1] 10 bytes, and B's recv sge[0] was 110
bytes, all the 110 bytes would be placed in B's recv sge[0].



> According to krping and my experience, to use send/recv operations, I
> have to convert host byte order to net byte order when preparing
> send_buf, and convert back to host byte order when recv_buf's data is
> ready.Yet, RDMA Read/Write does not has this overhead,right?


I don't really understand this question: If you're talking about payload
data, then you don't need to do conversions if both hosts are the same
byte order.


Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html