Pro/Cons of send/receive/read/write operations?

Pro/Cons of send/receive/read/write operations?

[ib-x2spCj9RiN0z5UmgcLIfJQ]

It seems there are several ways to transfer data via RDMA: Send/Send  
w/immediate, Receive, RDMA Write/RDMA Write w/immediate, the Atomic  
extensions, and then the various transport modes variations with some  
of these operations.  As an RDMA/IB newbie, it is somewhat confusing  
when determining the best operations for my application.  What are the  
various pros/cons of each?

For my application I will have multiple apps (probably around 4 for  
now) sending data to a single app.  I have 3 sets of data, packets if  
you like, that need to be sent, or written to, the target app.  Packet  
1, is small, only 16 bytes. Packet 2, will be a minimum of 4 MB and  
will likely be as large 16 MB, perhaps more.  Packet 3, will be a max,  
probably, of 4 MB.

Thanks,
CB

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Pro/Cons of send/receive/read/write operations?

[Jason Gunthorpe]

On Wed, Nov 10, 2010 at 10:14:16AM -0700, ib-x2spCj9RiN0z5UmgcLIfJQ@public.gmane.org wrote:

> It seems there are several ways to transfer data via RDMA: Send/Send
> w/immediate, Receive, RDMA Write/RDMA Write w/immediate, the Atomic
> extensions, and then the various transport modes variations with
> some of these operations.  As an RDMA/IB newbie, it is somewhat
> confusing when determining the best operations for my application.
> What are the various pros/cons of each?

Well, there really are quite a few trade offs, typically the main
issue for RDMA systems should be in managing the data buffers across
the whole application. A big cost in RDMA is registering buffers so
you want to design a scheme that minimizes how often that is done.

Some basic patterns you can consider:
 - SEND is used for small messaging and one of the small messages is
   'I want to send you a large message, give me parameters for RDMA
    WRITE'
 - Or above with 'here is where you can RDMA READ' from. Writes
   are better than reads, as there are HW limits on parallel read
   requests.
 - You use SEND for everything and allocate 16MB recv buffers
 - You construct a stream semantic (like TCP) using RDMA_WRITE. This
   is useful if the data is consumed in some way and then discarded
   right away.

You can start by reflecting on where the data is coming from and where
it is headed and use that to devise a buffer management strategy.

The choice of SEND/READ/WRITE will be more evident once you understand
how you want your data to flow. 

You almost always want to use RC connections, and if you needed to use
Atomic's you'd probably know already :)

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html