Re: (R)DMA in userspace

[Yann Droneaud <ydroneaud-RlY5vtjFyJ3QT0dZR+AlfA@public.gmane.org>]

Le vendredi 12 octobre 2012 à 11:12 +0200, Yann Droneaud a écrit :
> Hi,
> 
> Le jeudi 11 octobre 2012 à 13:44 -0700, Roland Dreier a écrit :
> > On Thu, Oct 11, 2012 at 8:34 AM, Animesh K Trivedi1 <ZRLATR-0BZ7OvD7h6o@public.gmane.orgcom> wrote:
> > >
> > > During memory memory registration, userspace buffers also go through same
> > > API calls (dma_map_sg_attrs(...)).
> > > What I am confused about why no such synchronization primitives are
> > > required in userspace before accessing
> > > an RDMA data buffer just after when an incoming write/recv (DMA on it) is
> > > finished? Who guarantees data
> > > freshness?
> > 
> > No one has really ever tried to deal with the issue of userspace RDMA on
> > a cache-incoherent architecture.  Basically if you try the current stack, the
> > in-kernel users (IPoIB etc) should be OK but libibverbs etc. will be completely
> > broken.
> > 
> 
> With the current ARMv7 Cortex-A9 / Cortex-A15 MPCore and the upcoming
> ARM 64 bits architecture eg ARMv8 aka Aarch64, one might want in the
> near future use RDMA (InfiniBand/RoCE) with them to create highly
> parallel system with low-power consumption. 
> 
> But this question and some reading about ARM memory management makes me
> feel pretty unsure of the ability to use RDMA (InfiniBand) on ARM.
> 
> In this article: "ARM, DMA, and memory management"
> http://lwn.net/Articles/440221/" 
> it is said that a memory page must not be mapped multiple time with
> different caching attributes.
> 
> This article take the point of Linaro's developers who want to upload
> texture to the GPU without holding them in caches. This behavior might
> also be applicable to RDMA as well: writing to a memory zone to be
> either local IBV_WR_SEND, local IBV_WR_RDMA_WRITE or remote
> IBV_WR_RDMA_READ, there's probably no need to keep it in cache.
> 
> You could also read this other article "CMA [contiguous memory
> allocator] and ARM" http://lwn.net/Articles/450286/
> and "ARM's multiply-mapped memory mess" http://lwn.net/Articles/409689/
> 
> After reading this, and not being an ARM expert, I'm asking myself about
> a possible RDMA (InfiniBand) support on ARM.
> 

I've found more information in "Implementing DMA on ARM SMP Systems",
Application Note 228
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dai0228a/index.html

ARMv7 Cortex-A9 seems to be a cache-coherent architecture.

Regards.

-- 
Yann Droneaud
OPTEYA



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html