From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yann Droneaud Subject: Re: (R)DMA in userspace Date: Fri, 12 Oct 2012 16:55:40 +0200 Message-ID: <1350053740.9068.1.camel@test.quest-ce.net> References: <1350033163.2291.22.camel@test.quest-ce.net> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: In-Reply-To: <1350033163.2291.22.camel-sQn2kEGNn0pFevvuwOF9vF6hYfS7NtTn@public.gmane.org> Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Cc: Roland Dreier , Animesh K Trivedi1 , Bernard Metzler List-Id: linux-rdma@vger.kernel.org Le vendredi 12 octobre 2012 =C3=A0 11:12 +0200, Yann Droneaud a =C3=A9c= rit : > Hi, >=20 > Le jeudi 11 octobre 2012 =C3=A0 13:44 -0700, Roland Dreier a =C3=A9cr= it : > > On Thu, Oct 11, 2012 at 8:34 AM, Animesh K Trivedi1 wrote: > > > > > > During memory memory registration, userspace buffers also go thro= ugh same > > > API calls (dma_map_sg_attrs(...)). > > > What I am confused about why no such synchronization primitives a= re > > > required in userspace before accessing > > > an RDMA data buffer just after when an incoming write/recv (DMA o= n it) is > > > finished? Who guarantees data > > > freshness? > >=20 > > No one has really ever tried to deal with the issue of userspace RD= MA on > > a cache-incoherent architecture. Basically if you try the current = stack, the > > in-kernel users (IPoIB etc) should be OK but libibverbs etc. will b= e completely > > broken. > >=20 >=20 > With the current ARMv7 Cortex-A9 / Cortex-A15 MPCore and the upcoming > ARM 64 bits architecture eg ARMv8 aka Aarch64, one might want in the > near future use RDMA (InfiniBand/RoCE) with them to create highly > parallel system with low-power consumption.=20 >=20 > But this question and some reading about ARM memory management makes = me > feel pretty unsure of the ability to use RDMA (InfiniBand) on ARM. >=20 > In this article: "ARM, DMA, and memory management" > http://lwn.net/Articles/440221/"=20 > it is said that a memory page must not be mapped multiple time with > different caching attributes. >=20 > This article take the point of Linaro's developers who want to upload > texture to the GPU without holding them in caches. This behavior migh= t > also be applicable to RDMA as well: writing to a memory zone to be > either local IBV_WR_SEND, local IBV_WR_RDMA_WRITE or remote > IBV_WR_RDMA_READ, there's probably no need to keep it in cache. >=20 > You could also read this other article "CMA [contiguous memory > allocator] and ARM" http://lwn.net/Articles/450286/ > and "ARM's multiply-mapped memory mess" http://lwn.net/Articles/40968= 9/ >=20 > After reading this, and not being an ARM expert, I'm asking myself ab= out > a possible RDMA (InfiniBand) support on ARM. >=20 I've found more information in "Implementing DMA on ARM SMP Systems", Application Note 228 http://infocenter.arm.com/help/index.jsp?topic=3D/com.arm.doc.dai0228a/= index.html ARMv7 Cortex-A9 seems to be a cache-coherent architecture. Regards. --=20 Yann Droneaud OPTEYA -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" i= n the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html