From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jason Gunthorpe Subject: Re: (R)DMA in userspace Date: Fri, 12 Oct 2012 17:10:28 -0600 Message-ID: <20121012231028.GC25541@obsidianresearch.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Or Gerlitz Cc: Roland Dreier , Animesh K Trivedi1 , linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Bernard Metzler List-Id: linux-rdma@vger.kernel.org On Thu, Oct 11, 2012 at 11:04:02PM +0200, Or Gerlitz wrote: > On Thu, Oct 11, 2012 at 10:44 PM, Roland Dreier wrote: > > > No one has really ever tried to deal with the issue of userspace > > RDMA on a cache-incoherent architecture. Basically if you try the > > current stack, the in-kernel users (IPoIB etc) should be OK but > > libibverbs etc. will be completely broken. > > I think the question might refer even to cache-coherent systems, e.g > in the kernel IB core and ULPs all buffers are dma mapped to/from the > device before/after they are touched by the CPU and vise versa, wheres > in user space, after the buffers are registered once, they are > repeatedly touched by the CPUs and provided to the HW for DMA, e.g all > user space buffers are treated like kernel DMA coherent ones. The answer is the same, userspace is designed to rely on a DMA cache coherent platform where the only requirement is to issue barrier instructions, which is done in the providers. I'm not sure supporting non-DMA-coherent is even possible with the verbs API, yes we could add the cache ops to the providers, the information is mostly there. However non-DMA-coherent system all require that once you start a DMA WRITE into a cache line *that line is never dirtied by the CPU* - which requires application support that is not even contemplated by verbs. Indeed, I wonder if all the kernel ULPs meet that restriction? Hopefully the forthcoming server grade ARMs are fully DMA coherent.. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html