From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from foss.arm.com (foss.arm.com [217.140.101.70]) by lists.ozlabs.org (Postfix) with ESMTP id 409RZR6Bt5zF2B8 for ; Tue, 27 Mar 2018 21:09:35 +1100 (AEDT) Date: Tue, 27 Mar 2018 11:09:44 +0100 From: Will Deacon To: Arnd Bergmann Cc: Benjamin Herrenschmidt , Jason Gunthorpe , Sinan Kaya , David Laight , Oliver , "open list:LINUX FOR POWERPC (32-BIT AND 64-BIT)" , "linux-rdma@vger.kernel.org" , "Paul E. McKenney" , Peter Zijlstra , Ingo Molnar , Jonathan Corbet Subject: Re: RFC on writel and writel_relaxed Message-ID: <20180327100944.GD29373@arm.com> References: <20180326210951.GD15554@ziepe.ca> <1522101717.7364.14.camel@kernel.crashing.org> <20180326222756.GJ15554@ziepe.ca> <1522141019.7364.43.camel@kernel.crashing.org> <20180327095745.GB29373@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On Tue, Mar 27, 2018 at 12:05:06PM +0200, Arnd Bergmann wrote: > On Tue, Mar 27, 2018 at 11:57 AM, Will Deacon wrote: > > > > > From db0daeaf94f0f6232f8206fc07a74211324b11d9 Mon Sep 17 00:00:00 2001 > > From: Will Deacon > > Date: Tue, 27 Mar 2018 10:49:58 +0100 > > Subject: [PATCH] docs/memory-barriers.txt: Fix broken DMA vs MMIO ordering > > example > > > > The section of memory-barriers.txt that describes the dma_Xmb() barriers > > has an incorrect example claiming that a wmb() is required after writing > > to coherent memory in order for those writes to be visible to a device > > before a subsequent MMIO access using writel() can reach the device. > > > > In fact, this ordering guarantee is provided (at significant cost on some > > architectures such as arm and power) by writel, so the wmb() is not > > necessary. writel_relaxed exists for cases where this ordering is not > > required. > > > > Fix the example and update the text to make this clearer. > > > > Cc: Benjamin Herrenschmidt > > Cc: Arnd Bergmann > > Cc: Jason Gunthorpe > > Cc: "Paul E. McKenney" > > Cc: Peter Zijlstra > > Cc: Ingo Molnar > > Cc: Jonathan Corbet > > Reported-by: Sinan Kaya > > Signed-off-by: Will Deacon > > --- > > Documentation/memory-barriers.txt | 18 ++++++++++-------- > > 1 file changed, 10 insertions(+), 8 deletions(-) > > > > diff --git a/Documentation/memory-barriers.txt b/Documentation/memory-barriers.txt > > index a863009849a3..2556b4b0e6f9 100644 > > --- a/Documentation/memory-barriers.txt > > +++ b/Documentation/memory-barriers.txt > > @@ -1909,9 +1909,6 @@ There are some more advanced barrier functions: > > /* assign ownership */ > > desc->status = DEVICE_OWN; > > > > - /* force memory to sync before notifying device via MMIO */ > > - wmb(); > > - > > /* notify device of new descriptors */ > > writel(DESC_NOTIFY, doorbell); > > } > > @@ -1919,11 +1916,16 @@ There are some more advanced barrier functions: > > The dma_rmb() allows us guarantee the device has released ownership > > before we read the data from the descriptor, and the dma_wmb() allows > > us to guarantee the data is written to the descriptor before the device > > - can see it now has ownership. The wmb() is needed to guarantee that the > > - cache coherent memory writes have completed before attempting a write to > > - the cache incoherent MMIO region. > > - > > - See Documentation/DMA-API.txt for more information on consistent memory. > > + can see it now has ownership. Note that, when using writel(), a prior > > + wmb() is not needed to guarantee that the cache coherent memory writes > > + have completed before writing to the cache incoherent MMIO region. > > + If this ordering between incoherent MMIO and coherent memory regions > > + is not required, writel_relaxed() can be used instead and is significantly > > + cheaper on some weakly-ordered architectures. > > I think that's a great improvement, but I'm a bit worried about recommending > writel_relaxed() too much: I've seen a lot of drivers that just always use > writel_relaxed() over write(), and some of them get that wrong when they > don't understand the difference but end up using DMA without explicit > barriers anyway. > > Also, having an architecture-independent driver use wmb()+writel_relaxed() > ends up being more expensive than just using write(). Not sure how to > best phrase it though. Perhaps I add reword that with a simple example to say: If this ordering between incoherent MMIO and coherent memory regions is not required (e.g. in a sequence of accesses all to the MMIO region) [...] since that seems to be the usual case where the _relaxed accessors help. Will