From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <benh@kernel.crashing.org>
Received: from gate.crashing.org (gate.crashing.org [63.228.1.57])
 (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by lists.ozlabs.org (Postfix) with ESMTPS id 406kcM3JBlzF21d
 for <linuxppc-dev@lists.ozlabs.org>; Fri, 23 Mar 2018 11:16:50 +1100 (AEDT)
Message-ID: <1521764168.16434.324.camel@kernel.crashing.org>
Subject: Re: RFC on writel and writel_relaxed
From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
To: Sinan Kaya <okaya@codeaurora.org>, Oliver <oohall@gmail.com>
Cc: linuxppc dev list <linuxppc-dev@lists.ozlabs.org>,
 "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>, Marc Zyngier
 <marc.zyngier@arm.com>, Will Deacon <will.deacon@arm.com>
Date: Fri, 23 Mar 2018 11:16:08 +1100
In-Reply-To: <5ccdb208-4664-0a7f-df5d-2e12cbe4c239@codeaurora.org>
References: <3611eabe-2999-1482-b2b4-6d216bbe4762@codeaurora.org>
 <CAOSf1CFmoK2XSn0=qAtiVEO9t4i-DHGgfieiedDfuO6E3tpkQw@mail.gmail.com>
 <4e5c745a-8b9b-959e-8893-d99cd6032484@codeaurora.org>
 <1521692689.16434.293.camel@kernel.crashing.org>
 <CAOSf1CFv1HHCL3YOVhRn2U=grdNjaQ=A4m3xwxN2Rek1-_TySg@mail.gmail.com>
 <1521726722.16434.312.camel@kernel.crashing.org>
 <5ccdb208-4664-0a7f-df5d-2e12cbe4c239@codeaurora.org>
Content-Type: text/plain; charset="UTF-8"
Mime-Version: 1.0
List-Id: Linux on PowerPC Developers Mail List <linuxppc-dev.lists.ozlabs.org>
List-Unsubscribe: <https://lists.ozlabs.org/options/linuxppc-dev>,
 <mailto:linuxppc-dev-request@lists.ozlabs.org?subject=unsubscribe>
List-Archive: <http://lists.ozlabs.org/pipermail/linuxppc-dev/>
List-Post: <mailto:linuxppc-dev@lists.ozlabs.org>
List-Help: <mailto:linuxppc-dev-request@lists.ozlabs.org?subject=help>
List-Subscribe: <https://lists.ozlabs.org/listinfo/linuxppc-dev>,
 <mailto:linuxppc-dev-request@lists.ozlabs.org?subject=subscribe>

On Thu, 2018-03-22 at 12:51 -0500, Sinan Kaya wrote:
> On 3/22/2018 8:52 AM, Benjamin Herrenschmidt wrote:
> > > > No, it's not sufficient.
> > 
> > Just to clarify ... barrier() is just a compiler barrier, it means the
> > compiler will generate things in the order they are written. This isn't
> > sufficient on archs with an OO memory model, where an actual memory
> > barrier instruction needs to be emited.
> 
> Surprisingly, ARM64 GCC compiler generates a write barrier as
> opposed to preventing code reordering.
> 
> I was curious if this is an ARM only thing or not. 

Are you sure of that ? I thought it's the ARM implementation of writel
that had an explicit write barrier in it:

#define writel(v,c)		({ __iowmb(); writel_relaxed((v),(c)); })

And __iowmb() is 

#define __iowmb()		wmb()

Note, I'm a bit dubious about this in ARM:

#define readl(c)		({ u32 __v = readl_relaxed(c); __iormb(); __v; }

Will, Marc, on powerpc, we put a sync *before* the read in readl etc...

The reasoning was there could be some DMA setup followed by a side
effect readl rather than a side effect writel to trigger a DMA. Granted
I wouldn't expect modern devices to be that stupid, but I have vague
memory of some devices back in the day having that sort of read ops.

In general, I though the model offerred by x86 and thus by Linux
readl/writel was full synchronization both before and after the MMIO,
vs either other MMIO or all other forms of ops (cachable memory, locks
etc...).

Also, can't the above readl_relaxed leak out of a lock ?

Cheers,
Ben.