Re: RFC on writel and writel_relaxed

[Jason Gunthorpe <jgg@ziepe.ca>]

On Fri, Mar 23, 2018 at 12:52:02AM +1100, Benjamin Herrenschmidt wrote:

> > >  - Make writel_relaxed() be a simple store without barriers, and
> > > readl_relaxed() be "eieio, read, eieio", thus allowing write combining
> > > to happen between successive writel_relaxed on WC space (no change on
> > > normal NC space) while maintaining the ordering between relaxed reads
> > > and writes. The flip side is a (slight) increased overhead of
> > > readl_relaxed.
> > 
> > Are there many drivers that actually do writeX() on WC space?
> > memory-barriers.txt
> > pretty much says that all bets are off and no ordering guarantees can be assumed
> > when using readX/writeX on prefetchable IO memory. It seems sketchy enough to
> > give me some pause, but maybe it works fine elsewhere.
> 
> I don't know whether any does it, but I want to provide a way for a
> driver to somewhat reliably obtain write combine semantics without
> having to hand code endian swap and other horrors involved with using
> __raw_* accessors.

Many of the drivers in drivers/infiniband work with write combining
memory.

The usual pattern is a desire to push 32 or 64 bytes to the WC BAR as
efficiently as possible, ideally in a single PCI-E TLP.

A memcpy_to_wc primitive could probably cover these use cases, no need
to redesign the IO accessors..

The WC memory is never read, so read/write order is not important to
any infiniband driver.

What is very important is keeping the WC behavior isolated within the
spinlock. WC to the same addresses cannot be permitted in this pattern:

   writel(addr = 0);
   mmiowmb();
   spin_unlock();
   spin_lock()
   writel(addr = 0);

The CPU must always generate two PCI-E TLPs to the device.

This is a super performance critical operation for most drivers and
directly impacts network performance.

Jason