Re: [PATCH] arm64: Enable PCI write-combine resources under sysfs

[Jason Gunthorpe <jgg@nvidia.com>]

On Tue, Sep 15, 2020 at 11:18:31AM +0100, Lorenzo Pieralisi wrote:
> On Tue, Sep 15, 2020 at 09:25:57AM +1000, Benjamin Herrenschmidt wrote:
> > On Mon, 2020-09-14 at 19:57 -0300, Jason Gunthorpe wrote:
> > > On Tue, Sep 15, 2020 at 08:00:27AM +1000, Benjamin Herrenschmidt wrote:
> > > > On Tue, 2020-09-15 at 07:42 +1000, Benjamin Herrenschmidt wrote:
> > > > > 
> > > > > > which is back to my original question, how do you do DMA using
> > > > > > /sys/xx/resources? Why not use VFIO like everything else?
> > > > > 
> > > > > Note: All this doesnt' change the fact that sys/xx/resources_wc
> > > > > exists
> > > > > for other archs and I see no reasons so far not to have it on ARM...
> > > > 
> > > > Also... it looks like VFIO also doesn't provide a way to do WC yet
> > > > unfortunately :-(
> > > 
> > > Yes, but if the driving reason for this patch is because a VFIO user
> > > like EFA DPDK is trying to work around VFIO limitations, then I'd say
> > > the VFIO mmap should be amended, and not so much worring about sysfs.
> > 
> > I don't think the two are exclusive.
> > 
> > > While there is no reason for ARM to not show the sysfs, it really
> > > should never be used. Modern kernels in secure boot don't even show
> > > it, for instance.
> > 
> > It's useful for random things, I've used it quite a bit in a previous
> > life for things like in-lab hw testing etc...  There's tooling out
> > there, esp. in the more 'embedded' side of thing that uses this, I
> > don't see a good reason not to provide the same level of functionality.
> > 
> > So Lorenzo, imho, we should merge the patch.
> 
> To sum it up:
> 
> (1) RDMA drivers need a new mapping function/attribute to define their
>     message push model. Actually the message model is not necessarily related
>     to write combining a la x86, so we should probably come up with a better
>     and consistent naming. Enabling this patchset may trigger performance
>     regressions on mellanox drivers on arm64 - this ought to be
>     addressed.

It is pretty clear now that the certain ARM chips that don't do write
combining with pgprot_writecombine will performance regress if they
are running a certain uncommon Mellanox configuration. I suspect these
deployments are all running the out of tree patch for DEVICE_GRE
though.

So, I wouldn't worry about it - but would prefer to see ARM folks
advance some proposal to accommodate those chips that need DEVICE_GRE.

Jason