From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Subject: Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory From: Benjamin Herrenschmidt Reply-To: benh@au1.ibm.com To: Jason Gunthorpe Cc: Dan Williams , Logan Gunthorpe , Linux Kernel Mailing List , linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma , linux-nvdimm , linux-block@vger.kernel.org, Stephen Bates , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Max Gurtovoy , =?ISO-8859-1?Q?J=E9r=F4me?= Glisse , Alex Williamson , Oliver OHalloran Date: Fri, 02 Mar 2018 09:06:52 +1100 In-Reply-To: <20180301205315.GJ19007@ziepe.ca> References: <20180228234006.21093-1-logang@deltatee.com> <1519876489.4592.3.camel@kernel.crashing.org> <1519876569.4592.4.camel@au1.ibm.com> <1519936477.4592.23.camel@au1.ibm.com> <1519936815.4592.25.camel@au1.ibm.com> <20180301205315.GJ19007@ziepe.ca> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Message-Id: <1519942012.4592.31.camel@au1.ibm.com> List-ID: On Thu, 2018-03-01 at 13:53 -0700, Jason Gunthorpe wrote: > On Fri, Mar 02, 2018 at 07:40:15AM +1100, Benjamin Herrenschmidt wrote: > > Also we need to be able to hard block MEMREMAP_WB mappings of non-RAM > > on ppc64 (maybe via an arch hook as it might depend on the processor > > family). Server powerpc cannot do cachable accesses on IO memory > > (unless it's special OpenCAPI or nVlink, but not on PCIe). > > I think you are right on this - even on x86 we must not create > cachable mappings of PCI BARs - there is no way that works the way > anyone would expect. > > I think this series doesn't have a problem here only because it never > touches the BAR pages with the CPU. > > BAR memory should be mapped into the CPU as WC at best on all arches.. Could be that x86 has the smarts to do the right thing, still trying to untangle the code :-) Cheers, Ben. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 8DD6D22546BB1 for ; Thu, 1 Mar 2018 14:00:54 -0800 (PST) Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w21M53Dj036504 for ; Thu, 1 Mar 2018 17:07:02 -0500 Received: from e06smtp14.uk.ibm.com (e06smtp14.uk.ibm.com [195.75.94.110]) by mx0b-001b2d01.pphosted.com with ESMTP id 2geqw250jg-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Thu, 01 Mar 2018 17:07:02 -0500 Received: from localhost by e06smtp14.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 1 Mar 2018 22:07:00 -0000 Subject: Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory From: Benjamin Herrenschmidt Date: Fri, 02 Mar 2018 09:06:52 +1100 In-Reply-To: <20180301205315.GJ19007@ziepe.ca> References: <20180228234006.21093-1-logang@deltatee.com> <1519876489.4592.3.camel@kernel.crashing.org> <1519876569.4592.4.camel@au1.ibm.com> <1519936477.4592.23.camel@au1.ibm.com> <1519936815.4592.25.camel@au1.ibm.com> <20180301205315.GJ19007@ziepe.ca> Mime-Version: 1.0 Message-Id: <1519942012.4592.31.camel@au1.ibm.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: benh@au1.ibm.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Jason Gunthorpe Cc: Jens Axboe , Keith Busch , Oliver OHalloran , Alex Williamson , linux-nvdimm , linux-rdma , linux-pci@vger.kernel.org, Linux Kernel Mailing List , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, =?ISO-8859-1?Q?J=E9r=F4me?= Glisse , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-ID: On Thu, 2018-03-01 at 13:53 -0700, Jason Gunthorpe wrote: > On Fri, Mar 02, 2018 at 07:40:15AM +1100, Benjamin Herrenschmidt wrote: > > Also we need to be able to hard block MEMREMAP_WB mappings of non-RAM > > on ppc64 (maybe via an arch hook as it might depend on the processor > > family). Server powerpc cannot do cachable accesses on IO memory > > (unless it's special OpenCAPI or nVlink, but not on PCIe). > > I think you are right on this - even on x86 we must not create > cachable mappings of PCI BARs - there is no way that works the way > anyone would expect. > > I think this series doesn't have a problem here only because it never > touches the BAR pages with the CPU. > > BAR memory should be mapped into the CPU as WC at best on all arches.. Could be that x86 has the smarts to do the right thing, still trying to untangle the code :-) Cheers, Ben. _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 From: benh@au1.ibm.com (Benjamin Herrenschmidt) Date: Fri, 02 Mar 2018 09:06:52 +1100 Subject: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory In-Reply-To: <20180301205315.GJ19007@ziepe.ca> References: <20180228234006.21093-1-logang@deltatee.com> <1519876489.4592.3.camel@kernel.crashing.org> <1519876569.4592.4.camel@au1.ibm.com> <1519936477.4592.23.camel@au1.ibm.com> <1519936815.4592.25.camel@au1.ibm.com> <20180301205315.GJ19007@ziepe.ca> Message-ID: <1519942012.4592.31.camel@au1.ibm.com> On Thu, 2018-03-01@13:53 -0700, Jason Gunthorpe wrote: > On Fri, Mar 02, 2018@07:40:15AM +1100, Benjamin Herrenschmidt wrote: > > Also we need to be able to hard block MEMREMAP_WB mappings of non-RAM > > on ppc64 (maybe via an arch hook as it might depend on the processor > > family). Server powerpc cannot do cachable accesses on IO memory > > (unless it's special OpenCAPI or nVlink, but not on PCIe). > > I think you are right on this - even on x86 we must not create > cachable mappings of PCI BARs - there is no way that works the way > anyone would expect. > > I think this series doesn't have a problem here only because it never > touches the BAR pages with the CPU. > > BAR memory should be mapped into the CPU as WC at best on all arches.. Could be that x86 has the smarts to do the right thing, still trying to untangle the code :-) Cheers, Ben. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Benjamin Herrenschmidt Subject: Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory Date: Fri, 02 Mar 2018 09:06:52 +1100 Message-ID: <1519942012.4592.31.camel@au1.ibm.com> References: <20180228234006.21093-1-logang@deltatee.com> <1519876489.4592.3.camel@kernel.crashing.org> <1519876569.4592.4.camel@au1.ibm.com> <1519936477.4592.23.camel@au1.ibm.com> <1519936815.4592.25.camel@au1.ibm.com> <20180301205315.GJ19007@ziepe.ca> Reply-To: benh-8fk3Idey6ehBDgjK7y7TUQ@public.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20180301205315.GJ19007-uk2M96/98Pc@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Jason Gunthorpe Cc: Jens Axboe , Keith Busch , Oliver OHalloran , Alex Williamson , linux-nvdimm , linux-rdma , linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Linux Kernel Mailing List , linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, =?ISO-8859-1?Q?J=E9r=F4me?= Glisse , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-Id: linux-rdma@vger.kernel.org On Thu, 2018-03-01 at 13:53 -0700, Jason Gunthorpe wrote: > On Fri, Mar 02, 2018 at 07:40:15AM +1100, Benjamin Herrenschmidt wrote: > > Also we need to be able to hard block MEMREMAP_WB mappings of non-RAM > > on ppc64 (maybe via an arch hook as it might depend on the processor > > family). Server powerpc cannot do cachable accesses on IO memory > > (unless it's special OpenCAPI or nVlink, but not on PCIe). > > I think you are right on this - even on x86 we must not create > cachable mappings of PCI BARs - there is no way that works the way > anyone would expect. > > I think this series doesn't have a problem here only because it never > touches the BAR pages with the CPU. > > BAR memory should be mapped into the CPU as WC at best on all arches.. Could be that x86 has the smarts to do the right thing, still trying to untangle the code :-) Cheers, Ben.