From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC4C4C433E7 for ; Wed, 2 Sep 2020 17:56:36 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A215220758 for ; Wed, 2 Sep 2020 17:56:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="zBFf5GYb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A215220758 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=GNC35yRH4PbZPeTAnmR8obwD8JfPcVn51eLlOKLf1jg=; b=zBFf5GYbIIz8SBxlnfL3/UW7N qk1Kv3ht7cor+4iT156yrv7a48egQtXG2d/rFTIuVEqZ6DeI6lw33Du60oENKvGTA5ud9mK8d2V/K 9thxV8n3OdWL+v6erv1yqBGlwqrZCgat8JwFxJv/h+pzQy5/93SFXu8bfh1epfa8H3jBfRQuL6F19 YC8beMXwaXdCk4oV6Aa/Y0gkkrLJQWHd+d64oaJlIlx3seENcbzh9++8KfE3ACF2p5ioITczgi/Sj y3Xyqo1hClqCDLNnpmEUU6VykbNoRCUq80Wk/G6Gp2Ls0ZA7BS1d0F1bCwJqD4TJhrpq5Z1GCLMlp QQF1LbiHQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kDWyL-0004ZQ-H6; Wed, 02 Sep 2020 17:55:01 +0000 Received: from foss.arm.com ([217.140.110.172]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kDWyH-0004Yt-KE for linux-arm-kernel@lists.infradead.org; Wed, 02 Sep 2020 17:54:59 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3EC43D6E; Wed, 2 Sep 2020 10:54:56 -0700 (PDT) Received: from e121166-lin.cambridge.arm.com (e121166-lin.cambridge.arm.com [10.1.196.255]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3D2CB3F66F; Wed, 2 Sep 2020 10:54:50 -0700 (PDT) Date: Wed, 2 Sep 2020 18:54:45 +0100 From: Lorenzo Pieralisi To: Catalin Marinas Subject: Re: [PATCH] arm64: Enable PCI write-combine resources under sysfs Message-ID: <20200902175445.GA31706@e121166-lin.cambridge.arm.com> References: <20200831151827.pumm2p54fyj7fz5s@amazon.com> <20200902113207.GA27676@e121166-lin.cambridge.arm.com> <20200902142922.xc4x6m33unkzewuh@amazon.com> <20200902164702.GA30611@e121166-lin.cambridge.arm.com> <20200902172156.GD16673@gaia> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200902172156.GD16673@gaia> User-Agent: Mutt/1.9.4 (2018-02-28) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200902_135458_132051_49FAF466 X-CRM114-Status: GOOD ( 37.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-pci@vger.kernel.org, Bjorn Helgaas , benh@kernel.crashing.org, will@kernel.org, linux-arm-kernel@lists.infradead.org, Clint Sbisa Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Sep 02, 2020 at 06:21:57PM +0100, Catalin Marinas wrote: > On Wed, Sep 02, 2020 at 05:47:02PM +0100, Lorenzo Pieralisi wrote: > > On Wed, Sep 02, 2020 at 02:29:22PM +0000, Clint Sbisa wrote: > > > On Wed, Sep 02, 2020 at 12:32:07PM +0100, Lorenzo Pieralisi wrote: > > > > On Mon, Aug 31, 2020 at 03:18:27PM +0000, Clint Sbisa wrote: > > > > > arm64 supports write-combine PCI mappings, so the appropriate define > > > > > has been added which will expose write-combine mappings under sysfs > > > > > for prefetchable PCI resources. > > > > > > > > > > Signed-off-by: Clint Sbisa > > > > > --- > > > > > arch/arm64/include/asm/pci.h | 1 + > > > > > 1 file changed, 1 insertion(+) > > > > > > > > > > diff --git a/arch/arm64/include/asm/pci.h b/arch/arm64/include/asm/pci.h > > > > > index 70b323cf8300..b33ca260e3c9 100644 > > > > > --- a/arch/arm64/include/asm/pci.h > > > > > +++ b/arch/arm64/include/asm/pci.h > > > > > @@ -17,6 +17,7 @@ > > > > > #define pcibios_assign_all_busses() \ > > > > > (pci_has_flag(PCI_REASSIGN_ALL_BUS)) > > > > > > > > > > +#define arch_can_pci_mmap_wc() 1 > > > > > > > > I am not comfortable with this blanket enable. Some existing drivers, > > > > eg: > > > > > > > > drivers/infiniband/hw/mlx5 > > > > > > > > use this macro to detect WC capability which again, it is x86 specific, > > > > on arm64 it means nothing and can have consequences on the driver > > > > operations. > > > > > > If that driver is fixed to check what it actually wants to check, would that > > > address your concern about the blanket enable? I don't see any other references > > > to this in kernel drivers and I think the documentation at > > > `filesystems/sysfs-pci.rst` outlines it pretty explicitly: > > > > > > Platforms which support write-combining maps of PCI resources must define > > > arch_can_pci_mmap_wc() which shall evaluate to non-zero at runtime when > > > write-combining is permitted. > > > > That's exactly the problem. I am asking you: what does "write-combining > > maps of PCI resources" mean ? > > > > I understand we do want weak ordering for prefetchable BAR mappings > > but my worry is that by exposing the resources as WC to user space > > we are giving user space the impression that those mappings mirror > > x86 WC mappings behaviour that is not true on ARM64. > > Would Device_GRE be close to the x86 WC better? It won't allow unaligned > accesses and that can be problematic for the user. OTOH, it doesn't > speculate reads, so it's safer from the hardware perspective. Thanks Catalin for chiming in, it may yes but I need to figure out the precise semantics of WC on x86 first. Actually *if* I read x86 specs correctly WC mappings allow speculative reads, which then would shift the issue on the PCI specs that allow marking read side effects BARs as prefetchable; in other words if an endpoint is designed with a prefetchable BAR that has read side effects this is already an issue on x86 in the current kernel. There is that, plus the usage of arch_can_pci_mmap_wc() in mellanox drivers which I suspect it is yet another interpretation of x86 write combine - I don't know what happens if we let arch_can_pci_mmap_wc() == 1 on both normalNC or deviceGRE mappings for pgprot_writecombine. I think it is worth getting to the bottom of this before applying this patch. Thanks, Lorenzo _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel