From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_ALL,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9496C433E2 for ; Thu, 3 Sep 2020 14:37:53 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7DB13206EB for ; Thu, 3 Sep 2020 14:37:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="oKHB50OL"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="HzA5iJto" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7DB13206EB Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=jT6Z7vBAtbwrRl+tGYqnoddkHJmcscdCmHc3CuOmq6k=; b=oKHB50OLkCDDVSCgEm9Sg6uYQ nM0xrYEB0l0k0GBQNXAaN2VCe9sH3RXcKFfu8xqKpbH3k9OWrIt/jAwwVLvwFOL8812CbHoRCVNvl HEh0UL6XChRCu7mJMJhrf/70gIAeO+gvgz0+0ovmP4YWYGBB8c02cSb1L8v2bzz0AEpK4A5aDFNOu IS+w84PW/QAlyUFJfVDmmC0sk01f/bfm7g4wQ2s9Rkj31gZRDQlzYhp8Auy8vYiyQ0OA1keWaI+dZ LShkWyV+Yw/vXhicw4IoDfyzN6L4mhEI9K2eOoMWLBqHo8KI7jNNQDU6L8n4TmXLBN4LcnGZLf1jg UvSwf9Skg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kDqLd-0000t9-H4; Thu, 03 Sep 2020 14:36:21 +0000 Received: from smtp-fw-6001.amazon.com ([52.95.48.154]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kDqLZ-0000sZ-UF for linux-arm-kernel@lists.infradead.org; Thu, 03 Sep 2020 14:36:18 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1599143778; x=1630679778; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=8yBGan8tgMpccZ4nTZypCZpwCtDtVD3XMtqoNCEo9E4=; b=HzA5iJtohvv42snHj/t4Z+/KnYktNvIqsWHasPSCtKOryIo2gDrcvJsu XItKldFvm0lLVBSDmaY3hlUSV4r4VoWcla1hmO5XKWigamEDgtAhBC/4Q tU3WJYEmwhQ1WXtW/hTRc9xTME1HyRJXNCeYNFVhB8Hch5JeLeb29HJ1w Y=; X-IronPort-AV: E=Sophos;i="5.76,387,1592870400"; d="scan'208";a="53246111" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-1e-27fb8269.us-east-1.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6001.iad6.amazon.com with ESMTP; 03 Sep 2020 14:36:13 +0000 Received: from EX13MTAUWA001.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1e-27fb8269.us-east-1.amazon.com (Postfix) with ESMTPS id AB8C2A21B4; Thu, 3 Sep 2020 14:36:10 +0000 (UTC) Received: from EX13D07UWA003.ant.amazon.com (10.43.160.35) by EX13MTAUWA001.ant.amazon.com (10.43.160.58) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 3 Sep 2020 14:36:09 +0000 Received: from EX13MTAUWA001.ant.amazon.com (10.43.160.58) by EX13D07UWA003.ant.amazon.com (10.43.160.35) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 3 Sep 2020 14:36:09 +0000 Received: from dev-dsk-csbisa-2a-37939146.us-west-2.amazon.com (172.19.34.216) by mail-relay.amazon.com (10.43.160.118) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 3 Sep 2020 14:36:09 +0000 Received: by dev-dsk-csbisa-2a-37939146.us-west-2.amazon.com (Postfix, from userid 800212) id B5DBA1AD; Thu, 3 Sep 2020 14:36:09 +0000 (UTC) Date: Thu, 3 Sep 2020 14:36:09 +0000 From: Clint Sbisa To: Lorenzo Pieralisi , Yishai Hadas , Guy Levy Subject: Re: [PATCH] arm64: Enable PCI write-combine resources under sysfs Message-ID: <20200903143609.didfh6jwt7etm7he@amazon.com> References: <20200831151827.pumm2p54fyj7fz5s@amazon.com> <20200902113207.GA27676@e121166-lin.cambridge.arm.com> <20200902142922.xc4x6m33unkzewuh@amazon.com> <20200902164702.GA30611@e121166-lin.cambridge.arm.com> <20200903110844.GB11284@e121166-lin.cambridge.arm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200903110844.GB11284@e121166-lin.cambridge.arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200903_103618_140529_6A1A4BF7 X-CRM114-Status: GOOD ( 32.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Benjamin Herrenschmidt , linux-pci@vger.kernel.org, Bjorn Helgaas , catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Adding some Mellanox folks to comment on their usage of arch_can_pci_mmap_wc(). On Thu, Sep 03, 2020 at 12:08:44PM +0100, Lorenzo Pieralisi wrote: > On Thu, Sep 03, 2020 at 09:07:00AM +1000, Benjamin Herrenschmidt wrote: > > On Wed, 2020-09-02 at 17:47 +0100, Lorenzo Pieralisi wrote: > > > Yes I do and I expressed them. > > > > > > The first concern is the WC ambiguity on non-x86 systems, it looks > > > like write combinining means everything and nothing at the same time > > > on != x86 arches. > > > > > > On x86 prefetchable BAR == WC mapping (still conditional on arch > > > features ie PAT, not a blanket enable). On ARM64 WC mapping currently > > > corresponds to normal NC memory and the PCIe specs allow read > > > side-effects BAR to be marked as prefetchable, I need to force PCI > > > sig > > > to remove the section I mentioned from the specifications because > > > there > > > is NO way it can be detected if a prefetchable BAR maps to read > > > side-effects memory. > > > > Im not sure I understand your sentence. It's been a long accepted rule > > in PCI land that "prefetchable" BARs means "no side effects" and in > > fact allows much more than just prefetching :-) > > I am referring to the nefarious: > > "Additional Guidance on the Prefetchable Bit in Memory Space BARs" > > I read it 100 times and I still have no idea how it can be implemented, > it sorts of acknowledges that read side-effects memory can be marked > as a prefetchable BAR *if* the system meets some criteria. > > As if endpoint designers knew the system where their endpoint is > plugged into (+ bit (3) in a BAR is read-only). > > I think that that implementation note must be removed from the > specifications - if anyone dares to follow it this whole > WC resource mapping can trigger trouble. > > Good news is that it would be trouble for all arches :) > > > > A kernel device driver would (hopefully) know, sysfs code that just > > > checks the prefetchable attribute and exports resource_WC does not. > > > > > > As I mentioned, if the mapping is done in a device specific driver it > > > can be vetted and there are not many drivers mapping BARs as > > > ioremap_wc(). > > > > It's been what other architectures have been doing for mroe than a > > decade without significant issues... I don't think you should worry too > > much about this. > > Minus what I wrote above, I agree with you. I'd still be able to > understand what this patch changes in the mellanox driver HW > handling though - not sure what they expect from arch_can_pci_mmap_wc() > returning 1. This seems to have been added broadly for x86, PPC, and ARM as part of initial WC support in the driver (37aa5c36 "IB/mlx5: Add UARs write-combining and non-cached mapping"). It was updated to use `arch_can_pci_mmap_wc()` later (1f3db161 "IB/mlx5: Generally use the WC auto detection test result"). Guy, Yishai, there are some concerns about difference the technical definition of WC and how WC is actually used. Can you comment on the usage of WC in mlx5 and which definition of WC the driver utilizes? We're unsure if a blanket enable for arm64 is safe in light of the driver's use of this function. Thanks, Clint > > Thanks, > Lorenzo _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel