From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9DCFC433E2 for ; Wed, 16 Sep 2020 08:01:09 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3DD9C2080C for ; Wed, 16 Sep 2020 08:01:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="a1PzTYFb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3DD9C2080C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.crashing.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Mime-Version:References:In-Reply-To:Date:To:From: Subject:Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=WUI78H5O+JqR/iGTfADIoVaqe+7DKCg4jL+Naa945Go=; b=a1PzTYFbi0SCKFPHWl6SlChmB L93M9qLBtyJPXz88HE9Goc1PEyNVZ6ZviH8zGWF/DVQdVx5pJgD9LKmgUHjUSsvQdWGuk/eBox/Zl WIOSkiKANS5Z0S5NpWD11bwS/K0en5O+dDiqpY8teFbHzjDB+stDswHAO1h/Fw95efX/2/Quk8Dsb ACOT+/4nN1zusvanRsPGDioSwopc4C5NaQV1qTqGqhHtqPu6XnglopMJfW5qun5I7FalFv9u52Xpa 7f3mliD/F9Q8b4q2+aGRZKHpWqwQancRGQ7O5m4j6GJZ3Gd+6wwgXqqoL5SLUisllh/6FU+pWjq0+ cCUGZEtzw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kISM5-0004Ba-0B; Wed, 16 Sep 2020 07:59:53 +0000 Received: from kernel.crashing.org ([76.164.61.194]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kISM1-0004AJ-9w for linux-arm-kernel@lists.infradead.org; Wed, 16 Sep 2020 07:59:50 +0000 Received: from localhost (gate.crashing.org [63.228.1.57]) (authenticated bits=0) by kernel.crashing.org (8.14.7/8.14.7) with ESMTP id 08G7xPbq020075 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 16 Sep 2020 02:59:29 -0500 Message-ID: <701012f288231d0d0733bf1c2c8fdbd9caa074fd.camel@kernel.crashing.org> Subject: Re: [PATCH] arm64: Enable PCI write-combine resources under sysfs From: Benjamin Herrenschmidt To: Jason Gunthorpe Date: Wed, 16 Sep 2020 17:59:24 +1000 In-Reply-To: <20200915234006.GI1573713@nvidia.com> References: <20200914141726.GA904879@nvidia.com> <20200914142406.k44zrnp2wdsandsp@amazon.com> <20200914143819.GC904879@nvidia.com> <375c478593945a416f3180c3773bcb5240d2e36c.camel@kernel.crashing.org> <1d6f2ceb8d3538c906a1fdb8cd3d4c74ccffa42e.camel@kernel.crashing.org> <20200914225740.GP904879@nvidia.com> <2b539df4c9ec703458e46da2fc879ee3b310b31c.camel@kernel.crashing.org> <20200915101831.GA2616@e121166-lin.cambridge.arm.com> <20200915110511.GQ904879@nvidia.com> <20200915234006.GI1573713@nvidia.com> X-Mailer: Evolution 3.28.5-0ubuntu0.18.04.2 Mime-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200916_035949_489187_00A2282D X-CRM114-Status: GOOD ( 29.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lorenzo Pieralisi , Leon Romanovsky , linux-pci@vger.kernel.org, Bjorn Helgaas , catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, Clint Sbisa Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, 2020-09-15 at 20:40 -0300, Jason Gunthorpe wrote: > Not quite, upstream kernel will never use WC on those > devices. DEVICE_GRE is not supported in upstream, > arch_can_pci_mmap_wc() is always false and the WC tester will always > fail. > > > With the patch, those device will now use MT_DEVICE_NC. > > Which doesn't do WC at all on some ARM implementations. Lovely... this is arm64 btw, still the case ? Also we could make this a variable rather than a constant and choose a more appropriate set of flags at boot time.... > > Why would that be a regression ? > > Using the WC submission flow when it doesn't work costs something like > 10% performance vs using the non-WC flow. You mean the driver uses a different path to the HW which ahs that overhead, not that MMIOs have that overhead right ? > Like I said, the case where the driver can't self test probably > doesn't intersect with the ARM implementations that can't do write > combining, and if it did, the users probably run the out of tree > driver that has the hacky stuff to make it use DEVICE_GRE. Ok. So you are saying to go for it and ignore that Mellanox case then ? :-) > > BTW. Lorenzo, why don't we use MT_DEVICE_GRE for pgprot_writecombine ? > > Its not supported on some chips ? > > It has alignment requirements drivers don't meet. We need a new > concept of "write combining and I promise to do aligned access" Ah yes, I remember. Right, we would need to provide new/better accessors for these kind of things. It's going to be a mess to find a common set that works for all archs. > > What on earth is pgprot_device() ? This is new ? On ARM it will be > > MT_DEVICE_nGnRE, so it allows posted write. It seems to match what > > ioremap does. Should then ioremap use it as well ? > > > > But it's only ever used for PCI mmap. Why is it different from > > pgprot_noncached() which disables posted writes (nE) ? > > > > Because a whole lot of drivers will use pgprot_noncached() explicitly > > in either mmap or vmap, with the expectation that it's somewhat the > > same as what ioremap does... > > *boggle* > > Only sysfs uses pci_mmap_resource_range() any other driver exposing > BAR pages, like VFIO dies not. Makes no sense at all it is different. > > Delete the ill defined pgprot_device() ? Nobody has complained > something is wrong with VFIO in the 6 years since it was added... I was wondering what it was, that's it ... Cheers, Ben. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel