From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EA983C10DC3 for ; Thu, 7 Dec 2023 10:14:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=gbu6sX17gCpgeF+LCcaT05nc85Sb+H1B5ZFI9ntLDXs=; b=uTDHvsdRgqFJwK u0KzuD8y33YQ6un7gdh+JyG59uZHeeOK2YFpv46BxrJNETp0wudWs0lPvWN1m48iIhyP3jRXaceGz a43Nj8NCr9fWNQSBpL5eHxvZvABgX6Aj5YXLhRU4wu6T2bWNgOB3xrJ6gfYdUyXOCZs8NmEDsd8yo 0HhwApVtXO33voS/I7knvANRVQrJZpEM3SgSq6U2RSqm91PwLG/4QPCdWWD+POvXbQKY607urbFrY wasU7k7XJQLuxG4kAfQRyKHIAi+oTK3UroHTuuI68MbIZiAx7acd2qlqdG7kRLctfi6ujzaYRZOmj gf0nZVvXE91u2Q3mHj0Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rBBOP-00CSAg-2t; Thu, 07 Dec 2023 10:14:05 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rBBOL-00CS9N-22 for linux-arm-kernel@lists.infradead.org; Thu, 07 Dec 2023 10:14:03 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id C95ED61F64; Thu, 7 Dec 2023 10:14:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EE0C4C433C8; Thu, 7 Dec 2023 10:13:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1701944040; bh=imy4ugx4p592rTLtYsew6LM8Qw7swHAvvWpZOmSRbzM=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=hhHuuJeqjenrJzluJ4waW5koAwpWLKrwzA15G7CY8Utv6hjNfvFBFFodpJSyTaKaF 7EN4QZkZ0wMz69gziLzI+1qWic/TNJhzwqb92SOYzocWHNq5gIwurKXze1emnQSkUu 5EaDIvOuY5Q9MPaZqjdoMnnNo9cbueXBY7gfxZidJC/YhLIrrDuLczNiynyULhIAXI +Ntbu8iq1JUNgGpTswSapQAtiLCQP87puO1UsFeWiY02poI233eYyilYZXLK6DrKQE vQEMmx2GQCZtFgFbx6pK6MBsP4SP7Y5Bg73JfNkZU6HgGJdvjt7ZSA9LYt+cuYd/lJ E6mNvF9PiP9pg== Date: Thu, 7 Dec 2023 11:13:52 +0100 From: Lorenzo Pieralisi To: Jason Gunthorpe Cc: Catalin Marinas , Marc Zyngier , ankita@nvidia.com, Shameerali Kolothum Thodi , oliver.upton@linux.dev, suzuki.poulose@arm.com, yuzenghui@huawei.com, will@kernel.org, ardb@kernel.org, akpm@linux-foundation.org, gshan@redhat.com, aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com, mochs@nvidia.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH v2 1/1] KVM: arm64: allow the VM to select DEVICE_* and NORMAL_NC for IO memory Message-ID: References: <20231205164318.GG2692119@nvidia.com> <20231205194822.GL2692119@nvidia.com> <20231206150556.GQ2692119@nvidia.com> <20231206153809.GS2692119@nvidia.com> <20231206164802.GT2692119@nvidia.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231206164802.GT2692119@nvidia.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231207_021401_754490_CC969532 X-CRM114-Status: GOOD ( 28.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Dec 06, 2023 at 12:48:02PM -0400, Jason Gunthorpe wrote: > On Wed, Dec 06, 2023 at 04:23:25PM +0000, Catalin Marinas wrote: > > On Wed, Dec 06, 2023 at 11:38:09AM -0400, Jason Gunthorpe wrote: > > > On Wed, Dec 06, 2023 at 04:18:05PM +0100, Lorenzo Pieralisi wrote: > > > > On Wed, Dec 06, 2023 at 11:05:56AM -0400, Jason Gunthorpe wrote: > > > > > On Wed, Dec 06, 2023 at 02:49:02PM +0000, Catalin Marinas wrote: > > > > > > BTW, on those Mellanox devices that require different attributes within > > > > > > a BAR, do they have a problem with speculative reads causing > > > > > > side-effects? > > > > > > > > > > Yes. We definitely have had that problem in the past on older > > > > > devices. VFIO must map the BAR using pgprot_device/noncached() into > > > > > the VMM, no other choice is functionally OK. > > > > > > > > Were those BARs tagged as prefetchable or non-prefetchable ? I assume the > > > > latter but please let me know if I am guessing wrong. > > > > > > I don't know it was quite old HW. Probably. > > > > > > Just because a BAR is not marked as prefetchable doesn't mean that the > > > device can't use NORMAL_NC on subsets of it. > > > > What about the other way around - would we have a prefetchable BAR that > > has portions which are unprefetchable? > > I would say possibly. > > Prefetch is a dead concept in PCIe, it was obsoleted in PCI-X about 20 > years ago. No PCIe system has ever done prefetch. > > There is a strong incentive to mark BAR's as prefetchable because it > allows 64 bit addressing in configurations with bridges. If by strong incentive you mean the "Additional guidance on the Prefetchable Bit in Memory Space BARs" in the PCI express specifications, I think it has been removed from the spec and the criteria that had to be met to implement it were basically impossible to fulfill on ARM systems, it did not make any sense in the first place. I agree on your statement related to the prefetchable concept but I believe that a prefetchable BAR containing regions that have read side-effects is essentially a borked design unless at system level speculative reads are prevented (as far as I understand the implementation note this could only be an endpoint integrated in a system where read speculation can't just happen (?)). Long story short: a PCIe card/device that can be plugged on any PCIe compliant system (x86, ARM or whatever) should not mark a BAR region with memory read side-effects as prefetchable, either that or I don't understand what the implementation note above was all about. AFAIK the prefetchable concept in PCIe is likely to be scrapped altogether in the not too distant future. Anyway, that was just for completeness (and to shed light on the BAR prefetchable bit usage). Lorenzo _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel