From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0A490C4167B for ; Wed, 6 Dec 2023 12:14:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1/eah2WASoTXfOdNJCd7dPy6V9UdLw0K8P1oY0h6f4A=; b=gqayOjJKGYI+ki Li4kp+ypD0WLWND2FuC6WAmSm5lVmMbvw08WL1XpQrD0Um4+4kUT4wpO0bRBDL0DR338E0qXEsBga u6sL4/3/vr+oQlu6vdRpAORY3G7xECAIZYLZmEyZGI6QY9WRlrN40ks/dcinKw2IyMe4hwFXNXhvJ TykGs5p8Y1yN/sTgHbesva8aZFsqXadWrMJ3/EGaThgggj9Rp9nmqWNmmmrZr1kPTwBfqX+2+ePkq ckH1KxHGrhLi74oIj8INZHDVdUWLHNECK4kPs6Zc9bEM9wGadjfhdoiglOzlXAlyVInHXqOLx63fR 1XPAG8OsjCTmlvZigiog==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAqnO-00AELx-0p; Wed, 06 Dec 2023 12:14:30 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAqnK-00AELB-2u for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 12:14:28 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by ams.source.kernel.org (Postfix) with ESMTP id 49DF9B82037; Wed, 6 Dec 2023 12:14:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6D996C433C7; Wed, 6 Dec 2023 12:14:20 +0000 (UTC) Date: Wed, 6 Dec 2023 12:14:18 +0000 From: Catalin Marinas To: Marc Zyngier Cc: Jason Gunthorpe , ankita@nvidia.com, Shameerali Kolothum Thodi , oliver.upton@linux.dev, suzuki.poulose@arm.com, yuzenghui@huawei.com, will@kernel.org, ardb@kernel.org, akpm@linux-foundation.org, gshan@redhat.com, aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com, mochs@nvidia.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, lpieralisi@kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH v2 1/1] KVM: arm64: allow the VM to select DEVICE_* and NORMAL_NC for IO memory Message-ID: References: <86fs0hatt3.wl-maz@kernel.org> <20231205130517.GD2692119@nvidia.com> <20231205164318.GG2692119@nvidia.com> <86bkb4bn2v.wl-maz@kernel.org> <86a5qobkt8.wl-maz@kernel.org> <868r67blwo.wl-maz@kernel.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <868r67blwo.wl-maz@kernel.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_041427_259224_CAD0F7BD X-CRM114-Status: GOOD ( 30.64 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Dec 06, 2023 at 11:39:03AM +0000, Marc Zyngier wrote: > On Tue, 05 Dec 2023 18:40:42 +0000, > Catalin Marinas wrote: > > On Tue, Dec 05, 2023 at 05:50:27PM +0000, Marc Zyngier wrote: > > > On Tue, 05 Dec 2023 17:33:01 +0000, > > > Catalin Marinas wrote: > > > > Ideally we should do this for vfio only but we don't have an easy > > > > way to convey this to KVM. > > > > > > But if we want to limit this to PCIe, we'll have to find out. The > > > initial proposal (a long while ago) had a flag conveying some > > > information, and I'd definitely feel more confident having something > > > like that. > > > > We can add a VM_PCI_IO in the high vma flags to be set by > > vfio_pci_core_mmap(), though it limits it to 64-bit architectures. KVM > > knows this is PCI and relaxes things a bit. It's not generic though if > > we need this later for something else. > > Either that, or something actually describing the attributes that VFIO > wants. > > And I very much want it to be a buy-in behaviour, not something that > automagically happens and changes the default behaviour for everyone > based on some hand-wavy assertions. > > If that means a userspace change, fine by me. The VMM better know what > is happening. Driving the attributes from a single point like the VFIO driver is indeed better. The problem is that write-combining on Arm doesn't come without speculative loads, otherwise we would have solved it by now. I also recall the VFIO maintainer pushing back on relaxing the pgprot_noncached() for the user mapping but I don't remember the reasons. We could do with a pgprot_maybewritecombine() or pgprot_writecombinenospec() (similar to Jason's idea but without changing the semantics of pgprot_device()). For the user mapping on arm64 this would be Device (even _GRE) since it can't disable speculation but stage 2 would leave the decision to the guest since the speculative loads aren't much different from committed loads done wrongly. If we want the VMM to drive this entirely, we could add a new mmap() flag like MAP_WRITECOMBINE or PROT_WRITECOMBINE. They do feel a bit weird but there is precedent with PROT_MTE to describe a memory type. One question is whether the VFIO driver still needs to have the knowledge and sanitise the requests from the VMM within a single BAR. If there are no security implications to such mappings, the VMM can map parts of the BAR as pgprot_noncached(), other parts as pgprot_writecombine() and KVM just follows them (similarly if we need a cacheable mapping). The latter has some benefits for DPDK but it's a lot more involved with having to add device-specific knowledge into the VMM. The VMM would also have to present the whole BAR contiguously to the guest even if there are different mapping attributes within the range. So a lot of MAP_FIXED uses. I'd rather leaving this decision with the guest than the VMM, it looks like more hassle to create those mappings. The VMM or the VFIO could only state write-combine and speculation allowed. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel