From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E6F21C4332F for ; Tue, 12 Dec 2023 17:47:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WCUTkiwsfkgLhv9MSYdkW2GZn4kOS3Dyxc+QEmJjK0U=; b=NsQMAPrh5rUnxo csbYyW5C6MhBPvGkiHaKRHB79vK2lfXl03FRYVfd8AuYqr49Bai6AnNopV7CisQtTwLFvDYRb5mnl 16wMD7JsWlkW//0WSA6MIjLlf9V/LR12ztmaIAkqXQQg3p7H6TroQ9lXiexMEDFnXdeDtgRwwZQF8 QUxHyPQR3bGiVJBy0YZ0U86pmQ+3UcRchZmPeYxl7O8fvTe674kDTMjfCziKLDcGgcWPLNXr5k+xe N8ElINLkzrl7rVTOOy+ms0AGK5Zt20c0FTy22LRi+oz8zm5KvP6sHal/vOzcjiKLnFxG0P84xhlR7 xoDWHzfooBIvysD+AbLA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rD6qF-00CQaz-1L; Tue, 12 Dec 2023 17:46:47 +0000 Received: from sin.source.kernel.org ([2604:1380:40e1:4800::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rD6qB-00CQaX-22 for linux-arm-kernel@lists.infradead.org; Tue, 12 Dec 2023 17:46:45 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id EB669CE1926; Tue, 12 Dec 2023 17:46:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 60BECC433C8; Tue, 12 Dec 2023 17:46:36 +0000 (UTC) Date: Tue, 12 Dec 2023 17:46:34 +0000 From: Catalin Marinas To: ankita@nvidia.com Cc: jgg@nvidia.com, maz@kernel.org, oliver.upton@linux.dev, suzuki.poulose@arm.com, yuzenghui@huawei.com, will@kernel.org, alex.williamson@redhat.com, kevin.tian@intel.com, yi.l.liu@intel.com, ardb@kernel.org, akpm@linux-foundation.org, gshan@redhat.com, linux-mm@kvack.org, lpieralisi@kernel.org, aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com, mochs@nvidia.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH v3 2/2] kvm: arm64: set io memory s2 pte as normalnc for vfio pci devices Message-ID: References: <20231208164709.23101-1-ankita@nvidia.com> <20231208164709.23101-3-ankita@nvidia.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231208164709.23101-3-ankita@nvidia.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231212_094644_007118_AC548040 X-CRM114-Status: GOOD ( 24.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Dec 08, 2023 at 10:17:09PM +0530, ankita@nvidia.com wrote: > arch/arm64/kvm/hyp/pgtable.c | 3 +++ > arch/arm64/kvm/mmu.c | 16 +++++++++++++--- > drivers/vfio/pci/vfio_pci_core.c | 3 ++- > include/linux/mm.h | 7 +++++++ > 4 files changed, 25 insertions(+), 4 deletions(-) It might be worth factoring out the vfio bits into a separate patch together with a bit of documentation around this new vma flag (up to Alex really). > diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c > index d4835d553c61..c8696c9e7a60 100644 > --- a/arch/arm64/kvm/hyp/pgtable.c > +++ b/arch/arm64/kvm/hyp/pgtable.c > @@ -722,6 +722,9 @@ static int stage2_set_prot_attr(struct kvm_pgtable *pgt, enum kvm_pgtable_prot p > kvm_pte_t attr; > u32 sh = KVM_PTE_LEAF_ATTR_LO_S2_SH_IS; > > + if (device && normal_nc) > + return -EINVAL; Ah, the comment Will and I made on patch 1 is handled here. Add a WARN_ON_ONCE() and please move this hunk to the first patch, it makes more sense there. > diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c > index d14504821b79..1ce1b6d89bf9 100644 > --- a/arch/arm64/kvm/mmu.c > +++ b/arch/arm64/kvm/mmu.c > @@ -1381,7 +1381,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, > int ret = 0; > bool write_fault, writable, force_pte = false; > bool exec_fault, mte_allowed; > - bool device = false; > + bool device = false, vfio_pci_device = false; I don't think the variable here should be named vfio_pci_device, the VM_* flag doesn't mention PCI. So just something like "vfio_allow_wc". > unsigned long mmu_seq; > struct kvm *kvm = vcpu->kvm; > struct kvm_mmu_memory_cache *memcache = &vcpu->arch.mmu_page_cache; > @@ -1472,6 +1472,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, > gfn = fault_ipa >> PAGE_SHIFT; > mte_allowed = kvm_vma_mte_allowed(vma); > > + vfio_pci_device = !!(vma->vm_flags & VM_VFIO_ALLOW_WC); Nitpick: no need for !!, you are assigning to a bool variable already. > diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c > index 1cbc990d42e0..c3f95ec7fc3a 100644 > --- a/drivers/vfio/pci/vfio_pci_core.c > +++ b/drivers/vfio/pci/vfio_pci_core.c > @@ -1863,7 +1863,8 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma > * See remap_pfn_range(), called from vfio_pci_fault() but we can't > * change vm_flags within the fault handler. Set them now. > */ > - vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); > + vm_flags_set(vma, VM_VFIO_ALLOW_WC | VM_IO | VM_PFNMAP | > + VM_DONTEXPAND | VM_DONTDUMP); Please add a comment here that write-combining is allowed to be enabled by the arch (KVM) code but the default user mmap() will still use pgprot_noncached(). > vma->vm_ops = &vfio_pci_mmap_ops; > > return 0; > diff --git a/include/linux/mm.h b/include/linux/mm.h > index a422cc123a2d..8d3c4820c492 100644 > --- a/include/linux/mm.h > +++ b/include/linux/mm.h > @@ -391,6 +391,13 @@ extern unsigned int kobjsize(const void *objp); > # define VM_UFFD_MINOR VM_NONE > #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */ > > +#ifdef CONFIG_64BIT > +#define VM_VFIO_ALLOW_WC_BIT 39 /* Convey KVM to map S2 NORMAL_NC */ This comment shouldn't be in the core header file. It knows nothing about S2 and Normal-NC, that's arm64 terminology. You can mention something like VFIO can use this flag hint that write-combining is allowed. > +#define VM_VFIO_ALLOW_WC BIT(VM_VFIO_ALLOW_WC_BIT) > +#else > +#define VM_VFIO_ALLOW_WC VM_NONE > +#endif And I think we need to add some documentation (is there any VFIO-specific doc) that describes what this flag actually means, what is permitted. For example, arm64 doesn't have write-combining without speculative fetches. So if one adds this flag to a new driver, they should know the implications. There's also an expectation that the actual driver (KVM guests) or maybe later DPDK can choose the safe non-cacheable or write-combine (Linux terminology) attributes for the BAR. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel