From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F3CEC433FE for ; Fri, 1 Apr 2022 16:24:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id C8BFD40B91; Fri, 1 Apr 2022 12:24:16 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@ziepe.ca Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N7pE5rm3ybCw; Fri, 1 Apr 2022 12:24:15 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id EB5FB4B1F2; Fri, 1 Apr 2022 12:24:12 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 0D9EE49EE3 for ; Fri, 1 Apr 2022 10:19:55 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uSaz+m+1TUsj for ; Fri, 1 Apr 2022 10:19:53 -0400 (EDT) Received: from mail-qk1-f182.google.com (mail-qk1-f182.google.com [209.85.222.182]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id E144F49EC0 for ; Fri, 1 Apr 2022 10:19:53 -0400 (EDT) Received: by mail-qk1-f182.google.com with SMTP id q200so2188908qke.7 for ; Fri, 01 Apr 2022 07:19:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ACCW8/bZraS0ZT4HrvKJE4Pim6an9ErxJI12ETZ6D8I=; b=etQwil6Hu1/NkwZEZI+ruAJC/+UQAuF2gjvBW+a4Ih820ydCI9emF5h5zgBE+pBPJV cuMaeIl8o+BQ30fEKqE0/wlfYBp+snjQpdzrj5JN4TG+yiRvP/oPCU3+kp6Cf01O5tJo KJcVxHHO3W1AuJK9ejBbe5awuERLWJNVwHF8oktiI38J+1x67GYhGWNV1N5WXpQ3C5Kh GR7suSkXiXE8inO+YigNntJAtBSWcDWuAzQkZl0vPVu0Og95P59hGBr8dfBPbAow2QFE cWslkhRfXsp/OPJK7IxYXWnENGPOs5PXsiHNQyXdEMHd+IDjV2DdPQ8IqIh9RCHDfOOC RCrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ACCW8/bZraS0ZT4HrvKJE4Pim6an9ErxJI12ETZ6D8I=; b=vgT2Ufe0R4QJUQ/QIfGVelSXjdF6ANTUBHXPtQCiBWlYjn2rOAq84cSpiVSqxuGmga xpmc4ZW7+Z1hXwpzaFbtgZ2r57fSyUrJdvVi4zWQObnNAOpZ8HYzk6Q0SmTKj9sTxxLd bcWu80I2PaRCjTjh9X9mfPMfC73gErJPc3PmB+F3YsuRhYyaE76F0nbY76jifro05AHN UR+CQKil9/pBmconDePyTNQgoynODwTLHLXLcgidBEFAnHQxBjXY+pcEgkTUHYmp1ckm VrybFaNjhLwpI+4RUaA2hjLLmc5d3ALajism6WQHSF+3pDJxFi3DqNNcj5KObCrd/kdR EySQ== X-Gm-Message-State: AOAM530QD6+Av43oXkdQgEK0PML8jFG9KdX4Fjfd8xvXC4HnKkYbcRD7 WbeXfGD0NsV0tl0To6jXO2T4uQ== X-Google-Smtp-Source: ABdhPJwUx/m0xIPLwWyjukTkeRO9Ht5F0EMHxDa8njMh8pspLckalcH4xcSuiH46I/amIZ6JQPChYA== X-Received: by 2002:a05:620a:4486:b0:67e:ccd4:990c with SMTP id x6-20020a05620a448600b0067eccd4990cmr6785363qkp.643.1648822793423; Fri, 01 Apr 2022 07:19:53 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-129.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.129]) by smtp.gmail.com with ESMTPSA id y11-20020ac85f4b000000b002e1e038a8fdsm1843780qta.13.2022.04.01.07.19.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Apr 2022 07:19:52 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1naI7z-00ATnc-C8; Fri, 01 Apr 2022 11:19:51 -0300 Date: Fri, 1 Apr 2022 11:19:51 -0300 From: Jason Gunthorpe To: xieming Subject: Re: [PATCH v2] kvm/arm64: fixed passthrough gpu into vm on arm64 Message-ID: <20220401141951.GP64706@ziepe.ca> References: <20220401090828.614167-1-xieming@kylinos.cn> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20220401090828.614167-1-xieming@kylinos.cn> X-Mailman-Approved-At: Fri, 01 Apr 2022 12:24:11 -0400 Cc: sashal@kernel.org, marc.zyngier@arm.com, catalin.marinas@arm.com, will.deacon@arm.com, linux@armlinux.org.uk, linux-kernel@vger.kernel.org, alex.williamson@redhat.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Fri, Apr 01, 2022 at 05:08:28PM +0800, xieming wrote: > diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c > index 51b791c750f1..6f66efb71743 100644 > +++ b/drivers/vfio/pci/vfio_pci.c > @@ -1452,7 +1452,14 @@ static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma) > } > > vma->vm_private_data = vdev; > +#ifdef CONFIG_ARM64 > + if (vfio_pci_is_vga(pdev)) > + vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot); > + else > + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); > +#else > vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); > +#endif This is a user visible change if VFIO starts making things write combining then userspace has to have different barriers around MMIO. Also this problem is bigger than just GPUs, lots of devices use write combining memory for their BARs and will do so inside VMs as well - so testing for 'pci_is_vga' is also not right. I think you need to solve this by having userspace somehow request the cachability type for the mmap (though I'm not sure how KVM will know what to do with that), or by having kvm always force write combining for all ioremaps.. > +/** > + * is_vma_write_combine - check if VMA is mapped with writecombine or not > + * Return true if VMA mapped with MT_NORMAL_NC otherwise fasle > + */ > +static inline bool is_vma_write_combine(struct vm_area_struct *vma) > +{ > + pteval_t pteval = pgprot_val(vma->vm_page_prot); > + > + return ((pteval & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_NC)); > +} Shouldn't KVM be copying the exact pgprot bits from the VMA to the KVM PTEs when it is mirroring them? eg the difference between pgprot_device and pgprot_noncached() seems relevant to preserve as well. > @@ -1209,7 +1221,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, > pfn = __phys_to_pfn(pa); > > for (addr = guest_ipa; addr < end; addr += PAGE_SIZE) { > - pte_t pte = pfn_pte(pfn, PAGE_S2_DEVICE); > + pte_t pte = pfn_pte(pfn, writecombine ? PAGE_S2_NC : PAGE_S2_DEVICE); Please don't send patches to the mailing list that are against such old kernels, this code was deleted in 2020. Jason _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5EB46C433EF for ; Fri, 1 Apr 2022 14:21:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hTk47wdB2ftVUEU+pibm0c/S3WdC80OIlobnVlAn3Qc=; b=wxe0z45IYiJkv2 LgpsqVgv7z4lKkoH5VhjH2nYktXGm2MVegaz6QLMS6N4+YRQTYnFEabgw307BHgRwL3r7lMmiiyL+ 5qz4ETjp3kMJJla9KYfbo+W1YGLSbAOaXaMyEAQuLPfgoiN7RMNqQW8eKkeuZFbzgqqHbf6scGaon 8JuuomxRPnlubRwxhWHn0cPOAOMmQfV+ywrrPm6ZYeGj8lvn0stGOG8LLazELqGmo4b9xnNMIi8OZ zyHHNit7oAFbrdJDoK8lD6/6kOCxzENfRkOrColHu1aFAaYzjWgIINFQGArT4bmiNzi4SVgllSmmu z8R5d9NMq+C+XzxenPDA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1naI8C-005yFz-Be; Fri, 01 Apr 2022 14:20:04 +0000 Received: from mail-qk1-x735.google.com ([2607:f8b0:4864:20::735]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1naI84-005yDD-Sq for linux-arm-kernel@lists.infradead.org; Fri, 01 Apr 2022 14:19:58 +0000 Received: by mail-qk1-x735.google.com with SMTP id 1so2216094qke.1 for ; Fri, 01 Apr 2022 07:19:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ACCW8/bZraS0ZT4HrvKJE4Pim6an9ErxJI12ETZ6D8I=; b=etQwil6Hu1/NkwZEZI+ruAJC/+UQAuF2gjvBW+a4Ih820ydCI9emF5h5zgBE+pBPJV cuMaeIl8o+BQ30fEKqE0/wlfYBp+snjQpdzrj5JN4TG+yiRvP/oPCU3+kp6Cf01O5tJo KJcVxHHO3W1AuJK9ejBbe5awuERLWJNVwHF8oktiI38J+1x67GYhGWNV1N5WXpQ3C5Kh GR7suSkXiXE8inO+YigNntJAtBSWcDWuAzQkZl0vPVu0Og95P59hGBr8dfBPbAow2QFE cWslkhRfXsp/OPJK7IxYXWnENGPOs5PXsiHNQyXdEMHd+IDjV2DdPQ8IqIh9RCHDfOOC RCrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ACCW8/bZraS0ZT4HrvKJE4Pim6an9ErxJI12ETZ6D8I=; b=PXdW+I2CgVR/f54S2NJCXjvvZvwZOIRoHawqJoXJdTwXzo+Onv8RyxT3ksuiscGg8U KtuWWXRRzw44evlP5Ej7Wq8EMXYvxufTsHsYQOfSnrCukr4Vi3RvUALW7nOQ7nGzaWWG jIQUArc3BNJ0C9RWJLj7E/DGUrsMoTise6PltmQcqN2pzS+PJIAsdpgVUhL2ZTV54DVK soq32tGh6yjm8F49sdCe3bpUtpnuODRUOixYQNjLxcS6JadpcetgZ3LMAWW2Jgl068iK WPQ46bc6FSsnK+ziIrkjdQdVuCYAG5Kn6kxHSnqQAx/lAcCBIvk2TyzTJBzJVgHONLiv 2ofw== X-Gm-Message-State: AOAM530vHKDDxEvZPIcfVywWZQzDVI68t8OeeuBrJ6ZFYMLhhgqPgXR9 +dcSWqHajhcEJ4T3Kmmxw5NvDg== X-Google-Smtp-Source: ABdhPJwUx/m0xIPLwWyjukTkeRO9Ht5F0EMHxDa8njMh8pspLckalcH4xcSuiH46I/amIZ6JQPChYA== X-Received: by 2002:a05:620a:4486:b0:67e:ccd4:990c with SMTP id x6-20020a05620a448600b0067eccd4990cmr6785363qkp.643.1648822793423; Fri, 01 Apr 2022 07:19:53 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-129.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.129]) by smtp.gmail.com with ESMTPSA id y11-20020ac85f4b000000b002e1e038a8fdsm1843780qta.13.2022.04.01.07.19.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Apr 2022 07:19:52 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1naI7z-00ATnc-C8; Fri, 01 Apr 2022 11:19:51 -0300 Date: Fri, 1 Apr 2022 11:19:51 -0300 From: Jason Gunthorpe To: xieming Cc: marc.zyngier@arm.com, linux@armlinux.org.uk, catalin.marinas@arm.com, will.deacon@arm.com, alex.williamson@redhat.com, sashal@kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2] kvm/arm64: fixed passthrough gpu into vm on arm64 Message-ID: <20220401141951.GP64706@ziepe.ca> References: <20220401090828.614167-1-xieming@kylinos.cn> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20220401090828.614167-1-xieming@kylinos.cn> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220401_071957_246742_3D8A5E51 X-CRM114-Status: GOOD ( 20.01 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Apr 01, 2022 at 05:08:28PM +0800, xieming wrote: > diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c > index 51b791c750f1..6f66efb71743 100644 > +++ b/drivers/vfio/pci/vfio_pci.c > @@ -1452,7 +1452,14 @@ static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma) > } > > vma->vm_private_data = vdev; > +#ifdef CONFIG_ARM64 > + if (vfio_pci_is_vga(pdev)) > + vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot); > + else > + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); > +#else > vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); > +#endif This is a user visible change if VFIO starts making things write combining then userspace has to have different barriers around MMIO. Also this problem is bigger than just GPUs, lots of devices use write combining memory for their BARs and will do so inside VMs as well - so testing for 'pci_is_vga' is also not right. I think you need to solve this by having userspace somehow request the cachability type for the mmap (though I'm not sure how KVM will know what to do with that), or by having kvm always force write combining for all ioremaps.. > +/** > + * is_vma_write_combine - check if VMA is mapped with writecombine or not > + * Return true if VMA mapped with MT_NORMAL_NC otherwise fasle > + */ > +static inline bool is_vma_write_combine(struct vm_area_struct *vma) > +{ > + pteval_t pteval = pgprot_val(vma->vm_page_prot); > + > + return ((pteval & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_NC)); > +} Shouldn't KVM be copying the exact pgprot bits from the VMA to the KVM PTEs when it is mirroring them? eg the difference between pgprot_device and pgprot_noncached() seems relevant to preserve as well. > @@ -1209,7 +1221,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, > pfn = __phys_to_pfn(pa); > > for (addr = guest_ipa; addr < end; addr += PAGE_SIZE) { > - pte_t pte = pfn_pte(pfn, PAGE_S2_DEVICE); > + pte_t pte = pfn_pte(pfn, writecombine ? PAGE_S2_NC : PAGE_S2_DEVICE); Please don't send patches to the mailing list that are against such old kernels, this code was deleted in 2020. Jason _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85A3DC433EF for ; Fri, 1 Apr 2022 14:19:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346792AbiDAOVq (ORCPT ); Fri, 1 Apr 2022 10:21:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240154AbiDAOVo (ORCPT ); Fri, 1 Apr 2022 10:21:44 -0400 Received: from mail-qk1-x72f.google.com (mail-qk1-x72f.google.com [IPv6:2607:f8b0:4864:20::72f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C0A14C7B2 for ; Fri, 1 Apr 2022 07:19:54 -0700 (PDT) Received: by mail-qk1-x72f.google.com with SMTP id v13so2205939qkv.3 for ; Fri, 01 Apr 2022 07:19:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ACCW8/bZraS0ZT4HrvKJE4Pim6an9ErxJI12ETZ6D8I=; b=etQwil6Hu1/NkwZEZI+ruAJC/+UQAuF2gjvBW+a4Ih820ydCI9emF5h5zgBE+pBPJV cuMaeIl8o+BQ30fEKqE0/wlfYBp+snjQpdzrj5JN4TG+yiRvP/oPCU3+kp6Cf01O5tJo KJcVxHHO3W1AuJK9ejBbe5awuERLWJNVwHF8oktiI38J+1x67GYhGWNV1N5WXpQ3C5Kh GR7suSkXiXE8inO+YigNntJAtBSWcDWuAzQkZl0vPVu0Og95P59hGBr8dfBPbAow2QFE cWslkhRfXsp/OPJK7IxYXWnENGPOs5PXsiHNQyXdEMHd+IDjV2DdPQ8IqIh9RCHDfOOC RCrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ACCW8/bZraS0ZT4HrvKJE4Pim6an9ErxJI12ETZ6D8I=; b=ulIkFLrnRjTu6txA1F8lPrP17CNIx0w4D6HY3uD+M4bE4J6JCLIaYAsGoZyTmS5dIG w3x6kUY/o7JL8f1Djs71yJIzwbl53thRY2bQp85/3exqCAhyNiinqMt+KKTyTrSA1N7g SPGVkG8YUnXu0Fc6JbGxo5jt0hfAwPTtaarn5uWYiAC6Z9BGqFhkNg2HNsBPco+4KXh2 c9Amaew5yLOugsBGqqLANtvlyENyj2QYJQnk10Ik+jiU5pq8Nr859qcdHJmjHxDmV4G5 0uc0sJFJZjtIl1pe8NBwTlsQeL/27Dr0IP/5ZUqUgZVQy1Q5Y4AI4tMthBzkntk6Ej8X D9Gw== X-Gm-Message-State: AOAM533GEFnC8x5v5MhC4AKc1M3TRNYb3s2AW/vmKo61rmBQfaIstb8n vgvvbTI4opJgUoQ/hstEnbjPvg== X-Google-Smtp-Source: ABdhPJwUx/m0xIPLwWyjukTkeRO9Ht5F0EMHxDa8njMh8pspLckalcH4xcSuiH46I/amIZ6JQPChYA== X-Received: by 2002:a05:620a:4486:b0:67e:ccd4:990c with SMTP id x6-20020a05620a448600b0067eccd4990cmr6785363qkp.643.1648822793423; Fri, 01 Apr 2022 07:19:53 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-129.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.129]) by smtp.gmail.com with ESMTPSA id y11-20020ac85f4b000000b002e1e038a8fdsm1843780qta.13.2022.04.01.07.19.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Apr 2022 07:19:52 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1naI7z-00ATnc-C8; Fri, 01 Apr 2022 11:19:51 -0300 Date: Fri, 1 Apr 2022 11:19:51 -0300 From: Jason Gunthorpe To: xieming Cc: marc.zyngier@arm.com, linux@armlinux.org.uk, catalin.marinas@arm.com, will.deacon@arm.com, alex.williamson@redhat.com, sashal@kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2] kvm/arm64: fixed passthrough gpu into vm on arm64 Message-ID: <20220401141951.GP64706@ziepe.ca> References: <20220401090828.614167-1-xieming@kylinos.cn> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220401090828.614167-1-xieming@kylinos.cn> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Apr 01, 2022 at 05:08:28PM +0800, xieming wrote: > diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c > index 51b791c750f1..6f66efb71743 100644 > +++ b/drivers/vfio/pci/vfio_pci.c > @@ -1452,7 +1452,14 @@ static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma) > } > > vma->vm_private_data = vdev; > +#ifdef CONFIG_ARM64 > + if (vfio_pci_is_vga(pdev)) > + vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot); > + else > + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); > +#else > vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); > +#endif This is a user visible change if VFIO starts making things write combining then userspace has to have different barriers around MMIO. Also this problem is bigger than just GPUs, lots of devices use write combining memory for their BARs and will do so inside VMs as well - so testing for 'pci_is_vga' is also not right. I think you need to solve this by having userspace somehow request the cachability type for the mmap (though I'm not sure how KVM will know what to do with that), or by having kvm always force write combining for all ioremaps.. > +/** > + * is_vma_write_combine - check if VMA is mapped with writecombine or not > + * Return true if VMA mapped with MT_NORMAL_NC otherwise fasle > + */ > +static inline bool is_vma_write_combine(struct vm_area_struct *vma) > +{ > + pteval_t pteval = pgprot_val(vma->vm_page_prot); > + > + return ((pteval & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_NC)); > +} Shouldn't KVM be copying the exact pgprot bits from the VMA to the KVM PTEs when it is mirroring them? eg the difference between pgprot_device and pgprot_noncached() seems relevant to preserve as well. > @@ -1209,7 +1221,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, > pfn = __phys_to_pfn(pa); > > for (addr = guest_ipa; addr < end; addr += PAGE_SIZE) { > - pte_t pte = pfn_pte(pfn, PAGE_S2_DEVICE); > + pte_t pte = pfn_pte(pfn, writecombine ? PAGE_S2_NC : PAGE_S2_DEVICE); Please don't send patches to the mailing list that are against such old kernels, this code was deleted in 2020. Jason