From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5EB46C433EF for ; Fri, 1 Apr 2022 14:21:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hTk47wdB2ftVUEU+pibm0c/S3WdC80OIlobnVlAn3Qc=; b=wxe0z45IYiJkv2 LgpsqVgv7z4lKkoH5VhjH2nYktXGm2MVegaz6QLMS6N4+YRQTYnFEabgw307BHgRwL3r7lMmiiyL+ 5qz4ETjp3kMJJla9KYfbo+W1YGLSbAOaXaMyEAQuLPfgoiN7RMNqQW8eKkeuZFbzgqqHbf6scGaon 8JuuomxRPnlubRwxhWHn0cPOAOMmQfV+ywrrPm6ZYeGj8lvn0stGOG8LLazELqGmo4b9xnNMIi8OZ zyHHNit7oAFbrdJDoK8lD6/6kOCxzENfRkOrColHu1aFAaYzjWgIINFQGArT4bmiNzi4SVgllSmmu z8R5d9NMq+C+XzxenPDA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1naI8C-005yFz-Be; Fri, 01 Apr 2022 14:20:04 +0000 Received: from mail-qk1-x735.google.com ([2607:f8b0:4864:20::735]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1naI84-005yDD-Sq for linux-arm-kernel@lists.infradead.org; Fri, 01 Apr 2022 14:19:58 +0000 Received: by mail-qk1-x735.google.com with SMTP id 1so2216094qke.1 for ; Fri, 01 Apr 2022 07:19:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ACCW8/bZraS0ZT4HrvKJE4Pim6an9ErxJI12ETZ6D8I=; b=etQwil6Hu1/NkwZEZI+ruAJC/+UQAuF2gjvBW+a4Ih820ydCI9emF5h5zgBE+pBPJV cuMaeIl8o+BQ30fEKqE0/wlfYBp+snjQpdzrj5JN4TG+yiRvP/oPCU3+kp6Cf01O5tJo KJcVxHHO3W1AuJK9ejBbe5awuERLWJNVwHF8oktiI38J+1x67GYhGWNV1N5WXpQ3C5Kh GR7suSkXiXE8inO+YigNntJAtBSWcDWuAzQkZl0vPVu0Og95P59hGBr8dfBPbAow2QFE cWslkhRfXsp/OPJK7IxYXWnENGPOs5PXsiHNQyXdEMHd+IDjV2DdPQ8IqIh9RCHDfOOC RCrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ACCW8/bZraS0ZT4HrvKJE4Pim6an9ErxJI12ETZ6D8I=; b=PXdW+I2CgVR/f54S2NJCXjvvZvwZOIRoHawqJoXJdTwXzo+Onv8RyxT3ksuiscGg8U KtuWWXRRzw44evlP5Ej7Wq8EMXYvxufTsHsYQOfSnrCukr4Vi3RvUALW7nOQ7nGzaWWG jIQUArc3BNJ0C9RWJLj7E/DGUrsMoTise6PltmQcqN2pzS+PJIAsdpgVUhL2ZTV54DVK soq32tGh6yjm8F49sdCe3bpUtpnuODRUOixYQNjLxcS6JadpcetgZ3LMAWW2Jgl068iK WPQ46bc6FSsnK+ziIrkjdQdVuCYAG5Kn6kxHSnqQAx/lAcCBIvk2TyzTJBzJVgHONLiv 2ofw== X-Gm-Message-State: AOAM530vHKDDxEvZPIcfVywWZQzDVI68t8OeeuBrJ6ZFYMLhhgqPgXR9 +dcSWqHajhcEJ4T3Kmmxw5NvDg== X-Google-Smtp-Source: ABdhPJwUx/m0xIPLwWyjukTkeRO9Ht5F0EMHxDa8njMh8pspLckalcH4xcSuiH46I/amIZ6JQPChYA== X-Received: by 2002:a05:620a:4486:b0:67e:ccd4:990c with SMTP id x6-20020a05620a448600b0067eccd4990cmr6785363qkp.643.1648822793423; Fri, 01 Apr 2022 07:19:53 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-129.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.129]) by smtp.gmail.com with ESMTPSA id y11-20020ac85f4b000000b002e1e038a8fdsm1843780qta.13.2022.04.01.07.19.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Apr 2022 07:19:52 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1naI7z-00ATnc-C8; Fri, 01 Apr 2022 11:19:51 -0300 Date: Fri, 1 Apr 2022 11:19:51 -0300 From: Jason Gunthorpe To: xieming Cc: marc.zyngier@arm.com, linux@armlinux.org.uk, catalin.marinas@arm.com, will.deacon@arm.com, alex.williamson@redhat.com, sashal@kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2] kvm/arm64: fixed passthrough gpu into vm on arm64 Message-ID: <20220401141951.GP64706@ziepe.ca> References: <20220401090828.614167-1-xieming@kylinos.cn> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20220401090828.614167-1-xieming@kylinos.cn> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220401_071957_246742_3D8A5E51 X-CRM114-Status: GOOD ( 20.01 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Apr 01, 2022 at 05:08:28PM +0800, xieming wrote: > diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c > index 51b791c750f1..6f66efb71743 100644 > +++ b/drivers/vfio/pci/vfio_pci.c > @@ -1452,7 +1452,14 @@ static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma) > } > > vma->vm_private_data = vdev; > +#ifdef CONFIG_ARM64 > + if (vfio_pci_is_vga(pdev)) > + vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot); > + else > + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); > +#else > vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); > +#endif This is a user visible change if VFIO starts making things write combining then userspace has to have different barriers around MMIO. Also this problem is bigger than just GPUs, lots of devices use write combining memory for their BARs and will do so inside VMs as well - so testing for 'pci_is_vga' is also not right. I think you need to solve this by having userspace somehow request the cachability type for the mmap (though I'm not sure how KVM will know what to do with that), or by having kvm always force write combining for all ioremaps.. > +/** > + * is_vma_write_combine - check if VMA is mapped with writecombine or not > + * Return true if VMA mapped with MT_NORMAL_NC otherwise fasle > + */ > +static inline bool is_vma_write_combine(struct vm_area_struct *vma) > +{ > + pteval_t pteval = pgprot_val(vma->vm_page_prot); > + > + return ((pteval & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_NC)); > +} Shouldn't KVM be copying the exact pgprot bits from the VMA to the KVM PTEs when it is mirroring them? eg the difference between pgprot_device and pgprot_noncached() seems relevant to preserve as well. > @@ -1209,7 +1221,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, > pfn = __phys_to_pfn(pa); > > for (addr = guest_ipa; addr < end; addr += PAGE_SIZE) { > - pte_t pte = pfn_pte(pfn, PAGE_S2_DEVICE); > + pte_t pte = pfn_pte(pfn, writecombine ? PAGE_S2_NC : PAGE_S2_DEVICE); Please don't send patches to the mailing list that are against such old kernels, this code was deleted in 2020. Jason _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel