From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1BFCC433DB for ; Fri, 22 Jan 2021 09:47:13 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 92DA9235E4 for ; Fri, 22 Jan 2021 09:47:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 92DA9235E4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Type: Content-Transfer-Encoding:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Message-ID:References:In-Reply-To:Subject:To:From: Date:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=xD+ZrLZuxEQFsLYZw4eg92QgywnGp2QSFS3hDJ6NNZs=; b=uPJZEFzXHMDOel0uHiy19llnH oc4KhnzH9avYFo2ali50BnBfmUBmdfXiDlJyKf1ET57yCTZ/auZH32HVvPcNHu7+aX+vZjxkg9Qwv vtzc5W08LFIMRnQxADVdcXJ6JKLFHbQY467zuHUMEn2Mb4l0PzQx8u8NuLSCQHJVWc9hahIABRGpa o6lsV15ZcsTnSQpzY2G+ClFZbhzJ0dMJrP4EooUJy4GXGcYMpxaYb/vMMAYC9tk3wHkla1lo1OfbI 86lYV7YegoWsXJDQXlDnxvnUxyiNgtEEDUeGfvb0FMNpfgAL6JSdF/VXS1auEABxj5G7Rz+2xOFAQ A3zHvb0uQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l2t0c-0006WG-2P; Fri, 22 Jan 2021 09:45:38 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l2t0U-0006Uh-1k for linux-arm-kernel@lists.infradead.org; Fri, 22 Jan 2021 09:45:31 +0000 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 045AE22DBF; Fri, 22 Jan 2021 09:45:29 +0000 (UTC) Received: from disco-boy.misterjones.org ([51.254.78.96] helo=www.loen.fr) by disco-boy.misterjones.org with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94) (envelope-from ) id 1l2t0Q-009NVo-MF; Fri, 22 Jan 2021 09:45:26 +0000 MIME-Version: 1.0 Date: Fri, 22 Jan 2021 09:45:26 +0000 From: Marc Zyngier To: Keqian Zhu Subject: Re: [RFC PATCH] kvm: arm64: Try stage2 block mapping for host device MMIO In-Reply-To: <20210122083650.21812-1-zhukeqian1@huawei.com> References: <20210122083650.21812-1-zhukeqian1@huawei.com> User-Agent: Roundcube Webmail/1.4.10 Message-ID: <09d89355cdbbd19c456699774a9a980a@kernel.org> X-Sender: maz@kernel.org X-SA-Exim-Connect-IP: 51.254.78.96 X-SA-Exim-Rcpt-To: zhukeqian1@huawei.com, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, james.morse@arm.com, robin.murphy@arm.com, joro@8bytes.org, daniel.lezcano@linaro.org, tglx@linutronix.de, suzuki.poulose@arm.com, julien.thierry.kdev@gmail.com, akpm@linux-foundation.org, alexios.zavras@intel.com, wanghaibin.wang@huawei.com, jiangkunkun@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210122_044530_269814_EC1671F7 X-CRM114-Status: GOOD ( 26.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , jiangkunkun@huawei.com, Andrew Morton , kvm@vger.kernel.org, Suzuki K Poulose , Catalin Marinas , Joerg Roedel , Daniel Lezcano , linux-kernel@vger.kernel.org, Alexios Zavras , James Morse , Julien Thierry , wanghaibin.wang@huawei.com, Thomas Gleixner , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, Robin Murphy Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 2021-01-22 08:36, Keqian Zhu wrote: > The MMIO region of a device maybe huge (GB level), try to use block > mapping in stage2 to speedup both map and unmap. > > Especially for unmap, it performs TLBI right after each invalidation > of PTE. If all mapping is of PAGE_SIZE, it takes much time to handle > GB level range. This is only on VM teardown, right? Or do you unmap the device more ofet? Can you please quantify the speedup and the conditions this occurs in? I have the feeling that we are just circling around another problem, which is that we could rely on a VM-wide TLBI when tearing down the guest. I worked on something like that[1] a long while ago, and parked it for some reason. Maybe it is worth reviving. [1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/elide-cmo-tlbi > > Signed-off-by: Keqian Zhu > --- > arch/arm64/include/asm/kvm_pgtable.h | 11 +++++++++++ > arch/arm64/kvm/hyp/pgtable.c | 15 +++++++++++++++ > arch/arm64/kvm/mmu.c | 12 ++++++++---- > 3 files changed, 34 insertions(+), 4 deletions(-) > > diff --git a/arch/arm64/include/asm/kvm_pgtable.h > b/arch/arm64/include/asm/kvm_pgtable.h > index 52ab38db04c7..2266ac45f10c 100644 > --- a/arch/arm64/include/asm/kvm_pgtable.h > +++ b/arch/arm64/include/asm/kvm_pgtable.h > @@ -82,6 +82,17 @@ struct kvm_pgtable_walker { > const enum kvm_pgtable_walk_flags flags; > }; > > +/** > + * kvm_supported_pgsize() - Get the max supported page size of a > mapping. > + * @pgt: Initialised page-table structure. > + * @addr: Virtual address at which to place the mapping. > + * @end: End virtual address of the mapping. > + * @phys: Physical address of the memory to map. > + * > + * The smallest return value is PAGE_SIZE. > + */ > +u64 kvm_supported_pgsize(struct kvm_pgtable *pgt, u64 addr, u64 end, > u64 phys); > + > /** > * kvm_pgtable_hyp_init() - Initialise a hypervisor stage-1 > page-table. > * @pgt: Uninitialised page-table structure to initialise. > diff --git a/arch/arm64/kvm/hyp/pgtable.c > b/arch/arm64/kvm/hyp/pgtable.c > index bdf8e55ed308..ab11609b9b13 100644 > --- a/arch/arm64/kvm/hyp/pgtable.c > +++ b/arch/arm64/kvm/hyp/pgtable.c > @@ -81,6 +81,21 @@ static bool kvm_block_mapping_supported(u64 addr, > u64 end, u64 phys, u32 level) > return IS_ALIGNED(addr, granule) && IS_ALIGNED(phys, granule); > } > > +u64 kvm_supported_pgsize(struct kvm_pgtable *pgt, u64 addr, u64 end, > u64 phys) > +{ > + u32 lvl; > + u64 pgsize = PAGE_SIZE; > + > + for (lvl = pgt->start_level; lvl < KVM_PGTABLE_MAX_LEVELS; lvl++) { > + if (kvm_block_mapping_supported(addr, end, phys, lvl)) { > + pgsize = kvm_granule_size(lvl); > + break; > + } > + } > + > + return pgsize; > +} > + > static u32 kvm_pgtable_idx(struct kvm_pgtable_walk_data *data, u32 > level) > { > u64 shift = kvm_granule_shift(level); > diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c > index 7d2257cc5438..80b403fc8e64 100644 > --- a/arch/arm64/kvm/mmu.c > +++ b/arch/arm64/kvm/mmu.c > @@ -499,7 +499,8 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) > int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, > phys_addr_t pa, unsigned long size, bool writable) > { > - phys_addr_t addr; > + phys_addr_t addr, end; > + unsigned long pgsize; > int ret = 0; > struct kvm_mmu_memory_cache cache = { 0, __GFP_ZERO, NULL, }; > struct kvm_pgtable *pgt = kvm->arch.mmu.pgt; > @@ -509,21 +510,24 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, > phys_addr_t guest_ipa, > > size += offset_in_page(guest_ipa); > guest_ipa &= PAGE_MASK; > + end = guest_ipa + size; > > - for (addr = guest_ipa; addr < guest_ipa + size; addr += PAGE_SIZE) { > + for (addr = guest_ipa; addr < end; addr += pgsize) { > ret = kvm_mmu_topup_memory_cache(&cache, > kvm_mmu_cache_min_pages(kvm)); > if (ret) > break; > > + pgsize = kvm_supported_pgsize(pgt, addr, end, pa); > + > spin_lock(&kvm->mmu_lock); > - ret = kvm_pgtable_stage2_map(pgt, addr, PAGE_SIZE, pa, prot, > + ret = kvm_pgtable_stage2_map(pgt, addr, pgsize, pa, prot, > &cache); > spin_unlock(&kvm->mmu_lock); > if (ret) > break; > > - pa += PAGE_SIZE; > + pa += pgsize; > } > > kvm_mmu_free_memory_cache(&cache); This otherwise looks neat enough. Thanks, M. -- Jazz is not dead. It just smells funny... _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel