From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16012C47E48 for ; Thu, 15 Jul 2021 10:50:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 008436120A for ; Thu, 15 Jul 2021 10:50:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241712AbhGOKxA convert rfc822-to-8bit (ORCPT ); Thu, 15 Jul 2021 06:53:00 -0400 Received: from mail.kernel.org ([198.145.29.99]:58456 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232055AbhGOKw7 (ORCPT ); Thu, 15 Jul 2021 06:52:59 -0400 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 84B0961002; Thu, 15 Jul 2021 10:50:06 +0000 (UTC) Received: from sofa.misterjones.org ([185.219.108.64] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1m3ywO-00DVgb-FB; Thu, 15 Jul 2021 11:50:04 +0100 Date: Thu, 15 Jul 2021 11:49:54 +0100 Message-ID: <87o8b3kg19.wl-maz@kernel.org> From: Marc Zyngier To: Qu Wenruo Cc: Robin Murphy , kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, Linux ARM Subject: Re: Any way to disable KVM VHE extension? In-Reply-To: <0e992d47-1f17-d49f-8341-670770ac49ef@suse.com> References: <37f873cf-1b39-ea7f-a5e7-6feb0200dd4c@suse.com> <0e992d47-1f17-d49f-8341-670770ac49ef@suse.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8BIT X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: wqu@suse.com, robin.murphy@arm.com, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Thu, 15 Jul 2021 10:44:32 +0100, Qu Wenruo wrote: > > > > On 2021/7/15 下午5:28, Robin Murphy wrote: > > On 2021-07-15 09:55, Qu Wenruo wrote: > >> Hi, > >> > >> Recently I'm playing around the Nvidia Xavier AGX board, which has > >> VHE extension support. > >> > >> In theory, considering the CPU and memory, it should be pretty > >> powerful compared to boards like RPI CM4. > >> > >> But to my surprise, KVM runs pretty poor on Xavier. > >> > >> Just booting the edk2 firmware could take over 10s, and 20s to > >> fully boot the kernel. > >> Even my VM on RPI CM4 has way faster boot time, even just running > >> on PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core. > >> > >> This is definitely out of my expectation, I double checked to be > >> sure that it's running in KVM mode. > >> > >> But further digging shows that, since Xavier AGX CPU supports VHE, > >> kvm is running in VHE mode other than HYP mode on CM4. > >> > >> Is there anyway to manually disable VHE mode to test the more > >> common HYP mode on Xavier? > > > > According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its > > low-level equivalent "id_aa64mmfr1.vh=0") on the command line should > > do that. > > Thanks for this one, I stupidly only searched modinfo of kvm, and > didn't even bother to search arch/arm64/kvm... > > > > > However I'd imagine the discrepancy is likely to be something more > > fundamental to the wildly different microarchitectures. There's > > certainly no harm in giving non-VHE a go for comparison, but I > > wouldn't be surprised if it turns out even slower... > > You're totally right, with nvhe mode, it's still the same slow speed. My experience with Denver (Nvidia's previous core) is that it is horribly slow when running KVM. I guess that the JIT-like microarch fares poorly with exceptions and save-restore operations. > BTW, what did you mean by the "wildly different microarch"? > Is ARMv8.2 arch that different from ARMv8 of RPI4? > > And any extra methods I could try to explore the reason of the slowness? > > At least RPI CM4 is beyond my expectation and is working pretty fine. > > Thanks, > Qu > > > > > Robin. > > > >> BTW, this is the dmesg related to KVM on Xavier, running v5.13 > >> upstream kernel, with 64K page size: > >> [    0.852357] kvm [1]: IPA Size Limit: 40 bits > >> [    0.857378] kvm [1]: vgic interrupt IRQ9 > >> [    0.862122] kvm: pmu event creation failed -2 And this isn't going to help finding out the bottleneck, as the kernel doesn't find a PMU. On Denver, once the PMU is enabled, profiling anything makes the whole thing even slower. At which point, I just parked the board and forgot about it. Thanks, M. -- Without deviation from the norm, progress is not possible.