Re: Any way to disable KVM VHE extension?

[Marc Zyngier <maz@kernel.org>]

On Thu, 15 Jul 2021 10:44:32 +0100,
Qu Wenruo <wqu@suse.com> wrote:
> 
> 
> 
> On 2021/7/15 下午5:28, Robin Murphy wrote:
> > On 2021-07-15 09:55, Qu Wenruo wrote:
> >> Hi,
> >> 
> >> Recently I'm playing around the Nvidia Xavier AGX board, which has
> >> VHE extension support.
> >> 
> >> In theory, considering the CPU and memory, it should be pretty
> >> powerful compared to boards like RPI CM4.
> >> 
> >> But to my surprise, KVM runs pretty poor on Xavier.
> >> 
> >> Just booting the edk2 firmware could take over 10s, and 20s to
> >> fully boot the kernel.
> >> Even my VM on RPI CM4 has way faster boot time, even just running
> >> on PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
> >> 
> >> This is definitely out of my expectation, I double checked to be
> >> sure that it's running in KVM mode.
> >> 
> >> But further digging shows that, since Xavier AGX CPU supports VHE,
> >> kvm is running in VHE mode other than HYP mode on CM4.
> >> 
> >> Is there anyway to manually disable VHE mode to test the more
> >> common HYP mode on Xavier?
> > 
> > According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its
> > low-level equivalent "id_aa64mmfr1.vh=0") on the command line should
> > do that.
> 
> Thanks for this one, I stupidly only searched modinfo of kvm, and
> didn't even bother to search arch/arm64/kvm...
> 
> > 
> > However I'd imagine the discrepancy is likely to be something more
> > fundamental to the wildly different microarchitectures. There's
> > certainly no harm in giving non-VHE a go for comparison, but I
> > wouldn't be surprised if it turns out even slower...
> 
> You're totally right, with nvhe mode, it's still the same slow speed.

My experience with Denver (Nvidia's previous core) is that it is
horribly slow when running KVM. I guess that the JIT-like microarch
fares poorly with exceptions and save-restore operations.

> BTW, what did you mean by the "wildly different microarch"?
> Is ARMv8.2 arch that different from ARMv8 of RPI4?
> 
> And any extra methods I could try to explore the reason of the slowness?
> 
> At least RPI CM4 is beyond my expectation and is working pretty fine.
> 
> Thanks,
> Qu
> 
> > 
> > Robin.
> > 
> >> BTW, this is the dmesg related to KVM on Xavier, running v5.13
> >> upstream kernel, with 64K page size:
> >> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
> >> [    0.857378] kvm [1]: vgic interrupt IRQ9
> >> [    0.862122] kvm: pmu event creation failed -2

And this isn't going to help finding out the bottleneck, as the kernel
doesn't find a PMU. On Denver, once the PMU is enabled, profiling
anything makes the whole thing even slower. At which point, I just
parked the board and forgot about it.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.