Re: [PATCH v10 00/59] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

From: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com>
To: Marc Zyngier <maz@kernel.org>,
	Miguel Luis <miguel.luis@oracle.com>,
	Eric Auger <eauger@redhat.com>
Cc: kvmarm@lists.linux.dev, kvm@vger.kernel.org,
	linux-arm-kernel@lists.infradead.org,
	Alexandru Elisei <alexandru.elisei@arm.com>,
	Andre Przywara <andre.przywara@arm.com>,
	Chase Conklin <chase.conklin@arm.com>,
	Christoffer Dall <christoffer.dall@arm.com>,
	Darren Hart <darren@os.amperecomputing.com>,
	Jintack Lim <jintack@cs.columbia.edu>,
	Russell King <rmk+kernel@armlinux.org.uk>,
	James Morse <james.morse@arm.com>,
	Suzuki K Poulose <suzuki.poulose@arm.com>,
	Oliver Upton <oliver.upton@linux.dev>,
	Zenghui Yu <yuzenghui@huawei.com>
Subject: Re: [PATCH v10 00/59] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support
Date: Fri, 7 Jul 2023 15:16:49 +0530	[thread overview]
Message-ID: <04ec8efb-33ff-153e-3be5-5c84a01bff2a@os.amperecomputing.com> (raw)
In-Reply-To: <a62dba1a-86b7-45ab-f6a8-f88b224af402@os.amperecomputing.com>

On 04-07-2023 06:01 pm, Ganapatrao Kulkarni wrote:
> 
> Hi Marc,
> 
> On 29-06-2023 12:33 pm, Marc Zyngier wrote:
>> Hi Ganapatrao,
>>
>> On Wed, 28 Jun 2023 07:45:55 +0100,
>> Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com> wrote:
>>>
>>>
>>> Hi Marc,
>>>
>>>
>>> On 15-05-2023 11:00 pm, Marc Zyngier wrote:
>>>> This is the 4th drop of NV support on arm64 for this year.
>>>>
>>>> For the previous episodes, see [1].
>>>>
>>>> What's changed:
>>>>
>>>> - New framework to track system register traps that are reinjected in
>>>>     guest EL2. It is expected to replace the discrete handling we have
>>>>     enjoyed so far, which didn't scale at all. This has already fixed a
>>>>     number of bugs that were hidden (a bunch of traps were never
>>>>     forwarded...). Still a work in progress, but this is going in the
>>>>     right direction.
>>>>
>>>> - Allow the L1 hypervisor to have a S2 that has an input larger than
>>>>     the L0 IPA space. This fixes a number of subtle issues, 
>>>> depending on
>>>>     how the initial guest was created.
>>>>
>>>> - Consequently, the patch series has gone longer again. Boo. But
>>>>     hopefully some of it is easier to review...
>>>>
>>>
>>> I am facing issue in booting NestedVM with V9 as well with 10 patchset.
>>>
>>> I have tried V9/V10 on Ampere platform using kvmtool and I could boot
>>> Guest-Hypervisor and then NestedVM without any issue.
>>> However when I try to boot using QEMU(not using EDK2/EFI),
>>> Guest-Hypervisor is booted with Fedora 37 using virtio disk. From
>>> Guest-Hypervisor console(or ssh shell), If I try to boot NestedVM,
>>> boot hangs very early stage of the boot.
>>>
>>> I did some debug using ftrace and it seems the Guest-Hypervisor is
>>> getting very high rate of arch-timer interrupts,
>>> due to that all CPU time is going on in serving the Guest-Hypervisor
>>> and it is never going back to NestedVM.
>>>
>>> I am using QEMU vanilla version v7.2.0 with top-up patches for NV [1]
>>
>> So I went ahead and gave QEMU a go. On my systems, *nothing* works (I
>> cannot even boot a L1 with 'virtualization=on" (the guest is stuck at
>> the point where virtio gets probed and waits for its first interrupt).
>>
>> Worse, booting a hVHE guest results in QEMU generating an assert as it
>> tries to inject an interrupt using the QEMU GICv3 model, something
>> that should *NEVER* be in use with KVM.
>>
>> With help from Eric, I got to a point where the hVHE guest could boot
>> as long as I kept injecting console interrupts, which is again a
>> symptom of the vGIC not being used.
>>
>> So something is *majorly* wrong with the QEMU patches. I don't know
>> what makes it possible for you to even boot the L1 - if the GIC is
>> external, injecting an interrupt in the L2 is simply impossible.
>>
>> Miguel, can you please investigate this?
>>
>> In the meantime, I'll add some code to the kernel side to refuse the
>> external interrupt controller configuration with NV. Hopefully that
>> will lead to some clues about what is going on.
> 
> Continued debugging of the issue and it seems the endless ptimer 
> interrupts on Ampere platform is due to some mess up of CVAL of ptimer, 
> resulting in interrupt triggered always when it is enabled.
> 
> I see function "timer_set_offset" called from kvm_arm_timer_set_reg in 
> QEMU case but there is no such calls in kvmtool boot.
> 
> If I comment the timer_set_offset calls in kvm_arm_timer_set_reg 
> function, then I could boot the Guest-Hypervisor then NestedVM from GH/L1.
> 
> I also observed in QEMU case, kvm_arm_timer_set_reg is called to set 
> CNT, CVAL and CTL of both vtimer and ptimer.
> Not sure why QEMU is setting these registers explicitly? need to dig.
> 

I don't see any direct ioctl calls to change any timer registers. Looks 
like it is happening from the emulation code(target/arm/helper.c)?

>>
>> Thanks,
>>
>>     M.
>>
> 
> Thanks,
> Ganapat
> 

Thanks,
Ganapat

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel