Re: LM regression: fce886a60207 KVM: arm64: Plumb the pKVM MMU in KVM

[Marc Zyngier <maz@kernel.org>]

On Mon, 05 May 2025 15:01:24 +0100,
Sebastian Ott <sebott@redhat.com> wrote:
> 
> On Mon, 5 May 2025, Marc Zyngier wrote:
> > On Mon, 05 May 2025 11:52:00 +0100,
> > Sebastian Ott <sebott@redhat.com> wrote:
> >> Doing back and forth migrations currently fails on arm after a couple iterations.
> >> During the failing migration KVM_RUN exits via guest_abort and returns -ENOMEM.
> >> I can reliably reproduce this by migrating between 2 qemu instances on an ampere
> >> altra machine. This fails after < 5 iterations. In this case qemu would spit out
> >> smth like this (other than that - nothing in the logs):
> >> 
> >> error: kvm run failed Cannot allocate memory
> >>  PC=0000aaaae7d48590 X00=0000aaaae80a2e00 X01=0000aaaae7ea2fc0
> >> X02=0000000001d3a5d0 X03=0000aaaae7eace8c X04=000000003b9aca00
> >> X05=000000000000004a X06=000000000000004a X07=0000000028000000
> >> X08=0000000000001d70 X09=0000000000000018 X10=000144b7d0000000
> >> X11=00ffffffffffffff X12=000000008378f367 X13=0000aaab1a202d70
> >> X14=0000000000000000 X15=0000000000000000 X16=0000ffffa2e2f7a8
> >> X17=0000ffffa2541f20 X18=000000000000a000 X19=84bfda6288cf2dd6
> >> X20=0000aaab1a1f1ce0 X21=000000007fffffff X22=0000ffffc5431788
> >> X23=0000aaab1a17db60 X24=0000ffffc5431770 X25=0000000100000000
> >> X26=0000004100000000 X27=0000000000000001 X28=0000aaab1a1f1c20
> >> X29=0000ffffc54316d0 X30=0000aaaae7f8cd24  SP=0000ffffc5431650
> >> PSTATE=20001000 --C- EL0t
> >> 
> >> Guest and host are otherwise idle, kvm is in normal VHE mode.
> >> 
> >> Git bisect points to (fce886a60207 "KVM: arm64: Plumb the pKVM MMU in KVM")
> >> I also double checked that by reverting this commit on top of 6.14.
> > 
> > Thanks for find the triggering commit. Can you further identify *what*
> > causes the -ENOMEM? The only new -ENOMEM in that patch is the one
> > added to topup_hyp_memcache(), which shouldn't be called.
> 
> The kvm_pgtable_stage2_map() call in user_mem_abort() returns -ENOMEM
> because the memcache pointer was not initialized!
> 
> It looks like smth like this without other conditions could do the trick:
> 
> if (!is_protected_kvm_enabled())
> 	memcache = &vcpu->arch.mmu_page_cache;
> else
> 	memcache = &vcpu->arch.pkvm_memcache;
> 
> (I'll try this now)

Right, we end-up with an uninitialised memcache variable. Why isn't
the compiler screaming?

I think you can indeed simply hoist the init of memcache very early
on, which should solve hopefully solve the issue.

> 
> > Also, a failure to allocate would leave some nastygram in the kernel
> > log, so it is unlikely to be an actual failure to allocate.
> > 
> > Is it the first KVM_RUN that fails after migration?
> 
> Nope, it happens on the side that triggers the migration.

Probably because splitting pages requires allocating some memory, and
all of a sudden you trigger the allocation from the memcache. Boo.

Thanks for spotting this, I'm looking forward to the fix!

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.