[PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

When a stage-2 permission fault doesn't report its IPA in HPFAR_EL2,
__get_fault_info() recovers it with an AT S1E1R/S1E1A on FAR_EL2. If that
fails, kvm_hyp_handle_memory_fault() goes straight back to the guest, on
the assumption that the replayed access will either succeed or take a
fault that the slow path can handle.

That assumption doesn't hold when the AT fails because its stage-1 walk
takes a stage-2 fault on one of the guest's page-table pages, while the
guest still holds a cached translation for the faulting VA. The guest
doesn't walk its page tables, so it never takes the S1PTW fault that
would let the slow path fix things up. It takes the same permission fault
again, the AT fails again, and the vCPU livelocks in the world switch.

This was observed on Apple M2, which has no hardware Access flag
management at stage 2, and no DIC, so execute permission is granted
lazily and the first instruction fetch from a page takes a permission
fault. With DAMON_STAT enabled (CONFIG_DAMON_STAT_ENABLED_DEFAULT=y),
kdamond keeps sampling guest memory, and damon_pmdp_mkold() ages a whole
PMD at a time through mmu_notifier_clear_young(). With THP-backed guest
memory, kvm_age_gfn() therefore clears the stage-2 AF of 32MB of guest
pages at once, without TLB invalidation. When that covers both freshly
loaded code and the page-table page that maps it, the first instruction
fetch from that code livelocks, and the vCPU only ever recovers when its
thread happens to be moved to another CPU. In all 52 instrumented
occurrences, the AT reported a stage-2 Access flag fault on a stage-1
table walk (PAR_EL1.{S,PTW} = {1,1}, FST = Access flag fault, level 3),
while an AT S12E1R on the same VA succeeded. With DAMON_STAT disabled,
the AT didn't fail once in 210 boots. Nothing in this is specific to
nested virtualisation, nor to instruction fetches: a write to a
read-only memslot hit it too.

A stage-1 fault can lead to the same livelock. A guest that ages its own
page tables without invalidating the TLB, as MGLRU does through
ptep_test_and_clear_young(), can keep using a cached translation while
the descriptor in memory has its Access flag clear. If such an access
takes a stage-2 permission fault (lazy execute permission, or dirty
logging), the AT reports a stage-1 Access flag fault (PAR_EL1.S = 0),
which the guest never takes either, as it doesn't walk its page tables.
This hasn't been observed, but it is why the invalidation below is done
whenever the AT fails, rather than only when PAR_EL1.PTW reports a
stage-2 fault on the walk. A guest that merely races with its own
page-table updates isn't harmed by it: at worst, it takes a translation
fault that it was allowed to take anyway.

Invalidate this VMID's TLB entries on the local CPU before replaying the
access, so that the guest has to walk its page tables and takes the
fault that the AT ran into: either a stage-2 fault on the page-table
page, with a valid HPFAR_EL2, which the slow path resolves, or a stage-1
fault, which the guest resolves itself. In all instrumented cases, the
next exit was an S1PTW Access flag fault on the page-table page. This
only happens when the AT fails, so the cost is negligible.

With this change, an EL2 UEFI guest that livelocked in 27 out of 210
boots didn't hang once in 420 boots.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
Cc: Marc Zyngier <maz@kernel.org>
Cc: Oliver Upton <oupton@kernel.org>
Cc: Fuad Tabba <fuad.tabba@linux.dev>
Cc: Joey Gouly <joey.gouly@arm.com>
Cc: Steffen Eiden <seiden@linux.ibm.com>
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Cc: Zenghui Yu <yuzenghui@huawei.com>

 arch/arm64/kvm/hyp/include/hyp/switch.h | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
index e0c55644224c..4ba61b71d511 100644
--- a/arch/arm64/kvm/hyp/include/hyp/switch.h
+++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
@@ -843,8 +843,21 @@ static inline bool kvm_hyp_handle_cp15_32(struct kvm_vcpu *vcpu, u64 *exit_code)
 static inline bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu,
 					       u64 *exit_code)
 {
-	if (!__populate_fault_info(vcpu))
+	if (!__populate_fault_info(vcpu)) {
+		/*
+		 * The AT couldn't resolve the IPA, typically because its
+		 * stage-1 walk took a stage-2 fault, such as an Access flag
+		 * fault on an aged page-table page. If the guest still has a
+		 * cached translation for the VA, it won't take that fault
+		 * itself, and would replay the same permission fault forever.
+		 * Drop this VMID's TLB entries on this CPU, so that the replay
+		 * walks the page tables and faults with a valid HPFAR_EL2.
+		 */
+		__tlbi(vmalls12e1);
+		dsb(nsh);
+		isb();
 		return true;
+	}
 
 	return false;
 }
-- 
2.56.0.317.g454f8302e1-goog

Re: [PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[Fuad Tabba]

Hi Ard,

On Mon, 02 Nov 2026 09:47:49 +0000, Ard Biesheuvel <ardb+git@google.com> wrote:
[...]
> When a stage-2 permission fault doesn't report its IPA in HPFAR_EL2,
> __get_fault_info() recovers it with an AT S1E1R/S1E1A on FAR_EL2. If that
> fails, kvm_hyp_handle_memory_fault() goes straight back to the guest, on
> the assumption that the replayed access will either succeed or take a
> fault that the slow path can handle.
[...]

Tested on QEMU (pKVM with protected and non-protected guests, VHE and
nVHE, and page_fault_test) and on an Apple M4 in nVHE, booting UEFI
guests with DAMON_STAT on. On the M4 I added a trace event to the new
branch, and it fired once in 120 boots, on a guest write to the UEFI
flash (a read-only memslot).

Since a guest can livelock on this, I was wondering if it should be Cc: stable.

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Tested-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad

Re: [PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[Will Deacon]

On Mon, Nov 02, 2026 at 10:47:49AM +0100, Ard Biesheuvel wrote:
> From: Ard Biesheuvel <ardb@kernel.org>
> 
> When a stage-2 permission fault doesn't report its IPA in HPFAR_EL2,
> __get_fault_info() recovers it with an AT S1E1R/S1E1A on FAR_EL2. If that
> fails, kvm_hyp_handle_memory_fault() goes straight back to the guest, on
> the assumption that the replayed access will either succeed or take a
> fault that the slow path can handle.
> 
> That assumption doesn't hold when the AT fails because its stage-1 walk
> takes a stage-2 fault on one of the guest's page-table pages, while the
> guest still holds a cached translation for the faulting VA. The guest
> doesn't walk its page tables, so it never takes the S1PTW fault that
> would let the slow path fix things up. It takes the same permission fault
> again, the AT fails again, and the vCPU livelocks in the world switch.
> 
> This was observed on Apple M2, which has no hardware Access flag
> management at stage 2, and no DIC, so execute permission is granted
> lazily and the first instruction fetch from a page takes a permission
> fault. With DAMON_STAT enabled (CONFIG_DAMON_STAT_ENABLED_DEFAULT=y),
> kdamond keeps sampling guest memory, and damon_pmdp_mkold() ages a whole
> PMD at a time through mmu_notifier_clear_young(). With THP-backed guest
> memory, kvm_age_gfn() therefore clears the stage-2 AF of 32MB of guest
> pages at once, without TLB invalidation. When that covers both freshly
> loaded code and the page-table page that maps it, the first instruction
> fetch from that code livelocks, and the vCPU only ever recovers when its
> thread happens to be moved to another CPU. In all 52 instrumented
> occurrences, the AT reported a stage-2 Access flag fault on a stage-1
> table walk (PAR_EL1.{S,PTW} = {1,1}, FST = Access flag fault, level 3),
> while an AT S12E1R on the same VA succeeded. With DAMON_STAT disabled,
> the AT didn't fail once in 210 boots. Nothing in this is specific to
> nested virtualisation, nor to instruction fetches: a write to a
> read-only memslot hit it too.
> 
> A stage-1 fault can lead to the same livelock. A guest that ages its own
> page tables without invalidating the TLB, as MGLRU does through
> ptep_test_and_clear_young(), can keep using a cached translation while
> the descriptor in memory has its Access flag clear. If such an access
> takes a stage-2 permission fault (lazy execute permission, or dirty
> logging), the AT reports a stage-1 Access flag fault (PAR_EL1.S = 0),
> which the guest never takes either, as it doesn't walk its page tables.
> This hasn't been observed, but it is why the invalidation below is done
> whenever the AT fails, rather than only when PAR_EL1.PTW reports a
> stage-2 fault on the walk. A guest that merely races with its own
> page-table updates isn't harmed by it: at worst, it takes a translation
> fault that it was allowed to take anyway.
> 
> Invalidate this VMID's TLB entries on the local CPU before replaying the
> access, so that the guest has to walk its page tables and takes the
> fault that the AT ran into: either a stage-2 fault on the page-table
> page, with a valid HPFAR_EL2, which the slow path resolves, or a stage-1
> fault, which the guest resolves itself. In all instrumented cases, the
> next exit was an S1PTW Access flag fault on the page-table page. This
> only happens when the AT fails, so the cost is negligible.
> 
> With this change, an EL2 UEFI guest that livelocked in 27 out of 210
> boots didn't hang once in 420 boots.
> 
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> ---
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Oliver Upton <oupton@kernel.org>
> Cc: Fuad Tabba <fuad.tabba@linux.dev>
> Cc: Joey Gouly <joey.gouly@arm.com>
> Cc: Steffen Eiden <seiden@linux.ibm.com>
> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
> Cc: Zenghui Yu <yuzenghui@huawei.com>
> 
>  arch/arm64/kvm/hyp/include/hyp/switch.h | 15 ++++++++++++++-
>  1 file changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
> index e0c55644224c..4ba61b71d511 100644
> --- a/arch/arm64/kvm/hyp/include/hyp/switch.h
> +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
> @@ -843,8 +843,21 @@ static inline bool kvm_hyp_handle_cp15_32(struct kvm_vcpu *vcpu, u64 *exit_code)
>  static inline bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu,
>  					       u64 *exit_code)
>  {
> -	if (!__populate_fault_info(vcpu))
> +	if (!__populate_fault_info(vcpu)) {

Presumably we need a barrier (DSB?) here, to ensure that the TLBI happens
after the AT instruction?

> +		/*
> +		 * The AT couldn't resolve the IPA, typically because its
> +		 * stage-1 walk took a stage-2 fault, such as an Access flag
> +		 * fault on an aged page-table page. If the guest still has a
> +		 * cached translation for the VA, it won't take that fault
> +		 * itself, and would replay the same permission fault forever.
> +		 * Drop this VMID's TLB entries on this CPU, so that the replay
> +		 * walks the page tables and faults with a valid HPFAR_EL2.
> +		 */
> +		__tlbi(vmalls12e1);

Can we get away with an ALLE1 here, or do we really need to blow away the
whole lot?

Will

Re: [PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[Marc Zyngier]

On Mon, 02 Nov 2026 13:59:12 +0000,
Will Deacon <will@kernel.org> wrote:
> 
> On Mon, Nov 02, 2026 at 10:47:49AM +0100, Ard Biesheuvel wrote:
> > From: Ard Biesheuvel <ardb@kernel.org>

[...]

> > diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
> > index e0c55644224c..4ba61b71d511 100644
> > --- a/arch/arm64/kvm/hyp/include/hyp/switch.h
> > +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
> > @@ -843,8 +843,21 @@ static inline bool kvm_hyp_handle_cp15_32(struct kvm_vcpu *vcpu, u64 *exit_code)
> >  static inline bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu,
> >  					       u64 *exit_code)
> >  {
> > -	if (!__populate_fault_info(vcpu))
> > +	if (!__populate_fault_info(vcpu)) {
> 
> Presumably we need a barrier (DSB?) here, to ensure that the TLBI happens
> after the AT instruction?
>
> > +		/*
> > +		 * The AT couldn't resolve the IPA, typically because its
> > +		 * stage-1 walk took a stage-2 fault, such as an Access flag
> > +		 * fault on an aged page-table page. If the guest still has a
> > +		 * cached translation for the VA, it won't take that fault
> > +		 * itself, and would replay the same permission fault forever.
> > +		 * Drop this VMID's TLB entries on this CPU, so that the replay
> > +		 * walks the page tables and faults with a valid HPFAR_EL2.
> > +		 */
> > +		__tlbi(vmalls12e1);
> 
> Can we get away with an ALLE1 here, or do we really need to blow away the
> whole lot?

ALLE1 blows EL1 translations for all VMIDs, while VMALLS12E1 is
localised to this VMID. Feels like the latter is the least invasive
option.

Alternatively, VMALLE1 or even VAE1 should be enough to get rid of the
S1 TLB.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[Will Deacon]

On Mon, Nov 02, 2026 at 02:09:25PM +0000, Marc Zyngier wrote:
> On Mon, 02 Nov 2026 13:59:12 +0000,
> Will Deacon <will@kernel.org> wrote:
> > 
> > On Mon, Nov 02, 2026 at 10:47:49AM +0100, Ard Biesheuvel wrote:
> > > From: Ard Biesheuvel <ardb@kernel.org>
> 
> [...]
> 
> > > diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
> > > index e0c55644224c..4ba61b71d511 100644
> > > --- a/arch/arm64/kvm/hyp/include/hyp/switch.h
> > > +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
> > > @@ -843,8 +843,21 @@ static inline bool kvm_hyp_handle_cp15_32(struct kvm_vcpu *vcpu, u64 *exit_code)
> > >  static inline bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu,
> > >  					       u64 *exit_code)
> > >  {
> > > -	if (!__populate_fault_info(vcpu))
> > > +	if (!__populate_fault_info(vcpu)) {
> > 
> > Presumably we need a barrier (DSB?) here, to ensure that the TLBI happens
> > after the AT instruction?
> >
> > > +		/*
> > > +		 * The AT couldn't resolve the IPA, typically because its
> > > +		 * stage-1 walk took a stage-2 fault, such as an Access flag
> > > +		 * fault on an aged page-table page. If the guest still has a
> > > +		 * cached translation for the VA, it won't take that fault
> > > +		 * itself, and would replay the same permission fault forever.
> > > +		 * Drop this VMID's TLB entries on this CPU, so that the replay
> > > +		 * walks the page tables and faults with a valid HPFAR_EL2.
> > > +		 */
> > > +		__tlbi(vmalls12e1);
> > 
> > Can we get away with an ALLE1 here, or do we really need to blow away the
> > whole lot?
> 
> ALLE1 blows EL1 translations for all VMIDs, while VMALLS12E1 is
> localised to this VMID. Feels like the latter is the least invasive
> option.
> 
> Alternatively, VMALLE1 or even VAE1 should be enough to get rid of the
> S1 TLB.

Ah, sorry, VMALLE1 is what I meant to ask about. I basically haven't
understood the need to invalidate the stage-2.

Will

Re: [PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[Fuad Tabba]

Hi Will,

On Mon, 02 Nov 2026 14:17:10 +0000, Will Deacon <will@kernel.org> wrote:
> On Mon, Nov 02, 2026 at 02:09:25PM +0000, Marc Zyngier wrote:
> > On Mon, 02 Nov 2026 13:59:12 +0000,
> > Will Deacon <will@kernel.org> wrote:
[...]
> > > > diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
[...]
> > > >  static inline bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu,
> > > >                                                 u64 *exit_code)
> > > >  {
> > > > -        if (!__populate_fault_info(vcpu))
> > > > +        if (!__populate_fault_info(vcpu)) {
> > >
> > > Presumably we need a barrier (DSB?) here, to ensure that the TLBI happens
> > > after the AT instruction?

I don't think the TLBI requires a barrier before it, because it
doesn't matter if the AT's walk allocates entries after the TLBI.
Anything it allocates matches memory, and the stage-2 descriptor the
AT faulted on has AF clear, so that descriptor can't be cached:

R_XCLRD: Translation table entries that generate a Translation
fault, an Address size fault, or an Access flag fault are never cached
in a TLB.

The stale entries the guest keeps hitting were already there before
the TLBI, and the dsb(nsh) and isb after it are what complete it
against them.

[... your other reply ... ]
> > Alternatively, VMALLE1 or even VAE1 should be enough to get rid of the
> > S1 TLB.
>
> Ah, sorry, VMALLE1 is what I meant to ask about. I basically haven't
> understood the need to invalidate the stage-2.

The guest's walk can also go through a stage-2 entry for its
page-table page, cached before the aging cleared that page's AF, and
nothing requires the AT to use that entry:

R_NPZWX: When an address translation instruction is executed, it is
IMPLEMENTATION DEFINED whether the result can be returned from a TLB,
or a translation table walk occurs.

So if my reading is correct, on a CPU whose AT walks memory instead,
the AT keeps failing while the guest's walk goes through the stale
entry, and the guest takes the same permission fault again. VMALLE1
and VAE1 only cover stage 1, so they'd leave that loop in place.

Cheers,
/fuad

Re: [PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[Ard Biesheuvel]

On Mon, 2 Nov 2026, at 16:57, Fuad Tabba wrote:
> Hi Will,
>
> On Mon, 02 Nov 2026 14:17:10 +0000, Will Deacon <will@kernel.org> wrote:
>> On Mon, Nov 02, 2026 at 02:09:25PM +0000, Marc Zyngier wrote:
>> [...]
>> > Alternatively, VMALLE1 or even VAE1 should be enough to get rid of the
>> > S1 TLB.
>>
>> Ah, sorry, VMALLE1 is what I meant to ask about. I basically haven't
>> understood the need to invalidate the stage-2.
>
> The guest's walk can also go through a stage-2 entry for its
> page-table page, cached before the aging cleared that page's AF, and
> nothing requires the AT to use that entry:
>
> R_NPZWX: When an address translation instruction is executed, it is
> IMPLEMENTATION DEFINED whether the result can be returned from a TLB,
> or a translation table walk occurs.
>
> So if my reading is correct, on a CPU whose AT walks memory instead,
> the AT keeps failing while the guest's walk goes through the stale
> entry, and the guest takes the same permission fault again. VMALLE1
> and VAE1 only cover stage 1, so they'd leave that loop in place.
>

This matches my findings on the Apple M2: TLB invalidation at stage 1
only (using VAAE1 on the FAR) does not solve the issue.

Re: [PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[Marc Zyngier]

On Mon, 02 Nov 2026 15:57:51 +0000,
Fuad Tabba <fuad.tabba@linux.dev> wrote:
> 
> Hi Will,
> 
> On Mon, 02 Nov 2026 14:17:10 +0000, Will Deacon <will@kernel.org> wrote:
> > On Mon, Nov 02, 2026 at 02:09:25PM +0000, Marc Zyngier wrote:
> > > On Mon, 02 Nov 2026 13:59:12 +0000,
> > > Will Deacon <will@kernel.org> wrote:
> [...]
> > > > > diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
> [...]
> > > > >  static inline bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu,
> > > > >                                                 u64 *exit_code)
> > > > >  {
> > > > > -        if (!__populate_fault_info(vcpu))
> > > > > +        if (!__populate_fault_info(vcpu)) {
> > > >
> > > > Presumably we need a barrier (DSB?) here, to ensure that the TLBI happens
> > > > after the AT instruction?
> 
> I don't think the TLBI requires a barrier before it, because it
> doesn't matter if the AT's walk allocates entries after the TLBI.
> Anything it allocates matches memory, and the stage-2 descriptor the
> AT faulted on has AF clear, so that descriptor can't be cached:
> 
> R_XCLRD: Translation table entries that generate a Translation
> fault, an Address size fault, or an Access flag fault are never cached
> in a TLB.
> 
> The stale entries the guest keeps hitting were already there before
> the TLBI, and the dsb(nsh) and isb after it are what complete it
> against them.
> 
> [... your other reply ... ]
> > > Alternatively, VMALLE1 or even VAE1 should be enough to get rid of the
> > > S1 TLB.
> >
> > Ah, sorry, VMALLE1 is what I meant to ask about. I basically haven't
> > understood the need to invalidate the stage-2.
> 
> The guest's walk can also go through a stage-2 entry for its
> page-table page, cached before the aging cleared that page's AF, and
> nothing requires the AT to use that entry:
> 
> R_NPZWX: When an address translation instruction is executed, it is
> IMPLEMENTATION DEFINED whether the result can be returned from a TLB,
> or a translation table walk occurs.
> 
> So if my reading is correct, on a CPU whose AT walks memory instead,
> the AT keeps failing while the guest's walk goes through the stale
> entry, and the guest takes the same permission fault again. VMALLE1
> and VAE1 only cover stage 1, so they'd leave that loop in place.

If we assume that the TLB covers VA->PA (which is a very likely case),
TLBI VAE1 should nuke the permission fault that brought the vcpu to
EL2, and instead walk the page tables, getting the access fault.

The only case I can imagine where invalidating S2 would matter is if
you are running under NV, because there is an independent S2 TLB (the
shadow S2), and then VMALLS12E1 makes total sense.

Ard, was this solely observed under NV? Or did you see it on a non NV
guest as well?

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[Ard Biesheuvel]

On Mon, 2 Nov 2026, at 17:52, Marc Zyngier wrote:
> On Mon, 02 Nov 2026 15:57:51 +0000,
> Fuad Tabba <fuad.tabba@linux.dev> wrote:
>> 
>> Hi Will,
>> 
>> On Mon, 02 Nov 2026 14:17:10 +0000, Will Deacon <will@kernel.org> wrote:
>> > On Mon, Nov 02, 2026 at 02:09:25PM +0000, Marc Zyngier wrote:
>> > > On Mon, 02 Nov 2026 13:59:12 +0000,
>> > > Will Deacon <will@kernel.org> wrote:
>> [...]
>> > > > > diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
>> [...]
>> > > > >  static inline bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu,
>> > > > >                                                 u64 *exit_code)
>> > > > >  {
>> > > > > -        if (!__populate_fault_info(vcpu))
>> > > > > +        if (!__populate_fault_info(vcpu)) {
>> > > >
>> > > > Presumably we need a barrier (DSB?) here, to ensure that the TLBI happens
>> > > > after the AT instruction?
>> 
>> I don't think the TLBI requires a barrier before it, because it
>> doesn't matter if the AT's walk allocates entries after the TLBI.
>> Anything it allocates matches memory, and the stage-2 descriptor the
>> AT faulted on has AF clear, so that descriptor can't be cached:
>> 
>> R_XCLRD: Translation table entries that generate a Translation
>> fault, an Address size fault, or an Access flag fault are never cached
>> in a TLB.
>> 
>> The stale entries the guest keeps hitting were already there before
>> the TLBI, and the dsb(nsh) and isb after it are what complete it
>> against them.
>> 
>> [... your other reply ... ]
>> > > Alternatively, VMALLE1 or even VAE1 should be enough to get rid of the
>> > > S1 TLB.
>> >
>> > Ah, sorry, VMALLE1 is what I meant to ask about. I basically haven't
>> > understood the need to invalidate the stage-2.
>> 
>> The guest's walk can also go through a stage-2 entry for its
>> page-table page, cached before the aging cleared that page's AF, and
>> nothing requires the AT to use that entry:
>> 
>> R_NPZWX: When an address translation instruction is executed, it is
>> IMPLEMENTATION DEFINED whether the result can be returned from a TLB,
>> or a translation table walk occurs.
>> 
>> So if my reading is correct, on a CPU whose AT walks memory instead,
>> the AT keeps failing while the guest's walk goes through the stale
>> entry, and the guest takes the same permission fault again. VMALLE1
>> and VAE1 only cover stage 1, so they'd leave that loop in place.
>
> If we assume that the TLB covers VA->PA (which is a very likely case),
> TLBI VAE1 should nuke the permission fault that brought the vcpu to
> EL2, and instead walk the page tables, getting the access fault.
>
> The only case I can imagine where invalidating S2 would matter is if
> you are running under NV, because there is an independent S2 TLB (the
> shadow S2), and then VMALLS12E1 makes total sense.
>
> Ard, was this solely observed under NV? Or did you see it on a non NV
> guest as well?
>

No it happens with guests running at EL1 too.

My experiments seem to indicate that on this uarch, the AT instruction
does not hit in the TLB, and so it can fail on AF==0 for an IPA that
is covered by a S2-only TLB entry (with AF==1 and XN==1 due to lazy exec).

So after the VAAE1, the S1 walk happily produces the same translation
we just invalidated, without taking a S1PTW fault, and the guest faults
again on the I-fetch that took us to hyp the first time around.

Re: [PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[Marc Zyngier]

On Mon, 02 Nov 2026 18:23:09 +0000,
"Ard Biesheuvel" <ardb@kernel.org> wrote:
> 
> 
> 
> On Mon, 2 Nov 2026, at 17:52, Marc Zyngier wrote:
> > On Mon, 02 Nov 2026 15:57:51 +0000,
> > Fuad Tabba <fuad.tabba@linux.dev> wrote:
> >> 
> >> Hi Will,
> >> 
> >> On Mon, 02 Nov 2026 14:17:10 +0000, Will Deacon <will@kernel.org> wrote:
> >> > On Mon, Nov 02, 2026 at 02:09:25PM +0000, Marc Zyngier wrote:
> >> > > On Mon, 02 Nov 2026 13:59:12 +0000,
> >> > > Will Deacon <will@kernel.org> wrote:
> >> [...]
> >> > > > > diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
> >> [...]
> >> > > > >  static inline bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu,
> >> > > > >                                                 u64 *exit_code)
> >> > > > >  {
> >> > > > > -        if (!__populate_fault_info(vcpu))
> >> > > > > +        if (!__populate_fault_info(vcpu)) {
> >> > > >
> >> > > > Presumably we need a barrier (DSB?) here, to ensure that the TLBI happens
> >> > > > after the AT instruction?
> >> 
> >> I don't think the TLBI requires a barrier before it, because it
> >> doesn't matter if the AT's walk allocates entries after the TLBI.
> >> Anything it allocates matches memory, and the stage-2 descriptor the
> >> AT faulted on has AF clear, so that descriptor can't be cached:
> >> 
> >> R_XCLRD: Translation table entries that generate a Translation
> >> fault, an Address size fault, or an Access flag fault are never cached
> >> in a TLB.
> >> 
> >> The stale entries the guest keeps hitting were already there before
> >> the TLBI, and the dsb(nsh) and isb after it are what complete it
> >> against them.
> >> 
> >> [... your other reply ... ]
> >> > > Alternatively, VMALLE1 or even VAE1 should be enough to get rid of the
> >> > > S1 TLB.
> >> >
> >> > Ah, sorry, VMALLE1 is what I meant to ask about. I basically haven't
> >> > understood the need to invalidate the stage-2.
> >> 
> >> The guest's walk can also go through a stage-2 entry for its
> >> page-table page, cached before the aging cleared that page's AF, and
> >> nothing requires the AT to use that entry:
> >> 
> >> R_NPZWX: When an address translation instruction is executed, it is
> >> IMPLEMENTATION DEFINED whether the result can be returned from a TLB,
> >> or a translation table walk occurs.
> >> 
> >> So if my reading is correct, on a CPU whose AT walks memory instead,
> >> the AT keeps failing while the guest's walk goes through the stale
> >> entry, and the guest takes the same permission fault again. VMALLE1
> >> and VAE1 only cover stage 1, so they'd leave that loop in place.
> >
> > If we assume that the TLB covers VA->PA (which is a very likely case),
> > TLBI VAE1 should nuke the permission fault that brought the vcpu to
> > EL2, and instead walk the page tables, getting the access fault.
> >
> > The only case I can imagine where invalidating S2 would matter is if
> > you are running under NV, because there is an independent S2 TLB (the
> > shadow S2), and then VMALLS12E1 makes total sense.
> >
> > Ard, was this solely observed under NV? Or did you see it on a non NV
> > guest as well?
> >
> 
> No it happens with guests running at EL1 too.

Damn!

> My experiments seem to indicate that on this uarch, the AT instruction
> does not hit in the TLB, and so it can fail on AF==0 for an IPA that
> is covered by a S2-only TLB entry (with AF==1 and XN==1 due to lazy exec).
> 
> So after the VAAE1, the S1 walk happily produces the same translation
> we just invalidated, without taking a S1PTW fault, and the guest faults
> again on the I-fetch that took us to hyp the first time around.

Ah, I think I see what can happen, even without split TLBs.

If you have a S2 walk cache that skips the descriptor where AF=0, then
the TLB refill just succeeds, and the cycle repeats. OK, I'm now
convinced we need S2 invalidation as well even in the VA->PA case. We
needed it for NV anyway, but hey...

Hopefully this doesn't happen too often.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[Will Deacon]

On Mon, Nov 02, 2026 at 06:41:15PM +0000, Marc Zyngier wrote:
> On Mon, 02 Nov 2026 18:23:09 +0000,
> "Ard Biesheuvel" <ardb@kernel.org> wrote:
> > My experiments seem to indicate that on this uarch, the AT instruction
> > does not hit in the TLB, and so it can fail on AF==0 for an IPA that
> > is covered by a S2-only TLB entry (with AF==1 and XN==1 due to lazy exec).
> > 
> > So after the VAAE1, the S1 walk happily produces the same translation
> > we just invalidated, without taking a S1PTW fault, and the guest faults
> > again on the I-fetch that took us to hyp the first time around.
> 
> Ah, I think I see what can happen, even without split TLBs.
> 
> If you have a S2 walk cache that skips the descriptor where AF=0, then
> the TLB refill just succeeds, and the cycle repeats. OK, I'm now
> convinced we need S2 invalidation as well even in the VA->PA case. We
> needed it for NV anyway, but hey...
> 
> Hopefully this doesn't happen too often.

Ard's observation in the commit message that kvm_mmu_notifier_clear_young()
doesn't do any TLB invalidation is also an interesting aspect to this, I
think. In that case, I don't think you need a particularly imaginative
TLB implementation: you just have AT always walking the S1 page-tables
in memory and seeing AF=0 at stage-2 when accessing a page-table page
whereas the MMU itself always hits in its (S2) TLB when doing the S1 walk.

For stage-1, we don't elide the TLBI quite like that -- instead, we elide
the trailing DSB (see __ptep_clear_flush_young() and the comment there).

Will

Re: [PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[Will Deacon]

On Mon, Nov 02, 2026 at 03:57:51PM +0000, Fuad Tabba wrote:
> On Mon, 02 Nov 2026 14:17:10 +0000, Will Deacon <will@kernel.org> wrote:
> > On Mon, Nov 02, 2026 at 02:09:25PM +0000, Marc Zyngier wrote:
> > > On Mon, 02 Nov 2026 13:59:12 +0000,
> > > Will Deacon <will@kernel.org> wrote:
> [...]
> > > > > diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
> [...]
> > > > >  static inline bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu,
> > > > >                                                 u64 *exit_code)
> > > > >  {
> > > > > -        if (!__populate_fault_info(vcpu))
> > > > > +        if (!__populate_fault_info(vcpu)) {
> > > >
> > > > Presumably we need a barrier (DSB?) here, to ensure that the TLBI happens
> > > > after the AT instruction?
> 
> I don't think the TLBI requires a barrier before it, because it
> doesn't matter if the AT's walk allocates entries after the TLBI.
> Anything it allocates matches memory, and the stage-2 descriptor the
> AT faulted on has AF clear, so that descriptor can't be cached:

I'm not sure that's relevant. The S1PTW case described by Ard in the
first part of his cover letter could look like this:

1. vCPU takes an S2 permission fault on VA 0x1000
	-> The permission faulting entry is in the TLB
2. Another vCPU unmaps a S1 page-table page used to translate 0x1000 from S2
3. EL2 does an AT S1E1R on 0x1000 as part of handling (1)
	-> Due to the uarch, this always walks the page-tables
4. The S1PTW from (3) faults at S2 because of (2)
	-> The AT fails

If we invalidate the TLB after step (4), then we will see the S1PTW
fault upon retrying the access that originally caused a permission fault
in (1). Without barriers, however, this invalidation could float up
between steps (1) and (2) and wouldn't be effective.

Now, having written it out, I think we would eventually resolve that by
replaying the faulting instruction a second time, but it feels odd to me
that we wouldn't just prevent this race once we know that the AT has
failed.

Will

Re: [PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[Marc Zyngier]

On Mon, 02 Nov 2026 09:47:49 +0000,
Ard Biesheuvel <ardb+git@google.com> wrote:
> 
> From: Ard Biesheuvel <ardb@kernel.org>
> 
> When a stage-2 permission fault doesn't report its IPA in HPFAR_EL2,
> __get_fault_info() recovers it with an AT S1E1R/S1E1A on FAR_EL2. If that
> fails, kvm_hyp_handle_memory_fault() goes straight back to the guest, on
> the assumption that the replayed access will either succeed or take a
> fault that the slow path can handle.
> 
> That assumption doesn't hold when the AT fails because its stage-1 walk
> takes a stage-2 fault on one of the guest's page-table pages, while the
> guest still holds a cached translation for the faulting VA. The guest
> doesn't walk its page tables, so it never takes the S1PTW fault that
> would let the slow path fix things up. It takes the same permission fault
> again, the AT fails again, and the vCPU livelocks in the world switch.

That's because the TLBs tend to cache VA->PA, while ATS1R is after the
IPA.  So you get the permission fault from the TLB, and the access
fault from AT. Rinse, repeat.

Amazing find, really. This code has been in this exact shape for 13
years, and it took that long to notice the issue.

[...]

> diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
> index e0c55644224c..4ba61b71d511 100644
> --- a/arch/arm64/kvm/hyp/include/hyp/switch.h
> +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
> @@ -843,8 +843,21 @@ static inline bool kvm_hyp_handle_cp15_32(struct kvm_vcpu *vcpu, u64 *exit_code)
>  static inline bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu,
>  					       u64 *exit_code)
>  {
> -	if (!__populate_fault_info(vcpu))
> +	if (!__populate_fault_info(vcpu)) {
> +		/*
> +		 * The AT couldn't resolve the IPA, typically because its
> +		 * stage-1 walk took a stage-2 fault, such as an Access flag
> +		 * fault on an aged page-table page. If the guest still has a
> +		 * cached translation for the VA, it won't take that fault
> +		 * itself, and would replay the same permission fault forever.
> +		 * Drop this VMID's TLB entries on this CPU, so that the replay
> +		 * walks the page tables and faults with a valid HPFAR_EL2.
> +		 */
> +		__tlbi(vmalls12e1);

As I was saying in my response to Will, I reckon this could be made
more focused on the faulting VA. Something like:

	__tlbi(vae1, read_sysreg_el2(SYS_FAR));

If you respin, a Cc: stable would be appreciated.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.