[PATCH] KVM: arm64: Always invalidate TLB for stage-2 permission faults

[PATCH] KVM: arm64: Always invalidate TLB for stage-2 permission faults

[Oliver Upton]

It is possible for multiple vCPUs to fault on the same IPA and attempt
to resolve the fault. One of the page table walks will actually update
the PTE and the rest will return -EAGAIN per our race detection scheme.
KVM elides the TLB invalidation on the racing threads as the return
value is nonzero.

Before commit a12ab1378a88 ("KVM: arm64: Use local TLBI on permission
relaxation") KVM always used broadcast TLB invalidations when handling
permission faults, which had the convenient property of making the
stage-2 updates visible to all CPUs in the system. However now we do a
local invalidation, and TLBI elision leads to vCPUs getting stuck in a
permission fault loop. Remember that the architecture permits the TLB to
cache translations that precipitate a permission fault.

Invalidate the TLB entry responsible for the permission fault if the
stage-2 descriptor has been relaxed, regardless of which thread actually
did the job.

Cc: stable@vger.kernel.org
Fixes: a12ab1378a88 ("KVM: arm64: Use local TLBI on permission relaxation")
Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
---
 arch/arm64/kvm/hyp/pgtable.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index f155b8c9e98c..286888751793 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -1314,7 +1314,7 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
 	ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level,
 				       KVM_PGTABLE_WALK_HANDLE_FAULT |
 				       KVM_PGTABLE_WALK_SHARED);
-	if (!ret)
+	if (!ret || ret == -EAGAIN)
 		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa_nsh, pgt->mmu, addr, level);
 	return ret;
 }

base-commit: ce9ecca0238b140b88f43859b211c9fdfd8e5b70
-- 
2.42.0.515.g380fc7ccd1-goog

Re: [PATCH] KVM: arm64: Always invalidate TLB for stage-2 permission faults

[Oliver Upton]

On Fri, Sep 22, 2023 at 10:32:29PM +0000, Oliver Upton wrote:
> It is possible for multiple vCPUs to fault on the same IPA and attempt
> to resolve the fault. One of the page table walks will actually update
> the PTE and the rest will return -EAGAIN per our race detection scheme.
> KVM elides the TLB invalidation on the racing threads as the return
> value is nonzero.
> 
> Before commit a12ab1378a88 ("KVM: arm64: Use local TLBI on permission
> relaxation") KVM always used broadcast TLB invalidations when handling
> permission faults, which had the convenient property of making the
> stage-2 updates visible to all CPUs in the system. However now we do a
> local invalidation, and TLBI elision leads to vCPUs getting stuck in a
> permission fault loop. Remember that the architecture permits the TLB to
> cache translations that precipitate a permission fault.

The effects of this are slightly overstated (got ahead of myself).
EAGAIN only crops up if the cmpxchg() fails, we return 0 if the PTE
didn't need to be updated.

On the subsequent permission fault we'll do the right thing and
invalidate the TLB, so this change is purely an optimization rather than
a correctness issue.

> Invalidate the TLB entry responsible for the permission fault if the
> stage-2 descriptor has been relaxed, regardless of which thread actually
> did the job.
> 
> Cc: stable@vger.kernel.org
> Fixes: a12ab1378a88 ("KVM: arm64: Use local TLBI on permission relaxation")

I'll drop the stable tag.

> Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
> ---
>  arch/arm64/kvm/hyp/pgtable.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
> index f155b8c9e98c..286888751793 100644
> --- a/arch/arm64/kvm/hyp/pgtable.c
> +++ b/arch/arm64/kvm/hyp/pgtable.c
> @@ -1314,7 +1314,7 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
>  	ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level,
>  				       KVM_PGTABLE_WALK_HANDLE_FAULT |
>  				       KVM_PGTABLE_WALK_SHARED);
> -	if (!ret)
> +	if (!ret || ret == -EAGAIN)
>  		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa_nsh, pgt->mmu, addr, level);
>  	return ret;
>  }
> 
> base-commit: ce9ecca0238b140b88f43859b211c9fdfd8e5b70
> -- 
> 2.42.0.515.g380fc7ccd1-goog
> 

-- 
Thanks,
Oliver

Re: [PATCH] KVM: arm64: Always invalidate TLB for stage-2 permission faults

[Marc Zyngier]

On Sat, 23 Sep 2023 00:08:21 +0100,
Oliver Upton <oliver.upton@linux.dev> wrote:
> 
> On Fri, Sep 22, 2023 at 10:32:29PM +0000, Oliver Upton wrote:
> > It is possible for multiple vCPUs to fault on the same IPA and attempt
> > to resolve the fault. One of the page table walks will actually update
> > the PTE and the rest will return -EAGAIN per our race detection scheme.
> > KVM elides the TLB invalidation on the racing threads as the return
> > value is nonzero.
> > 
> > Before commit a12ab1378a88 ("KVM: arm64: Use local TLBI on permission
> > relaxation") KVM always used broadcast TLB invalidations when handling
> > permission faults, which had the convenient property of making the
> > stage-2 updates visible to all CPUs in the system. However now we do a
> > local invalidation, and TLBI elision leads to vCPUs getting stuck in a
> > permission fault loop. Remember that the architecture permits the TLB to
> > cache translations that precipitate a permission fault.
> 
> The effects of this are slightly overstated (got ahead of myself).
> EAGAIN only crops up if the cmpxchg() fails, we return 0 if the PTE
> didn't need to be updated.
> 
> On the subsequent permission fault we'll do the right thing and
> invalidate the TLB, so this change is purely an optimization rather than
> a correctness issue.

Can you measure the actual effect of this change? In my (limited)
experience, I had to actually trick the guest into doing this, and
opportunistically invalidating TLBs didn't have any significant
benefit.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] KVM: arm64: Always invalidate TLB for stage-2 permission faults

[Oliver Upton]

On Sun, Sep 24, 2023 at 11:12:30AM +0100, Marc Zyngier wrote:
> On Sat, 23 Sep 2023 00:08:21 +0100,
> Oliver Upton <oliver.upton@linux.dev> wrote:
> > 
> > On Fri, Sep 22, 2023 at 10:32:29PM +0000, Oliver Upton wrote:
> > > It is possible for multiple vCPUs to fault on the same IPA and attempt
> > > to resolve the fault. One of the page table walks will actually update
> > > the PTE and the rest will return -EAGAIN per our race detection scheme.
> > > KVM elides the TLB invalidation on the racing threads as the return
> > > value is nonzero.
> > > 
> > > Before commit a12ab1378a88 ("KVM: arm64: Use local TLBI on permission
> > > relaxation") KVM always used broadcast TLB invalidations when handling
> > > permission faults, which had the convenient property of making the
> > > stage-2 updates visible to all CPUs in the system. However now we do a
> > > local invalidation, and TLBI elision leads to vCPUs getting stuck in a
> > > permission fault loop. Remember that the architecture permits the TLB to
> > > cache translations that precipitate a permission fault.
> > 
> > The effects of this are slightly overstated (got ahead of myself).
> > EAGAIN only crops up if the cmpxchg() fails, we return 0 if the PTE
> > didn't need to be updated.
> > 
> > On the subsequent permission fault we'll do the right thing and
> > invalidate the TLB, so this change is purely an optimization rather than
> > a correctness issue.
> 
> Can you measure the actual effect of this change? In my (limited)
> experience, I had to actually trick the guest into doing this, and
> opportunistically invalidating TLBs didn't have any significant
> benefit.

Sure. We were debugging some issues of vCPU hangs during post-copy
migration but that's more likely to be an issue with our VMM + out of
tree code.

Marginal improvements be damned, I'm still somewhat keen on doing the
TLB invalidation upon race detection anyway. Going back to the guest is
pointless, since in all likelihood we will hit the TLB entry that led to
the permission fault in the first place.

-- 
Thanks,
Oliver

Re: [PATCH] KVM: arm64: Always invalidate TLB for stage-2 permission faults

[Marc Zyngier]

On Tue, 26 Sep 2023 00:43:21 +0100,
Oliver Upton <oliver.upton@linux.dev> wrote:
> 
> On Sun, Sep 24, 2023 at 11:12:30AM +0100, Marc Zyngier wrote:
> > On Sat, 23 Sep 2023 00:08:21 +0100,
> > Oliver Upton <oliver.upton@linux.dev> wrote:
> > > 
> > > On Fri, Sep 22, 2023 at 10:32:29PM +0000, Oliver Upton wrote:
> > > > It is possible for multiple vCPUs to fault on the same IPA and attempt
> > > > to resolve the fault. One of the page table walks will actually update
> > > > the PTE and the rest will return -EAGAIN per our race detection scheme.
> > > > KVM elides the TLB invalidation on the racing threads as the return
> > > > value is nonzero.
> > > > 
> > > > Before commit a12ab1378a88 ("KVM: arm64: Use local TLBI on permission
> > > > relaxation") KVM always used broadcast TLB invalidations when handling
> > > > permission faults, which had the convenient property of making the
> > > > stage-2 updates visible to all CPUs in the system. However now we do a
> > > > local invalidation, and TLBI elision leads to vCPUs getting stuck in a
> > > > permission fault loop. Remember that the architecture permits the TLB to
> > > > cache translations that precipitate a permission fault.
> > > 
> > > The effects of this are slightly overstated (got ahead of myself).
> > > EAGAIN only crops up if the cmpxchg() fails, we return 0 if the PTE
> > > didn't need to be updated.
> > > 
> > > On the subsequent permission fault we'll do the right thing and
> > > invalidate the TLB, so this change is purely an optimization rather than
> > > a correctness issue.
> > 
> > Can you measure the actual effect of this change? In my (limited)
> > experience, I had to actually trick the guest into doing this, and
> > opportunistically invalidating TLBs didn't have any significant
> > benefit.
> 
> Sure. We were debugging some issues of vCPU hangs during post-copy
> migration but that's more likely to be an issue with our VMM + out of
> tree code.
> 
> Marginal improvements be damned, I'm still somewhat keen on doing the
> TLB invalidation upon race detection anyway. Going back to the guest is
> pointless, since in all likelihood we will hit the TLB entry that led to
> the permission fault in the first place.

I guess it completely depends on the size of the TLB. The machines I
deal with have a relatively small number of entries, and it doesn't
take much to fully evict them.

Now, all of that is probably irrelevant as there should be little
impact in performing the invalidation as long as it is local (unless
you trap, but that's another problem).

FWIW:

Acked-by: Marc Zyngier <maz@kernel.org>

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] KVM: arm64: Always invalidate TLB for stage-2 permission faults

[Oliver Upton]

On Fri, 22 Sep 2023 22:32:29 +0000, Oliver Upton wrote:
> It is possible for multiple vCPUs to fault on the same IPA and attempt
> to resolve the fault. One of the page table walks will actually update
> the PTE and the rest will return -EAGAIN per our race detection scheme.
> KVM elides the TLB invalidation on the racing threads as the return
> value is nonzero.
> 
> Before commit a12ab1378a88 ("KVM: arm64: Use local TLBI on permission
> relaxation") KVM always used broadcast TLB invalidations when handling
> permission faults, which had the convenient property of making the
> stage-2 updates visible to all CPUs in the system. However now we do a
> local invalidation, and TLBI elision leads to vCPUs getting stuck in a
> permission fault loop. Remember that the architecture permits the TLB to
> cache translations that precipitate a permission fault.
> 
> [...]

Applied to kvmarm/next, with the fixes and stable tag dropped.

[1/1] KVM: arm64: Always invalidate TLB for stage-2 permission faults
      https://git.kernel.org/kvmarm/kvmarm/c/5a6e594fc607

--
Best,
Oliver