* [PATCH v5 1/6] KVM: arm64: Use a variable for the canonical IPA in kvm_s2_fault_map()
2026-08-10 20:50 [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure) Wei-Lin Chang
@ 2026-08-10 20:50 ` Wei-Lin Chang
2026-08-10 20:50 ` [PATCH v5 2/6] KVM: arm64: nv: Introduce guest stage-2 tracking structures Wei-Lin Chang
` (6 subsequent siblings)
7 siblings, 0 replies; 25+ messages in thread
From: Wei-Lin Chang @ 2026-08-10 20:50 UTC (permalink / raw)
To: linux-arm-kernel, kvmarm, linux-kernel
Cc: Marc Zyngier, Oliver Upton, Fuad Tabba, Joey Gouly, Steffen Eiden,
Suzuki K Poulose, Zenghui Yu, Catalin Marinas, Will Deacon,
Lorenzo Stoakes, Itaru Kitayama, Wei-Lin Chang
Create a variable to store the canonical IPA, instead of calculating it
when needed. This will be useful when we need to use the canonical IPA
for guest stage-2 tracking later.
Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
---
arch/arm64/kvm/mmu.c | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 6c941aaa10c6..336dd8f7e8ab 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -2021,6 +2021,7 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
enum kvm_pgtable_walk_flags flags = KVM_PGTABLE_WALK_SHARED;
bool writable = prot & KVM_PGTABLE_PROT_W;
struct kvm *kvm = s2fd->vcpu->kvm;
+ phys_addr_t canonical_ipa;
struct kvm_pgtable *pgt;
long perm_fault_granule;
long mapping_size;
@@ -2039,6 +2040,7 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
mapping_size = s2vi->vma_pagesize;
pfn = s2vi->pfn;
gfn = s2vi->gfn;
+ canonical_ipa = gfn_to_gpa(get_canonical_gfn(s2fd, s2vi));
/*
* If we are not forced to use page mapping, check if we are
@@ -2057,6 +2059,7 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
goto out_unlock;
}
}
+ canonical_ipa = ALIGN_DOWN(canonical_ipa, mapping_size);
}
if (!perm_fault_granule && !s2vi->map_non_cacheable && kvm_has_mte(kvm))
@@ -2090,11 +2093,9 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
* making sure we adjust the canonical IPA if the mapping size has
* been updated (via a THP upgrade, for example).
*/
- if (writable && !ret) {
- phys_addr_t ipa = gfn_to_gpa(get_canonical_gfn(s2fd, s2vi));
- ipa &= ~(mapping_size - 1);
- mark_page_dirty_in_slot(kvm, s2fd->memslot, gpa_to_gfn(ipa));
- }
+ if (writable && !ret)
+ mark_page_dirty_in_slot(kvm, s2fd->memslot,
+ gpa_to_gfn(canonical_ipa));
if (ret != -EAGAIN)
return ret;
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v5 2/6] KVM: arm64: nv: Introduce guest stage-2 tracking structures
2026-08-10 20:50 [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure) Wei-Lin Chang
2026-08-10 20:50 ` [PATCH v5 1/6] KVM: arm64: Use a variable for the canonical IPA in kvm_s2_fault_map() Wei-Lin Chang
@ 2026-08-10 20:50 ` Wei-Lin Chang
2026-08-14 1:04 ` Itaru Kitayama
2026-08-10 20:50 ` [PATCH v5 3/6] KVM: arm64: nv: Track guest stage-2 mapping creation Wei-Lin Chang
` (5 subsequent siblings)
7 siblings, 1 reply; 25+ messages in thread
From: Wei-Lin Chang @ 2026-08-10 20:50 UTC (permalink / raw)
To: linux-arm-kernel, kvmarm, linux-kernel
Cc: Marc Zyngier, Oliver Upton, Fuad Tabba, Joey Gouly, Steffen Eiden,
Suzuki K Poulose, Zenghui Yu, Catalin Marinas, Will Deacon,
Lorenzo Stoakes, Itaru Kitayama, Wei-Lin Chang
In order to avoid unmapping all shadow stage-2 mappings when KVM
receives a MMU notifier unmap call, we have to keep track of the
canonical IPA -> nested IPA relationship of the shadow mappings
created. This essentially means tracking the guest's stage-2.
To do this, represent each mapping by struct kvm_guest_s2_mapping. It
stores the mapping's canonical IPA range and the nested IPA range using
two interval tree nodes. Both nodes will be inserted into their
respective interval trees called guest_s2_mappings. The canonical IPA
ranges will be stored in the tree within the canonical MMU, and the
nested IPA ranges will be stored in the corresponding nested MMU's tree.
For example:
struct kvm_guest_s2_mapping mapping1, mapping2;
---------------------> mapping2.canonical
| mapping1.canonical
| ^ (both stored in canonical mmu's tree)
| |
--*****-----------------------*****----------- CIPA
\\\\\ ||||| mapping1.nested_mmu
\\\\\ \\\\\ |
\\\\\ \\\\\ v
------\\\\\---------------------*****--------- NIPA #1 (nested mmu #1)
\\\\\ |
\\\\\ -> mapping1.nested
\\\\\ (stored in nested mmu #1's tree)
\\\\\
-----------*****------------------------------ NIPA #2 (nested mmu #2)
| ^
-> mapping2.nested |
(stored in nested mmu #2's tree) mapping2.nested_mmu
Using the trees we can look up nodes in either of the IPA spaces, and
for each node, find the corresponding range in the other IPA space from
the other node in the enclosing kvm_guest_s2_mapping.
Define kvm_guest_s2_mapping and the interval tree here. Guest stage-2
mapping tracking will come in subsequent patches.
Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
---
arch/arm64/include/asm/kvm_host.h | 17 +++++++++++++++++
arch/arm64/kvm/mmu.c | 30 ++++++++++++++++++++++++++++++
arch/arm64/kvm/nested.c | 1 +
3 files changed, 48 insertions(+)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index bae2c4f92ef5..0695c4ef93f1 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -14,6 +14,7 @@
#include <linux/arm-smccc.h>
#include <linux/bitmap.h>
#include <linux/types.h>
+#include <linux/interval_tree.h>
#include <linux/jump_label.h>
#include <linux/kvm_types.h>
#include <linux/maple_tree.h>
@@ -150,6 +151,16 @@ struct kvm_vmid {
atomic64_t id;
};
+/*
+ * Record of a guest stage-2 mapping, storing canonical and nested IPA
+ * ranges. Both ranges have the same size.
+ */
+struct kvm_guest_s2_mapping {
+ struct interval_tree_node canonical;
+ struct interval_tree_node nested;
+ struct kvm_s2_mmu *nested_mmu;
+};
+
struct kvm_s2_mmu {
struct kvm_vmid vmid;
@@ -227,6 +238,9 @@ struct kvm_s2_mmu {
*/
bool pending_unmap;
+ /* Guest s2 mapping records indexed in this MMU's IPA space. */
+ struct rb_root_cached guest_s2_mappings;
+
/*
* 0: Nobody is currently using this, check vttbr for validity
* >0: Somebody is actively using this.
@@ -326,6 +340,9 @@ struct kvm_arch {
size_t nested_mmus_size;
int nested_mmus_next;
+ /* Guest s2 tracking trees access serialization. */
+ spinlock_t guest_s2_tracking_lock;
+
/* Interrupt controller */
struct vgic_dist vgic;
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 336dd8f7e8ab..59b4f583240e 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -7,6 +7,7 @@
#include <linux/acpi.h>
#include <linux/mman.h>
#include <linux/kvm_host.h>
+#include <linux/interval_tree.h>
#include <linux/io.h>
#include <linux/hugetlb.h>
#include <linux/sched/signal.h>
@@ -1033,6 +1034,8 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
mmu->pgd_phys = __pa(pgt->pgd);
+ mmu->guest_s2_mappings = RB_ROOT_CACHED;
+
if (kvm_is_nested_s2_mmu(kvm, mmu))
kvm_init_nested_s2_mmu(mmu);
@@ -1122,10 +1125,32 @@ void stage2_unmap_vm(struct kvm *kvm)
srcu_read_unlock(&kvm->srcu, idx);
}
+static void guest_s2_tracking_destroy(struct kvm_s2_mmu *mmu,
+ struct rb_root_cached *tree)
+{
+ struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
+ struct kvm_guest_s2_mapping *mapping;
+ struct interval_tree_node *node;
+
+ while ((node = interval_tree_iter_first(tree, 0, ULONG_MAX))) {
+ interval_tree_remove(node, tree);
+
+ if (!kvm_is_nested_s2_mmu(kvm, mmu)) {
+ mapping = container_of(node, struct kvm_guest_s2_mapping,
+ canonical);
+ /* The canonical MMU is destroyed after the nested MMUs. */
+ kfree(mapping);
+ }
+
+ cond_resched();
+ }
+}
+
void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu)
{
struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
struct kvm_pgtable *pgt = NULL;
+ struct rb_root_cached mappings_tree;
write_lock(&kvm->mmu_lock);
pgt = mmu->pgt;
@@ -1138,12 +1163,17 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu)
if (kvm_is_nested_s2_mmu(kvm, mmu))
kvm_init_nested_s2_mmu(mmu);
+ mappings_tree = mmu->guest_s2_mappings;
+ mmu->guest_s2_mappings = RB_ROOT_CACHED;
+
write_unlock(&kvm->mmu_lock);
if (pgt) {
kvm_stage2_destroy(pgt);
kfree(pgt);
}
+
+ guest_s2_tracking_destroy(mmu, &mappings_tree);
}
static void hyp_mc_free_fn(void *addr, void *mc)
diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index dfb96edbdc43..744aacba61ae 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -49,6 +49,7 @@ void kvm_init_nested(struct kvm *kvm)
kvm->arch.nested_mmus = NULL;
kvm->arch.nested_mmus_size = 0;
atomic_set(&kvm->arch.vncr_map_count, 0);
+ spin_lock_init(&kvm->arch.guest_s2_tracking_lock);
}
static int init_nested_s2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* Re: [PATCH v5 2/6] KVM: arm64: nv: Introduce guest stage-2 tracking structures
2026-08-10 20:50 ` [PATCH v5 2/6] KVM: arm64: nv: Introduce guest stage-2 tracking structures Wei-Lin Chang
@ 2026-08-14 1:04 ` Itaru Kitayama
2026-08-14 10:42 ` Wei-Lin Chang
0 siblings, 1 reply; 25+ messages in thread
From: Itaru Kitayama @ 2026-08-14 1:04 UTC (permalink / raw)
To: Wei-Lin Chang
Cc: linux-arm-kernel, kvmarm, linux-kernel, Marc Zyngier,
Oliver Upton, Fuad Tabba, Joey Gouly, Steffen Eiden,
Suzuki K Poulose, Zenghui Yu, Catalin Marinas, Will Deacon,
Lorenzo Stoakes
On Mon, Aug 10, 2026 at 09:50:34PM +0100, Wei-Lin Chang wrote:
> In order to avoid unmapping all shadow stage-2 mappings when KVM
> receives a MMU notifier unmap call, we have to keep track of the
> canonical IPA -> nested IPA relationship of the shadow mappings
> created. This essentially means tracking the guest's stage-2.
>
> To do this, represent each mapping by struct kvm_guest_s2_mapping. It
> stores the mapping's canonical IPA range and the nested IPA range using
> two interval tree nodes. Both nodes will be inserted into their
> respective interval trees called guest_s2_mappings. The canonical IPA
> ranges will be stored in the tree within the canonical MMU, and the
> nested IPA ranges will be stored in the corresponding nested MMU's tree.
>
> For example:
>
> struct kvm_guest_s2_mapping mapping1, mapping2;
>
> ---------------------> mapping2.canonical
> | mapping1.canonical
> | ^ (both stored in canonical mmu's tree)
> | |
> --*****-----------------------*****----------- CIPA
> \\\\\ ||||| mapping1.nested_mmu
> \\\\\ \\\\\ |
> \\\\\ \\\\\ v
> ------\\\\\---------------------*****--------- NIPA #1 (nested mmu #1)
> \\\\\ |
> \\\\\ -> mapping1.nested
> \\\\\ (stored in nested mmu #1's tree)
> \\\\\
> -----------*****------------------------------ NIPA #2 (nested mmu #2)
> | ^
> -> mapping2.nested |
> (stored in nested mmu #2's tree) mapping2.nested_mmu
>
> Using the trees we can look up nodes in either of the IPA spaces, and
> for each node, find the corresponding range in the other IPA space from
> the other node in the enclosing kvm_guest_s2_mapping.
>
> Define kvm_guest_s2_mapping and the interval tree here. Guest stage-2
> mapping tracking will come in subsequent patches.
>
> Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
> ---
> arch/arm64/include/asm/kvm_host.h | 17 +++++++++++++++++
> arch/arm64/kvm/mmu.c | 30 ++++++++++++++++++++++++++++++
> arch/arm64/kvm/nested.c | 1 +
> 3 files changed, 48 insertions(+)
>
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index bae2c4f92ef5..0695c4ef93f1 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -14,6 +14,7 @@
> #include <linux/arm-smccc.h>
> #include <linux/bitmap.h>
> #include <linux/types.h>
> +#include <linux/interval_tree.h>
> #include <linux/jump_label.h>
> #include <linux/kvm_types.h>
> #include <linux/maple_tree.h>
> @@ -150,6 +151,16 @@ struct kvm_vmid {
> atomic64_t id;
> };
>
> +/*
> + * Record of a guest stage-2 mapping, storing canonical and nested IPA
> + * ranges. Both ranges have the same size.
> + */
> +struct kvm_guest_s2_mapping {
> + struct interval_tree_node canonical;
> + struct interval_tree_node nested;
> + struct kvm_s2_mmu *nested_mmu;
> +};
Is this to be used for normal (L1) guests? I guess this series is for
shadow stage 2 unmapping optimization, so not sure.
Thanks,
Itaru.
> +
> struct kvm_s2_mmu {
> struct kvm_vmid vmid;
>
> @@ -227,6 +238,9 @@ struct kvm_s2_mmu {
> */
> bool pending_unmap;
>
> + /* Guest s2 mapping records indexed in this MMU's IPA space. */
> + struct rb_root_cached guest_s2_mappings;
> +
> /*
> * 0: Nobody is currently using this, check vttbr for validity
> * >0: Somebody is actively using this.
> @@ -326,6 +340,9 @@ struct kvm_arch {
> size_t nested_mmus_size;
> int nested_mmus_next;
>
> + /* Guest s2 tracking trees access serialization. */
> + spinlock_t guest_s2_tracking_lock;
> +
> /* Interrupt controller */
> struct vgic_dist vgic;
>
> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index 336dd8f7e8ab..59b4f583240e 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c
> @@ -7,6 +7,7 @@
> #include <linux/acpi.h>
> #include <linux/mman.h>
> #include <linux/kvm_host.h>
> +#include <linux/interval_tree.h>
> #include <linux/io.h>
> #include <linux/hugetlb.h>
> #include <linux/sched/signal.h>
> @@ -1033,6 +1034,8 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
>
> mmu->pgd_phys = __pa(pgt->pgd);
>
> + mmu->guest_s2_mappings = RB_ROOT_CACHED;
> +
> if (kvm_is_nested_s2_mmu(kvm, mmu))
> kvm_init_nested_s2_mmu(mmu);
>
> @@ -1122,10 +1125,32 @@ void stage2_unmap_vm(struct kvm *kvm)
> srcu_read_unlock(&kvm->srcu, idx);
> }
>
> +static void guest_s2_tracking_destroy(struct kvm_s2_mmu *mmu,
> + struct rb_root_cached *tree)
> +{
> + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
> + struct kvm_guest_s2_mapping *mapping;
> + struct interval_tree_node *node;
> +
> + while ((node = interval_tree_iter_first(tree, 0, ULONG_MAX))) {
> + interval_tree_remove(node, tree);
> +
> + if (!kvm_is_nested_s2_mmu(kvm, mmu)) {
> + mapping = container_of(node, struct kvm_guest_s2_mapping,
> + canonical);
> + /* The canonical MMU is destroyed after the nested MMUs. */
> + kfree(mapping);
> + }
> +
> + cond_resched();
> + }
> +}
> +
> void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu)
> {
> struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
> struct kvm_pgtable *pgt = NULL;
> + struct rb_root_cached mappings_tree;
>
> write_lock(&kvm->mmu_lock);
> pgt = mmu->pgt;
> @@ -1138,12 +1163,17 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu)
> if (kvm_is_nested_s2_mmu(kvm, mmu))
> kvm_init_nested_s2_mmu(mmu);
>
> + mappings_tree = mmu->guest_s2_mappings;
> + mmu->guest_s2_mappings = RB_ROOT_CACHED;
> +
> write_unlock(&kvm->mmu_lock);
>
> if (pgt) {
> kvm_stage2_destroy(pgt);
> kfree(pgt);
> }
> +
> + guest_s2_tracking_destroy(mmu, &mappings_tree);
> }
>
> static void hyp_mc_free_fn(void *addr, void *mc)
> diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
> index dfb96edbdc43..744aacba61ae 100644
> --- a/arch/arm64/kvm/nested.c
> +++ b/arch/arm64/kvm/nested.c
> @@ -49,6 +49,7 @@ void kvm_init_nested(struct kvm *kvm)
> kvm->arch.nested_mmus = NULL;
> kvm->arch.nested_mmus_size = 0;
> atomic_set(&kvm->arch.vncr_map_count, 0);
> + spin_lock_init(&kvm->arch.guest_s2_tracking_lock);
> }
>
> static int init_nested_s2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
> --
> 2.43.0
>
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 2/6] KVM: arm64: nv: Introduce guest stage-2 tracking structures
2026-08-14 1:04 ` Itaru Kitayama
@ 2026-08-14 10:42 ` Wei-Lin Chang
2026-08-16 22:01 ` Itaru Kitayama
0 siblings, 1 reply; 25+ messages in thread
From: Wei-Lin Chang @ 2026-08-14 10:42 UTC (permalink / raw)
To: Itaru Kitayama
Cc: linux-arm-kernel, kvmarm, linux-kernel, Marc Zyngier,
Oliver Upton, Fuad Tabba, Joey Gouly, Steffen Eiden,
Suzuki K Poulose, Zenghui Yu, Catalin Marinas, Will Deacon,
Lorenzo Stoakes
Hi,
On Fri, Aug 14, 2026 at 10:04:57AM +0900, Itaru Kitayama wrote:
> On Mon, Aug 10, 2026 at 09:50:34PM +0100, Wei-Lin Chang wrote:
> > In order to avoid unmapping all shadow stage-2 mappings when KVM
> > receives a MMU notifier unmap call, we have to keep track of the
> > canonical IPA -> nested IPA relationship of the shadow mappings
> > created. This essentially means tracking the guest's stage-2.
> >
> > To do this, represent each mapping by struct kvm_guest_s2_mapping. It
> > stores the mapping's canonical IPA range and the nested IPA range using
> > two interval tree nodes. Both nodes will be inserted into their
> > respective interval trees called guest_s2_mappings. The canonical IPA
> > ranges will be stored in the tree within the canonical MMU, and the
> > nested IPA ranges will be stored in the corresponding nested MMU's tree.
> >
> > For example:
> >
> > struct kvm_guest_s2_mapping mapping1, mapping2;
> >
> > ---------------------> mapping2.canonical
> > | mapping1.canonical
> > | ^ (both stored in canonical mmu's tree)
> > | |
> > --*****-----------------------*****----------- CIPA
> > \\\\\ ||||| mapping1.nested_mmu
> > \\\\\ \\\\\ |
> > \\\\\ \\\\\ v
> > ------\\\\\---------------------*****--------- NIPA #1 (nested mmu #1)
> > \\\\\ |
> > \\\\\ -> mapping1.nested
> > \\\\\ (stored in nested mmu #1's tree)
> > \\\\\
> > -----------*****------------------------------ NIPA #2 (nested mmu #2)
> > | ^
> > -> mapping2.nested |
> > (stored in nested mmu #2's tree) mapping2.nested_mmu
> >
> > Using the trees we can look up nodes in either of the IPA spaces, and
> > for each node, find the corresponding range in the other IPA space from
> > the other node in the enclosing kvm_guest_s2_mapping.
> >
> > Define kvm_guest_s2_mapping and the interval tree here. Guest stage-2
> > mapping tracking will come in subsequent patches.
> >
> > Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
> > ---
[...]
> >
> > +/*
> > + * Record of a guest stage-2 mapping, storing canonical and nested IPA
> > + * ranges. Both ranges have the same size.
> > + */
> > +struct kvm_guest_s2_mapping {
> > + struct interval_tree_node canonical;
> > + struct interval_tree_node nested;
> > + struct kvm_s2_mmu *nested_mmu;
> > +};
>
> Is this to be used for normal (L1) guests? I guess this series is for
> shadow stage 2 unmapping optimization, so not sure.
Sorry I don't totally understand your question. Yes this series is
optimizing cases where a GPA (L1's PA) range have to be unmapped, from
an MMU notifier unmap call.
When we want to unmap a GPA range, the corresponding L2PAs must also be
unmapped from the shadow page tables. Before this series there is no way
of knowing what L2PAs are affected, so the code just calls
kvm_nested_s2_unmap() to unmap all shadow mappings.
Each struct kvm_guest_s2_mapping instance keeps one L1PA <-> L2PA
mapping record. For example:
canonical (L1PA space): [x, x+4K) <- node stored in canonical MMU tree
nested (L2PA space): [y, y+4K) <- node stored in nested MMU tree
Does this make sense?
Thanks,
Wei-Lin Chang
>
> Thanks,
> Itaru.
>
[...]
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 2/6] KVM: arm64: nv: Introduce guest stage-2 tracking structures
2026-08-14 10:42 ` Wei-Lin Chang
@ 2026-08-16 22:01 ` Itaru Kitayama
0 siblings, 0 replies; 25+ messages in thread
From: Itaru Kitayama @ 2026-08-16 22:01 UTC (permalink / raw)
To: Wei-Lin Chang
Cc: linux-arm-kernel, kvmarm, linux-kernel, Marc Zyngier,
Oliver Upton, Fuad Tabba, Joey Gouly, Steffen Eiden,
Suzuki K Poulose, Zenghui Yu, Catalin Marinas, Will Deacon,
Lorenzo Stoakes
On Fri, Aug 14, 2026 at 11:42:24AM +0100, Wei-Lin Chang wrote:
> Hi,
>
> On Fri, Aug 14, 2026 at 10:04:57AM +0900, Itaru Kitayama wrote:
> > On Mon, Aug 10, 2026 at 09:50:34PM +0100, Wei-Lin Chang wrote:
> > > In order to avoid unmapping all shadow stage-2 mappings when KVM
> > > receives a MMU notifier unmap call, we have to keep track of the
> > > canonical IPA -> nested IPA relationship of the shadow mappings
> > > created. This essentially means tracking the guest's stage-2.
> > >
> > > To do this, represent each mapping by struct kvm_guest_s2_mapping. It
> > > stores the mapping's canonical IPA range and the nested IPA range using
> > > two interval tree nodes. Both nodes will be inserted into their
> > > respective interval trees called guest_s2_mappings. The canonical IPA
> > > ranges will be stored in the tree within the canonical MMU, and the
> > > nested IPA ranges will be stored in the corresponding nested MMU's tree.
> > >
> > > For example:
> > >
> > > struct kvm_guest_s2_mapping mapping1, mapping2;
> > >
> > > ---------------------> mapping2.canonical
> > > | mapping1.canonical
> > > | ^ (both stored in canonical mmu's tree)
> > > | |
> > > --*****-----------------------*****----------- CIPA
> > > \\\\\ ||||| mapping1.nested_mmu
> > > \\\\\ \\\\\ |
> > > \\\\\ \\\\\ v
> > > ------\\\\\---------------------*****--------- NIPA #1 (nested mmu #1)
> > > \\\\\ |
> > > \\\\\ -> mapping1.nested
> > > \\\\\ (stored in nested mmu #1's tree)
> > > \\\\\
> > > -----------*****------------------------------ NIPA #2 (nested mmu #2)
> > > | ^
> > > -> mapping2.nested |
> > > (stored in nested mmu #2's tree) mapping2.nested_mmu
> > >
> > > Using the trees we can look up nodes in either of the IPA spaces, and
> > > for each node, find the corresponding range in the other IPA space from
> > > the other node in the enclosing kvm_guest_s2_mapping.
> > >
> > > Define kvm_guest_s2_mapping and the interval tree here. Guest stage-2
> > > mapping tracking will come in subsequent patches.
> > >
> > > Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
> > > ---
>
> [...]
>
> > >
> > > +/*
> > > + * Record of a guest stage-2 mapping, storing canonical and nested IPA
> > > + * ranges. Both ranges have the same size.
> > > + */
> > > +struct kvm_guest_s2_mapping {
> > > + struct interval_tree_node canonical;
> > > + struct interval_tree_node nested;
> > > + struct kvm_s2_mmu *nested_mmu;
> > > +};
> >
> > Is this to be used for normal (L1) guests? I guess this series is for
> > shadow stage 2 unmapping optimization, so not sure.
>
> Sorry I don't totally understand your question. Yes this series is
> optimizing cases where a GPA (L1's PA) range have to be unmapped, from
> an MMU notifier unmap call.
>
> When we want to unmap a GPA range, the corresponding L2PAs must also be
> unmapped from the shadow page tables. Before this series there is no way
> of knowing what L2PAs are affected, so the code just calls
> kvm_nested_s2_unmap() to unmap all shadow mappings.
>
> Each struct kvm_guest_s2_mapping instance keeps one L1PA <-> L2PA
> mapping record. For example:
>
> canonical (L1PA space): [x, x+4K) <- node stored in canonical MMU tree
> nested (L2PA space): [y, y+4K) <- node stored in nested MMU tree
>
> Does this make sense?
Yes, makes sense. I was mostly wondering about the sturct name,
kvm_guest_s2_mapping you introduced, since this is only needed for a guest
acting as a hypervisor. No strong opinion.
Thanks,
Itaru.
>
> Thanks,
> Wei-Lin Chang
>
> >
> > Thanks,
> > Itaru.
> >
>
> [...]
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v5 3/6] KVM: arm64: nv: Track guest stage-2 mapping creation
2026-08-10 20:50 [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure) Wei-Lin Chang
2026-08-10 20:50 ` [PATCH v5 1/6] KVM: arm64: Use a variable for the canonical IPA in kvm_s2_fault_map() Wei-Lin Chang
2026-08-10 20:50 ` [PATCH v5 2/6] KVM: arm64: nv: Introduce guest stage-2 tracking structures Wei-Lin Chang
@ 2026-08-10 20:50 ` Wei-Lin Chang
2026-08-10 20:50 ` [PATCH v5 4/6] KVM: arm64: nv: Track guest stage-2 mapping removal Wei-Lin Chang
` (4 subsequent siblings)
7 siblings, 0 replies; 25+ messages in thread
From: Wei-Lin Chang @ 2026-08-10 20:50 UTC (permalink / raw)
To: linux-arm-kernel, kvmarm, linux-kernel
Cc: Marc Zyngier, Oliver Upton, Fuad Tabba, Joey Gouly, Steffen Eiden,
Suzuki K Poulose, Zenghui Yu, Catalin Marinas, Will Deacon,
Lorenzo Stoakes, Itaru Kitayama, Wei-Lin Chang
During shadow stage-2 faults, in addition to creating mappings in the
shadow page tables, also allocate kvm_guest_s2_mapping objects, record
the mapping ranges, and insert them into the canonical and nested mmu's
guest_s2_mappings tree.
Note that because we allow parallel faulting, the interval trees could
store mappings that are not live in the shadow page tables. Storing a
superset of the live mappings is fine because we will only over-unmap
when we use this information later to do the targeted MMU notifier
unmap.
Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
---
arch/arm64/include/asm/kvm_nested.h | 3 +++
arch/arm64/kvm/mmu.c | 29 +++++++++++++++++++++++++++++
arch/arm64/kvm/nested.c | 25 +++++++++++++++++++++++++
3 files changed, 57 insertions(+)
diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h
index 012d711034d1..560b78b3f5ff 100644
--- a/arch/arm64/include/asm/kvm_nested.h
+++ b/arch/arm64/include/asm/kvm_nested.h
@@ -77,6 +77,9 @@ extern void kvm_s2_mmu_iterate_by_vmid(struct kvm *kvm, u16 vmid,
const union tlbi_info *info,
void (*)(struct kvm_s2_mmu *,
const union tlbi_info *));
+extern void kvm_record_guest_s2_mapping(struct kvm_s2_mmu *mmu, gpa_t canonical_ipa,
+ gpa_t nested_ipa, size_t map_size,
+ struct kvm_guest_s2_mapping *mapping);
extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu);
extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu);
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 59b4f583240e..cea968921041 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1642,6 +1642,7 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
enum kvm_pgtable_walk_flags flags = KVM_PGTABLE_WALK_SHARED;
enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
struct kvm_pgtable *pgt = s2fd->vcpu->arch.hw_mmu->pgt;
+ struct kvm_guest_s2_mapping *mapping = NULL;
unsigned long mmu_seq;
struct page *page;
struct kvm *kvm = s2fd->vcpu->kvm;
@@ -1655,6 +1656,11 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
ret = topup_mmu_memcache(s2fd->vcpu, memcache);
if (ret)
return ret;
+ if (kvm_is_nested_s2_mmu(kvm, pgt->mmu)) {
+ mapping = kmalloc_obj(struct kvm_guest_s2_mapping, GFP_KERNEL_ACCOUNT);
+ if (!mapping)
+ return -ENOMEM;
+ }
}
if (s2fd->nested)
@@ -1675,6 +1681,7 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
if (ret) {
kvm_prepare_memory_fault_exit(s2fd->vcpu, s2fd->fault_ipa, PAGE_SIZE,
write_fault, exec_fault, false);
+ kfree(mapping);
return ret;
}
@@ -1708,11 +1715,17 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
ret = KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, s2fd->fault_ipa, PAGE_SIZE,
__pfn_to_phys(pfn), prot,
memcache, flags);
+ if (!ret && kvm_is_nested_s2_mmu(kvm, pgt->mmu)) {
+ kvm_record_guest_s2_mapping(pgt->mmu, gfn << PAGE_SHIFT,
+ s2fd->fault_ipa, PAGE_SIZE, mapping);
+ mapping = NULL;
+ }
}
out_unlock:
kvm_release_faultin_page(kvm, page, !!ret, prot & KVM_PGTABLE_PROT_W);
kvm_fault_unlock(kvm);
+ kfree(mapping);
if ((prot & KVM_PGTABLE_PROT_W) && !ret)
mark_page_dirty_in_slot(kvm, s2fd->memslot, gfn);
@@ -2049,6 +2062,7 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
void *memcache)
{
enum kvm_pgtable_walk_flags flags = KVM_PGTABLE_WALK_SHARED;
+ struct kvm_guest_s2_mapping *mapping = NULL;
bool writable = prot & KVM_PGTABLE_PROT_W;
struct kvm *kvm = s2fd->vcpu->kvm;
phys_addr_t canonical_ipa;
@@ -2059,6 +2073,15 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
gfn_t gfn;
int ret;
+ if (kvm_is_nested_s2_mmu(kvm, s2fd->vcpu->arch.hw_mmu)) {
+ mapping = kmalloc_obj(struct kvm_guest_s2_mapping,
+ GFP_KERNEL_ACCOUNT);
+ if (!mapping) {
+ kvm_release_page_unused(s2vi->page);
+ return -ENOMEM;
+ }
+ }
+
kvm_fault_lock(kvm);
pgt = s2fd->vcpu->arch.hw_mmu->pgt;
ret = -EAGAIN;
@@ -2112,11 +2135,17 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
ret = KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, gfn_to_gpa(gfn), mapping_size,
__pfn_to_phys(pfn), prot,
memcache, flags);
+ if (!ret && kvm_is_nested_s2_mmu(kvm, pgt->mmu)) {
+ kvm_record_guest_s2_mapping(pgt->mmu, canonical_ipa,
+ gfn_to_gpa(gfn), mapping_size, mapping);
+ mapping = NULL;
+ }
}
out_unlock:
kvm_release_faultin_page(kvm, s2vi->page, !!ret, writable);
kvm_fault_unlock(kvm);
+ kfree(mapping);
/*
* Mark the page dirty only if the fault is handled successfully,
diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index 744aacba61ae..646b628bba17 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -5,6 +5,7 @@
*/
#include <linux/bitfield.h>
+#include <linux/interval_tree.h>
#include <linux/kvm.h>
#include <linux/kvm_host.h>
@@ -852,6 +853,30 @@ static struct kvm_s2_mmu *get_s2_mmu_nested(struct kvm_vcpu *vcpu)
return s2_mmu;
}
+void kvm_record_guest_s2_mapping(struct kvm_s2_mmu *mmu, gpa_t canonical_ipa,
+ gpa_t nested_ipa, size_t map_size,
+ struct kvm_guest_s2_mapping *mapping)
+{
+ struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
+
+ lockdep_assert_held_read(&kvm->mmu_lock);
+
+ if (WARN_ON(!IS_ALIGNED(canonical_ipa, map_size)))
+ canonical_ipa = ALIGN_DOWN(canonical_ipa, map_size);
+
+ mapping->canonical.start = canonical_ipa;
+ mapping->canonical.last = canonical_ipa + map_size - 1;
+
+ mapping->nested.start = nested_ipa;
+ mapping->nested.last = nested_ipa + map_size - 1;
+
+ mapping->nested_mmu = mmu;
+
+ guard(spinlock)(&kvm->arch.guest_s2_tracking_lock);
+ interval_tree_insert(&mapping->nested, &mmu->guest_s2_mappings);
+ interval_tree_insert(&mapping->canonical, &kvm->arch.mmu.guest_s2_mappings);
+}
+
void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu)
{
/* CnP being set denotes an invalid entry */
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v5 4/6] KVM: arm64: nv: Track guest stage-2 mapping removal
2026-08-10 20:50 [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure) Wei-Lin Chang
` (2 preceding siblings ...)
2026-08-10 20:50 ` [PATCH v5 3/6] KVM: arm64: nv: Track guest stage-2 mapping creation Wei-Lin Chang
@ 2026-08-10 20:50 ` Wei-Lin Chang
2026-08-10 20:50 ` [PATCH v5 5/6] KVM: arm64: nv: Avoid full shadow stage-2 unmap Wei-Lin Chang
` (3 subsequent siblings)
7 siblings, 0 replies; 25+ messages in thread
From: Wei-Lin Chang @ 2026-08-10 20:50 UTC (permalink / raw)
To: linux-arm-kernel, kvmarm, linux-kernel
Cc: Marc Zyngier, Oliver Upton, Fuad Tabba, Joey Gouly, Steffen Eiden,
Suzuki K Poulose, Zenghui Yu, Catalin Marinas, Will Deacon,
Lorenzo Stoakes, Itaru Kitayama, Wei-Lin Chang
kvm_stage2_unmap_range() is the helper to remove mappings from the
stage-2 page tables. It is called during guest TLBI handling, memslot
removal, nested mmu reuse, etc.
Teach it about the guest stage-2 tracking trees and remove mappings from
there when shadow mappings are removed. This keeps the tracking trees
from having stale mappings pile up.
Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
---
arch/arm64/include/asm/kvm_host.h | 5 ++++-
arch/arm64/include/asm/kvm_nested.h | 2 ++
arch/arm64/kvm/mmu.c | 23 +++++++++++++++++++++--
arch/arm64/kvm/nested.c | 28 ++++++++++++++++++++++++++++
4 files changed, 55 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 0695c4ef93f1..0bb83be1dd4f 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -340,7 +340,10 @@ struct kvm_arch {
size_t nested_mmus_size;
int nested_mmus_next;
- /* Guest s2 tracking trees access serialization. */
+ /*
+ * Serializes guest s2 tracking trees access when the mmu_lock
+ * is only held for read.
+ */
spinlock_t guest_s2_tracking_lock;
/* Interrupt controller */
diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h
index 560b78b3f5ff..ffa3fa01f3cd 100644
--- a/arch/arm64/include/asm/kvm_nested.h
+++ b/arch/arm64/include/asm/kvm_nested.h
@@ -80,6 +80,8 @@ extern void kvm_s2_mmu_iterate_by_vmid(struct kvm *kvm, u16 vmid,
extern void kvm_record_guest_s2_mapping(struct kvm_s2_mmu *mmu, gpa_t canonical_ipa,
gpa_t nested_ipa, size_t map_size,
struct kvm_guest_s2_mapping *mapping);
+extern void kvm_remove_guest_s2_mappings(struct kvm_s2_mmu *mmu,
+ gpa_t nipa, size_t size);
extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu);
extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu);
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index cea968921041..ddd1bbede227 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -314,6 +314,19 @@ static void invalidate_icache_guest_page(void *va, size_t size)
* we then fully enforce cacheability of RAM, no matter what the guest
* does.
*/
+
+static int kvm_pgtable_stage2_unmap_tracked(struct kvm_pgtable *pgt, u64 addr, u64 size)
+{
+ int ret;
+
+ ret = kvm_pgtable_stage2_unmap(pgt, addr, size);
+ if (ret)
+ return ret;
+
+ kvm_remove_guest_s2_mappings(pgt->mmu, addr, size);
+ return 0;
+}
+
/**
* __unmap_stage2_range -- Clear stage2 page table entries to unmap a range
* @mmu: The KVM stage-2 MMU pointer
@@ -331,11 +344,17 @@ static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64
{
struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
phys_addr_t end = start + size;
+ int (*fn)(struct kvm_pgtable *, u64, u64);
lockdep_assert_held_write(&kvm->mmu_lock);
WARN_ON(size & ~PAGE_MASK);
- WARN_ON(stage2_apply_range(mmu, start, end, KVM_PGT_FN(kvm_pgtable_stage2_unmap),
- may_block));
+
+ if (kvm_is_nested_s2_mmu(kvm, mmu))
+ fn = kvm_pgtable_stage2_unmap_tracked;
+ else
+ fn = KVM_PGT_FN(kvm_pgtable_stage2_unmap);
+
+ WARN_ON(stage2_apply_range(mmu, start, end, fn, may_block));
}
void kvm_stage2_unmap_range(struct kvm_s2_mmu *mmu, phys_addr_t start,
diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index 646b628bba17..2a4c86df404c 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -877,6 +877,34 @@ void kvm_record_guest_s2_mapping(struct kvm_s2_mmu *mmu, gpa_t canonical_ipa,
interval_tree_insert(&mapping->canonical, &kvm->arch.mmu.guest_s2_mappings);
}
+void kvm_remove_guest_s2_mappings(struct kvm_s2_mmu *mmu, gpa_t nipa,
+ size_t size)
+{
+ struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
+ struct interval_tree_node *node, *next;
+ struct kvm_guest_s2_mapping *mapping;
+ gpa_t nipa_end = nipa + size - 1;
+
+ /*
+ * Guest s2 tracking interval trees are only accessed while holding the
+ * mmu_lock, hence we don't have to take guest_s2_tracking_lock if the
+ * mmu_lock is held for write.
+ */
+ lockdep_assert_held_write(&kvm_s2_mmu_to_kvm(mmu)->mmu_lock);
+
+ node = interval_tree_iter_first(&mmu->guest_s2_mappings, nipa, nipa_end);
+ while (node) {
+ next = interval_tree_iter_next(node, nipa, nipa_end);
+ mapping = container_of(node, struct kvm_guest_s2_mapping,
+ nested);
+ interval_tree_remove(&mapping->nested, &mmu->guest_s2_mappings);
+ interval_tree_remove(&mapping->canonical,
+ &kvm->arch.mmu.guest_s2_mappings);
+ kfree(mapping);
+ node = next;
+ }
+}
+
void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu)
{
/* CnP being set denotes an invalid entry */
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v5 5/6] KVM: arm64: nv: Avoid full shadow stage-2 unmap
2026-08-10 20:50 [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure) Wei-Lin Chang
` (3 preceding siblings ...)
2026-08-10 20:50 ` [PATCH v5 4/6] KVM: arm64: nv: Track guest stage-2 mapping removal Wei-Lin Chang
@ 2026-08-10 20:50 ` Wei-Lin Chang
2026-08-10 20:50 ` [PATCH v5 6/6] KVM: arm64: Refactor kvm_unmap_gfn_range() with common variables Wei-Lin Chang
` (2 subsequent siblings)
7 siblings, 0 replies; 25+ messages in thread
From: Wei-Lin Chang @ 2026-08-10 20:50 UTC (permalink / raw)
To: linux-arm-kernel, kvmarm, linux-kernel
Cc: Marc Zyngier, Oliver Upton, Fuad Tabba, Joey Gouly, Steffen Eiden,
Suzuki K Poulose, Zenghui Yu, Catalin Marinas, Will Deacon,
Lorenzo Stoakes, Itaru Kitayama, Wei-Lin Chang
With guest stage-2 tracking in place, we can improve MMU notifier unmaps
from unmapping all existing shadow stage-2 mappings to only unmapping
the ones affected by the given canonical IPA range.
Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
---
arch/arm64/include/asm/kvm_nested.h | 2 ++
arch/arm64/kvm/mmu.c | 7 +++--
arch/arm64/kvm/nested.c | 47 +++++++++++++++++++++++++++--
3 files changed, 50 insertions(+), 6 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h
index ffa3fa01f3cd..4e7d89b6824b 100644
--- a/arch/arm64/include/asm/kvm_nested.h
+++ b/arch/arm64/include/asm/kvm_nested.h
@@ -170,6 +170,8 @@ extern int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu,
struct kvm_s2_trans *trans);
extern int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2);
extern void kvm_nested_s2_wp(struct kvm *kvm);
+extern void kvm_nested_unmap_cipa_range(struct kvm *kvm, gpa_t cipa,
+ size_t unmap_size, bool may_block);
extern void kvm_nested_s2_unmap(struct kvm *kvm, bool may_block);
extern void kvm_nested_s2_flush(struct kvm *kvm);
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index ddd1bbede227..241f020910d8 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -2518,8 +2518,9 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
__unmap_stage2_range(&kvm->arch.mmu, range->start << PAGE_SHIFT,
(range->end - range->start) << PAGE_SHIFT,
range->may_block);
-
- kvm_nested_s2_unmap(kvm, range->may_block);
+ kvm_nested_unmap_cipa_range(kvm, range->start << PAGE_SHIFT,
+ (range->end - range->start) << PAGE_SHIFT,
+ range->may_block);
return false;
}
@@ -2797,7 +2798,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
write_lock(&kvm->mmu_lock);
kvm_stage2_unmap_range(&kvm->arch.mmu, gpa, size, true);
- kvm_nested_s2_unmap(kvm, true);
+ kvm_nested_unmap_cipa_range(kvm, gpa, size, true);
write_unlock(&kvm->mmu_lock);
}
diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index 2a4c86df404c..0dc5824c8237 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -886,9 +886,8 @@ void kvm_remove_guest_s2_mappings(struct kvm_s2_mmu *mmu, gpa_t nipa,
gpa_t nipa_end = nipa + size - 1;
/*
- * Guest s2 tracking interval trees are only accessed while holding the
- * mmu_lock, hence we don't have to take guest_s2_tracking_lock if the
- * mmu_lock is held for write.
+ * See kvm_nested_unmap_cipa_range() for why guest_s2_tracking_lock
+ * isn't taken here.
*/
lockdep_assert_held_write(&kvm_s2_mmu_to_kvm(mmu)->mmu_lock);
@@ -1286,6 +1285,48 @@ void kvm_nested_s2_wp(struct kvm *kvm)
kvm_invalidate_vncr_ipa(kvm, 0, BIT(kvm->arch.mmu.pgt->ia_bits));
}
+void kvm_nested_unmap_cipa_range(struct kvm *kvm, gpa_t cipa, size_t unmap_size,
+ bool may_block)
+{
+ gpa_t cipa_end = cipa + unmap_size - 1;
+ struct kvm_guest_s2_mapping *mapping;
+ struct interval_tree_node *node;
+ size_t mapping_size;
+
+ /*
+ * Guest s2 tracking interval trees are only accessed while holding the
+ * mmu_lock, hence we don't have to take guest_s2_tracking_lock if the
+ * mmu_lock is held for write. This saves us from having to manually
+ * lock/unlock guest_s2_tracking_lock below around
+ * cond_resched_rwlock_write().
+ */
+ lockdep_assert_held_write(&kvm->mmu_lock);
+
+ if (!kvm->arch.nested_mmus_size)
+ return;
+
+ while ((node = interval_tree_iter_first(&kvm->arch.mmu.guest_s2_mappings,
+ cipa, cipa_end))) {
+ mapping = container_of(node, struct kvm_guest_s2_mapping,
+ canonical);
+ mapping_size = mapping->nested.last - mapping->nested.start + 1;
+
+ if (WARN_ON_ONCE(kvm_pgtable_stage2_unmap(mapping->nested_mmu->pgt,
+ mapping->nested.start,
+ mapping_size)))
+ return;
+
+ interval_tree_remove(node, &kvm->arch.mmu.guest_s2_mappings);
+ interval_tree_remove(&mapping->nested, &mapping->nested_mmu->guest_s2_mappings);
+ kfree(mapping);
+
+ if (may_block)
+ cond_resched_rwlock_write(&kvm->mmu_lock);
+ }
+
+ kvm_invalidate_vncr_ipa(kvm, cipa, cipa + unmap_size);
+}
+
void kvm_nested_s2_unmap(struct kvm *kvm, bool may_block)
{
int i;
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v5 6/6] KVM: arm64: Refactor kvm_unmap_gfn_range() with common variables
2026-08-10 20:50 [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure) Wei-Lin Chang
` (4 preceding siblings ...)
2026-08-10 20:50 ` [PATCH v5 5/6] KVM: arm64: nv: Avoid full shadow stage-2 unmap Wei-Lin Chang
@ 2026-08-10 20:50 ` Wei-Lin Chang
2026-08-12 2:12 ` [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure) Itaru Kitayama
2026-09-02 16:35 ` Wang Han
7 siblings, 0 replies; 25+ messages in thread
From: Wei-Lin Chang @ 2026-08-10 20:50 UTC (permalink / raw)
To: linux-arm-kernel, kvmarm, linux-kernel
Cc: Marc Zyngier, Oliver Upton, Fuad Tabba, Joey Gouly, Steffen Eiden,
Suzuki K Poulose, Zenghui Yu, Catalin Marinas, Will Deacon,
Lorenzo Stoakes, Itaru Kitayama, Wei-Lin Chang
__unmap_stage2_range() and kvm_nested_unmap_cipa_range() are using the
same arguments. Clean this up by using local variables.
Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
---
arch/arm64/kvm/mmu.c | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 241f020910d8..9d3ea44a894b 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -2512,15 +2512,16 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
{
+ gpa_t gpa = range->start << PAGE_SHIFT;
+ size_t size = (range->end - range->start) << PAGE_SHIFT;
+ bool may_block = range->may_block;
+
if (!kvm->arch.mmu.pgt || kvm_vm_is_protected(kvm))
return false;
- __unmap_stage2_range(&kvm->arch.mmu, range->start << PAGE_SHIFT,
- (range->end - range->start) << PAGE_SHIFT,
- range->may_block);
- kvm_nested_unmap_cipa_range(kvm, range->start << PAGE_SHIFT,
- (range->end - range->start) << PAGE_SHIFT,
- range->may_block);
+ __unmap_stage2_range(&kvm->arch.mmu, gpa, size, may_block);
+ kvm_nested_unmap_cipa_range(kvm, gpa, size, may_block);
+
return false;
}
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-08-10 20:50 [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure) Wei-Lin Chang
` (5 preceding siblings ...)
2026-08-10 20:50 ` [PATCH v5 6/6] KVM: arm64: Refactor kvm_unmap_gfn_range() with common variables Wei-Lin Chang
@ 2026-08-12 2:12 ` Itaru Kitayama
2026-09-02 16:35 ` Wang Han
7 siblings, 0 replies; 25+ messages in thread
From: Itaru Kitayama @ 2026-08-12 2:12 UTC (permalink / raw)
To: Wei-Lin Chang
Cc: linux-arm-kernel, kvmarm, linux-kernel, Marc Zyngier,
Oliver Upton, Fuad Tabba, Joey Gouly, Steffen Eiden,
Suzuki K Poulose, Zenghui Yu, Catalin Marinas, Will Deacon,
Lorenzo Stoakes
On Mon, Aug 10, 2026 at 09:50:32PM +0100, Wei-Lin Chang wrote:
> Hi,
>
> This is v5 of optimizing the shadow s2 mmu unmapping during MMU
> notifiers.
I've tested your series v5 on a Grace system. L2 booted into prompt
with the Ubuntu filesystem image, your two kvm selftest for nested
virtualization ran fine in L1, and also did stress-ng in L1:
projects $ sudo stress-ng --kvm 16 --cpu 16 --vm 8 --vma 8 --fork 8 --timeout 10m --verify --metrics-brief
stress-ng: info: [1053] setting to a 10 mins run per stressor
stress-ng: info: [1053] dispatching hogs: 16 kvm, 16 cpu, 8 vm, 8 vma, 8 fork
stress-ng: info: [1087] vm: using 32MB per stressor instance (total 256MB of 2.75GB available memory)
stress-ng: metrc: [1053] stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s
stress-ng: metrc: [1053] (secs) (secs) (secs) (real time) (usr+sys time)
stress-ng: metrc: [1053] kvm 81 602.31 250.98 357.55 0.13 0.13
stress-ng: metrc: [1053] cpu 63310 595.70 166.40 0.60 106.28 379.12
stress-ng: metrc: [1053] vm 4252063 600.90 38.98 56.59 7076.16 44491.45
stress-ng: metrc: [1053] vma 56633 601.76 12.64 157.96 94.11 331.97
stress-ng: metrc: [1053] fork 149 600.98 0.03 0.45 0.25 314.59
stress-ng: info: [1053] skipped: 0
stress-ng: info: [1053] passed: 56: kvm (16) cpu (16) vm (8) vma (8) fork (8)
stress-ng: info: [1053] failed: 0
stress-ng: info: [1053] metrics untrustworthy: 0
stress-ng: info: [1053] successful run completed in 10 mins 8.62 secs
Tested-by: Itaru Kitayama <itaru.kitayama@fujitsu.com>
Thanks,
Itaru.
>
> This time, a major overhaul is done to the implementation. After
> receiving some suggestions from Marc, I have identified that using the
> interval tree to store the guest stage-2 mappings solves many problems
> compared to using the maple tree.
>
> Interval Tree vs Maple Tree
> ===========================
>
> First of all, interval trees are capable of storing overlapping ranges,
> which is helpful when the L1 hypervisor maps something like:
>
> nested IPA [x, x+4K) -> canonical IPA [a, a+4K)
> nested IPA [y, y+2M) -> canonical IPA [a, a+2M)
>
> No problems with storing that in the interval tree with different nodes.
> We can avoid the maple tree UNKNOWN_IPA mechanism as a compromise.
>
> Second, ideally we would want to save the canonical IPA <-> nested IPA
> mapping in both directions to allow MMU notifier unmap speed up, and
> stale shadow mapping removals. If we use the maple tree, we'll have to
> have 2 separate trees, and make sure they store the same mappings, which
> isn't simple given the first point.
>
> On the other hand, by using this pattern:
>
> /* Record of a guest stage-2 mapping. */
> struct kvm_guest_s2_mapping {
> struct interval_tree_node canonical; // CIPA range of the mapping
> struct interval_tree_node nested; // NIPA range of the mapping
> struct kvm_s2_mmu *nested_mmu; // mmu of the NIPA space
> };
>
> and equip each mmu with an interval tree storing mapping records
> corresponding to the IPA space it represents, we can insert the
> respective nodes into the canonical IPA tree, and the corresponding
> nested IPA tree. This makes it trivial to find the range of the other
> IPA space from a range in one IPA space.
>
> Diagram to help understanding:
>
> struct kvm_guest_s2_mapping mapping1, mapping2;
>
> ---------------------> mapping2.canonical
> | mapping1.canonical
> | ^ (both stored in canonical mmu's tree)
> | |
> --*****-----------------------*****----------- CIPA
> \\\\\ ||||| mapping1.nested_mmu
> \\\\\ \\\\\ |
> \\\\\ \\\\\ v
> ------\\\\\---------------------*****--------- NIPA #1 (nested mmu #1)
> \\\\\ |
> \\\\\ -> mapping1.nested
> \\\\\ (stored in nested mmu #1's tree)
> \\\\\
> -----------*****------------------------------ NIPA #2 (nested mmu #2)
> | ^
> -> mapping2.nested |
> (stored in nested mmu #2's tree) mapping2.nested_mmu
>
> Third, maple tree does its own memory allocation. In the KVM stage-2
> fault path we only find out what the mapping ranges are after taking the
> KVM MMU lock, and the maple tree has to know the range and entry to be
> stored to preallocate, therefore in our case the maple tree is forced to
> only use GFP_NOWAIT, which isn't the best. With the interval tree the
> user does the memory management, and we can just allocate before taking
> the locks.
>
> Locking
> =======
>
> The guest_s2_tracking_lock serializes accesses to the tracking interval
> trees. It is taken after the mmu_lock. However in reality it is only
> taken after we take the read mmu_lock in the stage-2 fault path, as
> other accesses have the write mmu_lock already. This saves us some
> manual lock/unlocks.
>
> vCPU Stage-2 Fault Scalability Reduction
> ========================================
>
> KVM/arm64 is able to handle stage-2 faults from multiple vCPUs in
> parallel, thanks to the engineering done to the s2 pgtable code. However
> to safely insert mappings into the interval trees we have to serialize
> using the guest_s2_tracking_lock. We trade some performance in stage-2
> fault for faster MMU notifier unmaps, and keeping the unaffected shadow
> mappings.
>
> Memory Usage
> ============
>
> Each interval tree node is 48 bytes, and a kvm_guest_s2_mapping is 104
> bytes, residing in 128-byte slab objects. Each shadow stage-2 fault
> requires one kvm_guest_s2_mapping instance. This is 32MB for a fully 4KB
> mapped 1GB region, and 64KB for a 2MB mapped 1GB region.
>
> Series Structure
> ================
>
> Patch 1: Preparatory refactoring.
> Patch 2: Introduce data structures for guest stage-2 tracking.
> Patch 3-4: Guest stage-2 tracking addition and removal
> Patch 5: Avoid full unmap during MMU notifier unmap using the tracked
> guest stage-2 mapping information.
> Patch 6: Minor clean up.
>
> As this is a complete rework, I will omit the change log this time.
> Series is based on v7.2-rc5.
>
> Thanks!
>
> Link to v4: https://lore.kernel.org/kvmarm/20260714115926.2044757-1-weilin.chang@arm.com/
>
> Wei-Lin Chang (6):
> KVM: arm64: Use a variable for the canonical IPA in kvm_s2_fault_map()
> KVM: arm64: nv: Introduce guest stage-2 tracking structures
> KVM: arm64: nv: Track guest stage-2 mapping creation
> KVM: arm64: nv: Track guest stage-2 mapping removal
> KVM: arm64: nv: Avoid full shadow stage-2 unmap
> KVM: arm64: Refactor kvm_unmap_gfn_range() with common variables
>
> arch/arm64/include/asm/kvm_host.h | 20 ++++++
> arch/arm64/include/asm/kvm_nested.h | 7 ++
> arch/arm64/kvm/mmu.c | 105 ++++++++++++++++++++++++----
> arch/arm64/kvm/nested.c | 95 +++++++++++++++++++++++++
> 4 files changed, 215 insertions(+), 12 deletions(-)
>
> --
> 2.43.0
>
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-08-10 20:50 [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure) Wei-Lin Chang
` (6 preceding siblings ...)
2026-08-12 2:12 ` [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure) Itaru Kitayama
@ 2026-09-02 16:35 ` Wang Han
2026-09-03 7:43 ` Marc Zyngier
7 siblings, 1 reply; 25+ messages in thread
From: Wang Han @ 2026-09-02 16:35 UTC (permalink / raw)
To: weilin.chang
Cc: linux-arm-kernel, kvmarm, linux-kernel, maz, oupton, tabba,
joey.gouly, seiden, suzuki.poulose, catalin.marinas, will, ljs,
itaru.kitayama
Hi Wei-Lin,
I tested this series on a Yitian 710 system with an ARM Neoverse-N2 CPU
(128 CPUs, 2 NUMA nodes).
Test environment
----------------
L0 kernel: Linux v7.2-rc6
L1 guest: Ubuntu 26.04 LTS, kernel 7.0.0-27-generic (aarch64)
QEMU: 10.2.3
L0 NUMA balancing was enabled (`/proc/sys/kernel/numa_balancing=1`).
The host was booted with `kvm_arm.mode=nested`.
This series fixes a functional hang that is exposed when NUMA balancing is
enabled. The previous nested stage-2 unmap path is too slow for this
workload, making the performance problem user-visible: NUMA balancing can
leave the L1 guest unable to make progress and eventually hang during boot.
The L1 was started with 8 vCPUs and 32 GiB of RAM using:
qemu-system-aarch64 -smp 8 -m 32G \
-machine virt,accel=kvm,gic-version=3,virtualization=on \
-cpu host -nographic -enable-kvm \
-drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \
-drive if=pflash,format=raw,file=pflash1_bak.img \
-drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \
-nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \
-serial mon:stdio
With upstream v7.2-rc6 (075b74841bd0065a3bda3440873c747938e69b68),
L0 NUMA balancing enabled, and the same QEMU configuration, the L1 guest
hung during boot. The original L1 console reported:
[ 76.595764] watchdog: BUG: soft lockup - CPU#1 stuck for 45s! [k8s-dqlite:3068]
[ 76.595973] watchdog: BUG: soft lockup - CPU#0 stuck for 38s! [rs:main Q:Reg:1602]
[ 76.596181] watchdog: BUG: soft lockup - CPU#7 stuck for 45s! [kubelite:2672]
[ 76.596334] watchdog: BUG: soft lockup - CPU#5 stuck for 38s! [containerd:2375]
The corresponding L0 hung-task report was:
[Wed Sep 2 22:49:08 2026] INFO: task qemu-system-aar:14681 blocked in I/O wait for more than 120 seconds.
[Wed Sep 2 22:49:08 2026] Tainted: G E N 7.2.0-rc6-poluted_opt-7-2-numa #10.al8
[Wed Sep 2 22:49:08 2026] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 2 22:49:08 2026] task:qemu-system-aar state:D stack:0 pid:14681 tgid:14681 ppid:14680 task_flags:0x8400080 flags:0x00800000
[Wed Sep 2 22:49:08 2026] Call trace:
[Wed Sep 2 22:49:08 2026] __switch_to+0x128/0x168 (T)
[Wed Sep 2 22:49:08 2026] __schedule+0x278/0x910
[Wed Sep 2 22:49:08 2026] schedule+0x3c/0xe8
[Wed Sep 2 22:49:08 2026] io_schedule+0x44/0x68
[Wed Sep 2 22:49:08 2026] softleaf_entry_wait_on_locked+0x280/0x2d0
[Wed Sep 2 22:49:08 2026] migration_entry_wait+0xdc/0x140
[Wed Sep 2 22:49:08 2026] do_swap_page+0x834/0xd80
[Wed Sep 2 22:49:08 2026] handle_pte_fault+0x208/0x2b8
[Wed Sep 2 22:49:08 2026] __handle_mm_fault+0x228/0x528
[Wed Sep 2 22:49:08 2026] handle_mm_fault+0xdc/0x2d8
[Wed Sep 2 22:49:08 2026] do_page_fault+0x388/0x790
[Wed Sep 2 22:49:08 2026] do_translation_fault+0x4c/0x88
[Wed Sep 2 22:49:08 2026] do_mem_abort+0x4c/0xa0
[Wed Sep 2 22:49:08 2026] el0_da+0x54/0x178
[Wed Sep 2 22:49:08 2026] el0t_64_sync_handler+0xd0/0xe8
[Wed Sep 2 22:49:08 2026] el0t_64_sync+0x1ac/0x1b0
The same wait and stack were observed repeatedly; the hung-task report
recurred after 241 and 362 seconds.
I applied the v5 series to the same v7.2-rc6 baseline. With L0 NUMA
balancing still enabled and the same QEMU command line, the L1 guest
booted normally and could be operated through the serial console. QEMU
no longer hung. No new soft-lockup, hung-task, blocked-I/O,
migration-entry-wait, or softleaf-entry-wait message was observed in the
patched boot log.
As a control, the unmodified v7.2-rc6 kernel with
/proc/sys/kernel/numa_balancing=0 also booted the L1 with the same QEMU
configuration. This confirms that the patch removes the NUMA-balancing
failure mode in this setup rather than merely changing the guest setup.
Tested-by: Wang Han <wanghan@linux.alibaba.com>
Thanks,
Wang Han
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-09-02 16:35 ` Wang Han
@ 2026-09-03 7:43 ` Marc Zyngier
2026-09-03 13:28 ` Wei-Lin Chang
0 siblings, 1 reply; 25+ messages in thread
From: Marc Zyngier @ 2026-09-03 7:43 UTC (permalink / raw)
To: Wang Han
Cc: weilin.chang, linux-arm-kernel, kvmarm, linux-kernel, oupton,
tabba, joey.gouly, seiden, suzuki.poulose, catalin.marinas, will,
ljs, itaru.kitayama
On Wed, 02 Sep 2026 17:35:00 +0100,
Wang Han <wanghan@linux.alibaba.com> wrote:
>
> Hi Wei-Lin,
>
> I tested this series on a Yitian 710 system with an ARM Neoverse-N2 CPU
> (128 CPUs, 2 NUMA nodes).
>
> Test environment
> ----------------
>
> L0 kernel: Linux v7.2-rc6
> L1 guest: Ubuntu 26.04 LTS, kernel 7.0.0-27-generic (aarch64)
> QEMU: 10.2.3
>
> L0 NUMA balancing was enabled (`/proc/sys/kernel/numa_balancing=1`).
> The host was booted with `kvm_arm.mode=nested`.
>
> This series fixes a functional hang that is exposed when NUMA balancing is
> enabled. The previous nested stage-2 unmap path is too slow for this
> workload, making the performance problem user-visible: NUMA balancing can
> leave the L1 guest unable to make progress and eventually hang during boot.
>
> The L1 was started with 8 vCPUs and 32 GiB of RAM using:
>
> qemu-system-aarch64 -smp 8 -m 32G \
> -machine virt,accel=kvm,gic-version=3,virtualization=on \
> -cpu host -nographic -enable-kvm \
> -drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \
> -drive if=pflash,format=raw,file=pflash1_bak.img \
> -drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \
> -nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \
> -serial mon:stdio
>
Puzzling. If you are only running an L1 in VHE mode, there is no
shadow S2, and therefore nothing to unmap. For shadow S2s to be built
and affect the MMU notifiers, you need to run an L2.
So what are your actual test conditions?
M.
--
Jazz isn't dead. It just smells funny.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-09-03 7:43 ` Marc Zyngier
@ 2026-09-03 13:28 ` Wei-Lin Chang
2026-09-04 7:01 ` Shuai Xue
2026-09-04 7:49 ` Marc Zyngier
0 siblings, 2 replies; 25+ messages in thread
From: Wei-Lin Chang @ 2026-09-03 13:28 UTC (permalink / raw)
To: Marc Zyngier, Wang Han
Cc: linux-arm-kernel, kvmarm, linux-kernel, oupton, tabba, joey.gouly,
seiden, suzuki.poulose, catalin.marinas, will, ljs,
itaru.kitayama
On Thu, Sep 03, 2026 at 08:43:35AM +0100, Marc Zyngier wrote:
> On Wed, 02 Sep 2026 17:35:00 +0100,
> Wang Han <wanghan@linux.alibaba.com> wrote:
> >
> > Hi Wei-Lin,
> >
> > I tested this series on a Yitian 710 system with an ARM Neoverse-N2 CPU
> > (128 CPUs, 2 NUMA nodes).
> >
> > Test environment
> > ----------------
> >
> > L0 kernel: Linux v7.2-rc6
> > L1 guest: Ubuntu 26.04 LTS, kernel 7.0.0-27-generic (aarch64)
> > QEMU: 10.2.3
> >
> > L0 NUMA balancing was enabled (`/proc/sys/kernel/numa_balancing=1`).
> > The host was booted with `kvm_arm.mode=nested`.
> >
> > This series fixes a functional hang that is exposed when NUMA balancing is
> > enabled. The previous nested stage-2 unmap path is too slow for this
> > workload, making the performance problem user-visible: NUMA balancing can
> > leave the L1 guest unable to make progress and eventually hang during boot.
> >
> > The L1 was started with 8 vCPUs and 32 GiB of RAM using:
> >
> > qemu-system-aarch64 -smp 8 -m 32G \
> > -machine virt,accel=kvm,gic-version=3,virtualization=on \
> > -cpu host -nographic -enable-kvm \
> > -drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \
> > -drive if=pflash,format=raw,file=pflash1_bak.img \
> > -drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \
> > -nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \
> > -serial mon:stdio
> >
>
> Puzzling. If you are only running an L1 in VHE mode, there is no
> shadow S2, and therefore nothing to unmap. For shadow S2s to be built
> and affect the MMU notifiers, you need to run an L2.
I was thinking the same at first, but realized even with L1 in VHE mode
there is a small period of time where L1 runs in its EL1 during boot, so
one nested MMU will become valid for each vCPU. That causes
kvm_nested_s2_unmap() to iterate through the entire IPA space 8 times
(-smp 8).
What I am curious about is whether one single notifier unmap is enough
to hang L1, or were there multiple notifier unmaps.
QEMU with -machine virt uses 40 IPA bits only, unmapping that takes:
1024 (4KB pages, unmapping 1GB per iteration)
32768 (16KB pages, unmapping 32MB per iteration)
2048 (64KB pages, unmapping 512MB per iteration)
iterations for each page size. There aren't many mappings in each
iteration too. Does this really take that long on real hardware (even if
this must be done 8 times)?
Thanks,
Wei-Lin Chang
>
> So what are your actual test conditions?
>
> M.
>
> --
> Jazz isn't dead. It just smells funny.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-09-03 13:28 ` Wei-Lin Chang
@ 2026-09-04 7:01 ` Shuai Xue
2026-09-04 7:54 ` Marc Zyngier
2026-09-04 7:49 ` Marc Zyngier
1 sibling, 1 reply; 25+ messages in thread
From: Shuai Xue @ 2026-09-04 7:01 UTC (permalink / raw)
To: Wei-Lin Chang, Marc Zyngier, Wang Han
Cc: linux-arm-kernel, kvmarm, linux-kernel, oupton, tabba, joey.gouly,
seiden, suzuki.poulose, catalin.marinas, will, ljs,
itaru.kitayama
On 9/3/26 9:28 PM, Wei-Lin Chang wrote:
> On Thu, Sep 03, 2026 at 08:43:35AM +0100, Marc Zyngier wrote:
>> On Wed, 02 Sep 2026 17:35:00 +0100,
>> Wang Han <wanghan@linux.alibaba.com> wrote:
>>>
>>> Hi Wei-Lin,
>>>
>>> I tested this series on a Yitian 710 system with an ARM Neoverse-N2 CPU
>>> (128 CPUs, 2 NUMA nodes).
>>>
>>> Test environment
>>> ----------------
>>>
>>> L0 kernel: Linux v7.2-rc6
>>> L1 guest: Ubuntu 26.04 LTS, kernel 7.0.0-27-generic (aarch64)
>>> QEMU: 10.2.3
>>>
>>> L0 NUMA balancing was enabled (`/proc/sys/kernel/numa_balancing=1`).
>>> The host was booted with `kvm_arm.mode=nested`.
>>>
>>> This series fixes a functional hang that is exposed when NUMA balancing is
>>> enabled. The previous nested stage-2 unmap path is too slow for this
>>> workload, making the performance problem user-visible: NUMA balancing can
>>> leave the L1 guest unable to make progress and eventually hang during boot.
>>>
>>> The L1 was started with 8 vCPUs and 32 GiB of RAM using:
>>>
>>> qemu-system-aarch64 -smp 8 -m 32G \
>>> -machine virt,accel=kvm,gic-version=3,virtualization=on \
>>> -cpu host -nographic -enable-kvm \
>>> -drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \
>>> -drive if=pflash,format=raw,file=pflash1_bak.img \
>>> -drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \
>>> -nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \
>>> -serial mon:stdio
>>>
>>
>> Puzzling. If you are only running an L1 in VHE mode, there is no
>> shadow S2, and therefore nothing to unmap. For shadow S2s to be built
>> and affect the MMU notifiers, you need to run an L2.
>
> I was thinking the same at first, but realized even with L1 in VHE mode
> there is a small period of time where L1 runs in its EL1 during boot, so
> one nested MMU will become valid for each vCPU. That causes
> kvm_nested_s2_unmap() to iterate through the entire IPA space 8 times
> (-smp 8).
>
> What I am curious about is whether one single notifier unmap is enough
> to hang L1, or were there multiple notifier unmaps.
>
> QEMU with -machine virt uses 40 IPA bits only, unmapping that takes:
> 1024 (4KB pages, unmapping 1GB per iteration)
> 32768 (16KB pages, unmapping 32MB per iteration)
> 2048 (64KB pages, unmapping 512MB per iteration)
> iterations for each page size. There aren't many mappings in each
> iteration too. Does this really take that long on real hardware (even if
> this must be done 8 times)?
>
> Thanks,
> Wei-Lin Chang
>
>>
>> So what are your actual test conditions?
>>
>> M.
>>
Hi, Wei-Lin and Marc,
I was able to reproduce this issue and capture ftrace evidence that confirms
the root cause. Below is the analysis, trace log, and timing data.
## Problem
Environment:
- Host (L0): ARM64, KVM with virtualization=on (nested virtualization)
- Guest (L1): Ubuntu 26.04, 8 vCPUs / 32 GB
- Host NUMA balancing enabled, numad active
When booting the L1 QEMU guest, the L1 kernel hits a soft lockup during
early boot (~45 s):
[ 45.646468] watchdog: BUG: soft lockup - CPU#0 stuck for 32s!
[kworker/0:2:330]
[ 45.646882] watchdog: BUG: soft lockup - CPU#2 stuck for 29s!
[snap:1146]
[ 45.647093] watchdog: BUG: soft lockup - CPU#3 stuck for 29s!
[snap:1141]
[ 45.647242] watchdog: BUG: soft lockup - CPU#7 stuck for 29s!
[snap:1144]
At the same time, L0 dmesg reports the QEMU main thread blocked in D-state
for more than 120 s:
[11239.341817] INFO: task qemu-system-aar:170799 blocked in I/O wait for
more than 120 seconds.
...
softleaf_entry_wait_on_locked+0x280/0x2d0
migration_entry_wait+0xdc/0x140
do_swap_page+0x834/0xd80
handle_pte_fault+0x208/0x2b8
__handle_mm_fault+0x228/0x528
handle_mm_fault+0xdc/0x2d8
do_page_fault+0x244/0x790
do_translation_fault+0x4c/0x88
do_mem_abort+0x4c/0xa0
el1_abort+0x50/0x80
el1h_64_sync_handler+0x50/0x108
el1h_64_sync+0x80/0x88
do_sys_poll+0x224/0x290
__arm64_sys_ppoll+0xa4/0x130
Note: This is not a 100% reproducible failure. In my automated loop the hang
reproduced on the 2nd boot attempt, but other attempts ran for 6–9 minutes
without hitting it. The bug is clearly timing-dependent on NUMA migration
activity during the L1 boot window.
## Root cause
The issue is in arch/arm64/kvm/nested.c:
void kvm_nested_s2_unmap(struct kvm *kvm, bool may_block)
{
int i;
lockdep_assert_held_write(&kvm->mmu_lock);
if (!kvm->arch.nested_mmus_size)
return;
for (i = 0; i < kvm->arch.nested_mmus_size; i++) {
struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
if (kvm_s2_mmu_valid(mmu))
kvm_stage2_unmap_range(mmu, 0, kvm_phys_size(mmu), may_block);
}
kvm_invalidate_vncr_ipa(kvm, 0, BIT(kvm->arch.mmu.pgt->ia_bits));
}
When L0 NUMA balancing migrates a page belonging to the QEMU process, the
MMU notifier path calls kvm_unmap_gfn_range():
bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
{
...
__unmap_stage2_range(&kvm->arch.mmu, range->start << PAGE_SHIFT,
(range->end - range->start) << PAGE_SHIFT,
range->may_block);
kvm_nested_s2_unmap(kvm, range->may_block); /* full unmap */
return false;
}
kvm_handle_hva_range() takes kvm->mmu_lock for writing before invoking the
handler and releases it only after the handler returns
(virt/kvm/kvm_main.c:622-642). Therefore, the entire kvm_nested_s2_unmap()
runs under kvm->mmu_lock.
The problem is that kvm_nested_s2_unmap() does not unmap only the affected
GPA range. Instead, it unmaps the entire IPA space (0 to kvm_phys_size(mmu))
for every nested S2 MMU. With nested virtualization enabled, this is very
expensive.
## Trace evidence
I captured function_graph traces for kvm_unmap_gfn_range,
kvm_nested_s2_unmap, and kvm_stage2_unmap_range, plus mm_migrate_pages
tracepoints.
1. numad-triggered full unmap
64) numad-1888 | | /* set_migration_pte:
addr=fff790579000, pte=3040c319d9680 order=0 */
64) numad-1888 | | kvm_unmap_gfn_range() {
64) numad-1888 | | kvm_nested_s2_unmap() {
64) numad-1888 | @ 877415.7 us | kvm_stage2_unmap_range();
64) numad-1888 | @ 877418.2 us | }
64) numad-1888 | @ 877422.0 us | }
Each set_migration_pte line is a single-page NUMA migration. Yet each
migration triggers one full kvm_nested_s2_unmap() that takes 877 ms.
Subsequent calls show per-page unmap durations between 841 ms and 1.16 s.
2. QEMU threads blocked as well
23) qemu-sy-227844 | | kvm_unmap_gfn_range() {
23) qemu-sy-227844 | | kvm_nested_s2_unmap() {
23) qemu-sy-227844 | $ 1170521 us | kvm_stage2_unmap_range();
23) qemu-sy-227844 | $ 1170529 us | }
23) qemu-sy-227844 | $ 1170542 us | }
QEMU's own threads also get stuck in the same full unmap, with one call
reaching 6.67 s.
3. Statistics
┌──────────────┬───────┬──────────────────────────────────────┐
│ Thread │ Calls │ kvm_nested_s2_unmap duration │
├──────────────┼───────┼──────────────────────────────────────┤
│ numad-1888 │ 59 │ min 0.84 s / avg 1.02 s / max 1.16 s │
├──────────────┼───────┼──────────────────────────────────────┤
│ QEMU threads │ 34 │ min 1.14 s / avg 5.14 s / max 6.67 s │
└──────────────┴───────┴──────────────────────────────────────┘
numad migrates pages one after another; each page holds kvm->mmu_lock for
about one second. QEMU vCPU threads cannot acquire mmu_lock and also block
on migration_entry_wait. The L1 guest vCPUs make no forward progress, and
the watchdog fires.
Soft lockup threshold check
Read directly inside the L1 guest:
root@ubuntu-vm:~# cat /proc/sys/kernel/watchdog_thresh
10
So the soft lockup threshold is 2 * watchdog_thresh = 20 s.
The L1 guest reported stuck times of 29~32 s, which exceeds the 20 s
threshold.
## Conclusion
The root cause is confirmed: kvm_nested_s2_unmap() performs a full IPA space
unmap in the MMU notifier path instead of unmapping only the affected
GPA/CPAI range. The interval-tree-based precise range unmap approach is the
right fix.
Please consider applying the patch that replaces the full unmap with
kvm_nested_unmap_cipa_range() to avoid scanning the entire nested stage-2
page table on every NUMA migration.
Thanks,
Shuai
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-09-04 7:01 ` Shuai Xue
@ 2026-09-04 7:54 ` Marc Zyngier
2026-09-05 15:35 ` Shuai Xue
0 siblings, 1 reply; 25+ messages in thread
From: Marc Zyngier @ 2026-09-04 7:54 UTC (permalink / raw)
To: Shuai Xue
Cc: Wei-Lin Chang, Wang Han, linux-arm-kernel, kvmarm, linux-kernel,
oupton, tabba, joey.gouly, seiden, suzuki.poulose,
catalin.marinas, will, ljs, itaru.kitayama
On Fri, 04 Sep 2026 08:01:24 +0100,
Shuai Xue <xueshuai@linux.alibaba.com> wrote:
>
>
>
> On 9/3/26 9:28 PM, Wei-Lin Chang wrote:
> > On Thu, Sep 03, 2026 at 08:43:35AM +0100, Marc Zyngier wrote:
> >> On Wed, 02 Sep 2026 17:35:00 +0100,
> >> Wang Han <wanghan@linux.alibaba.com> wrote:
> >>>
> >>> Hi Wei-Lin,
> >>>
> >>> I tested this series on a Yitian 710 system with an ARM Neoverse-N2 CPU
> >>> (128 CPUs, 2 NUMA nodes).
> >>>
> >>> Test environment
> >>> ----------------
> >>>
> >>> L0 kernel: Linux v7.2-rc6
> >>> L1 guest: Ubuntu 26.04 LTS, kernel 7.0.0-27-generic (aarch64)
> >>> QEMU: 10.2.3
> >>>
> >>> L0 NUMA balancing was enabled (`/proc/sys/kernel/numa_balancing=1`).
> >>> The host was booted with `kvm_arm.mode=nested`.
> >>>
> >>> This series fixes a functional hang that is exposed when NUMA balancing is
> >>> enabled. The previous nested stage-2 unmap path is too slow for this
> >>> workload, making the performance problem user-visible: NUMA balancing can
> >>> leave the L1 guest unable to make progress and eventually hang during boot.
> >>>
> >>> The L1 was started with 8 vCPUs and 32 GiB of RAM using:
> >>>
> >>> qemu-system-aarch64 -smp 8 -m 32G \
> >>> -machine virt,accel=kvm,gic-version=3,virtualization=on \
> >>> -cpu host -nographic -enable-kvm \
> >>> -drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \
> >>> -drive if=pflash,format=raw,file=pflash1_bak.img \
> >>> -drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \
> >>> -nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \
> >>> -serial mon:stdio
> >>>
> >>
> >> Puzzling. If you are only running an L1 in VHE mode, there is no
> >> shadow S2, and therefore nothing to unmap. For shadow S2s to be built
> >> and affect the MMU notifiers, you need to run an L2.
> >
> > I was thinking the same at first, but realized even with L1 in VHE mode
> > there is a small period of time where L1 runs in its EL1 during boot, so
> > one nested MMU will become valid for each vCPU. That causes
> > kvm_nested_s2_unmap() to iterate through the entire IPA space 8 times
> > (-smp 8).
> >
> > What I am curious about is whether one single notifier unmap is enough
> > to hang L1, or were there multiple notifier unmaps.
> >
> > QEMU with -machine virt uses 40 IPA bits only, unmapping that takes:
> > 1024 (4KB pages, unmapping 1GB per iteration)
> > 32768 (16KB pages, unmapping 32MB per iteration)
> > 2048 (64KB pages, unmapping 512MB per iteration)
> > iterations for each page size. There aren't many mappings in each
> > iteration too. Does this really take that long on real hardware (even if
> > this must be done 8 times)?
> >
> > Thanks,
> > Wei-Lin Chang
> >
> >>
> >> So what are your actual test conditions?
> >>
> >> M.
> >>
>
> Hi, Wei-Lin and Marc,
>
> I was able to reproduce this issue and capture ftrace evidence that confirms
> the root cause. Below is the analysis, trace log, and timing data.
[...]
> Each set_migration_pte line is a single-page NUMA migration. Yet each
> migration triggers one full kvm_nested_s2_unmap() that takes 877 ms.
And why is it taking so long? It should be *empty* after the first
iteration.
[...]
> ## Conclusion
>
> The root cause is confirmed: kvm_nested_s2_unmap() performs a full IPA space
> unmap in the MMU notifier path instead of unmapping only the affected
> GPA/CPAI range. The interval-tree-based precise range unmap approach is the
> right fix.
No. This just indicates that this is papering over a bigger problem,
and your AI is jumping to conclusions.
> Please consider applying the patch that replaces the full unmap with
> kvm_nested_unmap_cipa_range() to avoid scanning the entire nested stage-2
> page table on every NUMA migration.
Not until we get to the bottom of this issue.
M.
--
Jazz isn't dead. It just smells funny.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-09-04 7:54 ` Marc Zyngier
@ 2026-09-05 15:35 ` Shuai Xue
2026-09-06 10:42 ` Marc Zyngier
0 siblings, 1 reply; 25+ messages in thread
From: Shuai Xue @ 2026-09-05 15:35 UTC (permalink / raw)
To: Marc Zyngier
Cc: Wei-Lin Chang, Wang Han, linux-arm-kernel, kvmarm, linux-kernel,
oupton, tabba, joey.gouly, seiden, suzuki.poulose,
catalin.marinas, will, ljs, itaru.kitayama
On 9/4/26 3:54 PM, Marc Zyngier wrote:
> On Fri, 04 Sep 2026 08:01:24 +0100,
> Shuai Xue <xueshuai@linux.alibaba.com> wrote:
>>
>>
>>
>> On 9/3/26 9:28 PM, Wei-Lin Chang wrote:
>>> On Thu, Sep 03, 2026 at 08:43:35AM +0100, Marc Zyngier wrote:
>>>> On Wed, 02 Sep 2026 17:35:00 +0100,
>>>> Wang Han <wanghan@linux.alibaba.com> wrote:
>>>>>
>>>>> Hi Wei-Lin,
>>>>>
>>>>> I tested this series on a Yitian 710 system with an ARM Neoverse-N2 CPU
>>>>> (128 CPUs, 2 NUMA nodes).
>>>>>
>>>>> Test environment
>>>>> ----------------
>>>>>
>>>>> L0 kernel: Linux v7.2-rc6
>>>>> L1 guest: Ubuntu 26.04 LTS, kernel 7.0.0-27-generic (aarch64)
>>>>> QEMU: 10.2.3
>>>>>
>>>>> L0 NUMA balancing was enabled (`/proc/sys/kernel/numa_balancing=1`).
>>>>> The host was booted with `kvm_arm.mode=nested`.
>>>>>
>>>>> This series fixes a functional hang that is exposed when NUMA balancing is
>>>>> enabled. The previous nested stage-2 unmap path is too slow for this
>>>>> workload, making the performance problem user-visible: NUMA balancing can
>>>>> leave the L1 guest unable to make progress and eventually hang during boot.
>>>>>
>>>>> The L1 was started with 8 vCPUs and 32 GiB of RAM using:
>>>>>
>>>>> qemu-system-aarch64 -smp 8 -m 32G \
>>>>> -machine virt,accel=kvm,gic-version=3,virtualization=on \
>>>>> -cpu host -nographic -enable-kvm \
>>>>> -drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \
>>>>> -drive if=pflash,format=raw,file=pflash1_bak.img \
>>>>> -drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \
>>>>> -nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \
>>>>> -serial mon:stdio
>>>>>
>>>>
>>>> Puzzling. If you are only running an L1 in VHE mode, there is no
>>>> shadow S2, and therefore nothing to unmap. For shadow S2s to be built
>>>> and affect the MMU notifiers, you need to run an L2.
>>>
>>> I was thinking the same at first, but realized even with L1 in VHE mode
>>> there is a small period of time where L1 runs in its EL1 during boot, so
>>> one nested MMU will become valid for each vCPU. That causes
>>> kvm_nested_s2_unmap() to iterate through the entire IPA space 8 times
>>> (-smp 8).
>>>
>>> What I am curious about is whether one single notifier unmap is enough
>>> to hang L1, or were there multiple notifier unmaps.
>>>
>>> QEMU with -machine virt uses 40 IPA bits only, unmapping that takes:
>>> 1024 (4KB pages, unmapping 1GB per iteration)
>>> 32768 (16KB pages, unmapping 32MB per iteration)
>>> 2048 (64KB pages, unmapping 512MB per iteration)
>>> iterations for each page size. There aren't many mappings in each
>>> iteration too. Does this really take that long on real hardware (even if
>>> this must be done 8 times)?
>>>
>>> Thanks,
>>> Wei-Lin Chang
>>>
>>>>
>>>> So what are your actual test conditions?
>>>>
>>>> M.
>>>>
>>
>> Hi, Wei-Lin and Marc,
>>
>> I was able to reproduce this issue and capture ftrace evidence that confirms
>> the root cause. Below is the analysis, trace log, and timing data.
>
> [...]
>
>> Each set_migration_pte line is a single-page NUMA migration. Yet each
>> migration triggers one full kvm_nested_s2_unmap() that takes 877 ms.
>
> And why is it taking so long? It should be *empty* after the first
> iteration.
Good question.
After dive into the details trace, let to try to answer the question.
Yes, it is empty -- and that is exactly the point: the 877ms is paid *for*
an empty table.
The cost is not in the walk and not in clearing PTEs;
**it is 262,144 broadcast TLB invalidations**, one at the end of every
1GB chunk, each ~3.3us. Since v6.6 the cost of an unmap is
proportional to the size of the IPA range, not to the number of
mappings; an empty table pays in full.
>
> [...]
>
>> ## Conclusion
>>
>> The root cause is confirmed: kvm_nested_s2_unmap() performs a full IPA space
>> unmap in the MMU notifier path instead of unmapping only the affected
>> GPA/CPAI range. The interval-tree-based precise range unmap approach is the
>> right fix.
>
> No. This just indicates that this is papering over a bigger problem,
> and your AI is jumping to conclusions.
Sorry for the jumping up.
The culprit is 7657ea920c54 ("KVM: arm64: Use TLBI range-based
instructions for unmap", v6.6). kvm_pgtable_stage2_unmap() ends
*every* call with an unconditional kvm_tlb_flush_vmid_range(), whether
or not the walk cleared a single PTE:
ret = kvm_pgtable_walk(pgt, addr, size, &walker);
if (stage2_unmap_defer_tlb_flush(pgt))
/* Perform the deferred TLB invalidations */
kvm_tlb_flush_vmid_range(pgt->mmu, addr, size);
stage2_apply_range() calls it once per 1GB chunk, and the nested MMU
covers the guest PARange -- 48 bits here, so 262,144 calls per
kvm_nested_s2_unmap(). Each broadcast is one IPAS2E1IS (range) plus one VMALLE1IS
plus two DSB(ish), ~3.2-3.4us without ftrace:
877ms / 262,144 chunks = 3.35us per chunk
which is exactly the per-broadcast cost.
All numbers below are from
ftrace function_graph on the same workload as Wang Han reported
(Yitian 710, 128 CPUs, Neoverse-N2, L0 v7.2-rc6 with
kvm_arm.mode=nested, QEMU virt machine, 8 vCPUs / 32G, 4K host
pages), and can be checked against the trace (pre-change kernel, ~90s
window):
$ grep -c 'kvm_nested_s2_unmap() {' trace.txt
31 # callbacks in the window
$ grep -c 'kvm_pgtable_stage2_unmap() {' trace.txt
8476033 # one call per 1GB chunk
$ grep -c '__kvm_tlb_flush_vmid_range();' trace.txt
8476033 # one broadcast per chunk
The chunk count and the broadcast count are *equal*: every chunk call
ends in a broadcast, mapped or not. Per chunk, instrumented:
kvm_pgtable_stage2_unmap() median 5.097us
kvm_tlb_flush_vmid_range() median 4.645us
__kvm_tlb_flush_vmid_range() p50 4.0us / p90 5.0 / p99 6.25us
so ~4.6us of every ~5.1us chunk is the flush; the walk of the empty
chunk is ~0.4us. A representative uncontended episode, from the numad
thread that executes the unmap itself: 1312ms total, of which 1147ms
(87%) inside kvm_tlb_flush_vmid_range() and 106ms (8%) walking. And
only 289 of the 8,476,033 chunk calls (0.003%) exceed 20us -- there is
nothing in this table.
Raw sample, pre-change kernel (every chunk is a frame, because it
contains the traced flush):
13) numad-1888 | | kvm_pgtable_stage2_unmap() {
13) numad-1888 | | kvm_tlb_flush_vmid_range() {
13) numad-1888 | 3.240 us | __kvm_tlb_flush_vmid_range();
13) numad-1888 | 3.660 us | }
13) numad-1888 | 4.060 us | }
To confirm that the table is empty, we made the flush conditional on
the walk having actually cleared a valid leaf PTE (candidate fix
below; nothing else changes). The flush count then becomes a direct
detector of "did this chunk clear anything?":
- The first full-IPA unmap of the boot issued 2 flushes -- the
boot-time EL1-period mappings, the few dozen pages you predicted,
living in two 1GB chunks.
- All 35 subsequent full-IPA unmaps in that run issued *zero*
flushes. So yes: empty after the first iteration, exactly as you
say. Before the change, each of those empty iterations still cost
877ms, because the broadcast does not depend on the walk having
cleared anything.
- The same empty walk now measures 0.208us per chunk (median), i.e.
~55ms of traced chunk time for a full 48-bit sweep:
24) qemu-sy-25907 | | kvm_nested_s2_unmap() {
24) qemu-sy-25907 | | kvm_stage2_unmap_range() {
24) qemu-sy-25907 | 2.740 us | kvm_pgtable_stage2_unmap();
24) qemu-sy-25907 | 0.300 us | kvm_pgtable_stage2_unmap();
24) qemu-sy-25907 | 0.260 us | kvm_pgtable_stage2_unmap();
24) qemu-sy-25907 | 0.240 us | kvm_pgtable_stage2_unmap();
while a canonical-S2 chunk that really does clear the migrated page
still flushes:
13) qemu-sy-25900 | | kvm_stage2_unmap_range() {
13) qemu-sy-25900 | | kvm_pgtable_stage2_unmap() {
13) qemu-sy-25900 | | kvm_tlb_flush_vmid_range() {
13) qemu-sy-25900 | 4.180 us | __kvm_tlb_flush_vmid_range();
13) qemu-sy-25900 | 4.780 us | }
The candidate fix is below. Table entries
(KVM_PGTABLE_WALK_TABLE_POST) are unaffected: stage2_unmap_put_pte()
keeps issuing the immediate __kvm_tlb_flush_vmid_ipa() for them, and
their child leaves are counted as leaves within the same walk, so any
call that clears something still flushes a superset of what it
cleared. This restores the pre-v6.6 semantics -- cost proportional to
what is mapped. With it, an empty nested unmap costs ~110ms
instrumented (the pure walk); getting to "exactly zero" would
additionally require kvm_nested_s2_unmap() to skip nested MMUs that
are valid but empty.
Not proposing to apply anything as-is -- this is the measurement that
answers "why is it taking so long", plus the minimal change that
confirms it.
Happy to share the full traces if that is useful.
Thanks,
Shuai
---8<---
Subject: [PATCH] KVM: arm64: Skip deferred TLB invalidation for empty unmaps
kvm_pgtable_stage2_unmap() issues kvm_tlb_flush_vmid_range()
unconditionally at the end of every call, whether or not the walk
cleared any PTE. As stage2_apply_range() calls it once per 1GB chunk,
unmapping a sparse (or empty) range costs one broadcast per chunk:
for a 48-bit IPA space that is 262,144 broadcasts, ~0.9s on a
Neoverse-N2, paid by every mmu-notifier callback. This is the
dominant cost of kvm_nested_s2_unmap() in the NUMA-balancing
workload, where the nested shadow S2 is all but empty.
Only perform the deferred invalidation when the walk actually
cleared a valid leaf PTE, restoring the pre-v6.6 semantics of the
cost being proportional to what is mapped. Table entries keep the
immediate __kvm_tlb_flush_vmid_ipa() issued from
stage2_unmap_put_pte(), and their child leaves are counted within
the same walk, so any call that clears anything still flushes a
superset of the range it cleared.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -891,11 +891,17 @@ static bool stage2_unmap_defer_tlb_flush(struct kvm_pgtable *pgt)
return system_supports_tlb_range() && cpus_have_final_cap(ARM64_HAS_STAGE2_FWB);
}
+struct stage2_unmap_data {
+ struct kvm_pgtable *pgt;
+ u64 unmapped_leaves;
+};
+
static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx,
struct kvm_s2_mmu *mmu,
struct kvm_pgtable_mm_ops *mm_ops)
{
- struct kvm_pgtable *pgt = ctx->arg;
+ struct stage2_unmap_data *data = ctx->arg;
+ struct kvm_pgtable *pgt = data->pgt;
/*
* Clear the existing PTE, and perform break-before-make if it was
@@ -1155,7 +1161,8 @@ int kvm_pgtable_stage2_annotate(struct kvm_pgtable *pgt, u64 addr, u64 size,
static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
enum kvm_pgtable_walk_flags visit)
{
- struct kvm_pgtable *pgt = ctx->arg;
+ struct stage2_unmap_data *data = ctx->arg;
+ struct kvm_pgtable *pgt = data->pgt;
struct kvm_s2_mmu *mmu = pgt->mmu;
struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops;
kvm_pte_t *childp = NULL;
@@ -1183,6 +1190,9 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
* block entry and rely on the remaining portions being faulted
* back lazily.
*/
+ if (kvm_pte_valid(ctx->old) && !kvm_pte_table(ctx->old, ctx->level))
+ data->unmapped_leaves++;
+
stage2_unmap_put_pte(ctx, mmu, mm_ops);
if (need_flush && mm_ops->dcache_clean_inval_poc)
@@ -1198,14 +1208,18 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size)
{
int ret;
+ struct stage2_unmap_data data = {
+ .pgt = pgt,
+ .unmapped_leaves = 0,
+ };
struct kvm_pgtable_walker walker = {
.cb = stage2_unmap_walker,
- .arg = pgt,
+ .arg = &data,
.flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST,
};
ret = kvm_pgtable_walk(pgt, addr, size, &walker);
- if (stage2_unmap_defer_tlb_flush(pgt))
+ if (stage2_unmap_defer_tlb_flush(pgt) && data.unmapped_leaves)
/* Perform the deferred TLB invalidations */
kvm_tlb_flush_vmid_range(pgt->mmu, addr, size);
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-09-05 15:35 ` Shuai Xue
@ 2026-09-06 10:42 ` Marc Zyngier
0 siblings, 0 replies; 25+ messages in thread
From: Marc Zyngier @ 2026-09-06 10:42 UTC (permalink / raw)
To: Shuai Xue
Cc: Wei-Lin Chang, Wang Han, linux-arm-kernel, kvmarm, linux-kernel,
oupton, tabba, joey.gouly, seiden, suzuki.poulose,
catalin.marinas, will, ljs, itaru.kitayama
On Sat, 05 Sep 2026 16:35:01 +0100,
Shuai Xue <xueshuai@linux.alibaba.com> wrote:
>
>
>
> On 9/4/26 3:54 PM, Marc Zyngier wrote:
> > On Fri, 04 Sep 2026 08:01:24 +0100,
> > Shuai Xue <xueshuai@linux.alibaba.com> wrote:
> >>
> >>
> >>
> >> On 9/3/26 9:28 PM, Wei-Lin Chang wrote:
> >>> On Thu, Sep 03, 2026 at 08:43:35AM +0100, Marc Zyngier wrote:
> >>>> On Wed, 02 Sep 2026 17:35:00 +0100,
> >>>> Wang Han <wanghan@linux.alibaba.com> wrote:
> >>>>>
> >>>>> Hi Wei-Lin,
> >>>>>
> >>>>> I tested this series on a Yitian 710 system with an ARM Neoverse-N2 CPU
> >>>>> (128 CPUs, 2 NUMA nodes).
> >>>>>
> >>>>> Test environment
> >>>>> ----------------
> >>>>>
> >>>>> L0 kernel: Linux v7.2-rc6
> >>>>> L1 guest: Ubuntu 26.04 LTS, kernel 7.0.0-27-generic (aarch64)
> >>>>> QEMU: 10.2.3
> >>>>>
> >>>>> L0 NUMA balancing was enabled (`/proc/sys/kernel/numa_balancing=1`).
> >>>>> The host was booted with `kvm_arm.mode=nested`.
> >>>>>
> >>>>> This series fixes a functional hang that is exposed when NUMA balancing is
> >>>>> enabled. The previous nested stage-2 unmap path is too slow for this
> >>>>> workload, making the performance problem user-visible: NUMA balancing can
> >>>>> leave the L1 guest unable to make progress and eventually hang during boot.
> >>>>>
> >>>>> The L1 was started with 8 vCPUs and 32 GiB of RAM using:
> >>>>>
> >>>>> qemu-system-aarch64 -smp 8 -m 32G \
> >>>>> -machine virt,accel=kvm,gic-version=3,virtualization=on \
> >>>>> -cpu host -nographic -enable-kvm \
> >>>>> -drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \
> >>>>> -drive if=pflash,format=raw,file=pflash1_bak.img \
> >>>>> -drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \
> >>>>> -nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \
> >>>>> -serial mon:stdio
> >>>>>
> >>>>
> >>>> Puzzling. If you are only running an L1 in VHE mode, there is no
> >>>> shadow S2, and therefore nothing to unmap. For shadow S2s to be built
> >>>> and affect the MMU notifiers, you need to run an L2.
> >>>
> >>> I was thinking the same at first, but realized even with L1 in VHE mode
> >>> there is a small period of time where L1 runs in its EL1 during boot, so
> >>> one nested MMU will become valid for each vCPU. That causes
> >>> kvm_nested_s2_unmap() to iterate through the entire IPA space 8 times
> >>> (-smp 8).
> >>>
> >>> What I am curious about is whether one single notifier unmap is enough
> >>> to hang L1, or were there multiple notifier unmaps.
> >>>
> >>> QEMU with -machine virt uses 40 IPA bits only, unmapping that takes:
> >>> 1024 (4KB pages, unmapping 1GB per iteration)
> >>> 32768 (16KB pages, unmapping 32MB per iteration)
> >>> 2048 (64KB pages, unmapping 512MB per iteration)
> >>> iterations for each page size. There aren't many mappings in each
> >>> iteration too. Does this really take that long on real hardware (even if
> >>> this must be done 8 times)?
> >>>
> >>> Thanks,
> >>> Wei-Lin Chang
> >>>
> >>>>
> >>>> So what are your actual test conditions?
> >>>>
> >>>> M.
> >>>>
> >>
> >> Hi, Wei-Lin and Marc,
> >>
> >> I was able to reproduce this issue and capture ftrace evidence that confirms
> >> the root cause. Below is the analysis, trace log, and timing data.
> >
> > [...]
> >
> >> Each set_migration_pte line is a single-page NUMA migration. Yet each
> >> migration triggers one full kvm_nested_s2_unmap() that takes 877 ms.
> >
> > And why is it taking so long? It should be *empty* after the first
> > iteration.
>
> Good question.
>
> After dive into the details trace, let to try to answer the question.
>
> Yes, it is empty -- and that is exactly the point: the 877ms is paid *for*
> an empty table.
>
> The cost is not in the walk and not in clearing PTEs;
> **it is 262,144 broadcast TLB invalidations**, one at the end of every
> 1GB chunk, each ~3.3us. Since v6.6 the cost of an unmap is
> proportional to the size of the IPA range, not to the number of
> mappings; an empty table pays in full.
>
> >
> > [...]
> >
> >> ## Conclusion
> >>
> >> The root cause is confirmed: kvm_nested_s2_unmap() performs a full IPA space
> >> unmap in the MMU notifier path instead of unmapping only the affected
> >> GPA/CPAI range. The interval-tree-based precise range unmap approach is the
> >> right fix.
> >
> > No. This just indicates that this is papering over a bigger problem,
> > and your AI is jumping to conclusions.
>
> Sorry for the jumping up.
>
> The culprit is 7657ea920c54 ("KVM: arm64: Use TLBI range-based
> instructions for unmap", v6.6). kvm_pgtable_stage2_unmap() ends
> *every* call with an unconditional kvm_tlb_flush_vmid_range(), whether
> or not the walk cleared a single PTE:
>
> ret = kvm_pgtable_walk(pgt, addr, size, &walker);
> if (stage2_unmap_defer_tlb_flush(pgt))
> /* Perform the deferred TLB invalidations */
> kvm_tlb_flush_vmid_range(pgt->mmu, addr, size);
>
> stage2_apply_range() calls it once per 1GB chunk, and the nested MMU
> covers the guest PARange -- 48 bits here, so 262,144 calls per
> kvm_nested_s2_unmap(). Each broadcast is one IPAS2E1IS (range) plus one VMALLE1IS
> plus two DSB(ish), ~3.2-3.4us without ftrace:
>
> 877ms / 262,144 chunks = 3.35us per chunk
>
> which is exactly the per-broadcast cost.
Right. That's pretty compelling, thanks for digging into this. Your
proposed approach (counting the invalidated regions) is interesting,
but I don't think it is the correct one.
The real issue here is that we treat a full S2 unmap as if it was a
set of ranges. This is what needs fixing, because we can invalidate
the whole thing with exactly *ONE* TLBI.
This is even more important once you run an L2, as L1 will also
perform its own TLB invalidation, and we want to avoid having trapping
pointlessly. This also propagates in the way we handle TLB emulation.
[...]
> The candidate fix is below. Table entries
> (KVM_PGTABLE_WALK_TABLE_POST) are unaffected: stage2_unmap_put_pte()
> keeps issuing the immediate __kvm_tlb_flush_vmid_ipa() for them, and
> their child leaves are counted as leaves within the same walk, so any
> call that clears something still flushes a superset of what it
> cleared. This restores the pre-v6.6 semantics -- cost proportional to
> what is mapped. With it, an empty nested unmap costs ~110ms
> instrumented (the pure walk); getting to "exactly zero" would
> additionally require kvm_nested_s2_unmap() to skip nested MMUs that
> are valid but empty.
That's an interesting remark. I guess we could add some extra tracking
for that, but let's see what we can do about the above first.
I've hacked something together and pushed the result at [1]
(compile-tested only). I'd appreciate it if you could put it to the
test with your setup.
Thanks,
M.
[1] https://web.git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/unmap-vmall
--
Jazz isn't dead. It just smells funny.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-09-03 13:28 ` Wei-Lin Chang
2026-09-04 7:01 ` Shuai Xue
@ 2026-09-04 7:49 ` Marc Zyngier
2026-09-04 11:37 ` Wei-Lin Chang
2026-09-04 22:42 ` Wei-Lin Chang
1 sibling, 2 replies; 25+ messages in thread
From: Marc Zyngier @ 2026-09-04 7:49 UTC (permalink / raw)
To: Wei-Lin Chang
Cc: Wang Han, linux-arm-kernel, kvmarm, linux-kernel, oupton, tabba,
joey.gouly, seiden, suzuki.poulose, catalin.marinas, will, ljs,
itaru.kitayama
On Thu, 03 Sep 2026 14:28:16 +0100,
Wei-Lin Chang <weilin.chang@arm.com> wrote:
>
> On Thu, Sep 03, 2026 at 08:43:35AM +0100, Marc Zyngier wrote:
> > On Wed, 02 Sep 2026 17:35:00 +0100,
> > Wang Han <wanghan@linux.alibaba.com> wrote:
> > >
> > > Hi Wei-Lin,
> > >
> > > I tested this series on a Yitian 710 system with an ARM Neoverse-N2 CPU
> > > (128 CPUs, 2 NUMA nodes).
> > >
> > > Test environment
> > > ----------------
> > >
> > > L0 kernel: Linux v7.2-rc6
> > > L1 guest: Ubuntu 26.04 LTS, kernel 7.0.0-27-generic (aarch64)
> > > QEMU: 10.2.3
> > >
> > > L0 NUMA balancing was enabled (`/proc/sys/kernel/numa_balancing=1`).
> > > The host was booted with `kvm_arm.mode=nested`.
> > >
> > > This series fixes a functional hang that is exposed when NUMA balancing is
> > > enabled. The previous nested stage-2 unmap path is too slow for this
> > > workload, making the performance problem user-visible: NUMA balancing can
> > > leave the L1 guest unable to make progress and eventually hang during boot.
> > >
> > > The L1 was started with 8 vCPUs and 32 GiB of RAM using:
> > >
> > > qemu-system-aarch64 -smp 8 -m 32G \
> > > -machine virt,accel=kvm,gic-version=3,virtualization=on \
> > > -cpu host -nographic -enable-kvm \
> > > -drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \
> > > -drive if=pflash,format=raw,file=pflash1_bak.img \
> > > -drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \
> > > -nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \
> > > -serial mon:stdio
> > >
> >
> > Puzzling. If you are only running an L1 in VHE mode, there is no
> > shadow S2, and therefore nothing to unmap. For shadow S2s to be built
> > and affect the MMU notifiers, you need to run an L2.
>
> I was thinking the same at first, but realized even with L1 in VHE mode
> there is a small period of time where L1 runs in its EL1 during boot, so
> one nested MMU will become valid for each vCPU. That causes
> kvm_nested_s2_unmap() to iterate through the entire IPA space 8 times
> (-smp 8).
It should be one nested MMU for the whole VM, not one per vcpu.
That's assuming they share the same VMID+VTCR.
> What I am curious about is whether one single notifier unmap is enough
> to hang L1, or were there multiple notifier unmaps.
>
> QEMU with -machine virt uses 40 IPA bits only, unmapping that takes:
> 1024 (4KB pages, unmapping 1GB per iteration)
> 32768 (16KB pages, unmapping 32MB per iteration)
> 2048 (64KB pages, unmapping 512MB per iteration)
> iterations for each page size. There aren't many mappings in each
> iteration too. Does this really take that long on real hardware (even if
> this must be done 8 times)?
This should be close to being at zero cost, so something else is
amiss.
Could you please have a look?
M.
--
Jazz isn't dead. It just smells funny.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-09-04 7:49 ` Marc Zyngier
@ 2026-09-04 11:37 ` Wei-Lin Chang
2026-09-04 22:42 ` Wei-Lin Chang
1 sibling, 0 replies; 25+ messages in thread
From: Wei-Lin Chang @ 2026-09-04 11:37 UTC (permalink / raw)
To: Marc Zyngier
Cc: Wang Han, linux-arm-kernel, kvmarm, linux-kernel, oupton, tabba,
joey.gouly, seiden, suzuki.poulose, catalin.marinas, will, ljs,
itaru.kitayama
On Fri, Sep 04, 2026 at 08:49:14AM +0100, Marc Zyngier wrote:
[...]
> > > >
> > > > The L1 was started with 8 vCPUs and 32 GiB of RAM using:
> > > >
> > > > qemu-system-aarch64 -smp 8 -m 32G \
> > > > -machine virt,accel=kvm,gic-version=3,virtualization=on \
> > > > -cpu host -nographic -enable-kvm \
> > > > -drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \
> > > > -drive if=pflash,format=raw,file=pflash1_bak.img \
> > > > -drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \
> > > > -nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \
> > > > -serial mon:stdio
> > > >
> > >
> > > Puzzling. If you are only running an L1 in VHE mode, there is no
> > > shadow S2, and therefore nothing to unmap. For shadow S2s to be built
> > > and affect the MMU notifiers, you need to run an L2.
> >
> > I was thinking the same at first, but realized even with L1 in VHE mode
> > there is a small period of time where L1 runs in its EL1 during boot, so
> > one nested MMU will become valid for each vCPU. That causes
> > kvm_nested_s2_unmap() to iterate through the entire IPA space 8 times
> > (-smp 8).
>
> It should be one nested MMU for the whole VM, not one per vcpu.
> That's assuming they share the same VMID+VTCR.
(sigh) Yes, rookie mistake from me as always.
>
> > What I am curious about is whether one single notifier unmap is enough
> > to hang L1, or were there multiple notifier unmaps.
> >
> > QEMU with -machine virt uses 40 IPA bits only, unmapping that takes:
> > 1024 (4KB pages, unmapping 1GB per iteration)
> > 32768 (16KB pages, unmapping 32MB per iteration)
> > 2048 (64KB pages, unmapping 512MB per iteration)
> > iterations for each page size. There aren't many mappings in each
> > iteration too. Does this really take that long on real hardware (even if
> > this must be done 8 times)?
>
> This should be close to being at zero cost, so something else is
> amiss.
>
> Could you please have a look?
Of course.
Thanks,
Wei-Lin Chang
>
> M.
>
> --
> Jazz isn't dead. It just smells funny.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-09-04 7:49 ` Marc Zyngier
2026-09-04 11:37 ` Wei-Lin Chang
@ 2026-09-04 22:42 ` Wei-Lin Chang
2026-09-05 13:48 ` Marc Zyngier
2026-09-05 15:49 ` Shuai Xue
1 sibling, 2 replies; 25+ messages in thread
From: Wei-Lin Chang @ 2026-09-04 22:42 UTC (permalink / raw)
To: Marc Zyngier
Cc: Wang Han, linux-arm-kernel, kvmarm, linux-kernel, oupton, tabba,
joey.gouly, seiden, suzuki.poulose, catalin.marinas, will, ljs,
itaru.kitayama
On Fri, Sep 04, 2026 at 08:49:14AM +0100, Marc Zyngier wrote:
[...]
> > > >
> > > > The L1 was started with 8 vCPUs and 32 GiB of RAM using:
> > > >
> > > > qemu-system-aarch64 -smp 8 -m 32G \
> > > > -machine virt,accel=kvm,gic-version=3,virtualization=on \
> > > > -cpu host -nographic -enable-kvm \
> > > > -drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \
> > > > -drive if=pflash,format=raw,file=pflash1_bak.img \
> > > > -drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \
> > > > -nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \
> > > > -serial mon:stdio
> > > >
> > >
> > > Puzzling. If you are only running an L1 in VHE mode, there is no
> > > shadow S2, and therefore nothing to unmap. For shadow S2s to be built
> > > and affect the MMU notifiers, you need to run an L2.
> >
> > I was thinking the same at first, but realized even with L1 in VHE mode
> > there is a small period of time where L1 runs in its EL1 during boot, so
> > one nested MMU will become valid for each vCPU. That causes
> > kvm_nested_s2_unmap() to iterate through the entire IPA space 8 times
> > (-smp 8).
>
> It should be one nested MMU for the whole VM, not one per vcpu.
> That's assuming they share the same VMID+VTCR.
>
> > What I am curious about is whether one single notifier unmap is enough
> > to hang L1, or were there multiple notifier unmaps.
> >
> > QEMU with -machine virt uses 40 IPA bits only, unmapping that takes:
> > 1024 (4KB pages, unmapping 1GB per iteration)
> > 32768 (16KB pages, unmapping 32MB per iteration)
> > 2048 (64KB pages, unmapping 512MB per iteration)
> > iterations for each page size. There aren't many mappings in each
> > iteration too. Does this really take that long on real hardware (even if
> > this must be done 8 times)?
>
> This should be close to being at zero cost, so something else is
> amiss.
>
Guess what? I am completely wrong once again.
The complete range of the nested mmus isn't about the value
supplied by KVM_VM_TYPE_ARM_IPA_SIZE. Instead it is the PARange exposed
to the guest. That's because the guest hypervisor is allowed to expose
whatever memory map it likes to its nested guests, as long as it is
within the guest's PARange. This is explained clearly in
init_nested_s2_mmu().
Now the PARange exposed to guests should be much larger than 40bits. On
the neoverse-n2 it should be 48bits (256TB), therefore the iterations
required for each unmapping would be 256x:
262144 (4KB pages, unmapping 1GB per iteration)
8388608 (16KB pages, unmapping 32MB per iteration)
524288 (64KB pages, unmapping 512MB per iteration)
I think these numbers match the hang time Shuai have shown.
Just to demonstrate the difference I have tried manually limiting
kvm_ipa_limit to 40, and tested with software emulation. Before the
manual change a kvm_nested_s2_unmap() finished after 42 seconds, after
limiting kvm_ipa_limit kvm_nested_s2_unmap() can finish in less than a
second.
With this, I think there aren't other underlying issues, it just is that
slow unfortunately..
Thanks,
Wei-Lin Chang
> Could you please have a look?
>
> M.
>
> --
> Jazz isn't dead. It just smells funny.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-09-04 22:42 ` Wei-Lin Chang
@ 2026-09-05 13:48 ` Marc Zyngier
2026-09-05 15:49 ` Shuai Xue
1 sibling, 0 replies; 25+ messages in thread
From: Marc Zyngier @ 2026-09-05 13:48 UTC (permalink / raw)
To: Wei-Lin Chang
Cc: Wang Han, linux-arm-kernel, kvmarm, linux-kernel, oupton, tabba,
joey.gouly, seiden, suzuki.poulose, catalin.marinas, will, ljs,
itaru.kitayama
On Fri, 04 Sep 2026 23:42:15 +0100,
Wei-Lin Chang <weilin.chang@arm.com> wrote:
>
> On Fri, Sep 04, 2026 at 08:49:14AM +0100, Marc Zyngier wrote:
>
> [...]
>
> > > > >
> > > > > The L1 was started with 8 vCPUs and 32 GiB of RAM using:
> > > > >
> > > > > qemu-system-aarch64 -smp 8 -m 32G \
> > > > > -machine virt,accel=kvm,gic-version=3,virtualization=on \
> > > > > -cpu host -nographic -enable-kvm \
> > > > > -drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \
> > > > > -drive if=pflash,format=raw,file=pflash1_bak.img \
> > > > > -drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \
> > > > > -nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \
> > > > > -serial mon:stdio
> > > > >
> > > >
> > > > Puzzling. If you are only running an L1 in VHE mode, there is no
> > > > shadow S2, and therefore nothing to unmap. For shadow S2s to be built
> > > > and affect the MMU notifiers, you need to run an L2.
> > >
> > > I was thinking the same at first, but realized even with L1 in VHE mode
> > > there is a small period of time where L1 runs in its EL1 during boot, so
> > > one nested MMU will become valid for each vCPU. That causes
> > > kvm_nested_s2_unmap() to iterate through the entire IPA space 8 times
> > > (-smp 8).
> >
> > It should be one nested MMU for the whole VM, not one per vcpu.
> > That's assuming they share the same VMID+VTCR.
> >
> > > What I am curious about is whether one single notifier unmap is enough
> > > to hang L1, or were there multiple notifier unmaps.
> > >
> > > QEMU with -machine virt uses 40 IPA bits only, unmapping that takes:
> > > 1024 (4KB pages, unmapping 1GB per iteration)
> > > 32768 (16KB pages, unmapping 32MB per iteration)
> > > 2048 (64KB pages, unmapping 512MB per iteration)
> > > iterations for each page size. There aren't many mappings in each
> > > iteration too. Does this really take that long on real hardware (even if
> > > this must be done 8 times)?
> >
> > This should be close to being at zero cost, so something else is
> > amiss.
> >
>
> Guess what? I am completely wrong once again.
>
> The complete range of the nested mmus isn't about the value
> supplied by KVM_VM_TYPE_ARM_IPA_SIZE. Instead it is the PARange exposed
> to the guest. That's because the guest hypervisor is allowed to expose
> whatever memory map it likes to its nested guests, as long as it is
> within the guest's PARange. This is explained clearly in
> init_nested_s2_mmu().
>
> Now the PARange exposed to guests should be much larger than 40bits. On
> the neoverse-n2 it should be 48bits (256TB), therefore the iterations
> required for each unmapping would be 256x:
>
> 262144 (4KB pages, unmapping 1GB per iteration)
> 8388608 (16KB pages, unmapping 32MB per iteration)
> 524288 (64KB pages, unmapping 512MB per iteration)
>
> I think these numbers match the hang time Shuai have shown.
>
> Just to demonstrate the difference I have tried manually limiting
> kvm_ipa_limit to 40, and tested with software emulation. Before the
> manual change a kvm_nested_s2_unmap() finished after 42 seconds, after
> limiting kvm_ipa_limit kvm_nested_s2_unmap() can finish in less than a
> second.
>
> With this, I think there aren't other underlying issues, it just is that
> slow unfortunately..
I don't buy this. There should be very little mapped in the shadow S2,
a few dozen pages at most. As you noticed, there is very little the
kernel executes at EL1 before upgrading to EL2. Invalidating this
should be cheap. Furthermore, once invalidated after the last vcpu
has booted, the cost should be exactly zero.
It seems that these two assumptions are not holding up, and I wonder
why.
M.
--
Jazz isn't dead. It just smells funny.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-09-04 22:42 ` Wei-Lin Chang
2026-09-05 13:48 ` Marc Zyngier
@ 2026-09-05 15:49 ` Shuai Xue
2026-09-05 23:48 ` Wei-Lin Chang
1 sibling, 1 reply; 25+ messages in thread
From: Shuai Xue @ 2026-09-05 15:49 UTC (permalink / raw)
To: Wei-Lin Chang, Marc Zyngier
Cc: Wang Han, linux-arm-kernel, kvmarm, linux-kernel, oupton, tabba,
joey.gouly, seiden, suzuki.poulose, catalin.marinas, will, ljs,
itaru.kitayama
On 9/5/26 6:42 AM, Wei-Lin Chang wrote:
> On Fri, Sep 04, 2026 at 08:49:14AM +0100, Marc Zyngier wrote:
>
> [...]
>
>>>>>
>>>>> The L1 was started with 8 vCPUs and 32 GiB of RAM using:
>>>>>
>>>>> qemu-system-aarch64 -smp 8 -m 32G \
>>>>> -machine virt,accel=kvm,gic-version=3,virtualization=on \
>>>>> -cpu host -nographic -enable-kvm \
>>>>> -drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \
>>>>> -drive if=pflash,format=raw,file=pflash1_bak.img \
>>>>> -drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \
>>>>> -nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \
>>>>> -serial mon:stdio
>>>>>
>>>>
>>>> Puzzling. If you are only running an L1 in VHE mode, there is no
>>>> shadow S2, and therefore nothing to unmap. For shadow S2s to be built
>>>> and affect the MMU notifiers, you need to run an L2.
>>>
>>> I was thinking the same at first, but realized even with L1 in VHE mode
>>> there is a small period of time where L1 runs in its EL1 during boot, so
>>> one nested MMU will become valid for each vCPU. That causes
>>> kvm_nested_s2_unmap() to iterate through the entire IPA space 8 times
>>> (-smp 8).
>>
>> It should be one nested MMU for the whole VM, not one per vcpu.
>> That's assuming they share the same VMID+VTCR.
>>
>>> What I am curious about is whether one single notifier unmap is enough
>>> to hang L1, or were there multiple notifier unmaps.
>>>
>>> QEMU with -machine virt uses 40 IPA bits only, unmapping that takes:
>>> 1024 (4KB pages, unmapping 1GB per iteration)
>>> 32768 (16KB pages, unmapping 32MB per iteration)
>>> 2048 (64KB pages, unmapping 512MB per iteration)
>>> iterations for each page size. There aren't many mappings in each
>>> iteration too. Does this really take that long on real hardware (even if
>>> this must be done 8 times)?
>>
>> This should be close to being at zero cost, so something else is
>> amiss.
>>
>
> Guess what? I am completely wrong once again.
>
> The complete range of the nested mmus isn't about the value
> supplied by KVM_VM_TYPE_ARM_IPA_SIZE. Instead it is the PARange exposed
> to the guest. That's because the guest hypervisor is allowed to expose
> whatever memory map it likes to its nested guests, as long as it is
> within the guest's PARange. This is explained clearly in
> init_nested_s2_mmu().
>
> Now the PARange exposed to guests should be much larger than 40bits. On
> the neoverse-n2 it should be 48bits (256TB), therefore the iterations
> required for each unmapping would be 256x:
>
> 262144 (4KB pages, unmapping 1GB per iteration)
> 8388608 (16KB pages, unmapping 32MB per iteration)
> 524288 (64KB pages, unmapping 512MB per iteration)
>
> I think these numbers match the hang time Shuai have shown.
>
> Just to demonstrate the difference I have tried manually limiting
> kvm_ipa_limit to 40, and tested with software emulation. Before the
> manual change a kvm_nested_s2_unmap() finished after 42 seconds, after
> limiting kvm_ipa_limit kvm_nested_s2_unmap() can finish in less than a
> second.
Hi Wei-Lin,
Your PARange analysis is what this whole diagnosis rests on -- the
48-bit/256x arithmetic is exactly right, and our ftrace data confirms
it to the chunk: every full-IPA kvm_nested_s2_unmap() in our traces
walks precisely 262,144 1GB chunks, the number from your table. The
kvm_ipa_limit=40 experiment also pointed straight at the answer. I'd
only push back on the last sentence:
>
> With this, I think there aren't other underlying issues, it just is that
> slow unfortunately..
I don't think it is "just that slow" -- I think it is a specific
regression, and your own data localizes it better than the conclusion
suggests.
The thing is, kvm_ipa_limit=40 cannot tell *what* each chunk spends
its ~3.3us on, because the walk is also per-chunk: cutting the chunk
count by 256x shrinks both terms together, whichever one dominates.
The experiment that separates them is one where the chunk count does
not change at all.
We ran that experiment. Details are in my reply to Marc, but the
short version:
- Before: 877ms per callback, 262,144 chunk calls, 262,144
broadcasts. Chunk count and broadcast count are exactly equal
(8,476,033 = 8,476,033 over the trace window) -- every chunk
ends in a broadcast, mapped or not. Per chunk, ~4.6us of the
~5.1us is inside kvm_tlb_flush_vmid_range(); the walk of an
empty chunk is ~0.2us. In an uncontended numad episode, 87% of
the 1312ms is flush time and 8% is walk time.
- After conditioning the flush on the walk having actually cleared
a valid leaf (the candidate fix in my other mail): same 262,144
chunks, same walk, ~110ms instrumented. Nothing about the walk
changed -- ~3.1us of the per-chunk 3.3us was the broadcast.
- And the table really is empty, as Marc says: with the flush
count acting as a detector, the first full-IPA unmap of the boot
issues 2 flushes (the EL1-boot-period mappings, a few dozen
pages in two 1GB chunks); all 35 subsequent ones issue zero.
The code history agrees. Before 7657ea920c54, the TLBI lived inside
stage2_put_pte()'s 'if (kvm_pte_valid(ctx->old))' -- an empty walk
issued zero invalidations, ever. v6.6 hoisted the invalidation out
to the end of kvm_pgtable_stage2_unmap() to batch it into one range
TLBI per call, but the condition got dropped on the way. So "since
v6.6, the cost is proportional to the IPA range, not to the number
of mappings" -- which is precisely why the empty table pays in full,
and why I'd frame it as a fixable regression rather than an inherent
cost.
None of this subtracts from your PARange finding -- it depends on
it. The 42s -> <1s result was the 256x chunk scaling; the remaining
<1s is what the conditional flush removes.
Thanks,
Shuai
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-09-05 15:49 ` Shuai Xue
@ 2026-09-05 23:48 ` Wei-Lin Chang
2026-09-06 2:16 ` Shuai Xue
0 siblings, 1 reply; 25+ messages in thread
From: Wei-Lin Chang @ 2026-09-05 23:48 UTC (permalink / raw)
To: Shuai Xue, Marc Zyngier
Cc: Wang Han, linux-arm-kernel, kvmarm, linux-kernel, oupton, tabba,
joey.gouly, seiden, suzuki.poulose, catalin.marinas, will, ljs,
itaru.kitayama
On Sat, Sep 05, 2026 at 11:49:32PM +0800, Shuai Xue wrote:
[...]
>
> Hi Wei-Lin,
>
> Your PARange analysis is what this whole diagnosis rests on -- the
> 48-bit/256x arithmetic is exactly right, and our ftrace data confirms
> it to the chunk: every full-IPA kvm_nested_s2_unmap() in our traces
> walks precisely 262,144 1GB chunks, the number from your table. The
> kvm_ipa_limit=40 experiment also pointed straight at the answer. I'd
> only push back on the last sentence:
>
> >
> > With this, I think there aren't other underlying issues, it just is that
> > slow unfortunately..
>
>
> I don't think it is "just that slow" -- I think it is a specific
> regression, and your own data localizes it better than the conclusion
> suggests.
>
> The thing is, kvm_ipa_limit=40 cannot tell *what* each chunk spends
> its ~3.3us on, because the walk is also per-chunk: cutting the chunk
> count by 256x shrinks both terms together, whichever one dominates.
> The experiment that separates them is one where the chunk count does
> not change at all.
>
> We ran that experiment. Details are in my reply to Marc, but the
> short version:
>
> - Before: 877ms per callback, 262,144 chunk calls, 262,144
> broadcasts. Chunk count and broadcast count are exactly equal
> (8,476,033 = 8,476,033 over the trace window) -- every chunk
> ends in a broadcast, mapped or not. Per chunk, ~4.6us of the
> ~5.1us is inside kvm_tlb_flush_vmid_range(); the walk of an
> empty chunk is ~0.2us. In an uncontended numad episode, 87% of
> the 1312ms is flush time and 8% is walk time.
>
> - After conditioning the flush on the walk having actually cleared
> a valid leaf (the candidate fix in my other mail): same 262,144
> chunks, same walk, ~110ms instrumented. Nothing about the walk
> changed -- ~3.1us of the per-chunk 3.3us was the broadcast.
>
> - And the table really is empty, as Marc says: with the flush
> count acting as a detector, the first full-IPA unmap of the boot
> issues 2 flushes (the EL1-boot-period mappings, a few dozen
> pages in two 1GB chunks); all 35 subsequent ones issue zero.
>
> The code history agrees. Before 7657ea920c54, the TLBI lived inside
> stage2_put_pte()'s 'if (kvm_pte_valid(ctx->old))' -- an empty walk
> issued zero invalidations, ever. v6.6 hoisted the invalidation out
> to the end of kvm_pgtable_stage2_unmap() to batch it into one range
> TLBI per call, but the condition got dropped on the way. So "since
> v6.6, the cost is proportional to the IPA range, not to the number
> of mappings" -- which is precisely why the empty table pays in full,
> and why I'd frame it as a fixable regression rather than an inherent
> cost.
>
> None of this subtracts from your PARange finding -- it depends on
> it. The 42s -> <1s result was the 256x chunk scaling; the remaining
> <1s is what the conditional flush removes.
Thanks for the analysis!
I completely agree. My finding was only a small part of the full
picture. I stopped at finding out the real iteration count and didn't
realize 3-5us is a long time for a kvm_pgtable_stage2_unmap() on a empty
table (thanks for the learning opportunity).
So, because there are no relevant tlb entries created in the first
place, the per iteration TLBIs and barriers are mostly doing nothing
useful and cause a 10x slowdown.
Thanks,
Wei-Lin Chang
>
> Thanks,
> Shuai
>
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)
2026-09-05 23:48 ` Wei-Lin Chang
@ 2026-09-06 2:16 ` Shuai Xue
0 siblings, 0 replies; 25+ messages in thread
From: Shuai Xue @ 2026-09-06 2:16 UTC (permalink / raw)
To: Wei-Lin Chang, Marc Zyngier
Cc: Wang Han, linux-arm-kernel, kvmarm, linux-kernel, oupton, tabba,
joey.gouly, seiden, suzuki.poulose, catalin.marinas, will, ljs,
itaru.kitayama
On 9/6/26 7:48 AM, Wei-Lin Chang wrote:
> On Sat, Sep 05, 2026 at 11:49:32PM +0800, Shuai Xue wrote:
>
> [...]
>
>>
>> Hi Wei-Lin,
>>
>> Your PARange analysis is what this whole diagnosis rests on -- the
>> 48-bit/256x arithmetic is exactly right, and our ftrace data confirms
>> it to the chunk: every full-IPA kvm_nested_s2_unmap() in our traces
>> walks precisely 262,144 1GB chunks, the number from your table. The
>> kvm_ipa_limit=40 experiment also pointed straight at the answer. I'd
>> only push back on the last sentence:
>>
>>>
>>> With this, I think there aren't other underlying issues, it just is that
>>> slow unfortunately..
>>
>>
>> I don't think it is "just that slow" -- I think it is a specific
>> regression, and your own data localizes it better than the conclusion
>> suggests.
>>
>> The thing is, kvm_ipa_limit=40 cannot tell *what* each chunk spends
>> its ~3.3us on, because the walk is also per-chunk: cutting the chunk
>> count by 256x shrinks both terms together, whichever one dominates.
>> The experiment that separates them is one where the chunk count does
>> not change at all.
>>
>> We ran that experiment. Details are in my reply to Marc, but the
>> short version:
>>
>> - Before: 877ms per callback, 262,144 chunk calls, 262,144
>> broadcasts. Chunk count and broadcast count are exactly equal
>> (8,476,033 = 8,476,033 over the trace window) -- every chunk
>> ends in a broadcast, mapped or not. Per chunk, ~4.6us of the
>> ~5.1us is inside kvm_tlb_flush_vmid_range(); the walk of an
>> empty chunk is ~0.2us. In an uncontended numad episode, 87% of
>> the 1312ms is flush time and 8% is walk time.
>>
>> - After conditioning the flush on the walk having actually cleared
>> a valid leaf (the candidate fix in my other mail): same 262,144
>> chunks, same walk, ~110ms instrumented. Nothing about the walk
>> changed -- ~3.1us of the per-chunk 3.3us was the broadcast.
>>
>> - And the table really is empty, as Marc says: with the flush
>> count acting as a detector, the first full-IPA unmap of the boot
>> issues 2 flushes (the EL1-boot-period mappings, a few dozen
>> pages in two 1GB chunks); all 35 subsequent ones issue zero.
>>
>> The code history agrees. Before 7657ea920c54, the TLBI lived inside
>> stage2_put_pte()'s 'if (kvm_pte_valid(ctx->old))' -- an empty walk
>> issued zero invalidations, ever. v6.6 hoisted the invalidation out
>> to the end of kvm_pgtable_stage2_unmap() to batch it into one range
>> TLBI per call, but the condition got dropped on the way. So "since
>> v6.6, the cost is proportional to the IPA range, not to the number
>> of mappings" -- which is precisely why the empty table pays in full,
>> and why I'd frame it as a fixable regression rather than an inherent
>> cost.
>>
>> None of this subtracts from your PARange finding -- it depends on
>> it. The 42s -> <1s result was the 256x chunk scaling; the remaining
>> <1s is what the conditional flush removes.
>
> Thanks for the analysis!
>
> I completely agree. My finding was only a small part of the full
> picture. I stopped at finding out the real iteration count and didn't
> realize 3-5us is a long time for a kvm_pgtable_stage2_unmap() on a empty
> table (thanks for the learning opportunity).
>
> So, because there are no relevant tlb entries created in the first
> place, the per iteration TLBIs and barriers are mostly doing nothing
> useful and cause a 10x slowdown.
Hi Marc, Wei-Lin,
Thanks for pushing back on this. Let me try to give the
full picture (Wei-Lin, please correct me if I get any of it wrong).
## 1. The cost of one kvm_nested_s2_unmap()
per-callback cost = (IPA range / 1GB) x per-chunk cost
With numbers:
chunks (range) per-chunk cost per callback
current 262,144 ~3.55us (3.35 broadcast ~877ms
(48-bit IPA) + 0.2 walk, always)
The first factor is what Wei-Lin identified: the nested MMU covers
the guest PARange -- 48 bits here, not the 40 bits QEMU asked for --
so stage2_apply_range() calls kvm_pgtable_stage2_unmap() once per 1GB
chunk. The second factor is where the surprise was hiding: the
broadcast is unconditional, whether or not the chunk contained
anything.
Since both factors are large, there are two independent ways to bring
the product down:
chunks (range) per-chunk cost per callback
current 262,144 ~3.55us (always) ~877ms
precise unmap ~1 ~3.55us (always) ~us
(nested layer,
Wei-Lin's
interval-tree)
conditional 262,144 ~0.2us when empty ~110ms
flush (~3.55us if it
(pgtable layer, cleared something)
below)
Each attack a different factor. Wei-Lin's patch shrinks the range to
the single chunk containing the migrated page; the conditional flush
keeps the full-range walk but stops paying the broadcast for chunks
that cleared nothing. Either one alone brings the per-callback cost
far below the point where consecutive callbacks can accumulate a
watchdog-relevant stall; they also compose.
## 2. Conditioning the flush: the how and the why
We tried the pgtable-layer variant: the walker counts valid leaf PTEs
it actually clears, and the deferred flush fires only if that count
is non-zero:
ret = kvm_pgtable_walk(pgt, addr, size, &walker);
if (stage2_unmap_defer_tlb_flush(pgt) && data.unmapped_leaves)
kvm_tlb_flush_vmid_range(pgt->mmu, addr, size);
Correctness: a non-zero count means at least one valid leaf was
cleared, and the flush covers the whole [addr, addr+size) range, a
superset of everything this call cleared. A zero count means no valid
PTE was cleared, so no core can hold a stale translation this call is
responsible for. Table entries keep their immediate
__kvm_tlb_flush_vmid_ipa() from stage2_unmap_put_pte(), and their
child leaves are counted as leaves within the same walk.
Same L1 QEMU bootup test, five 6-minute runs: per-callback 877ms -> ~110ms
instrumented (~40-80ms uninstrumented, the pure walk); broadcasts
8,476,033 -> 221 in total, of which 2 were for actual nested mappings
and the rest canonical flushes of the pages NUMA balancing really
migrated; the soft lockups are gone (0/5 runs). And it is a
pgtable-layer change, so every other sparse-range unmap caller --
canonical S2, pKVM -- stops paying the per-chunk broadcast whenever
the range is much larger than what is mapped in it.
## 3. Where the unconditional broadcast came from
The condition was not dropped deliberately; it is a casualty of
7657ea920c54 ("KVM: arm64: Use TLBI range-based instructions for
unmap", v6.6). That commit batched the per-PTE TLBI into one range
TLBI at the end of each kvm_pgtable_stage2_unmap() call. But in the
old stage2_put_pte(), the "only if a valid PTE was cleared" condition
was simply the *position* of the TLBI inside the 'if
(kvm_pte_valid(ctx->old))' block -- there was nothing to move, so
nothing was moved:
dense chunk empty chunk
pre-v6.6 262,144 TLBIs 0 TLBIs
since v6.6 1 range TLBI 1 range TLBI
The dense column is a 260,000x win -- the case the commit was written
for (unmapping a fully populated memslot). The empty column is a
0 -> 1 regression per chunk, invisible as long as nobody unmaps
ranges that are simultaneously huge and empty. Before v6.6, the cost
of an unmap was proportional to the number of mappings; since, it is
proportional to the size of the IPA range. Nested virtualisation
plus NUMA balancing is, as far as we know, the first caller to do
exactly that, per notifier event, while holding the mmu_lock for
write -- 262,144 x 3.35us is the ~877ms we measured, and the vCPUs'
stall accumulates across consecutive callbacks until it crosses the
guest watchdog threshold (one event: 7 of 8 guest CPUs reporting
"stuck for 24-28s" within the same millisecond).
So the conditional flush is not an optimisation so much as restoring
the lost condition -- making empty unmaps behave like pre-v6.6 again.
## 4. The question
Do you think restoring the lost condition -- i.e. the pgtable-layer
fix above, conditioning the deferred flush on the walk having
actually cleared a valid leaf -- is worth submitting as a proper fix
for 7657ea920c54?
As far as we can tell it is a ~20-line change that
restores the pre-v6.6 semantics with no new ones: empty and sparse
unmaps cost the walk again, dense unmaps keep the range-TLBI win.
Thanks,
Shuai
>
> Thanks,
> Wei-Lin Chang
>
>>
>> Thanks,
>> Shuai
>>
>>
^ permalink raw reply [flat|nested] 25+ messages in thread