[PATCH v3 00/11] KVM Dirty-bit cleaning hw accelerator (HACDBS)

[PATCH v3 00/11] KVM Dirty-bit cleaning hw accelerator (HACDBS)

[Leonardo Bras]

Disclaimer: While this patchset is buildable and testable on it's own,
it is not ready to be merged, as it depends on bits from another
patchset that will superseed the one in [1], and will enable HDBSS.
See note below on patch 1.

My expectation on sharing this earlier is to gather feedback on
implementation and testing methods, to make sure it's sure it's ready
when [1] becomes ready. It should work for both lazy-splitting 
and eager-splitting scenarios.

===

Create an arch-generic dirty-bit cleaning acceleration interface, which
compiles-out if the arch does not implement it, and creates no new API.
Using that, implement an arm64 accelerator based on HACDBS.

This implementation is able to accelerate the cleaning on both
dirty-bitmap and dirty-ring tracking mechanisms on KVM.

Patch 1 is here just to make this testable, as this patchset
depends on bits from HDBSS that are not upstream yet. It is composed
of a bunch of bits that I collected across HDBSS patches so this can
work, so few free to ignore it on review. Patch 1 enables eager-splitting
by default, but that's for testing purposes only, this should work with 
both eager and lazy splitting, as both mark the page to be clean with
the DBM bit. 

This version relies on polling HACDBS registers to verify if the processing
is done, an thus HACDBSIRQ is not enabled/used.

Kernel v7.2-rc5 + this patchset builds properly, passing both kvm selftests
for dirty-bit tracking[2] and a qemu live migration test, with both
HW HACDBS enabled or disabled.

On terms of performance improvement, tests were done using
dirty_log_perf_test[3] to measure the time spent on the following ioctl:
a - KVM_GET_DIRTY_LOG,     using dirty-bitmap without manual protect
    command:    ./dirty_log_perf_test -m 3 -m 6 -m 12 -g
b - KVM_CLEAR_DIRTY_LOG,   using dirty-bitmap with manual protect
    command:    ./dirty_log_perf_test -m 3 -m 6 -m 12
c - KVM_RESET_DIRTY_RINGS, using dirty-ring, using 4096 entries/vcpu.
    command:    ./dirty_log_perf_test -m 3 -m 6 -m 12 -d 4096

Tests ran in the model show that runtime was reduced by:
-(a) 87.36% (0.08% stdev) for 1GB memory, and
     87.22% (0.01% stdev) for 3GB memory
-(b) 85.72% (0.08% stdev) for 1GB memory, and
     86.10% (0.04% stdev) for 3GB memory
-(b) 70.08% (0.24% stdev) for 1GB memory, and
     69.99% (0.03% stdev) for 3GB memory

Which means those syscalls are from 3.3x to 8x faster.
Above numbers already take into account the improvements in S2
hugepage-splitting that is implemented by [4].

Please let me know of any question :)

Thanks for reviewing!
Leo


Changes since v2:
- HACDBS completion is detected by polling registers instead of HACDBSIRQ
  - No need for ACPI enablement any longer
  - Fixes issue with possibly waiting forever on wfi()
- Cleaning routine now protected by preempt_notifiers,
  - Save reg status on preemption, and restore when it's back.
  - Was preempt_disable(), which was not good in case of long buffers
- Improved error paths in case of invalid memslot/range on dirty_ring
- Added dsb() barrier to make sure the cpu finishes writting to array
  before HACDBS starts.
- Fixed page-splitting skipping ranges for dirty-bit.
- Patches 1 & 2 from v2 merged into patch 1 of v3, as it's only about 
  setting enough bits to test it. 
v2 Link: https://lore.kernel.org/all/20260629111820.1873540-1-leo.bras@arm.com/

Changes since v1:
- Improvements on splitting with manual protect, skipping when cleaning
  pages from the same level-2 entry. (new patch)
- Got the correct concept of chunk_size and thus:
  - Corrected it to a reasonable chunk to do page splitting before
    rescheduling, considering new improvements from [4].
  - TTWL is not based in chunk size, so fix it in LAST_LEVEL
- Minor fixes, removing debugging traces
v1 Link: https://lore.kernel.org/all/20260430111424.3479613-2-leo.bras@arm.com/


[1]: https://lore.kernel.org/all/20260709104026.2612599-1-zhengtian10@huawei.com/
[2]: dirty_log_test && dirty_log_perf_test
[3]: using this patchset to enable dirty-ring on dirty_log_perf_test:
     https://lore.kernel.org/all/20260629105950.1790259-1-leo.bras@arm.com/
[4]: https://lore.kernel.org/all/20260618131447.764085-1-leo.bras@arm.com/

Leonardo Bras (11):
  KVM: arm64: HDBSS bits
  arm64/cpufeature: Add system-wide FEAT_HACDBS detection
  arm64/sysreg: Add HACDBS consumer and base registers
  KVM: arm64: dirty_bit: Add base FEAT_HACDBS cleaning routine
  kvm: Add arch-generic interface for hw-accelerated dirty-bitmap
    cleaning
  KVM: arm64: Add hardware-accelerated dirty-bitmap cleaning routine
  KVM: arm64: Dirty-bitmap: avoid splitting previously split blocks
  kvm/dirty_ring: Introduce get_memslot and move helpers to header
  kvm/dirty_ring: Add arch-generic interface for hw-accelerated
    dirty-ring cleaning
  KVM: arm64: Add hardware-accelerated dirty-ring cleaning routine
  KVM: arm64: Enable KVM_HW_DIRTY_BIT

 arch/arm64/include/asm/cpufeature.h    |  10 +
 arch/arm64/include/asm/kvm_dirty_bit.h |  49 +++
 arch/arm64/include/asm/kvm_pgtable.h   |   3 +
 include/linux/kvm_dirty_bit.h          |  34 ++
 include/linux/kvm_dirty_ring.h         |  12 +
 include/linux/kvm_host.h               |   3 +
 arch/arm64/kernel/cpufeature.c         |  20 ++
 arch/arm64/kvm/dirty_bit.c             | 438 +++++++++++++++++++++++++
 arch/arm64/kvm/hyp/pgtable.c           |  15 +-
 arch/arm64/kvm/mmu.c                   |  12 +-
 virt/kvm/dirty_ring.c                  |  35 +-
 virt/kvm/kvm_main.c                    |  13 +-
 arch/arm64/kvm/Kconfig                 |   1 +
 arch/arm64/kvm/Makefile                |   2 +-
 arch/arm64/tools/cpucaps               |   2 +
 arch/arm64/tools/sysreg                |  30 ++
 virt/kvm/Kconfig                       |   3 +
 17 files changed, 660 insertions(+), 22 deletions(-)
 create mode 100644 arch/arm64/include/asm/kvm_dirty_bit.h
 create mode 100644 include/linux/kvm_dirty_bit.h
 create mode 100644 arch/arm64/kvm/dirty_bit.c


base-commit: f5098b6bae761e346ebcd9da7f95622c04733cff
-- 
2.55.0

[PATCH v3 01/11] KVM: arm64: HDBSS bits

[Leonardo Bras]

All those bits should come from a future version of HDBSS patchset:
https://lore.kernel.org/all/20260709104026.2612599-1-zhengtian10@huawei.com/

I added them here in order to fulfill the dependencies and be able to
easily build and test this patchset, but this particular patch should *not*
be merged upstream.

It also sets a default nonzero value for chunksize, which will force eager
splitting, and allow this patchset to be properly tested. While
lazy-splitting should be possible in the next HDBSS patchset, it introduces
no new behaviours in how HACDBS shoud work, so having eager splitting
should be enough for testing.

Signed-off-by: Leonardo Bras <leo.bras@arm.com>
---
 arch/arm64/include/asm/cpufeature.h    |  5 +++++
 arch/arm64/include/asm/kvm_dirty_bit.h | 12 ++++++++++++
 arch/arm64/include/asm/kvm_pgtable.h   |  3 +++
 arch/arm64/kernel/cpufeature.c         | 12 ++++++++++++
 arch/arm64/kvm/dirty_bit.c             | 16 ++++++++++++++++
 arch/arm64/kvm/hyp/pgtable.c           | 15 +++++++++++++--
 arch/arm64/kvm/mmu.c                   |  8 ++++++--
 arch/arm64/kvm/Makefile                |  2 +-
 arch/arm64/tools/cpucaps               |  1 +
 9 files changed, 69 insertions(+), 5 deletions(-)
 create mode 100644 arch/arm64/include/asm/kvm_dirty_bit.h
 create mode 100644 arch/arm64/kvm/dirty_bit.c

diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index a57870fa96db..bdfab086fd94 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -856,20 +856,25 @@ static inline bool system_supports_poe(void)
 static inline bool system_supports_gcs(void)
 {
 	return alternative_has_cap_unlikely(ARM64_HAS_GCS);
 }
 
 static inline bool system_supports_haft(void)
 {
 	return cpus_have_final_cap(ARM64_HAFT);
 }
 
+static inline bool system_supports_hdbss(void)
+{
+	return cpus_have_final_cap(ARM64_HAS_HDBSS);
+}
+
 static __always_inline bool system_supports_mpam(void)
 {
 	return alternative_has_cap_unlikely(ARM64_MPAM);
 }
 
 static __always_inline bool system_supports_mpam_hcr(void)
 {
 	return alternative_has_cap_unlikely(ARM64_MPAM_HCR);
 }
 
diff --git a/arch/arm64/include/asm/kvm_dirty_bit.h b/arch/arm64/include/asm/kvm_dirty_bit.h
new file mode 100644
index 000000000000..dd16438f0651
--- /dev/null
+++ b/arch/arm64/include/asm/kvm_dirty_bit.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Ltd.
+ * Author: Leonardo Bras <leo.bras@arm.com>
+ */
+
+#ifndef __ARM64_KVM_DIRTY_BIT_H__
+#define __ARM64_KVM_DIRTY_BIT_H__
+
+#include <asm/kvm_pgtable.h>
+
+#endif /* __ARM64_KVM_DIRTY_BIT_H__ */
diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
index 41a8687938eb..646ff88e0258 100644
--- a/arch/arm64/include/asm/kvm_pgtable.h
+++ b/arch/arm64/include/asm/kvm_pgtable.h
@@ -86,20 +86,22 @@ typedef u64 kvm_pte_t;
 #define KVM_PTE_LEAF_ATTR_HI		GENMASK(63, 50)
 
 #define KVM_PTE_LEAF_ATTR_HI_SW		GENMASK(58, 55)
 
 #define KVM_PTE_LEAF_ATTR_HI_S1_XN	BIT(54)
 #define KVM_PTE_LEAF_ATTR_HI_S1_UXN	BIT(54)
 #define KVM_PTE_LEAF_ATTR_HI_S1_PXN	BIT(53)
 
 #define KVM_PTE_LEAF_ATTR_HI_S2_XN	GENMASK(54, 53)
 
+#define KVM_PTE_LEAF_ATTR_HI_S2_DBM	BIT(51)
+
 #define KVM_PTE_LEAF_ATTR_HI_S1_GP	BIT(50)
 
 #define KVM_PTE_LEAF_ATTR_S2_PERMS	(KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R | \
 					 KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W | \
 					 KVM_PTE_LEAF_ATTR_HI_S2_XN)
 
 /* pKVM invalid pte encodings */
 #define KVM_INVALID_PTE_TYPE_MASK	GENMASK(63, 60)
 #define KVM_INVALID_PTE_ANNOT_MASK	~(KVM_PTE_VALID | \
 					  KVM_INVALID_PTE_TYPE_MASK)
@@ -246,20 +248,21 @@ struct kvm_pgtable_mm_ops {
 };
 
 /**
  * enum kvm_pgtable_stage2_flags - Stage-2 page-table flags.
  * @KVM_PGTABLE_S2_IDMAP:	Only use identity mappings.
  * @KVM_PGTABLE_S2_AS_S1:	Final memory attributes are that of Stage-1.
  */
 enum kvm_pgtable_stage2_flags {
 	KVM_PGTABLE_S2_IDMAP			= BIT(0),
 	KVM_PGTABLE_S2_AS_S1			= BIT(1),
+	KVM_PGTABLE_S2_DBM			= BIT(2),
 };
 
 /**
  * enum kvm_pgtable_prot - Page-table permissions and attributes.
  * @KVM_PGTABLE_PROT_UX:	Unprivileged execute permission.
  * @KVM_PGTABLE_PROT_PX:	Privileged execute permission.
  * @KVM_PGTABLE_PROT_X:		Privileged and unprivileged execute permission.
  * @KVM_PGTABLE_PROT_W:		Write permission.
  * @KVM_PGTABLE_PROT_R:		Read permission.
  * @KVM_PGTABLE_PROT_DEVICE:	Device attributes.
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 9a22df0c5120..aa327eebaf1c 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -2124,20 +2124,25 @@ static bool has_nested_virt_support(const struct arm64_cpu_capabilities *cap,
 
 	return true;
 }
 
 static bool hvhe_possible(const struct arm64_cpu_capabilities *entry,
 			  int __unused)
 {
 	return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_HVHE);
 }
 
+static bool has_vhe_hdbss(const struct arm64_cpu_capabilities *entry, int cope)
+{
+	return is_kernel_in_hyp_mode() && has_cpuid_feature(entry, cope);
+}
+
 bool cpu_supports_bbml2_noabort(void)
 {
 	/*
 	 * We want to allow usage of BBML2 in as wide a range of kernel contexts
 	 * as possible. This list is therefore an allow-list of known-good
 	 * implementations that both support BBML2 and additionally, fulfill the
 	 * extra constraint of never generating TLB conflict aborts when using
 	 * the relaxed BBML2 semantics (such aborts make use of BBML2 in certain
 	 * kernel contexts difficult to prove safe against recursive aborts).
 	 *
@@ -2774,20 +2779,27 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		 * cannot be emulated in software (no access fault will occur).
 		 * Therefore this should be used only if it's supported system
 		 * wide.
 		 */
 		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
 		.capability = ARM64_HAFT,
 		.matches = has_cpuid_feature,
 		ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, HAFDBS, HAFT)
 	},
 #endif
+	{
+		.desc = "Hardware Dirty state tracking structure (HDBSS)",
+		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+		.capability = ARM64_HAS_HDBSS,
+		.matches = has_vhe_hdbss,
+		ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, HAFDBS, HDBSS)
+	},
 	{
 		.desc = "CRC32 instructions",
 		.capability = ARM64_HAS_CRC32,
 		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
 		.matches = has_cpuid_feature,
 		ARM64_CPUID_FIELDS(ID_AA64ISAR0_EL1, CRC32, IMP)
 	},
 	{
 		.desc = "Speculative Store Bypassing Safe (SSBS)",
 		.capability = ARM64_SSBS,
diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c
new file mode 100644
index 000000000000..32fe938d6bf7
--- /dev/null
+++ b/arch/arm64/kvm/dirty_bit.c
@@ -0,0 +1,16 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Ltd.
+ * Author: Leonardo Bras <leo.bras@arm.com>
+ */
+
+#include <asm/kvm_dirty_bit.h>
+
+/* HDBSS entry field definitions */
+#define HDBSS_ENTRY_VALID BIT(0)
+#define HDBSS_ENTRY_TTWL_SHIFT (1)
+#define HDBSS_ENTRY_TTWL_MASK (GENMASK(3, 1))
+#define HDBSS_ENTRY_TTWL(x) \
+	(((x) << HDBSS_ENTRY_TTWL_SHIFT) & HDBSS_ENTRY_TTWL_MASK)
+#define HDBSS_ENTRY_TTWL_RESV HDBSS_ENTRY_TTWL(-4)
+#define HDBSS_ENTRY_IPA GENMASK_ULL(55, 12)
diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index b74dd5ce1efd..0e2ab3fc6910 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -724,23 +724,27 @@ static int stage2_set_prot_attr(struct kvm_pgtable *pgt, enum kvm_pgtable_prot p
 		attr = KVM_S2_MEMATTR(pgt, NORMAL);
 	}
 
 	r = stage2_set_xn_attr(prot, &attr);
 	if (r)
 		return r;
 
 	if (prot & KVM_PGTABLE_PROT_R)
 		attr |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R;
 
-	if (prot & KVM_PGTABLE_PROT_W)
+	if (prot & KVM_PGTABLE_PROT_W) {
 		attr |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W;
 
+		if (pgt->flags & KVM_PGTABLE_S2_DBM)
+			attr |= KVM_PTE_LEAF_ATTR_HI_S2_DBM;
+	}
+
 	if (!kvm_lpa2_is_enabled())
 		attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S2_SH, sh);
 
 	attr |= KVM_PTE_LEAF_ATTR_LO_S2_AF;
 	attr |= prot & KVM_PTE_LEAF_ATTR_HI_SW;
 	*ptep = attr;
 
 	return 0;
 }
 
@@ -1360,23 +1364,27 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
 	kvm_pte_t xn = 0, set = 0, clr = 0;
 	s8 level;
 	int ret;
 
 	if (prot & KVM_PTE_LEAF_ATTR_HI_SW)
 		return -EINVAL;
 
 	if (prot & KVM_PGTABLE_PROT_R)
 		set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R;
 
-	if (prot & KVM_PGTABLE_PROT_W)
+	if (prot & KVM_PGTABLE_PROT_W) {
 		set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W;
 
+		if (pgt->flags & KVM_PGTABLE_S2_DBM)
+			set |= KVM_PTE_LEAF_ATTR_HI_S2_DBM;
+	}
+
 	if (prot & KVM_PGTABLE_PROT_X) {
 		ret = stage2_set_xn_attr(prot, &xn);
 		if (ret)
 			return ret;
 
 		set |= xn & KVM_PTE_LEAF_ATTR_HI_S2_XN;
 		clr |= ~xn & KVM_PTE_LEAF_ATTR_HI_S2_XN;
 	}
 
 	ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level, flags);
@@ -1581,20 +1589,23 @@ int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
 	u64 vtcr = mmu->vtcr;
 	u32 ia_bits = VTCR_EL2_IPA(vtcr);
 	u32 sl0 = FIELD_GET(VTCR_EL2_SL0_MASK, vtcr);
 	s8 start_level = VTCR_EL2_TGRAN_SL0_BASE - sl0;
 
 	pgd_sz = kvm_pgd_pages(ia_bits, start_level) * PAGE_SIZE;
 	pgt->pgd = (kvm_pteref_t)mm_ops->zalloc_pages_exact(pgd_sz);
 	if (!pgt->pgd)
 		return -ENOMEM;
 
+	if (system_supports_hdbss())
+		flags |= KVM_PGTABLE_S2_DBM;
+
 	pgt->ia_bits		= ia_bits;
 	pgt->start_level	= start_level;
 	pgt->mm_ops		= mm_ops;
 	pgt->mmu		= mmu;
 	pgt->flags		= flags;
 	pgt->force_pte_cb	= force_pte_cb;
 
 	/* Ensure zeroed PGD pages are visible to the hardware walker */
 	dsb(ishst);
 	return 0;
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 6c941aaa10c6..e086c01a9325 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1020,22 +1020,26 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
 
 	mmu->last_vcpu_ran = alloc_percpu(typeof(*mmu->last_vcpu_ran));
 	if (!mmu->last_vcpu_ran) {
 		err = -ENOMEM;
 		goto out_destroy_pgtable;
 	}
 
 	for_each_possible_cpu(cpu)
 		*per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1;
 
-	 /* The eager page splitting is disabled by default */
-	mmu->split_page_chunk_size = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT;
+	 /* The eager page splitting is disabled by default if system has no HDBSS */
+	if (system_supports_hdbss())
+		mmu->split_page_chunk_size = 256 * 1024 * PAGE_SIZE;
+	else
+		mmu->split_page_chunk_size = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT;
+
 	mmu->split_page_cache.gfp_zero = __GFP_ZERO;
 
 	mmu->pgd_phys = __pa(pgt->pgd);
 
 	if (kvm_is_nested_s2_mmu(kvm, mmu))
 		kvm_init_nested_s2_mmu(mmu);
 
 	return 0;
 
 out_destroy_pgtable:
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 59612d2f277c..6faacd857346 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -17,21 +17,21 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \
 	 inject_fault.o va_layout.o handle_exit.o config.o \
 	 guest.o debug.o reset.o sys_regs.o stacktrace.o \
 	 vgic-sys-reg-v3.o fpsimd.o pkvm.o \
 	 arch_timer.o trng.o vmid.o emulate-nested.o nested.o at.o \
 	 vgic/vgic.o vgic/vgic-init.o \
 	 vgic/vgic-irqfd.o vgic/vgic-v2.o \
 	 vgic/vgic-v3.o vgic/vgic-v4.o \
 	 vgic/vgic-mmio.o vgic/vgic-mmio-v2.o \
 	 vgic/vgic-mmio-v3.o vgic/vgic-kvm-device.o \
 	 vgic/vgic-its.o vgic/vgic-debug.o vgic/vgic-v3-nested.o \
-	 vgic/vgic-v5.o
+	 vgic/vgic-v5.o dirty_bit.o
 
 kvm-$(CONFIG_HW_PERF_EVENTS)  += pmu-emul.o pmu.o
 kvm-$(CONFIG_ARM64_PTR_AUTH)  += pauth.o
 kvm-$(CONFIG_PTDUMP_STAGE2_DEBUGFS) += ptdump.o
 
 kvm-$(CONFIG_NVHE_EL2_TRACING) += hyp_trace.o
 
 always-y := hyp_constants.h hyp-constants.s
 
 define rule_gen_hyp_constants
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index 9b85a84f6fd4..a87706c9d160 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -62,20 +62,21 @@ HAS_RASV1P1_EXTN
 HAS_RNG
 HAS_SB
 HAS_STAGE2_FWB
 HAS_TCR2
 HAS_TIDCP1
 HAS_TLB_RANGE
 HAS_VA52
 HAS_VIRT_HOST_EXTN
 HAS_WFXT
 HAS_XNX
+HAS_HDBSS
 HAFT
 HW_DBM
 KVM_HVHE
 KVM_PROTECTED_MODE
 MISMATCHED_CACHE_TYPE
 MPAM
 MPAM_HCR
 MTE
 MTE_ASYMM
 MTE_FAR
-- 
2.55.0

[PATCH v3 02/11] arm64/cpufeature: Add system-wide FEAT_HACDBS detection

[Leonardo Bras]

FEAT_HACDBS will only be used for dirty-bit cleaning if it is detected in
all running cpus.

Signed-off-by: Leonardo Bras <leo.bras@arm.com>
---
 arch/arm64/include/asm/cpufeature.h | 5 +++++
 arch/arm64/kernel/cpufeature.c      | 8 ++++++++
 arch/arm64/tools/cpucaps            | 1 +
 3 files changed, 14 insertions(+)

diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index bdfab086fd94..620ae4cddb76 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -861,20 +861,25 @@ static inline bool system_supports_gcs(void)
 static inline bool system_supports_haft(void)
 {
 	return cpus_have_final_cap(ARM64_HAFT);
 }
 
 static inline bool system_supports_hdbss(void)
 {
 	return cpus_have_final_cap(ARM64_HAS_HDBSS);
 }
 
+static inline bool system_supports_hacdbs(void)
+{
+	return cpus_have_final_cap(ARM64_HACDBS);
+}
+
 static __always_inline bool system_supports_mpam(void)
 {
 	return alternative_has_cap_unlikely(ARM64_MPAM);
 }
 
 static __always_inline bool system_supports_mpam_hcr(void)
 {
 	return alternative_has_cap_unlikely(ARM64_MPAM_HCR);
 }
 
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index aa327eebaf1c..62f56bbd0a65 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -516,20 +516,21 @@ static const struct arm64_ftr_bits ftr_id_aa64mmfr3[] = {
 		       FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64MMFR3_EL1_S1POE_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64MMFR3_EL1_S1PIE_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR3_EL1_SCTLRX_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64MMFR3_EL1_TCRX_SHIFT, 4, 0),
 	ARM64_FTR_END,
 };
 
 static const struct arm64_ftr_bits ftr_id_aa64mmfr4[] = {
 	S_ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR4_EL1_E2H0_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR4_EL1_NV_frac_SHIFT, 4, 0),
+	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR4_EL1_HACDBS_SHIFT, 4, 0),
 	ARM64_FTR_END,
 };
 
 static const struct arm64_ftr_bits ftr_ctr[] = {
 	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_EXACT, 31, 1, 1), /* RES1 */
 	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, CTR_EL0_DIC_SHIFT, 1, 1),
 	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, CTR_EL0_IDC_SHIFT, 1, 1),
 	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_HIGHER_OR_ZERO_SAFE, CTR_EL0_CWG_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_HIGHER_OR_ZERO_SAFE, CTR_EL0_ERG_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, CTR_EL0_DminLine_SHIFT, 4, 1),
@@ -2764,20 +2765,27 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 	{
 		.desc = "Hardware dirty bit management",
 		.type = ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE,
 		.capability = ARM64_HW_DBM,
 		.matches = has_hw_dbm,
 		.cpu_enable = cpu_enable_hw_dbm,
 		.cpus = &dbm_cpus,
 		ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, HAFDBS, DBM)
 	},
 #endif
+	{
+		.desc = "Hardware dirty bit Cleaning",
+		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+		.capability = ARM64_HACDBS,
+		.matches = has_cpuid_feature,
+		ARM64_CPUID_FIELDS(ID_AA64MMFR4_EL1, HACDBS, IMP)
+	},
 #ifdef CONFIG_ARM64_HAFT
 	{
 		.desc = "Hardware managed Access Flag for Table Descriptors",
 		/*
 		 * Contrary to the page/block access flag, the table access flag
 		 * cannot be emulated in software (no access fault will occur).
 		 * Therefore this should be used only if it's supported system
 		 * wide.
 		 */
 		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index a87706c9d160..bd2c0bb98da6 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -65,20 +65,21 @@ HAS_STAGE2_FWB
 HAS_TCR2
 HAS_TIDCP1
 HAS_TLB_RANGE
 HAS_VA52
 HAS_VIRT_HOST_EXTN
 HAS_WFXT
 HAS_XNX
 HAS_HDBSS
 HAFT
 HW_DBM
+HACDBS
 KVM_HVHE
 KVM_PROTECTED_MODE
 MISMATCHED_CACHE_TYPE
 MPAM
 MPAM_HCR
 MTE
 MTE_ASYMM
 MTE_FAR
 MTE_STORE_ONLY
 SME
-- 
2.55.0

[PATCH v3 03/11] arm64/sysreg: Add HACDBS consumer and base registers

[Leonardo Bras]

They will be used on a later commit to make use of the FEAT_HACDBS
mechanism if available.

Signed-off-by: Leonardo Bras <leo.bras@arm.com>
Reviewed-by: Mark Brown <broonie@kernel.org>
---
 arch/arm64/tools/sysreg | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg
index 7cb61aca3797..50ec9751b44d 100644
--- a/arch/arm64/tools/sysreg
+++ b/arch/arm64/tools/sysreg
@@ -4627,20 +4627,50 @@ EndEnum
 Enum	9:8	IRGN0
 	0b00	NC
 	0b01	WBWA
 	0b10	WT
 	0b11	WBnWA
 EndEnum
 Field	7:6	SL0
 Field	5:0	T0SZ
 EndSysreg
 
+Sysreg	HACDBSBR_EL2	3	4	2	3	4
+Res0	63:56
+Field	55:12	BADDR
+Field	11	EN
+Res0	10:4
+UnsignedEnum	3:0	SZ
+	0b0000	4K
+	0b0001	8K
+	0b0010	16K
+	0b0011	32K
+	0b0100	64K
+	0b0101	128K
+	0b0110	256K
+	0b0111	512K
+	0b1000	1M
+	0b1001	2M
+EndEnum
+EndSysreg
+
+Sysreg	HACDBSCONS_EL2	3	4	2	3	5
+UnsignedEnum	63:62	ERR_REASON
+	0b00	NOF
+	0b01	STRUCTF
+	0b10	IPAF
+	0b11	IPAHACF
+EndEnum
+Res0	61:19
+Field	18:0	INDEX
+EndSysreg
+
 Sysreg	GCSCR_EL2	3	4	2	5	0
 Fields	GCSCR_ELx
 EndSysreg
 
 Sysreg	GCSPR_EL2	3	4	2	5	1
 Fields	GCSPR_ELx
 EndSysreg
 
 Sysreg	HDBSSBR_EL2	3	4	2	3	2
 Res0	63:56
-- 
2.55.0

[PATCH v3 04/11] KVM: arm64: dirty_bit: Add base FEAT_HACDBS cleaning routine

[Leonardo Bras]

Introduce the basic cleaning routine that is going to be used for both
dirty-bitmap and dirty-ring routines.

It sets the required registers with the input buffer, and wait for
HACDBS to finish, which means either the task is done, or there was some
error during processing.

It is ran with preemption disabled, as a task being scheduled in could
change the translation registers used by HACDBS and end up corrupting the
current dirty-bit tracking and the sched-in task's S2 pagetables.

Signed-off-by: Leonardo Bras <leo.bras@arm.com>
---
 arch/arm64/kvm/dirty_bit.c | 176 +++++++++++++++++++++++++++++++++++++
 1 file changed, 176 insertions(+)

diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c
index 32fe938d6bf7..33826bbd16d9 100644
--- a/arch/arm64/kvm/dirty_bit.c
+++ b/arch/arm64/kvm/dirty_bit.c
@@ -1,16 +1,192 @@
 // SPDX-License-Identifier: GPL-2.0-only
 /*
  * Copyright (C) 2026 ARM Ltd.
  * Author: Leonardo Bras <leo.bras@arm.com>
  */
 
 #include <asm/kvm_dirty_bit.h>
+#include <asm/kvm_mmu.h>
 
 /* HDBSS entry field definitions */
 #define HDBSS_ENTRY_VALID BIT(0)
 #define HDBSS_ENTRY_TTWL_SHIFT (1)
 #define HDBSS_ENTRY_TTWL_MASK (GENMASK(3, 1))
 #define HDBSS_ENTRY_TTWL(x) \
 	(((x) << HDBSS_ENTRY_TTWL_SHIFT) & HDBSS_ENTRY_TTWL_MASK)
 #define HDBSS_ENTRY_TTWL_RESV HDBSS_ENTRY_TTWL(-4)
 #define HDBSS_ENTRY_IPA GENMASK_ULL(55, 12)
+
+struct hacdbs_save {
+	u64 hacdbsbr_el2;
+	u64 hacdbscons_el2;
+	u64 vttbr_el2;
+	u64 vtcr_el2;
+	bool hcr_vm_set;
+};
+
+struct hacdbs_sched_save {
+	struct preempt_notifier notifier;
+	struct hacdbs_save save;
+};
+
+static int hacdbs_last_entry(int size)
+{
+	u64 cons = read_sysreg_s(SYS_HACDBSCONS_EL2);
+	int index = FIELD_GET(HACDBSCONS_EL2_INDEX, cons);
+
+	switch (FIELD_GET(HACDBSCONS_EL2_ERR_REASON, cons)) {
+	case HACDBSCONS_EL2_ERR_REASON_NOF:
+		return size;
+	case HACDBSCONS_EL2_ERR_REASON_IPAHACF:
+		/* When size not a power of two >= 4k, exit with reserved TTLW */
+		if (index >= size)
+			return size;
+
+		fallthrough;
+	default:
+		/* In case of error, INDEX should point the faulty entry */
+		return index;
+	}
+}
+
+static void hacdbs_start(u64 *hw_entries, int size)
+{
+	u64 br;
+	/* Each entry is 8 bytes */
+	int size_b = size * sizeof(hw_entries[0]);
+	int size_p2 = max(roundup_pow_of_two(size_b), PAGE_SIZE);
+
+	/* If not using the full size of the array, put a stop entry at the end */
+	if (size_b < size_p2)
+		hw_entries[size] = HDBSS_ENTRY_VALID | HDBSS_ENTRY_TTWL_RESV;
+
+	sysreg_clear_set_s(SYS_HACDBSCONS_EL2,
+			   HACDBSCONS_EL2_ERR_REASON | HACDBSCONS_EL2_INDEX, 0);
+
+	br = (virt_to_phys(hw_entries) & HACDBSBR_EL2_BADDR_MASK) |
+	     FIELD_PREP(HACDBSBR_EL2_SZ, ilog2(size_p2) - 12) |
+	     FIELD_PREP(HACDBSBR_EL2_EN, 1);
+
+	/* All writes to the array have to complete before starting */
+	dsb(ishst);
+
+	write_sysreg_s(br, SYS_HACDBSBR_EL2);
+	isb();
+}
+
+static int hacdbs_stop(int size)
+{
+	int idx;
+
+	write_sysreg_s(0, SYS_HACDBSBR_EL2);
+	isb();
+
+	idx = hacdbs_last_entry(size);
+
+	return idx;
+}
+
+static void hacdbs_sched_in(struct preempt_notifier *notifier, int cpu)
+{
+	struct hacdbs_sched_save *save = container_of(notifier,
+						      struct hacdbs_sched_save,
+						      notifier);
+
+	write_sysreg_s(save->save.vtcr_el2, SYS_VTCR_EL2);
+	write_sysreg_s(save->save.vttbr_el2, SYS_VTTBR_EL2);
+	isb();
+
+	if (save->save.hcr_vm_set) {
+		sysreg_clear_set_hcr(0, HCR_EL2_VM);
+		isb();
+	}
+
+	write_sysreg_s(save->save.hacdbscons_el2, SYS_HACDBSCONS_EL2);
+	write_sysreg_s(save->save.hacdbsbr_el2, SYS_HACDBSBR_EL2);
+	isb();
+}
+
+static void hacdbs_sched_out(struct preempt_notifier *notifier,
+			     struct task_struct *next)
+{
+	struct hacdbs_sched_save *save = container_of(notifier,
+						      struct hacdbs_sched_save,
+						      notifier);
+
+	if (read_sysreg_s(SYS_HACDBSBR_EL2) & HACDBSBR_EL2_EN) {
+		save->save.hacdbsbr_el2 = HACDBSBR_EL2_EN;
+		sysreg_clear_set_s(SYS_HACDBSBR_EL2, HACDBSBR_EL2_EN, 0);
+		isb();
+	} else {
+		save->save.hacdbsbr_el2 = 0;
+	}
+
+	save->save.hacdbscons_el2 = read_sysreg_s(SYS_HACDBSCONS_EL2);
+	save->save.hacdbsbr_el2 |= read_sysreg_s(SYS_HACDBSBR_EL2);
+	save->save.vttbr_el2 = read_sysreg_s(SYS_VTTBR_EL2);
+	save->save.vtcr_el2 = read_sysreg_s(SYS_VTCR_EL2);
+
+	if (read_sysreg_s(SYS_HCR_EL2) & HCR_EL2_VM) {
+		sysreg_clear_set_hcr(HCR_EL2_VM, 0);
+		isb();
+		save->save.hcr_vm_set = true;
+	} else {
+		save->save.hcr_vm_set = false;
+	}
+}
+
+static struct preempt_ops hacdbs_preempt_ops = {
+	.sched_in = hacdbs_sched_in,
+	.sched_out = hacdbs_sched_out,
+};
+
+/*
+ * Clears dirty-bits for an array of pages (hw_entries) using HACDBS
+ * Returns the number of items cleaned from the array. If returns value < size,
+ *	there was an error in the processing.
+ */
+static int dirty_bit_clear(struct kvm *kvm, u64 *hw_entries, int size)
+{
+	int ret;
+	u64 cons;
+	struct hacdbs_sched_save save;
+
+	preempt_notifier_init(&save.notifier, &hacdbs_preempt_ops);
+	preempt_disable();
+	preempt_notifier_register(&save.notifier);
+	preempt_enable();
+
+	__load_stage2(&kvm->arch.mmu);
+	sysreg_clear_set_hcr(0, HCR_EL2_VM);
+	isb();
+
+	hacdbs_start(hw_entries, size);
+
+	do {
+		cons = read_sysreg_s(SYS_HACDBSCONS_EL2);
+		if (FIELD_GET(HACDBSCONS_EL2_ERR_REASON, cons))
+			break;
+
+		if (FIELD_GET(HACDBSCONS_EL2_INDEX, cons) >= size)
+			break;
+
+		udelay(1);
+	} while (true);
+
+	ret = hacdbs_stop(size);
+
+	sysreg_clear_set_hcr(HCR_EL2_VM, 0);
+	isb();
+
+	/*
+	 * No DSB is needed here, as kvm_flush_remote_tlbs_memslot() that happens
+	 * later in generic dirty-cleaning code already performs a DSB before
+	 * doing the TLBI.
+	 */
+
+	preempt_disable();
+	preempt_notifier_unregister(&save.notifier);
+	preempt_enable();
+
+	return ret;
+}
-- 
2.55.0

[PATCH v3 05/11] kvm: Add arch-generic interface for hw-accelerated dirty-bitmap cleaning

[Leonardo Bras]

Introduce kvm_arch_dirty_log_clear() that allow implementation of
arch-specific hardware-accelerated dirty-log routines.

A call to that is added on both kvm_get_dirty_log_protect() and
kvm_clear_dirty_log_protect() and will fall back to software version if
not implemented, or any error was detected in the arch-specific routine.

For an arch to implement this function, it's required to provide an
asm/kvm_dirty_bit.h and have CONFIG_HAVE_KVM_HW_DIRTY_BIT=y on building.

If the arch does not implement it, and thus lack above config, the
introduced snippet is expected to be compiled-out and have zero impact at
runtime.

Signed-off-by: Leonardo Bras <leo.bras@arm.com>
---
 include/linux/kvm_dirty_bit.h | 27 +++++++++++++++++++++++++++
 virt/kvm/kvm_main.c           | 13 ++++++++++++-
 virt/kvm/Kconfig              |  3 +++
 3 files changed, 42 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/kvm_dirty_bit.h

diff --git a/include/linux/kvm_dirty_bit.h b/include/linux/kvm_dirty_bit.h
new file mode 100644
index 000000000000..fa4f6b67b623
--- /dev/null
+++ b/include/linux/kvm_dirty_bit.h
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Ltd.
+ * Author: Leonardo Bras <leo.bras@arm.com>
+ */
+
+#ifndef __KVM_DIRTY_BIT_H__
+#define __KVM_DIRTY_BIT_H__
+
+#ifndef CONFIG_HAVE_KVM_HW_DIRTY_BIT
+
+static inline int kvm_arch_dirty_log_clear(struct kvm *kvm,
+					   struct kvm_memory_slot *memslot,
+					   struct kvm_clear_dirty_log *log,
+					   unsigned long *bitmap,
+					   bool *flush)
+{
+	return -ENXIO;
+}
+
+#else /* CONFIG_HAVE_KVM_HW_DIRTY_BIT */
+
+#include <asm/kvm_dirty_bit.h>
+
+#endif /* CONFIG_HAVE_KVM_HW_DIRTY_BIT */
+
+#endif /* __KVM_DIRTY_BIT_H__ */
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 45e784462ec6..7a13ef69b6e3 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -58,20 +58,21 @@
 #include "async_pf.h"
 #include "kvm_mm.h"
 #include "vfio.h"
 
 #include <trace/events/ipi.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/kvm.h>
 
 #include <linux/kvm_dirty_ring.h>
+#include <linux/kvm_dirty_bit.h>
 
 
 /* Worst case buffer size needed for holding an integer. */
 #define ITOA_MAX_LEN 12
 
 MODULE_AUTHOR("Qumranet");
 MODULE_DESCRIPTION("Kernel-based Virtual Machine (KVM) Hypervisor");
 MODULE_LICENSE("GPL");
 
 /* Architectures should define their poll value according to the halt latency */
@@ -2255,39 +2256,44 @@ static int kvm_get_dirty_log_protect(struct kvm *kvm, struct kvm_dirty_log *log)
 		 * is some code duplication between this function and
 		 * kvm_get_dirty_log, but hopefully all architecture
 		 * transition to kvm_get_dirty_log_protect and kvm_get_dirty_log
 		 * can be eliminated.
 		 */
 		dirty_bitmap_buffer = dirty_bitmap;
 	} else {
 		dirty_bitmap_buffer = kvm_second_dirty_bitmap(memslot);
 		memset(dirty_bitmap_buffer, 0, n);
 
+		if (kvm_arch_dirty_log_clear(kvm, memslot, NULL,
+					     dirty_bitmap_buffer, &flush) >= 0)
+			goto out;
+
 		KVM_MMU_LOCK(kvm);
 		for (i = 0; i < n / sizeof(long); i++) {
 			unsigned long mask;
 			gfn_t offset;
 
 			if (!dirty_bitmap[i])
 				continue;
 
 			flush = true;
 			mask = xchg(&dirty_bitmap[i], 0);
 			dirty_bitmap_buffer[i] = mask;
 
 			offset = i * BITS_PER_LONG;
 			kvm_arch_mmu_enable_log_dirty_pt_masked(kvm, memslot,
 								offset, mask);
 		}
 		KVM_MMU_UNLOCK(kvm);
 	}
 
+out:
 	if (flush)
 		kvm_flush_remote_tlbs_memslot(kvm, memslot);
 
 	if (copy_to_user(log->dirty_bitmap, dirty_bitmap_buffer, n))
 		return -EFAULT;
 	return 0;
 }
 
 
 /**
@@ -2366,45 +2372,50 @@ static int kvm_clear_dirty_log_protect(struct kvm *kvm,
 	    (log->num_pages < memslot->npages - log->first_page && (log->num_pages & 63)))
 	    return -EINVAL;
 
 	kvm_arch_sync_dirty_log(kvm, memslot);
 
 	flush = false;
 	dirty_bitmap_buffer = kvm_second_dirty_bitmap(memslot);
 	if (copy_from_user(dirty_bitmap_buffer, log->dirty_bitmap, n))
 		return -EFAULT;
 
+	if (kvm_arch_dirty_log_clear(kvm, memslot, log, dirty_bitmap_buffer,
+				     &flush) >= 0)
+		goto out;
+
 	KVM_MMU_LOCK(kvm);
 	for (offset = log->first_page, i = offset / BITS_PER_LONG,
 		 n = DIV_ROUND_UP(log->num_pages, BITS_PER_LONG); n--;
 	     i++, offset += BITS_PER_LONG) {
 		unsigned long mask = *dirty_bitmap_buffer++;
 		atomic_long_t *p = (atomic_long_t *) &dirty_bitmap[i];
 		if (!mask)
 			continue;
 
 		mask &= atomic_long_fetch_andnot(mask, p);
 
 		/*
 		 * mask contains the bits that really have been cleared.  This
 		 * never includes any bits beyond the length of the memslot (if
 		 * the length is not aligned to 64 pages), therefore it is not
 		 * a problem if userspace sets them in log->dirty_bitmap.
 		*/
 		if (mask) {
 			flush = true;
+
 			kvm_arch_mmu_enable_log_dirty_pt_masked(kvm, memslot,
 								offset, mask);
 		}
 	}
 	KVM_MMU_UNLOCK(kvm);
-
+out:
 	if (flush)
 		kvm_flush_remote_tlbs_memslot(kvm, memslot);
 
 	return 0;
 }
 
 static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
 					struct kvm_clear_dirty_log *log)
 {
 	int r;
diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig
index 794976b88c6f..f8757b5b84b3 100644
--- a/virt/kvm/Kconfig
+++ b/virt/kvm/Kconfig
@@ -13,20 +13,23 @@ config HAVE_KVM_PFNCACHE
 
 config HAVE_KVM_IRQCHIP
        bool
 
 config HAVE_KVM_IRQ_ROUTING
        bool
 
 config HAVE_KVM_DIRTY_RING
        bool
 
+config HAVE_KVM_HW_DIRTY_BIT
+       bool
+
 # Only strongly ordered architectures can select this, as it doesn't
 # put any explicit constraint on userspace ordering. They can also
 # select the _ACQ_REL version.
 config HAVE_KVM_DIRTY_RING_TSO
        bool
        select HAVE_KVM_DIRTY_RING
        depends on X86
 
 # Weakly ordered architectures can only select this, advertising
 # to userspace the additional ordering requirements.
-- 
2.55.0

[PATCH v3 06/11] KVM: arm64: Add hardware-accelerated dirty-bitmap cleaning routine

[Leonardo Bras]

Implement arm64 version of kvm_arch_dirty_log_clear() making use of
FEAT_HACDBS.

It works by transversing the dirty-bitmap and converting the set bits into
HDBSS entries in a 64-page blocks granularity.

The resulting HDBSS array is then fed to the HACDBS mechanism that walks
the pagetable marking writable-dirty pages as writable-clean.

In case of error, rewrite all unprocessed entries, including the faulting
one, to the dirty-bitmap and fall back to generic software cleaning.

In case of the options to "manual protect + init set" are enabled, do
the hugepage splitting in the same fashion as the generic software
cleaning, i.e. in 64-page blocks. For that, remove the static qualifier
from kvm_mmu_split_huge_pages() and make the function available on
kvm_host.h.

Signed-off-by: Leonardo Bras <leo.bras@arm.com>
---
 arch/arm64/include/asm/kvm_dirty_bit.h |  24 ++++
 include/linux/kvm_host.h               |   3 +
 arch/arm64/kvm/dirty_bit.c             | 163 +++++++++++++++++++++++++
 arch/arm64/kvm/mmu.c                   |   4 +-
 4 files changed, 192 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_dirty_bit.h b/arch/arm64/include/asm/kvm_dirty_bit.h
index dd16438f0651..d368a23a84b3 100644
--- a/arch/arm64/include/asm/kvm_dirty_bit.h
+++ b/arch/arm64/include/asm/kvm_dirty_bit.h
@@ -2,11 +2,35 @@
 /*
  * Copyright (C) 2026 ARM Ltd.
  * Author: Leonardo Bras <leo.bras@arm.com>
  */
 
 #ifndef __ARM64_KVM_DIRTY_BIT_H__
 #define __ARM64_KVM_DIRTY_BIT_H__
 
 #include <asm/kvm_pgtable.h>
 
+int __kvm_arch_dirty_log_clear(struct kvm *kvm,
+			       struct kvm_memory_slot *memslot,
+			       struct kvm_clear_dirty_log *log,
+			       unsigned long *bitmap,
+			       bool *flush);
+
+static inline bool kvm_arch_dirty_clear_enabled(struct kvm *kvm)
+{
+	return system_supports_hacdbs() &&  kvm->arch.mmu.pgt &&
+	       (kvm->arch.mmu.pgt->flags & KVM_PGTABLE_S2_DBM);
+}
+
+static inline int kvm_arch_dirty_log_clear(struct kvm *kvm,
+					   struct kvm_memory_slot *memslot,
+					   struct kvm_clear_dirty_log *log,
+					   unsigned long *bitmap,
+					   bool *flush)
+{
+	if (!kvm_arch_dirty_clear_enabled(kvm))
+		return -EPERM;
+
+	return __kvm_arch_dirty_log_clear(kvm, memslot, log, bitmap, flush);
+}
+
 #endif /* __ARM64_KVM_DIRTY_BIT_H__ */
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index ab8cfaec82d3..7ea6ed7ce203 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1662,20 +1662,23 @@ void kvm_arch_disable_virtualization_cpu(void);
 bool kvm_vcpu_has_events(struct kvm_vcpu *vcpu);
 int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu);
 bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu);
 int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu);
 bool kvm_arch_dy_runnable(struct kvm_vcpu *vcpu);
 bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);
 bool kvm_arch_vcpu_preempted_in_kernel(struct kvm_vcpu *vcpu);
 void kvm_arch_pre_destroy_vm(struct kvm *kvm);
 void kvm_arch_create_vm_debugfs(struct kvm *kvm);
 
+int kvm_mmu_split_huge_pages(struct kvm *kvm, phys_addr_t addr,
+			     phys_addr_t end);
+
 #ifndef __KVM_HAVE_ARCH_VM_ALLOC
 /*
  * All architectures that want to use vzalloc currently also
  * need their own kvm_arch_alloc_vm implementation.
  */
 static inline struct kvm *kvm_arch_alloc_vm(void)
 {
 	return kzalloc_obj(struct kvm, GFP_KERNEL_ACCOUNT);
 }
 #endif
diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c
index 33826bbd16d9..5a6266205295 100644
--- a/arch/arm64/kvm/dirty_bit.c
+++ b/arch/arm64/kvm/dirty_bit.c
@@ -183,10 +183,173 @@ static int dirty_bit_clear(struct kvm *kvm, u64 *hw_entries, int size)
 	 * later in generic dirty-cleaning code already performs a DSB before
 	 * doing the TLBI.
 	 */
 
 	preempt_disable();
 	preempt_notifier_unregister(&save.notifier);
 	preempt_enable();
 
 	return ret;
 }
+
+static inline void hdbss_to_bitmap(u64 *hdbss_array, int start, int end,
+				   unsigned long *dirty_bitmap,
+				   unsigned long long offset)
+{
+	u64 w = (gpa_to_gfn(hdbss_array[start]) - offset) / BITS_PER_LONG;
+	u64 mask = 0;
+	int idx = start;
+
+	/*
+	 * Entries should be in order, since they were extracted from
+	 * the dirty-bitmap, so batching the atomic writes is efficient.
+	 */
+
+	do {
+		u64 entry = (gpa_to_gfn(hdbss_array[idx]) - offset);
+
+		if (entry / BITS_PER_LONG == w) {
+			mask |= BIT(entry % BITS_PER_LONG);
+		} else {
+			atomic_long_or(mask, (atomic_long_t *)&dirty_bitmap[w]);
+			w = entry / BITS_PER_LONG;
+			mask = BIT(entry % BITS_PER_LONG);
+		}
+	} while (++idx < end);
+	atomic_long_or(mask, (atomic_long_t *)&dirty_bitmap[w]);
+}
+
+static inline int mask_to_hdbss(unsigned long *mask, u64 *hw_entries, const gfn_t offset,
+				u64 ttwl, int idx, int entries_sz)
+{
+	while (idx < entries_sz) {
+		int j = __ffs(*mask);
+		u64 a = gfn_to_gpa(offset + j);
+
+		hw_entries[idx++] = (a & HDBSS_ENTRY_IPA) |
+				    ttwl |
+				    HDBSS_ENTRY_VALID;
+
+		*mask &= ~BIT(j);
+		if (!*mask)
+			break;
+	}
+
+	return idx;
+}
+
+int __kvm_arch_dirty_log_clear(struct kvm *kvm,
+			       struct kvm_memory_slot *memslot,
+			       struct kvm_clear_dirty_log *log,
+			       unsigned long *bitmap,
+			       bool *flush)
+{
+	int ret = 0;
+	int idx = 0;
+	unsigned long *dirty_bitmap = memslot->dirty_bitmap;
+	u64 *hw_entries;
+	const int entries_sz = PAGE_SIZE / sizeof(*hw_entries);
+	u64 ttwl;
+	u64 start, end;
+	gfn_t base_gfn;
+
+	hw_entries = kmalloc_objs(u64, entries_sz, GFP_KERNEL);
+	if (!hw_entries)
+		return -ENOMEM;
+
+	ttwl = HDBSS_ENTRY_TTWL(KVM_PGTABLE_LAST_LEVEL);
+
+	if (log) {
+		start = log->first_page / BITS_PER_LONG;
+		end = start + DIV_ROUND_UP(log->num_pages, BITS_PER_LONG);
+		base_gfn = memslot->base_gfn + log->first_page % BITS_PER_LONG;
+	} else {
+		start = 0;
+		end = kvm_dirty_bitmap_bytes(memslot) / sizeof(long);
+		base_gfn = memslot->base_gfn;
+	}
+
+	write_lock(&kvm->mmu_lock);
+
+	for (unsigned long i = start; i < end; i++) {
+		unsigned long mask;
+		gfn_t offset;
+		atomic_long_t *p;
+
+		if (log) { /* Clean only what is in the input bitmap */
+			mask = bitmap[i - start];
+			if (!mask)
+				continue;
+
+			p = (atomic_long_t *)&dirty_bitmap[i];
+			mask &= atomic_long_fetch_andnot(mask, p);
+		} else { /* Clean everything, save to output bitmap */
+			if (!dirty_bitmap[i])
+				continue;
+
+			mask = xchg(&dirty_bitmap[i], 0);
+			bitmap[i] = mask;
+		}
+
+		if (!mask)
+			continue;
+
+		offset = base_gfn + i * BITS_PER_LONG;
+
+		if (kvm_dirty_log_manual_protect_and_init_set(kvm))
+			kvm_mmu_split_huge_pages(kvm,
+						 gfn_to_gpa(offset + __ffs(mask)),
+						 gfn_to_gpa(offset + __fls(mask) + 1));
+
+		do {
+			idx = mask_to_hdbss(&mask, hw_entries, offset, ttwl, idx, entries_sz);
+			if (idx >= entries_sz) {
+				ret = dirty_bit_clear(kvm, hw_entries, idx);
+				*flush = *flush || ret > 0;
+				if (ret != idx) {
+					/* Restore mask as before processing */
+					atomic_long_or(mask, (atomic_long_t *)&dirty_bitmap[i]);
+					end = i + 1;
+					goto out_err;
+				}
+				idx = 0;
+			}
+		} while (mask);
+	}
+
+	if (idx != 0) {
+		ret = dirty_bit_clear(kvm, hw_entries, idx);
+		*flush = *flush || ret > 0;
+	}
+out_err:
+	if (unlikely(idx > 0 && ret != idx)) {
+		/*
+		 * In case there is an error and not all entries in HACDBS get
+		 * cleaned, we have to mark the dirty bits back in the bitmap,
+		 * as that will be used by the software routine.
+		 */
+
+		if (ret < idx) {
+			if (!log) {
+				/*
+				 * We have to restore everything, cleaned up to
+				 * now, because software routine uses xchg, and
+				 * something may have been dirtied in the meantime
+				 */
+				for (int i = start; i < end; i++)
+					atomic_long_or(bitmap[i],
+						       (atomic_long_t *)&dirty_bitmap[i]);
+			} else {
+				/* Restore only what have not been cleaned yet */
+				hdbss_to_bitmap(hw_entries, ret, idx, dirty_bitmap,
+						memslot->base_gfn);
+			}
+		}
+
+		ret = -EAGAIN;
+	}
+
+	write_unlock(&kvm->mmu_lock);
+	kfree(hw_entries);
+
+	return ret;
+}
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index e086c01a9325..2f9d90c35668 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -110,22 +110,22 @@ static bool need_split_memcache_topup_or_resched(struct kvm *kvm)
 
 	if (need_resched() || rwlock_needbreak(&kvm->mmu_lock))
 		return true;
 
 	chunk_size = kvm->arch.mmu.split_page_chunk_size;
 	min = kvm_mmu_split_nr_page_tables(chunk_size);
 	cache = &kvm->arch.mmu.split_page_cache;
 	return kvm_mmu_memory_cache_nr_free_objects(cache) < min;
 }
 
-static int kvm_mmu_split_huge_pages(struct kvm *kvm, phys_addr_t addr,
-				    phys_addr_t end)
+int kvm_mmu_split_huge_pages(struct kvm *kvm, phys_addr_t addr,
+			     phys_addr_t end)
 {
 	struct kvm_mmu_memory_cache *cache;
 	struct kvm_pgtable *pgt;
 	int ret, cache_capacity;
 	u64 next, chunk_size;
 
 	lockdep_assert_held_write(&kvm->mmu_lock);
 
 	chunk_size = kvm->arch.mmu.split_page_chunk_size;
 	cache_capacity = kvm_mmu_split_nr_page_tables(chunk_size);
-- 
2.55.0

[PATCH v3 07/11] KVM: arm64: Dirty-bitmap: avoid splitting previously split blocks

[Leonardo Bras]

If previous dirty-clean already split a block, then avoid calling the
split helper on that block again.

Signed-off-by: Leonardo Bras <leo.bras@arm.com>
---
 arch/arm64/kvm/dirty_bit.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c
index 5a6266205295..d7e624f9b630 100644
--- a/arch/arm64/kvm/dirty_bit.c
+++ b/arch/arm64/kvm/dirty_bit.c
@@ -244,20 +244,21 @@ int __kvm_arch_dirty_log_clear(struct kvm *kvm,
 			       bool *flush)
 {
 	int ret = 0;
 	int idx = 0;
 	unsigned long *dirty_bitmap = memslot->dirty_bitmap;
 	u64 *hw_entries;
 	const int entries_sz = PAGE_SIZE / sizeof(*hw_entries);
 	u64 ttwl;
 	u64 start, end;
 	gfn_t base_gfn;
+	gpa_t last_split = 0;
 
 	hw_entries = kmalloc_objs(u64, entries_sz, GFP_KERNEL);
 	if (!hw_entries)
 		return -ENOMEM;
 
 	ttwl = HDBSS_ENTRY_TTWL(KVM_PGTABLE_LAST_LEVEL);
 
 	if (log) {
 		start = log->first_page / BITS_PER_LONG;
 		end = start + DIV_ROUND_UP(log->num_pages, BITS_PER_LONG);
@@ -288,24 +289,29 @@ int __kvm_arch_dirty_log_clear(struct kvm *kvm,
 
 			mask = xchg(&dirty_bitmap[i], 0);
 			bitmap[i] = mask;
 		}
 
 		if (!mask)
 			continue;
 
 		offset = base_gfn + i * BITS_PER_LONG;
 
-		if (kvm_dirty_log_manual_protect_and_init_set(kvm))
-			kvm_mmu_split_huge_pages(kvm,
-						 gfn_to_gpa(offset + __ffs(mask)),
-						 gfn_to_gpa(offset + __fls(mask) + 1));
+		if (kvm_dirty_log_manual_protect_and_init_set(kvm)) {
+			gpa_t split_start = gfn_to_gpa(offset + __ffs(mask));
+			gpa_t split_end =  gfn_to_gpa(offset + __fls(mask) + 1);
+
+			if (split_end > last_split) {
+				kvm_mmu_split_huge_pages(kvm, split_start, split_end);
+				last_split = ALIGN_DOWN(split_end - 1, PMD_SIZE) + PMD_SIZE;
+			}
+		}
 
 		do {
 			idx = mask_to_hdbss(&mask, hw_entries, offset, ttwl, idx, entries_sz);
 			if (idx >= entries_sz) {
 				ret = dirty_bit_clear(kvm, hw_entries, idx);
 				*flush = *flush || ret > 0;
 				if (ret != idx) {
 					/* Restore mask as before processing */
 					atomic_long_or(mask, (atomic_long_t *)&dirty_bitmap[i]);
 					end = i + 1;
-- 
2.55.0

[PATCH v3 08/11] kvm/dirty_ring: Introduce get_memslot and move helpers to header

[Leonardo Bras]

Dirty-ring entry struct carries a slot number which is used to return
a struct memslot*. That struct carry important information such as
memory offset of that memory slot, which is used to calculate the guest
physical address of the page which needs to be marked clean.

In order to fetch that memslot information without duplicating code, split
that part of kvm_reset_dirty_gfn() into a new helper
kvm_dirty_ring_get_memslot().

Along with that, make it available on kvm_dirty_ring.h other helpers such
as kvm_dirty_gfn_harvested() and kvm_dirty_gfn_set_invalid() which will be
useful on implementing arch specific dirty-ring cleaning accelerators.

Signed-off-by: Leonardo Bras <leo.bras@arm.com>
---
 include/linux/kvm_dirty_ring.h | 12 ++++++++++++
 virt/kvm/dirty_ring.c          | 30 ++++++++++++++++--------------
 2 files changed, 28 insertions(+), 14 deletions(-)

diff --git a/include/linux/kvm_dirty_ring.h b/include/linux/kvm_dirty_ring.h
index eb10d87adf7d..190d97fce4a4 100644
--- a/include/linux/kvm_dirty_ring.h
+++ b/include/linux/kvm_dirty_ring.h
@@ -77,18 +77,30 @@ bool kvm_use_dirty_bitmap(struct kvm *kvm);
 bool kvm_arch_allow_write_without_running_vcpu(struct kvm *kvm);
 u32 kvm_dirty_ring_get_rsvd_entries(struct kvm *kvm);
 int kvm_dirty_ring_alloc(struct kvm *kvm, struct kvm_dirty_ring *ring,
 			 int index, u32 size);
 int kvm_dirty_ring_reset(struct kvm *kvm, struct kvm_dirty_ring *ring,
 			 int *nr_entries_reset);
 void kvm_dirty_ring_push(struct kvm_vcpu *vcpu, u32 slot, u64 offset);
 
 bool kvm_dirty_ring_check_request(struct kvm_vcpu *vcpu);
 
+static inline bool kvm_dirty_gfn_harvested(struct kvm_dirty_gfn *gfn)
+{
+	return smp_load_acquire(&gfn->flags) & KVM_DIRTY_GFN_F_RESET;
+}
+
+static inline void kvm_dirty_gfn_set_invalid(struct kvm_dirty_gfn *gfn)
+{
+	smp_store_release(&gfn->flags, 0);
+}
+
+struct kvm_memory_slot *kvm_dirty_ring_get_memslot(struct kvm *kvm, u32 slot);
+
 /* for use in vm_operations_struct */
 struct page *kvm_dirty_ring_get_page(struct kvm_dirty_ring *ring, u32 offset);
 
 void kvm_dirty_ring_free(struct kvm_dirty_ring *ring);
 
 #endif /* CONFIG_HAVE_KVM_DIRTY_RING */
 
 #endif	/* KVM_DIRTY_RING_H */
diff --git a/virt/kvm/dirty_ring.c b/virt/kvm/dirty_ring.c
index 572b854edf74..807c45fb1fc7 100644
--- a/virt/kvm/dirty_ring.c
+++ b/virt/kvm/dirty_ring.c
@@ -43,32 +43,44 @@ static u32 kvm_dirty_ring_used(struct kvm_dirty_ring *ring)
 static bool kvm_dirty_ring_soft_full(struct kvm_dirty_ring *ring)
 {
 	return kvm_dirty_ring_used(ring) >= ring->soft_limit;
 }
 
 static bool kvm_dirty_ring_full(struct kvm_dirty_ring *ring)
 {
 	return kvm_dirty_ring_used(ring) >= ring->size;
 }
 
-static void kvm_reset_dirty_gfn(struct kvm *kvm, u32 slot, u64 offset, u64 mask)
+static inline struct kvm_memory_slot *
+__kvm_dirty_ring_get_memslot(struct kvm *kvm, u32 slot)
 {
-	struct kvm_memory_slot *memslot;
 	int as_id, id;
 
 	as_id = slot >> 16;
 	id = (u16)slot;
 
 	if (as_id >= kvm_arch_nr_memslot_as_ids(kvm) || id >= KVM_USER_MEM_SLOTS)
-		return;
+		return NULL;
 
-	memslot = id_to_memslot(__kvm_memslots(kvm, as_id), id);
+	return id_to_memslot(__kvm_memslots(kvm, as_id), id);
+}
+
+struct kvm_memory_slot *kvm_dirty_ring_get_memslot(struct kvm *kvm, u32 slot)
+{
+	return __kvm_dirty_ring_get_memslot(kvm, slot);
+}
+
+static void kvm_reset_dirty_gfn(struct kvm *kvm, u32 slot, u64 offset, u64 mask)
+{
+	struct kvm_memory_slot *memslot;
+
+	memslot = __kvm_dirty_ring_get_memslot(kvm, slot);
 
 	if (!memslot || offset >= memslot->npages ||
 	    offset + __fls(mask) >= memslot->npages)
 		return;
 
 	KVM_MMU_LOCK(kvm);
 	kvm_arch_mmu_enable_log_dirty_pt_masked(kvm, memslot, offset, mask);
 	KVM_MMU_UNLOCK(kvm);
 }
 
@@ -81,35 +93,25 @@ int kvm_dirty_ring_alloc(struct kvm *kvm, struct kvm_dirty_ring *ring,
 
 	ring->size = size / sizeof(struct kvm_dirty_gfn);
 	ring->soft_limit = ring->size - kvm_dirty_ring_get_rsvd_entries(kvm);
 	ring->dirty_index = 0;
 	ring->reset_index = 0;
 	ring->index = index;
 
 	return 0;
 }
 
-static inline void kvm_dirty_gfn_set_invalid(struct kvm_dirty_gfn *gfn)
-{
-	smp_store_release(&gfn->flags, 0);
-}
-
 static inline void kvm_dirty_gfn_set_dirtied(struct kvm_dirty_gfn *gfn)
 {
 	gfn->flags = KVM_DIRTY_GFN_F_DIRTY;
 }
 
-static inline bool kvm_dirty_gfn_harvested(struct kvm_dirty_gfn *gfn)
-{
-	return smp_load_acquire(&gfn->flags) & KVM_DIRTY_GFN_F_RESET;
-}
-
 int kvm_dirty_ring_reset(struct kvm *kvm, struct kvm_dirty_ring *ring,
 			 int *nr_entries_reset)
 {
 	/*
 	 * To minimize mmu_lock contention, batch resets for harvested entries
 	 * whose gfns are in the same slot, and are within N frame numbers of
 	 * each other, where N is the number of bits in an unsigned long.  For
 	 * simplicity, process the current set of entries when the next entry
 	 * can't be included in the batch.
 	 *
-- 
2.55.0

[PATCH v3 09/11] kvm/dirty_ring: Add arch-generic interface for hw-accelerated dirty-ring cleaning

[Leonardo Bras]

Introduce kvm_arch_dirty_ring_clear() that allow implementation of
arch-specific hardware-accelerated dirty-ring routines.

A call to that is added on kvm_dirty_ring_reset() and will fall back to
software version if not implemented, or any error was detected in the
arch-specific routine.

For an arch to implement this function, it's required to provide it in a
asm/kvm_dirty_bit.h and have CONFIG_HAVE_KVM_HW_DIRTY_BIT=y on building.

If the arch does not implement it, and thus lack above config, the
introduced snippet is expected to be compiled-out and have zero impact at
runtime.

Signed-off-by: Leonardo Bras <leo.bras@arm.com>
---
 include/linux/kvm_dirty_bit.h | 7 +++++++
 virt/kvm/dirty_ring.c         | 5 +++++
 2 files changed, 12 insertions(+)

diff --git a/include/linux/kvm_dirty_bit.h b/include/linux/kvm_dirty_bit.h
index fa4f6b67b623..8492979d694e 100644
--- a/include/linux/kvm_dirty_bit.h
+++ b/include/linux/kvm_dirty_bit.h
@@ -11,17 +11,24 @@
 
 static inline int kvm_arch_dirty_log_clear(struct kvm *kvm,
 					   struct kvm_memory_slot *memslot,
 					   struct kvm_clear_dirty_log *log,
 					   unsigned long *bitmap,
 					   bool *flush)
 {
 	return -ENXIO;
 }
 
+static inline int kvm_arch_dirty_ring_clear(struct kvm *kvm,
+					    struct kvm_dirty_ring *ring,
+					    int *nr_entries_reset)
+{
+	return -ENXIO;
+}
+
 #else /* CONFIG_HAVE_KVM_HW_DIRTY_BIT */
 
 #include <asm/kvm_dirty_bit.h>
 
 #endif /* CONFIG_HAVE_KVM_HW_DIRTY_BIT */
 
 #endif /* __KVM_DIRTY_BIT_H__ */
diff --git a/virt/kvm/dirty_ring.c b/virt/kvm/dirty_ring.c
index 807c45fb1fc7..3fa62894f3d2 100644
--- a/virt/kvm/dirty_ring.c
+++ b/virt/kvm/dirty_ring.c
@@ -1,20 +1,21 @@
 // SPDX-License-Identifier: GPL-2.0-only
 /*
  * KVM dirty ring implementation
  *
  * Copyright 2019 Red Hat, Inc.
  */
 #include <linux/kvm_host.h>
 #include <linux/kvm.h>
 #include <linux/vmalloc.h>
 #include <linux/kvm_dirty_ring.h>
+#include <linux/kvm_dirty_bit.h>
 #include <trace/events/kvm.h>
 #include "kvm_mm.h"
 
 int __weak kvm_cpu_dirty_log_size(struct kvm *kvm)
 {
 	return 0;
 }
 
 u32 kvm_dirty_ring_get_rsvd_entries(struct kvm *kvm)
 {
@@ -126,20 +127,23 @@ int kvm_dirty_ring_reset(struct kvm *kvm, struct kvm_dirty_ring *ring,
 	struct kvm_dirty_gfn *entry;
 
 	/*
 	 * Ensure concurrent calls to KVM_RESET_DIRTY_RINGS are serialized,
 	 * e.g. so that KVM fully resets all entries processed by a given call
 	 * before returning to userspace.  Holding slots_lock also protects
 	 * the various memslot accesses.
 	 */
 	lockdep_assert_held(&kvm->slots_lock);
 
+	if (kvm_arch_dirty_ring_clear(kvm, ring, nr_entries_reset) >= 0)
+		goto out;
+
 	while (likely((*nr_entries_reset) < INT_MAX)) {
 		if (signal_pending(current))
 			return -EINTR;
 
 		entry = &ring->dirty_gfns[ring->reset_index & (ring->size - 1)];
 
 		if (!kvm_dirty_gfn_harvested(entry))
 			break;
 
 		next_slot = READ_ONCE(entry->slot);
@@ -206,20 +210,21 @@ int kvm_dirty_ring_reset(struct kvm *kvm, struct kvm_dirty_ring *ring,
 	 * always be left pending.
 	 */
 	if (mask)
 		kvm_reset_dirty_gfn(kvm, cur_slot, cur_offset, mask);
 
 	/*
 	 * The request KVM_REQ_DIRTY_RING_SOFT_FULL will be cleared
 	 * by the VCPU thread next time when it enters the guest.
 	 */
 
+out:
 	trace_kvm_dirty_ring_reset(ring);
 
 	return 0;
 }
 
 void kvm_dirty_ring_push(struct kvm_vcpu *vcpu, u32 slot, u64 offset)
 {
 	struct kvm_dirty_ring *ring = &vcpu->dirty_ring;
 	struct kvm_dirty_gfn *entry;
 
-- 
2.55.0

[PATCH v3 10/11] KVM: arm64: Add hardware-accelerated dirty-ring cleaning routine

[Leonardo Bras]

Implement arm64 version of kvm_arch_dirty_ring_clear() making use of
FEAT_HACDBS.

It works by transversing the dirty-ring and converting its entries into
HDBSS entries based on the slot offset.

The resulting HDBSS array is then fed to the HACDBS mechanism that walks
the pagetable marking writable-dirty pages as writable-clean.

Only successfully cleaned entries are set as invalid on the dirty-ring, so
in case of error, falling back to generic software cleaning will take care
of any remaining entry in the dirty-ring.

Signed-off-by: Leonardo Bras <leo.bras@arm.com>
---
 arch/arm64/include/asm/kvm_dirty_bit.h | 13 +++++
 arch/arm64/kvm/dirty_bit.c             | 77 ++++++++++++++++++++++++++
 2 files changed, 90 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_dirty_bit.h b/arch/arm64/include/asm/kvm_dirty_bit.h
index d368a23a84b3..f785a7aad9e8 100644
--- a/arch/arm64/include/asm/kvm_dirty_bit.h
+++ b/arch/arm64/include/asm/kvm_dirty_bit.h
@@ -8,29 +8,42 @@
 #define __ARM64_KVM_DIRTY_BIT_H__
 
 #include <asm/kvm_pgtable.h>
 
 int __kvm_arch_dirty_log_clear(struct kvm *kvm,
 			       struct kvm_memory_slot *memslot,
 			       struct kvm_clear_dirty_log *log,
 			       unsigned long *bitmap,
 			       bool *flush);
 
+int __kvm_arch_dirty_ring_clear(struct kvm *kvm, struct kvm_dirty_ring *ring,
+				int *nr_entries_reset);
+
 static inline bool kvm_arch_dirty_clear_enabled(struct kvm *kvm)
 {
 	return system_supports_hacdbs() &&  kvm->arch.mmu.pgt &&
 	       (kvm->arch.mmu.pgt->flags & KVM_PGTABLE_S2_DBM);
 }
 
 static inline int kvm_arch_dirty_log_clear(struct kvm *kvm,
 					   struct kvm_memory_slot *memslot,
 					   struct kvm_clear_dirty_log *log,
 					   unsigned long *bitmap,
 					   bool *flush)
 {
 	if (!kvm_arch_dirty_clear_enabled(kvm))
 		return -EPERM;
 
 	return __kvm_arch_dirty_log_clear(kvm, memslot, log, bitmap, flush);
 }
 
+static inline int kvm_arch_dirty_ring_clear(struct kvm *kvm,
+					    struct kvm_dirty_ring *ring,
+					    int *nr_entries_reset)
+{
+	if (!kvm_arch_dirty_clear_enabled(kvm))
+		return -EPERM;
+
+	return __kvm_arch_dirty_ring_clear(kvm, ring, nr_entries_reset);
+}
+
 #endif /* __ARM64_KVM_DIRTY_BIT_H__ */
diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c
index d7e624f9b630..ca9cfefe0893 100644
--- a/arch/arm64/kvm/dirty_bit.c
+++ b/arch/arm64/kvm/dirty_bit.c
@@ -352,10 +352,87 @@ int __kvm_arch_dirty_log_clear(struct kvm *kvm,
 		}
 
 		ret = -EAGAIN;
 	}
 
 	write_unlock(&kvm->mmu_lock);
 	kfree(hw_entries);
 
 	return ret;
 }
+
+int __kvm_arch_dirty_ring_clear(struct kvm *kvm, struct kvm_dirty_ring *ring,
+				int *nr_entries_reset)
+{
+	u64 *hw_entries __free(kfree) = NULL;
+	u64 ttwl;
+	s64 cur_slot = S64_MAX;
+	int i, ret;
+	struct kvm_memory_slot *memslot;
+
+	if (signal_pending(current))
+		return -EINTR;
+
+	ttwl = HDBSS_ENTRY_TTWL(KVM_PGTABLE_LAST_LEVEL);
+
+	hw_entries = kmalloc(max(ring->size * sizeof(u64), PAGE_SIZE), GFP_KERNEL);
+	if (!hw_entries)
+		return -ENOMEM;
+
+	for (i = 0; i < ring->size; i++) {
+		struct kvm_dirty_gfn *entry;
+		gfn_t gfn;
+		u32 entry_slot;
+		u64 entry_offset;
+
+		entry = &ring->dirty_gfns[(ring->reset_index + i) &
+					  (ring->size - 1)];
+
+		if (!kvm_dirty_gfn_harvested(entry))
+			break;
+
+		entry_slot = READ_ONCE(entry->slot);
+		entry_offset = READ_ONCE(entry->offset);
+
+		if (entry_slot != cur_slot) {
+			memslot = kvm_dirty_ring_get_memslot(kvm, entry_slot);
+			if (!memslot)
+				return -EFAULT;
+
+			cur_slot = entry_slot;
+		}
+
+		if (entry_offset >= memslot->npages)
+			return -EFAULT;
+
+		gfn = memslot->base_gfn + entry_offset;
+
+		hw_entries[i] = (gfn_to_gpa(gfn) & HDBSS_ENTRY_IPA) |
+				  ttwl | HDBSS_ENTRY_VALID;
+	}
+
+	if (i == 0)
+		return 0;
+
+	write_lock(&kvm->mmu_lock);
+	ret = dirty_bit_clear(kvm, hw_entries, i);
+	write_unlock(&kvm->mmu_lock);
+
+	/* Set as invalid all successfully cleaned entries */
+	for (int j = 0; j < ret; j++) {
+		struct kvm_dirty_gfn *entry;
+
+		entry = &ring->dirty_gfns[(ring->reset_index + j) &
+					  (ring->size - 1)];
+
+		kvm_dirty_gfn_set_invalid(entry);
+	}
+
+	/* In case of error, try software cleaning from the faulting entry */
+	ring->reset_index += ret;
+	*nr_entries_reset += ret;
+
+	if (ret < i)
+		return -EAGAIN;
+
+	return ret;
+}
-- 
2.55.0

[PATCH v3 11/11] KVM: arm64: Enable KVM_HW_DIRTY_BIT

[Leonardo Bras]

Set the corresponding bit to enable hardware accelerated dirty-bitmap and
dirty-ring cleaning for arm64. Actually using acceleration depends on the
cpus enabling FEAT_HACDBS as well as the pre-requisite features for it,
such as FEAT_HDBSS and FEAT_HAFDBS.

Signed-off-by: Leonardo Bras <leo.bras@arm.com>
---
 arch/arm64/kvm/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index 449154f9a485..3264d8ca093e 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -27,20 +27,21 @@ menuconfig KVM
 	select VIRT_XFER_TO_GUEST_WORK
 	select KVM_VFIO
 	select HAVE_KVM_DIRTY_RING_ACQ_REL
 	select NEED_KVM_DIRTY_RING_WITH_BITMAP
 	select HAVE_KVM_MSI
 	select HAVE_KVM_IRQCHIP
 	select HAVE_KVM_IRQ_ROUTING
 	select HAVE_KVM_IRQ_BYPASS
 	select HAVE_KVM_READONLY_MEM
 	select HAVE_KVM_VCPU_RUN_PID_CHANGE
+	select HAVE_KVM_HW_DIRTY_BIT
 	select SCHED_INFO
 	select GUEST_PERF_EVENTS if PERF_EVENTS
 	select KVM_GUEST_MEMFD
 	help
 	  Support hosting virtualized guest machines.
 
 	  If unsure, say N.
 
 if KVM
 
-- 
2.55.0

Re: [PATCH v3 01/11] KVM: arm64: HDBSS bits

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 6 potential issue(s) to consider:
- [High] The patch is explicitly marked by the author as a test hack that should not be merged upstream.
- [High] Hardware Dirty Bit Management (DBM) is not cleared when write-protecting pages, allowing guests to bypass software write-protection (such as userfaultfd-wp or software dirty logging).
- [Medium] Left shifting a negative signed integer is undefined behavior in C.
- [Medium] The hardcoded split_page_chunk_size is excessively large, causing massive memory allocations under the mmu_lock.
- [Low] HDBSS entry macros are defined in a `.c` file instead of a header file, making them inaccessible to other subsystems.
- [Low] Typo in the parameter name of `has_vhe_hdbss`.
--

--- Patch [1]: [PATCH v3 01/11] KVM: arm64: HDBSS bits ---
commit 125f04d66539bf05e0255567036409a407e48004
Author: Leonardo Bras <leo.bras@arm.com>
Subject: KVM: arm64: HDBSS bits

A brief summary:
This commit introduces preliminary support for the Hardware Dirty state
tracking structure (HDBSS) feature on ARM64. It adds CPU feature detection,
sets the DBM stage-2 page table flag when write protection is relaxed, and
configures eager page splitting chunk sizes for testing purposes.

> I added them here in order to fulfill the dependencies and be able to
> easily build and test this patchset, but this particular patch should *not*
> be merged upstream.

[Severity: High]
Since this is explicitly marked as a test hack meant only for local building
and testing, should this patch be dropped from the series before it gets
merged upstream?

[ ... ]

> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> --- a/arch/arm64/kernel/cpufeature.c
> +++ b/arch/arm64/kernel/cpufeature.c
> @@ -2131,6 +2131,11 @@ static bool hvhe_possible(const struct arm64_cpu_capabilities *entry,
>  	return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_HVHE);
>  }
>  
> +static bool has_vhe_hdbss(const struct arm64_cpu_capabilities *entry, int cope)
                                                                             ^^^^
[Severity: Low]
Is this intended to be named scope? The standard convention for capability
matching functions in cpufeature.c uses scope for this parameter.

[ ... ]

> diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c
> new file mode 100644
> --- /dev/null
> +++ b/arch/arm64/kvm/dirty_bit.c
> @@ -0,0 +1,16 @@
[ ... ]
> +#include <asm/kvm_dirty_bit.h>
> +
> +/* HDBSS entry field definitions */
> +#define HDBSS_ENTRY_VALID BIT(0)
> +#define HDBSS_ENTRY_TTWL_SHIFT (1)
> +#define HDBSS_ENTRY_TTWL_MASK (GENMASK(3, 1))
> +#define HDBSS_ENTRY_TTWL(x) \
> +	(((x) << HDBSS_ENTRY_TTWL_SHIFT) & HDBSS_ENTRY_TTWL_MASK)
> +#define HDBSS_ENTRY_TTWL_RESV HDBSS_ENTRY_TTWL(-4)

[Severity: Low]
Because these HDBSS entry macros are defined directly in the .c file rather
than in the newly created header arch/arm64/include/asm/kvm_dirty_bit.h, they
are inaccessible to other subsystems. Should they be moved to the header file
so dependent patches can use them?

[Severity: Medium]
Will HDBSS_ENTRY_TTWL_RESV trigger undefined behavior here? The macro expands
to ((-4) << 1), and left shifting a negative signed integer is undefined
behavior in C (prior to C23), which can lead to shift-out-of-bounds splats
at runtime.

> +#define HDBSS_ENTRY_IPA GENMASK_ULL(55, 12)

[ ... ]

> diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
> --- a/arch/arm64/kvm/hyp/pgtable.c
> +++ b/arch/arm64/kvm/hyp/pgtable.c
> @@ -1367,9 +1371,13 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
>  	if (prot & KVM_PGTABLE_PROT_R)
>  		set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R;
>  
> -	if (prot & KVM_PGTABLE_PROT_W)
> +	if (prot & KVM_PGTABLE_PROT_W) {
>  		set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W;
>  
> +		if (pgt->flags & KVM_PGTABLE_S2_DBM)
> +			set |= KVM_PTE_LEAF_ATTR_HI_S2_DBM;
> +	}
> +

[Severity: High]
Does kvm_pgtable_stage2_wrprotect() also need to be updated to clear the DBM
bit? 

Currently, kvm_pgtable_stage2_wrprotect() clears the 
KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W bit to trap writes, but if 
KVM_PTE_LEAF_ATTR_HI_S2_DBM is left set, the hardware will autonomously update 
the PTE to allow writes without generating a permission fault. Could this 
allow guests to bypass software write-protection mechanisms like 
userfaultfd-wp or KVM software dirty page logging?

>  	if (prot & KVM_PGTABLE_PROT_X) {
>  		ret = stage2_set_xn_attr(prot, &xn);
>  		if (ret)

[ ... ]

> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c
> @@ -1027,8 +1027,12 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
>  	for_each_possible_cpu(cpu)
>  		*per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1;
>  
> -	 /* The eager page splitting is disabled by default */
> -	mmu->split_page_chunk_size = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT;
> +	 /* The eager page splitting is disabled by default if system has no HDBSS */
> +	if (system_supports_hdbss())
> +		mmu->split_page_chunk_size = 256 * 1024 * PAGE_SIZE;

[Severity: Medium]
Is this hardcoded split_page_chunk_size excessively large? This evaluates to
1GB on 4KB page systems and 16GB on 64KB page systems. When splitting this
chunk, kvm_mmu_topup_memory_cache might attempt to allocate a massive amount
of page tables under the mmu_lock. Could this cause severe latency spikes or
allocation failures under memory pressure?

> +	else
> +		mmu->split_page_chunk_size = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT;
> +
>  	mmu->split_page_cache.gfp_zero = __GFP_ZERO;
>  
>  	mmu->pgd_phys = __pa(pgt->pgd);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729104548.3439958-1-leo.bras@arm.com?part=1

Re: [PATCH v3 04/11] KVM: arm64: dirty_bit: Add base FEAT_HACDBS cleaning routine

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 8 potential issue(s) to consider:
- [Critical] `hacdbs_sched_in` restores a potentially stale VMID from `save.vttbr_el2`, leading to cross-VM memory corruption.
- [High] Direct EL2 system register accesses from host code without a VHE check causes kernel panics on nVHE systems.
- [High] Writes to control-plane system registers are batched and not immediately followed by an `isb()` barrier.
- [High] Missing synchronization barrier (`DSB`) before changing the translation context after disabling the HACDBS asynchronous engine.
- [High] The hardware polling loop in `dirty_bit_clear` lacks a timeout or signal check, risking an unkillable task hang.
- [Medium] Using `virt_to_phys()` on a dynamic buffer array which could be non-contiguous (e.g. allocated with vmalloc).
- [Medium] The commit message explicitly asserts the code runs with preemption disabled, but the implementation explicitly enables preemption.
- [Low] Left-shift of a negative integer in macro definition.
--

--- Patch [4]: [PATCH v3 04/11] KVM: arm64: dirty_bit: Add base FEAT_HACDBS cleaning routine ---
commit b837f416f1871772a3bac546015e6fcac9c7accb
Author: Leonardo Bras <leo.bras@arm.com>

KVM: arm64: dirty_bit: Add base FEAT_HACDBS cleaning routine

Introduce the basic cleaning routine that is going to be used for both
dirty-bitmap and dirty-ring routines. It sets the required registers with
the input buffer, and wait for HACDBS to finish.

> It is ran with preemption disabled, as a task being scheduled in could
> change the translation registers used by HACDBS and end up corrupting the
> current dirty-bit tracking and the sched-in task's S2 pagetables.

[Severity: Medium]
The commit message asserts that the code runs with preemption disabled, but
looking at dirty_bit_clear(), preemption is explicitly enabled immediately
after registering the notifier:

dirty_bit_clear() {
    preempt_disable();
    preempt_notifier_register(&save.notifier);
    preempt_enable();
}

Is this description accurate, or does the implementation contradict the
commit message's synchronization premise?

> diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c
> index 32fe938d6bf70..33826bbd16d91 100644
> --- a/arch/arm64/kvm/dirty_bit.c
> +++ b/arch/arm64/kvm/dirty_bit.c
> @@ -5,6 +5,7 @@
>   */
>  
>  #include <asm/kvm_dirty_bit.h>
> +#include <asm/kvm_mmu.h>
>  
>  /* HDBSS entry field definitions */
>  #define HDBSS_ENTRY_VALID BIT(0)
> @@ -14,3 +15,178 @@
>  	(((x) << HDBSS_ENTRY_TTWL_SHIFT) & HDBSS_ENTRY_TTWL_MASK)
>  #define HDBSS_ENTRY_TTWL_RESV HDBSS_ENTRY_TTWL(-4)

[Severity: Low]
Does this code trigger undefined behavior by left-shifting a negative
signed integer (-4) via the HDBSS_ENTRY_TTWL macro?

[ ... ]

> +static void hacdbs_start(u64 *hw_entries, int size)
> +{
> +	u64 br;
> +	/* Each entry is 8 bytes */
> +	int size_b = size * sizeof(hw_entries[0]);
> +	int size_p2 = max(roundup_pow_of_two(size_b), PAGE_SIZE);
> +
> +	/* If not using the full size of the array, put a stop entry at the end */
> +	if (size_b < size_p2)
> +		hw_entries[size] = HDBSS_ENTRY_VALID | HDBSS_ENTRY_TTWL_RESV;
> +
> +	sysreg_clear_set_s(SYS_HACDBSCONS_EL2,
> +			   HACDBSCONS_EL2_ERR_REASON | HACDBSCONS_EL2_INDEX, 0);

[Severity: High]
Can this sequence leave the CPU executing with architecturally undefined
pipeline state?

In hacdbs_start(), the write to the control-plane system register
SYS_HACDBSCONS_EL2 is followed by pointer arithmetic and a dsb() rather
than an isb() as the very next instruction.

> +
> +	br = (virt_to_phys(hw_entries) & HACDBSBR_EL2_BADDR_MASK) |
> +	     FIELD_PREP(HACDBSBR_EL2_SZ, ilog2(size_p2) - 12) |
> +	     FIELD_PREP(HACDBSBR_EL2_EN, 1);

[Severity: Medium]
Will this return an invalid physical address if hw_entries is allocated via
vmalloc() or kvmalloc()?

Using virt_to_phys() on a dynamic buffer array assumes it resides in the
physically contiguous linear map, which could lead to hardware DMA targeting
the wrong physical memory pages.

> +
> +	/* All writes to the array have to complete before starting */
> +	dsb(ishst);
> +
> +	write_sysreg_s(br, SYS_HACDBSBR_EL2);

[Severity: High]
Does this code cause a kernel panic on nVHE systems?

In hacdbs_start(), directly writing to the EL2 system register
SYS_HACDBSBR_EL2 from the KVM host kernel executing at EL1 without a
has_vhe() guard will trigger an immediate Undefined Instruction trap.

> +	isb();
> +}

[ ... ]

> +static void hacdbs_sched_in(struct preempt_notifier *notifier, int cpu)
> +{
> +	struct hacdbs_sched_save *save = container_of(notifier,
> +						      struct hacdbs_sched_save,
> +						      notifier);
> +
> +	write_sysreg_s(save->save.vtcr_el2, SYS_VTCR_EL2);
> +	write_sysreg_s(save->save.vttbr_el2, SYS_VTTBR_EL2);

[Severity: High]
Could this direct EL2 register access cause a panic on nVHE systems?

Similar to hacdbs_start(), hacdbs_sched_in() accesses SYS_VTCR_EL2 and
SYS_VTTBR_EL2 from EL1 without a VHE check.

[Severity: High]
Is a synchronization barrier missing here?

In hacdbs_sched_in(), the write to SYS_VTCR_EL2 is immediately followed by a
write to SYS_VTTBR_EL2 rather than an isb(), meaning subsequent instructions
may observe undefined pipeline state.

[Severity: Critical]
Can this restore a stale VMID and cause cross-VM memory corruption?

hacdbs_sched_in() restores the cached save.vttbr_el2 (which contains the
VMID). Since this is a generic host thread, if KVM's VMID allocator rolls
over while the task sleeps, the VMID could be reassigned.

This could cause the HACDBS engine to operate on another VM's memory.

> +	isb();
> +
> +	if (save->save.hcr_vm_set) {
> +		sysreg_clear_set_hcr(0, HCR_EL2_VM);
> +		isb();
> +	}
> +
> +	write_sysreg_s(save->save.hacdbscons_el2, SYS_HACDBSCONS_EL2);
> +	write_sysreg_s(save->save.hacdbsbr_el2, SYS_HACDBSBR_EL2);

[Severity: High]
Does this sequence lack an immediate isb() after writing SYS_HACDBSCONS_EL2,
allowing the CPU to execute the SYS_HACDBSBR_EL2 write with undefined
pipeline state?

> +	isb();
> +}
> +
> +static void hacdbs_sched_out(struct preempt_notifier *notifier,
> +			     struct task_struct *next)
> +{
> +	struct hacdbs_sched_save *save = container_of(notifier,
> +						      struct hacdbs_sched_save,
> +						      notifier);
> +
> +	if (read_sysreg_s(SYS_HACDBSBR_EL2) & HACDBSBR_EL2_EN) {
> +		save->save.hacdbsbr_el2 = HACDBSBR_EL2_EN;
> +		sysreg_clear_set_s(SYS_HACDBSBR_EL2, HACDBSBR_EL2_EN, 0);
> +		isb();
> +	} else {
> +		save->save.hacdbsbr_el2 = 0;
> +	}
> +
> +	save->save.hacdbscons_el2 = read_sysreg_s(SYS_HACDBSCONS_EL2);
> +	save->save.hacdbsbr_el2 |= read_sysreg_s(SYS_HACDBSBR_EL2);
> +	save->save.vttbr_el2 = read_sysreg_s(SYS_VTTBR_EL2);

[Severity: Critical]
Is it safe to cache the VTTBR (including the VMID) into task-local memory
here in hacdbs_sched_out()?

As mentioned for hacdbs_sched_in(), if a VMID rollover occurs while this
task is preempted, the cached VMID becomes stale.

> +	save->save.vtcr_el2 = read_sysreg_s(SYS_VTCR_EL2);
> +
> +	if (read_sysreg_s(SYS_HCR_EL2) & HCR_EL2_VM) {
> +		sysreg_clear_set_hcr(HCR_EL2_VM, 0);

[Severity: High]
Does this code change the translation regime (clearing HCR_EL2_VM) without
a preceding dsb()?

The engine was disabled earlier in hacdbs_sched_out() (by clearing
HACDBSBR_EL2_EN), but without a dsb(), the asynchronous engine might not be
definitively halted before the translation context changes.

> +		isb();
> +		save->save.hcr_vm_set = true;
> +	} else {
> +		save->save.hcr_vm_set = false;
> +	}
> +}

[ ... ]

> +static int dirty_bit_clear(struct kvm *kvm, u64 *hw_entries, int size)
> +{
> +	int ret;
> +	u64 cons;
> +	struct hacdbs_sched_save save;
> +
> +	preempt_notifier_init(&save.notifier, &hacdbs_preempt_ops);
> +	preempt_disable();
> +	preempt_notifier_register(&save.notifier);
> +	preempt_enable();
> +
> +	__load_stage2(&kvm->arch.mmu);
> +	sysreg_clear_set_hcr(0, HCR_EL2_VM);
> +	isb();
> +
> +	hacdbs_start(hw_entries, size);
> +
> +	do {
> +		cons = read_sysreg_s(SYS_HACDBSCONS_EL2);
> +		if (FIELD_GET(HACDBSCONS_EL2_ERR_REASON, cons))
> +			break;
> +
> +		if (FIELD_GET(HACDBSCONS_EL2_INDEX, cons) >= size)
> +			break;
> +
> +		udelay(1);
> +	} while (true);

[Severity: High]
Could this continuous hardware polling loop hang the CPU permanently?

In dirty_bit_clear(), this loop waits for the hardware engine to finish, but
it lacks a timeout mechanism, bounded retry, or a fatal_signal_pending()
check.

> +
> +	ret = hacdbs_stop(size);
> +
> +	sysreg_clear_set_hcr(HCR_EL2_VM, 0);

[Severity: High]
Can this translation regime change lead to speculative execution or out of
context memory accesses?

In dirty_bit_clear(), hacdbs_stop() clears the enable bit, but clearing
HCR_EL2_VM immediately after lacks an intervening dsb() to ensure the
hardware engine is definitively halted.

> +	isb();
> +
> +	/*
> +	 * No DSB is needed here, as kvm_flush_remote_tlbs_memslot() that happens
> +	 * later in generic dirty-cleaning code already performs a DSB before
> +	 * doing the TLBI.
> +	 */
> +
> +	preempt_disable();
> +	preempt_notifier_unregister(&save.notifier);
> +	preempt_enable();
> +
> +	return ret;
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729104548.3439958-1-leo.bras@arm.com?part=4

Re: [PATCH v3 05/11] kvm: Add arch-generic interface for hw-accelerated dirty-bitmap cleaning

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Critical] Successful hardware dirty log extraction completely bypasses the generic software dirty bitmap, dropping internally-dirtied pages.
- [Critical] Hardware dirty bits are completely ignored when `KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2` is enabled.
- [High] Fallback software loop clobbers partial hardware dirty log output and leaks garbage data if `kvm_arch_dirty_log_clear()` fails.
- [Medium] The `kvm_arch_dirty_log_clear` API overloads two distinct behaviors based on an undocumented `NULL` parameter, risking a NULL pointer dereference in arch implementations.
--

--- Patch [5]: [PATCH v3 05/11] kvm: Add arch-generic interface for hw-accelerated dirty-bitmap cleaning ---
commit 4c5b443921b3fcc7c6b76fa46a7c11e8755ad50f
Author: Leonardo Bras <leo.bras@arm.com>

kvm: Add arch-generic interface for hw-accelerated dirty-bitmap cleaning

This patch introduces kvm_arch_dirty_log_clear() to allow architectures
to implement hardware-accelerated dirty-bitmap clearing. It adds hooks in
both kvm_get_dirty_log_protect() and kvm_clear_dirty_log_protect() to try
the hardware implementation first, falling back to the software loop if it
fails or is unimplemented.

> diff --git a/include/linux/kvm_dirty_bit.h b/include/linux/kvm_dirty_bit.h
> new file mode 100644
> index 0000000000000..fa4f6b67b6235
> --- /dev/null
> +++ b/include/linux/kvm_dirty_bit.h

[ ... ]

> +static inline int kvm_arch_dirty_log_clear(struct kvm *kvm,
> +					   struct kvm_memory_slot *memslot,
> +					   struct kvm_clear_dirty_log *log,
> +					   unsigned long *bitmap,
> +					   bool *flush)

[Severity: Medium]
The signature here strongly implies the log parameter is a valid clear
operation struct. However, when this is called from
kvm_get_dirty_log_protect(), NULL is passed for the log parameter.

Could this establish an undocumented API contract where new arch
implementations blindly dereference the log pointer and cause a host
kernel panic?

> +{
> +	return -ENXIO;
> +}

[ ... ]

> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 45e784462ec6e..7a13ef69b6e3e 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c

[ ... ]

> @@ -2262,6 +2263,10 @@ static int kvm_get_dirty_log_protect(struct kvm *kvm, struct kvm_dirty_log *log)
>  		dirty_bitmap_buffer = kvm_second_dirty_bitmap(memslot);
>  		memset(dirty_bitmap_buffer, 0, n);
>  
> +		if (kvm_arch_dirty_log_clear(kvm, memslot, NULL,
> +					     dirty_bitmap_buffer, &flush) >= 0)

[Severity: Critical]
Since this hardware clear is invoked inside the else block of the
if (kvm->manual_dirty_log_protect) check, are hardware dirty bits
completely ignored when manual protection is enabled?

Modern VMMs often enable manual dirty log protection by default. Will
skipping the hardware tracking here result in userspace only receiving
the software dirty bitmap, potentially causing silent memory corruption
on the destination after a live migration?

[Severity: Medium]
This is the call site passing NULL for the log parameter.

As mentioned above, does this pose a risk of NULL pointer dereferences
for architectural implementations expecting a valid
struct kvm_clear_dirty_log pointer?

> +			goto out;
> +

[Severity: Critical]
If kvm_arch_dirty_log_clear() succeeds, this goto out bypasses the generic
software dirty bitmap processing loop completely.

Does this mean KVM's internally-dirtied pages, such as those from MMIO
emulation or virtual timers marked via mark_page_dirty(), are silently
dropped from the dirty log?

[Severity: High]
If the arch-specific hardware routine fails partway through and returns
an error code, could it leave partial hardware bits in dirty_bitmap_buffer?

If it falls through to the software loop, the software loop might overwrite
hardware bits with software bits (if the software bit is 1) and preserve
garbage hardware bits (if the software bit is 0). Should
dirty_bitmap_buffer be cleared on failure?

>  		KVM_MMU_LOCK(kvm);
>  		for (i = 0; i < n / sizeof(long); i++) {
>  			unsigned long mask;

[ ... ]

> @@ -2373,6 +2379,10 @@ static int kvm_clear_dirty_log_protect(struct kvm *kvm,
>  	if (copy_from_user(dirty_bitmap_buffer, log->dirty_bitmap, n))
>  		return -EFAULT;
>  
> +	if (kvm_arch_dirty_log_clear(kvm, memslot, log, dirty_bitmap_buffer,
> +				     &flush) >= 0)

[Severity: High]
If the arch hook modifies dirty_bitmap_buffer before failing and
returning an error, does it corrupt the mask used by the fallback
clear loop?

The fallback software loop uses dirty_bitmap_buffer to decide which bits
to clear. If it contains garbage from a failed hardware clear, could this
cause the wrong pages to be cleared?

> +		goto out;
> +

[Severity: Critical]
Similar to the GET path, this goto out bypasses the loop that clears bits
in memslot->dirty_bitmap using atomic_long_fetch_andnot().

If the hardware clear succeeds, will the software bits remain set
indefinitely since they are bypassed from being cleared from KVM's
internal tracking?

>  	KVM_MMU_LOCK(kvm);
>  	for (offset = log->first_page, i = offset / BITS_PER_LONG,
>  		 n = DIV_ROUND_UP(log->num_pages, BITS_PER_LONG); n--;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729104548.3439958-1-leo.bras@arm.com?part=5

Re: [PATCH v3 02/11] arm64/cpufeature: Add system-wide FEAT_HACDBS detection

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] Missing KVM ID register filtering exposes FEAT_HACDBS to guests, preventing VMMs from hiding the feature and breaking VM migration.
- [Low] The new capability HACDBS is not alphabetically sorted in the cpucaps list.
--

--- Patch [2]: [PATCH v3 02/11] arm64/cpufeature: Add system-wide FEAT_HACDBS detection ---
commit 79552fba5f033b2becee395a64d0ccf3fbde7b1a
Author: Leonardo Bras <leo.bras@arm.com>

arm64/cpufeature: Add system-wide FEAT_HACDBS detection

This commit adds system-wide CPU feature detection for FEAT_HACDBS
(Hardware dirty bit Cleaning). It updates the cpufeature structures
and cpucaps list to track the feature across the system.

> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> index aa327eebaf1ca..62f56bbd0a65c 100644
> --- a/arch/arm64/kernel/cpufeature.c
> +++ b/arch/arm64/kernel/cpufeature.c
> @@ -523,6 +523,7 @@ static const struct arm64_ftr_bits ftr_id_aa64mmfr3[] = {
>  static const struct arm64_ftr_bits ftr_id_aa64mmfr4[] = {
>  	S_ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR4_EL1_E2H0_SHIFT, 4, 0),
>  	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR4_EL1_NV_frac_SHIFT, 4, 0),
> +	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR4_EL1_HACDBS_SHIFT, 4, 0),

[Severity: Critical]
Does exposing this feature here break VM live migration for KVM?

By adding HACDBS to the host's sanitised ID register, it is subsequently
read by KVM to form the guest's ID register limit.

Looking at arch/arm64/kvm/sys_regs.c:__kvm_read_sanitised_id_reg(), there is
no case handling SYS_ID_AA64MMFR4_EL1, which means the host's HACDBS value
unconditionally leaks into the guest's limit.

Furthermore, looking at sys_reg_descs in arch/arm64/kvm/sys_regs.c:

	ID_WRITABLE(ID_AA64MMFR4_EL1, ID_AA64MMFR4_EL1_NV_frac),

The writable mask for ID_AA64MMFR4_EL1 does not include HACDBS. Because KVM
strictly enforces that non-writable fields must match the destination host's
limit, a VMM attempting to restore a saved VM state will fail if the source
and destination hosts have different HACDBS support.

Could this be addressed by either masking HACDBS from the guest view or
making the field writable by the VMM?

>  	ARM64_FTR_END,
>  };

[ ... ]

> diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
> index a87706c9d160e..bd2c0bb98da61 100644
> --- a/arch/arm64/tools/cpucaps
> +++ b/arch/arm64/tools/cpucaps
> @@ -72,6 +72,7 @@ HAS_XNX
>  HAS_HDBSS
>  HAFT
>  HW_DBM
> +HACDBS

[Severity: Low]
This isn't a bug, but HACDBS is inserted out of alphabetical order here.

Does this violate the "# Internal CPU capabilities constants, keep this list
sorted" comment at the top of the arch/arm64/tools/cpucaps file?

>  KVM_HVHE
>  KVM_PROTECTED_MODE
>  MISMATCHED_CACHE_TYPE

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729104548.3439958-1-leo.bras@arm.com?part=2

Re: [PATCH v3 07/11] KVM: arm64: Dirty-bitmap: avoid splitting previously split blocks

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Dropping `mmu_lock` during eager page splitting creates a race window that can permanently lose hardware-managed dirty bits (FEAT_HAFDBS), leading to silent guest data corruption during live migration.
--

--- Patch [7]: [PATCH v3 07/11] KVM: arm64: Dirty-bitmap: avoid splitting previously split blocks ---
commit fb963a566e307a780c669718446f8bf7c2c6ef94
Author: Leonardo Bras <leo.bras@arm.com>

KVM: arm64: Dirty-bitmap: avoid splitting previously split blocks

If previous dirty-clean already split a block, then avoid calling the
split helper on that block again.

> diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c
> index 5a6266205295e..d7e624f9b6308 100644
> --- a/arch/arm64/kvm/dirty_bit.c
> +++ b/arch/arm64/kvm/dirty_bit.c
> @@ -295,10 +296,15 @@ int __kvm_arch_dirty_log_clear(struct kvm *kvm,
>  
>  		offset = base_gfn + i * BITS_PER_LONG;
>  
> -		if (kvm_dirty_log_manual_protect_and_init_set(kvm))
> -			kvm_mmu_split_huge_pages(kvm,
> -						 gfn_to_gpa(offset + __ffs(mask)),
> -						 gfn_to_gpa(offset + __fls(mask) + 1));
> +		if (kvm_dirty_log_manual_protect_and_init_set(kvm)) {
> +			gpa_t split_start = gfn_to_gpa(offset + __ffs(mask));
> +			gpa_t split_end =  gfn_to_gpa(offset + __fls(mask) + 1);
> +
> +			if (split_end > last_split) {
> +				kvm_mmu_split_huge_pages(kvm, split_start, split_end);

[Severity: High]
Can this sequence lose dirty state due to a race with hardware dirty bit
management?

In __kvm_arch_dirty_log_clear(), the software dirty_bitmap for the target pages
is fetched and cleared before reaching this point.

When kvm_mmu_split_huge_pages() is called, it can temporarily drop
kvm->mmu_lock to allocate memory.

During this unlocked window, a vCPU could locklessly write to the page. With
FEAT_HAFDBS enabled, the hardware automatically sets the PTE dirty bit without
trapping to KVM.

After the lock is reacquired, the loop proceeds to dirty_bit_clear(), which
unconditionally clears the hardware PTE dirty bit (via FEAT_HACDBS).

Because the software bitmap was already cleared and the hardware state is wiped
without feeding the dirty state back, could a write during the unlocked window
be permanently lost if userspace reads the page data before the clear operation?
The next KVM_GET_DIRTY_LOG would not see it.

> +				last_split = ALIGN_DOWN(split_end - 1, PMD_SIZE) + PMD_SIZE;
> +			}
> +		}
>  
>  		do {
>  			idx = mask_to_hdbss(&mask, hw_entries, offset, ttwl, idx, entries_sz);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729104548.3439958-1-leo.bras@arm.com?part=7

Re: [PATCH v3 09/11] kvm/dirty_ring: Add arch-generic interface for hw-accelerated dirty-ring cleaning

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Low] The monolithic `#ifndef CONFIG_HAVE_KVM_HW_DIRTY_BIT` guard causes a build failure for architectures that implement only one of the hardware-accelerated dirty logging hooks.
- [Low] The undocumented API contract of `kvm_arch_dirty_ring_clear` silently shifts the responsibility of updating KVM dirty ring state to the architecture hook.
- [Low] Missing forward declaration for `struct kvm_dirty_ring` in `include/linux/kvm_dirty_bit.h`.
--

--- Patch [9]: [PATCH v3 09/11] kvm/dirty_ring: Add arch-generic interface for hw-accelerated dirty-ring cleaning ---
commit 54695fde1ebfc7f27e678570e500f9d9eb6ecff5
Author: Leonardo Bras <leo.bras@arm.com>

kvm/dirty_ring: Add arch-generic interface for hw-accelerated dirty-ring cleaning

Introduce kvm_arch_dirty_ring_clear() to allow architecture-specific
hardware-accelerated dirty-ring routines. The call is added to
kvm_dirty_ring_reset() and falls back to a software version if not
implemented or if an error is detected.

> diff --git a/include/linux/kvm_dirty_bit.h b/include/linux/kvm_dirty_bit.h
> index fa4f6b67b6235..8492979d694e7 100644
> --- a/include/linux/kvm_dirty_bit.h
> +++ b/include/linux/kvm_dirty_bit.h
> @@ -18,6 +18,13 @@ static inline int kvm_arch_dirty_log_clear(struct kvm *kvm,
>  	return -ENXIO;
>  }
>  
> +static inline int kvm_arch_dirty_ring_clear(struct kvm *kvm,
> +					    struct kvm_dirty_ring *ring,

[Severity: Low]
Does this struct need a forward declaration?

If this header is included before include/linux/kvm_dirty_ring.h, might this
cause a compiler warning for struct kvm_dirty_ring being declared inside a
parameter list?

> +					    int *nr_entries_reset)
> +{
> +	return -ENXIO;
> +}
> +
>  #else /* CONFIG_HAVE_KVM_HW_DIRTY_BIT */

[Severity: Low]
Could this monolithic config guard cause a build failure if an architecture
enables CONFIG_HAVE_KVM_HW_DIRTY_BIT to provide kvm_arch_dirty_log_clear(),
but does not implement kvm_arch_dirty_ring_clear()?

Because the fallback for kvm_arch_dirty_ring_clear() is excluded when the
config is enabled, omitting it in the arch-specific header would leave it
undeclared.

>  
>  #include <asm/kvm_dirty_bit.h>
> diff --git a/virt/kvm/dirty_ring.c b/virt/kvm/dirty_ring.c
> index 807c45fb1fc7b..3fa62894f3d24 100644
> --- a/virt/kvm/dirty_ring.c
> +++ b/virt/kvm/dirty_ring.c

[ ... ]

> @@ -133,6 +134,9 @@ int kvm_dirty_ring_reset(struct kvm *kvm, struct kvm_dirty_ring *ring,
>  	 */
>  	lockdep_assert_held(&kvm->slots_lock);
>  
> +	if (kvm_arch_dirty_ring_clear(kvm, ring, nr_entries_reset) >= 0)
> +		goto out;
> +

[Severity: Low]
Does this bypass implicitly require the arch-specific hook to manually update
the internal state of the dirty ring?

If an architecture implements kvm_arch_dirty_ring_clear() and returns success,
this goto skips the generic software loop. The arch hook would need to
manually increment ring->reset_index, update *nr_entries_reset, and call
kvm_dirty_gfn_set_invalid() for each processed entry to avoid corrupting the
ring state. Is this API contract expected to be documented here?

>  	while (likely((*nr_entries_reset) < INT_MAX)) {
>  		if (signal_pending(current))
>  			return -EINTR;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729104548.3439958-1-leo.bras@arm.com?part=9

Re: [PATCH v3 06/11] KVM: arm64: Add hardware-accelerated dirty-bitmap cleaning routine

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] Dropping `mmu_lock` during eager page splitting allows concurrent guest writes to be permanently lost.
- [Critical] Direct EL2 system register access from EL1 host kernel triggers an UNDEFINED exception on nVHE/pKVM systems.
- [High] Infinite hardware polling loop with preemption disabled risks hard lockups and renders the preempt notifier dead code.
--

--- Patch [6]: [PATCH v3 06/11] KVM: arm64: Add hardware-accelerated dirty-bitmap cleaning routine ---
commit 67bd2745169abaea5a656577d6b7add50b3e09f1
Author: Leonardo Bras <leo.bras@arm.com>

KVM: arm64: Add hardware-accelerated dirty-bitmap cleaning routine

Implement arm64 version of kvm_arch_dirty_log_clear() making use of FEAT_HACDBS.
It converts dirty bitmap entries into HDBSS entries in 64-page blocks and feeds
them to the HACDBS mechanism to clear dirty states. It also supports hugepage
splitting.

> diff --git a/arch/arm64/include/asm/kvm_dirty_bit.h b/arch/arm64/include/asm/kvm_dirty_bit.h
> index dd16438f06518..d368a23a84b30 100644
> --- a/arch/arm64/include/asm/kvm_dirty_bit.h
> +++ b/arch/arm64/include/asm/kvm_dirty_bit.h
[ ... ]
> +static inline bool kvm_arch_dirty_clear_enabled(struct kvm *kvm)
> +{
> +	return system_supports_hacdbs() &&  kvm->arch.mmu.pgt &&
> +	       (kvm->arch.mmu.pgt->flags & KVM_PGTABLE_S2_DBM);
> +}

[Severity: Critical]
Does this check need to verify if the kernel is running in VHE mode?

If this executes on an nVHE or pKVM system (where the host runs at EL1), 
and the hardware supports HACDBS, the ioctl will proceed and call 
dirty_bit_clear(). That function directly accesses EL2 system registers 
like SYS_HACDBSCONS_EL2, which will trap or UNDEF when accessed from EL1.

> diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c
> index 33826bbd16d91..5a6266205295e 100644
> --- a/arch/arm64/kvm/dirty_bit.c
> +++ b/arch/arm64/kvm/dirty_bit.c
[ ... ]
> +int __kvm_arch_dirty_log_clear(struct kvm *kvm,
> +			       struct kvm_memory_slot *memslot,
> +			       struct kvm_clear_dirty_log *log,
> +			       unsigned long *bitmap,
> +			       bool *flush)
> +{
[ ... ]
> +	write_lock(&kvm->mmu_lock);
> +
> +	for (unsigned long i = start; i < end; i++) {
> +		unsigned long mask;
> +		gfn_t offset;
> +		atomic_long_t *p;
> +
> +		if (log) { /* Clean only what is in the input bitmap */
> +			mask = bitmap[i - start];
> +			if (!mask)
> +				continue;
> +
> +			p = (atomic_long_t *)&dirty_bitmap[i];
> +			mask &= atomic_long_fetch_andnot(mask, p);
> +		} else { /* Clean everything, save to output bitmap */
[ ... ]
> +		if (kvm_dirty_log_manual_protect_and_init_set(kvm))
> +			kvm_mmu_split_huge_pages(kvm,
> +						 gfn_to_gpa(offset + __ffs(mask)),
> +						 gfn_to_gpa(offset + __fls(mask) + 1));

[Severity: Critical]
Could this lead to silent memory corruption due to dropped locks?

The software dirty bit is cleared above via atomic_long_fetch_andnot(). 
Then we call kvm_mmu_split_huge_pages(), which drops mmu_lock internally:

kvm_mmu_split_huge_pages()
  need_split_memcache_topup_or_resched()
  write_unlock(&kvm->mmu_lock);
  cond_resched();

While the lock is dropped, a vCPU can write to the page without trapping
because the hardware DBM bit is still set. Once the lock is reacquired, 
dirty_bit_clear() clears the hardware DBM bit. The write gets lost because
both software and hardware bits are now clean.

> +		do {
> +			idx = mask_to_hdbss(&mask, hw_entries, offset, ttwl, idx, entries_sz);
> +			if (idx >= entries_sz) {
> +				ret = dirty_bit_clear(kvm, hw_entries, idx);

[Severity: High]
Is there a risk of a hard lockup here during hardware polling?

Looking at dirty_bit_clear(), it polls SYS_HACDBSCONS_EL2 in a while (true)
loop without a timeout fallback. 

Since __kvm_arch_dirty_log_clear() takes write_lock(&kvm->mmu_lock) before 
calling dirty_bit_clear(), preemption is disabled on non-RT kernels. This 
makes the preempt_notifier registered inside dirty_bit_clear() dead code 
and guarantees the CPU will spin forever if the hardware hangs.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729104548.3439958-1-leo.bras@arm.com?part=6

Re: [PATCH v3 10/11] KVM: arm64: Add hardware-accelerated dirty-ring cleaning routine

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [Critical] Hardware-accelerated dirty ring clear directly accesses EL2 system registers from the host kernel, causing fatal unhandled exceptions on nVHE and pKVM systems.
- [High] The `hw_entries` buffer is allocated using kmalloc, which does not guarantee the page alignment strictly required by the `SYS_HACDBSBR_EL2` hardware register.

Pre-existing issues:
- [High] Writes to control-plane system registers are batched without an intervening `isb()`, violating ARM64 architectural synchronization requirements.
--

--- Patch [10]: [PATCH v3 10/11] KVM: arm64: Add hardware-accelerated dirty-ring cleaning routine ---
commit f9fc2a3f2ff348143923dea638ecd996e369580d
Author: Leonardo Bras <leo.bras@arm.com>

KVM: arm64: Add hardware-accelerated dirty-ring cleaning routine
    
Implement arm64 version of kvm_arch_dirty_ring_clear() making use of
FEAT_HACDBS.
    
It works by transversing the dirty-ring and converting its entries into
HDBSS entries based on the slot offset.
    
The resulting HDBSS array is then fed to the HACDBS mechanism that walks
the pagetable marking writable-dirty pages as writable-clean.

> diff --git a/arch/arm64/include/asm/kvm_dirty_bit.h b/arch/arm64/include/asm/kvm_dirty_bit.h
> index d368a23a84b30..f785a7aad9e88 100644
> --- a/arch/arm64/include/asm/kvm_dirty_bit.h
> +++ b/arch/arm64/include/asm/kvm_dirty_bit.h
> @@ -15,6 +15,9 @@ int __kvm_arch_dirty_log_clear(struct kvm *kvm,
>  			       unsigned long *bitmap,
>  			       bool *flush);
>  
> +int __kvm_arch_dirty_ring_clear(struct kvm *kvm, struct kvm_dirty_ring *ring,
> +				int *nr_entries_reset);
> +
>  static inline bool kvm_arch_dirty_clear_enabled(struct kvm *kvm)
>  {
>  	return system_supports_hacdbs() &&  kvm->arch.mmu.pgt &&
>  	       (kvm->arch.mmu.pgt->flags & KVM_PGTABLE_S2_DBM);
>  }

[Severity: Critical]
Does this gating function also need to verify the VHE execution mode?

By checking only system_supports_hacdbs(), it appears this permits the
accelerated path to be invoked on nVHE and pKVM host systems.

Since dirty_bit_clear() performs direct accesses to EL2 system registers
from the host kernel:

arch/arm64/kvm/dirty_bit.c:dirty_bit_clear() {
    ...
    __load_stage2(&kvm->arch.mmu);
    sysreg_clear_set_hcr(0, HCR_EL2_VM);
    ...
}

Wouldn't this trigger an unhandled hypervisor trap or undefined instruction
exception when triggered by userspace on nVHE systems?

[ ... ]
> diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c
> index d7e624f9b6308..ca9cfefe0893b 100644
> --- a/arch/arm64/kvm/dirty_bit.c
> +++ b/arch/arm64/kvm/dirty_bit.c
[ ... ]
> @@ -359,3 +359,80 @@ int __kvm_arch_dirty_log_clear(struct kvm *kvm,
>  
>  	return ret;
>  }
> +
> +int __kvm_arch_dirty_ring_clear(struct kvm *kvm, struct kvm_dirty_ring *ring,
> +				int *nr_entries_reset)
> +{
> +	u64 *hw_entries __free(kfree) = NULL;
> +	u64 ttwl;
> +	s64 cur_slot = S64_MAX;
> +	int i, ret;
> +	struct kvm_memory_slot *memslot;
> +
> +	if (signal_pending(current))
> +		return -EINTR;
> +
> +	ttwl = HDBSS_ENTRY_TTWL(KVM_PGTABLE_LAST_LEVEL);
> +
> +	hw_entries = kmalloc(max(ring->size * sizeof(u64), PAGE_SIZE), GFP_KERNEL);

[Severity: High]
Could this allocation cause the hardware accelerator to read from the wrong
memory address?

When debugging features like SLUB debugging are enabled, kmalloc allocations
are not strictly page-aligned because of prepended metadata.

The SYS_HACDBSBR_EL2 hardware register requires a page-aligned physical
base address. When hacdbs_start() masks out the offset using
HACDBSBR_EL2_BADDR_MASK, an unaligned base address gets truncated.

Will this result in the hardware reading from the start of the physical
page instead of the actual hw_entries array? It might be necessary to use
alloc_pages_exact() or __get_free_pages() here to guarantee alignment.

> +	if (!hw_entries)
> +		return -ENOMEM;
> +
[ ... ]
> +	if (i == 0)
> +		return 0;
> +
> +	write_lock(&kvm->mmu_lock);
> +	ret = dirty_bit_clear(kvm, hw_entries, i);

[Severity: High]
This is a pre-existing issue, but is it possible that the hardware routines
called here violate architectural synchronization requirements?

Looking at the underlying functions, writes to control-plane system
registers seem to be batched without an intervening isb() as the very next
instruction.

For example, in hacdbs_start():

arch/arm64/kvm/dirty_bit.c:hacdbs_start()
    ...
    sysreg_clear_set_s(SYS_HACDBSCONS_EL2,
                       HACDBSCONS_EL2_ERR_REASON | HACDBSCONS_EL2_INDEX, 0);
                       
    br = (virt_to_phys(hw_entries) & HACDBSBR_EL2_BADDR_MASK) | ...
    ...

And in hacdbs_sched_in():

arch/arm64/kvm/dirty_bit.c:hacdbs_sched_in()
    ...
    write_sysreg_s(save->save.vtcr_el2, SYS_VTCR_EL2);
    write_sysreg_s(save->save.vttbr_el2, SYS_VTTBR_EL2);
    isb();

Could this allow the CPU to execute the intervening instructions in an
architecturally undefined pipeline state?

> +	write_unlock(&kvm->mmu_lock);
> +
> +	/* Set as invalid all successfully cleaned entries */
[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729104548.3439958-1-leo.bras@arm.com?part=10

Re: [PATCH v3 11/11] KVM: arm64: Enable KVM_HW_DIRTY_BIT

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Critical] Direct accesses to EL2 system registers from EL1 cause host panics on nVHE and pKVM configurations.
- [High] Writes to control-plane system registers are missing immediate `isb()` synchronization instructions.
- [High] Unbounded hardware polling loop with preemption disabled.
- [Medium] Preempt notifier is registered within an atomic section where preemption is inherently disabled.
--

--- Patch [11]: [PATCH v3 11/11] KVM: arm64: Enable KVM_HW_DIRTY_BIT ---
commit 9dec1efff0823ef68d8ee166e3fa545263373c0f
Author: Leonardo Bras <leo.bras@arm.com>

KVM: arm64: Enable KVM_HW_DIRTY_BIT
    
Set the corresponding bit to enable hardware accelerated dirty-bitmap and
dirty-ring cleaning for arm64. Actually using acceleration depends on the
cpus enabling FEAT_HACDBS as well as the pre-requisite features for it,
such as FEAT_HDBSS and FEAT_HAFDBS.

> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
> index 449154f9a4852..3264d8ca093e3 100644
> --- a/arch/arm64/kvm/Kconfig
> +++ b/arch/arm64/kvm/Kconfig
> @@ -34,6 +34,7 @@ menuconfig KVM
>  	select HAVE_KVM_IRQ_BYPASS
>  	select HAVE_KVM_READONLY_MEM
>  	select HAVE_KVM_VCPU_RUN_PID_CHANGE
> +	select HAVE_KVM_HW_DIRTY_BIT
>  	select SCHED_INFO
>  	select GUEST_PERF_EVENTS if PERF_EVENTS
>  	select KVM_GUEST_MEMFD
> diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c
> new file mode 100644
> index 0000000000000..ca9cfefe0893b
> --- /dev/null
> +++ b/arch/arm64/kvm/dirty_bit.c
> @@ -0,0 +1,438 @@
[ ... ]
> +static void hacdbs_start(u64 *hw_entries, int size)
> +{
> +	u64 br;
> +	/* Each entry is 8 bytes */
> +	int size_b = size * sizeof(hw_entries[0]);
> +	int size_p2 = max(roundup_pow_of_two(size_b), PAGE_SIZE);
> +
> +	/* If not using the full size of the array, put a stop entry at the end */
> +	if (size_b < size_p2)
> +		hw_entries[size] = HDBSS_ENTRY_VALID | HDBSS_ENTRY_TTWL_RESV;
> +
> +	sysreg_clear_set_s(SYS_HACDBSCONS_EL2,
> +			   HACDBSCONS_EL2_ERR_REASON | HACDBSCONS_EL2_INDEX, 0);

[Severity: Critical]
Can this code cause a host panic on nVHE and pKVM configurations?

The code directly attempts to read and write EL2 registers, which triggers
a synchronous undefined exception at EL1. Since kvm_arch_dirty_clear_enabled()
only checks for system_supports_hacdbs() and does not enforce has_vhe(),
could an unprivileged userspace application reliably crash the host by
triggering this on non-VHE hardware?

[Severity: High]
Does this code allow the CPU pipeline to observe architecturally undefined
state? 

The ARM64 subsystem rules require an isb() as the very next instruction
following a control-plane sysreg write. Here, SYS_HACDBSCONS_EL2 is modified
but several instructions and variable assignments occur before an isb() is
reached.

> +
> +	br = (virt_to_phys(hw_entries) & HACDBSBR_EL2_BADDR_MASK) |
> +	     FIELD_PREP(HACDBSBR_EL2_SZ, ilog2(size_p2) - 12) |
> +	     FIELD_PREP(HACDBSBR_EL2_EN, 1);
> +
> +	/* All writes to the array have to complete before starting */
> +	dsb(ishst);
> +
> +	write_sysreg_s(br, SYS_HACDBSBR_EL2);
> +	isb();
> +}
[ ... ]
> +static void hacdbs_sched_in(struct preempt_notifier *notifier, int cpu)
> +{
> +	struct hacdbs_sched_save *save = container_of(notifier,
> +						      struct hacdbs_sched_save,
> +						      notifier);
> +
> +	write_sysreg_s(save->save.vtcr_el2, SYS_VTCR_EL2);
> +	write_sysreg_s(save->save.vttbr_el2, SYS_VTTBR_EL2);
> +	isb();

[Severity: High]
Can this missing synchronization lead to unpredictable execution?

The ARM64 subsystem rules require an isb() as the very next instruction
following a control-plane sysreg write, but SYS_VTCR_EL2 is written followed
by SYS_VTTBR_EL2 before the isb() is issued.

> +
> +	if (save->save.hcr_vm_set) {
> +		sysreg_clear_set_hcr(0, HCR_EL2_VM);
> +		isb();
> +	}
> +
> +	write_sysreg_s(save->save.hacdbscons_el2, SYS_HACDBSCONS_EL2);
> +	write_sysreg_s(save->save.hacdbsbr_el2, SYS_HACDBSBR_EL2);
> +	isb();

[Severity: High]
Is another isb() required here?

SYS_HACDBSCONS_EL2 is written followed by SYS_HACDBSBR_EL2 without
an intervening isb(). Does this sequence allow the CPU pipeline to observe
stale configurations?

> +}
[ ... ]
> +static int dirty_bit_clear(struct kvm *kvm, u64 *hw_entries, int size)
> +{
> +	int ret;
> +	u64 cons;
> +	struct hacdbs_sched_save save;
> +
> +	preempt_notifier_init(&save.notifier, &hacdbs_preempt_ops);
> +	preempt_disable();
> +	preempt_notifier_register(&save.notifier);
> +	preempt_enable();

[Severity: Medium]
Are these preempt notifiers effectively dead code on standard kernels?

The calling functions (__kvm_arch_dirty_log_clear and
__kvm_arch_dirty_ring_clear) acquire write_lock(&kvm->mmu_lock) before
this code runs.

Since the mmu_lock rwlock already disables preemption, this region cannot be
preempted. If the hardware cleaning takes a long time, won't the thread
stall rather than cleanly context switching?

> +
> +	__load_stage2(&kvm->arch.mmu);
> +	sysreg_clear_set_hcr(0, HCR_EL2_VM);
> +	isb();

[Severity: Critical]
Does this code also trigger a synchronous undefined exception at EL1?

Directly attempting to access EL2 registers from EL1 causes traps on non-VHE
hardware.

> +
> +	hacdbs_start(hw_entries, size);
> +
> +	do {
> +		cons = read_sysreg_s(SYS_HACDBSCONS_EL2);
> +		if (FIELD_GET(HACDBSCONS_EL2_ERR_REASON, cons))
> +			break;
> +
> +		if (FIELD_GET(HACDBSCONS_EL2_INDEX, cons) >= size)
> +			break;
> +
> +		udelay(1);
> +	} while (true);

[Severity: High]
Could this unbounded polling loop cause a hard lockup?

Since this runs under write_lock(&kvm->mmu_lock) with preemption disabled,
if the hardware accelerator hangs, takes unusually long, or fails to set an
error reason, the CPU will stall indefinitely. Should there be a timeout or
fallback mechanism here?

> +
> +	ret = hacdbs_stop(size);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729104548.3439958-1-leo.bras@arm.com?part=11