[PATCH v2 00/18] KVM: arm64: Introduce pKVM hypervisor heap allocator

[PATCH v2 00/18] KVM: arm64: Introduce pKVM hypervisor heap allocator

[Vincent Donnefort]

pKVM historically lacked a dynamic memory allocator: all hypervisor-side
VM and VCPU structures had to be sized on the host, allocated as
contiguous pages and donated to the hypervisor.

This design tightly coupled the hypervisor's memory footprint to
host-side constraints, complicated memory reclaim, and severely
restricted VM scalability.

This patch series introduces a dynamically-mapped custom heap allocator
(hyp_allocator) to the pKVM hypervisor. The initial users are the
pkvm_hyp_vm and pkvm_hyp_vcpu structs, and the hypervisor tracing
metadata.

In the near future, this heap allocator is expected to be leveraged to
support SVE in protected VMs and in the distant future, it will also
support dynamic device assignment.

By moving to a hypervisor-managed dynamic allocator, we also allow
deduplicating the donation/reclaim path of EL2-private structures.

The main building blocks for this series are:

1. pkvm_hyp_req:
----------------
When the hypervisor heap allocator goes out of memory (-ENOMEM), it
suspends the hypercall, embeds a PKVM_HYP_REQ_HYP_ALLOC top-up request
into the SMCCC HVC return registers, and exits back to the host.

This building block will also be useful for the future huge-mapping
support in protected guests, allowing EL2 to raise requests such as
block splitting back to the host.

2. hyp_allocator:
----------------
This heap allocator manages a reserved VA space range, dynamically
mapping and unmapping physical pages on-demand to minimise the pKVM
hypervisor footprint. As memory is reclaimed and relinquished to the
host, unmapped holes are introduced within the VA space. To prevent
orphan mapped regions, neighboring unused chunks cannot be merged if
they are separated by an unmapped region.

The allocator chunk metadata is stored directly into the VA space range.
To minimize metadata overhead, chunks only link to each other via a
relative 32-bit offset.

A simple hardening of the metadata is added via a simple 32-bit hash.

3. shrinker:
------------
As the heap allocator isn't reclaimed actively on VM or tracing
teardown, a shrinker is added to allow the host to reclaim unused memory
from the hypervisor when the host is under heavy memory pressure.

v1 -> v2:
  - Rebased series on 7.2-rc2.
  - Use scope-based hyp_spinlock.
  - Fix best_missing/best_data_size priority in hyp_allocator_find_efficient_chunk() (Sashiko)
  - Fix missing free_hyp_memcache() in pkvm_hyp_topup() (Sashiko)
  - Fix unused selftest_init() warning when !CONFIG_NVHE_EL2_DEBUG (Sashiko)
  - Fix missing shrinker_free() in teardown_hyp_mode() (Sashiko)

v1: https://lore.kernel.org/r/20260520152650.4107895-1-vdonnefort@google.com

Vincent Donnefort (18):
  KVM: arm64: Add __pkvm_private_range_pa
  KVM: arm64: Add pkvm_remove_mappings
  KVM: arm64: Add __hyp_allocator_map for the pKVM hyp
  KVM: arm64: Add a heap allocator for the pKVM hyp
  KVM: arm64: Allow kvm_hyp_memcache usage outside of stage-2
  KVM: arm64: Add topup interface for the pKVM heap allocator
  KVM: arm64: Add pkvm_hyp_req infrastructure
  KVM: arm64: Handle PKVM_HYP_REQ_HYP_ALLOC request
  KVM: arm64: Add reclaim interface for the pKVM heap alloc
  KVM: arm64: Add selftests for the pKVM heap allocator
  KVM: arm64: Add a shrinker for pKVM
  KVM: arm64: Filter out non-kernel addresses in kern_hyp_va
  KVM: arm64: Move hyp_vm refcount into the structure
  KVM: arm64: Use noclear for PGD in __pkvm_init_vm error path
  KVM: arm64: Alloc pkvm_hyp_vm using pKVM heap allocator
  KVM: arm64: Alloc pkvm_hyp_vcpu using pKVM heap allocator
  KVM: arm64: Reject hyp trace descriptors with fewer CPUs than
    hyp_nr_cpus
  KVM: arm64: Alloc simple_buffer_page using pKVM hyp allocator

 arch/arm64/include/asm/kvm_asm.h           |    4 +
 arch/arm64/include/asm/kvm_host.h          |   14 +-
 arch/arm64/include/asm/kvm_mmu.h           |    3 +
 arch/arm64/include/asm/kvm_pkvm.h          |  104 ++
 arch/arm64/kvm/arm.c                       |   26 +-
 arch/arm64/kvm/hyp/hyp-constants.c         |    2 -
 arch/arm64/kvm/hyp/include/nvhe/alloc.h    |   24 +
 arch/arm64/kvm/hyp/include/nvhe/mm.h       |    3 +
 arch/arm64/kvm/hyp/include/nvhe/pkvm.h     |   19 +-
 arch/arm64/kvm/hyp/include/nvhe/spinlock.h |    5 +
 arch/arm64/kvm/hyp/nvhe/Makefile           |    2 +-
 arch/arm64/kvm/hyp/nvhe/alloc.c            | 1217 ++++++++++++++++++++
 arch/arm64/kvm/hyp/nvhe/hyp-main.c         |  124 +-
 arch/arm64/kvm/hyp/nvhe/mm.c               |   33 +
 arch/arm64/kvm/hyp/nvhe/pkvm.c             |  111 +-
 arch/arm64/kvm/hyp/nvhe/setup.c            |    5 +
 arch/arm64/kvm/hyp/nvhe/trace.c            |   66 +-
 arch/arm64/kvm/hyp_trace.c                 |   16 +-
 arch/arm64/kvm/mmu.c                       |    4 +-
 arch/arm64/kvm/pkvm.c                      |  136 ++-
 arch/arm64/kvm/trace_pkvm.h                |   37 +
 21 files changed, 1801 insertions(+), 154 deletions(-)
 create mode 100644 arch/arm64/kvm/hyp/include/nvhe/alloc.h
 create mode 100644 arch/arm64/kvm/hyp/nvhe/alloc.c
 create mode 100644 arch/arm64/kvm/trace_pkvm.h


base-commit: 8cdeaa50eae8dad34885515f62559ee83e7e8dda
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 01/18] KVM: arm64: Add __pkvm_private_range_pa

[Vincent Donnefort]

Mappings in the pKVM private range are not identity mapped, making the
standard __hyp_pa() unsuitable for translating these addresses.

Introduce __pkvm_private_range_pa() to resolve physical addresses for
this range by walking the hypervisor page-table. This will be useful for
the upcoming pKVM heap allocator.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h
index 6e83ce35c2f2..d3137c16b632 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/mm.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h
@@ -30,5 +30,6 @@ int __pkvm_create_private_mapping(phys_addr_t phys, size_t size,
 				  unsigned long *haddr);
 int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr);
 int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr);
+phys_addr_t __pkvm_private_range_pa(void *va);
 
 #endif /* __KVM_HYP_MM_H */
diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c
index 3b0bee496bff..773426a68d2d 100644
--- a/arch/arm64/kvm/hyp/nvhe/mm.c
+++ b/arch/arm64/kvm/hyp/nvhe/mm.c
@@ -502,3 +502,17 @@ int refill_memcache(struct kvm_hyp_memcache *mc, unsigned long min_pages,
 
 	return ret;
 }
+
+phys_addr_t __pkvm_private_range_pa(void *va)
+{
+	kvm_pte_t pte = 0;
+	s8 level;
+
+	hyp_spin_lock(&pkvm_pgd_lock);
+	WARN_ON(kvm_pgtable_get_leaf(&pkvm_pgtable, (u64)va, &pte, &level));
+	hyp_spin_unlock(&pkvm_pgd_lock);
+
+	WARN_ON(!kvm_pte_valid(pte));
+
+	return kvm_pte_to_phys(pte) + offset_in_page(va);
+}
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 02/18] KVM: arm64: Add pkvm_remove_mappings

[Vincent Donnefort]

Add the counterpart to pkvm_create_mappings(), allowing previously
mapped ranges to be removed. This will be useful for the upcoming pKVM
heap allocator to manage its private mappings.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h
index d3137c16b632..725bb0fb941d 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/mm.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h
@@ -25,6 +25,7 @@ int hyp_back_vmemmap(phys_addr_t back);
 int pkvm_cpu_set_vector(enum arm64_hyp_spectre_vector slot);
 int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot);
 int pkvm_create_mappings_locked(void *from, void *to, enum kvm_pgtable_prot prot);
+void pkvm_remove_mappings(void *from, void *to);
 int __pkvm_create_private_mapping(phys_addr_t phys, size_t size,
 				  enum kvm_pgtable_prot prot,
 				  unsigned long *haddr);
diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c
index 773426a68d2d..ec59da1322a7 100644
--- a/arch/arm64/kvm/hyp/nvhe/mm.c
+++ b/arch/arm64/kvm/hyp/nvhe/mm.c
@@ -146,6 +146,20 @@ int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot)
 	return ret;
 }
 
+void pkvm_remove_mappings(void *from, void *to)
+{
+	u64 size;
+
+	to = PTR_ALIGN(to, PAGE_SIZE);
+	from = PTR_ALIGN_DOWN(from, PAGE_SIZE);
+	size = (u64)to - (u64)from;
+	WARN_ON(from > to);
+
+	hyp_spin_lock(&pkvm_pgd_lock);
+	WARN_ON(kvm_pgtable_hyp_unmap(&pkvm_pgtable, (u64)from, size) != size);
+	hyp_spin_unlock(&pkvm_pgd_lock);
+}
+
 int hyp_back_vmemmap(phys_addr_t back)
 {
 	unsigned long i, start, size, end = 0;
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 03/18] KVM: arm64: Add __hyp_allocator_map for the pKVM hyp

[Vincent Donnefort]

In preparation for the pKVM heap allocator, introduce __hyp_allocator_map()
to map a single physical page to a given virtual address in the hypervisor
stage-1 page-table.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h
index 725bb0fb941d..98a7774b541c 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/mm.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h
@@ -32,5 +32,6 @@ int __pkvm_create_private_mapping(phys_addr_t phys, size_t size,
 int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr);
 int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr);
 phys_addr_t __pkvm_private_range_pa(void *va);
+int __hyp_allocator_map(void *va, phys_addr_t phys);
 
 #endif /* __KVM_HYP_MM_H */
diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c
index ec59da1322a7..c1c01f81ac5f 100644
--- a/arch/arm64/kvm/hyp/nvhe/mm.c
+++ b/arch/arm64/kvm/hyp/nvhe/mm.c
@@ -160,6 +160,11 @@ void pkvm_remove_mappings(void *from, void *to)
 	hyp_spin_unlock(&pkvm_pgd_lock);
 }
 
+int __hyp_allocator_map(void *va, phys_addr_t phys)
+{
+	return __pkvm_create_mappings((unsigned long)va, PAGE_SIZE, phys, PAGE_HYP);
+}
+
 int hyp_back_vmemmap(phys_addr_t back)
 {
 	unsigned long i, start, size, end = 0;
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 04/18] KVM: arm64: Add a heap allocator for the pKVM hyp

[Vincent Donnefort]

Currently, memory used by the hypervisor comes from donations that are
embedded within HVCs. e.g. __pkvm_init_vm()'s hyp_vm.

This is cumbersome: the host needs to know the size of those
struct, the memory must be page-aligned and physically contiguous which
may be difficult to satisfy when host memory is highly fragmented.

Create a heap allocator to manage VA-contiguous memory. This allocator
grows upward, recycles unused chunks of memory and provides a simple API
to allocate and free:

  hyp_alloc(size), hyp_free(addr)

This heap allocator also manages the underlying physical memory,
allowing the host to top up the allocator's pool and reclaim memory.

  hyp_alloc_topup(), hyp_alloc_reclaim().

Pages remain mapped in the allocator's VA-space as long as they are not
reclaimed.

When the allocator runs out of memory, hyp_alloc() fails and
hyp_alloc_errno() returns -ENOMEM to signal that a top-up is required.

Additionally, harden the allocator by hashing chunk headers to detect
metadata corruption.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/kvm/hyp/include/nvhe/alloc.h b/arch/arm64/kvm/hyp/include/nvhe/alloc.h
new file mode 100644
index 000000000000..8f87a63f8946
--- /dev/null
+++ b/arch/arm64/kvm/hyp/include/nvhe/alloc.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __KVM_NVHE_ALLOC__
+#define __KVM_NVHE_ALLOC__
+#include <linux/types.h>
+
+#include <asm/kvm_host.h>
+
+void *hyp_alloc(size_t size);
+int hyp_alloc_errno(void);
+u32 hyp_alloc_topup_needed(void);
+void hyp_free(void *addr);
+
+int hyp_alloc_init(size_t size);
+int hyp_alloc_topup(struct kvm_hyp_memcache *host_mc);
+unsigned long hyp_alloc_reclaimable(void);
+void hyp_alloc_reclaim(struct kvm_hyp_memcache *host_mc, unsigned long target);
+#endif
diff --git a/arch/arm64/kvm/hyp/include/nvhe/spinlock.h b/arch/arm64/kvm/hyp/include/nvhe/spinlock.h
index 7c7ea8c55405..c0b3a1a48e45 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/spinlock.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/spinlock.h
@@ -16,6 +16,7 @@
 #include <asm/alternative.h>
 #include <asm/lse.h>
 #include <asm/rwonce.h>
+#include <linux/cleanup.h>
 
 typedef union hyp_spinlock {
 	u32	__val;
@@ -122,4 +123,8 @@ static inline void hyp_assert_lock_held(hyp_spinlock_t *lock)
 static inline void hyp_assert_lock_held(hyp_spinlock_t *lock) { }
 #endif
 
+DEFINE_LOCK_GUARD_1(hyp_spinlock, hyp_spinlock_t,
+		    hyp_spin_lock(_T->lock),
+		    hyp_spin_unlock(_T->lock))
+
 #endif /* __ARM64_KVM_NVHE_SPINLOCK_H__ */
diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
index f57450ebcb49..985fe3343f4c 100644
--- a/arch/arm64/kvm/hyp/nvhe/Makefile
+++ b/arch/arm64/kvm/hyp/nvhe/Makefile
@@ -23,7 +23,7 @@ lib-objs := $(addprefix ../../../lib/, $(lib-objs))
 CFLAGS_switch.nvhe.o += -Wno-override-init
 
 hyp-obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \
-	 hyp-main.o hyp-smp.o psci-relay.o early_alloc.o page_alloc.o \
+	 hyp-main.o hyp-smp.o psci-relay.o alloc.o early_alloc.o page_alloc.o \
 	 cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o
 hyp-obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \
 	 ../hyp-entry.o ../exception.o ../pgtable.o ../vgic-v5-sr.o
diff --git a/arch/arm64/kvm/hyp/nvhe/alloc.c b/arch/arm64/kvm/hyp/nvhe/alloc.c
new file mode 100644
index 000000000000..07ef0a13855f
--- /dev/null
+++ b/arch/arm64/kvm/hyp/nvhe/alloc.c
@@ -0,0 +1,1033 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 Google LLC
+ * Author: Vincent Donnefort <vdonnefort@google.com>
+ *
+ * This heap allocator manages a reserved VA space range, dynamically mapping
+ * and unmapping physical pages on-demand to minimise the pKVM hypervisor
+ * footprint. As memory is reclaimed and relinquished to the host, unmapped
+ * holes are introduced within the VA space. To prevent orphans mapped regions,
+ * neighboring unused chunks cannot be merged if they are separated by an
+ * unmapped region.
+ *
+ */
+
+#include <nvhe/alloc.h>
+#include <nvhe/mem_protect.h>
+#include <nvhe/mm.h>
+#include <nvhe/spinlock.h>
+
+#include <linux/build_bug.h>
+#include <linux/hash.h>
+
+#define MIN_ALLOC_SIZE 8UL /* Must be a power of two */
+
+/**
+ * struct chunk_hdr - Chunk header
+ * @next:       offset from this chunk header to the next one.
+ * @prev:       offset from this chunk header to the previous one.
+ * @__unmapped: Internal field containing the offset to the unmapped page
+ *              boundary, multiplexed with the allocation state flag.
+ * @hash:       Hash computed over the chunk header.
+ */
+struct chunk_hdr {
+	u32			next;
+	u32			prev;
+#define USED_BIT_MASK		1U
+	u32			__unmapped;
+	u32			hash;
+	char			data[];
+} __aligned(MIN_ALLOC_SIZE);
+
+/**
+ * struct hyp_allocator - Heap allocator
+ * @start:		Start in the allocator's reserved virtual address range.
+ * @end:		End in the allocator's reserved virtual address range.
+ * @last_used:		Pointer to the end of the last used chunk. This is
+ *			necessary for the last chunk in the list as the
+ *			allocated size of a chunk is derived from the next one.
+ * @first_unmapped:	Pointer to the first unmapped page in the
+ *			allocator's range. This is only necessary and
+ *			updated when no chunk is in the list.
+ * @head:		Head of the chunk list.
+ * @tail:		Tail of the chunk list.
+ * @mc:			Memcache containing pre-allocated pages for mapping.
+ * @lock:		Spinlock protecting the allocator state.
+ * @errno:		Per-CPU error code for allocation failures.
+ * @topup_needed:	Per-CPU page counter needed to top-up the memcache.
+ */
+struct hyp_allocator {
+	void			*start;
+	void			*end;
+	void			*last_used;
+	void			*first_unmapped;
+	struct chunk_hdr	*head;
+	struct chunk_hdr	*tail;
+	struct kvm_hyp_memcache	mc;
+	hyp_spinlock_t		lock;
+	int __percpu		*errno;
+	u32 __percpu		*topup_needed;
+};
+
+static u32 chunk_hash_compute(const struct chunk_hdr *chunk)
+{
+	u32 hash = 0;
+
+	BUILD_BUG_ON(sizeof(*chunk) != 16);
+
+	hash ^= hash_64(*(const u64 *)chunk, 32);
+	hash ^= hash_32(chunk->__unmapped, 32);
+	return hash;
+}
+
+static void chunk_set_hash(struct chunk_hdr *chunk)
+{
+	if (chunk)
+		chunk->hash = chunk_hash_compute(chunk);
+}
+
+static void chunk_check_hash(const struct chunk_hdr *chunk)
+{
+	if (chunk)
+		WARN_ON(chunk->hash != chunk_hash_compute(chunk));
+}
+
+static bool chunk_is_used(const struct chunk_hdr *chunk)
+{
+	return !!(chunk->__unmapped & USED_BIT_MASK);
+}
+
+static void chunk_set_used(struct chunk_hdr *chunk)
+{
+	chunk->__unmapped |= USED_BIT_MASK;
+}
+
+static void chunk_set_unused(struct chunk_hdr *chunk)
+{
+	chunk->__unmapped &= ~USED_BIT_MASK;
+}
+
+static void *chunk_unmapped(const struct chunk_hdr *chunk)
+{
+	u32 offset = chunk->__unmapped & ~USED_BIT_MASK;
+
+	if (!offset)
+		return NULL;
+
+	return (void *)chunk + offset;
+}
+
+static void __chunk_set_unmapped(struct chunk_hdr *chunk, u32 unmapped)
+{
+	chunk->__unmapped = unmapped | (chunk_is_used(chunk) ? USED_BIT_MASK : 0);
+}
+
+static void chunk_set_unmapped(struct chunk_hdr *chunk, void *unmapped)
+{
+	WARN_ON(!PAGE_ALIGNED(unmapped));
+
+	if (unmapped) {
+		WARN_ON((void *)chunk > unmapped);
+		__chunk_set_unmapped(chunk, unmapped - (void *)chunk);
+	} else {
+		__chunk_set_unmapped(chunk, 0);
+	}
+}
+
+static void *chunk_data(const struct chunk_hdr *chunk)
+{
+	return (void *)&chunk->data;
+}
+
+static struct chunk_hdr *__chunk_next(const struct chunk_hdr *chunk)
+{
+	if (!chunk->next)
+		return NULL;
+
+	return (struct chunk_hdr *)((void *)chunk + chunk->next);
+}
+
+static struct chunk_hdr *__chunk_prev(const struct chunk_hdr *chunk)
+{
+	if (!chunk->prev)
+		return NULL;
+
+	return (struct chunk_hdr *)((void *)chunk - chunk->prev);
+}
+
+static void chunk_set_next(struct chunk_hdr *chunk, struct chunk_hdr *next)
+{
+	if (!chunk)
+		return;
+
+	if (next) {
+		WARN_ON(chunk > next);
+		chunk->next = (void *)next - (void *)chunk;
+	} else {
+		chunk->next = 0;
+	}
+}
+
+static void chunk_set_prev(struct chunk_hdr *chunk, struct chunk_hdr *prev)
+{
+	if (!chunk)
+		return;
+
+	if (prev) {
+		WARN_ON(chunk < prev);
+		chunk->prev = (void *)chunk - (void *)prev;
+	} else {
+		chunk->prev = 0;
+	}
+}
+
+static struct chunk_hdr *chunk_get_next(const struct chunk_hdr *chunk)
+{
+	struct chunk_hdr *next = __chunk_next(chunk);
+
+	chunk_check_hash(next);
+	return next;
+}
+
+static struct chunk_hdr *chunk_get_prev(const struct chunk_hdr *chunk)
+{
+	struct chunk_hdr *prev = __chunk_prev(chunk);
+
+	chunk_check_hash(prev);
+	return prev;
+}
+
+static struct chunk_hdr *chunk_get(struct chunk_hdr *chunk)
+{
+	chunk_check_hash(chunk);
+	return chunk;
+}
+
+#define chunk_hdr_size() \
+	offsetof(struct chunk_hdr, data)
+
+#define chunk_min_size() \
+	(chunk_hdr_size() + MIN_ALLOC_SIZE)
+
+static size_t chunk_data_size(const struct chunk_hdr *chunk, struct hyp_allocator *allocator)
+{
+	struct chunk_hdr *next = chunk_get_next(chunk);
+	void *end;
+
+	if (next)
+		end = (void *)next;
+	else
+		end = allocator->end;
+
+	return end - chunk_data(chunk);
+}
+
+static size_t chunk_mapped_data_size(const struct chunk_hdr *chunk, struct hyp_allocator *allocator)
+{
+	void *unmapped = chunk_unmapped(chunk);
+
+	if (!unmapped)
+		return chunk_data_size(chunk, allocator);
+
+	return unmapped - chunk_data(chunk);
+}
+
+static size_t chunk_used_size(const struct chunk_hdr *chunk, struct hyp_allocator *allocator)
+{
+	struct chunk_hdr *next = chunk_get_next(chunk);
+
+	if (!chunk_is_used(chunk))
+		return 0;
+
+	if (next)
+		return chunk_mapped_data_size(chunk, allocator);
+
+	return allocator->last_used - chunk_data(chunk);
+}
+
+static void chunk_list_insert(struct chunk_hdr *chunk, struct chunk_hdr *prev)
+{
+	struct chunk_hdr *next = NULL;
+
+	WARN_ON(!chunk);
+
+	if (prev) {
+		next = chunk_get_next(prev);
+		chunk_set_next(prev, chunk);
+		chunk_set_hash(prev);
+	}
+
+	if (next) {
+		chunk_set_prev(next, chunk);
+		chunk_set_hash(next);
+	}
+
+	chunk_set_next(chunk, next);
+	chunk_set_prev(chunk, prev);
+}
+
+static void chunk_list_del(struct chunk_hdr *chunk)
+{
+	struct chunk_hdr *prev, *next;
+
+	WARN_ON(!chunk);
+
+	prev = chunk_get_prev(chunk);
+	next = chunk_get_next(chunk);
+
+	if (prev) {
+		chunk_set_next(prev, next);
+		chunk_set_hash(prev);
+	}
+
+	if (next) {
+		chunk_set_prev(next, prev);
+		chunk_set_hash(next);
+	}
+}
+
+/*
+ * Return a fixup start address for chunk creation. It makes sure the chunk
+ * header doesn't cross any page boundary and that it leaves enough space at the
+ * start of page. This is intended to prevent orphan mapped regions during chunk
+ * memory reclaim
+ */
+static void *chunk_start(void *start)
+{
+	void *page = PTR_ALIGN(start, PAGE_SIZE);
+
+	if (page - start < chunk_hdr_size())
+		return page;
+
+	page = PTR_ALIGN_DOWN(start, PAGE_SIZE);
+	if (start - page < chunk_min_size())
+		return page + chunk_min_size();
+
+	return start;
+}
+
+static void hyp_allocator_set_errno(struct hyp_allocator *allocator, int errno)
+{
+	*this_cpu_ptr(allocator->errno) = errno;
+}
+
+static int hyp_allocator_errno(struct hyp_allocator *allocator)
+{
+	int *errno = this_cpu_ptr(allocator->errno);
+	int ret = *errno;
+
+	*errno = 0;
+
+	return ret;
+}
+
+static int hyp_allocator_map(struct hyp_allocator *allocator, struct chunk_hdr *chunk,
+			     struct chunk_hdr *next,
+			     void *addr, void *end)
+{
+	void *unmapped = chunk ? chunk_unmapped(chunk) : allocator->first_unmapped;
+
+	/*
+	 * hyp_allocator_can_create_chunk() already validates addr/end
+	 * belong to the chunk.
+	 */
+	WARN_ON(end <= addr);
+
+	/* The chunk does not span an unmapped region */
+	if (!unmapped)
+		return 0;
+
+	while (unmapped < end) {
+		void *page = pop_hyp_memcache(&allocator->mc, hyp_phys_to_virt);
+		int ret;
+
+		if (!page) {
+			end = PTR_ALIGN(end, PAGE_SIZE);
+			*this_cpu_ptr(allocator->topup_needed) =
+				(unsigned long)(end - unmapped) >> PAGE_SHIFT;
+			return -ENOMEM;
+		}
+
+		ret = __hyp_allocator_map(unmapped, hyp_virt_to_phys(page));
+		if (ret) {
+			push_hyp_memcache(&allocator->mc, page, hyp_virt_to_phys);
+			return ret;
+		}
+
+		unmapped += PAGE_SIZE;
+
+		/*
+		 * Reset the unmap field if we've reached the next chunk or the
+		 * allocator boundary.
+		 */
+		if (unmapped == (next ?: allocator->end))
+			unmapped = 0;
+
+		if (chunk) {
+			chunk_set_unmapped(chunk, unmapped);
+			chunk_set_hash(chunk);
+		} else {
+			allocator->first_unmapped = unmapped;
+		}
+
+		if (!unmapped)
+			break;
+	}
+
+	return 0;
+}
+
+static void hyp_allocator_unmap(struct hyp_allocator *allocator, struct chunk_hdr *chunk,
+				void *addr, void *end)
+{
+	void *unmap = addr;
+
+	/*
+	 * hyp_allocator_chunk_reclaimable() already computes valid addr/end, no
+	 * need to check them again
+	 */
+	WARN_ON(end <= addr);
+
+	while (unmap < end) {
+		phys_addr_t pa = __pkvm_private_range_pa((void *)unmap);
+		void *page = hyp_phys_to_virt(pa);
+
+		push_hyp_memcache(&allocator->mc, page, hyp_virt_to_phys);
+		unmap += PAGE_SIZE;
+	}
+
+	pkvm_remove_mappings((void *)addr, (void *)(end));
+
+	if (chunk) {
+		chunk_set_unmapped(chunk, addr);
+		chunk_set_hash(chunk);
+	} else {
+		allocator->first_unmapped = addr;
+	}
+}
+
+static bool hyp_allocator_can_create_chunk(struct hyp_allocator *allocator,
+					   const struct chunk_hdr *prev,
+					   const struct chunk_hdr *next,
+					   void *addr, void *end)
+{
+	void *page, *unmapped;
+
+	if (addr < allocator->start || end > allocator->end)
+		return false;
+
+	/* First chunk created must be installed at allocator->start */
+	if (!prev)
+		return addr == allocator->start;
+
+	/* Must not overwrite the next chunk */
+	if (next && end > (void *)next)
+		return false;
+
+	/* Must not overwrite the previous chunk */
+	if (addr < (chunk_data(prev) + chunk_used_size(prev, allocator)))
+		return false;
+
+	/* Header must not cross page boundaries */
+	page = PTR_ALIGN(addr, PAGE_SIZE);
+	if (page != addr && (page - addr) < chunk_hdr_size())
+		return false;
+
+	/* Must leave a minimum distance from a page-start to maximise reclaim */
+	page = PTR_ALIGN_DOWN(addr, PAGE_SIZE);
+	if (page != addr && (addr - page) < chunk_min_size())
+		return false;
+
+	unmapped = chunk_unmapped(prev);
+	if (!unmapped)
+		return true;
+
+	/* Must never create an orphan mapped region */
+	if (addr > unmapped)
+		return false;
+
+	return true;
+}
+
+/*
+ * Tries to create a new chunk in the allocator whose header starts at @addr and
+ * whose data finishes at @end.
+ */
+static struct chunk_hdr *hyp_allocator_create_chunk(struct hyp_allocator *allocator,
+						    struct chunk_hdr *prev, void *addr,
+						    void *end, bool used)
+{
+	struct chunk_hdr *next, *chunk = addr;
+	void *unmapped;
+	int ret;
+
+	if (end > allocator->end)
+		return ERR_PTR(-E2BIG);
+
+	next = prev ? chunk_get_next(prev) : NULL;
+	if (!hyp_allocator_can_create_chunk(allocator, prev, next, addr, end))
+		return ERR_PTR(-EINVAL);
+
+	ret = hyp_allocator_map(allocator, prev, next, addr, end);
+	if (ret)
+		return ERR_PTR(ret);
+
+	memset(chunk, 0, sizeof(*chunk));
+	if (used)
+		chunk_set_used(chunk);
+	else
+		chunk_set_unused(chunk);
+
+	/* First chunk, first allocation */
+	if (!prev) {
+		chunk_set_unmapped(chunk, allocator->first_unmapped);
+		chunk_list_insert(chunk, NULL);
+		chunk_set_hash(chunk);
+
+		allocator->last_used = end;
+		allocator->head = allocator->tail = chunk;
+		return chunk;
+	}
+
+	/* Last chunk in the list */
+	if (!next) {
+		allocator->last_used = end;
+		allocator->tail = chunk;
+	}
+
+	/* Inherit prev's unmapped region */
+	unmapped = chunk_unmapped(prev);
+	chunk_set_unmapped(chunk, unmapped);
+	chunk_list_insert(chunk, prev);
+	chunk_set_hash(chunk);
+
+	chunk_set_unmapped(prev, 0);
+	chunk_set_hash(prev);
+
+	return chunk;
+}
+
+static bool hyp_allocator_can_destroy_chunk(struct hyp_allocator *allocator,
+					    const struct chunk_hdr *prev,
+					    const struct chunk_hdr *next,
+					    const struct chunk_hdr *chunk)
+{
+	if (chunk_is_used(chunk))
+		return false;
+
+	/* Last chunk in the allocator */
+	if (!prev)
+		return true;
+
+	/* Can't merge down unless we are the last one in the list */
+	if (next && chunk_is_used(prev))
+		return false;
+
+	/* Must never create an orphan mapped region */
+	if (chunk_unmapped(prev))
+		return false;
+
+	return true;
+}
+
+static int hyp_allocator_destroy_chunk(struct hyp_allocator *allocator,
+				       struct chunk_hdr *prev,
+				       struct chunk_hdr *chunk)
+{
+	struct chunk_hdr *next;
+
+	next = prev ? chunk_get_next(chunk) : NULL;
+	if (!hyp_allocator_can_destroy_chunk(allocator, prev, next, chunk))
+		return -EINVAL;
+
+	/* Last chunk in the allocator */
+	if (!prev) {
+		allocator->first_unmapped = chunk_unmapped(chunk);
+		allocator->head = allocator->tail = NULL;
+		return 0;
+	}
+
+	/* Last chunk in the list */
+	if (!next) {
+		allocator->last_used = chunk;
+		allocator->tail = prev;
+	}
+
+	chunk_set_unmapped(prev, chunk_unmapped(chunk));
+	chunk_set_hash(prev);
+	chunk_list_del(chunk);
+
+	return 0;
+}
+
+/*
+ * Return the best unused chunk for recycling, that is the smallest chunk
+ * fitting the allocation which needs to use the least unmapped region.
+ */
+static struct chunk_hdr *hyp_allocator_find_efficient_chunk(struct hyp_allocator *allocator,
+							    size_t size)
+{
+	struct chunk_hdr *chunk, *best_chunk = NULL;
+	size_t best_data_size = SIZE_MAX;
+	size_t best_missing = SIZE_MAX;
+
+	chunk = allocator->head;
+	while (chunk) {
+		size_t missing, mapped, data_size;
+
+		if (chunk_is_used(chunk))
+			goto next;
+
+		data_size = chunk_data_size(chunk, allocator);
+		if (data_size < size)
+			goto next;
+
+		mapped = chunk_mapped_data_size(chunk, allocator);
+		missing = (size > mapped) ? DIV_ROUND_UP(size - mapped, PAGE_SIZE) : 0;
+		if (missing > best_missing)
+			goto next;
+
+		if (missing == best_missing && data_size >= best_data_size)
+			goto next;
+
+		best_missing = missing;
+		best_data_size = data_size;
+		best_chunk = chunk;
+
+next:
+		chunk = chunk_get_next(chunk);
+	}
+
+	return best_chunk;
+}
+
+static struct chunk_hdr *hyp_allocator_reuse_chunk(struct hyp_allocator *allocator,
+						   struct chunk_hdr *chunk, size_t size)
+{
+	struct chunk_hdr *next = chunk_get_next(chunk);
+	void *start, *end, *split, *split_end;
+	int ret;
+
+	start = chunk_data(chunk);
+	end = start + size;
+
+	/* Last chunk in the list, no need to split */
+	if (!next) {
+		split = split_end = NULL;
+		allocator->last_used = chunk_data(chunk) + size;
+	} else {
+		split = chunk_start(end);
+		split_end = split + chunk_min_size();
+
+		if (!hyp_allocator_can_create_chunk(allocator, chunk, next, split, split_end))
+			split = split_end = NULL;
+	}
+
+	/* Batch the mapping of the reused chunk and the split */
+	ret = hyp_allocator_map(allocator, chunk, next, chunk_data(chunk), split ? split_end : end);
+	if (ret)
+		return ERR_PTR(ret);
+
+	if (split)
+		WARN_ON(IS_ERR_OR_NULL(
+			hyp_allocator_create_chunk(allocator, chunk, split, split_end, false)));
+
+	chunk_set_used(chunk);
+	chunk_set_hash(chunk);
+
+	return chunk;
+}
+
+static struct chunk_hdr *hyp_allocator_alloc_chunk(struct hyp_allocator *allocator, size_t size)
+{
+	struct chunk_hdr *chunk;
+	void *start, *end;
+
+	/* First allocation */
+	if (!allocator->head) {
+		start = allocator->start;
+		end = start + chunk_hdr_size() + size;
+		return hyp_allocator_create_chunk(allocator, NULL, start, end, true);
+	}
+
+	chunk = hyp_allocator_find_efficient_chunk(allocator, size);
+
+	/* Nothing found, create a new chunk at the end in the list */
+	if (!chunk) {
+		start = chunk_start(chunk_data(allocator->tail) +
+				    chunk_used_size(allocator->tail, allocator));
+		end = start + chunk_hdr_size() + size;
+		return hyp_allocator_create_chunk(allocator, allocator->tail, start, end, true);
+	}
+
+	return hyp_allocator_reuse_chunk(allocator, chunk, size);
+}
+
+static void *hyp_allocator_alloc(struct hyp_allocator *allocator, size_t size)
+{
+	struct chunk_hdr *chunk;
+
+	size = max(size, MIN_ALLOC_SIZE);
+
+	/* Ensure we do not overflow ALIGN(MIN_ALLOC_SIZE) */
+	if (size > U32_MAX) {
+		hyp_allocator_set_errno(allocator, -E2BIG);
+		return NULL;
+	}
+
+	size = ALIGN(size, MIN_ALLOC_SIZE);
+	if (size > (allocator->end - allocator->start - chunk_hdr_size())) {
+		hyp_allocator_set_errno(allocator, -E2BIG);
+		return NULL;
+	}
+
+#ifdef CONFIG_NVHE_EL2_DEBUG
+	/* The allocator can modify the hyp stage-1 */
+	if (WARN_ON(hyp_spin_is_locked(&pkvm_pgd_lock))) {
+		hyp_allocator_set_errno(allocator, -EINVAL);
+		return NULL;
+	}
+#endif
+	scoped_guard(hyp_spinlock, &allocator->lock) {
+		chunk = hyp_allocator_alloc_chunk(allocator, size);
+		if (IS_ERR_OR_NULL(chunk)) {
+			hyp_allocator_set_errno(allocator,
+						IS_ERR(chunk) ? PTR_ERR(chunk) : -EINVAL);
+			return NULL;
+		}
+	}
+
+	memset(chunk_data(chunk), 0, size);
+	return chunk_data(chunk);
+}
+
+static void hyp_allocator_free(struct hyp_allocator *allocator, void *data)
+{
+	struct chunk_hdr *chunk, *next, *prev;
+
+	if (!data)
+		return;
+
+	WARN_ON(!IS_ALIGNED((unsigned long)data, MIN_ALLOC_SIZE));
+	WARN_ON(data >= allocator->end || data < allocator->start + chunk_hdr_size());
+
+	guard(hyp_spinlock)(&allocator->lock);
+
+	chunk = chunk_get(container_of(data, struct chunk_hdr, data));
+	WARN_ON(!chunk_is_used(chunk));
+	chunk_set_unused(chunk);
+	chunk_set_hash(chunk);
+
+	next = chunk_get_next(chunk);
+	if (next)
+		hyp_allocator_destroy_chunk(allocator, chunk, next);
+
+	prev = chunk_get_prev(chunk);
+	if (prev)
+		hyp_allocator_destroy_chunk(allocator, prev, chunk);
+}
+
+static unsigned long hyp_allocator_chunk_reclaimable(struct hyp_allocator *allocator,
+						     const struct chunk_hdr *chunk,
+						     u64 *__addr, u64 *__end)
+{
+	struct chunk_hdr *next;
+	void *addr, *end;
+
+	/* Last chunk in the allocator */
+	if (chunk == allocator->head && chunk == allocator->tail && !chunk_is_used(chunk)) {
+		addr = (void *)chunk;
+		end = chunk_unmapped(chunk);
+		if (!end)
+			end = allocator->end;
+		goto end;
+	}
+
+	next = chunk_get_next(chunk);
+
+	/* Last chunk in the list we can reclaim, even if used */
+	if (!next) {
+		addr = chunk_data(chunk) + chunk_used_size(chunk, allocator);
+		addr = PTR_ALIGN(addr, PAGE_SIZE);
+		end = chunk_unmapped(chunk);
+		if (!end)
+			end = allocator->end;
+		goto end;
+	}
+
+	if (chunk_is_used(chunk))
+		return 0;
+
+	addr = PTR_ALIGN(chunk_data(chunk), PAGE_SIZE);
+	end = chunk_unmapped(chunk);
+	if (!end)
+		end = PTR_ALIGN_DOWN(next, PAGE_SIZE);
+
+end:
+	if (addr >= end)
+		return 0;
+
+	if (__end)
+		*__end = (u64)end;
+	if (__addr)
+		*__addr = (u64)addr;
+
+	return (end - addr) >> PAGE_SHIFT;
+}
+
+static void hyp_allocator_reclaim_chunk(struct hyp_allocator *allocator, struct chunk_hdr *chunk,
+					void *addr, void *end)
+{
+	struct chunk_hdr *next;
+
+	WARN_ON(end <= addr);
+
+	/* We are about to destroy the last chunk in the allocator */
+	if (addr == allocator->start) {
+		allocator->tail = allocator->head = chunk = NULL;
+		goto unmap;
+	}
+
+	next = chunk_get_next(chunk);
+
+	/*
+	 * Split the reclaimed chunk at the next page boundary,
+	 * this ensures no orphan mapped region is created. Splitting at the page boundary is always
+	 * possible because chunks always leave a minimum distance to the page start.
+	 *
+	 *  +--------------+
+	 *  |______________|
+	 *  |______________|<- Next chunk
+	 *  |_ _ _ __ _ _ _|
+	 *  |              |<- Page-aligned split
+	 *  +--------------+
+	 *  +--------------+
+	 *  |              |
+	 *  |              |<- Page reclaimed
+	 *  |              |
+	 *  |              |
+	 *  +--------------+
+	 *  +--------------+
+	 *  |              |
+	 *  |______________|
+	 *  |______________|<- Chunk to split
+	 *  |              |
+	 *  +--------------+
+	 */
+	if (next && !chunk_unmapped(chunk) && next != end)
+		WARN_ON(IS_ERR_OR_NULL(hyp_allocator_create_chunk(allocator, chunk, end, next,
+								  false)));
+unmap:
+	hyp_allocator_unmap(allocator, chunk, addr, end);
+}
+
+/*
+ * Return the best reclaimable chunk which is the highest chunk in the list
+ * with the biggest reclaimable region.
+ */
+static struct chunk_hdr *hyp_allocator_find_reclaimable_chunk(struct hyp_allocator *allocator,
+							      u64 *addr, u64 *end)
+{
+	struct chunk_hdr *chunk, *best_chunk = NULL;
+	unsigned long best_reclaimable = 0;
+
+	chunk = allocator->head;
+	while (chunk) {
+		u64 __addr, __end;
+		unsigned long reclaimable = hyp_allocator_chunk_reclaimable(allocator, chunk,
+									    &__addr, &__end);
+
+		/* Favour the top biggest chunks */
+		if (reclaimable && reclaimable >= best_reclaimable) {
+			best_reclaimable = reclaimable;
+			best_chunk = chunk;
+			*addr = __addr;
+			*end = __end;
+		}
+
+		chunk = chunk_get_next(chunk);
+	}
+
+	return best_chunk;
+}
+
+static unsigned long hyp_allocator_drain_memcache(struct hyp_allocator *allocator,
+						  struct kvm_hyp_memcache *host_mc,
+						  unsigned long target)
+{
+	struct kvm_hyp_memcache *mc = &allocator->mc;
+	unsigned long drained = 0;
+
+	while (target && mc->nr_pages) {
+		void *page = pop_hyp_memcache(mc, hyp_phys_to_virt);
+
+		memset(page, 0, PAGE_SIZE);
+		kvm_flush_dcache_to_poc(page, PAGE_SIZE);
+		push_hyp_memcache(host_mc, page, hyp_virt_to_phys);
+		WARN_ON(__pkvm_hyp_donate_host(hyp_virt_to_pfn(page), 1));
+
+		target--;
+		drained++;
+	}
+
+	return drained;
+}
+
+static void hyp_allocator_reclaim(struct hyp_allocator *allocator, struct kvm_hyp_memcache *host_mc,
+				  unsigned long target)
+{
+	if (!target)
+		return;
+
+	guard(hyp_spinlock)(&allocator->lock);
+
+	target -= hyp_allocator_drain_memcache(allocator, host_mc, target);
+	if (!target)
+		return;
+
+	do {
+		unsigned long reclaimable;
+		struct chunk_hdr *chunk;
+		u64 addr, end;
+
+		chunk = hyp_allocator_find_reclaimable_chunk(allocator, &addr, &end);
+		if (!chunk)
+			break;
+
+		reclaimable = min((end - addr) >> PAGE_SHIFT, target);
+		addr = end - (reclaimable << PAGE_SHIFT);
+		hyp_allocator_reclaim_chunk(allocator, chunk, (void *)addr, (void *)end);
+
+		target -= reclaimable;
+	} while (target);
+
+	hyp_allocator_drain_memcache(allocator, host_mc, ULONG_MAX);
+}
+
+static unsigned long hyp_allocator_reclaimable(struct hyp_allocator *allocator)
+{
+	unsigned long reclaimable = 0;
+	struct chunk_hdr *chunk;
+
+	guard(hyp_spinlock)(&allocator->lock);
+
+	chunk = allocator->head;
+	while (chunk) {
+		reclaimable += hyp_allocator_chunk_reclaimable(allocator, chunk, NULL, NULL);
+		chunk = chunk_get_next(chunk);
+	}
+
+	return reclaimable;
+}
+
+static int hyp_allocator_topup(struct hyp_allocator *allocator,
+			       struct kvm_hyp_memcache *host_mc)
+{
+	struct kvm_hyp_memcache *alloc_mc = &allocator->mc;
+
+	guard(hyp_spinlock)(&allocator->lock);
+	return refill_memcache(alloc_mc, host_mc->nr_pages + alloc_mc->nr_pages, host_mc);
+}
+
+static u32 hyp_allocator_topup_needed(struct hyp_allocator *allocator)
+{
+	u32 *topup_needed = this_cpu_ptr(allocator->topup_needed);
+	u32 ret = *topup_needed;
+
+	*topup_needed = 0;
+
+	return ret;
+}
+
+static int hyp_allocator_init(struct hyp_allocator *allocator, size_t size)
+{
+	unsigned long start;
+	int ret;
+
+	size = PAGE_ALIGN(size);
+
+	/* constrained by chunk_hdr u32 types */
+	if (size > U32_MAX || !size)
+		return -EINVAL;
+
+	ret = pkvm_alloc_private_va_range(size, &start);
+	if (ret)
+		return ret;
+
+	allocator->first_unmapped = allocator->start = (void *)start;
+	allocator->end = allocator->start + size;
+	hyp_spin_lock_init(&allocator->lock);
+
+	return 0;
+}
+
+static DEFINE_PER_CPU(int, __hyp_allocator_errno);
+static DEFINE_PER_CPU(u32, __hyp_allocator_topup_needed);
+
+static struct hyp_allocator hyp_allocator = {
+	.errno = &__hyp_allocator_errno,
+	.topup_needed = &__hyp_allocator_topup_needed,
+};
+
+/**
+ * hyp_alloc() - Allocate memory from the heap allocator
+ *
+ * @size:	Allocation size in bytes.
+ *
+ * Return: A pointer to the allocated memory on success, else NULL.
+ */
+void *hyp_alloc(size_t size)
+{
+	return hyp_allocator_alloc(&hyp_allocator, size);
+}
+
+/**
+ * hyp_free() - Free memory allocated with hyp_alloc()
+ *
+ * @data:	Address returned by the original hyp_alloc().
+ *
+ * The use of any other address than one returned by hyp_alloc() will cause a
+ * hypervisor panic.
+ */
+void hyp_free(void *data)
+{
+	hyp_allocator_free(&hyp_allocator, data);
+}
+
+/**
+ * hyp_alloc_errno() - Read the errno on allocation error
+ *
+ * Get the return code from an allocation failure.
+ *
+ * Return: -ENOMEM if the allocator needs a refill from the host, -E2BIG if
+ * there is no VA space left else 0.
+ */
+int hyp_alloc_errno(void)
+{
+	return hyp_allocator_errno(&hyp_allocator);
+}
+
+int hyp_alloc_init(size_t size)
+{
+	return hyp_allocator_init(&hyp_allocator, size);
+}
+
+void hyp_alloc_reclaim(struct kvm_hyp_memcache *mc, unsigned long target)
+{
+	hyp_allocator_reclaim(&hyp_allocator, mc, target);
+}
+
+unsigned long hyp_alloc_reclaimable(void)
+{
+	return hyp_allocator_reclaimable(&hyp_allocator);
+}
+
+int hyp_alloc_topup(struct kvm_hyp_memcache *host_mc)
+{
+	return hyp_allocator_topup(&hyp_allocator, host_mc);
+}
+
+u32 hyp_alloc_topup_needed(void)
+{
+	return hyp_allocator_topup_needed(&hyp_allocator);
+}
diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c
index 75b00c323310..1c10cf78c869 100644
--- a/arch/arm64/kvm/hyp/nvhe/setup.c
+++ b/arch/arm64/kvm/hyp/nvhe/setup.c
@@ -10,6 +10,7 @@
 #include <asm/kvm_pgtable.h>
 #include <asm/kvm_pkvm.h>
 
+#include <nvhe/alloc.h>
 #include <nvhe/early_alloc.h>
 #include <nvhe/ffa.h>
 #include <nvhe/gfp.h>
@@ -363,6 +364,10 @@ int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long *per_cpu_bas
 	if (ret)
 		return ret;
 
+	ret = hyp_alloc_init(SZ_128M);
+	if (ret)
+		return ret;
+
 	update_nvhe_init_params();
 
 	/* Jump in the idmap page to switch to the new page-tables */
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 05/18] KVM: arm64: Allow kvm_hyp_memcache usage outside of stage-2

[Vincent Donnefort]

Although currently limited to guest stage-2 page-table allocations,
struct kvm_hyp_memcache is a useful primitive for passing a list of
discontiguous pages between host and hypervisor.

Introduce init_hyp_memcache() to initialise a generic hyp memcache, and
init_hyp_stage2_memcache() for stage-2 specific memcaches. The generic
initialiser will be used to top up the upcoming pKVM heap allocator.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index bae2c4f92ef5..caea8e9986d1 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -91,9 +91,22 @@ struct kvm_hyp_memcache {
 	struct pkvm_mapping *mapping; /* only used from EL1 */
 
 #define	HYP_MEMCACHE_ACCOUNT_STAGE2	BIT(1)
+#define	HYP_MEMCACHE_ACCOUNT_KMEMCG	BIT(2)
 	unsigned long flags;
 };
 
+static inline void init_hyp_memcache(struct kvm_hyp_memcache *mc)
+{
+	memset(mc, 0, sizeof(*mc));
+	mc->mapping = ZERO_SIZE_PTR; /* Prevent allocation, solely useful for stage2 memcache */
+}
+
+static inline void init_hyp_stage2_memcache(struct kvm_hyp_memcache *mc)
+{
+	memset(mc, 0, sizeof(*mc));
+	mc->flags = HYP_MEMCACHE_ACCOUNT_STAGE2 | HYP_MEMCACHE_ACCOUNT_KMEMCG;
+}
+
 static inline void push_hyp_memcache(struct kvm_hyp_memcache *mc,
 				     phys_addr_t *p,
 				     phys_addr_t (*to_pa)(void *virt))
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 6c941aaa10c6..fb38821f2037 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1159,9 +1159,11 @@ static void hyp_mc_free_fn(void *addr, void *mc)
 static void *hyp_mc_alloc_fn(void *mc)
 {
 	struct kvm_hyp_memcache *memcache = mc;
+	gfp_t gfp = (memcache->flags & HYP_MEMCACHE_ACCOUNT_KMEMCG) ?
+		    GFP_KERNEL_ACCOUNT : GFP_KERNEL;
 	void *addr;
 
-	addr = (void *)__get_free_page(GFP_KERNEL_ACCOUNT);
+	addr = (void *)__get_free_page(gfp);
 	if (addr && memcache->flags & HYP_MEMCACHE_ACCOUNT_STAGE2)
 		kvm_account_pgtable_pages(addr, 1);
 
diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index 053e4f733e4b..8324a6a1bc48 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -111,7 +111,7 @@ static int __pkvm_create_hyp_vcpu(struct kvm_vcpu *vcpu)
 	void *hyp_vcpu;
 	int ret;
 
-	vcpu->arch.pkvm_memcache.flags |= HYP_MEMCACHE_ACCOUNT_STAGE2;
+	init_hyp_stage2_memcache(&vcpu->arch.pkvm_memcache);
 
 	hyp_vcpu = alloc_pages_exact(hyp_vcpu_sz, GFP_KERNEL_ACCOUNT);
 	if (!hyp_vcpu)
@@ -172,7 +172,7 @@ static int __pkvm_create_hyp_vm(struct kvm *kvm)
 		goto free_vm;
 
 	kvm->arch.pkvm.is_created = true;
-	kvm->arch.pkvm.stage2_teardown_mc.flags |= HYP_MEMCACHE_ACCOUNT_STAGE2;
+	init_hyp_stage2_memcache(&kvm->arch.pkvm.stage2_teardown_mc);
 	kvm_account_pgtable_pages(pgd, pgd_sz / PAGE_SIZE);
 
 	return 0;
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 06/18] KVM: arm64: Add topup interface for the pKVM heap allocator

[Vincent Donnefort]

Introduce a host HVC interface and a host side helper to allow refilling
the pKVM heap allocator.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index 043495f7fc78..681b7bf8ac08 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -114,6 +114,7 @@ enum __kvm_host_smccc_func {
 	__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_load,
 	__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_put,
 	__KVM_HOST_SMCCC_FUNC___pkvm_tlb_flush_vmid,
+	__KVM_HOST_SMCCC_FUNC___pkvm_hyp_topup,
 
 	MARKER(__KVM_HOST_SMCCC_FUNC_MAX)
 };
diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h
index 74fedd9c5ff0..870f7870bb62 100644
--- a/arch/arm64/include/asm/kvm_pkvm.h
+++ b/arch/arm64/include/asm/kvm_pkvm.h
@@ -17,6 +17,10 @@
 
 #define HYP_MEMBLOCK_REGIONS 128
 
+enum pkvm_topup_id {
+	PKVM_TOPUP_HYP_ALLOC,
+};
+
 int pkvm_init_host_vm(struct kvm *kvm, unsigned long type);
 int pkvm_create_hyp_vm(struct kvm *kvm);
 bool pkvm_hyp_vm_is_created(struct kvm *kvm);
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
index d3c69de698f4..e270a974fa37 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
@@ -15,6 +15,7 @@
 #include <asm/kvm_hypevents.h>
 #include <asm/kvm_mmu.h>
 
+#include <nvhe/alloc.h>
 #include <nvhe/ffa.h>
 #include <nvhe/mem_protect.h>
 #include <nvhe/mm.h>
@@ -632,6 +633,30 @@ static void handle___pkvm_finalize_teardown_vm(struct kvm_cpu_context *host_ctxt
 	cpu_reg(host_ctxt, 1) = __pkvm_finalize_teardown_vm(handle);
 }
 
+static void handle___pkvm_hyp_topup(struct kvm_cpu_context *host_ctxt)
+{
+	DECLARE_REG(enum pkvm_topup_id, id, host_ctxt, 1);
+	DECLARE_REG(phys_addr_t, head, host_ctxt, 2);
+	DECLARE_REG(unsigned long, nr_pages, host_ctxt, 3);
+	struct kvm_hyp_memcache host_mc = {
+		.head = head,
+		.nr_pages = nr_pages,
+	};
+	int ret;
+
+	switch (id) {
+	case PKVM_TOPUP_HYP_ALLOC:
+		ret = hyp_alloc_topup(&host_mc);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	cpu_reg(host_ctxt, 1) = ret;
+	cpu_reg(host_ctxt, 2) = host_mc.head;
+	cpu_reg(host_ctxt, 3) = host_mc.nr_pages;
+}
+
 static void handle___tracing_load(struct kvm_cpu_context *host_ctxt)
 {
 	DECLARE_REG(unsigned long, desc_hva, host_ctxt, 1);
@@ -762,6 +787,7 @@ static const hcall_t host_hcall[] = {
 	HANDLE_FUNC(__pkvm_vcpu_load),
 	HANDLE_FUNC(__pkvm_vcpu_put),
 	HANDLE_FUNC(__pkvm_tlb_flush_vmid),
+	HANDLE_FUNC(__pkvm_hyp_topup),
 };
 
 static void handle_host_hcall(struct kvm_cpu_context *host_ctxt)
diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index 8324a6a1bc48..e2c8714e8ccc 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -85,6 +85,33 @@ void __init kvm_hyp_reserve(void)
 		 hyp_mem_base);
 }
 
+static int pkvm_hyp_topup(enum pkvm_topup_id id, unsigned long nr_pages)
+{
+	struct kvm_hyp_memcache mc;
+	struct arm_smccc_res res;
+	int ret;
+
+	init_hyp_memcache(&mc);
+	ret = topup_hyp_memcache(&mc, nr_pages);
+	if (ret)
+		goto err;
+
+	arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(__pkvm_hyp_topup), id, mc.head,
+			  mc.nr_pages, &res);
+	if (WARN_ON_ONCE(res.a0 != SMCCC_RET_SUCCESS)) {
+		ret = -EINVAL;
+		goto err;
+	}
+
+	ret = res.a1;
+	mc.head = res.a2;
+	mc.nr_pages = res.a3;
+
+err:
+	free_hyp_memcache(&mc);
+	return ret;
+}
+
 static void __pkvm_destroy_hyp_vm(struct kvm *kvm)
 {
 	if (pkvm_hyp_vm_is_created(kvm)) {
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 07/18] KVM: arm64: Add pkvm_hyp_req infrastructure

[Vincent Donnefort]

Introduce a struct pkvm_hyp_req to enable the pKVM hypervisor to request
resources from the host.

Provide serialisation helpers to transport these requests via SMCCC
registers (starting from a2):

  pkvm_hyp_req_to_smccc() to encode into the SMCCC args.
  smccc_to_pkvm_hyp_req() to decode them.

When the hypervisor raises a request, the host must handle it and retry
the HVC. To automate this sequence, introduce the pkvm_call_hyp_req()
macro. This intercepts pending requests, invokes the handler and retries
the HVC.

Additionally, introduce a trace event to track the handling of these
requests.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h
index 870f7870bb62..0be1469e7874 100644
--- a/arch/arm64/include/asm/kvm_pkvm.h
+++ b/arch/arm64/include/asm/kvm_pkvm.h
@@ -203,6 +203,95 @@ struct pkvm_mapping {
 	u64 __subtree_last;	/* Internal member for interval tree */
 };
 
+enum pkvm_hyp_req_type {
+	PKVM_HYP_NO_REQ = 0,
+	__PKVM_HYP_REQ_TYPE_MAX,
+};
+
+#define PKVM_HYP_REQ_SMCCC_ARG_SIZE_MAX \
+	(sizeof(struct arm_smccc_res) - offsetof(struct arm_smccc_res, a2) - 1)
+
+struct pkvm_hyp_req {
+	u8 type;
+	union {
+		struct {
+			u32	nr_pages;
+		} mem;
+		struct {
+			/* Helper for SMCCC encoding/decoding */
+			u8	args[PKVM_HYP_REQ_SMCCC_ARG_SIZE_MAX];
+		} args;
+	};
+};
+
+static inline size_t pkvm_hyp_req_arg_size(u8 type)
+{
+	switch (type) {
+	case PKVM_HYP_NO_REQ:
+		return 0;
+	default:
+		WARN_ON(1);
+	}
+
+	return 0;
+}
+
+/* Encode the pending pkvm_hyp_req type into the SMCCC args */
+static inline void
+pkvm_hyp_req_to_smccc(struct kvm_cpu_context *host_ctxt, struct pkvm_hyp_req *req)
+{
+	u8 *dst, type = req->type;
+	size_t size;
+
+	if (type == PKVM_HYP_NO_REQ || type >= __PKVM_HYP_REQ_TYPE_MAX) {
+		host_ctxt->regs.regs[2] = 0;
+		return;
+	}
+
+	size = pkvm_hyp_req_arg_size(type);
+	if (WARN_ON(size > PKVM_HYP_REQ_SMCCC_ARG_SIZE_MAX))
+		return;
+
+	dst = (u8 *)&host_ctxt->regs.regs[2];
+	*dst = type;
+
+	memcpy(dst + 1, &req->args, size);
+}
+
+/* Return true if a pkvm_hyp_req has been decoded from the SMCCC args */
+static inline bool smccc_to_pkvm_hyp_req(struct pkvm_hyp_req *req, struct arm_smccc_res *res)
+{
+	u8 *src = (u8 *)res + offsetof(struct arm_smccc_res, a2);
+	u8 type = *src;
+
+	if (type == PKVM_HYP_NO_REQ || type >= __PKVM_HYP_REQ_TYPE_MAX)
+		return false;
+
+	req->type = type;
+	memcpy(&req->args, src + 1, pkvm_hyp_req_arg_size(type));
+
+	return true;
+}
+
+int __pkvm_handle_smccc_req(struct arm_smccc_res *res);
+
+#define pkvm_call_hyp_req(f, ...)								\
+({												\
+	struct arm_smccc_res __res;								\
+	int __ret;										\
+	do {											\
+		__ret = -1;									\
+		arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(f), ##__VA_ARGS__, &__res);		\
+		if (WARN_ON(__res.a0 != SMCCC_RET_SUCCESS))					\
+			break;									\
+		__ret = __res.a1;								\
+		if (!__ret)									\
+			break;									\
+		__ret = __pkvm_handle_smccc_req(&__res);					\
+	} while (!__ret);									\
+	__ret;											\
+})
+
 int pkvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
 			     struct kvm_pgtable_mm_ops *mm_ops);
 void pkvm_pgtable_stage2_destroy_range(struct kvm_pgtable *pgt,
diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index e2c8714e8ccc..5ba8b81b8c6c 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -16,6 +16,9 @@
 
 #include "hyp_constants.h"
 
+#define CREATE_TRACE_POINTS
+#include "trace_pkvm.h"
+
 DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized);
 
 static struct memblock_region *hyp_memory = kvm_nvhe_sym(hyp_memory);
@@ -617,3 +620,25 @@ bool pkvm_force_reclaim_guest_page(phys_addr_t phys)
 
 	return !ret || ret == -EAGAIN;
 }
+
+static int pkvm_handle_hyp_req(struct pkvm_hyp_req *req)
+{
+	int ret = -EINVAL;
+
+	switch (req->type) {
+	}
+
+	trace_kvm_handle_pkvm_hyp_req(req, ret);
+
+	return ret;
+}
+
+int __pkvm_handle_smccc_req(struct arm_smccc_res *res)
+{
+	struct pkvm_hyp_req req;
+
+	if (smccc_to_pkvm_hyp_req(&req, res))
+		return pkvm_handle_hyp_req(&req);
+
+	return res->a1;
+}
diff --git a/arch/arm64/kvm/trace_pkvm.h b/arch/arm64/kvm/trace_pkvm.h
new file mode 100644
index 000000000000..4bf57c12e7de
--- /dev/null
+++ b/arch/arm64/kvm/trace_pkvm.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#if !defined(_TRACE_PKVM_ARM64_KVM_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_PKVM_ARM64_KVM_H
+
+#include <linux/tracepoint.h>
+#include <asm/kvm_pkvm.h>
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM kvm
+
+TRACE_EVENT(kvm_handle_pkvm_hyp_req,
+	TP_PROTO(struct pkvm_hyp_req *req, int ret),
+	TP_ARGS(req, ret),
+
+	TP_STRUCT__entry(
+		__field(u8,	type)
+		__field(int,	ret)
+	),
+
+	TP_fast_assign(
+		__entry->type = req->type;
+		__entry->ret = ret;
+	),
+
+	TP_printk("type: %u ret: %d",
+		  __entry->type, __entry->ret)
+);
+
+#endif /* _TRACE_PKVM_ARM64_KVM_H */
+
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE trace_pkvm
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 08/18] KVM: arm64: Handle PKVM_HYP_REQ_HYP_ALLOC request

[Vincent Donnefort]

Introduce a new pkvm_hyp_request type asking the host to top up the pKVM
heap allocator.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h
index 0be1469e7874..99daa985856f 100644
--- a/arch/arm64/include/asm/kvm_pkvm.h
+++ b/arch/arm64/include/asm/kvm_pkvm.h
@@ -205,6 +205,7 @@ struct pkvm_mapping {
 
 enum pkvm_hyp_req_type {
 	PKVM_HYP_NO_REQ = 0,
+	PKVM_HYP_REQ_HYP_ALLOC,
 	__PKVM_HYP_REQ_TYPE_MAX,
 };
 
@@ -226,9 +227,13 @@ struct pkvm_hyp_req {
 
 static inline size_t pkvm_hyp_req_arg_size(u8 type)
 {
+	struct pkvm_hyp_req *req;
+
 	switch (type) {
 	case PKVM_HYP_NO_REQ:
 		return 0;
+	case PKVM_HYP_REQ_HYP_ALLOC:
+		return sizeof(req->mem);
 	default:
 		WARN_ON(1);
 	}
diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index 5ba8b81b8c6c..adc418076cb0 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -626,6 +626,9 @@ static int pkvm_handle_hyp_req(struct pkvm_hyp_req *req)
 	int ret = -EINVAL;
 
 	switch (req->type) {
+	case PKVM_HYP_REQ_HYP_ALLOC:
+		ret = pkvm_hyp_topup(PKVM_TOPUP_HYP_ALLOC, req->mem.nr_pages);
+		break;
 	}
 
 	trace_kvm_handle_pkvm_hyp_req(req, ret);
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 09/18] KVM: arm64: Add reclaim interface for the pKVM heap alloc

[Vincent Donnefort]

Introduce a host interface to reclaim donated memory from the pKVM heap
allocator back to the host.

It specifically provides two helpers that will make it easier to
create a shrinker for pKVM:

  pkvm_hyp_reclaimable()
  pkvm_hyp_relaim()

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index 681b7bf8ac08..b427ef790b15 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -115,6 +115,8 @@ enum __kvm_host_smccc_func {
 	__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_put,
 	__KVM_HOST_SMCCC_FUNC___pkvm_tlb_flush_vmid,
 	__KVM_HOST_SMCCC_FUNC___pkvm_hyp_topup,
+	__KVM_HOST_SMCCC_FUNC___pkvm_hyp_reclaim,
+	__KVM_HOST_SMCCC_FUNC___pkvm_hyp_reclaimable,
 
 	MARKER(__KVM_HOST_SMCCC_FUNC_MAX)
 };
diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h
index 99daa985856f..3a8590d1793e 100644
--- a/arch/arm64/include/asm/kvm_pkvm.h
+++ b/arch/arm64/include/asm/kvm_pkvm.h
@@ -21,6 +21,9 @@ enum pkvm_topup_id {
 	PKVM_TOPUP_HYP_ALLOC,
 };
 
+unsigned long pkvm_hyp_reclaim(enum pkvm_topup_id id, unsigned long target);
+unsigned long pkvm_hyp_reclaimable(enum pkvm_topup_id id);
+
 int pkvm_init_host_vm(struct kvm *kvm, unsigned long type);
 int pkvm_create_hyp_vm(struct kvm *kvm);
 bool pkvm_hyp_vm_is_created(struct kvm *kvm);
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
index e270a974fa37..8fbebe64a93d 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
@@ -657,6 +657,42 @@ static void handle___pkvm_hyp_topup(struct kvm_cpu_context *host_ctxt)
 	cpu_reg(host_ctxt, 3) = host_mc.nr_pages;
 }
 
+static void handle___pkvm_hyp_reclaim(struct kvm_cpu_context *host_ctxt)
+{
+	DECLARE_REG(enum pkvm_topup_id, id, host_ctxt, 1);
+	DECLARE_REG(unsigned long, target, host_ctxt, 2);
+	struct kvm_hyp_memcache host_mc = {};
+	int ret = 0;
+
+	switch (id) {
+	case PKVM_TOPUP_HYP_ALLOC:
+		hyp_alloc_reclaim(&host_mc, target);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	cpu_reg(host_ctxt, 1) = ret;
+	cpu_reg(host_ctxt, 2) = host_mc.head;
+	cpu_reg(host_ctxt, 3) = host_mc.nr_pages;
+}
+
+static void handle___pkvm_hyp_reclaimable(struct kvm_cpu_context *host_ctxt)
+{
+	DECLARE_REG(enum pkvm_topup_id, id, host_ctxt, 1);
+	unsigned long reclaimable = 0;
+
+	switch (id) {
+	case PKVM_TOPUP_HYP_ALLOC:
+		reclaimable = hyp_alloc_reclaimable();
+		break;
+	default:
+		reclaimable = 0;
+	}
+
+	cpu_reg(host_ctxt, 1) = reclaimable;
+}
+
 static void handle___tracing_load(struct kvm_cpu_context *host_ctxt)
 {
 	DECLARE_REG(unsigned long, desc_hva, host_ctxt, 1);
@@ -788,6 +824,8 @@ static const hcall_t host_hcall[] = {
 	HANDLE_FUNC(__pkvm_vcpu_put),
 	HANDLE_FUNC(__pkvm_tlb_flush_vmid),
 	HANDLE_FUNC(__pkvm_hyp_topup),
+	HANDLE_FUNC(__pkvm_hyp_reclaim),
+	HANDLE_FUNC(__pkvm_hyp_reclaimable),
 };
 
 static void handle_host_hcall(struct kvm_cpu_context *host_ctxt)
diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index adc418076cb0..eb4c60a0db5a 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -115,6 +115,28 @@ static int pkvm_hyp_topup(enum pkvm_topup_id id, unsigned long nr_pages)
 	return ret;
 }
 
+unsigned long pkvm_hyp_reclaim(enum pkvm_topup_id id, unsigned long target)
+{
+	struct kvm_hyp_memcache mc;
+	struct arm_smccc_res res;
+	unsigned long reclaimed;
+
+	arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(__pkvm_hyp_reclaim), id, target, &res);
+	if (WARN_ON_ONCE(res.a0 != SMCCC_RET_SUCCESS) || WARN_ON_ONCE(res.a1))
+		return 0;
+
+	init_hyp_memcache(&mc);
+	mc.head = res.a2;
+	mc.nr_pages = reclaimed = res.a3;
+	free_hyp_memcache(&mc);
+
+	return reclaimed;
+}
+
+unsigned long pkvm_hyp_reclaimable(enum pkvm_topup_id id)
+{
+	return kvm_call_hyp_nvhe(__pkvm_hyp_reclaimable, id);
+}
 static void __pkvm_destroy_hyp_vm(struct kvm *kvm)
 {
 	if (pkvm_hyp_vm_is_created(kvm)) {
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 10/18] KVM: arm64: Add selftests for the pKVM heap allocator

[Vincent Donnefort]

Introduce a comprehensive runtime selftest for the pKVM hypervisor heap
allocator, executed during init when CONFIG_NVHE_EL2_DEBUG is enabled.

The selftest runs entirely at EL2 and exercises allocator's core
mechanisms:

  * over-sized allocations
  * basic allocation and alignment
  * chunk recycling, splitting, merging
  * memory reclaiming
  * memory topup

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index b427ef790b15..07a46860c8b2 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -117,6 +117,7 @@ enum __kvm_host_smccc_func {
 	__KVM_HOST_SMCCC_FUNC___pkvm_hyp_topup,
 	__KVM_HOST_SMCCC_FUNC___pkvm_hyp_reclaim,
 	__KVM_HOST_SMCCC_FUNC___pkvm_hyp_reclaimable,
+	__KVM_HOST_SMCCC_FUNC___pkvm_hyp_alloc_selftest,
 
 	MARKER(__KVM_HOST_SMCCC_FUNC_MAX)
 };
diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h
index 3a8590d1793e..5837c2d7928a 100644
--- a/arch/arm64/include/asm/kvm_pkvm.h
+++ b/arch/arm64/include/asm/kvm_pkvm.h
@@ -19,6 +19,7 @@
 
 enum pkvm_topup_id {
 	PKVM_TOPUP_HYP_ALLOC,
+	PKVM_TOPUP_HYP_ALLOC_SELFTEST,
 };
 
 unsigned long pkvm_hyp_reclaim(enum pkvm_topup_id id, unsigned long target);
@@ -209,6 +210,7 @@ struct pkvm_mapping {
 enum pkvm_hyp_req_type {
 	PKVM_HYP_NO_REQ = 0,
 	PKVM_HYP_REQ_HYP_ALLOC,
+	PKVM_HYP_REQ_HYP_ALLOC_SELFTEST,
 	__PKVM_HYP_REQ_TYPE_MAX,
 };
 
@@ -236,6 +238,7 @@ static inline size_t pkvm_hyp_req_arg_size(u8 type)
 	case PKVM_HYP_NO_REQ:
 		return 0;
 	case PKVM_HYP_REQ_HYP_ALLOC:
+	case PKVM_HYP_REQ_HYP_ALLOC_SELFTEST:
 		return sizeof(req->mem);
 	default:
 		WARN_ON(1);
diff --git a/arch/arm64/kvm/hyp/include/nvhe/alloc.h b/arch/arm64/kvm/hyp/include/nvhe/alloc.h
index 8f87a63f8946..329250dad6f6 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/alloc.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/alloc.h
@@ -14,4 +14,11 @@ int hyp_alloc_init(size_t size);
 int hyp_alloc_topup(struct kvm_hyp_memcache *host_mc);
 unsigned long hyp_alloc_reclaimable(void);
 void hyp_alloc_reclaim(struct kvm_hyp_memcache *host_mc, unsigned long target);
+
+#ifdef CONFIG_NVHE_EL2_DEBUG
+int hyp_allocator_selftest(void);
+u32 hyp_alloc_selftest_topup_needed(void);
+int hyp_alloc_selftest_topup(struct kvm_hyp_memcache *host_mc);
+void hyp_alloc_selftest_reclaim(struct kvm_hyp_memcache *host_mc, unsigned long target);
+#endif
 #endif
diff --git a/arch/arm64/kvm/hyp/nvhe/alloc.c b/arch/arm64/kvm/hyp/nvhe/alloc.c
index 07ef0a13855f..e7114dde9142 100644
--- a/arch/arm64/kvm/hyp/nvhe/alloc.c
+++ b/arch/arm64/kvm/hyp/nvhe/alloc.c
@@ -1007,9 +1007,17 @@ int hyp_alloc_errno(void)
 	return hyp_allocator_errno(&hyp_allocator);
 }
 
+static int selftest_init(void);
+
 int hyp_alloc_init(size_t size)
 {
-	return hyp_allocator_init(&hyp_allocator, size);
+	int ret;
+
+	ret = hyp_allocator_init(&hyp_allocator, size);
+	if (ret)
+		return ret;
+
+	return selftest_init();
 }
 
 void hyp_alloc_reclaim(struct kvm_hyp_memcache *mc, unsigned long target)
@@ -1031,3 +1039,179 @@ u32 hyp_alloc_topup_needed(void)
 {
 	return hyp_allocator_topup_needed(&hyp_allocator);
 }
+
+#ifdef CONFIG_NVHE_EL2_DEBUG
+#define SELFTEST_MAX_PAGES 6
+#define SELFTEST_MAX_SIZE (PAGE_SIZE * SELFTEST_MAX_PAGES)
+
+static DEFINE_PER_CPU(int, __selftest_errno);
+static DEFINE_PER_CPU(u32, __selftest_topup_needed);
+
+static struct hyp_allocator selftest_allocator = {
+	.errno = &__selftest_errno,
+	.topup_needed = &__selftest_topup_needed,
+	.lock = __HYP_SPIN_LOCK_UNLOCKED,
+};
+
+int hyp_alloc_selftest_topup(struct kvm_hyp_memcache *host_mc)
+{
+	return hyp_allocator_topup(&selftest_allocator, host_mc);
+}
+
+void hyp_alloc_selftest_reclaim(struct kvm_hyp_memcache *host_mc, unsigned long target)
+{
+	hyp_allocator_reclaim(&selftest_allocator, host_mc, target);
+}
+
+u32 hyp_alloc_selftest_topup_needed(void)
+{
+	return hyp_allocator_topup_needed(&selftest_allocator);
+}
+
+static int selftest_init(void)
+{
+	return hyp_allocator_init(&selftest_allocator, SELFTEST_MAX_SIZE);
+}
+
+static void *selftest_alloc(size_t size)
+{
+	return hyp_allocator_alloc(&selftest_allocator, size);
+}
+
+static void selftest_free(void *addr)
+{
+	hyp_allocator_free(&selftest_allocator, addr);
+}
+
+static int selftest_errno(void)
+{
+	return hyp_allocator_errno(&selftest_allocator);
+}
+
+int hyp_allocator_selftest(void)
+{
+	struct hyp_allocator *allocator = &selftest_allocator;
+	static DEFINE_HYP_SPINLOCK(selftest_lock);
+	struct kvm_hyp_memcache host_mc = { };
+	void *addr1, *addr2, *addr3, *addr4;
+	int ret;
+
+	guard(hyp_spinlock)(&selftest_lock);
+
+	if (allocator->mc.nr_pages < SELFTEST_MAX_PAGES) {
+		*this_cpu_ptr(allocator->topup_needed) = SELFTEST_MAX_PAGES -
+							 allocator->mc.nr_pages;
+		return -ENOMEM;
+	}
+
+	selftest_alloc(SELFTEST_MAX_SIZE);
+	if (selftest_errno() != -E2BIG)
+		return -EINVAL;
+
+	selftest_alloc(SIZE_MAX);
+	if (selftest_errno() != -E2BIG)
+		return -EINVAL;
+
+	/* Test first chunk */
+	addr1 = selftest_alloc(0);
+	if (!addr1 || addr1 != (void *)allocator->start + chunk_hdr_size())
+		return -EINVAL;
+
+	/* Test second contiguous chunk with unaligned size */
+	addr2 = selftest_alloc(MIN_ALLOC_SIZE + 1);
+	if (!addr2)
+		return -EINVAL;
+	addr3 = selftest_alloc(0);
+	if (!addr3 ||
+	    addr3 != addr2 + (2 * MIN_ALLOC_SIZE) + chunk_hdr_size())
+		return -EINVAL;
+
+	selftest_free(addr3);
+
+	/* Test chunk recycling */
+	selftest_free(addr1);
+	if (addr1 != selftest_alloc(0))
+		return -EINVAL;
+
+	/* Test chunk forward merging */
+	addr3 = selftest_alloc(0);
+	selftest_free(addr2);
+	selftest_free(addr1);
+	if (addr1 != selftest_alloc(MIN_ALLOC_SIZE * 2))
+		return -EINVAL;
+
+	selftest_free(addr1);
+
+	/* Test chunk splitting */
+	if (addr1 != selftest_alloc(0))
+		return -EINVAL;
+	if (addr2 != selftest_alloc(0))
+		return -EINVAL;
+
+	/* Test chunk backward merging */
+	selftest_free(addr1);
+	selftest_free(addr2);
+	if (addr1 != selftest_alloc(MIN_ALLOC_SIZE * 2))
+		return -EINVAL;
+
+	selftest_free(addr1);
+
+	/* Test chunk 3-way merging */
+	addr1 = selftest_alloc(0);
+	addr2 = selftest_alloc(0);
+	addr4 = selftest_alloc(0);
+	selftest_free(addr1);
+	selftest_free(addr3);
+	selftest_free(addr2);
+	if (addr1 != selftest_alloc(MIN_ALLOC_SIZE * 3))
+		return -EINVAL;
+
+	selftest_free(addr4);
+	selftest_free(addr1);
+
+	/* Test reclaiming */
+	if (addr1 != selftest_alloc(0))
+		return -EINVAL;
+	if (addr2 != selftest_alloc(PAGE_SIZE * 2))
+		return -EINVAL;
+	addr3 = selftest_alloc(0);
+	addr4 = selftest_alloc(PAGE_SIZE);
+
+	/* Test reclaiming the last chunk of the list */
+	selftest_free(addr4);
+	hyp_allocator_reclaim(allocator, &host_mc, SELFTEST_MAX_PAGES);
+	if (host_mc.nr_pages != SELFTEST_MAX_PAGES - 3)
+		return -EINVAL;
+
+	/* Test punching a hole in the middle of a free chunk ... */
+	selftest_free(addr2);
+	hyp_allocator_reclaim(allocator, &host_mc, SELFTEST_MAX_PAGES);
+	if (host_mc.nr_pages != SELFTEST_MAX_PAGES - 2)
+		return -EINVAL;
+
+	if (selftest_alloc(PAGE_SIZE))
+		return -EINVAL;
+	if (selftest_errno() != -ENOMEM)
+		return -EINVAL;
+
+	/* ... and to refill this hole */
+	ret = hyp_allocator_topup(allocator, &host_mc);
+	if (ret)
+		return ret;
+	/* Chunk at addr2 was made smaller by the reclaim */
+	if (addr2 != selftest_alloc(PAGE_SIZE))
+		return -EINVAL;
+
+	/* Test reclaiming the entire allocator from the host */
+	selftest_free(addr3);
+	selftest_free(addr2);
+	selftest_free(addr1);
+	if (addr1 != selftest_alloc(SELFTEST_MAX_PAGES * PAGE_SIZE - chunk_hdr_size()))
+		return -EINVAL;
+	selftest_free(addr1);
+
+	return 0;
+}
+#else
+static int selftest_init(void) { return 0; }
+#endif
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
index 8fbebe64a93d..105313bb3dd9 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
@@ -633,6 +633,28 @@ static void handle___pkvm_finalize_teardown_vm(struct kvm_cpu_context *host_ctxt
 	cpu_reg(host_ctxt, 1) = __pkvm_finalize_teardown_vm(handle);
 }
 
+#ifdef CONFIG_NVHE_EL2_DEBUG
+static void handle___pkvm_hyp_alloc_selftest(struct kvm_cpu_context *host_ctxt)
+{
+	struct pkvm_hyp_req req = { .type = PKVM_HYP_NO_REQ };
+	int ret;
+
+	ret = hyp_allocator_selftest();
+	if (ret == -ENOMEM) {
+		req.type = PKVM_HYP_REQ_HYP_ALLOC_SELFTEST;
+		req.mem.nr_pages = hyp_alloc_selftest_topup_needed();
+	}
+
+	cpu_reg(host_ctxt, 1) = ret;
+	pkvm_hyp_req_to_smccc(host_ctxt, &req);
+}
+#else
+static void handle___pkvm_hyp_alloc_selftest(struct kvm_cpu_context *host_ctxt)
+{
+	cpu_reg(host_ctxt, 1) = -EPERM;
+}
+#endif
+
 static void handle___pkvm_hyp_topup(struct kvm_cpu_context *host_ctxt)
 {
 	DECLARE_REG(enum pkvm_topup_id, id, host_ctxt, 1);
@@ -648,6 +670,11 @@ static void handle___pkvm_hyp_topup(struct kvm_cpu_context *host_ctxt)
 	case PKVM_TOPUP_HYP_ALLOC:
 		ret = hyp_alloc_topup(&host_mc);
 		break;
+#ifdef CONFIG_NVHE_EL2_DEBUG
+	case PKVM_TOPUP_HYP_ALLOC_SELFTEST:
+		ret = hyp_alloc_selftest_topup(&host_mc);
+		break;
+#endif
 	default:
 		ret = -EINVAL;
 	}
@@ -668,6 +695,11 @@ static void handle___pkvm_hyp_reclaim(struct kvm_cpu_context *host_ctxt)
 	case PKVM_TOPUP_HYP_ALLOC:
 		hyp_alloc_reclaim(&host_mc, target);
 		break;
+#ifdef CONFIG_NVHE_EL2_DEBUG
+	case PKVM_TOPUP_HYP_ALLOC_SELFTEST:
+		hyp_alloc_selftest_reclaim(&host_mc, target);
+		break;
+#endif
 	default:
 		ret = -EINVAL;
 	}
@@ -826,6 +858,7 @@ static const hcall_t host_hcall[] = {
 	HANDLE_FUNC(__pkvm_hyp_topup),
 	HANDLE_FUNC(__pkvm_hyp_reclaim),
 	HANDLE_FUNC(__pkvm_hyp_reclaimable),
+	HANDLE_FUNC(__pkvm_hyp_alloc_selftest),
 };
 
 static void handle_host_hcall(struct kvm_cpu_context *host_ctxt)
diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index eb4c60a0db5a..f7e1846d6093 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -137,6 +137,7 @@ unsigned long pkvm_hyp_reclaimable(enum pkvm_topup_id id)
 {
 	return kvm_call_hyp_nvhe(__pkvm_hyp_reclaimable, id);
 }
+
 static void __pkvm_destroy_hyp_vm(struct kvm *kvm)
 {
 	if (pkvm_hyp_vm_is_created(kvm)) {
@@ -325,6 +326,22 @@ static int __init pkvm_drop_host_privileges(void)
 	return ret;
 }
 
+static void __init pkvm_selftests(void)
+{
+#ifdef CONFIG_NVHE_EL2_DEBUG
+	int ret = pkvm_call_hyp_req(__pkvm_hyp_alloc_selftest);
+	unsigned long reclaimed;
+
+	reclaimed = pkvm_hyp_reclaim(PKVM_TOPUP_HYP_ALLOC_SELFTEST, ULONG_MAX);
+
+	/* On failure, not all the pages may be reclaimable */
+	if (!ret)
+		WARN_ON(reclaimed != 6 /* SELFTEST_MAX_PAGES */);
+	else
+		kvm_err("pKVM hyp allocator selftest failed (%d)\n", ret);
+#endif
+}
+
 static int __init finalize_pkvm(void)
 {
 	int ret;
@@ -345,6 +362,9 @@ static int __init finalize_pkvm(void)
 	if (ret)
 		pr_err("Failed to finalize Hyp protection: %d\n", ret);
 
+	if (!ret)
+		pkvm_selftests();
+
 	return ret;
 }
 device_initcall_sync(finalize_pkvm);
@@ -651,6 +671,9 @@ static int pkvm_handle_hyp_req(struct pkvm_hyp_req *req)
 	case PKVM_HYP_REQ_HYP_ALLOC:
 		ret = pkvm_hyp_topup(PKVM_TOPUP_HYP_ALLOC, req->mem.nr_pages);
 		break;
+	case PKVM_HYP_REQ_HYP_ALLOC_SELFTEST:
+		ret = pkvm_hyp_topup(PKVM_TOPUP_HYP_ALLOC_SELFTEST, req->mem.nr_pages);
+		break;
 	}
 
 	trace_kvm_handle_pkvm_hyp_req(req, ret);
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 11/18] KVM: arm64: Add a shrinker for pKVM

[Vincent Donnefort]

Integrate the pKVM memory reclaim interface with the host's memory
management subsystem.

This allows the host to automatically recover unused memory fom the
hypervisor's heap allocator when the host is under memory pressure.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 50adfff75be8..3dc2b9d2c322 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -122,6 +122,8 @@ static bool vgic_present, kvm_arm_initialised;
 
 static DEFINE_PER_CPU(unsigned char, kvm_hyp_initialized);
 
+static struct shrinker *pkvm_shrinker;
+
 bool is_kvm_arm_initialised(void)
 {
 	return kvm_arm_initialised;
@@ -2526,8 +2528,8 @@ static void __init teardown_hyp_mode(void)
 		}
 
 		free_pages(kvm_nvhe_sym(kvm_arm_hyp_percpu_base)[cpu], nvhe_percpu_order());
-
 	}
+	shrinker_free(pkvm_shrinker);
 }
 
 static int __init do_pkvm_init(u32 hyp_va_bits)
@@ -2713,6 +2715,18 @@ static void pkvm_hyp_init_ptrauth(void)
 	}
 }
 
+static unsigned long
+pkvm_shrinker_count(struct shrinker *shrink, struct shrink_control *sc)
+{
+	return pkvm_hyp_reclaimable(PKVM_TOPUP_HYP_ALLOC) ?: SHRINK_EMPTY;
+}
+
+static unsigned long
+pkvm_shrinker_scan(struct shrinker *shrink, struct shrink_control *sc)
+{
+	return pkvm_hyp_reclaim(PKVM_TOPUP_HYP_ALLOC, sc->nr_to_scan);
+}
+
 /* Inits Hyp-mode on all online CPUs */
 static int __init init_hyp_mode(void)
 {
@@ -2883,6 +2897,16 @@ static int __init init_hyp_mode(void)
 			kvm_err("Failed to init hyp memory protection\n");
 			goto out_err;
 		}
+
+		pkvm_shrinker = shrinker_alloc(0, "pkvm");
+		if (pkvm_shrinker) {
+			pkvm_shrinker->count_objects = pkvm_shrinker_count;
+			pkvm_shrinker->scan_objects = pkvm_shrinker_scan;
+			shrinker_register(pkvm_shrinker);
+		} else {
+			kvm_err("Failed to register shrinker for pKVM\n");
+		}
+
 	}
 
 	return 0;
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 12/18] KVM: arm64: Filter out non-kernel addresses in kern_hyp_va

[Vincent Donnefort]

kern_hyp_va() is idempotent for the hypervisor linear space. This is
handy for nVHE hypervisor callers handling kvm_vcpu or kvm_arch
pointers. Those pointers can originate from the hypervisor space (when
protected mode is enabled, we don't trust the kernel and the hypervisor
uses its own copy) or from the kernel space (we do trust the kernel in
"non-protected" nVHE).

This idempotence does not hold for addresses within the hypervisor
private range, like the ones you get from the pKVM heap allocator
(hyp_alloc()). To resolve this, filter out non-kernel addresses based on
PAGE_OFFSET.

Leave the assembly version untouched as it has no current users.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 6eae7e7e2a68..d60e5f2de10c 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -126,6 +126,9 @@ static __always_inline unsigned long __kern_hyp_va(unsigned long v)
  * replace the instructions with `nop`s.
  */
 #ifndef __KVM_VHE_HYPERVISOR__
+	if (!is_ttbr1_addr(v))
+		return v;
+
 	asm volatile(ALTERNATIVE_CB("and %0, %0, #1\n"         /* mask with va_mask */
 				    "ror %0, %0, #1\n"         /* rotate to the first tag bit */
 				    "add %0, %0, #0\n"         /* insert the low 12 bits of the tag */
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 13/18] KVM: arm64: Move hyp_vm refcount into the structure

[Vincent Donnefort]

In preparation for allocating hyp_vm using the pKVM heap allocator
(hyp_alloc()), move its reference count out of the page metadata
(vmemmap) and place it into the structure itself. This transition is
necessary because hyp_alloc() allows multiple small objects to share the
same physical page.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
index c904647d2f76..624367d0ef5b 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
@@ -41,6 +41,7 @@ struct pkvm_hyp_vm {
 	struct kvm_pgtable pgt;
 	struct kvm_pgtable_mm_ops mm_ops;
 	struct hyp_pool pool;
+	unsigned short refcount;
 	hyp_spinlock_t lock;
 
 	/* Array of the hyp vCPU structures for this VM. */
@@ -65,6 +66,18 @@ static inline bool pkvm_hyp_vm_is_protected(struct pkvm_hyp_vm *hyp_vm)
 	return kvm_vm_is_protected(&hyp_vm->kvm);
 }
 
+static inline void pkvm_hyp_vm_ref_inc(struct pkvm_hyp_vm *hyp_vm)
+{
+	BUG_ON(hyp_vm->refcount == USHRT_MAX);
+	hyp_vm->refcount++;
+}
+
+static inline void pkvm_hyp_vm_ref_dec(struct pkvm_hyp_vm *hyp_vm)
+{
+	BUG_ON(!hyp_vm->refcount);
+	hyp_vm->refcount--;
+}
+
 void pkvm_hyp_vm_table_init(void *tbl);
 
 int __pkvm_reserve_vm(void);
diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c
index 3b2c4fbc34d8..d9ab58226889 100644
--- a/arch/arm64/kvm/hyp/nvhe/pkvm.c
+++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c
@@ -278,7 +278,7 @@ struct pkvm_hyp_vcpu *pkvm_load_hyp_vcpu(pkvm_handle_t handle,
 	}
 
 	hyp_vcpu->loaded_hyp_vcpu = this_cpu_ptr(&loaded_hyp_vcpu);
-	hyp_page_ref_inc(hyp_virt_to_page(hyp_vm));
+	pkvm_hyp_vm_ref_inc(hyp_vm);
 unlock:
 	hyp_spin_unlock(&vm_table_lock);
 
@@ -294,7 +294,7 @@ void pkvm_put_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
 	hyp_spin_lock(&vm_table_lock);
 	hyp_vcpu->loaded_hyp_vcpu = NULL;
 	__this_cpu_write(loaded_hyp_vcpu, NULL);
-	hyp_page_ref_dec(hyp_virt_to_page(hyp_vm));
+	pkvm_hyp_vm_ref_dec(hyp_vm);
 	hyp_spin_unlock(&vm_table_lock);
 }
 
@@ -311,7 +311,7 @@ struct pkvm_hyp_vm *get_pkvm_hyp_vm(pkvm_handle_t handle)
 	hyp_spin_lock(&vm_table_lock);
 	hyp_vm = get_vm_by_handle(handle);
 	if (hyp_vm)
-		hyp_page_ref_inc(hyp_virt_to_page(hyp_vm));
+		pkvm_hyp_vm_ref_inc(hyp_vm);
 	hyp_spin_unlock(&vm_table_lock);
 
 	return hyp_vm;
@@ -320,7 +320,7 @@ struct pkvm_hyp_vm *get_pkvm_hyp_vm(pkvm_handle_t handle)
 void put_pkvm_hyp_vm(struct pkvm_hyp_vm *hyp_vm)
 {
 	hyp_spin_lock(&vm_table_lock);
-	hyp_page_ref_dec(hyp_virt_to_page(hyp_vm));
+	pkvm_hyp_vm_ref_dec(hyp_vm);
 	hyp_spin_unlock(&vm_table_lock);
 }
 
@@ -966,7 +966,7 @@ static struct pkvm_hyp_vm *get_pkvm_unref_hyp_vm_locked(pkvm_handle_t handle)
 	hyp_assert_lock_held(&vm_table_lock);
 
 	hyp_vm = get_vm_by_handle(handle);
-	if (!hyp_vm || hyp_page_count(hyp_vm))
+	if (!hyp_vm || hyp_vm->refcount)
 		return NULL;
 
 	return hyp_vm;
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 14/18] KVM: arm64: Use noclear for PGD in __pkvm_init_vm error path

[Vincent Donnefort]

In the error path of __pkvm_init_vm(), use unmap_donated_memory_noclear()
instead of the clearing variant to release the donated stage-2 PGD back
to the host.

This intends to eliminate the clearing variant of
unmap_donated_memory(), as zeroing the PGD memory before returning it to
the host is unnecessary in this failure path.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c
index d9ab58226889..0c82f1ddb93d 100644
--- a/arch/arm64/kvm/hyp/nvhe/pkvm.c
+++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c
@@ -861,7 +861,7 @@ int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva,
 	kvm_guest_destroy_stage2(hyp_vm);
 err_remove_mappings:
 	unmap_donated_memory(hyp_vm, vm_size);
-	unmap_donated_memory(pgd, pgd_size);
+	unmap_donated_memory_noclear(pgd, pgd_size);
 err_unpin_kvm:
 	hyp_unpin_shared_mem(host_kvm, host_kvm + 1);
 	return ret;
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 15/18] KVM: arm64: Alloc pkvm_hyp_vm using pKVM heap allocator

[Vincent Donnefort]

Transition the allocation of the hypervisor VM state structure
(pkvm_hyp_vm) from the host to the hypervisor using
the new pKVM heap allocator (hyp_alloc()).

Previously, the host was responsible for calculating the size of,
allocating, and donating memory for pkvm_hyp_vm during VM creation. With
the heap allocator in place, the hypervisor now allocates this structure
dynamically at EL2.

Use the pkvm_call_hyp_req() wrapper in the host to invoke
__pkvm_init_vm, which automatically handles any top-up requests if the
hypervisor runs out of heap memory during allocation.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/kvm/hyp/hyp-constants.c b/arch/arm64/kvm/hyp/hyp-constants.c
index b257a3b4bfc5..501ab35a3840 100644
--- a/arch/arm64/kvm/hyp/hyp-constants.c
+++ b/arch/arm64/kvm/hyp/hyp-constants.c
@@ -7,7 +7,6 @@
 int main(void)
 {
 	DEFINE(STRUCT_HYP_PAGE_SIZE,	sizeof(struct hyp_page));
-	DEFINE(PKVM_HYP_VM_SIZE,	sizeof(struct pkvm_hyp_vm));
 	DEFINE(PKVM_HYP_VCPU_SIZE,	sizeof(struct pkvm_hyp_vcpu));
 	return 0;
 }
diff --git a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
index 624367d0ef5b..8e930c8729af 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
@@ -82,8 +82,7 @@ void pkvm_hyp_vm_table_init(void *tbl);
 
 int __pkvm_reserve_vm(void);
 void __pkvm_unreserve_vm(pkvm_handle_t handle);
-int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva,
-		   unsigned long pgd_hva);
+int __pkvm_init_vm(struct kvm *host_kvm, void *pgd);
 int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu,
 		     unsigned long vcpu_hva);
 
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
index 105313bb3dd9..16f963aedd70 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
@@ -575,14 +575,30 @@ static void handle___pkvm_unreserve_vm(struct kvm_cpu_context *host_ctxt)
 	__pkvm_unreserve_vm(handle);
 }
 
+static void errno_to_smccc(int ret, struct kvm_cpu_context *host_ctxt)
+{
+	struct pkvm_hyp_req req = { .type = PKVM_HYP_NO_REQ };
+
+	switch (ret) {
+	case -ENOMEM:
+		req.type = PKVM_HYP_REQ_HYP_ALLOC;
+		req.mem.nr_pages = hyp_alloc_topup_needed();
+		break;
+	}
+
+	cpu_reg(host_ctxt, 1) = ret;
+	pkvm_hyp_req_to_smccc(host_ctxt, &req);
+}
+
 static void handle___pkvm_init_vm(struct kvm_cpu_context *host_ctxt)
 {
 	DECLARE_REG(struct kvm *, host_kvm, host_ctxt, 1);
-	DECLARE_REG(unsigned long, vm_hva, host_ctxt, 2);
-	DECLARE_REG(unsigned long, pgd_hva, host_ctxt, 3);
+	DECLARE_REG(unsigned long, pgd_hva, host_ctxt, 2);
+	void *pgd;
 
 	host_kvm = kern_hyp_va(host_kvm);
-	cpu_reg(host_ctxt, 1) = __pkvm_init_vm(host_kvm, vm_hva, pgd_hva);
+	pgd = (void *)kern_hyp_va(pgd_hva);
+	errno_to_smccc(__pkvm_init_vm(host_kvm, pgd), host_ctxt);
 }
 
 static void handle___pkvm_init_vcpu(struct kvm_cpu_context *host_ctxt)
diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c
index 0c82f1ddb93d..2f616a89f8b3 100644
--- a/arch/arm64/kvm/hyp/nvhe/pkvm.c
+++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c
@@ -11,6 +11,7 @@
 
 #include <asm/kvm_emulate.h>
 
+#include <nvhe/alloc.h>
 #include <nvhe/mem_protect.h>
 #include <nvhe/memory.h>
 #include <nvhe/pkvm.h>
@@ -797,24 +798,22 @@ void teardown_selftest_vm(void)
  * Unmap the donated memory from the host at stage 2.
  *
  * host_kvm: A pointer to the host's struct kvm.
- * vm_hva: The host va of the area being donated for the VM state.
- *	   Must be page aligned.
- * pgd_hva: The host va of the area being donated for the stage-2 PGD for
- *	    the VM. Must be page aligned. Its size is implied by the VM's
- *	    VTCR.
+ * pgd: The va of the area being donated for the stage-2 PGD for the VM. Must
+ *      be page aligned. Its size is implied by the VM's VTCR.
  *
  * Return 0 success, negative error code on failure.
  */
-int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva,
-		   unsigned long pgd_hva)
+int __pkvm_init_vm(struct kvm *host_kvm, void *pgd)
 {
 	struct pkvm_hyp_vm *hyp_vm = NULL;
 	size_t vm_size, pgd_size;
 	unsigned int nr_vcpus;
 	pkvm_handle_t handle;
-	void *pgd = NULL;
 	int ret;
 
+	if (!PAGE_ALIGNED(pgd))
+		return -EINVAL;
+
 	ret = hyp_pin_shared_mem(host_kvm, host_kvm + 1);
 	if (ret)
 		return ret;
@@ -834,15 +833,15 @@ int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva,
 	vm_size = pkvm_get_hyp_vm_size(nr_vcpus);
 	pgd_size = kvm_pgtable_stage2_pgd_size(host_mmu.arch.mmu.vtcr);
 
-	ret = -ENOMEM;
+	hyp_vm = hyp_alloc(vm_size);
+	if (!hyp_vm) {
+		ret = hyp_alloc_errno();
+		goto err_unpin_kvm;
+	}
 
-	hyp_vm = map_donated_memory(vm_hva, vm_size);
-	if (!hyp_vm)
-		goto err_remove_mappings;
-
-	pgd = map_donated_memory_noclear(pgd_hva, pgd_size);
-	if (!pgd)
-		goto err_remove_mappings;
+	ret = __pkvm_host_donate_hyp(hyp_virt_to_pfn(pgd), PAGE_ALIGN(pgd_size) >> PAGE_SHIFT);
+	if (ret)
+		goto err_free_hyp_vm;
 
 	init_pkvm_hyp_vm(host_kvm, hyp_vm, nr_vcpus, handle);
 
@@ -860,8 +859,9 @@ int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva,
 err_destroy_stage2:
 	kvm_guest_destroy_stage2(hyp_vm);
 err_remove_mappings:
-	unmap_donated_memory(hyp_vm, vm_size);
 	unmap_donated_memory_noclear(pgd, pgd_size);
+err_free_hyp_vm:
+	hyp_free(hyp_vm);
 err_unpin_kvm:
 	hyp_unpin_shared_mem(host_kvm, host_kvm + 1);
 	return ret;
@@ -997,7 +997,6 @@ int __pkvm_finalize_teardown_vm(pkvm_handle_t handle)
 	struct pkvm_hyp_vm *hyp_vm;
 	struct kvm *host_kvm;
 	unsigned int idx;
-	size_t vm_size;
 	int err;
 
 	hyp_spin_lock(&vm_table_lock);
@@ -1040,8 +1039,7 @@ int __pkvm_finalize_teardown_vm(pkvm_handle_t handle)
 		teardown_donated_memory(mc, hyp_vcpu, sizeof(*hyp_vcpu));
 	}
 
-	vm_size = pkvm_get_hyp_vm_size(hyp_vm->kvm.created_vcpus);
-	teardown_donated_memory(mc, hyp_vm, vm_size);
+	hyp_free(hyp_vm);
 	hyp_unpin_shared_mem(host_kvm, host_kvm + 1);
 	return 0;
 
diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index f7e1846d6093..8a8b5b84825c 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -191,8 +191,8 @@ static int __pkvm_create_hyp_vcpu(struct kvm_vcpu *vcpu)
  */
 static int __pkvm_create_hyp_vm(struct kvm *kvm)
 {
-	size_t pgd_sz, hyp_vm_sz;
-	void *pgd, *hyp_vm;
+	size_t pgd_sz;
+	void *pgd;
 	int ret;
 
 	if (kvm->created_vcpus < 1)
@@ -209,28 +209,15 @@ static int __pkvm_create_hyp_vm(struct kvm *kvm)
 	if (!pgd)
 		return -ENOMEM;
 
-	/* Allocate memory to donate to hyp for vm and vcpu pointers. */
-	hyp_vm_sz = PAGE_ALIGN(size_add(PKVM_HYP_VM_SIZE,
-					size_mul(sizeof(void *),
-						 kvm->created_vcpus)));
-	hyp_vm = alloc_pages_exact(hyp_vm_sz, GFP_KERNEL_ACCOUNT);
-	if (!hyp_vm) {
-		ret = -ENOMEM;
-		goto free_pgd;
-	}
-
-	/* Donate the VM memory to hyp and let hyp initialize it. */
-	ret = kvm_call_hyp_nvhe(__pkvm_init_vm, kvm, hyp_vm, pgd);
+	ret = pkvm_call_hyp_req(__pkvm_init_vm, kvm, pgd);
 	if (ret)
-		goto free_vm;
+		goto free_pgd;
 
 	kvm->arch.pkvm.is_created = true;
 	init_hyp_stage2_memcache(&kvm->arch.pkvm.stage2_teardown_mc);
 	kvm_account_pgtable_pages(pgd, pgd_sz / PAGE_SIZE);
 
 	return 0;
-free_vm:
-	free_pages_exact(hyp_vm, hyp_vm_sz);
 free_pgd:
 	free_pages_exact(pgd, pgd_sz);
 	return ret;
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 16/18] KVM: arm64: Alloc pkvm_hyp_vcpu using pKVM heap allocator

[Vincent Donnefort]

Transition the allocation of the hypervisor vCPU state structure
(pkvm_hyp_vcpu) from the host to the hypervisor using the new pKVM heap
allocator (hyp_alloc()).

Previously, the host was responsible for calculating the size of,
allocating, and donating memory for pkvm_hyp_vcpu during VM creation.
With the heap allocator in place, the hypervisor now allocates this
structure dynamically at EL2.

Use the pkvm_call_hyp_req() wrapper in the host to invoke
__pkvm_create_hyp_vcpu, which automatically handles any top-up requests
if the hypervisor runs out of heap memory during allocation.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index caea8e9986d1..95e05cee0f3a 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -268,7 +268,6 @@ typedef u16 pkvm_handle_t;
 
 struct kvm_protected_vm {
 	pkvm_handle_t handle;
-	struct kvm_hyp_memcache teardown_mc;
 	struct kvm_hyp_memcache stage2_teardown_mc;
 	bool is_protected;
 	bool is_created;
diff --git a/arch/arm64/kvm/hyp/hyp-constants.c b/arch/arm64/kvm/hyp/hyp-constants.c
index 501ab35a3840..b2caae21f271 100644
--- a/arch/arm64/kvm/hyp/hyp-constants.c
+++ b/arch/arm64/kvm/hyp/hyp-constants.c
@@ -7,6 +7,5 @@
 int main(void)
 {
 	DEFINE(STRUCT_HYP_PAGE_SIZE,	sizeof(struct hyp_page));
-	DEFINE(PKVM_HYP_VCPU_SIZE,	sizeof(struct pkvm_hyp_vcpu));
 	return 0;
 }
diff --git a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
index 8e930c8729af..cfb6e409bf49 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
@@ -83,8 +83,7 @@ void pkvm_hyp_vm_table_init(void *tbl);
 int __pkvm_reserve_vm(void);
 void __pkvm_unreserve_vm(pkvm_handle_t handle);
 int __pkvm_init_vm(struct kvm *host_kvm, void *pgd);
-int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu,
-		     unsigned long vcpu_hva);
+int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu);
 
 int __pkvm_reclaim_dying_guest_page(pkvm_handle_t handle, u64 gfn);
 int __pkvm_start_teardown_vm(pkvm_handle_t handle);
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
index 16f963aedd70..1315129dcd1f 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
@@ -605,10 +605,9 @@ static void handle___pkvm_init_vcpu(struct kvm_cpu_context *host_ctxt)
 {
 	DECLARE_REG(pkvm_handle_t, handle, host_ctxt, 1);
 	DECLARE_REG(struct kvm_vcpu *, host_vcpu, host_ctxt, 2);
-	DECLARE_REG(unsigned long, vcpu_hva, host_ctxt, 3);
 
 	host_vcpu = kern_hyp_va(host_vcpu);
-	cpu_reg(host_ctxt, 1) = __pkvm_init_vcpu(handle, host_vcpu, vcpu_hva);
+	errno_to_smccc(__pkvm_init_vcpu(handle, host_vcpu), host_ctxt);
 }
 
 static void handle___pkvm_vcpu_in_poison_fault(struct kvm_cpu_context *host_ctxt)
diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c
index 2f616a89f8b3..45ec77e9eca8 100644
--- a/arch/arm64/kvm/hyp/nvhe/pkvm.c
+++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c
@@ -645,30 +645,6 @@ static size_t pkvm_get_hyp_vm_size(unsigned int nr_vcpus)
 		size_mul(sizeof(struct pkvm_hyp_vcpu *), nr_vcpus));
 }
 
-static void *map_donated_memory_noclear(unsigned long host_va, size_t size)
-{
-	void *va = (void *)kern_hyp_va(host_va);
-
-	if (!PAGE_ALIGNED(va))
-		return NULL;
-
-	if (__pkvm_host_donate_hyp(hyp_virt_to_pfn(va),
-				   PAGE_ALIGN(size) >> PAGE_SHIFT))
-		return NULL;
-
-	return va;
-}
-
-static void *map_donated_memory(unsigned long host_va, size_t size)
-{
-	void *va = map_donated_memory_noclear(host_va, size);
-
-	if (va)
-		memset(va, 0, size);
-
-	return va;
-}
-
 static void __unmap_donated_memory(void *va, size_t size)
 {
 	kvm_flush_dcache_to_poc(va, size);
@@ -676,15 +652,6 @@ static void __unmap_donated_memory(void *va, size_t size)
 				       PAGE_ALIGN(size) >> PAGE_SHIFT));
 }
 
-static void unmap_donated_memory(void *va, size_t size)
-{
-	if (!va)
-		return;
-
-	memset(va, 0, size);
-	__unmap_donated_memory(va, size);
-}
-
 static void unmap_donated_memory_noclear(void *va, size_t size)
 {
 	if (!va)
@@ -896,16 +863,15 @@ static int register_hyp_vcpu(struct pkvm_hyp_vm *hyp_vm,
 	return 0;
 }
 
-int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu,
-		     unsigned long vcpu_hva)
+int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu)
 {
 	struct pkvm_hyp_vcpu *hyp_vcpu;
 	struct pkvm_hyp_vm *hyp_vm;
 	int ret;
 
-	hyp_vcpu = map_donated_memory(vcpu_hva, sizeof(*hyp_vcpu));
+	hyp_vcpu = hyp_alloc(sizeof(*hyp_vcpu));
 	if (!hyp_vcpu)
-		return -ENOMEM;
+		return hyp_alloc_errno();
 
 	hyp_spin_lock(&vm_table_lock);
 
@@ -926,24 +892,12 @@ int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu,
 	}
 unlock:
 	hyp_spin_unlock(&vm_table_lock);
-
 	if (ret)
-		unmap_donated_memory(hyp_vcpu, sizeof(*hyp_vcpu));
+		hyp_free(hyp_vcpu);
+
 	return ret;
 }
 
-static void
-teardown_donated_memory(struct kvm_hyp_memcache *mc, void *addr, size_t size)
-{
-	size = PAGE_ALIGN(size);
-	memset(addr, 0, size);
-
-	for (void *start = addr; start < addr + size; start += PAGE_SIZE)
-		push_hyp_memcache(mc, start, hyp_virt_to_phys);
-
-	unmap_donated_memory_noclear(addr, size);
-}
-
 int __pkvm_reclaim_dying_guest_page(pkvm_handle_t handle, u64 gfn)
 {
 	struct pkvm_hyp_vm *hyp_vm = get_pkvm_hyp_vm(handle);
@@ -993,7 +947,7 @@ int __pkvm_start_teardown_vm(pkvm_handle_t handle)
 
 int __pkvm_finalize_teardown_vm(pkvm_handle_t handle)
 {
-	struct kvm_hyp_memcache *mc, *stage2_mc;
+	struct kvm_hyp_memcache *stage2_mc;
 	struct pkvm_hyp_vm *hyp_vm;
 	struct kvm *host_kvm;
 	unsigned int idx;
@@ -1014,7 +968,6 @@ int __pkvm_finalize_teardown_vm(pkvm_handle_t handle)
 	hyp_spin_unlock(&vm_table_lock);
 
 	/* Reclaim guest pages (including page-table pages) */
-	mc = &host_kvm->arch.pkvm.teardown_mc;
 	stage2_mc = &host_kvm->arch.pkvm.stage2_teardown_mc;
 	reclaim_pgtable_pages(hyp_vm, stage2_mc);
 	unpin_host_vcpus(hyp_vm->vcpus, hyp_vm->kvm.created_vcpus);
@@ -1036,7 +989,7 @@ int __pkvm_finalize_teardown_vm(pkvm_handle_t handle)
 			unmap_donated_memory_noclear(addr, PAGE_SIZE);
 		}
 
-		teardown_donated_memory(mc, hyp_vcpu, sizeof(*hyp_vcpu));
+		hyp_free(hyp_vcpu);
 	}
 
 	hyp_free(hyp_vm);
diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index 8a8b5b84825c..1899b05d9fc2 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -153,28 +153,19 @@ static void __pkvm_destroy_hyp_vm(struct kvm *kvm)
 
 	kvm->arch.pkvm.handle = 0;
 	kvm->arch.pkvm.is_created = false;
-	free_hyp_memcache(&kvm->arch.pkvm.teardown_mc);
 	free_hyp_memcache(&kvm->arch.pkvm.stage2_teardown_mc);
 }
 
 static int __pkvm_create_hyp_vcpu(struct kvm_vcpu *vcpu)
 {
-	size_t hyp_vcpu_sz = PAGE_ALIGN(PKVM_HYP_VCPU_SIZE);
 	pkvm_handle_t handle = vcpu->kvm->arch.pkvm.handle;
-	void *hyp_vcpu;
 	int ret;
 
 	init_hyp_stage2_memcache(&vcpu->arch.pkvm_memcache);
 
-	hyp_vcpu = alloc_pages_exact(hyp_vcpu_sz, GFP_KERNEL_ACCOUNT);
-	if (!hyp_vcpu)
-		return -ENOMEM;
-
-	ret = kvm_call_hyp_nvhe(__pkvm_init_vcpu, handle, vcpu, hyp_vcpu);
+	ret = pkvm_call_hyp_req(__pkvm_init_vcpu, handle, vcpu);
 	if (!ret)
 		vcpu_set_flag(vcpu, VCPU_PKVM_FINALIZED);
-	else
-		free_pages_exact(hyp_vcpu, hyp_vcpu_sz);
 
 	return ret;
 }
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 17/18] KVM: arm64: Reject hyp trace descriptors with fewer CPUs than hyp_nr_cpus

[Vincent Donnefort]

While a trace descriptor with fewer CPUs than hyp_nr_cpus currently
causes no functional issue, such a configuration is invalid and would
cause the hypervisor to skip ring buffer initialization when allocating
backing pages in EL2.

Enforce nr_cpus == hyp_nr_cpus during trace descriptor validation to
guarantee that every possible CPU has a valid ring buffer descriptor.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/kvm/hyp/nvhe/trace.c b/arch/arm64/kvm/hyp/nvhe/trace.c
index e7e150ab265f..6e716295247a 100644
--- a/arch/arm64/kvm/hyp/nvhe/trace.c
+++ b/arch/arm64/kvm/hyp/nvhe/trace.c
@@ -181,6 +181,9 @@ static bool hyp_trace_desc_is_valid(struct hyp_trace_desc *desc, size_t desc_siz
 	desc_end = (void *)desc + desc_size;
 	nr_bpages = desc->bpages_backing_size / sizeof(struct simple_buffer_page);
 
+	if (desc->trace_buffer_desc.nr_cpus != hyp_nr_cpus)
+		return false;
+
 	for_each_ring_buffer_desc(rb_desc, cpu, &desc->trace_buffer_desc) {
 		/* Can we read nr_page_va? */
 		if ((void *)rb_desc + struct_size(rb_desc, page_va, 0) > desc_end)
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

[PATCH v2 18/18] KVM: arm64: Alloc simple_buffer_page using pKVM hyp allocator

[Vincent Donnefort]

In protected mode, transition the allocation of the simple_ring_buffer
structures from the host to the hypervisor using the new pKVM heap
allocator.

Previously, the host allocated and donated a contiguous backing memory
for these structures. In pKVM the hypervisor can now allocate them
dynamically.

Use the pkvm_call_hyp_req() wrapper in the host to invoke
__tracing_load, which automatically handles any top-up requests if the
hypervisor runs out of heap memory during allocation.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
index 1315129dcd1f..090319bddf86 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
@@ -745,7 +745,7 @@ static void handle___tracing_load(struct kvm_cpu_context *host_ctxt)
 	DECLARE_REG(unsigned long, desc_hva, host_ctxt, 1);
 	DECLARE_REG(size_t, desc_size, host_ctxt, 2);
 
-	cpu_reg(host_ctxt, 1) = __tracing_load(desc_hva, desc_size);
+	errno_to_smccc(__tracing_load(desc_hva, desc_size), host_ctxt);
 }
 
 static void handle___tracing_unload(struct kvm_cpu_context *host_ctxt)
diff --git a/arch/arm64/kvm/hyp/nvhe/trace.c b/arch/arm64/kvm/hyp/nvhe/trace.c
index 6e716295247a..bcfe06bba29b 100644
--- a/arch/arm64/kvm/hyp/nvhe/trace.c
+++ b/arch/arm64/kvm/hyp/nvhe/trace.c
@@ -4,6 +4,7 @@
  * Author: Vincent Donnefort <vdonnefort@google.com>
  */
 
+#include <nvhe/alloc.h>
 #include <nvhe/clock.h>
 #include <nvhe/mem_protect.h>
 #include <nvhe/mm.h>
@@ -62,18 +63,34 @@ static void __release_host_mem(void *start, u64 size)
 	WARN_ON(__pkvm_hyp_donate_host(hyp_virt_to_pfn(start), size >> PAGE_SHIFT));
 }
 
-static int hyp_trace_buffer_load_bpage_backing(struct hyp_trace_buffer *trace_buffer,
-					       struct hyp_trace_desc *desc)
+static int hyp_trace_buffer_alloc_bpages(struct hyp_trace_buffer *trace_buffer,
+					 struct hyp_trace_desc *desc)
 {
-	void *start = (void *)kern_hyp_va(desc->bpages_backing_start);
-	size_t size = desc->bpages_backing_size;
+	void *start;
+	size_t size;
 	int ret;
 
-	ret = __admit_host_mem(start, size);
-	if (ret)
-		return ret;
+	if (is_protected_kvm_enabled()) {
+		struct ring_buffer_desc *rb_desc;
+		int cpu;
 
-	memset(start, 0, size);
+		size = 0;
+		for_each_ring_buffer_desc(rb_desc, cpu, &desc->trace_buffer_desc)
+			size += rb_desc->nr_page_va * sizeof(struct simple_buffer_page);
+
+		start = hyp_alloc(size);
+		if (!start)
+			return hyp_alloc_errno();
+	} else {
+		start = (void *)kern_hyp_va(desc->bpages_backing_start);
+		size = desc->bpages_backing_size;
+
+		ret = __admit_host_mem(start, size);
+		if (ret)
+			return ret;
+
+		memset(start, 0, size);
+	}
 
 	trace_buffer->bpages_backing_start = start;
 	trace_buffer->bpages_backing_size = size;
@@ -81,7 +98,7 @@ static int hyp_trace_buffer_load_bpage_backing(struct hyp_trace_buffer *trace_bu
 	return 0;
 }
 
-static void hyp_trace_buffer_unload_bpage_backing(struct hyp_trace_buffer *trace_buffer)
+static void hyp_trace_buffer_free_bpages(struct hyp_trace_buffer *trace_buffer)
 {
 	void *start = trace_buffer->bpages_backing_start;
 	size_t size = trace_buffer->bpages_backing_size;
@@ -89,9 +106,12 @@ static void hyp_trace_buffer_unload_bpage_backing(struct hyp_trace_buffer *trace
 	if (!size)
 		return;
 
-	memset(start, 0, size);
-
-	__release_host_mem(start, size);
+	if (is_protected_kvm_enabled()) {
+		hyp_free(start);
+	} else {
+		memset(start, 0, size);
+		__release_host_mem(start, size);
+	}
 
 	trace_buffer->bpages_backing_start = 0;
 	trace_buffer->bpages_backing_size = 0;
@@ -128,7 +148,7 @@ static void hyp_trace_buffer_unload(struct hyp_trace_buffer *trace_buffer)
 		simple_ring_buffer_unload_mm(per_cpu_ptr(trace_buffer->simple_rbs, cpu),
 					     __unpin_shared_page);
 
-	hyp_trace_buffer_unload_bpage_backing(trace_buffer);
+	hyp_trace_buffer_free_bpages(trace_buffer);
 }
 
 static int hyp_trace_buffer_load(struct hyp_trace_buffer *trace_buffer,
@@ -143,7 +163,7 @@ static int hyp_trace_buffer_load(struct hyp_trace_buffer *trace_buffer,
 	if (hyp_trace_buffer_loaded(trace_buffer))
 		return -EINVAL;
 
-	ret = hyp_trace_buffer_load_bpage_backing(trace_buffer, desc);
+	ret = hyp_trace_buffer_alloc_bpages(trace_buffer, desc);
 	if (ret)
 		return ret;
 
@@ -168,18 +188,16 @@ static bool hyp_trace_desc_is_valid(struct hyp_trace_desc *desc, size_t desc_siz
 {
 	struct ring_buffer_desc *rb_desc;
 	unsigned int cpu;
-	size_t nr_bpages;
 	void *desc_end;
 
 	if (!is_protected_kvm_enabled())
 		return true;
 
 	/*
-	 * Both desc_size and bpages_backing_size are untrusted host-provided
-	 * values. We rely on __pkvm_host_donate_hyp() to enforce their validity.
+	 * desc_size is an untrusted host-provided value. We rely on
+	 * __pkvm_host_donate_hyp() to enforce its validity.
 	 */
 	desc_end = (void *)desc + desc_size;
-	nr_bpages = desc->bpages_backing_size / sizeof(struct simple_buffer_page);
 
 	if (desc->trace_buffer_desc.nr_cpus != hyp_nr_cpus)
 		return false;
@@ -193,17 +211,8 @@ static bool hyp_trace_desc_is_valid(struct hyp_trace_desc *desc, size_t desc_siz
 		if ((void *)rb_desc + struct_size(rb_desc, page_va, rb_desc->nr_page_va) > desc_end)
 			return false;
 
-		/* Overflow bpages backing memory? */
-		if (nr_bpages < rb_desc->nr_page_va)
-			return false;
-
-		if (cpu >= hyp_nr_cpus)
-			return false;
-
 		if (cpu != rb_desc->cpu)
 			return false;
-
-		nr_bpages -= rb_desc->nr_page_va;
 	}
 
 	return true;
diff --git a/arch/arm64/kvm/hyp_trace.c b/arch/arm64/kvm/hyp_trace.c
index 2411b4c32932..405ccd954524 100644
--- a/arch/arm64/kvm/hyp_trace.c
+++ b/arch/arm64/kvm/hyp_trace.c
@@ -13,6 +13,7 @@
 #include <asm/kvm_host.h>
 #include <asm/kvm_hyptrace.h>
 #include <asm/kvm_mmu.h>
+#include <asm/kvm_pkvm.h>
 
 #include "hyp_trace.h"
 
@@ -157,10 +158,18 @@ static void __unshare_page(unsigned long va)
 
 static int hyp_trace_buffer_alloc_bpages_backing(struct hyp_trace_buffer *trace_buffer, size_t size)
 {
-	int nr_bpages = (PAGE_ALIGN(size) / PAGE_SIZE) + 1;
 	size_t backing_size;
+	int nr_bpages;
 	void *start;
 
+	/* pKVM uses hyp_alloc() to allocate struct simple_buffer_page */
+	if (is_protected_kvm_enabled()) {
+		trace_buffer->desc->bpages_backing_start = 0;
+		trace_buffer->desc->bpages_backing_size = 0;
+		return 0;
+	}
+
+	nr_bpages = (PAGE_ALIGN(size) / PAGE_SIZE) + 1;
 	backing_size = PAGE_ALIGN(sizeof(struct simple_buffer_page) * nr_bpages *
 				  num_possible_cpus());
 
@@ -176,6 +185,9 @@ static int hyp_trace_buffer_alloc_bpages_backing(struct hyp_trace_buffer *trace_
 
 static void hyp_trace_buffer_free_bpages_backing(struct hyp_trace_buffer *trace_buffer)
 {
+	if (!trace_buffer->desc->bpages_backing_start)
+		return;
+
 	free_pages_exact((void *)trace_buffer->desc->bpages_backing_start,
 			 trace_buffer->desc->bpages_backing_size);
 }
@@ -264,7 +276,7 @@ static struct trace_buffer_desc *hyp_trace_load(unsigned long size, void *priv)
 	if (ret)
 		goto err_free_buffer;
 
-	ret = kvm_call_hyp_nvhe(__tracing_load, (unsigned long)desc, desc_size);
+	ret = pkvm_call_hyp_req(__tracing_load, (unsigned long)desc, desc_size);
 	if (ret)
 		goto err_unload_pages;
 
-- 
2.55.0.rc2.803.g1fd1e6609c-goog

Re: [PATCH v2 04/18] KVM: arm64: Add a heap allocator for the pKVM hyp

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] hyp_allocator_map remaps already-mapped pages and corrupts unmapped tracker state due to unaligned next chunks.
- [High] Fatal EL2 WARN_ON creates a dead-branch trap and panics the hypervisor on global lock contention in hyp_allocator_alloc.
--

commit d07913766ae6b18b86e593f40b4fc4a8165e2722
Author: Vincent Donnefort <vdonnefort@google.com>

KVM: arm64: Add a heap allocator for the pKVM hyp

This commit creates a heap allocator to manage VA-contiguous memory for the
hypervisor, replacing donations embedded within HVCs. It supports allocating,
freeing, topping up from a host memcache, and reclaiming memory.

> diff --git a/arch/arm64/kvm/hyp/nvhe/alloc.c b/arch/arm64/kvm/hyp/nvhe/alloc.c
> index 0000000000000..07ef0a13855f7
> --- /dev/null
> +++ b/arch/arm64/kvm/hyp/nvhe/alloc.c

[ ... ]

> +static int hyp_allocator_map(struct hyp_allocator *allocator, struct chunk_hdr *chunk,
> +			     struct chunk_hdr *next,
> +			     void *addr, void *end)
> +{
> +	void *unmapped = chunk ? chunk_unmapped(chunk) : allocator->first_unmapped;
> +
> +	/*
> +	 * hyp_allocator_can_create_chunk() already validates addr/end
> +	 * belong to the chunk.
> +	 */
> +	WARN_ON(end <= addr);
> +
> +	/* The chunk does not span an unmapped region */
> +	if (!unmapped)
> +		return 0;
> +
> +	while (unmapped < end) {
> +		void *page = pop_hyp_memcache(&allocator->mc, hyp_phys_to_virt);
> +		int ret;
> +
> +		if (!page) {
> +			end = PTR_ALIGN(end, PAGE_SIZE);
> +			*this_cpu_ptr(allocator->topup_needed) =
> +				(unsigned long)(end - unmapped) >> PAGE_SHIFT;
> +			return -ENOMEM;
> +		}
> +
> +		ret = __hyp_allocator_map(unmapped, hyp_virt_to_phys(page));

[Severity: Critical]
Does this loop correctly handle unaligned next chunks when re-mapping
reclaimed memory?

If the allocator reclaims a chunk where the next chunk is unaligned (e.g., at
PAGE_SIZE + 32), the unmapped hole is safely truncated to PAGE_SIZE by
PTR_ALIGN_DOWN(next, PAGE_SIZE). However, if the chunk is later reused with
an end address extending into the partially shared page (e.g., PAGE_SIZE + 16):

The loop in hyp_allocator_map() iterates while (unmapped < end). When unmapped
reaches the end of the true hole (PAGE_SIZE), the loop continues because
PAGE_SIZE < PAGE_SIZE + 16. It then pops a new physical page and maps it at
PAGE_SIZE, overwriting the existing active PTE containing the next chunk.

Additionally, the unmapped == next reset check fails because of the unaligned
next pointer, propagating the corrupted tracker state.

[ ... ]

> +static void *hyp_allocator_alloc(struct hyp_allocator *allocator, size_t size)
> +{
> +	struct chunk_hdr *chunk;
> +
> +	size = max(size, MIN_ALLOC_SIZE);
> +
> +	/* Ensure we do not overflow ALIGN(MIN_ALLOC_SIZE) */
> +	if (size > U32_MAX) {
> +		hyp_allocator_set_errno(allocator, -E2BIG);
> +		return NULL;
> +	}
> +
> +	size = ALIGN(size, MIN_ALLOC_SIZE);
> +	if (size > (allocator->end - allocator->start - chunk_hdr_size())) {
> +		hyp_allocator_set_errno(allocator, -E2BIG);
> +		return NULL;
> +	}
> +
> +#ifdef CONFIG_NVHE_EL2_DEBUG
> +	/* The allocator can modify the hyp stage-1 */
> +	if (WARN_ON(hyp_spin_is_locked(&pkvm_pgd_lock))) {
> +		hyp_allocator_set_errno(allocator, -EINVAL);
> +		return NULL;
> +	}
> +#endif

[Severity: High]
Can this lock check trigger a hypervisor panic under normal SMP contention?

At EL2, WARN_ON() expands to a fatal BRK instruction, meaning the error
handling path here acts as a dead-branch trap and will never execute.

Furthermore, hyp_spin_is_locked() returns true if the lock is held by any CPU,
not just the local one. If host userspace triggers concurrent VM creations or
VCPU initializations, driving concurrent EL2 hypercalls to hyp_alloc(), CPU A
might hold pkvm_pgd_lock while CPU B concurrently evaluates this WARN_ON().

This converts standard SMP lock contention into a fatal host denial of service.

> +	scoped_guard(hyp_spinlock, &allocator->lock) {
> +		chunk = hyp_allocator_alloc_chunk(allocator, size);
> +		if (IS_ERR_OR_NULL(chunk)) {
> +			hyp_allocator_set_errno(allocator,
> +						IS_ERR(chunk) ? PTR_ERR(chunk) : -EINVAL);
> +			return NULL;
> +		}
> +	}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260706175415.2604046-1-vdonnefort@google.com?part=4

Re: [PATCH v2 11/18] KVM: arm64: Add a shrinker for pKVM

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The pKVM memory shrinker is registered before pKVM is finalized, exposing 'pKVM-finalised-only' hypercalls to early execution which triggers a WARN_ON and potential boot panic.
--

commit 32c5640182999493c354bd3285732dfbfa3fd23d
Author: Vincent Donnefort <vdonnefort@google.com>

KVM: arm64: Add a shrinker for pKVM

Integrate the pKVM memory reclaim interface with the host's memory
management subsystem. This allows the host to automatically recover unused
memory fom the hypervisor's heap allocator when the host is under memory
pressure.

> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c

[ ... ]

> @@ -2713,6 +2715,18 @@ static void pkvm_hyp_init_ptrauth(void)
>  	}
>  }
>  
> +static unsigned long
> +pkvm_shrinker_count(struct shrinker *shrink, struct shrink_control *sc)
> +{
> +	return pkvm_hyp_reclaimable(PKVM_TOPUP_HYP_ALLOC) ?: SHRINK_EMPTY;
> +}
> +
> +static unsigned long
> +pkvm_shrinker_scan(struct shrinker *shrink, struct shrink_control *sc)
> +{
> +	return pkvm_hyp_reclaim(PKVM_TOPUP_HYP_ALLOC, sc->nr_to_scan);
> +}
> +

[ ... ]

> @@ -2883,6 +2897,16 @@ static int __init init_hyp_mode(void)
>  			kvm_err("Failed to init hyp memory protection\n");
>  			goto out_err;
>  		}
> +
> +		pkvm_shrinker = shrinker_alloc(0, "pkvm");
> +		if (pkvm_shrinker) {
> +			pkvm_shrinker->count_objects = pkvm_shrinker_count;
> +			pkvm_shrinker->scan_objects = pkvm_shrinker_scan;
> +			shrinker_register(pkvm_shrinker);
> +		} else {
> +			kvm_err("Failed to register shrinker for pKVM\n");
> +		}
> +
>  	}
>  
>  	return 0;

[Severity: High]
Could registering the shrinker here expose pKVM-finalised-only hypercalls
to early execution?

Since init_hyp_mode() runs at device_initcall, the shrinker becomes globally
visible before pKVM is finalized at device_initcall_sync in finalize_pkvm().

If memory pressure triggers the shrinker in this window,
pkvm_shrinker_count() and pkvm_shrinker_scan() will invoke reclaim
hypercalls. Because pKVM is not yet finalized, the EL2 handler will reject
hypercalls beyond __KVM_HOST_SMCCC_FUNC_PKVM_ONLY with
SMCCC_RET_NOT_SUPPORTED.

Will this failure cause the host-side kvm_call_hyp_nvhe macro to trigger a
WARN_ON, potentially panicking the host if panic_on_warn is set?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260706175415.2604046-1-vdonnefort@google.com?part=11

Re: [PATCH v2 06/18] KVM: arm64: Add topup interface for the pKVM heap allocator

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [Low] Unused static function `pkvm_hyp_topup` causes build failure.

Pre-existing issues:
- [High] `__pkvm_host_donate_hyp()` handles allocator failures with `WARN_ON()`, allowing userspace to trivially panic the hypervisor on memory exhaustion.
--

commit ed5697a7fc612043a616e5de99f36c231db3d5dd
Author: Vincent Donnefort <vdonnefort@google.com>

KVM: arm64: Add topup interface for the pKVM heap allocator

Introduce a host HVC interface and a host side helper to allow refilling
the pKVM heap allocator.

> diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
> index 8324a6a1bc48b..e2c8714e8ccc0 100644
> --- a/arch/arm64/kvm/pkvm.c
> +++ b/arch/arm64/kvm/pkvm.c
[ ... ]
> @@ -85,6 +85,33 @@ void __init kvm_hyp_reserve(void)
>  		 hyp_mem_base);
>  }
>  
> +static int pkvm_hyp_topup(enum pkvm_topup_id id, unsigned long nr_pages)
> +{
> +	struct kvm_hyp_memcache mc;
> +	struct arm_smccc_res res;

[Severity: Low]
Does this cause a build failure with CONFIG_WERROR=y?

This static function is added here but doesn't appear to be called anywhere
in this patch, which usually triggers a -Wunused-function warning. While it
may be used in a subsequent patch in the series, each patch should compile
independently to preserve git bisectability.

[Severity: High]
This is a pre-existing issue, but can host userspace crash the hypervisor
by exhausting EL2 memory?

In arch/arm64/kvm/hyp/nvhe/mem_protect.c:__pkvm_host_donate_hyp():

    WARN_ON(pkvm_create_mappings_locked(virt, virt + size, PAGE_HYP));
    WARN_ON(host_stage2_set_owner_locked(phys, size, PKVM_ID_HYP));

At EL2 in nVHE/pKVM, WARN_ON() expands to BRK and panics the hypervisor. If
pkvm_create_mappings_locked() requires a new page table level and the EL2
buddy allocator is exhausted, it returns -ENOMEM. 

By wrapping this in WARN_ON(), a fallible allocator outcome is transformed
into a fatal EL2 panic. This allows host userspace to trivially panic the
host kernel by exhausting EL2 memory, such as by creating numerous VMs.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260706175415.2604046-1-vdonnefort@google.com?part=6

Re: [PATCH v2 14/18] KVM: arm64: Use noclear for PGD in __pkvm_init_vm error path

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Using `unmap_donated_memory_noclear()` for the PGD on the error path leaks EL2 virtual addresses to the untrusted host, bypassing hypervisor ASLR.
--

commit d94f8fc664be6ec0ac22803e7fe54acff9cf8af1
Author: Vincent Donnefort <vdonnefort@google.com>

KVM: arm64: Use noclear for PGD in __pkvm_init_vm error path
    
This commit modifies the error path of __pkvm_init_vm() to use
unmap_donated_memory_noclear() instead of unmap_donated_memory() when
returning the donated stage-2 PGD to the host.

> diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c
> index d9ab582268895..0c82f1ddb93df 100644
> --- a/arch/arm64/kvm/hyp/nvhe/pkvm.c
> +++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c
> @@ -861,7 +861,7 @@ int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva,
>  	kvm_guest_destroy_stage2(hyp_vm);
>  err_remove_mappings:
>  	unmap_donated_memory(hyp_vm, vm_size);
> -	unmap_donated_memory(pgd, pgd_size);
> +	unmap_donated_memory_noclear(pgd, pgd_size);

[Severity: High]
Does using unmap_donated_memory_noclear() here leak EL2 virtual addresses to
the untrusted host?

If the host deliberately provides an invalid handle before triggering this
hypercall, insert_vm_table_entry() will fail and execution will take this
error path.

Before reaching this error path, kvm_guest_prepare_stage2() initializes the
donated PGD memory as a buddy pool. During this process, __hyp_attach_page()
uses page_add_to_list() to write EL2 list_head pointers into the free pages.

By using the noclear variant here, the memory is returned to the host without
zeroing those pointers. The host can then read this returned memory to
extract valid EL2 virtual addresses, which breaks hypervisor KASLR.

Should this remain unmap_donated_memory() to ensure these internal pointers
are properly scrubbed before the host regains access?

>  err_unpin_kvm:
>  	hyp_unpin_shared_mem(host_kvm, host_kvm + 1);
>  	return ret;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260706175415.2604046-1-vdonnefort@google.com?part=14

Re: [PATCH v2 18/18] KVM: arm64: Alloc simple_buffer_page using pKVM hyp allocator

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Missing validation of host-supplied `nr_page_va` causes EL2 memory leak and host infinite loop.
--

commit 20e38c941fd5614b7d02825f2350aeac0276e149
Author: Vincent Donnefort <vdonnefort@google.com>

KVM: arm64: Alloc simple_buffer_page using pKVM hyp allocator
    
This patch transitions the allocation of the simple_ring_buffer structures
from the host to the hypervisor in protected mode using the new pKVM heap
allocator. It uses pkvm_call_hyp_req() to handle potential top-up requests
during allocation.

> diff --git a/arch/arm64/kvm/hyp/nvhe/trace.c b/arch/arm64/kvm/hyp/nvhe/trace.c
> index 6e716295247ad..bcfe06bba29bb 100644
> --- a/arch/arm64/kvm/hyp/nvhe/trace.c
> +++ b/arch/arm64/kvm/hyp/nvhe/trace.c

[ ... ]

> @@ -62,18 +63,34 @@ static void __release_host_mem(void *start, u64 size)
>  	WARN_ON(__pkvm_hyp_donate_host(hyp_virt_to_pfn(start), size >> PAGE_SHIFT));
>  }
>  
> -static int hyp_trace_buffer_load_bpage_backing(struct hyp_trace_buffer *trace_buffer,
> -					       struct hyp_trace_desc *desc)
> +static int hyp_trace_buffer_alloc_bpages(struct hyp_trace_buffer *trace_buffer,
> +					 struct hyp_trace_desc *desc)
>  {
> -	void *start = (void *)kern_hyp_va(desc->bpages_backing_start);
> -	size_t size = desc->bpages_backing_size;
> +	void *start;
> +	size_t size;
>  	int ret;
>  
> -	ret = __admit_host_mem(start, size);
> -	if (ret)
> -		return ret;
> +	if (is_protected_kvm_enabled()) {
> +		struct ring_buffer_desc *rb_desc;
> +		int cpu;
> +
> +		size = 0;
> +		for_each_ring_buffer_desc(rb_desc, cpu, &desc->trace_buffer_desc)
> +			size += rb_desc->nr_page_va * sizeof(struct simple_buffer_page);
> +
> +		start = hyp_alloc(size);

[Severity: High]
Is there a missing bounds check on the host-supplied nr_page_va here?

If the host sets nr_page_va to 0 for all CPUs, size will be 0, but
hyp_alloc(0) will still succeed by allocating MIN_ALLOC_SIZE.

> +		if (!start)
> +			return hyp_alloc_errno();
> +	} else {
> +		start = (void *)kern_hyp_va(desc->bpages_backing_start);
> +		size = desc->bpages_backing_size;
>  
> -	memset(start, 0, size);
> +		ret = __admit_host_mem(start, size);
> +		if (ret)
> +			return ret;
> +
> +		memset(start, 0, size);
> +	}

[ ... ]

> @@ -81,7 +98,7 @@ static int hyp_trace_buffer_load_bpage_backing(struct hyp_trace_buffer *trace_bu
>  	return 0;
>  }
>  
> -static void hyp_trace_buffer_unload_bpage_backing(struct hyp_trace_buffer *trace_buffer)
> +static void hyp_trace_buffer_free_bpages(struct hyp_trace_buffer *trace_buffer)
>  {
>  	void *start = trace_buffer->bpages_backing_start;
>  	size_t size = trace_buffer->bpages_backing_size;
> @@ -89,9 +106,12 @@ static void hyp_trace_buffer_unload_bpage_backing(struct hyp_trace_buffer *trace
>  	if (!size)
>  		return;

[Severity: High]
Will this early return permanently leak the memory allocated above?

If size evaluated to 0 during allocation, simple_ring_buffer_init_mm()
fails with -EINVAL and triggers this cleanup path. Since size is 0, this
returns early without calling hyp_free(), leaking the MIN_ALLOC_SIZE chunk
in the EL2 heap.

Could exhausting the EL2 heap this way cause a subsequent 0-page topup
request back to the host, resulting in an infinite loop in the host CPU's
pkvm_call_hyp_req() macro?

> -	memset(start, 0, size);
> -
> -	__release_host_mem(start, size);
> +	if (is_protected_kvm_enabled()) {
> +		hyp_free(start);
> +	} else {
> +		memset(start, 0, size);
> +		__release_host_mem(start, size);
> +	}
>  
>  	trace_buffer->bpages_backing_start = 0;
>  	trace_buffer->bpages_backing_size = 0;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260706175415.2604046-1-vdonnefort@google.com?part=18

Re: [PATCH v2 15/18] KVM: arm64: Alloc pkvm_hyp_vm using pKVM heap allocator

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Memcg accounting bypass in EL2 heap top-up allows host memory exhaustion
--

commit 66032df021feb40dc76572f7b2d6e0341f8b7e65
Author: Vincent Donnefort <vdonnefort@google.com>

KVM: arm64: Alloc pkvm_hyp_vm using pKVM heap allocator

Transition the allocation of the hypervisor VM state structure
(pkvm_hyp_vm) from the host to the hypervisor using the new pKVM heap
allocator (hyp_alloc()).

> diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
> index f7e1846d6093f..8a8b5b84825c7 100644
> --- a/arch/arm64/kvm/pkvm.c
> +++ b/arch/arm64/kvm/pkvm.c
[ ... ]
> @@ -209,28 +209,15 @@ static int __pkvm_create_hyp_vm(struct kvm *kvm)
>  	if (!pgd)
>  		return -ENOMEM;
>  
> -	/* Allocate memory to donate to hyp for vm and vcpu pointers. */
> -	hyp_vm_sz = PAGE_ALIGN(size_add(PKVM_HYP_VM_SIZE,
> -					size_mul(sizeof(void *),
> -						 kvm->created_vcpus)));
> -	hyp_vm = alloc_pages_exact(hyp_vm_sz, GFP_KERNEL_ACCOUNT);
> -	if (!hyp_vm) {
> -		ret = -ENOMEM;
> -		goto free_pgd;
> -	}
> -
> -	/* Donate the VM memory to hyp and let hyp initialize it. */
> -	ret = kvm_call_hyp_nvhe(__pkvm_init_vm, kvm, hyp_vm, pgd);
> +	ret = pkvm_call_hyp_req(__pkvm_init_vm, kvm, pgd);

[Severity: High]
Does this change bypass memory cgroup accounting for the VM allocation?

Previously, the host allocated hyp_vm using GFP_KERNEL_ACCOUNT. With this
patch, the allocation is deferred to EL2, and if the EL2 heap is
depleted, it triggers a top-up request handled by pkvm_call_hyp_req()
and pkvm_hyp_topup() on the host.

Looking at how pkvm_hyp_topup() initializes the memcache:

init_hyp_memcache(&mc);
ret = topup_hyp_memcache(&mc, nr_pages);

The init_hyp_memcache() helper zeros the memcache flags, meaning it omits
the HYP_MEMCACHE_ACCOUNT_KMEMCG flag. As a result, the subsequent host page
allocations in hyp_mc_alloc_fn() will use a plain GFP_KERNEL without
__GFP_ACCOUNT.

Could an unprivileged container repeatedly trigger KVM_CREATE_VM to exhaust
host memory without being restricted by its memory cgroup limits?

>  	if (ret)
> -		goto free_vm;
> +		goto free_pgd;
>  
>  	kvm->arch.pkvm.is_created = true;
>  	init_hyp_stage2_memcache(&kvm->arch.pkvm.stage2_teardown_mc);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260706175415.2604046-1-vdonnefort@google.com?part=15

Re: [PATCH v2 16/18] KVM: arm64: Alloc pkvm_hyp_vcpu using pKVM heap allocator

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] The pKVM heap allocator (`hyp_alloc`) guarantees only 8-byte alignment, which violates the strict 16-byte alignment requirement of `struct pkvm_hyp_vcpu`.
- [High] Delegating `pkvm_hyp_vcpu` allocation to the global EL2 heap bypasses memory cgroup (memcg) accounting, allowing unprivileged userspace to exceed memory limits.
--

commit 6cb252c4c2be901676d82bdcc257dd6e52864352
Author: Vincent Donnefort <vdonnefort@google.com>

KVM: arm64: Alloc pkvm_hyp_vcpu using pKVM heap allocator

Transition the allocation of the hypervisor vCPU state structure
(pkvm_hyp_vcpu) from the host to the hypervisor using the new pKVM heap
allocator (hyp_alloc()).

> diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c
> index 2f616a89f8b35..45ec77e9eca82 100644
> --- a/arch/arm64/kvm/hyp/nvhe/pkvm.c
> +++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c

[ ... ]

> @@ -896,16 +863,15 @@ static int register_hyp_vcpu(struct pkvm_hyp_vm *hyp_vm,
>  	return 0;
>  }
>  
> -int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu,
> -		     unsigned long vcpu_hva)
> +int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu)
>  {
>  	struct pkvm_hyp_vcpu *hyp_vcpu;
>  	struct pkvm_hyp_vm *hyp_vm;
>  	int ret;
>  
> -	hyp_vcpu = map_donated_memory(vcpu_hva, sizeof(*hyp_vcpu));
> +	hyp_vcpu = hyp_alloc(sizeof(*hyp_vcpu));

[Severity: High]
Will this allocation guarantee the 16-byte alignment required by
struct pkvm_hyp_vcpu?

The struct embeds a struct kvm_vcpu, which contains a struct
user_fpsimd_state with a __uint128_t vregs[32] array. The C ABI and
AAPCS64 mandate strict 16-byte alignment for 128-bit types.

If hyp_alloc() only guarantees 8-byte alignment (since its MIN_ALLOC_SIZE
is 8), returning an 8-byte aligned pointer could trigger an alignment fault
during compiler-generated SIMD instructions (like LDP Qx, Qy) when
saving or restoring FPSIMD state, potentially leading to a hypervisor panic.

>  	if (!hyp_vcpu)
> -		return -ENOMEM;
> +		return hyp_alloc_errno();
>  
>  	hyp_spin_lock(&vm_table_lock);

[ ... ]

> diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
> index 8a8b5b84825c7..1899b05d9fc22 100644
> --- a/arch/arm64/kvm/pkvm.c
> +++ b/arch/arm64/kvm/pkvm.c

[ ... ]

> @@ -153,28 +153,19 @@ static void __pkvm_destroy_hyp_vm(struct kvm *kvm)
>  
>  	kvm->arch.pkvm.handle = 0;
>  	kvm->arch.pkvm.is_created = false;
> -	free_hyp_memcache(&kvm->arch.pkvm.teardown_mc);
>  	free_hyp_memcache(&kvm->arch.pkvm.stage2_teardown_mc);
>  }
>  
>  static int __pkvm_create_hyp_vcpu(struct kvm_vcpu *vcpu)
>  {
> -	size_t hyp_vcpu_sz = PAGE_ALIGN(PKVM_HYP_VCPU_SIZE);
>  	pkvm_handle_t handle = vcpu->kvm->arch.pkvm.handle;
> -	void *hyp_vcpu;
>  	int ret;
>  
>  	init_hyp_stage2_memcache(&vcpu->arch.pkvm_memcache);
>  
> -	hyp_vcpu = alloc_pages_exact(hyp_vcpu_sz, GFP_KERNEL_ACCOUNT);
> -	if (!hyp_vcpu)
> -		return -ENOMEM;
> -
> -	ret = kvm_call_hyp_nvhe(__pkvm_init_vcpu, handle, vcpu, hyp_vcpu);
> +	ret = pkvm_call_hyp_req(__pkvm_init_vcpu, handle, vcpu);

[Severity: High]
Does this change bypass memory cgroup (memcg) accounting for the vCPU
allocations?

Previously, alloc_pages_exact() used GFP_KERNEL_ACCOUNT, which ensured the
memory was charged to the allocating process. By delegating the allocation
to the EL2 heap via pkvm_call_hyp_req(), a heap depletion will trigger a
top-up request (PKVM_HYP_REQ_HYP_ALLOC).

The host handler for this top-up (pkvm_hyp_topup()) uses
init_hyp_memcache(), which defaults to GFP_KERNEL and omits the
HYP_MEMCACHE_ACCOUNT_KMEMCG flag.

Could this allow an unprivileged userspace process to bypass memcg limits
and exhaust system memory by creating numerous VMs and vCPUs?

>  	if (!ret)
>  		vcpu_set_flag(vcpu, VCPU_PKVM_FINALIZED);
> -	else
> -		free_pages_exact(hyp_vcpu, hyp_vcpu_sz);
>  
>  	return ret;
>  }

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260706175415.2604046-1-vdonnefort@google.com?part=16

Re: [PATCH v2 04/18] KVM: arm64: Add a heap allocator for the pKVM hyp

[Vincent Donnefort]

On Mon, Jul 06, 2026 at 06:12:41PM +0000, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
> - [Critical] hyp_allocator_map remaps already-mapped pages and corrupts unmapped tracker state due to unaligned next chunks.
> - [High] Fatal EL2 WARN_ON creates a dead-branch trap and panics the hypervisor on global lock contention in hyp_allocator_alloc.
> --
> 
> commit d07913766ae6b18b86e593f40b4fc4a8165e2722
> Author: Vincent Donnefort <vdonnefort@google.com>
> 
> KVM: arm64: Add a heap allocator for the pKVM hyp
> 
> This commit creates a heap allocator to manage VA-contiguous memory for the
> hypervisor, replacing donations embedded within HVCs. It supports allocating,
> freeing, topping up from a host memcache, and reclaiming memory.
> 
> > diff --git a/arch/arm64/kvm/hyp/nvhe/alloc.c b/arch/arm64/kvm/hyp/nvhe/alloc.c
> > index 0000000000000..07ef0a13855f7
> > --- /dev/null
> > +++ b/arch/arm64/kvm/hyp/nvhe/alloc.c
> 
> [ ... ]
> 
> > +static int hyp_allocator_map(struct hyp_allocator *allocator, struct chunk_hdr *chunk,
> > +			     struct chunk_hdr *next,
> > +			     void *addr, void *end)
> > +{
> > +	void *unmapped = chunk ? chunk_unmapped(chunk) : allocator->first_unmapped;
> > +
> > +	/*
> > +	 * hyp_allocator_can_create_chunk() already validates addr/end
> > +	 * belong to the chunk.
> > +	 */
> > +	WARN_ON(end <= addr);
> > +
> > +	/* The chunk does not span an unmapped region */
> > +	if (!unmapped)
> > +		return 0;
> > +
> > +	while (unmapped < end) {
> > +		void *page = pop_hyp_memcache(&allocator->mc, hyp_phys_to_virt);
> > +		int ret;
> > +
> > +		if (!page) {
> > +			end = PTR_ALIGN(end, PAGE_SIZE);
> > +			*this_cpu_ptr(allocator->topup_needed) =
> > +				(unsigned long)(end - unmapped) >> PAGE_SHIFT;
> > +			return -ENOMEM;
> > +		}
> > +
> > +		ret = __hyp_allocator_map(unmapped, hyp_virt_to_phys(page));
> 
> [Severity: Critical]
> Does this loop correctly handle unaligned next chunks when re-mapping
> reclaimed memory?
> 
> If the allocator reclaims a chunk where the next chunk is unaligned (e.g., at
> PAGE_SIZE + 32), the unmapped hole is safely truncated to PAGE_SIZE by
> PTR_ALIGN_DOWN(next, PAGE_SIZE). However, if the chunk is later reused with
> an end address extending into the partially shared page (e.g., PAGE_SIZE + 16):
> 
> The loop in hyp_allocator_map() iterates while (unmapped < end). When unmapped
> reaches the end of the true hole (PAGE_SIZE), the loop continues because
> PAGE_SIZE < PAGE_SIZE + 16. It then pops a new physical page and maps it at
> PAGE_SIZE, overwriting the existing active PTE containing the next chunk.
> 
> Additionally, the unmapped == next reset check fails because of the unaligned
> next pointer, propagating the corrupted tracker state.

I don't believe this can happen. Chunks __always__ describe a mapped section,
followed (or not) by an unmapped one. So if we are adding mappings to a chunk,
next has to be PAGE_ALIGNED.

However, I could add a WARN_ON() here.


> 
> [ ... ]
> 
> > +static void *hyp_allocator_alloc(struct hyp_allocator *allocator, size_t size)
> > +{
> > +	struct chunk_hdr *chunk;
> > +
> > +	size = max(size, MIN_ALLOC_SIZE);
> > +
> > +	/* Ensure we do not overflow ALIGN(MIN_ALLOC_SIZE) */
> > +	if (size > U32_MAX) {
> > +		hyp_allocator_set_errno(allocator, -E2BIG);
> > +		return NULL;
> > +	}
> > +
> > +	size = ALIGN(size, MIN_ALLOC_SIZE);
> > +	if (size > (allocator->end - allocator->start - chunk_hdr_size())) {
> > +		hyp_allocator_set_errno(allocator, -E2BIG);
> > +		return NULL;
> > +	}
> > +
> > +#ifdef CONFIG_NVHE_EL2_DEBUG
> > +	/* The allocator can modify the hyp stage-1 */
> > +	if (WARN_ON(hyp_spin_is_locked(&pkvm_pgd_lock))) {
> > +		hyp_allocator_set_errno(allocator, -EINVAL);
> > +		return NULL;
> > +	}
> > +#endif
> 
> [Severity: High]
> Can this lock check trigger a hypervisor panic under normal SMP contention?
> 
> At EL2, WARN_ON() expands to a fatal BRK instruction, meaning the error
> handling path here acts as a dead-branch trap and will never execute.
> 
> Furthermore, hyp_spin_is_locked() returns true if the lock is held by any CPU,
> not just the local one. If host userspace triggers concurrent VM creations or
> VCPU initializations, driving concurrent EL2 hypercalls to hyp_alloc(), CPU A
> might hold pkvm_pgd_lock while CPU B concurrently evaluates this WARN_ON().
> 
> This converts standard SMP lock contention into a fatal host denial of service.

Duh, this hyp_spin_is_locked() is of course utterly broken! I'll add a
comment for hyp_alloc() instead.

> 
> > +	scoped_guard(hyp_spinlock, &allocator->lock) {
> > +		chunk = hyp_allocator_alloc_chunk(allocator, size);
> > +		if (IS_ERR_OR_NULL(chunk)) {
> > +			hyp_allocator_set_errno(allocator,
> > +						IS_ERR(chunk) ? PTR_ERR(chunk) : -EINVAL);
> > +			return NULL;
> > +		}
> > +	}
> 
> -- 
> Sashiko AI review · https://sashiko.dev/#/patchset/20260706175415.2604046-1-vdonnefort@google.com?part=4

Re: [PATCH v2 01/18] KVM: arm64: Add __pkvm_private_range_pa

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> Mappings in the pKVM private range are not identity mapped, making the
> standard __hyp_pa() unsuitable for translating these addresses.
>
> Introduce __pkvm_private_range_pa() to resolve physical addresses for
> this range by walking the hypervisor page-table. This will be useful for
> the upcoming pKVM heap allocator.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
>
> diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h
> index 6e83ce35c2f2..d3137c16b632 100644
> --- a/arch/arm64/kvm/hyp/include/nvhe/mm.h
> +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h
> @@ -30,5 +30,6 @@ int __pkvm_create_private_mapping(phys_addr_t phys, size_t size,
>                                   unsigned long *haddr);
>  int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr);
>  int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr);
> +phys_addr_t __pkvm_private_range_pa(void *va);
>
>  #endif /* __KVM_HYP_MM_H */
> diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c
> index 3b0bee496bff..773426a68d2d 100644
> --- a/arch/arm64/kvm/hyp/nvhe/mm.c
> +++ b/arch/arm64/kvm/hyp/nvhe/mm.c
> @@ -502,3 +502,17 @@ int refill_memcache(struct kvm_hyp_memcache *mc, unsigned long min_pages,
>
>         return ret;
>  }
> +
> +phys_addr_t __pkvm_private_range_pa(void *va)
> +{
> +       kvm_pte_t pte = 0;
> +       s8 level;
> +
> +       hyp_spin_lock(&pkvm_pgd_lock);
> +       WARN_ON(kvm_pgtable_get_leaf(&pkvm_pgtable, (u64)va, &pte, &level));

nit: can you pass NULL instead of level?

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad


> +       hyp_spin_unlock(&pkvm_pgd_lock);
> +
> +       WARN_ON(!kvm_pte_valid(pte));
> +
> +       return kvm_pte_to_phys(pte) + offset_in_page(va);
> +}
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 02/18] KVM: arm64: Add pkvm_remove_mappings

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> Add the counterpart to pkvm_create_mappings(), allowing previously
> mapped ranges to be removed. This will be useful for the upcoming pKVM
> heap allocator to manage its private mappings.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad

>
> diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h
> index d3137c16b632..725bb0fb941d 100644
> --- a/arch/arm64/kvm/hyp/include/nvhe/mm.h
> +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h
> @@ -25,6 +25,7 @@ int hyp_back_vmemmap(phys_addr_t back);
>  int pkvm_cpu_set_vector(enum arm64_hyp_spectre_vector slot);
>  int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot);
>  int pkvm_create_mappings_locked(void *from, void *to, enum kvm_pgtable_prot prot);
> +void pkvm_remove_mappings(void *from, void *to);
>  int __pkvm_create_private_mapping(phys_addr_t phys, size_t size,
>                                   enum kvm_pgtable_prot prot,
>                                   unsigned long *haddr);
> diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c
> index 773426a68d2d..ec59da1322a7 100644
> --- a/arch/arm64/kvm/hyp/nvhe/mm.c
> +++ b/arch/arm64/kvm/hyp/nvhe/mm.c
> @@ -146,6 +146,20 @@ int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot)
>         return ret;
>  }
>
> +void pkvm_remove_mappings(void *from, void *to)
> +{
> +       u64 size;
> +
> +       to = PTR_ALIGN(to, PAGE_SIZE);
> +       from = PTR_ALIGN_DOWN(from, PAGE_SIZE);
> +       size = (u64)to - (u64)from;
> +       WARN_ON(from > to);
> +
> +       hyp_spin_lock(&pkvm_pgd_lock);
> +       WARN_ON(kvm_pgtable_hyp_unmap(&pkvm_pgtable, (u64)from, size) != size);
> +       hyp_spin_unlock(&pkvm_pgd_lock);
> +}
> +
>  int hyp_back_vmemmap(phys_addr_t back)
>  {
>         unsigned long i, start, size, end = 0;
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 03/18] KVM: arm64: Add __hyp_allocator_map for the pKVM hyp

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> In preparation for the pKVM heap allocator, introduce __hyp_allocator_map()
> to map a single physical page to a given virtual address in the hypervisor
> stage-1 page-table.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
>
> diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h
> index 725bb0fb941d..98a7774b541c 100644
> --- a/arch/arm64/kvm/hyp/include/nvhe/mm.h
> +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h
> @@ -32,5 +32,6 @@ int __pkvm_create_private_mapping(phys_addr_t phys, size_t size,
>  int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr);
>  int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr);
>  phys_addr_t __pkvm_private_range_pa(void *va);
> +int __hyp_allocator_map(void *va, phys_addr_t phys);
>
>  #endif /* __KVM_HYP_MM_H */
> diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c
> index ec59da1322a7..c1c01f81ac5f 100644
> --- a/arch/arm64/kvm/hyp/nvhe/mm.c
> +++ b/arch/arm64/kvm/hyp/nvhe/mm.c
> @@ -160,6 +160,11 @@ void pkvm_remove_mappings(void *from, void *to)
>         hyp_spin_unlock(&pkvm_pgd_lock);
>  }
>
> +int __hyp_allocator_map(void *va, phys_addr_t phys)
> +{
> +       return __pkvm_create_mappings((unsigned long)va, PAGE_SIZE, phys, PAGE_HYP);
> +}
> +

nit: __hyp_allocator_map() is named for its caller rather than what it does
(map a single hyp page). Its siblings here (pkvm_create_mappings,
pkvm_remove_mappings) are named for the operation. Consider a name like
pkvm_map_page() so it reads as a generic primitive.

That said, naming is hard:

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad

>  int hyp_back_vmemmap(phys_addr_t back)
>  {
>         unsigned long i, start, size, end = 0;
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 04/18] KVM: arm64: Add a heap allocator for the pKVM hyp

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:

...

> +static u32 chunk_hash_compute(const struct chunk_hdr *chunk)
> +{
> +       u32 hash = 0;
> +
> +       BUILD_BUG_ON(sizeof(*chunk) != 16);
> +
> +       hash ^= hash_64(*(const u64 *)chunk, 32);
> +       hash ^= hash_32(chunk->__unmapped, 32);

Could you state the threat model in the commit msg? The VA range is
hyp-private and the hash is unkeyed (anything that can write EL2
memory recomputes it), so this only catches _accidental_ corruption,
i.e. a debug aid.

Since that's the goal, should it be gated on CONFIG_NVHE_EL2_DEBUG
like the other checks here? As written it's O(n) hashing per
hyp_alloc(), in production, for a check that only fires on a hyp bug.
And a plain XOR/additive checksum would be cheaper and just as good
for accidental corruption.

> +       return hash;
> +}
> +

...


> diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c
> index 75b00c323310..1c10cf78c869 100644
> --- a/arch/arm64/kvm/hyp/nvhe/setup.c
> +++ b/arch/arm64/kvm/hyp/nvhe/setup.c
> @@ -10,6 +10,7 @@
>  #include <asm/kvm_pgtable.h>
>  #include <asm/kvm_pkvm.h>
>
> +#include <nvhe/alloc.h>
>  #include <nvhe/early_alloc.h>
>  #include <nvhe/ffa.h>
>  #include <nvhe/gfp.h>
> @@ -363,6 +364,10 @@ int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long *per_cpu_bas
>         if (ret)
>                 return ret;
>
> +       ret = hyp_alloc_init(SZ_128M);

Where does 128M come from? A comment would help.

Cheers,
/fuad



> +       if (ret)
> +               return ret;
> +
>         update_nvhe_init_params();
>
>         /* Jump in the idmap page to switch to the new page-tables */
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 05/18] KVM: arm64: Allow kvm_hyp_memcache usage outside of stage-2

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> Although currently limited to guest stage-2 page-table allocations,
> struct kvm_hyp_memcache is a useful primitive for passing a list of
> discontiguous pages between host and hypervisor.
>
> Introduce init_hyp_memcache() to initialise a generic hyp memcache, and
> init_hyp_stage2_memcache() for stage-2 specific memcaches. The generic
> initialiser will be used to top up the upcoming pKVM heap allocator.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
>
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index bae2c4f92ef5..caea8e9986d1 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -91,9 +91,22 @@ struct kvm_hyp_memcache {
>         struct pkvm_mapping *mapping; /* only used from EL1 */
>
>  #define        HYP_MEMCACHE_ACCOUNT_STAGE2     BIT(1)
> +#define        HYP_MEMCACHE_ACCOUNT_KMEMCG     BIT(2)
>         unsigned long flags;
>  };
>
> +static inline void init_hyp_memcache(struct kvm_hyp_memcache *mc)
> +{
> +       memset(mc, 0, sizeof(*mc));
> +       mc->mapping = ZERO_SIZE_PTR; /* Prevent allocation, solely useful for stage2 memcache */
> +}
> +
> +static inline void init_hyp_stage2_memcache(struct kvm_hyp_memcache *mc)
> +{
> +       memset(mc, 0, sizeof(*mc));
> +       mc->flags = HYP_MEMCACHE_ACCOUNT_STAGE2 | HYP_MEMCACHE_ACCOUNT_KMEMCG;
> +}
> +
>  static inline void push_hyp_memcache(struct kvm_hyp_memcache *mc,
>                                      phys_addr_t *p,
>                                      phys_addr_t (*to_pa)(void *virt))
> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index 6c941aaa10c6..fb38821f2037 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c
> @@ -1159,9 +1159,11 @@ static void hyp_mc_free_fn(void *addr, void *mc)
>  static void *hyp_mc_alloc_fn(void *mc)
>  {
>         struct kvm_hyp_memcache *memcache = mc;
> +       gfp_t gfp = (memcache->flags & HYP_MEMCACHE_ACCOUNT_KMEMCG) ?
> +                   GFP_KERNEL_ACCOUNT : GFP_KERNEL;
>         void *addr;
>
> -       addr = (void *)__get_free_page(GFP_KERNEL_ACCOUNT);
> +       addr = (void *)__get_free_page(gfp);

Generic memcaches are not accounted anymore. That looks intentional
and right since heap pages are hyp-global, not attributable to the
topup'ing task. Maybe worth a line in the commit message confirming
this is deliberate?

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad

>         if (addr && memcache->flags & HYP_MEMCACHE_ACCOUNT_STAGE2)
>                 kvm_account_pgtable_pages(addr, 1);
>
> diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
> index 053e4f733e4b..8324a6a1bc48 100644
> --- a/arch/arm64/kvm/pkvm.c
> +++ b/arch/arm64/kvm/pkvm.c
> @@ -111,7 +111,7 @@ static int __pkvm_create_hyp_vcpu(struct kvm_vcpu *vcpu)
>         void *hyp_vcpu;
>         int ret;
>
> -       vcpu->arch.pkvm_memcache.flags |= HYP_MEMCACHE_ACCOUNT_STAGE2;
> +       init_hyp_stage2_memcache(&vcpu->arch.pkvm_memcache);
>
>         hyp_vcpu = alloc_pages_exact(hyp_vcpu_sz, GFP_KERNEL_ACCOUNT);
>         if (!hyp_vcpu)
> @@ -172,7 +172,7 @@ static int __pkvm_create_hyp_vm(struct kvm *kvm)
>                 goto free_vm;
>
>         kvm->arch.pkvm.is_created = true;
> -       kvm->arch.pkvm.stage2_teardown_mc.flags |= HYP_MEMCACHE_ACCOUNT_STAGE2;
> +       init_hyp_stage2_memcache(&kvm->arch.pkvm.stage2_teardown_mc);
>         kvm_account_pgtable_pages(pgd, pgd_sz / PAGE_SIZE);
>
>         return 0;
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 06/18] KVM: arm64: Add topup interface for the pKVM heap allocator

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> Introduce a host HVC interface and a host side helper to allow refilling
> the pKVM heap allocator.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
>
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index 043495f7fc78..681b7bf8ac08 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -114,6 +114,7 @@ enum __kvm_host_smccc_func {
>         __KVM_HOST_SMCCC_FUNC___pkvm_vcpu_load,
>         __KVM_HOST_SMCCC_FUNC___pkvm_vcpu_put,
>         __KVM_HOST_SMCCC_FUNC___pkvm_tlb_flush_vmid,
> +       __KVM_HOST_SMCCC_FUNC___pkvm_hyp_topup,
>
>         MARKER(__KVM_HOST_SMCCC_FUNC_MAX)
>  };
> diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h
> index 74fedd9c5ff0..870f7870bb62 100644
> --- a/arch/arm64/include/asm/kvm_pkvm.h
> +++ b/arch/arm64/include/asm/kvm_pkvm.h
> @@ -17,6 +17,10 @@
>
>  #define HYP_MEMBLOCK_REGIONS 128
>
> +enum pkvm_topup_id {
> +       PKVM_TOPUP_HYP_ALLOC,
> +};
> +
>  int pkvm_init_host_vm(struct kvm *kvm, unsigned long type);
>  int pkvm_create_hyp_vm(struct kvm *kvm);
>  bool pkvm_hyp_vm_is_created(struct kvm *kvm);
> diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
> index d3c69de698f4..e270a974fa37 100644
> --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
> +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
> @@ -15,6 +15,7 @@
>  #include <asm/kvm_hypevents.h>
>  #include <asm/kvm_mmu.h>
>
> +#include <nvhe/alloc.h>
>  #include <nvhe/ffa.h>
>  #include <nvhe/mem_protect.h>
>  #include <nvhe/mm.h>
> @@ -632,6 +633,30 @@ static void handle___pkvm_finalize_teardown_vm(struct kvm_cpu_context *host_ctxt
>         cpu_reg(host_ctxt, 1) = __pkvm_finalize_teardown_vm(handle);
>  }
>
> +static void handle___pkvm_hyp_topup(struct kvm_cpu_context *host_ctxt)

Sashiko caught this. Suggest moving just this helper to patch 8
alongside its caller.

Cheers,
/fuad

> +{
> +       DECLARE_REG(enum pkvm_topup_id, id, host_ctxt, 1);
> +       DECLARE_REG(phys_addr_t, head, host_ctxt, 2);
> +       DECLARE_REG(unsigned long, nr_pages, host_ctxt, 3);
> +       struct kvm_hyp_memcache host_mc = {
> +               .head = head,
> +               .nr_pages = nr_pages,
> +       };
> +       int ret;
> +
> +       switch (id) {
> +       case PKVM_TOPUP_HYP_ALLOC:
> +               ret = hyp_alloc_topup(&host_mc);
> +               break;
> +       default:
> +               ret = -EINVAL;
> +       }
> +
> +       cpu_reg(host_ctxt, 1) = ret;
> +       cpu_reg(host_ctxt, 2) = host_mc.head;
> +       cpu_reg(host_ctxt, 3) = host_mc.nr_pages;
> +}
> +
>  static void handle___tracing_load(struct kvm_cpu_context *host_ctxt)
>  {
>         DECLARE_REG(unsigned long, desc_hva, host_ctxt, 1);
> @@ -762,6 +787,7 @@ static const hcall_t host_hcall[] = {
>         HANDLE_FUNC(__pkvm_vcpu_load),
>         HANDLE_FUNC(__pkvm_vcpu_put),
>         HANDLE_FUNC(__pkvm_tlb_flush_vmid),
> +       HANDLE_FUNC(__pkvm_hyp_topup),
>  };
>
>  static void handle_host_hcall(struct kvm_cpu_context *host_ctxt)
> diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
> index 8324a6a1bc48..e2c8714e8ccc 100644
> --- a/arch/arm64/kvm/pkvm.c
> +++ b/arch/arm64/kvm/pkvm.c
> @@ -85,6 +85,33 @@ void __init kvm_hyp_reserve(void)
>                  hyp_mem_base);
>  }
>
> +static int pkvm_hyp_topup(enum pkvm_topup_id id, unsigned long nr_pages)
> +{
> +       struct kvm_hyp_memcache mc;
> +       struct arm_smccc_res res;
> +       int ret;
> +
> +       init_hyp_memcache(&mc);
> +       ret = topup_hyp_memcache(&mc, nr_pages);
> +       if (ret)
> +               goto err;
> +
> +       arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(__pkvm_hyp_topup), id, mc.head,
> +                         mc.nr_pages, &res);
> +       if (WARN_ON_ONCE(res.a0 != SMCCC_RET_SUCCESS)) {
> +               ret = -EINVAL;
> +               goto err;
> +       }
> +
> +       ret = res.a1;
> +       mc.head = res.a2;
> +       mc.nr_pages = res.a3;
> +
> +err:
> +       free_hyp_memcache(&mc);
> +       return ret;
> +}
> +
>  static void __pkvm_destroy_hyp_vm(struct kvm *kvm)
>  {
>         if (pkvm_hyp_vm_is_created(kvm)) {
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 07/18] KVM: arm64: Add pkvm_hyp_req infrastructure

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> Introduce a struct pkvm_hyp_req to enable the pKVM hypervisor to request
> resources from the host.
>
> Provide serialisation helpers to transport these requests via SMCCC
> registers (starting from a2):
>
>   pkvm_hyp_req_to_smccc() to encode into the SMCCC args.
>   smccc_to_pkvm_hyp_req() to decode them.
>
> When the hypervisor raises a request, the host must handle it and retry
> the HVC. To automate this sequence, introduce the pkvm_call_hyp_req()
> macro. This intercepts pending requests, invokes the handler and retries
> the HVC.
>
> Additionally, introduce a trace event to track the handling of these
> requests.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad

>
> diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h
> index 870f7870bb62..0be1469e7874 100644
> --- a/arch/arm64/include/asm/kvm_pkvm.h
> +++ b/arch/arm64/include/asm/kvm_pkvm.h
> @@ -203,6 +203,95 @@ struct pkvm_mapping {
>         u64 __subtree_last;     /* Internal member for interval tree */
>  };
>
> +enum pkvm_hyp_req_type {
> +       PKVM_HYP_NO_REQ = 0,
> +       __PKVM_HYP_REQ_TYPE_MAX,
> +};
> +
> +#define PKVM_HYP_REQ_SMCCC_ARG_SIZE_MAX \
> +       (sizeof(struct arm_smccc_res) - offsetof(struct arm_smccc_res, a2) - 1)
> +
> +struct pkvm_hyp_req {
> +       u8 type;
> +       union {
> +               struct {
> +                       u32     nr_pages;
> +               } mem;
> +               struct {
> +                       /* Helper for SMCCC encoding/decoding */
> +                       u8      args[PKVM_HYP_REQ_SMCCC_ARG_SIZE_MAX];
> +               } args;
> +       };
> +};
> +
> +static inline size_t pkvm_hyp_req_arg_size(u8 type)
> +{
> +       switch (type) {
> +       case PKVM_HYP_NO_REQ:
> +               return 0;
> +       default:
> +               WARN_ON(1);
> +       }
> +
> +       return 0;
> +}
> +
> +/* Encode the pending pkvm_hyp_req type into the SMCCC args */
> +static inline void
> +pkvm_hyp_req_to_smccc(struct kvm_cpu_context *host_ctxt, struct pkvm_hyp_req *req)
> +{
> +       u8 *dst, type = req->type;
> +       size_t size;
> +
> +       if (type == PKVM_HYP_NO_REQ || type >= __PKVM_HYP_REQ_TYPE_MAX) {
> +               host_ctxt->regs.regs[2] = 0;
> +               return;
> +       }
> +
> +       size = pkvm_hyp_req_arg_size(type);
> +       if (WARN_ON(size > PKVM_HYP_REQ_SMCCC_ARG_SIZE_MAX))
> +               return;
> +
> +       dst = (u8 *)&host_ctxt->regs.regs[2];
> +       *dst = type;
> +
> +       memcpy(dst + 1, &req->args, size);
> +}
> +
> +/* Return true if a pkvm_hyp_req has been decoded from the SMCCC args */
> +static inline bool smccc_to_pkvm_hyp_req(struct pkvm_hyp_req *req, struct arm_smccc_res *res)
> +{
> +       u8 *src = (u8 *)res + offsetof(struct arm_smccc_res, a2);
> +       u8 type = *src;
> +
> +       if (type == PKVM_HYP_NO_REQ || type >= __PKVM_HYP_REQ_TYPE_MAX)
> +               return false;
> +
> +       req->type = type;
> +       memcpy(&req->args, src + 1, pkvm_hyp_req_arg_size(type));
> +
> +       return true;
> +}
> +
> +int __pkvm_handle_smccc_req(struct arm_smccc_res *res);
> +
> +#define pkvm_call_hyp_req(f, ...)                                                              \
> +({                                                                                             \
> +       struct arm_smccc_res __res;                                                             \
> +       int __ret;                                                                              \
> +       do {                                                                                    \
> +               __ret = -1;                                                                     \
> +               arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(f), ##__VA_ARGS__, &__res);               \
> +               if (WARN_ON(__res.a0 != SMCCC_RET_SUCCESS))                                     \
> +                       break;                                                                  \
> +               __ret = __res.a1;                                                               \
> +               if (!__ret)                                                                     \
> +                       break;                                                                  \
> +               __ret = __pkvm_handle_smccc_req(&__res);                                        \
> +       } while (!__ret);                                                                       \
> +       __ret;                                                                                  \
> +})
> +
>  int pkvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
>                              struct kvm_pgtable_mm_ops *mm_ops);
>  void pkvm_pgtable_stage2_destroy_range(struct kvm_pgtable *pgt,
> diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
> index e2c8714e8ccc..5ba8b81b8c6c 100644
> --- a/arch/arm64/kvm/pkvm.c
> +++ b/arch/arm64/kvm/pkvm.c
> @@ -16,6 +16,9 @@
>
>  #include "hyp_constants.h"
>
> +#define CREATE_TRACE_POINTS
> +#include "trace_pkvm.h"
> +
>  DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized);
>
>  static struct memblock_region *hyp_memory = kvm_nvhe_sym(hyp_memory);
> @@ -617,3 +620,25 @@ bool pkvm_force_reclaim_guest_page(phys_addr_t phys)
>
>         return !ret || ret == -EAGAIN;
>  }
> +
> +static int pkvm_handle_hyp_req(struct pkvm_hyp_req *req)
> +{
> +       int ret = -EINVAL;
> +
> +       switch (req->type) {
> +       }
> +
> +       trace_kvm_handle_pkvm_hyp_req(req, ret);
> +
> +       return ret;
> +}
> +
> +int __pkvm_handle_smccc_req(struct arm_smccc_res *res)
> +{
> +       struct pkvm_hyp_req req;
> +
> +       if (smccc_to_pkvm_hyp_req(&req, res))
> +               return pkvm_handle_hyp_req(&req);
> +
> +       return res->a1;
> +}
> diff --git a/arch/arm64/kvm/trace_pkvm.h b/arch/arm64/kvm/trace_pkvm.h
> new file mode 100644
> index 000000000000..4bf57c12e7de
> --- /dev/null
> +++ b/arch/arm64/kvm/trace_pkvm.h
> @@ -0,0 +1,37 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#if !defined(_TRACE_PKVM_ARM64_KVM_H) || defined(TRACE_HEADER_MULTI_READ)
> +#define _TRACE_PKVM_ARM64_KVM_H
> +
> +#include <linux/tracepoint.h>
> +#include <asm/kvm_pkvm.h>
> +
> +#undef TRACE_SYSTEM
> +#define TRACE_SYSTEM kvm
> +
> +TRACE_EVENT(kvm_handle_pkvm_hyp_req,
> +       TP_PROTO(struct pkvm_hyp_req *req, int ret),
> +       TP_ARGS(req, ret),
> +
> +       TP_STRUCT__entry(
> +               __field(u8,     type)
> +               __field(int,    ret)
> +       ),
> +
> +       TP_fast_assign(
> +               __entry->type = req->type;
> +               __entry->ret = ret;
> +       ),
> +
> +       TP_printk("type: %u ret: %d",
> +                 __entry->type, __entry->ret)
> +);
> +
> +#endif /* _TRACE_PKVM_ARM64_KVM_H */
> +
> +#undef TRACE_INCLUDE_PATH
> +#define TRACE_INCLUDE_PATH .
> +#undef TRACE_INCLUDE_FILE
> +#define TRACE_INCLUDE_FILE trace_pkvm
> +
> +/* This part must be outside protection */
> +#include <trace/define_trace.h>
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 09/18] KVM: arm64: Add reclaim interface for the pKVM heap alloc

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:

...

> +
> +unsigned long pkvm_hyp_reclaimable(enum pkvm_topup_id id)
> +{
> +       return kvm_call_hyp_nvhe(__pkvm_hyp_reclaimable, id);
> +}

Missing new line.

With that fixed:

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad

>  static void __pkvm_destroy_hyp_vm(struct kvm *kvm)
>  {
>         if (pkvm_hyp_vm_is_created(kvm)) {
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 08/18] KVM: arm64: Handle PKVM_HYP_REQ_HYP_ALLOC request

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> Introduce a new pkvm_hyp_request type asking the host to top up the pKVM
> heap allocator.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
>
> diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h
> index 0be1469e7874..99daa985856f 100644
> --- a/arch/arm64/include/asm/kvm_pkvm.h
> +++ b/arch/arm64/include/asm/kvm_pkvm.h
> @@ -205,6 +205,7 @@ struct pkvm_mapping {
>
>  enum pkvm_hyp_req_type {
>         PKVM_HYP_NO_REQ = 0,
> +       PKVM_HYP_REQ_HYP_ALLOC,

Now that the enum has a request type the tracepoint can actually fire
for, would you be able to print it symbolically:
TRACE_DEFINE_ENUM(PKVM_HYP_REQ_HYP_ALLOC) plus __print_symbolic() in
kvm_handle_pkvm_hyp_req's TP_printk, so traces read HYP_ALLOC rather
than a raw number. I'm referring to trace_pkvm.h from the previous
patch.

Cheers,
/fuad



>         __PKVM_HYP_REQ_TYPE_MAX,
>  };
>
> @@ -226,9 +227,13 @@ struct pkvm_hyp_req {
>
>  static inline size_t pkvm_hyp_req_arg_size(u8 type)
>  {
> +       struct pkvm_hyp_req *req;
> +
>         switch (type) {
>         case PKVM_HYP_NO_REQ:
>                 return 0;
> +       case PKVM_HYP_REQ_HYP_ALLOC:
> +               return sizeof(req->mem);
>         default:
>                 WARN_ON(1);
>         }
> diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
> index 5ba8b81b8c6c..adc418076cb0 100644
> --- a/arch/arm64/kvm/pkvm.c
> +++ b/arch/arm64/kvm/pkvm.c
> @@ -626,6 +626,9 @@ static int pkvm_handle_hyp_req(struct pkvm_hyp_req *req)
>         int ret = -EINVAL;
>
>         switch (req->type) {
> +       case PKVM_HYP_REQ_HYP_ALLOC:
> +               ret = pkvm_hyp_topup(PKVM_TOPUP_HYP_ALLOC, req->mem.nr_pages);
> +               break;
>         }
>
>         trace_kvm_handle_pkvm_hyp_req(req, ret);
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 03/18] KVM: arm64: Add __hyp_allocator_map for the pKVM hyp

[Vincent Donnefort]

On Tue, Jul 14, 2026 at 06:53:55PM +0100, Fuad Tabba wrote:
> On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
> >
> > In preparation for the pKVM heap allocator, introduce __hyp_allocator_map()
> > to map a single physical page to a given virtual address in the hypervisor
> > stage-1 page-table.
> >
> > Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
> >
> > diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h
> > index 725bb0fb941d..98a7774b541c 100644
> > --- a/arch/arm64/kvm/hyp/include/nvhe/mm.h
> > +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h
> > @@ -32,5 +32,6 @@ int __pkvm_create_private_mapping(phys_addr_t phys, size_t size,
> >  int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr);
> >  int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr);
> >  phys_addr_t __pkvm_private_range_pa(void *va);
> > +int __hyp_allocator_map(void *va, phys_addr_t phys);
> >
> >  #endif /* __KVM_HYP_MM_H */
> > diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c
> > index ec59da1322a7..c1c01f81ac5f 100644
> > --- a/arch/arm64/kvm/hyp/nvhe/mm.c
> > +++ b/arch/arm64/kvm/hyp/nvhe/mm.c
> > @@ -160,6 +160,11 @@ void pkvm_remove_mappings(void *from, void *to)
> >         hyp_spin_unlock(&pkvm_pgd_lock);
> >  }
> >
> > +int __hyp_allocator_map(void *va, phys_addr_t phys)
> > +{
> > +       return __pkvm_create_mappings((unsigned long)va, PAGE_SIZE, phys, PAGE_HYP);
> > +}
> > +
> 
> nit: __hyp_allocator_map() is named for its caller rather than what it does
> (map a single hyp page). Its siblings here (pkvm_create_mappings,
> pkvm_remove_mappings) are named for the operation. Consider a name like
> pkvm_map_page() so it reads as a generic primitive.

IIRC (it was a few years ago) this was a suggestion from Quentin who didn't want
to expose too much of mm and suggested to make it clear that was a function
private to the allocator.

But looking at what we have now. perhaps 

int __pkvm_map_private_pages(phys_addr_t phys, void *addr, size_t size);

And this could verify that addr is actually aprt of the private range

WDYS?

> 
> That said, naming is hard:
> 
> Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
> 
> Cheers,
> /fuad
> 
> >  int hyp_back_vmemmap(phys_addr_t back)
> >  {
> >         unsigned long i, start, size, end = 0;
> > --
> > 2.55.0.rc2.803.g1fd1e6609c-goog
> >

Re: [PATCH v2 03/18] KVM: arm64: Add __hyp_allocator_map for the pKVM hyp

[Fuad Tabba]

On Wed, 15 Jul 2026 at 09:58, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> On Tue, Jul 14, 2026 at 06:53:55PM +0100, Fuad Tabba wrote:
> > On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
> > >
> > > In preparation for the pKVM heap allocator, introduce __hyp_allocator_map()
> > > to map a single physical page to a given virtual address in the hypervisor
> > > stage-1 page-table.
> > >
> > > Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
> > >
> > > diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h
> > > index 725bb0fb941d..98a7774b541c 100644
> > > --- a/arch/arm64/kvm/hyp/include/nvhe/mm.h
> > > +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h
> > > @@ -32,5 +32,6 @@ int __pkvm_create_private_mapping(phys_addr_t phys, size_t size,
> > >  int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr);
> > >  int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr);
> > >  phys_addr_t __pkvm_private_range_pa(void *va);
> > > +int __hyp_allocator_map(void *va, phys_addr_t phys);
> > >
> > >  #endif /* __KVM_HYP_MM_H */
> > > diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c
> > > index ec59da1322a7..c1c01f81ac5f 100644
> > > --- a/arch/arm64/kvm/hyp/nvhe/mm.c
> > > +++ b/arch/arm64/kvm/hyp/nvhe/mm.c
> > > @@ -160,6 +160,11 @@ void pkvm_remove_mappings(void *from, void *to)
> > >         hyp_spin_unlock(&pkvm_pgd_lock);
> > >  }
> > >
> > > +int __hyp_allocator_map(void *va, phys_addr_t phys)
> > > +{
> > > +       return __pkvm_create_mappings((unsigned long)va, PAGE_SIZE, phys, PAGE_HYP);
> > > +}
> > > +
> >
> > nit: __hyp_allocator_map() is named for its caller rather than what it does
> > (map a single hyp page). Its siblings here (pkvm_create_mappings,
> > pkvm_remove_mappings) are named for the operation. Consider a name like
> > pkvm_map_page() so it reads as a generic primitive.
>
> IIRC (it was a few years ago) this was a suggestion from Quentin who didn't want
> to expose too much of mm and suggested to make it clear that was a function
> private to the allocator.
>
> But looking at what we have now. perhaps
>
> int __pkvm_map_private_pages(phys_addr_t phys, void *addr, size_t size);
>
> And this could verify that addr is actually aprt of the private range
>
> WDYS?

I think this sounds better. That was a nit, so up to you.

Thanks,
/fuad

>
> >
> > That said, naming is hard:
> >
> > Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
> >
> > Cheers,
> > /fuad
> >
> > >  int hyp_back_vmemmap(phys_addr_t back)
> > >  {
> > >         unsigned long i, start, size, end = 0;
> > > --
> > > 2.55.0.rc2.803.g1fd1e6609c-goog
> > >

Re: [PATCH v2 04/18] KVM: arm64: Add a heap allocator for the pKVM hyp

[Fuad Tabba]

On Tue, 14 Jul 2026 at 19:35, Fuad Tabba <fuad.tabba@linux.dev> wrote:
>
> On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> ...
>
> > +static u32 chunk_hash_compute(const struct chunk_hdr *chunk)
> > +{
> > +       u32 hash = 0;
> > +
> > +       BUILD_BUG_ON(sizeof(*chunk) != 16);
> > +
> > +       hash ^= hash_64(*(const u64 *)chunk, 32);
> > +       hash ^= hash_32(chunk->__unmapped, 32);
>
> Could you state the threat model in the commit msg? The VA range is
> hyp-private and the hash is unkeyed (anything that can write EL2
> memory recomputes it), so this only catches _accidental_ corruption,
> i.e. a debug aid.
>
> Since that's the goal, should it be gated on CONFIG_NVHE_EL2_DEBUG
> like the other checks here? As written it's O(n) hashing per
> hyp_alloc(), in production, for a check that only fires on a hyp bug.
> And a plain XOR/additive checksum would be cheaper and just as good
> for accidental corruption.

Thinking about it, I was wrong to say that it would be just as good,
slightly cheaper, but it would detect things like swaps.

Still, I think this should be gated.

Cheers,
/fuad

>
> > +       return hash;
> > +}
> > +
>
> ...
>
>
> > diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c
> > index 75b00c323310..1c10cf78c869 100644
> > --- a/arch/arm64/kvm/hyp/nvhe/setup.c
> > +++ b/arch/arm64/kvm/hyp/nvhe/setup.c
> > @@ -10,6 +10,7 @@
> >  #include <asm/kvm_pgtable.h>
> >  #include <asm/kvm_pkvm.h>
> >
> > +#include <nvhe/alloc.h>
> >  #include <nvhe/early_alloc.h>
> >  #include <nvhe/ffa.h>
> >  #include <nvhe/gfp.h>
> > @@ -363,6 +364,10 @@ int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long *per_cpu_bas
> >         if (ret)
> >                 return ret;
> >
> > +       ret = hyp_alloc_init(SZ_128M);
>
> Where does 128M come from? A comment would help.
>
> Cheers,
> /fuad
>
>
>
> > +       if (ret)
> > +               return ret;
> > +
> >         update_nvhe_init_params();
> >
> >         /* Jump in the idmap page to switch to the new page-tables */
> > --
> > 2.55.0.rc2.803.g1fd1e6609c-goog
> >

Re: [PATCH v2 10/18] KVM: arm64: Add selftests for the pKVM heap allocator

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> Introduce a comprehensive runtime selftest for the pKVM hypervisor heap
> allocator, executed during init when CONFIG_NVHE_EL2_DEBUG is enabled.
>
> The selftest runs entirely at EL2 and exercises allocator's core
> mechanisms:
>
>   * over-sized allocations
>   * basic allocation and alignment
>   * chunk recycling, splitting, merging
>   * memory reclaiming
>   * memory topup
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad

>
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index b427ef790b15..07a46860c8b2 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -117,6 +117,7 @@ enum __kvm_host_smccc_func {
>         __KVM_HOST_SMCCC_FUNC___pkvm_hyp_topup,
>         __KVM_HOST_SMCCC_FUNC___pkvm_hyp_reclaim,
>         __KVM_HOST_SMCCC_FUNC___pkvm_hyp_reclaimable,
> +       __KVM_HOST_SMCCC_FUNC___pkvm_hyp_alloc_selftest,
>
>         MARKER(__KVM_HOST_SMCCC_FUNC_MAX)
>  };
> diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h
> index 3a8590d1793e..5837c2d7928a 100644
> --- a/arch/arm64/include/asm/kvm_pkvm.h
> +++ b/arch/arm64/include/asm/kvm_pkvm.h
> @@ -19,6 +19,7 @@
>
>  enum pkvm_topup_id {
>         PKVM_TOPUP_HYP_ALLOC,
> +       PKVM_TOPUP_HYP_ALLOC_SELFTEST,
>  };
>
>  unsigned long pkvm_hyp_reclaim(enum pkvm_topup_id id, unsigned long target);
> @@ -209,6 +210,7 @@ struct pkvm_mapping {
>  enum pkvm_hyp_req_type {
>         PKVM_HYP_NO_REQ = 0,
>         PKVM_HYP_REQ_HYP_ALLOC,
> +       PKVM_HYP_REQ_HYP_ALLOC_SELFTEST,
>         __PKVM_HYP_REQ_TYPE_MAX,
>  };
>
> @@ -236,6 +238,7 @@ static inline size_t pkvm_hyp_req_arg_size(u8 type)
>         case PKVM_HYP_NO_REQ:
>                 return 0;
>         case PKVM_HYP_REQ_HYP_ALLOC:
> +       case PKVM_HYP_REQ_HYP_ALLOC_SELFTEST:
>                 return sizeof(req->mem);
>         default:
>                 WARN_ON(1);
> diff --git a/arch/arm64/kvm/hyp/include/nvhe/alloc.h b/arch/arm64/kvm/hyp/include/nvhe/alloc.h
> index 8f87a63f8946..329250dad6f6 100644
> --- a/arch/arm64/kvm/hyp/include/nvhe/alloc.h
> +++ b/arch/arm64/kvm/hyp/include/nvhe/alloc.h
> @@ -14,4 +14,11 @@ int hyp_alloc_init(size_t size);
>  int hyp_alloc_topup(struct kvm_hyp_memcache *host_mc);
>  unsigned long hyp_alloc_reclaimable(void);
>  void hyp_alloc_reclaim(struct kvm_hyp_memcache *host_mc, unsigned long target);
> +
> +#ifdef CONFIG_NVHE_EL2_DEBUG
> +int hyp_allocator_selftest(void);
> +u32 hyp_alloc_selftest_topup_needed(void);
> +int hyp_alloc_selftest_topup(struct kvm_hyp_memcache *host_mc);
> +void hyp_alloc_selftest_reclaim(struct kvm_hyp_memcache *host_mc, unsigned long target);
> +#endif
>  #endif
> diff --git a/arch/arm64/kvm/hyp/nvhe/alloc.c b/arch/arm64/kvm/hyp/nvhe/alloc.c
> index 07ef0a13855f..e7114dde9142 100644
> --- a/arch/arm64/kvm/hyp/nvhe/alloc.c
> +++ b/arch/arm64/kvm/hyp/nvhe/alloc.c
> @@ -1007,9 +1007,17 @@ int hyp_alloc_errno(void)
>         return hyp_allocator_errno(&hyp_allocator);
>  }
>
> +static int selftest_init(void);
> +
>  int hyp_alloc_init(size_t size)
>  {
> -       return hyp_allocator_init(&hyp_allocator, size);
> +       int ret;
> +
> +       ret = hyp_allocator_init(&hyp_allocator, size);
> +       if (ret)
> +               return ret;
> +
> +       return selftest_init();
>  }
>
>  void hyp_alloc_reclaim(struct kvm_hyp_memcache *mc, unsigned long target)
> @@ -1031,3 +1039,179 @@ u32 hyp_alloc_topup_needed(void)
>  {
>         return hyp_allocator_topup_needed(&hyp_allocator);
>  }
> +
> +#ifdef CONFIG_NVHE_EL2_DEBUG
> +#define SELFTEST_MAX_PAGES 6
> +#define SELFTEST_MAX_SIZE (PAGE_SIZE * SELFTEST_MAX_PAGES)
> +
> +static DEFINE_PER_CPU(int, __selftest_errno);
> +static DEFINE_PER_CPU(u32, __selftest_topup_needed);
> +
> +static struct hyp_allocator selftest_allocator = {
> +       .errno = &__selftest_errno,
> +       .topup_needed = &__selftest_topup_needed,
> +       .lock = __HYP_SPIN_LOCK_UNLOCKED,
> +};
> +
> +int hyp_alloc_selftest_topup(struct kvm_hyp_memcache *host_mc)
> +{
> +       return hyp_allocator_topup(&selftest_allocator, host_mc);
> +}
> +
> +void hyp_alloc_selftest_reclaim(struct kvm_hyp_memcache *host_mc, unsigned long target)
> +{
> +       hyp_allocator_reclaim(&selftest_allocator, host_mc, target);
> +}
> +
> +u32 hyp_alloc_selftest_topup_needed(void)
> +{
> +       return hyp_allocator_topup_needed(&selftest_allocator);
> +}
> +
> +static int selftest_init(void)
> +{
> +       return hyp_allocator_init(&selftest_allocator, SELFTEST_MAX_SIZE);
> +}
> +
> +static void *selftest_alloc(size_t size)
> +{
> +       return hyp_allocator_alloc(&selftest_allocator, size);
> +}
> +
> +static void selftest_free(void *addr)
> +{
> +       hyp_allocator_free(&selftest_allocator, addr);
> +}
> +
> +static int selftest_errno(void)
> +{
> +       return hyp_allocator_errno(&selftest_allocator);
> +}
> +
> +int hyp_allocator_selftest(void)
> +{
> +       struct hyp_allocator *allocator = &selftest_allocator;
> +       static DEFINE_HYP_SPINLOCK(selftest_lock);
> +       struct kvm_hyp_memcache host_mc = { };
> +       void *addr1, *addr2, *addr3, *addr4;
> +       int ret;
> +
> +       guard(hyp_spinlock)(&selftest_lock);
> +
> +       if (allocator->mc.nr_pages < SELFTEST_MAX_PAGES) {
> +               *this_cpu_ptr(allocator->topup_needed) = SELFTEST_MAX_PAGES -
> +                                                        allocator->mc.nr_pages;
> +               return -ENOMEM;
> +       }
> +
> +       selftest_alloc(SELFTEST_MAX_SIZE);
> +       if (selftest_errno() != -E2BIG)
> +               return -EINVAL;
> +
> +       selftest_alloc(SIZE_MAX);
> +       if (selftest_errno() != -E2BIG)
> +               return -EINVAL;
> +
> +       /* Test first chunk */
> +       addr1 = selftest_alloc(0);
> +       if (!addr1 || addr1 != (void *)allocator->start + chunk_hdr_size())
> +               return -EINVAL;
> +
> +       /* Test second contiguous chunk with unaligned size */
> +       addr2 = selftest_alloc(MIN_ALLOC_SIZE + 1);
> +       if (!addr2)
> +               return -EINVAL;
> +       addr3 = selftest_alloc(0);
> +       if (!addr3 ||
> +           addr3 != addr2 + (2 * MIN_ALLOC_SIZE) + chunk_hdr_size())
> +               return -EINVAL;
> +
> +       selftest_free(addr3);
> +
> +       /* Test chunk recycling */
> +       selftest_free(addr1);
> +       if (addr1 != selftest_alloc(0))
> +               return -EINVAL;
> +
> +       /* Test chunk forward merging */
> +       addr3 = selftest_alloc(0);
> +       selftest_free(addr2);
> +       selftest_free(addr1);
> +       if (addr1 != selftest_alloc(MIN_ALLOC_SIZE * 2))
> +               return -EINVAL;
> +
> +       selftest_free(addr1);
> +
> +       /* Test chunk splitting */
> +       if (addr1 != selftest_alloc(0))
> +               return -EINVAL;
> +       if (addr2 != selftest_alloc(0))
> +               return -EINVAL;
> +
> +       /* Test chunk backward merging */
> +       selftest_free(addr1);
> +       selftest_free(addr2);
> +       if (addr1 != selftest_alloc(MIN_ALLOC_SIZE * 2))
> +               return -EINVAL;
> +
> +       selftest_free(addr1);
> +
> +       /* Test chunk 3-way merging */
> +       addr1 = selftest_alloc(0);
> +       addr2 = selftest_alloc(0);
> +       addr4 = selftest_alloc(0);
> +       selftest_free(addr1);
> +       selftest_free(addr3);
> +       selftest_free(addr2);
> +       if (addr1 != selftest_alloc(MIN_ALLOC_SIZE * 3))
> +               return -EINVAL;
> +
> +       selftest_free(addr4);
> +       selftest_free(addr1);
> +
> +       /* Test reclaiming */
> +       if (addr1 != selftest_alloc(0))
> +               return -EINVAL;
> +       if (addr2 != selftest_alloc(PAGE_SIZE * 2))
> +               return -EINVAL;
> +       addr3 = selftest_alloc(0);
> +       addr4 = selftest_alloc(PAGE_SIZE);
> +
> +       /* Test reclaiming the last chunk of the list */
> +       selftest_free(addr4);
> +       hyp_allocator_reclaim(allocator, &host_mc, SELFTEST_MAX_PAGES);
> +       if (host_mc.nr_pages != SELFTEST_MAX_PAGES - 3)
> +               return -EINVAL;
> +
> +       /* Test punching a hole in the middle of a free chunk ... */
> +       selftest_free(addr2);
> +       hyp_allocator_reclaim(allocator, &host_mc, SELFTEST_MAX_PAGES);
> +       if (host_mc.nr_pages != SELFTEST_MAX_PAGES - 2)
> +               return -EINVAL;
> +
> +       if (selftest_alloc(PAGE_SIZE))
> +               return -EINVAL;
> +       if (selftest_errno() != -ENOMEM)
> +               return -EINVAL;
> +
> +       /* ... and to refill this hole */
> +       ret = hyp_allocator_topup(allocator, &host_mc);
> +       if (ret)
> +               return ret;
> +       /* Chunk at addr2 was made smaller by the reclaim */
> +       if (addr2 != selftest_alloc(PAGE_SIZE))
> +               return -EINVAL;
> +
> +       /* Test reclaiming the entire allocator from the host */
> +       selftest_free(addr3);
> +       selftest_free(addr2);
> +       selftest_free(addr1);
> +       if (addr1 != selftest_alloc(SELFTEST_MAX_PAGES * PAGE_SIZE - chunk_hdr_size()))
> +               return -EINVAL;
> +       selftest_free(addr1);
> +
> +       return 0;
> +}
> +#else
> +static int selftest_init(void) { return 0; }
> +#endif
> diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
> index 8fbebe64a93d..105313bb3dd9 100644
> --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
> +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
> @@ -633,6 +633,28 @@ static void handle___pkvm_finalize_teardown_vm(struct kvm_cpu_context *host_ctxt
>         cpu_reg(host_ctxt, 1) = __pkvm_finalize_teardown_vm(handle);
>  }
>
> +#ifdef CONFIG_NVHE_EL2_DEBUG
> +static void handle___pkvm_hyp_alloc_selftest(struct kvm_cpu_context *host_ctxt)
> +{
> +       struct pkvm_hyp_req req = { .type = PKVM_HYP_NO_REQ };
> +       int ret;
> +
> +       ret = hyp_allocator_selftest();
> +       if (ret == -ENOMEM) {
> +               req.type = PKVM_HYP_REQ_HYP_ALLOC_SELFTEST;
> +               req.mem.nr_pages = hyp_alloc_selftest_topup_needed();
> +       }
> +
> +       cpu_reg(host_ctxt, 1) = ret;
> +       pkvm_hyp_req_to_smccc(host_ctxt, &req);
> +}
> +#else
> +static void handle___pkvm_hyp_alloc_selftest(struct kvm_cpu_context *host_ctxt)
> +{
> +       cpu_reg(host_ctxt, 1) = -EPERM;
> +}
> +#endif
> +
>  static void handle___pkvm_hyp_topup(struct kvm_cpu_context *host_ctxt)
>  {
>         DECLARE_REG(enum pkvm_topup_id, id, host_ctxt, 1);
> @@ -648,6 +670,11 @@ static void handle___pkvm_hyp_topup(struct kvm_cpu_context *host_ctxt)
>         case PKVM_TOPUP_HYP_ALLOC:
>                 ret = hyp_alloc_topup(&host_mc);
>                 break;
> +#ifdef CONFIG_NVHE_EL2_DEBUG
> +       case PKVM_TOPUP_HYP_ALLOC_SELFTEST:
> +               ret = hyp_alloc_selftest_topup(&host_mc);
> +               break;
> +#endif
>         default:
>                 ret = -EINVAL;
>         }
> @@ -668,6 +695,11 @@ static void handle___pkvm_hyp_reclaim(struct kvm_cpu_context *host_ctxt)
>         case PKVM_TOPUP_HYP_ALLOC:
>                 hyp_alloc_reclaim(&host_mc, target);
>                 break;
> +#ifdef CONFIG_NVHE_EL2_DEBUG
> +       case PKVM_TOPUP_HYP_ALLOC_SELFTEST:
> +               hyp_alloc_selftest_reclaim(&host_mc, target);
> +               break;
> +#endif
>         default:
>                 ret = -EINVAL;
>         }
> @@ -826,6 +858,7 @@ static const hcall_t host_hcall[] = {
>         HANDLE_FUNC(__pkvm_hyp_topup),
>         HANDLE_FUNC(__pkvm_hyp_reclaim),
>         HANDLE_FUNC(__pkvm_hyp_reclaimable),
> +       HANDLE_FUNC(__pkvm_hyp_alloc_selftest),
>  };
>
>  static void handle_host_hcall(struct kvm_cpu_context *host_ctxt)
> diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
> index eb4c60a0db5a..f7e1846d6093 100644
> --- a/arch/arm64/kvm/pkvm.c
> +++ b/arch/arm64/kvm/pkvm.c
> @@ -137,6 +137,7 @@ unsigned long pkvm_hyp_reclaimable(enum pkvm_topup_id id)
>  {
>         return kvm_call_hyp_nvhe(__pkvm_hyp_reclaimable, id);
>  }
> +
>  static void __pkvm_destroy_hyp_vm(struct kvm *kvm)
>  {
>         if (pkvm_hyp_vm_is_created(kvm)) {
> @@ -325,6 +326,22 @@ static int __init pkvm_drop_host_privileges(void)
>         return ret;
>  }
>
> +static void __init pkvm_selftests(void)
> +{
> +#ifdef CONFIG_NVHE_EL2_DEBUG
> +       int ret = pkvm_call_hyp_req(__pkvm_hyp_alloc_selftest);
> +       unsigned long reclaimed;
> +
> +       reclaimed = pkvm_hyp_reclaim(PKVM_TOPUP_HYP_ALLOC_SELFTEST, ULONG_MAX);
> +
> +       /* On failure, not all the pages may be reclaimable */
> +       if (!ret)
> +               WARN_ON(reclaimed != 6 /* SELFTEST_MAX_PAGES */);
> +       else
> +               kvm_err("pKVM hyp allocator selftest failed (%d)\n", ret);
> +#endif
> +}
> +
>  static int __init finalize_pkvm(void)
>  {
>         int ret;
> @@ -345,6 +362,9 @@ static int __init finalize_pkvm(void)
>         if (ret)
>                 pr_err("Failed to finalize Hyp protection: %d\n", ret);
>
> +       if (!ret)
> +               pkvm_selftests();
> +
>         return ret;
>  }
>  device_initcall_sync(finalize_pkvm);
> @@ -651,6 +671,9 @@ static int pkvm_handle_hyp_req(struct pkvm_hyp_req *req)
>         case PKVM_HYP_REQ_HYP_ALLOC:
>                 ret = pkvm_hyp_topup(PKVM_TOPUP_HYP_ALLOC, req->mem.nr_pages);
>                 break;
> +       case PKVM_HYP_REQ_HYP_ALLOC_SELFTEST:
> +               ret = pkvm_hyp_topup(PKVM_TOPUP_HYP_ALLOC_SELFTEST, req->mem.nr_pages);
> +               break;
>         }
>
>         trace_kvm_handle_pkvm_hyp_req(req, ret);
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>
>

Re: [PATCH v2 11/18] KVM: arm64: Add a shrinker for pKVM

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> Integrate the pKVM memory reclaim interface with the host's memory
> management subsystem.
>
> This allows the host to automatically recover unused memory fom the
> hypervisor's heap allocator when the host is under memory pressure.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
>
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index 50adfff75be8..3dc2b9d2c322 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -122,6 +122,8 @@ static bool vgic_present, kvm_arm_initialised;
>
>  static DEFINE_PER_CPU(unsigned char, kvm_hyp_initialized);
>
> +static struct shrinker *pkvm_shrinker;
> +
>  bool is_kvm_arm_initialised(void)
>  {
>         return kvm_arm_initialised;
> @@ -2526,8 +2528,8 @@ static void __init teardown_hyp_mode(void)
>                 }
>
>                 free_pages(kvm_nvhe_sym(kvm_arm_hyp_percpu_base)[cpu], nvhe_percpu_order());
> -
>         }
> +       shrinker_free(pkvm_shrinker);
>  }
>
>  static int __init do_pkvm_init(u32 hyp_va_bits)
> @@ -2713,6 +2715,18 @@ static void pkvm_hyp_init_ptrauth(void)
>         }
>  }
>
> +static unsigned long
> +pkvm_shrinker_count(struct shrinker *shrink, struct shrink_control *sc)
> +{
> +       return pkvm_hyp_reclaimable(PKVM_TOPUP_HYP_ALLOC) ?: SHRINK_EMPTY;
> +}
> +
> +static unsigned long
> +pkvm_shrinker_scan(struct shrinker *shrink, struct shrink_control *sc)
> +{
> +       return pkvm_hyp_reclaim(PKVM_TOPUP_HYP_ALLOC, sc->nr_to_scan);
> +}
> +
>  /* Inits Hyp-mode on all online CPUs */
>  static int __init init_hyp_mode(void)
>  {
> @@ -2883,6 +2897,16 @@ static int __init init_hyp_mode(void)
>                         kvm_err("Failed to init hyp memory protection\n");
>                         goto out_err;
>                 }
> +
> +               pkvm_shrinker = shrinker_alloc(0, "pkvm");
> +               if (pkvm_shrinker) {
> +                       pkvm_shrinker->count_objects = pkvm_shrinker_count;
> +                       pkvm_shrinker->scan_objects = pkvm_shrinker_scan;
> +                       shrinker_register(pkvm_shrinker);
> +               } else {
> +                       kvm_err("Failed to register shrinker for pKVM\n");
> +               }
> +
>         }

Sashiko might be onto something here. The shrinker goes live in
init_hyp_mode(), but kvm_protected_mode_initialized isn't set until
finalize_pkvm()...

Cheers,
/fuad

>
>         return 0;
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 12/18] KVM: arm64: Filter out non-kernel addresses in kern_hyp_va

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> kern_hyp_va() is idempotent for the hypervisor linear space. This is
> handy for nVHE hypervisor callers handling kvm_vcpu or kvm_arch
> pointers. Those pointers can originate from the hypervisor space (when
> protected mode is enabled, we don't trust the kernel and the hypervisor
> uses its own copy) or from the kernel space (we do trust the kernel in
> "non-protected" nVHE).
>
> This idempotence does not hold for addresses within the hypervisor
> private range, like the ones you get from the pKVM heap allocator
> (hyp_alloc()). To resolve this, filter out non-kernel addresses based on
> PAGE_OFFSET.
>
> Leave the assembly version untouched as it has no current users.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad

>
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index 6eae7e7e2a68..d60e5f2de10c 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -126,6 +126,9 @@ static __always_inline unsigned long __kern_hyp_va(unsigned long v)
>   * replace the instructions with `nop`s.
>   */
>  #ifndef __KVM_VHE_HYPERVISOR__
> +       if (!is_ttbr1_addr(v))
> +               return v;
> +
>         asm volatile(ALTERNATIVE_CB("and %0, %0, #1\n"         /* mask with va_mask */
>                                     "ror %0, %0, #1\n"         /* rotate to the first tag bit */
>                                     "add %0, %0, #0\n"         /* insert the low 12 bits of the tag */
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 13/18] KVM: arm64: Move hyp_vm refcount into the structure

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> In preparation for allocating hyp_vm using the pKVM heap allocator
> (hyp_alloc()), move its reference count out of the page metadata
> (vmemmap) and place it into the structure itself. This transition is
> necessary because hyp_alloc() allows multiple small objects to share the
> same physical page.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad


>
> diff --git a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
> index c904647d2f76..624367d0ef5b 100644
> --- a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
> +++ b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
> @@ -41,6 +41,7 @@ struct pkvm_hyp_vm {
>         struct kvm_pgtable pgt;
>         struct kvm_pgtable_mm_ops mm_ops;
>         struct hyp_pool pool;
> +       unsigned short refcount;
>         hyp_spinlock_t lock;
>
>         /* Array of the hyp vCPU structures for this VM. */
> @@ -65,6 +66,18 @@ static inline bool pkvm_hyp_vm_is_protected(struct pkvm_hyp_vm *hyp_vm)
>         return kvm_vm_is_protected(&hyp_vm->kvm);
>  }
>
> +static inline void pkvm_hyp_vm_ref_inc(struct pkvm_hyp_vm *hyp_vm)
> +{
> +       BUG_ON(hyp_vm->refcount == USHRT_MAX);
> +       hyp_vm->refcount++;
> +}
> +
> +static inline void pkvm_hyp_vm_ref_dec(struct pkvm_hyp_vm *hyp_vm)
> +{
> +       BUG_ON(!hyp_vm->refcount);
> +       hyp_vm->refcount--;
> +}
> +
>  void pkvm_hyp_vm_table_init(void *tbl);
>
>  int __pkvm_reserve_vm(void);
> diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c
> index 3b2c4fbc34d8..d9ab58226889 100644
> --- a/arch/arm64/kvm/hyp/nvhe/pkvm.c
> +++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c
> @@ -278,7 +278,7 @@ struct pkvm_hyp_vcpu *pkvm_load_hyp_vcpu(pkvm_handle_t handle,
>         }
>
>         hyp_vcpu->loaded_hyp_vcpu = this_cpu_ptr(&loaded_hyp_vcpu);
> -       hyp_page_ref_inc(hyp_virt_to_page(hyp_vm));
> +       pkvm_hyp_vm_ref_inc(hyp_vm);
>  unlock:
>         hyp_spin_unlock(&vm_table_lock);
>
> @@ -294,7 +294,7 @@ void pkvm_put_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
>         hyp_spin_lock(&vm_table_lock);
>         hyp_vcpu->loaded_hyp_vcpu = NULL;
>         __this_cpu_write(loaded_hyp_vcpu, NULL);
> -       hyp_page_ref_dec(hyp_virt_to_page(hyp_vm));
> +       pkvm_hyp_vm_ref_dec(hyp_vm);
>         hyp_spin_unlock(&vm_table_lock);
>  }
>
> @@ -311,7 +311,7 @@ struct pkvm_hyp_vm *get_pkvm_hyp_vm(pkvm_handle_t handle)
>         hyp_spin_lock(&vm_table_lock);
>         hyp_vm = get_vm_by_handle(handle);
>         if (hyp_vm)
> -               hyp_page_ref_inc(hyp_virt_to_page(hyp_vm));
> +               pkvm_hyp_vm_ref_inc(hyp_vm);
>         hyp_spin_unlock(&vm_table_lock);
>
>         return hyp_vm;
> @@ -320,7 +320,7 @@ struct pkvm_hyp_vm *get_pkvm_hyp_vm(pkvm_handle_t handle)
>  void put_pkvm_hyp_vm(struct pkvm_hyp_vm *hyp_vm)
>  {
>         hyp_spin_lock(&vm_table_lock);
> -       hyp_page_ref_dec(hyp_virt_to_page(hyp_vm));
> +       pkvm_hyp_vm_ref_dec(hyp_vm);
>         hyp_spin_unlock(&vm_table_lock);
>  }
>
> @@ -966,7 +966,7 @@ static struct pkvm_hyp_vm *get_pkvm_unref_hyp_vm_locked(pkvm_handle_t handle)
>         hyp_assert_lock_held(&vm_table_lock);
>
>         hyp_vm = get_vm_by_handle(handle);
> -       if (!hyp_vm || hyp_page_count(hyp_vm))
> +       if (!hyp_vm || hyp_vm->refcount)
>                 return NULL;
>
>         return hyp_vm;
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 11/18] KVM: arm64: Add a shrinker for pKVM

[Vincent Donnefort]

On Wed, Jul 15, 2026 at 12:20:51PM +0100, Fuad Tabba wrote:
> On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
> >
> > Integrate the pKVM memory reclaim interface with the host's memory
> > management subsystem.
> >
> > This allows the host to automatically recover unused memory fom the
> > hypervisor's heap allocator when the host is under memory pressure.
> >
> > Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
> >
> > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> > index 50adfff75be8..3dc2b9d2c322 100644
> > --- a/arch/arm64/kvm/arm.c
> > +++ b/arch/arm64/kvm/arm.c
> > @@ -122,6 +122,8 @@ static bool vgic_present, kvm_arm_initialised;
> >
> >  static DEFINE_PER_CPU(unsigned char, kvm_hyp_initialized);
> >
> > +static struct shrinker *pkvm_shrinker;
> > +
> >  bool is_kvm_arm_initialised(void)
> >  {
> >         return kvm_arm_initialised;
> > @@ -2526,8 +2528,8 @@ static void __init teardown_hyp_mode(void)
> >                 }
> >
> >                 free_pages(kvm_nvhe_sym(kvm_arm_hyp_percpu_base)[cpu], nvhe_percpu_order());
> > -
> >         }
> > +       shrinker_free(pkvm_shrinker);
> >  }
> >
> >  static int __init do_pkvm_init(u32 hyp_va_bits)
> > @@ -2713,6 +2715,18 @@ static void pkvm_hyp_init_ptrauth(void)
> >         }
> >  }
> >
> > +static unsigned long
> > +pkvm_shrinker_count(struct shrinker *shrink, struct shrink_control *sc)
> > +{
> > +       return pkvm_hyp_reclaimable(PKVM_TOPUP_HYP_ALLOC) ?: SHRINK_EMPTY;
> > +}
> > +
> > +static unsigned long
> > +pkvm_shrinker_scan(struct shrinker *shrink, struct shrink_control *sc)
> > +{
> > +       return pkvm_hyp_reclaim(PKVM_TOPUP_HYP_ALLOC, sc->nr_to_scan);
> > +}
> > +
> >  /* Inits Hyp-mode on all online CPUs */
> >  static int __init init_hyp_mode(void)
> >  {
> > @@ -2883,6 +2897,16 @@ static int __init init_hyp_mode(void)
> >                         kvm_err("Failed to init hyp memory protection\n");
> >                         goto out_err;
> >                 }
> > +
> > +               pkvm_shrinker = shrinker_alloc(0, "pkvm");
> > +               if (pkvm_shrinker) {
> > +                       pkvm_shrinker->count_objects = pkvm_shrinker_count;
> > +                       pkvm_shrinker->scan_objects = pkvm_shrinker_scan;
> > +                       shrinker_register(pkvm_shrinker);
> > +               } else {
> > +                       kvm_err("Failed to register shrinker for pKVM\n");
> > +               }
> > +
> >         }
> 
> Sashiko might be onto something here. The shrinker goes live in
> init_hyp_mode(), but kvm_protected_mode_initialized isn't set until
> finalize_pkvm()...

I doubt the shrinker can be triggered that early... BUT 

The hyp tracing is a user of this hyp allocator and I have been working on
enabling early tracing [1] so I shall move those HVCs as well!

[1] https://lore.kernel.org/all/20260708075435.47419-2-vdonnefort@google.com/

> 
> Cheers,
> /fuad
> 
> >
> >         return 0;
> > --
> > 2.55.0.rc2.803.g1fd1e6609c-goog
> >

Re: [PATCH v2 14/18] KVM: arm64: Use noclear for PGD in __pkvm_init_vm error path

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> In the error path of __pkvm_init_vm(), use unmap_donated_memory_noclear()
> instead of the clearing variant to release the donated stage-2 PGD back
> to the host.
>
> This intends to eliminate the clearing variant of
> unmap_donated_memory(), as zeroing the PGD memory before returning it to
> the host is unnecessary in this failure path.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
>
> diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c
> index d9ab58226889..0c82f1ddb93d 100644
> --- a/arch/arm64/kvm/hyp/nvhe/pkvm.c
> +++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c
> @@ -861,7 +861,7 @@ int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva,
>         kvm_guest_destroy_stage2(hyp_vm);
>  err_remove_mappings:
>         unmap_donated_memory(hyp_vm, vm_size);
> -       unmap_donated_memory(pgd, pgd_size);
> +       unmap_donated_memory_noclear(pgd, pgd_size);


We spoke about this in person before you sent the series and I thought
it was ok. Sashiko's review made me doubt at first, since the freed
blocks carry EL2 list_head pointers that the noclear variant hands
back to the host.

You mentioned that the mapping isn't secret (later when we discussed
Sashiko's review in person), and I checked: kvm_compute_layout() runs
at EL1 on the host and rolls the tag itself. The host already knows
the PA of every page it donated, so it can compute those VAs anyway.
So I don't think this is a real leak.

However, this relies on the pool and hyp_vm both being donated
linear-map memory here. Once hyp_vm moves to hyp_alloc() in the
private range, the free_area anchor becomes a private-range VA, and I
haven't checked whether that's host-derivable.

Given it's the init error path and not hot, clearing costs basically
nothing, so I'd lean towards just keeping unmap_donated_memory()
rather than reasoning about derivability.

What do you think?



/fuad

>  err_unpin_kvm:
>         hyp_unpin_shared_mem(host_kvm, host_kvm + 1);
>         return ret;
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 15/18] KVM: arm64: Alloc pkvm_hyp_vm using pKVM heap allocator

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, 'Vincent Donnefort' via kernel-team
<kernel-team@android.com> wrote:

...

> --- a/arch/arm64/kvm/pkvm.c
> +++ b/arch/arm64/kvm/pkvm.c
> @@ -191,8 +191,8 @@ static int __pkvm_create_hyp_vcpu(struct kvm_vcpu *vcpu)
>   */
>  static int __pkvm_create_hyp_vm(struct kvm *kvm)
>  {
> -       size_t pgd_sz, hyp_vm_sz;
> -       void *pgd, *hyp_vm;
> +       size_t pgd_sz;
> +       void *pgd;
>         int ret;
>
>         if (kvm->created_vcpus < 1)
> @@ -209,28 +209,15 @@ static int __pkvm_create_hyp_vm(struct kvm *kvm)
>         if (!pgd)
>                 return -ENOMEM;
>
> -       /* Allocate memory to donate to hyp for vm and vcpu pointers. */
> -       hyp_vm_sz = PAGE_ALIGN(size_add(PKVM_HYP_VM_SIZE,
> -                                       size_mul(sizeof(void *),
> -                                                kvm->created_vcpus)));
> -       hyp_vm = alloc_pages_exact(hyp_vm_sz, GFP_KERNEL_ACCOUNT);
> -       if (!hyp_vm) {
> -               ret = -ENOMEM;
> -               goto free_pgd;
> -       }
> -
> -       /* Donate the VM memory to hyp and let hyp initialize it. */
> -       ret = kvm_call_hyp_nvhe(__pkvm_init_vm, kvm, hyp_vm, pgd);
> +       ret = pkvm_call_hyp_req(__pkvm_init_vm, kvm, pgd);

Sashiko might be right on the memcg point: hyp_vm used to be
GFP_KERNEL_ACCOUNT and now comes from the heap unaccounted, while pgd
next to it is still accounted, so it's inconsistent.

If we do want to fix it, would it go in pkvm_hyp_topup() to cover all
consumers rather than here?

Cheers,
/fuad

Re: [PATCH v2 15/18] KVM: arm64: Alloc pkvm_hyp_vm using pKVM heap allocator

[Vincent Donnefort]

On Wed, Jul 15, 2026 at 01:33:00PM +0100, Fuad Tabba wrote:
> On Mon, 6 Jul 2026 at 18:54, 'Vincent Donnefort' via kernel-team
> <kernel-team@android.com> wrote:
> 
> ...
> 
> > --- a/arch/arm64/kvm/pkvm.c
> > +++ b/arch/arm64/kvm/pkvm.c
> > @@ -191,8 +191,8 @@ static int __pkvm_create_hyp_vcpu(struct kvm_vcpu *vcpu)
> >   */
> >  static int __pkvm_create_hyp_vm(struct kvm *kvm)
> >  {
> > -       size_t pgd_sz, hyp_vm_sz;
> > -       void *pgd, *hyp_vm;
> > +       size_t pgd_sz;
> > +       void *pgd;
> >         int ret;
> >
> >         if (kvm->created_vcpus < 1)
> > @@ -209,28 +209,15 @@ static int __pkvm_create_hyp_vm(struct kvm *kvm)
> >         if (!pgd)
> >                 return -ENOMEM;
> >
> > -       /* Allocate memory to donate to hyp for vm and vcpu pointers. */
> > -       hyp_vm_sz = PAGE_ALIGN(size_add(PKVM_HYP_VM_SIZE,
> > -                                       size_mul(sizeof(void *),
> > -                                                kvm->created_vcpus)));
> > -       hyp_vm = alloc_pages_exact(hyp_vm_sz, GFP_KERNEL_ACCOUNT);
> > -       if (!hyp_vm) {
> > -               ret = -ENOMEM;
> > -               goto free_pgd;
> > -       }
> > -
> > -       /* Donate the VM memory to hyp and let hyp initialize it. */
> > -       ret = kvm_call_hyp_nvhe(__pkvm_init_vm, kvm, hyp_vm, pgd);
> > +       ret = pkvm_call_hyp_req(__pkvm_init_vm, kvm, pgd);
> 
> Sashiko might be right on the memcg point: hyp_vm used to be
> GFP_KERNEL_ACCOUNT and now comes from the heap unaccounted, while pgd
> next to it is still accounted, so it's inconsistent.

Yep that is correct but there is no way around that:

 * Donated memory can now be shared be shared by several VM (or other parts of
   the hypervisor)

 * Donated memory is also not reclaimed until the shrinker fires.

So we don't really have a way to account for that meta-data anymore.

(I should at least mention this in the commit message)

> 
> If we do want to fix it, would it go in pkvm_hyp_topup() to cover all
> consumers rather than here?
> 
> Cheers,
> /fuad

Re: [PATCH v2 15/18] KVM: arm64: Alloc pkvm_hyp_vm using pKVM heap allocator

[Fuad Tabba]

> > Sashiko might be right on the memcg point: hyp_vm used to be
> > GFP_KERNEL_ACCOUNT and now comes from the heap unaccounted, while pgd
> > next to it is still accounted, so it's inconsistent.
>
> Yep that is correct but there is no way around that:
>
>  * Donated memory can now be shared be shared by several VM (or other parts of
>    the hypervisor)
>
>  * Donated memory is also not reclaimed until the shrinker fires.
>
> So we don't really have a way to account for that meta-data anymore.
>
> (I should at least mention this in the commit message)

Sgtm.
/fuad

>
> >
> > If we do want to fix it, would it go in pkvm_hyp_topup() to cover all
> > consumers rather than here?
> >
> > Cheers,
> > /fuad

Re: [PATCH v2 16/18] KVM: arm64: Alloc pkvm_hyp_vcpu using pKVM heap allocator

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:

...
> -int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu,
> -                    unsigned long vcpu_hva)
> +int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu)
>  {
>         struct pkvm_hyp_vcpu *hyp_vcpu;
>         struct pkvm_hyp_vm *hyp_vm;
>         int ret;
>
> -       hyp_vcpu = map_donated_memory(vcpu_hva, sizeof(*hyp_vcpu));
> +       hyp_vcpu = hyp_alloc(sizeof(*hyp_vcpu));

On Sashiko's alignment point: hyp_alloc only guarantees 8-byte
alignment while pkvm_hyp_vcpu wants 16 (the __uint128_t vregs[32]),
and the old map_donated_memory path was page-aligned, so it's a real
reduction.

But if my understanding of the ARM ARM is correct, and please note
that I just consulted the spec when replying to this, so it's a big
*if*: I don't think it faults. We run EL2 with SCTLR_EL2.A == 0
(INIT_SCTLR_EL2_MMU_ON has SA but not A), and FPSIMD save/restore is
plain LDR/STR/LDP/STP of Q registers. From the ARM ARM (DDI 0487M.c,
B2.8.2.1.1):

"When the value of SCTLR_ELx.A applicable to the current Exception
level is 0: An unaligned access is performed."

and, covering LDP/STP:

"For these instructions, the definition of an unaligned access is
based on the size of the accessed elements, not the overall size of
the memory access. This affects SIMD and SVE element and structure
loads and stores, and also load/store pair instructions."

However, the alignment here is still weaker than the
kmalloc/alloc_pages callers relied on , so it might be cleaner to give
hyp_alloc a 16-byte minimum alignment at the allocator level rather
than per-caller. What do you think?

Cheers,
/fuad


>         if (!hyp_vcpu)
> -               return -ENOMEM;
> +               return hyp_alloc_errno();
>
>         hyp_spin_lock(&vm_table_lock);
>
> @@ -926,24 +892,12 @@ int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu,
>         }
>  unlock:
>         hyp_spin_unlock(&vm_table_lock);
> -
>         if (ret)
> -               unmap_donated_memory(hyp_vcpu, sizeof(*hyp_vcpu));
> +               hyp_free(hyp_vcpu);
> +
>         return ret;
>  }
>
> -static void
> -teardown_donated_memory(struct kvm_hyp_memcache *mc, void *addr, size_t size)
> -{
> -       size = PAGE_ALIGN(size);
> -       memset(addr, 0, size);
> -
> -       for (void *start = addr; start < addr + size; start += PAGE_SIZE)
> -               push_hyp_memcache(mc, start, hyp_virt_to_phys);
> -
> -       unmap_donated_memory_noclear(addr, size);
> -}
> -
>  int __pkvm_reclaim_dying_guest_page(pkvm_handle_t handle, u64 gfn)
>  {
>         struct pkvm_hyp_vm *hyp_vm = get_pkvm_hyp_vm(handle);
> @@ -993,7 +947,7 @@ int __pkvm_start_teardown_vm(pkvm_handle_t handle)
>
>  int __pkvm_finalize_teardown_vm(pkvm_handle_t handle)
>  {
> -       struct kvm_hyp_memcache *mc, *stage2_mc;
> +       struct kvm_hyp_memcache *stage2_mc;
>         struct pkvm_hyp_vm *hyp_vm;
>         struct kvm *host_kvm;
>         unsigned int idx;
> @@ -1014,7 +968,6 @@ int __pkvm_finalize_teardown_vm(pkvm_handle_t handle)
>         hyp_spin_unlock(&vm_table_lock);
>
>         /* Reclaim guest pages (including page-table pages) */
> -       mc = &host_kvm->arch.pkvm.teardown_mc;
>         stage2_mc = &host_kvm->arch.pkvm.stage2_teardown_mc;
>         reclaim_pgtable_pages(hyp_vm, stage2_mc);
>         unpin_host_vcpus(hyp_vm->vcpus, hyp_vm->kvm.created_vcpus);
> @@ -1036,7 +989,7 @@ int __pkvm_finalize_teardown_vm(pkvm_handle_t handle)
>                         unmap_donated_memory_noclear(addr, PAGE_SIZE);
>                 }
>
> -               teardown_donated_memory(mc, hyp_vcpu, sizeof(*hyp_vcpu));
> +               hyp_free(hyp_vcpu);
>         }
>
>         hyp_free(hyp_vm);
> diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
> index 8a8b5b84825c..1899b05d9fc2 100644
> --- a/arch/arm64/kvm/pkvm.c
> +++ b/arch/arm64/kvm/pkvm.c
> @@ -153,28 +153,19 @@ static void __pkvm_destroy_hyp_vm(struct kvm *kvm)
>
>         kvm->arch.pkvm.handle = 0;
>         kvm->arch.pkvm.is_created = false;
> -       free_hyp_memcache(&kvm->arch.pkvm.teardown_mc);
>         free_hyp_memcache(&kvm->arch.pkvm.stage2_teardown_mc);
>  }
>
>  static int __pkvm_create_hyp_vcpu(struct kvm_vcpu *vcpu)
>  {
> -       size_t hyp_vcpu_sz = PAGE_ALIGN(PKVM_HYP_VCPU_SIZE);
>         pkvm_handle_t handle = vcpu->kvm->arch.pkvm.handle;
> -       void *hyp_vcpu;
>         int ret;
>
>         init_hyp_stage2_memcache(&vcpu->arch.pkvm_memcache);
>
> -       hyp_vcpu = alloc_pages_exact(hyp_vcpu_sz, GFP_KERNEL_ACCOUNT);
> -       if (!hyp_vcpu)
> -               return -ENOMEM;
> -
> -       ret = kvm_call_hyp_nvhe(__pkvm_init_vcpu, handle, vcpu, hyp_vcpu);
> +       ret = pkvm_call_hyp_req(__pkvm_init_vcpu, handle, vcpu);
>         if (!ret)
>                 vcpu_set_flag(vcpu, VCPU_PKVM_FINALIZED);
> -       else
> -               free_pages_exact(hyp_vcpu, hyp_vcpu_sz);
>
>         return ret;
>  }
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 17/18] KVM: arm64: Reject hyp trace descriptors with fewer CPUs than hyp_nr_cpus

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> While a trace descriptor with fewer CPUs than hyp_nr_cpus currently
> causes no functional issue, such a configuration is invalid and would
> cause the hypervisor to skip ring buffer initialization when allocating
> backing pages in EL2.
>
> Enforce nr_cpus == hyp_nr_cpus during trace descriptor validation to
> guarantee that every possible CPU has a valid ring buffer descriptor.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad

>
> diff --git a/arch/arm64/kvm/hyp/nvhe/trace.c b/arch/arm64/kvm/hyp/nvhe/trace.c
> index e7e150ab265f..6e716295247a 100644
> --- a/arch/arm64/kvm/hyp/nvhe/trace.c
> +++ b/arch/arm64/kvm/hyp/nvhe/trace.c
> @@ -181,6 +181,9 @@ static bool hyp_trace_desc_is_valid(struct hyp_trace_desc *desc, size_t desc_siz
>         desc_end = (void *)desc + desc_size;
>         nr_bpages = desc->bpages_backing_size / sizeof(struct simple_buffer_page);
>
> +       if (desc->trace_buffer_desc.nr_cpus != hyp_nr_cpus)
> +               return false;
> +
>         for_each_ring_buffer_desc(rb_desc, cpu, &desc->trace_buffer_desc) {
>                 /* Can we read nr_page_va? */
>                 if ((void *)rb_desc + struct_size(rb_desc, page_va, 0) > desc_end)
> --
> 2.55.0.rc2.803.g1fd1e6609c-goog
>

Re: [PATCH v2 18/18] KVM: arm64: Alloc simple_buffer_page using pKVM hyp allocator

[Fuad Tabba]

On Mon, 6 Jul 2026 at 18:54, Vincent Donnefort <vdonnefort@google.com> wrote:

...

> @@ -193,17 +211,8 @@ static bool hyp_trace_desc_is_valid(struct hyp_trace_desc *desc, size_t desc_siz
>                 if ((void *)rb_desc + struct_size(rb_desc, page_va, rb_desc->nr_page_va) > desc_end)
>                         return false;
>
> -               /* Overflow bpages backing memory? */
> -               if (nr_bpages < rb_desc->nr_page_va)
> -                       return false;
> -
> -               if (cpu >= hyp_nr_cpus)
> -                       return false;
> -
>                 if (cpu != rb_desc->cpu)
>                         return false;

Sashiko flagged this. Nothing here bounds nr_page_va from below, so
with nr_page_va == 0 on every CPU the bpage size sums to 0.
hyp_alloc() still hands back a chunk, but bpages_backing_size is
stored as 0, so when init_mm() fails on nr_page_va < 3 the free path
takes its if (!size) return and never frees it. The host can repeat
it. Would validating nr_page_va >= 3 here work?

I don't think the rest of Sashiko's note holds: the zero-size path
succeeds rather than hitting -ENOMEM. And it's only reachable from the
host kernel, not userspace.

Cheers,
/fuad