[PATCH v1 0/4] trace_hyp_printk() for pKVM/nVHE hypervisor

[PATCH v1 0/4] trace_hyp_printk() for pKVM/nVHE hypervisor

[Vincent Donnefort]

Hi all,

This series adds a hypervisor event "hyp_printk" which enables
developers to log pretty much anything into the hypervisor tracing
buffer, just like the kernel function trace_printk().

This enables rich logging from the hypervisor, while leaving all the
string parsing burden to the kernel. This has been the main way of
debugging pKVM in Android.

Even though not strictly related to trace_hyp_printk, I have added the
following two patches:

  * KVM: arm64: Allow early calls to pKVM host_share/unshare_hyp

    This one mainly intends to support one of the new features I have
    posted here [1], which allows to enable tracing as early as
    possible. I have added it here to limit cross-posting.

  * KVM: arm64: Move kvm_define_hypevents.h to arch/arm64/kvm/

    This one is just a cleanup.

[1] https://lore.kernel.org/all/20260605163825.1762953-1-vdonnefort@google.com/

Vincent Donnefort (4):
  KVM: arm64: Allow early calls to pKVM host_share/unshare_hyp
  KVM: arm64: Move kvm_define_hypevents.h to arch/arm64/kvm/
  tracing/remotes: Add REMOTE_EVENT_CUSTOM_PRINTK() helper
  KVM: arm64: Add hyp_printk event to nVHE/pKVM hyp

 arch/arm64/include/asm/kvm_asm.h              |  4 +-
 arch/arm64/include/asm/kvm_hypevents.h        | 14 ++++
 arch/arm64/include/asm/kvm_hyptrace.h         |  8 +++
 arch/arm64/kernel/image-vars.h                |  1 +
 arch/arm64/kernel/vmlinux.lds.S               |  4 ++
 .../define_hypevents.h}                       |  0
 .../kvm/hyp/include/nvhe/define_events.h      |  2 -
 arch/arm64/kvm/hyp/include/nvhe/trace.h       | 65 +++++++++++++++++++
 arch/arm64/kvm/hyp/nvhe/events.c              |  6 ++
 arch/arm64/kvm/hyp/nvhe/hyp-main.c            |  2 +-
 arch/arm64/kvm/hyp_trace.c                    | 60 ++++++++++++++++-
 include/trace/define_remote_events.h          | 19 +++++-
 12 files changed, 176 insertions(+), 9 deletions(-)
 rename arch/arm64/{include/asm/kvm_define_hypevents.h => kvm/define_hypevents.h} (100%)


base-commit: 4549871118cf616eecdd2d939f78e3b9e1dddc48
-- 
2.54.0.1136.gdb2ca164c4-goog

[PATCH v1 1/4] KVM: arm64: Allow early calls to pKVM host_share/unshare_hyp

[Vincent Donnefort]

The hypervisor tracing for pKVM relies on the __pkvm_host_share_hyp and
__pkvm_host_unshare_hyp HVCs. In order to start tracing as early as
possible, allow those two HVCs before the host is deprivileged.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index 043495f7fc78..fb049c40d04f 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -89,12 +89,12 @@ enum __kvm_host_smccc_func {
 	__KVM_HOST_SMCCC_FUNC___vgic_v3_restore_vmcr_aprs,
 	__KVM_HOST_SMCCC_FUNC___vgic_v5_save_apr,
 	__KVM_HOST_SMCCC_FUNC___vgic_v5_restore_vmcr_apr,
+	__KVM_HOST_SMCCC_FUNC___pkvm_host_share_hyp,
+	__KVM_HOST_SMCCC_FUNC___pkvm_host_unshare_hyp,
 
 	MARKER(__KVM_HOST_SMCCC_FUNC_PKVM_ONLY),
 
 	/* Hypercalls that are available only when pKVM has finalised. */
-	__KVM_HOST_SMCCC_FUNC___pkvm_host_share_hyp,
-	__KVM_HOST_SMCCC_FUNC___pkvm_host_unshare_hyp,
 	__KVM_HOST_SMCCC_FUNC___pkvm_host_donate_guest,
 	__KVM_HOST_SMCCC_FUNC___pkvm_host_share_guest,
 	__KVM_HOST_SMCCC_FUNC___pkvm_host_unshare_guest,
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
index 06db299c37a8..f0c52667cf52 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
@@ -721,9 +721,9 @@ static const hcall_t host_hcall[] = {
 	HANDLE_FUNC(__vgic_v3_restore_vmcr_aprs),
 	HANDLE_FUNC(__vgic_v5_save_apr),
 	HANDLE_FUNC(__vgic_v5_restore_vmcr_apr),
-
 	HANDLE_FUNC(__pkvm_host_share_hyp),
 	HANDLE_FUNC(__pkvm_host_unshare_hyp),
+
 	HANDLE_FUNC(__pkvm_host_donate_guest),
 	HANDLE_FUNC(__pkvm_host_share_guest),
 	HANDLE_FUNC(__pkvm_host_unshare_guest),
-- 
2.54.0.1136.gdb2ca164c4-goog

[PATCH v1 2/4] KVM: arm64: Move kvm_define_hypevents.h to arch/arm64/kvm/

[Vincent Donnefort]

kvm_define_hypevents.h is used to define the kernel-side structures for
hypervisor events. It doesn't need to be used anywhere else than in
hyp_trace.c.

Move it to arch/arm64/kvm/

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/include/asm/kvm_define_hypevents.h b/arch/arm64/kvm/define_hypevents.h
similarity index 100%
rename from arch/arm64/include/asm/kvm_define_hypevents.h
rename to arch/arm64/kvm/define_hypevents.h
diff --git a/arch/arm64/kvm/hyp_trace.c b/arch/arm64/kvm/hyp_trace.c
index c4b3ee552131..821bc93ecdd1 100644
--- a/arch/arm64/kvm/hyp_trace.c
+++ b/arch/arm64/kvm/hyp_trace.c
@@ -391,7 +391,7 @@ static struct trace_remote_callbacks trace_remote_callbacks = {
 
 static const char *__hyp_enter_exit_reason_str(u8 reason);
 
-#include <asm/kvm_define_hypevents.h>
+#include "define_hypevents.h"
 
 static const char *__hyp_enter_exit_reason_str(u8 reason)
 {
-- 
2.54.0.1136.gdb2ca164c4-goog

[PATCH v1 3/4] tracing/remotes: Add REMOTE_EVENT_CUSTOM_PRINTK() helper

[Vincent Donnefort]

The current REMOTE_EVENT() takes as a __printk argument a string format
and a list of arguments, such as RE_STRUCT("foo=%d bar=%d", foo, bar).
Add a REMOTE_EVENT_CUSTOM_PRINTK() where the __printk argument can be a
function. This intends to support the creation of a "printk" event for
the arm64 nVHE/pKVM hypervisor with a dynamic prototype and by extension
a dynamic print format.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/include/trace/define_remote_events.h b/include/trace/define_remote_events.h
index 676e803dc144..4f4d58e37b84 100644
--- a/include/trace/define_remote_events.h
+++ b/include/trace/define_remote_events.h
@@ -35,17 +35,26 @@ do {											\
 
 #define RE_PRINTK(__args...) __args
 
-#define REMOTE_EVENT(__name, __id, __struct, __printk)					\
-	REMOTE_EVENT_FORMAT(__name, __struct);						\
+#define REMOTE_EVENT_PRINT_FUNC(__name, __printk)					\
 	static void remote_event_print_##__name(void *evt, struct trace_seq *seq)	\
 	{										\
 		struct remote_event_format_##__name __maybe_unused *__entry = evt;	\
 		trace_seq_puts(seq, #__name);						\
-		remote_printk(__printk);						\
+		__printk;								\
 	}
+
+#define REMOTE_EVENT(__name, __id, __struct, __printk)					\
+	REMOTE_EVENT_FORMAT(__name, __struct);						\
+	REMOTE_EVENT_PRINT_FUNC(__name, remote_printk(__printk))
+
+#define REMOTE_EVENT_CUSTOM_PRINTK(__name, __id, __struct, __printk)			\
+	REMOTE_EVENT_FORMAT(__name, __struct);						\
+	REMOTE_EVENT_PRINT_FUNC(__name, __printk)
+
 #include REMOTE_EVENT_INCLUDE(REMOTE_EVENT_INCLUDE_FILE)
 
 #undef REMOTE_EVENT
+#undef REMOTE_EVENT_CUSTOM_PRINTK
 #undef RE_PRINTK
 #undef re_field
 #define re_field(__type, __field)							\
@@ -70,4 +79,8 @@ do {											\
 		.print_fmt	= remote_event_print_fmt_##__name,			\
 		.print		= remote_event_print_##__name,				\
 	}
+
+#define REMOTE_EVENT_CUSTOM_PRINTK(__name, __id, __struct, __printk)			\
+	REMOTE_EVENT(__name, __id, RE_STRUCT(__struct), "Unknown")
+
 #include REMOTE_EVENT_INCLUDE(REMOTE_EVENT_INCLUDE_FILE)
-- 
2.54.0.1136.gdb2ca164c4-goog

[PATCH v1 4/4] KVM: arm64: Add hyp_printk event to nVHE/pKVM hyp

[Vincent Donnefort]

Create an event to allow developers to log pretty much anything into the
hypervisor tracing buffer with trace_hyp_printk(), just like the kernel
tracing has the function trace_printk().

  trace_hyp_printk("foobar");
  trace_hyp_printk("foo=%d", foo);
  trace_hyp_printk("foo=%d bar=0x%016llx", foo, bar);

To ensure writing into the tracing buffer is fast, store the string
format into a kernel-accessible ELF section. The hypervisor only has to
write into the event the string ID, which is the delta between the
hyp_string_fmt and the start of the ELF section.

To not waste tracing buffer data, use a dynamic size. Each
argument is 8 bytes and the in-kernel printing function can simply know
how many of them there are by looking at the event length.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

diff --git a/arch/arm64/include/asm/kvm_hypevents.h b/arch/arm64/include/asm/kvm_hypevents.h
index 743c49bd878f..8465b523cb1d 100644
--- a/arch/arm64/include/asm/kvm_hypevents.h
+++ b/arch/arm64/include/asm/kvm_hypevents.h
@@ -57,4 +57,18 @@ HYP_EVENT(selftest,
 	),
 	RE_PRINTK("id=%llu", __entry->id)
 );
+
+/*
+ * trace_hyp_printk() has too many specificities to be declared with HYP_EVENT().
+ * However, we can use a REMOTE_EVENT macro to automatically do the declaration
+ * for the kernel side.
+ */
+REMOTE_EVENT_CUSTOM_PRINTK(hyp_printk,
+	0, /* id will be overwritten during hyp event init */
+	RE_STRUCT(
+		re_field(u16, fmt_id)
+		re_field(u64, args[])
+	),
+	__hyp_trace_printk(evt, seq)
+);
 #endif
diff --git a/arch/arm64/include/asm/kvm_hyptrace.h b/arch/arm64/include/asm/kvm_hyptrace.h
index de133b735f72..46097105fdd8 100644
--- a/arch/arm64/include/asm/kvm_hyptrace.h
+++ b/arch/arm64/include/asm/kvm_hyptrace.h
@@ -23,4 +23,12 @@ extern struct remote_event __hyp_events_end[];
 extern struct hyp_event_id __hyp_event_ids_start[];
 extern struct hyp_event_id __hyp_event_ids_end[];
 
+#define HYP_STRING_FMT_MAX_SIZE 128
+
+struct hyp_string_fmt {
+	const char	fmt[HYP_STRING_FMT_MAX_SIZE];
+};
+
+extern struct hyp_string_fmt __hyp_string_fmts_start[];
+extern struct hyp_string_fmt __hyp_string_fmts_end[];
 #endif
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index d4c7d45ae6bc..ec03621d7a81 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -141,6 +141,7 @@ KVM_NVHE_ALIAS(__hyp_rodata_end);
 #ifdef CONFIG_NVHE_EL2_TRACING
 KVM_NVHE_ALIAS(__hyp_event_ids_start);
 KVM_NVHE_ALIAS(__hyp_event_ids_end);
+KVM_NVHE_ALIAS(__hyp_string_fmts_start);
 #endif
 
 /* pKVM static key */
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index e1ac876200a3..b6d62642b6bd 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -324,6 +324,10 @@ SECTIONS
 		__hyp_events_start = .;
 		*(SORT(_hyp_events.*))
 		__hyp_events_end = .;
+
+		__hyp_string_fmts_start = .;
+		*(_hyp_string_fmts)
+		__hyp_string_fmts_end = .;
 	}
 #endif
 	/*
diff --git a/arch/arm64/kvm/hyp/include/nvhe/define_events.h b/arch/arm64/kvm/hyp/include/nvhe/define_events.h
index 776d4c6cb702..370e8c2d39fe 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/define_events.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/define_events.h
@@ -10,5 +10,3 @@
 #define HYP_EVENT_MULTI_READ
 #include <asm/kvm_hypevents.h>
 #undef HYP_EVENT_MULTI_READ
-
-#undef HYP_EVENT
diff --git a/arch/arm64/kvm/hyp/include/nvhe/trace.h b/arch/arm64/kvm/hyp/include/nvhe/trace.h
index 8813ff250f8e..3d0b5c634bb3 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/trace.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/trace.h
@@ -46,6 +46,69 @@ static inline pid_t __tracing_get_vcpu_pid(struct kvm_cpu_context *host_ctxt)
 void *tracing_reserve_entry(unsigned long length);
 void tracing_commit_entry(void);
 
+/*
+ * The trace_hyp_printk boilerplate is too fiddly to be declared with
+ * HYP_EVENT():
+ *
+ * The string format is stored into a kernel-accessible ELF section. The
+ * hypervisor only writes the format ID.
+ *
+ * The function has a variadic prototype. We have no easy way to know each
+ * argument width so they must all cast to u64.
+ */
+#define REMOTE_EVENT_CUSTOM_PRINTK(...)
+
+#define __TO_U64_0()
+#define __TO_U64_1(x)			, (u64)(x)
+#define __TO_U64_2(x, ...)		, (u64)(x) __TO_U64_1(__VA_ARGS__)
+#define __TO_U64_3(x, ...)		, (u64)(x) __TO_U64_2(__VA_ARGS__)
+#define __TO_U64_4(x, ...)		, (u64)(x) __TO_U64_3(__VA_ARGS__)
+#define __TO_U64_5(x, ...)		, (u64)(x) __TO_U64_4(__VA_ARGS__)
+#define __TO_U64_6(x, ...)		, (u64)(x) __TO_U64_5(__VA_ARGS__)
+#define __TO_U64_7(x, ...)		, (u64)(x) __TO_U64_6(__VA_ARGS__)
+#define __TO_U64_8(x, ...)		, (u64)(x) __TO_U64_7(__VA_ARGS__)
+
+#define __TO_U64_X(N, ...)		CONCATENATE(__TO_U64_, N)(__VA_ARGS__)
+#define __TO_U64(...)			__TO_U64_X(COUNT_ARGS(__VA_ARGS__), ##__VA_ARGS__)
+
+REMOTE_EVENT_FORMAT(hyp_printk, HE_STRUCT(he_field(u16, fmt_id) he_field(u64, args[])));
+extern struct hyp_event_id hyp_event_id_hyp_printk;
+
+static __always_inline void __trace_hyp_printk(struct hyp_string_fmt *fmt, int nr_args, ...)
+{
+	struct remote_event_format_hyp_printk *entry;
+	va_list va;
+	int i;
+
+	if (!atomic_read(&hyp_event_id_hyp_printk.enabled))
+		return;
+
+	entry = tracing_reserve_entry(struct_size(entry, args, nr_args));
+	if (!entry)
+		return;
+
+	entry->hdr.id = hyp_event_id_hyp_printk.id;
+	entry->fmt_id = fmt - __hyp_string_fmts_start;
+
+	va_start(va, nr_args);
+	for (i = 0; i < nr_args; i++)
+		entry->args[i] = va_arg(va, u64);
+	va_end(va);
+
+	tracing_commit_entry();
+}
+
+
+#define trace_hyp_printk(__fmt, __args...)						\
+do {											\
+	static struct hyp_string_fmt __used __section("_hyp_string_fmts") fmt = {	\
+		.fmt = __fmt								\
+	};										\
+	BUILD_BUG_ON(sizeof(__fmt) > HYP_STRING_FMT_MAX_SIZE);				\
+	/* __TO_U64 prepends a comma if there are arguments */				\
+	__trace_hyp_printk(&fmt, COUNT_ARGS(__args) __TO_U64(__args));			\
+} while (0)
+
 int __tracing_load(unsigned long desc_va, size_t desc_size);
 void __tracing_unload(void);
 int __tracing_enable(bool enable);
@@ -58,6 +121,8 @@ static inline void *tracing_reserve_entry(unsigned long length) { return NULL; }
 static inline void tracing_commit_entry(void) { }
 #define HYP_EVENT(__name, __proto, __struct, __assign, __printk)      \
 	static inline void trace_##__name(__proto) {}
+#define REMOTE_EVENT_CUSTOM_PRINTK(...)
+#define trace_hyp_printk(fmt, args...) do { } while (0)
 
 static inline int __tracing_load(unsigned long desc_va, size_t desc_size) { return -ENODEV; }
 static inline void __tracing_unload(void) { }
diff --git a/arch/arm64/kvm/hyp/nvhe/events.c b/arch/arm64/kvm/hyp/nvhe/events.c
index add9383aadb5..12223d2e3618 100644
--- a/arch/arm64/kvm/hyp/nvhe/events.c
+++ b/arch/arm64/kvm/hyp/nvhe/events.c
@@ -9,6 +9,12 @@
 
 #include <nvhe/define_events.h>
 
+/*
+ * The hyp_printk event is not declared with HYP_EVENT in kvm_hypevents.h,
+ * so we manually add the boilerplate here.
+ */
+HYP_EVENT(hyp_printk, 0, 0, 0, 0);
+
 int __tracing_enable_event(unsigned short id, bool enable)
 {
 	struct hyp_event_id *event_id = &__hyp_event_ids_start[id];
diff --git a/arch/arm64/kvm/hyp_trace.c b/arch/arm64/kvm/hyp_trace.c
index 821bc93ecdd1..187a8a295d6f 100644
--- a/arch/arm64/kvm/hyp_trace.c
+++ b/arch/arm64/kvm/hyp_trace.c
@@ -391,6 +391,9 @@ static struct trace_remote_callbacks trace_remote_callbacks = {
 
 static const char *__hyp_enter_exit_reason_str(u8 reason);
 
+struct remote_event_format_hyp_printk;
+static void __hyp_trace_printk(struct remote_event_format_hyp_printk *entry, struct trace_seq *seq);
+
 #include "define_hypevents.h"
 
 static const char *__hyp_enter_exit_reason_str(u8 reason)
@@ -409,6 +412,61 @@ static const char *__hyp_enter_exit_reason_str(u8 reason)
 	return strs[min(reason, HYP_REASON_UNKNOWN)];
 }
 
+static void __hyp_trace_printk(struct remote_event_format_hyp_printk *entry, struct trace_seq *seq)
+{
+	const char *fmt = (const char *)(&__hyp_string_fmts_start[entry->fmt_id]);
+	struct ring_buffer_event *evt = (void *)entry - RB_EVNT_HDR_SIZE;
+	int nr_args;
+
+	trace_seq_putc(seq, ' ');
+
+	if ((void *)fmt >= (void *)__hyp_string_fmts_end) {
+		trace_seq_printf(seq, "Unknown hyp_string_fmt ID %d\n", entry->fmt_id);
+		return;
+	}
+
+	nr_args = (ring_buffer_event_length(evt) -
+		   offsetof(struct remote_event_format_hyp_printk, args)) / sizeof(entry->args[0]);
+	switch (nr_args) {
+	case 0:
+		trace_seq_printf(seq, fmt);
+		break;
+	case 1:
+		trace_seq_printf(seq, fmt, entry->args[0]);
+		break;
+	case 2:
+		trace_seq_printf(seq, fmt, entry->args[0], entry->args[1]);
+		break;
+	case 3:
+		trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2]);
+		break;
+	case 4:
+		trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2],
+				 entry->args[3]);
+		break;
+	case 5:
+		trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2],
+				 entry->args[3], entry->args[4]);
+		break;
+	case 6:
+		trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2],
+				 entry->args[3], entry->args[4], entry->args[5]);
+		break;
+	case 7:
+		trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2],
+				 entry->args[3], entry->args[4], entry->args[5], entry->args[6]);
+		break;
+	default:
+		trace_seq_printf(seq, fmt, entry->args[0], entry->args[1],
+				 entry->args[2], entry->args[3], entry->args[4], entry->args[5],
+				 entry->args[6], entry->args[7]);
+		break;
+	}
+
+	if (seq->buffer[trace_seq_used(seq) - 1] != '\n')
+		trace_seq_putc(seq, '\n');
+}
+
 static void __init hyp_trace_init_events(void)
 {
 	struct hyp_event_id *hyp_event_id = __hyp_event_ids_start;
-- 
2.54.0.1136.gdb2ca164c4-goog

Re: [PATCH v1 0/4] trace_hyp_printk() for pKVM/nVHE hypervisor

[Fuad Tabba]

Hi Vincent,

On Fri, 12 Jun 2026 at 15:22, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> Hi all,
>
> This series adds a hypervisor event "hyp_printk" which enables
> developers to log pretty much anything into the hypervisor tracing
> buffer, just like the kernel function trace_printk().
>
> This enables rich logging from the hypervisor, while leaving all the
> string parsing burden to the kernel. This has been the main way of
> debugging pKVM in Android.

I tested the series on v7.1-rc7 under QEMU (cortex-a53 CPU, pKVM nVHE):
- Booted a host under pKVM with a non-protected kvmtool guest (npVM)
and a protected kvmtool guest (pVM).
- Functional test: added a temporary trace_hyp_printk() call site in
handle___kvm_vcpu_run() with 0-arg, 1-arg, and 2-arg calls. Mounted
tracefs, enabled the hyp_printk event, ran a kvmtool guest to trigger
vcpu_run, read the trace buffer. All expected entries appeared with
correctly formatted output.

One question: kvm_hyp_trace_init() returns early when
is_kernel_in_hyp_mode() is true. On VHE-capable hardware, pKVM uses
hVHE. So it seams that the entire hyp tracing subsystem (not just
hyp_printk) is non-functional in hVHE mode. Is hVHE support
intentionally deferred?

Cheers,
/fuad

>
> Even though not strictly related to trace_hyp_printk, I have added the
> following two patches:
>
>   * KVM: arm64: Allow early calls to pKVM host_share/unshare_hyp
>
>     This one mainly intends to support one of the new features I have
>     posted here [1], which allows to enable tracing as early as
>     possible. I have added it here to limit cross-posting.
>
>   * KVM: arm64: Move kvm_define_hypevents.h to arch/arm64/kvm/
>
>     This one is just a cleanup.
>
> [1] https://lore.kernel.org/all/20260605163825.1762953-1-vdonnefort@google.com/
>
> Vincent Donnefort (4):
>   KVM: arm64: Allow early calls to pKVM host_share/unshare_hyp
>   KVM: arm64: Move kvm_define_hypevents.h to arch/arm64/kvm/
>   tracing/remotes: Add REMOTE_EVENT_CUSTOM_PRINTK() helper
>   KVM: arm64: Add hyp_printk event to nVHE/pKVM hyp
>
>  arch/arm64/include/asm/kvm_asm.h              |  4 +-
>  arch/arm64/include/asm/kvm_hypevents.h        | 14 ++++
>  arch/arm64/include/asm/kvm_hyptrace.h         |  8 +++
>  arch/arm64/kernel/image-vars.h                |  1 +
>  arch/arm64/kernel/vmlinux.lds.S               |  4 ++
>  .../define_hypevents.h}                       |  0
>  .../kvm/hyp/include/nvhe/define_events.h      |  2 -
>  arch/arm64/kvm/hyp/include/nvhe/trace.h       | 65 +++++++++++++++++++
>  arch/arm64/kvm/hyp/nvhe/events.c              |  6 ++
>  arch/arm64/kvm/hyp/nvhe/hyp-main.c            |  2 +-
>  arch/arm64/kvm/hyp_trace.c                    | 60 ++++++++++++++++-
>  include/trace/define_remote_events.h          | 19 +++++-
>  12 files changed, 176 insertions(+), 9 deletions(-)
>  rename arch/arm64/{include/asm/kvm_define_hypevents.h => kvm/define_hypevents.h} (100%)
>
>
> base-commit: 4549871118cf616eecdd2d939f78e3b9e1dddc48
> --
> 2.54.0.1136.gdb2ca164c4-goog
>

Re: [PATCH v1 1/4] KVM: arm64: Allow early calls to pKVM host_share/unshare_hyp

[Fuad Tabba]

On Fri, 12 Jun 2026 at 15:22, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> The hypervisor tracing for pKVM relies on the __pkvm_host_share_hyp and
> __pkvm_host_unshare_hyp HVCs. In order to start tracing as early as
> possible, allow those two HVCs before the host is deprivileged.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

To my good friend Sashiko:
https://lore.kernel.org/all/20260529121755.2923500-1-tabba@google.com/

The hyp_trace_load() issue seems legit though.

As for this patch itself:

Reviewed-by: Fuad Tabba <tabba@google.com>
Tested-by: Fuad Tabba <tabba@google.com>

Cheers,
/fuad


>
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index 043495f7fc78..fb049c40d04f 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -89,12 +89,12 @@ enum __kvm_host_smccc_func {
>         __KVM_HOST_SMCCC_FUNC___vgic_v3_restore_vmcr_aprs,
>         __KVM_HOST_SMCCC_FUNC___vgic_v5_save_apr,
>         __KVM_HOST_SMCCC_FUNC___vgic_v5_restore_vmcr_apr,
> +       __KVM_HOST_SMCCC_FUNC___pkvm_host_share_hyp,
> +       __KVM_HOST_SMCCC_FUNC___pkvm_host_unshare_hyp,
>
>         MARKER(__KVM_HOST_SMCCC_FUNC_PKVM_ONLY),
>
>         /* Hypercalls that are available only when pKVM has finalised. */
> -       __KVM_HOST_SMCCC_FUNC___pkvm_host_share_hyp,
> -       __KVM_HOST_SMCCC_FUNC___pkvm_host_unshare_hyp,
>         __KVM_HOST_SMCCC_FUNC___pkvm_host_donate_guest,
>         __KVM_HOST_SMCCC_FUNC___pkvm_host_share_guest,
>         __KVM_HOST_SMCCC_FUNC___pkvm_host_unshare_guest,
> diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
> index 06db299c37a8..f0c52667cf52 100644
> --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
> +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
> @@ -721,9 +721,9 @@ static const hcall_t host_hcall[] = {
>         HANDLE_FUNC(__vgic_v3_restore_vmcr_aprs),
>         HANDLE_FUNC(__vgic_v5_save_apr),
>         HANDLE_FUNC(__vgic_v5_restore_vmcr_apr),
> -
>         HANDLE_FUNC(__pkvm_host_share_hyp),
>         HANDLE_FUNC(__pkvm_host_unshare_hyp),
> +
>         HANDLE_FUNC(__pkvm_host_donate_guest),
>         HANDLE_FUNC(__pkvm_host_share_guest),
>         HANDLE_FUNC(__pkvm_host_unshare_guest),
> --
> 2.54.0.1136.gdb2ca164c4-goog
>

Re: [PATCH v1 2/4] KVM: arm64: Move kvm_define_hypevents.h to arch/arm64/kvm/

[Fuad Tabba]

Hi Vincent,

On Fri, 12 Jun 2026 at 15:22, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> kvm_define_hypevents.h is used to define the kernel-side structures for
> hypervisor events. It doesn't need to be used anywhere else than in
> hyp_trace.c.
>
> Move it to arch/arm64/kvm/

nit: rename and move

Reviewed-by: Fuad Tabba <tabba@google.com>
Tested-by: Fuad Tabba <tabba@google.com>

Cheers,
/fuad

>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
>
> diff --git a/arch/arm64/include/asm/kvm_define_hypevents.h b/arch/arm64/kvm/define_hypevents.h
> similarity index 100%
> rename from arch/arm64/include/asm/kvm_define_hypevents.h
> rename to arch/arm64/kvm/define_hypevents.h
> diff --git a/arch/arm64/kvm/hyp_trace.c b/arch/arm64/kvm/hyp_trace.c
> index c4b3ee552131..821bc93ecdd1 100644
> --- a/arch/arm64/kvm/hyp_trace.c
> +++ b/arch/arm64/kvm/hyp_trace.c
> @@ -391,7 +391,7 @@ static struct trace_remote_callbacks trace_remote_callbacks = {
>
>  static const char *__hyp_enter_exit_reason_str(u8 reason);
>
> -#include <asm/kvm_define_hypevents.h>
> +#include "define_hypevents.h"
>
>  static const char *__hyp_enter_exit_reason_str(u8 reason)
>  {
> --
> 2.54.0.1136.gdb2ca164c4-goog
>

Re: [PATCH v1 3/4] tracing/remotes: Add REMOTE_EVENT_CUSTOM_PRINTK() helper

[Fuad Tabba]

On Fri, 12 Jun 2026 at 15:22, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> The current REMOTE_EVENT() takes as a __printk argument a string format
> and a list of arguments, such as RE_STRUCT("foo=%d bar=%d", foo, bar).
> Add a REMOTE_EVENT_CUSTOM_PRINTK() where the __printk argument can be a
> function. This intends to support the creation of a "printk" event for
> the arm64 nVHE/pKVM hypervisor with a dynamic prototype and by extension
> a dynamic print format.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>

Reviewed-by: Fuad Tabba <tabba@google.com>
Tested-by: Fuad Tabba <tabba@google.com>

Cheers,
/fuad

>
> diff --git a/include/trace/define_remote_events.h b/include/trace/define_remote_events.h
> index 676e803dc144..4f4d58e37b84 100644
> --- a/include/trace/define_remote_events.h
> +++ b/include/trace/define_remote_events.h
> @@ -35,17 +35,26 @@ do {                                                                                        \
>
>  #define RE_PRINTK(__args...) __args
>
> -#define REMOTE_EVENT(__name, __id, __struct, __printk)                                 \
> -       REMOTE_EVENT_FORMAT(__name, __struct);                                          \
> +#define REMOTE_EVENT_PRINT_FUNC(__name, __printk)                                      \
>         static void remote_event_print_##__name(void *evt, struct trace_seq *seq)       \
>         {                                                                               \
>                 struct remote_event_format_##__name __maybe_unused *__entry = evt;      \
>                 trace_seq_puts(seq, #__name);                                           \
> -               remote_printk(__printk);                                                \
> +               __printk;                                                               \
>         }
> +
> +#define REMOTE_EVENT(__name, __id, __struct, __printk)                                 \
> +       REMOTE_EVENT_FORMAT(__name, __struct);                                          \
> +       REMOTE_EVENT_PRINT_FUNC(__name, remote_printk(__printk))
> +
> +#define REMOTE_EVENT_CUSTOM_PRINTK(__name, __id, __struct, __printk)                   \
> +       REMOTE_EVENT_FORMAT(__name, __struct);                                          \
> +       REMOTE_EVENT_PRINT_FUNC(__name, __printk)
> +
>  #include REMOTE_EVENT_INCLUDE(REMOTE_EVENT_INCLUDE_FILE)
>
>  #undef REMOTE_EVENT
> +#undef REMOTE_EVENT_CUSTOM_PRINTK
>  #undef RE_PRINTK
>  #undef re_field
>  #define re_field(__type, __field)                                                      \
> @@ -70,4 +79,8 @@ do {                                                                                  \
>                 .print_fmt      = remote_event_print_fmt_##__name,                      \
>                 .print          = remote_event_print_##__name,                          \
>         }
> +
> +#define REMOTE_EVENT_CUSTOM_PRINTK(__name, __id, __struct, __printk)                   \
> +       REMOTE_EVENT(__name, __id, RE_STRUCT(__struct), "Unknown")
> +
>  #include REMOTE_EVENT_INCLUDE(REMOTE_EVENT_INCLUDE_FILE)
> --
> 2.54.0.1136.gdb2ca164c4-goog
>

Re: [PATCH v1 4/4] KVM: arm64: Add hyp_printk event to nVHE/pKVM hyp

[Fuad Tabba]

Hi Vincent,

On Fri, 12 Jun 2026 at 15:22, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> Create an event to allow developers to log pretty much anything into the
> hypervisor tracing buffer with trace_hyp_printk(), just like the kernel
> tracing has the function trace_printk().
>
>   trace_hyp_printk("foobar");
>   trace_hyp_printk("foo=%d", foo);
>   trace_hyp_printk("foo=%d bar=0x%016llx", foo, bar);
>
> To ensure writing into the tracing buffer is fast, store the string
> format into a kernel-accessible ELF section. The hypervisor only has to
> write into the event the string ID, which is the delta between the
> hyp_string_fmt and the start of the ELF section.
>
> To not waste tracing buffer data, use a dynamic size. Each
> argument is 8 bytes and the in-kernel printing function can simply know
> how many of them there are by looking at the event length.
>
> Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
>
> diff --git a/arch/arm64/include/asm/kvm_hypevents.h b/arch/arm64/include/asm/kvm_hypevents.h
> index 743c49bd878f..8465b523cb1d 100644
> --- a/arch/arm64/include/asm/kvm_hypevents.h
> +++ b/arch/arm64/include/asm/kvm_hypevents.h
> @@ -57,4 +57,18 @@ HYP_EVENT(selftest,
>         ),
>         RE_PRINTK("id=%llu", __entry->id)
>  );
> +
> +/*
> + * trace_hyp_printk() has too many specificities to be declared with HYP_EVENT().
> + * However, we can use a REMOTE_EVENT macro to automatically do the declaration
> + * for the kernel side.
> + */
> +REMOTE_EVENT_CUSTOM_PRINTK(hyp_printk,
> +       0, /* id will be overwritten during hyp event init */
> +       RE_STRUCT(
> +               re_field(u16, fmt_id)
> +               re_field(u64, args[])
> +       ),
> +       __hyp_trace_printk(evt, seq)
> +);
>  #endif
> diff --git a/arch/arm64/include/asm/kvm_hyptrace.h b/arch/arm64/include/asm/kvm_hyptrace.h
> index de133b735f72..46097105fdd8 100644
> --- a/arch/arm64/include/asm/kvm_hyptrace.h
> +++ b/arch/arm64/include/asm/kvm_hyptrace.h
> @@ -23,4 +23,12 @@ extern struct remote_event __hyp_events_end[];
>  extern struct hyp_event_id __hyp_event_ids_start[];
>  extern struct hyp_event_id __hyp_event_ids_end[];
>
> +#define HYP_STRING_FMT_MAX_SIZE 128
> +
> +struct hyp_string_fmt {
> +       const char      fmt[HYP_STRING_FMT_MAX_SIZE];
> +};
> +
> +extern struct hyp_string_fmt __hyp_string_fmts_start[];
> +extern struct hyp_string_fmt __hyp_string_fmts_end[];
>  #endif
> diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
> index d4c7d45ae6bc..ec03621d7a81 100644
> --- a/arch/arm64/kernel/image-vars.h
> +++ b/arch/arm64/kernel/image-vars.h
> @@ -141,6 +141,7 @@ KVM_NVHE_ALIAS(__hyp_rodata_end);
>  #ifdef CONFIG_NVHE_EL2_TRACING
>  KVM_NVHE_ALIAS(__hyp_event_ids_start);
>  KVM_NVHE_ALIAS(__hyp_event_ids_end);
> +KVM_NVHE_ALIAS(__hyp_string_fmts_start);
>  #endif
>
>  /* pKVM static key */
> diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
> index e1ac876200a3..b6d62642b6bd 100644
> --- a/arch/arm64/kernel/vmlinux.lds.S
> +++ b/arch/arm64/kernel/vmlinux.lds.S
> @@ -324,6 +324,10 @@ SECTIONS
>                 __hyp_events_start = .;
>                 *(SORT(_hyp_events.*))
>                 __hyp_events_end = .;
> +
> +               __hyp_string_fmts_start = .;
> +               *(_hyp_string_fmts)
> +               __hyp_string_fmts_end = .;
>         }
>  #endif
>         /*
> diff --git a/arch/arm64/kvm/hyp/include/nvhe/define_events.h b/arch/arm64/kvm/hyp/include/nvhe/define_events.h
> index 776d4c6cb702..370e8c2d39fe 100644
> --- a/arch/arm64/kvm/hyp/include/nvhe/define_events.h
> +++ b/arch/arm64/kvm/hyp/include/nvhe/define_events.h
> @@ -10,5 +10,3 @@
>  #define HYP_EVENT_MULTI_READ
>  #include <asm/kvm_hypevents.h>
>  #undef HYP_EVENT_MULTI_READ
> -
> -#undef HYP_EVENT
> diff --git a/arch/arm64/kvm/hyp/include/nvhe/trace.h b/arch/arm64/kvm/hyp/include/nvhe/trace.h
> index 8813ff250f8e..3d0b5c634bb3 100644
> --- a/arch/arm64/kvm/hyp/include/nvhe/trace.h
> +++ b/arch/arm64/kvm/hyp/include/nvhe/trace.h
> @@ -46,6 +46,69 @@ static inline pid_t __tracing_get_vcpu_pid(struct kvm_cpu_context *host_ctxt)
>  void *tracing_reserve_entry(unsigned long length);
>  void tracing_commit_entry(void);
>
> +/*
> + * The trace_hyp_printk boilerplate is too fiddly to be declared with
> + * HYP_EVENT():
> + *
> + * The string format is stored into a kernel-accessible ELF section. The
> + * hypervisor only writes the format ID.
> + *
> + * The function has a variadic prototype. We have no easy way to know each
> + * argument width so they must all cast to u64.
> + */
> +#define REMOTE_EVENT_CUSTOM_PRINTK(...)
> +
> +#define __TO_U64_0()
> +#define __TO_U64_1(x)                  , (u64)(x)
> +#define __TO_U64_2(x, ...)             , (u64)(x) __TO_U64_1(__VA_ARGS__)
> +#define __TO_U64_3(x, ...)             , (u64)(x) __TO_U64_2(__VA_ARGS__)
> +#define __TO_U64_4(x, ...)             , (u64)(x) __TO_U64_3(__VA_ARGS__)
> +#define __TO_U64_5(x, ...)             , (u64)(x) __TO_U64_4(__VA_ARGS__)
> +#define __TO_U64_6(x, ...)             , (u64)(x) __TO_U64_5(__VA_ARGS__)
> +#define __TO_U64_7(x, ...)             , (u64)(x) __TO_U64_6(__VA_ARGS__)
> +#define __TO_U64_8(x, ...)             , (u64)(x) __TO_U64_7(__VA_ARGS__)
> +
> +#define __TO_U64_X(N, ...)             CONCATENATE(__TO_U64_, N)(__VA_ARGS__)
> +#define __TO_U64(...)                  __TO_U64_X(COUNT_ARGS(__VA_ARGS__), ##__VA_ARGS__)
> +
> +REMOTE_EVENT_FORMAT(hyp_printk, HE_STRUCT(he_field(u16, fmt_id) he_field(u64, args[])));
> +extern struct hyp_event_id hyp_event_id_hyp_printk;
> +
> +static __always_inline void __trace_hyp_printk(struct hyp_string_fmt *fmt, int nr_args, ...)
> +{
> +       struct remote_event_format_hyp_printk *entry;
> +       va_list va;
> +       int i;
> +
> +       if (!atomic_read(&hyp_event_id_hyp_printk.enabled))
> +               return;
> +
> +       entry = tracing_reserve_entry(struct_size(entry, args, nr_args));
> +       if (!entry)
> +               return;
> +
> +       entry->hdr.id = hyp_event_id_hyp_printk.id;
> +       entry->fmt_id = fmt - __hyp_string_fmts_start;

nit: fmt_id is u16, the subtraction is ptrdiff_t. Silent truncation if
the section ever has more than 65536 entries. Not realistic today, but
a WARN_ON on the section size in hyp_trace_init_events() would catch
it at boot for free.

Reviewed-by: Fuad Tabba <tabba@google.com>
Tested-by: Fuad Tabba <tabba@google.com>

Cheers,
/fuad


> +
> +       va_start(va, nr_args);
> +       for (i = 0; i < nr_args; i++)
> +               entry->args[i] = va_arg(va, u64);
> +       va_end(va);
> +
> +       tracing_commit_entry();
> +}
> +
> +
> +#define trace_hyp_printk(__fmt, __args...)                                             \
> +do {                                                                                   \
> +       static struct hyp_string_fmt __used __section("_hyp_string_fmts") fmt = {       \
> +               .fmt = __fmt                                                            \
> +       };                                                                              \
> +       BUILD_BUG_ON(sizeof(__fmt) > HYP_STRING_FMT_MAX_SIZE);                          \
> +       /* __TO_U64 prepends a comma if there are arguments */                          \
> +       __trace_hyp_printk(&fmt, COUNT_ARGS(__args) __TO_U64(__args));                  \
> +} while (0)
> +
>  int __tracing_load(unsigned long desc_va, size_t desc_size);
>  void __tracing_unload(void);
>  int __tracing_enable(bool enable);
> @@ -58,6 +121,8 @@ static inline void *tracing_reserve_entry(unsigned long length) { return NULL; }
>  static inline void tracing_commit_entry(void) { }
>  #define HYP_EVENT(__name, __proto, __struct, __assign, __printk)      \
>         static inline void trace_##__name(__proto) {}
> +#define REMOTE_EVENT_CUSTOM_PRINTK(...)
> +#define trace_hyp_printk(fmt, args...) do { } while (0)
>
>  static inline int __tracing_load(unsigned long desc_va, size_t desc_size) { return -ENODEV; }
>  static inline void __tracing_unload(void) { }
> diff --git a/arch/arm64/kvm/hyp/nvhe/events.c b/arch/arm64/kvm/hyp/nvhe/events.c
> index add9383aadb5..12223d2e3618 100644
> --- a/arch/arm64/kvm/hyp/nvhe/events.c
> +++ b/arch/arm64/kvm/hyp/nvhe/events.c
> @@ -9,6 +9,12 @@
>
>  #include <nvhe/define_events.h>
>
> +/*
> + * The hyp_printk event is not declared with HYP_EVENT in kvm_hypevents.h,
> + * so we manually add the boilerplate here.
> + */
> +HYP_EVENT(hyp_printk, 0, 0, 0, 0);
> +
>  int __tracing_enable_event(unsigned short id, bool enable)
>  {
>         struct hyp_event_id *event_id = &__hyp_event_ids_start[id];
> diff --git a/arch/arm64/kvm/hyp_trace.c b/arch/arm64/kvm/hyp_trace.c
> index 821bc93ecdd1..187a8a295d6f 100644
> --- a/arch/arm64/kvm/hyp_trace.c
> +++ b/arch/arm64/kvm/hyp_trace.c
> @@ -391,6 +391,9 @@ static struct trace_remote_callbacks trace_remote_callbacks = {
>
>  static const char *__hyp_enter_exit_reason_str(u8 reason);
>
> +struct remote_event_format_hyp_printk;
> +static void __hyp_trace_printk(struct remote_event_format_hyp_printk *entry, struct trace_seq *seq);
> +
>  #include "define_hypevents.h"
>
>  static const char *__hyp_enter_exit_reason_str(u8 reason)
> @@ -409,6 +412,61 @@ static const char *__hyp_enter_exit_reason_str(u8 reason)
>         return strs[min(reason, HYP_REASON_UNKNOWN)];
>  }
>
> +static void __hyp_trace_printk(struct remote_event_format_hyp_printk *entry, struct trace_seq *seq)
> +{
> +       const char *fmt = (const char *)(&__hyp_string_fmts_start[entry->fmt_id]);
> +       struct ring_buffer_event *evt = (void *)entry - RB_EVNT_HDR_SIZE;
> +       int nr_args;
> +
> +       trace_seq_putc(seq, ' ');
> +
> +       if ((void *)fmt >= (void *)__hyp_string_fmts_end) {
> +               trace_seq_printf(seq, "Unknown hyp_string_fmt ID %d\n", entry->fmt_id);
> +               return;
> +       }
> +
> +       nr_args = (ring_buffer_event_length(evt) -
> +                  offsetof(struct remote_event_format_hyp_printk, args)) / sizeof(entry->args[0]);
> +       switch (nr_args) {
> +       case 0:
> +               trace_seq_printf(seq, fmt);
> +               break;
> +       case 1:
> +               trace_seq_printf(seq, fmt, entry->args[0]);
> +               break;
> +       case 2:
> +               trace_seq_printf(seq, fmt, entry->args[0], entry->args[1]);
> +               break;
> +       case 3:
> +               trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2]);
> +               break;
> +       case 4:
> +               trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2],
> +                                entry->args[3]);
> +               break;
> +       case 5:
> +               trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2],
> +                                entry->args[3], entry->args[4]);
> +               break;
> +       case 6:
> +               trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2],
> +                                entry->args[3], entry->args[4], entry->args[5]);
> +               break;
> +       case 7:
> +               trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2],
> +                                entry->args[3], entry->args[4], entry->args[5], entry->args[6]);
> +               break;
> +       default:
> +               trace_seq_printf(seq, fmt, entry->args[0], entry->args[1],
> +                                entry->args[2], entry->args[3], entry->args[4], entry->args[5],
> +                                entry->args[6], entry->args[7]);
> +               break;
> +       }
> +
> +       if (seq->buffer[trace_seq_used(seq) - 1] != '\n')
> +               trace_seq_putc(seq, '\n');
> +}
> +
>  static void __init hyp_trace_init_events(void)
>  {
>         struct hyp_event_id *hyp_event_id = __hyp_event_ids_start;
> --
> 2.54.0.1136.gdb2ca164c4-goog
>

Re: [PATCH v1 0/4] trace_hyp_printk() for pKVM/nVHE hypervisor

[Vincent Donnefort]

On Sun, Jun 14, 2026 at 01:57:56PM +0100, Fuad Tabba wrote:
> Hi Vincent,
> 
> On Fri, 12 Jun 2026 at 15:22, Vincent Donnefort <vdonnefort@google.com> wrote:
> >
> > Hi all,
> >
> > This series adds a hypervisor event "hyp_printk" which enables
> > developers to log pretty much anything into the hypervisor tracing
> > buffer, just like the kernel function trace_printk().
> >
> > This enables rich logging from the hypervisor, while leaving all the
> > string parsing burden to the kernel. This has been the main way of
> > debugging pKVM in Android.
> 
> I tested the series on v7.1-rc7 under QEMU (cortex-a53 CPU, pKVM nVHE):
> - Booted a host under pKVM with a non-protected kvmtool guest (npVM)
> and a protected kvmtool guest (pVM).
> - Functional test: added a temporary trace_hyp_printk() call site in
> handle___kvm_vcpu_run() with 0-arg, 1-arg, and 2-arg calls. Mounted
> tracefs, enabled the hyp_printk event, ran a kvmtool guest to trigger
> vcpu_run, read the trace buffer. All expected entries appeared with
> correctly formatted output.

Thanks for the testing!

> 
> One question: kvm_hyp_trace_init() returns early when
> is_kernel_in_hyp_mode() is true. On VHE-capable hardware, pKVM uses
> hVHE. So it seams that the entire hyp tracing subsystem (not just
> hyp_printk) is non-functional in hVHE mode. Is hVHE support
> intentionally deferred?

You got me scared for a moment but I did try hVHE and it seems alright:

  [    5.369985] kvm [1]: Protected hVHE mode initialized successfully

  $ ls /sys/kernel/tracing/remotes/hypervisor/

is_kernel_in_hyp_mode() just checks if the kernel is running at EL2 which it
shouldn't in the hVHE case?

> 
> Cheers,
> /fuad
> 
> >
> > Even though not strictly related to trace_hyp_printk, I have added the
> > following two patches:
> >
> >   * KVM: arm64: Allow early calls to pKVM host_share/unshare_hyp
> >
> >     This one mainly intends to support one of the new features I have
> >     posted here [1], which allows to enable tracing as early as
> >     possible. I have added it here to limit cross-posting.
> >
> >   * KVM: arm64: Move kvm_define_hypevents.h to arch/arm64/kvm/
> >
> >     This one is just a cleanup.
> >
> > [1] https://lore.kernel.org/all/20260605163825.1762953-1-vdonnefort@google.com/
> >
> > Vincent Donnefort (4):
> >   KVM: arm64: Allow early calls to pKVM host_share/unshare_hyp
> >   KVM: arm64: Move kvm_define_hypevents.h to arch/arm64/kvm/
> >   tracing/remotes: Add REMOTE_EVENT_CUSTOM_PRINTK() helper
> >   KVM: arm64: Add hyp_printk event to nVHE/pKVM hyp
> >
> >  arch/arm64/include/asm/kvm_asm.h              |  4 +-
> >  arch/arm64/include/asm/kvm_hypevents.h        | 14 ++++
> >  arch/arm64/include/asm/kvm_hyptrace.h         |  8 +++
> >  arch/arm64/kernel/image-vars.h                |  1 +
> >  arch/arm64/kernel/vmlinux.lds.S               |  4 ++
> >  .../define_hypevents.h}                       |  0
> >  .../kvm/hyp/include/nvhe/define_events.h      |  2 -
> >  arch/arm64/kvm/hyp/include/nvhe/trace.h       | 65 +++++++++++++++++++
> >  arch/arm64/kvm/hyp/nvhe/events.c              |  6 ++
> >  arch/arm64/kvm/hyp/nvhe/hyp-main.c            |  2 +-
> >  arch/arm64/kvm/hyp_trace.c                    | 60 ++++++++++++++++-
> >  include/trace/define_remote_events.h          | 19 +++++-
> >  12 files changed, 176 insertions(+), 9 deletions(-)
> >  rename arch/arm64/{include/asm/kvm_define_hypevents.h => kvm/define_hypevents.h} (100%)
> >
> >
> > base-commit: 4549871118cf616eecdd2d939f78e3b9e1dddc48
> > --
> > 2.54.0.1136.gdb2ca164c4-goog
> >

Re: [PATCH v1 4/4] KVM: arm64: Add hyp_printk event to nVHE/pKVM hyp

[Vincent Donnefort]

On Sun, Jun 14, 2026 at 04:25:00PM +0100, Fuad Tabba wrote:
> Hi Vincent,
> 
> On Fri, 12 Jun 2026 at 15:22, Vincent Donnefort <vdonnefort@google.com> wrote:
> >
> > Create an event to allow developers to log pretty much anything into the
> > hypervisor tracing buffer with trace_hyp_printk(), just like the kernel
> > tracing has the function trace_printk().
> >
> >   trace_hyp_printk("foobar");
> >   trace_hyp_printk("foo=%d", foo);
> >   trace_hyp_printk("foo=%d bar=0x%016llx", foo, bar);
> >
> > To ensure writing into the tracing buffer is fast, store the string
> > format into a kernel-accessible ELF section. The hypervisor only has to
> > write into the event the string ID, which is the delta between the
> > hyp_string_fmt and the start of the ELF section.
> >
> > To not waste tracing buffer data, use a dynamic size. Each
> > argument is 8 bytes and the in-kernel printing function can simply know
> > how many of them there are by looking at the event length.
> >
> > Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
> >
> > diff --git a/arch/arm64/include/asm/kvm_hypevents.h b/arch/arm64/include/asm/kvm_hypevents.h
> > index 743c49bd878f..8465b523cb1d 100644
> > --- a/arch/arm64/include/asm/kvm_hypevents.h
> > +++ b/arch/arm64/include/asm/kvm_hypevents.h
> > @@ -57,4 +57,18 @@ HYP_EVENT(selftest,
> >         ),
> >         RE_PRINTK("id=%llu", __entry->id)
> >  );
> > +
> > +/*
> > + * trace_hyp_printk() has too many specificities to be declared with HYP_EVENT().
> > + * However, we can use a REMOTE_EVENT macro to automatically do the declaration
> > + * for the kernel side.
> > + */
> > +REMOTE_EVENT_CUSTOM_PRINTK(hyp_printk,
> > +       0, /* id will be overwritten during hyp event init */
> > +       RE_STRUCT(
> > +               re_field(u16, fmt_id)
> > +               re_field(u64, args[])
> > +       ),
> > +       __hyp_trace_printk(evt, seq)
> > +);
> >  #endif
> > diff --git a/arch/arm64/include/asm/kvm_hyptrace.h b/arch/arm64/include/asm/kvm_hyptrace.h
> > index de133b735f72..46097105fdd8 100644
> > --- a/arch/arm64/include/asm/kvm_hyptrace.h
> > +++ b/arch/arm64/include/asm/kvm_hyptrace.h
> > @@ -23,4 +23,12 @@ extern struct remote_event __hyp_events_end[];
> >  extern struct hyp_event_id __hyp_event_ids_start[];
> >  extern struct hyp_event_id __hyp_event_ids_end[];
> >
> > +#define HYP_STRING_FMT_MAX_SIZE 128
> > +
> > +struct hyp_string_fmt {
> > +       const char      fmt[HYP_STRING_FMT_MAX_SIZE];
> > +};
> > +
> > +extern struct hyp_string_fmt __hyp_string_fmts_start[];
> > +extern struct hyp_string_fmt __hyp_string_fmts_end[];
> >  #endif
> > diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
> > index d4c7d45ae6bc..ec03621d7a81 100644
> > --- a/arch/arm64/kernel/image-vars.h
> > +++ b/arch/arm64/kernel/image-vars.h
> > @@ -141,6 +141,7 @@ KVM_NVHE_ALIAS(__hyp_rodata_end);
> >  #ifdef CONFIG_NVHE_EL2_TRACING
> >  KVM_NVHE_ALIAS(__hyp_event_ids_start);
> >  KVM_NVHE_ALIAS(__hyp_event_ids_end);
> > +KVM_NVHE_ALIAS(__hyp_string_fmts_start);
> >  #endif
> >
> >  /* pKVM static key */
> > diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
> > index e1ac876200a3..b6d62642b6bd 100644
> > --- a/arch/arm64/kernel/vmlinux.lds.S
> > +++ b/arch/arm64/kernel/vmlinux.lds.S
> > @@ -324,6 +324,10 @@ SECTIONS
> >                 __hyp_events_start = .;
> >                 *(SORT(_hyp_events.*))
> >                 __hyp_events_end = .;
> > +
> > +               __hyp_string_fmts_start = .;
> > +               *(_hyp_string_fmts)
> > +               __hyp_string_fmts_end = .;
> >         }
> >  #endif
> >         /*
> > diff --git a/arch/arm64/kvm/hyp/include/nvhe/define_events.h b/arch/arm64/kvm/hyp/include/nvhe/define_events.h
> > index 776d4c6cb702..370e8c2d39fe 100644
> > --- a/arch/arm64/kvm/hyp/include/nvhe/define_events.h
> > +++ b/arch/arm64/kvm/hyp/include/nvhe/define_events.h
> > @@ -10,5 +10,3 @@
> >  #define HYP_EVENT_MULTI_READ
> >  #include <asm/kvm_hypevents.h>
> >  #undef HYP_EVENT_MULTI_READ
> > -
> > -#undef HYP_EVENT
> > diff --git a/arch/arm64/kvm/hyp/include/nvhe/trace.h b/arch/arm64/kvm/hyp/include/nvhe/trace.h
> > index 8813ff250f8e..3d0b5c634bb3 100644
> > --- a/arch/arm64/kvm/hyp/include/nvhe/trace.h
> > +++ b/arch/arm64/kvm/hyp/include/nvhe/trace.h
> > @@ -46,6 +46,69 @@ static inline pid_t __tracing_get_vcpu_pid(struct kvm_cpu_context *host_ctxt)
> >  void *tracing_reserve_entry(unsigned long length);
> >  void tracing_commit_entry(void);
> >
> > +/*
> > + * The trace_hyp_printk boilerplate is too fiddly to be declared with
> > + * HYP_EVENT():
> > + *
> > + * The string format is stored into a kernel-accessible ELF section. The
> > + * hypervisor only writes the format ID.
> > + *
> > + * The function has a variadic prototype. We have no easy way to know each
> > + * argument width so they must all cast to u64.
> > + */
> > +#define REMOTE_EVENT_CUSTOM_PRINTK(...)
> > +
> > +#define __TO_U64_0()
> > +#define __TO_U64_1(x)                  , (u64)(x)
> > +#define __TO_U64_2(x, ...)             , (u64)(x) __TO_U64_1(__VA_ARGS__)
> > +#define __TO_U64_3(x, ...)             , (u64)(x) __TO_U64_2(__VA_ARGS__)
> > +#define __TO_U64_4(x, ...)             , (u64)(x) __TO_U64_3(__VA_ARGS__)
> > +#define __TO_U64_5(x, ...)             , (u64)(x) __TO_U64_4(__VA_ARGS__)
> > +#define __TO_U64_6(x, ...)             , (u64)(x) __TO_U64_5(__VA_ARGS__)
> > +#define __TO_U64_7(x, ...)             , (u64)(x) __TO_U64_6(__VA_ARGS__)
> > +#define __TO_U64_8(x, ...)             , (u64)(x) __TO_U64_7(__VA_ARGS__)
> > +
> > +#define __TO_U64_X(N, ...)             CONCATENATE(__TO_U64_, N)(__VA_ARGS__)
> > +#define __TO_U64(...)                  __TO_U64_X(COUNT_ARGS(__VA_ARGS__), ##__VA_ARGS__)
> > +
> > +REMOTE_EVENT_FORMAT(hyp_printk, HE_STRUCT(he_field(u16, fmt_id) he_field(u64, args[])));
> > +extern struct hyp_event_id hyp_event_id_hyp_printk;
> > +
> > +static __always_inline void __trace_hyp_printk(struct hyp_string_fmt *fmt, int nr_args, ...)
> > +{
> > +       struct remote_event_format_hyp_printk *entry;
> > +       va_list va;
> > +       int i;
> > +
> > +       if (!atomic_read(&hyp_event_id_hyp_printk.enabled))
> > +               return;
> > +
> > +       entry = tracing_reserve_entry(struct_size(entry, args, nr_args));
> > +       if (!entry)
> > +               return;
> > +
> > +       entry->hdr.id = hyp_event_id_hyp_printk.id;
> > +       entry->fmt_id = fmt - __hyp_string_fmts_start;
> 
> nit: fmt_id is u16, the subtraction is ptrdiff_t. Silent truncation if
> the section ever has more than 65536 entries. Not realistic today, but
> a WARN_ON on the section size in hyp_trace_init_events() would catch
> it at boot for free.

Ack. That wouldn't cost anything. I'll add that in a v2

> 
> Reviewed-by: Fuad Tabba <tabba@google.com>
> Tested-by: Fuad Tabba <tabba@google.com>
> 
> Cheers,
> /fuad
> 
> 
> > +
> > +       va_start(va, nr_args);
> > +       for (i = 0; i < nr_args; i++)
> > +               entry->args[i] = va_arg(va, u64);
> > +       va_end(va);
> > +
> > +       tracing_commit_entry();
> > +}
> > +
> > +
> > +#define trace_hyp_printk(__fmt, __args...)                                             \
> > +do {                                                                                   \
> > +       static struct hyp_string_fmt __used __section("_hyp_string_fmts") fmt = {       \
> > +               .fmt = __fmt                                                            \
> > +       };                                                                              \
> > +       BUILD_BUG_ON(sizeof(__fmt) > HYP_STRING_FMT_MAX_SIZE);                          \
> > +       /* __TO_U64 prepends a comma if there are arguments */                          \
> > +       __trace_hyp_printk(&fmt, COUNT_ARGS(__args) __TO_U64(__args));                  \
> > +} while (0)
> > +
> >  int __tracing_load(unsigned long desc_va, size_t desc_size);
> >  void __tracing_unload(void);
> >  int __tracing_enable(bool enable);
> > @@ -58,6 +121,8 @@ static inline void *tracing_reserve_entry(unsigned long length) { return NULL; }
> >  static inline void tracing_commit_entry(void) { }
> >  #define HYP_EVENT(__name, __proto, __struct, __assign, __printk)      \
> >         static inline void trace_##__name(__proto) {}
> > +#define REMOTE_EVENT_CUSTOM_PRINTK(...)
> > +#define trace_hyp_printk(fmt, args...) do { } while (0)
> >
> >  static inline int __tracing_load(unsigned long desc_va, size_t desc_size) { return -ENODEV; }
> >  static inline void __tracing_unload(void) { }
> > diff --git a/arch/arm64/kvm/hyp/nvhe/events.c b/arch/arm64/kvm/hyp/nvhe/events.c
> > index add9383aadb5..12223d2e3618 100644
> > --- a/arch/arm64/kvm/hyp/nvhe/events.c
> > +++ b/arch/arm64/kvm/hyp/nvhe/events.c
> > @@ -9,6 +9,12 @@
> >
> >  #include <nvhe/define_events.h>
> >
> > +/*
> > + * The hyp_printk event is not declared with HYP_EVENT in kvm_hypevents.h,
> > + * so we manually add the boilerplate here.
> > + */
> > +HYP_EVENT(hyp_printk, 0, 0, 0, 0);
> > +
> >  int __tracing_enable_event(unsigned short id, bool enable)
> >  {
> >         struct hyp_event_id *event_id = &__hyp_event_ids_start[id];
> > diff --git a/arch/arm64/kvm/hyp_trace.c b/arch/arm64/kvm/hyp_trace.c
> > index 821bc93ecdd1..187a8a295d6f 100644
> > --- a/arch/arm64/kvm/hyp_trace.c
> > +++ b/arch/arm64/kvm/hyp_trace.c
> > @@ -391,6 +391,9 @@ static struct trace_remote_callbacks trace_remote_callbacks = {
> >
> >  static const char *__hyp_enter_exit_reason_str(u8 reason);
> >
> > +struct remote_event_format_hyp_printk;
> > +static void __hyp_trace_printk(struct remote_event_format_hyp_printk *entry, struct trace_seq *seq);
> > +
> >  #include "define_hypevents.h"
> >
> >  static const char *__hyp_enter_exit_reason_str(u8 reason)
> > @@ -409,6 +412,61 @@ static const char *__hyp_enter_exit_reason_str(u8 reason)
> >         return strs[min(reason, HYP_REASON_UNKNOWN)];
> >  }
> >
> > +static void __hyp_trace_printk(struct remote_event_format_hyp_printk *entry, struct trace_seq *seq)
> > +{
> > +       const char *fmt = (const char *)(&__hyp_string_fmts_start[entry->fmt_id]);
> > +       struct ring_buffer_event *evt = (void *)entry - RB_EVNT_HDR_SIZE;
> > +       int nr_args;
> > +
> > +       trace_seq_putc(seq, ' ');
> > +
> > +       if ((void *)fmt >= (void *)__hyp_string_fmts_end) {
> > +               trace_seq_printf(seq, "Unknown hyp_string_fmt ID %d\n", entry->fmt_id);
> > +               return;
> > +       }
> > +
> > +       nr_args = (ring_buffer_event_length(evt) -
> > +                  offsetof(struct remote_event_format_hyp_printk, args)) / sizeof(entry->args[0]);
> > +       switch (nr_args) {
> > +       case 0:
> > +               trace_seq_printf(seq, fmt);
> > +               break;
> > +       case 1:
> > +               trace_seq_printf(seq, fmt, entry->args[0]);
> > +               break;
> > +       case 2:
> > +               trace_seq_printf(seq, fmt, entry->args[0], entry->args[1]);
> > +               break;
> > +       case 3:
> > +               trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2]);
> > +               break;
> > +       case 4:
> > +               trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2],
> > +                                entry->args[3]);
> > +               break;
> > +       case 5:
> > +               trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2],
> > +                                entry->args[3], entry->args[4]);
> > +               break;
> > +       case 6:
> > +               trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2],
> > +                                entry->args[3], entry->args[4], entry->args[5]);
> > +               break;
> > +       case 7:
> > +               trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2],
> > +                                entry->args[3], entry->args[4], entry->args[5], entry->args[6]);
> > +               break;
> > +       default:
> > +               trace_seq_printf(seq, fmt, entry->args[0], entry->args[1],
> > +                                entry->args[2], entry->args[3], entry->args[4], entry->args[5],
> > +                                entry->args[6], entry->args[7]);
> > +               break;
> > +       }
> > +
> > +       if (seq->buffer[trace_seq_used(seq) - 1] != '\n')
> > +               trace_seq_putc(seq, '\n');
> > +}
> > +
> >  static void __init hyp_trace_init_events(void)
> >  {
> >         struct hyp_event_id *hyp_event_id = __hyp_event_ids_start;
> > --
> > 2.54.0.1136.gdb2ca164c4-goog
> >

Re: [PATCH v1 0/4] trace_hyp_printk() for pKVM/nVHE hypervisor

[Fuad Tabba]

On Mon, 15 Jun 2026 at 09:28, Vincent Donnefort <vdonnefort@google.com> wrote:
>
> On Sun, Jun 14, 2026 at 01:57:56PM +0100, Fuad Tabba wrote:
> > Hi Vincent,
> >
> > On Fri, 12 Jun 2026 at 15:22, Vincent Donnefort <vdonnefort@google.com> wrote:
> > >
> > > Hi all,
> > >
> > > This series adds a hypervisor event "hyp_printk" which enables
> > > developers to log pretty much anything into the hypervisor tracing
> > > buffer, just like the kernel function trace_printk().
> > >
> > > This enables rich logging from the hypervisor, while leaving all the
> > > string parsing burden to the kernel. This has been the main way of
> > > debugging pKVM in Android.
> >
> > I tested the series on v7.1-rc7 under QEMU (cortex-a53 CPU, pKVM nVHE):
> > - Booted a host under pKVM with a non-protected kvmtool guest (npVM)
> > and a protected kvmtool guest (pVM).
> > - Functional test: added a temporary trace_hyp_printk() call site in
> > handle___kvm_vcpu_run() with 0-arg, 1-arg, and 2-arg calls. Mounted
> > tracefs, enabled the hyp_printk event, ran a kvmtool guest to trigger
> > vcpu_run, read the trace buffer. All expected entries appeared with
> > correctly formatted output.
>
> Thanks for the testing!
>
> >
> > One question: kvm_hyp_trace_init() returns early when
> > is_kernel_in_hyp_mode() is true. On VHE-capable hardware, pKVM uses
> > hVHE. So it seams that the entire hyp tracing subsystem (not just
> > hyp_printk) is non-functional in hVHE mode. Is hVHE support
> > intentionally deferred?
>
> You got me scared for a moment but I did try hVHE and it seems alright:
>
>   [    5.369985] kvm [1]: Protected hVHE mode initialized successfully
>
>   $ ls /sys/kernel/tracing/remotes/hypervisor/
>
> is_kernel_in_hyp_mode() just checks if the kernel is running at EL2 which it
> shouldn't in the hVHE case?

I'll try again and report back, must be user error.

/fuad

>
> >
> > Cheers,
> > /fuad
> >
> > >
> > > Even though not strictly related to trace_hyp_printk, I have added the
> > > following two patches:
> > >
> > >   * KVM: arm64: Allow early calls to pKVM host_share/unshare_hyp
> > >
> > >     This one mainly intends to support one of the new features I have
> > >     posted here [1], which allows to enable tracing as early as
> > >     possible. I have added it here to limit cross-posting.
> > >
> > >   * KVM: arm64: Move kvm_define_hypevents.h to arch/arm64/kvm/
> > >
> > >     This one is just a cleanup.
> > >
> > > [1] https://lore.kernel.org/all/20260605163825.1762953-1-vdonnefort@google.com/
> > >
> > > Vincent Donnefort (4):
> > >   KVM: arm64: Allow early calls to pKVM host_share/unshare_hyp
> > >   KVM: arm64: Move kvm_define_hypevents.h to arch/arm64/kvm/
> > >   tracing/remotes: Add REMOTE_EVENT_CUSTOM_PRINTK() helper
> > >   KVM: arm64: Add hyp_printk event to nVHE/pKVM hyp
> > >
> > >  arch/arm64/include/asm/kvm_asm.h              |  4 +-
> > >  arch/arm64/include/asm/kvm_hypevents.h        | 14 ++++
> > >  arch/arm64/include/asm/kvm_hyptrace.h         |  8 +++
> > >  arch/arm64/kernel/image-vars.h                |  1 +
> > >  arch/arm64/kernel/vmlinux.lds.S               |  4 ++
> > >  .../define_hypevents.h}                       |  0
> > >  .../kvm/hyp/include/nvhe/define_events.h      |  2 -
> > >  arch/arm64/kvm/hyp/include/nvhe/trace.h       | 65 +++++++++++++++++++
> > >  arch/arm64/kvm/hyp/nvhe/events.c              |  6 ++
> > >  arch/arm64/kvm/hyp/nvhe/hyp-main.c            |  2 +-
> > >  arch/arm64/kvm/hyp_trace.c                    | 60 ++++++++++++++++-
> > >  include/trace/define_remote_events.h          | 19 +++++-
> > >  12 files changed, 176 insertions(+), 9 deletions(-)
> > >  rename arch/arm64/{include/asm/kvm_define_hypevents.h => kvm/define_hypevents.h} (100%)
> > >
> > >
> > > base-commit: 4549871118cf616eecdd2d939f78e3b9e1dddc48
> > > --
> > > 2.54.0.1136.gdb2ca164c4-goog
> > >

Re: [PATCH v1 0/4] trace_hyp_printk() for pKVM/nVHE hypervisor

[Fuad Tabba]

On Mon, 15 Jun 2026 at 09:30, Fuad Tabba <tabba@google.com> wrote:
>
> On Mon, 15 Jun 2026 at 09:28, Vincent Donnefort <vdonnefort@google.com> wrote:
> >
> > On Sun, Jun 14, 2026 at 01:57:56PM +0100, Fuad Tabba wrote:
> > > Hi Vincent,
> > >
> > > On Fri, 12 Jun 2026 at 15:22, Vincent Donnefort <vdonnefort@google.com> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > This series adds a hypervisor event "hyp_printk" which enables
> > > > developers to log pretty much anything into the hypervisor tracing
> > > > buffer, just like the kernel function trace_printk().
> > > >
> > > > This enables rich logging from the hypervisor, while leaving all the
> > > > string parsing burden to the kernel. This has been the main way of
> > > > debugging pKVM in Android.
> > >
> > > I tested the series on v7.1-rc7 under QEMU (cortex-a53 CPU, pKVM nVHE):
> > > - Booted a host under pKVM with a non-protected kvmtool guest (npVM)
> > > and a protected kvmtool guest (pVM).
> > > - Functional test: added a temporary trace_hyp_printk() call site in
> > > handle___kvm_vcpu_run() with 0-arg, 1-arg, and 2-arg calls. Mounted
> > > tracefs, enabled the hyp_printk event, ran a kvmtool guest to trigger
> > > vcpu_run, read the trace buffer. All expected entries appeared with
> > > correctly formatted output.
> >
> > Thanks for the testing!
> >
> > >
> > > One question: kvm_hyp_trace_init() returns early when
> > > is_kernel_in_hyp_mode() is true. On VHE-capable hardware, pKVM uses
> > > hVHE. So it seams that the entire hyp tracing subsystem (not just
> > > hyp_printk) is non-functional in hVHE mode. Is hVHE support
> > > intentionally deferred?
> >
> > You got me scared for a moment but I did try hVHE and it seems alright:
> >
> >   [    5.369985] kvm [1]: Protected hVHE mode initialized successfully
> >
> >   $ ls /sys/kernel/tracing/remotes/hypervisor/
> >
> > is_kernel_in_hyp_mode() just checks if the kernel is running at EL2 which it
> > shouldn't in the hVHE case?
>
> I'll try again and report back, must be user error.

Confirmed, user error. I must have messed something up with the hVHE test.

My Tested-by for the series covers that now too.

Sorry for the noise.
/fuad
>
> /fuad
>
> >
> > >
> > > Cheers,
> > > /fuad
> > >
> > > >
> > > > Even though not strictly related to trace_hyp_printk, I have added the
> > > > following two patches:
> > > >
> > > >   * KVM: arm64: Allow early calls to pKVM host_share/unshare_hyp
> > > >
> > > >     This one mainly intends to support one of the new features I have
> > > >     posted here [1], which allows to enable tracing as early as
> > > >     possible. I have added it here to limit cross-posting.
> > > >
> > > >   * KVM: arm64: Move kvm_define_hypevents.h to arch/arm64/kvm/
> > > >
> > > >     This one is just a cleanup.
> > > >
> > > > [1] https://lore.kernel.org/all/20260605163825.1762953-1-vdonnefort@google.com/
> > > >
> > > > Vincent Donnefort (4):
> > > >   KVM: arm64: Allow early calls to pKVM host_share/unshare_hyp
> > > >   KVM: arm64: Move kvm_define_hypevents.h to arch/arm64/kvm/
> > > >   tracing/remotes: Add REMOTE_EVENT_CUSTOM_PRINTK() helper
> > > >   KVM: arm64: Add hyp_printk event to nVHE/pKVM hyp
> > > >
> > > >  arch/arm64/include/asm/kvm_asm.h              |  4 +-
> > > >  arch/arm64/include/asm/kvm_hypevents.h        | 14 ++++
> > > >  arch/arm64/include/asm/kvm_hyptrace.h         |  8 +++
> > > >  arch/arm64/kernel/image-vars.h                |  1 +
> > > >  arch/arm64/kernel/vmlinux.lds.S               |  4 ++
> > > >  .../define_hypevents.h}                       |  0
> > > >  .../kvm/hyp/include/nvhe/define_events.h      |  2 -
> > > >  arch/arm64/kvm/hyp/include/nvhe/trace.h       | 65 +++++++++++++++++++
> > > >  arch/arm64/kvm/hyp/nvhe/events.c              |  6 ++
> > > >  arch/arm64/kvm/hyp/nvhe/hyp-main.c            |  2 +-
> > > >  arch/arm64/kvm/hyp_trace.c                    | 60 ++++++++++++++++-
> > > >  include/trace/define_remote_events.h          | 19 +++++-
> > > >  12 files changed, 176 insertions(+), 9 deletions(-)
> > > >  rename arch/arm64/{include/asm/kvm_define_hypevents.h => kvm/define_hypevents.h} (100%)
> > > >
> > > >
> > > > base-commit: 4549871118cf616eecdd2d939f78e3b9e1dddc48
> > > > --
> > > > 2.54.0.1136.gdb2ca164c4-goog
> > > >