[PATCH v2 0/6] panic, x86/vmware: Preserve crash logs before kdump

[PATCH v2 0/6] panic, x86/vmware: Preserve crash logs before kdump

[Zack Rusin]

A VMware guest may have no persistent storage or working userspace after
a crash. Preserve its newest kernel log tail in the host's vmware.log
before entering kdump, without enabling every panic notifier by default.

Following Petr's suggestion, add one generic panic_pre_kdump_list with
VMware as its first client. A set-once helper covers panic() and direct
crash-kexec entry. Fatal oopses can reach kdump without calling panic(),
so patch 2 adds that call after capturing the original registers under
the existing kexec lock. The late panic call follows sys_info() and
precedes the kmsg dumpers.

Patch 3 adds Guilherme's suggested escape hatch for debugging kdump
failures involving early callbacks. With panic_pre_kdump_postpone set,
the list runs before kdump only when crash_kexec_post_notifiers is also
set. This can omit hypervisor crash handling as well as diagnostics.
The late panic call remains eligible. This policy is a separate patch;
I can omit patch 3 if it is not wanted.

The logger preallocates an 8 KiB buffer and bounds its register-based
RPC transfer and retries. Michael suggested increasing the buffer
because 4 KiB can omit useful stack-trace context. Ordinary guests
enable logging by default; encrypted guests must opt in. Use
bool/proc_dobool for the sysctl as Joel requested. The potentially
terminating REPORTGUESTCRASH event remains a separate late dumper,
suppressed whenever a crash image is loaded. The v1 assignment to
crash_kexec_post_notifiers is dropped.

Tested on VMware Workstation and ESXi with ordinary guests, and on
ESXi with SEV-SNP and TDX guests.

v1:
https://lore.kernel.org/r/cover.1788414671.git.zack.rusin@broadcom.com

Zack Rusin (6):
  panic: Add a notifier chain for pre-kdump callbacks
  crash: Notify pre-kdump callbacks before switching kernels
  panic: Allow postponing pre-kdump notifiers
  x86/vmware: Add a bounded pre-kdump log sender
  x86/vmware: Add the vmware_record_panic_msg sysctl
  x86/vmware: Report guest crashes after kmsg dumpers

 .../admin-guide/kernel-parameters.txt         |  14 +
 Documentation/admin-guide/sysctl/kernel.rst   |  20 ++
 arch/x86/include/asm/vmware.h                 |   2 +
 arch/x86/kernel/cpu/vmware.c                  | 250 ++++++++++++++++++
 include/linux/panic_notifier.h                |   4 +
 kernel/crash_core.c                           |   3 +
 kernel/panic.c                                |  28 ++
 7 files changed, 321 insertions(+)


base-commit: fd73f4a6659897191fa0d40695fe370925dd3780

[PATCH v2 1/6] panic: Add a notifier chain for pre-kdump callbacks

[Zack Rusin]

Selected crash callbacks should not require running every panic notifier
before kdump. Add a separate atomic chain with an at-most-once dispatch
helper, and invoke it before panic reaches the kmsg dumpers.

The helper permits another call from the crash path without repeating
callbacks or recursing into them. Document that callbacks may run in NMI
context, with other CPUs running and the kexec lock held.

Link: https://lore.kernel.org/r/aquttpMVKX8e6zGB@pathway.suse.cz
Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>
---
New in v2: separate pre-kdump chain, with a set-once dispatch guard.

 include/linux/panic_notifier.h |  3 +++
 kernel/panic.c                 | 26 ++++++++++++++++++++++++++
 2 files changed, 29 insertions(+)

diff --git a/include/linux/panic_notifier.h b/include/linux/panic_notifier.h
index 41e32483d7a7..3ad390764a41 100644
--- a/include/linux/panic_notifier.h
+++ b/include/linux/panic_notifier.h
@@ -6,6 +6,9 @@
 #include <linux/types.h>
 
 extern struct atomic_notifier_head panic_notifier_list;
+extern struct atomic_notifier_head panic_pre_kdump_list;
+
+void panic_notify_pre_kdump(char *msg);
 
 extern bool crash_kexec_post_notifiers;
 
diff --git a/kernel/panic.c b/kernel/panic.c
index 213725b612aa..d78e90abb192 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -82,6 +82,10 @@ ATOMIC_NOTIFIER_HEAD(panic_notifier_list);
 
 EXPORT_SYMBOL(panic_notifier_list);
 
+struct atomic_notifier_head panic_pre_kdump_list =
+	ATOMIC_NOTIFIER_INIT(panic_pre_kdump_list);
+EXPORT_SYMBOL_GPL(panic_pre_kdump_list);
+
 static void panic_print_deprecated(void)
 {
 	pr_info_once("Kernel: The 'panic_print' parameter is now deprecated. Please use 'panic_sys_info' and 'panic_console_replay' instead.\n");
@@ -567,6 +571,27 @@ static void panic_other_cpus_shutdown(bool crash_kexec)
 		crash_smp_send_stop();
 }
 
+/**
+ * panic_notify_pre_kdump - invoke selected callbacks before kdump
+ * @msg: Panic message, or NULL for direct crash-kexec entry
+ *
+ * Dispatch at most once per boot, including recursive entry. Callbacks run
+ * with interrupts disabled, possibly in NMI context, before other CPUs have
+ * necessarily stopped. The kexec lock may be held. They must not sleep,
+ * allocate, wait for other CPUs, or call into kexec management operations.
+ * The message is read-only and may only be used during the callback.
+ */
+void panic_notify_pre_kdump(char *msg)
+{
+	static atomic_t started = ATOMIC_INIT(0);
+	unsigned long flags;
+
+	local_irq_save(flags);
+	if (!atomic_xchg(&started, 1))
+		atomic_notifier_call_chain(&panic_pre_kdump_list, 0, msg);
+	local_irq_restore(flags);
+}
+
 /**
  * vpanic - halt the system
  * @fmt: The text string to print
@@ -682,6 +707,7 @@ void vpanic(const char *fmt, va_list args)
 
 	sys_info(panic_print);
 
+	panic_notify_pre_kdump(buf);
 	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);
 
 	/*

[PATCH v2 2/6] crash: Notify pre-kdump callbacks before switching kernels

[Zack Rusin]

Fatal x86 oopses can call crash_kexec() without reaching panic(). Run the
pre-kdump chain from __crash_kexec() as well, after finding a loaded image
under the kexec lock and capturing the original registers.

This covers direct crash entry without moving all panic notifiers ahead
of kdump. A missing image or failed trylock leaves callbacks uncalled;
the shared guard skips callbacks already invoked by panic().

Link: https://lore.kernel.org/r/aquttpMVKX8e6zGB@pathway.suse.cz
Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>
---
New in v2: cover direct crash-kexec entry in a separate core patch.

 kernel/crash_core.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/crash_core.c b/kernel/crash_core.c
index 2b36aa9fade0..5d9fe9e00f39 100644
--- a/kernel/crash_core.c
+++ b/kernel/crash_core.c
@@ -23,6 +23,7 @@
 #include <linux/objtool.h>
 #include <linux/delay.h>
 #include <linux/panic.h>
+#include <linux/panic_notifier.h>
 
 #include <asm/page.h>
 #include <asm/sections.h>
@@ -139,6 +140,7 @@ void __noclone __crash_kexec(struct pt_regs *regs)
 			struct pt_regs fixed_regs;
 
 			crash_setup_regs(&fixed_regs, regs);
+			panic_notify_pre_kdump(NULL);
 			crash_save_vmcoreinfo();
 			machine_crash_shutdown(&fixed_regs);
 			crash_cma_clear_pending_dma();

[PATCH v2 3/6] panic: Allow postponing pre-kdump notifiers

[Zack Rusin]

Provide an escape hatch for debugging kdump failures that may involve
pre-kdump callbacks. Add panic_pre_kdump_postpone to skip their early
invocation in crash core without consuming the once-only guard.

When enabled, the chain runs before kdump only when
crash_kexec_post_notifiers is also enabled. Leave the late panic call
unchanged so callbacks remain eligible if that path is reached.
Skipping the callbacks may prevent hypervisor crash handling.

Suggested-by: Guilherme G. Piccoli <gpiccoli@igalia.com>
Link: https://lore.kernel.org/r/bc0bdd14-5e6a-0676-cd04-14fb0bb93cd5@igalia.com
Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>
---
New in v2: optional postponement policy suggested by Guilherme.

 Documentation/admin-guide/kernel-parameters.txt | 14 ++++++++++++++
 include/linux/panic_notifier.h                  |  1 +
 kernel/crash_core.c                             |  3 ++-
 kernel/panic.c                                  |  2 ++
 4 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 33cd30996e47..3d722df30838 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -4904,6 +4904,20 @@ Kernel parameters
 			This option should only be used for systems with the above
 			constraints as it might cause the panic operation to be less reliable.
 
+	panic_pre_kdump_postpone=
+			[KNL] Defer pre-kdump notifiers to the late panic path.
+			They run before kdump only when
+			crash_kexec_post_notifiers is also enabled. If panic
+			reaches the late path without entering kdump, they
+			can still run there.
+			Intended for debugging kexec/crash-dump failures.
+			Skipping these callbacks may prevent correct crash
+			handling by the hypervisor.
+			Format: <bool>
+			Default: 0
+			Writable at runtime via
+			/sys/module/kernel/parameters/panic_pre_kdump_postpone.
+
 	panic_print=	Bitmask for printing system info when panic happens.
 			User can chose combination of the following bits:
 
diff --git a/include/linux/panic_notifier.h b/include/linux/panic_notifier.h
index 3ad390764a41..655ee5c77181 100644
--- a/include/linux/panic_notifier.h
+++ b/include/linux/panic_notifier.h
@@ -11,5 +11,6 @@ extern struct atomic_notifier_head panic_pre_kdump_list;
 void panic_notify_pre_kdump(char *msg);
 
 extern bool crash_kexec_post_notifiers;
+extern bool panic_pre_kdump_postpone;
 
 #endif	/* _LINUX_PANIC_NOTIFIERS_H */
diff --git a/kernel/crash_core.c b/kernel/crash_core.c
index 5d9fe9e00f39..d6b2293dd0fe 100644
--- a/kernel/crash_core.c
+++ b/kernel/crash_core.c
@@ -140,7 +140,8 @@ void __noclone __crash_kexec(struct pt_regs *regs)
 			struct pt_regs fixed_regs;
 
 			crash_setup_regs(&fixed_regs, regs);
-			panic_notify_pre_kdump(NULL);
+			if (!READ_ONCE(panic_pre_kdump_postpone))
+				panic_notify_pre_kdump(NULL);
 			crash_save_vmcoreinfo();
 			machine_crash_shutdown(&fixed_regs);
 			crash_cma_clear_pending_dma();
diff --git a/kernel/panic.c b/kernel/panic.c
index d78e90abb192..65d86594e038 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -62,6 +62,7 @@ static int pause_on_oops;
 static int pause_on_oops_flag;
 static DEFINE_SPINLOCK(pause_on_oops_lock);
 bool crash_kexec_post_notifiers;
+bool panic_pre_kdump_postpone;
 int panic_on_warn __read_mostly;
 unsigned long panic_on_taint;
 bool panic_on_taint_nousertaint = false;
@@ -1243,6 +1244,7 @@ core_param(panic, panic_timeout, int, 0644);
 core_param(pause_on_oops, pause_on_oops, int, 0644);
 core_param(panic_on_warn, panic_on_warn, int, 0644);
 core_param(crash_kexec_post_notifiers, crash_kexec_post_notifiers, bool, 0644);
+core_param(panic_pre_kdump_postpone, panic_pre_kdump_postpone, bool, 0644);
 core_param(panic_console_replay, panic_console_replay, bool, 0644);
 
 static int panic_print_set(const char *val, const struct kernel_param *kp)

[PATCH v2 4/6] x86/vmware: Add a bounded pre-kdump log sender

[Zack Rusin]

Preserve the newest kernel log tail in the host's vmware.log, which can
outlive a guest with no working userspace or persistent storage.

Allocate an 8 KiB buffer during early init and register on
panic_pre_kdump_list. A 4 KiB buffer can omit useful stack-trace
context. Use a fixed byte budget so transfer work does not scale
with PAGE_SIZE.

Copy the newest records, prefix them with "log ", and send one RPC over
the low-bandwidth register interface. Bound the transfer to 8 KiB and
at most three checkpoint attempts; zero-fill a partial payload word and
close the channel after every successful open.

The callback allocates nothing and emits no printk messages. Enable it
for ordinary guests, leaving encrypted guests disabled until explicitly
enabled by the sysctl. Skip initialization without printk support, and
leave logging disabled after allocation or registration failure.

Suggested-by: Michael Kelley <mhklinux@outlook.com>
Link: https://lore.kernel.org/r/SN6PR02MB41574A11D309A0530C63E0FED4842@SN6PR02MB4157.namprd02.prod.outlook.com
Link: https://lore.kernel.org/r/20260309235250.2611115-3-alexey.makhalov@broadcom.com
Co-developed-by: Bo Gan <bo.gan@broadcom.com>
Signed-off-by: Bo Gan <bo.gan@broadcom.com>
Co-developed-by: Alexey Makhalov <alexey.makhalov@broadcom.com>
Signed-off-by: Alexey Makhalov <alexey.makhalov@broadcom.com>
Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>
---
v2: use the pre-kdump notifier, establish safe guest defaults, and
increase the fixed buffer to 8 KiB as Michael suggested.
Maaz's v1 Reviewed-by is omitted for renewed review of the new callback.

 arch/x86/include/asm/vmware.h |   1 +
 arch/x86/kernel/cpu/vmware.c  | 173 ++++++++++++++++++++++++++++++++++
 2 files changed, 174 insertions(+)

diff --git a/arch/x86/include/asm/vmware.h b/arch/x86/include/asm/vmware.h
index 4220dae14a2d..598d4cd448ec 100644
--- a/arch/x86/include/asm/vmware.h
+++ b/arch/x86/include/asm/vmware.h
@@ -57,6 +57,7 @@
 #define VMWARE_HYPERVISOR_MAGIC		0x564d5868U
 
 #define VMWARE_CMD_GETVERSION		10
+#define VMWARE_CMD_MESSAGE		30
 #define VMWARE_CMD_GETHZ		45
 #define VMWARE_CMD_GETVCPU_INFO		68
 #define VMWARE_CMD_STEALCLOCK		91
diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
index 34b73573b108..ccf8dc84b30e 100644
--- a/arch/x86/kernel/cpu/vmware.c
+++ b/arch/x86/kernel/cpu/vmware.c
@@ -24,11 +24,16 @@
 #include <linux/dmi.h>
 #include <linux/init.h>
 #include <linux/export.h>
+#include <linux/kmsg_dump.h>
+#include <linux/mm.h>
+#include <linux/panic_notifier.h>
 #include <linux/clocksource.h>
 #include <linux/cpu.h>
 #include <linux/efi.h>
 #include <linux/reboot.h>
+#include <linux/sizes.h>
 #include <linux/static_call.h>
+#include <linux/wordpart.h>
 #include <linux/sched/cputime.h>
 #include <asm/div64.h>
 #include <asm/x86_init.h>
@@ -52,6 +57,23 @@
 #define STEALCLOCK_DISABLED        0
 #define STEALCLOCK_ENABLED         1
 
+#define VMWARE_MSG_STATUS_SUCCESS	BIT(16)
+#define VMWARE_MSG_STATUS_CPT		BIT(20)
+
+#define VMWARE_RPCI_PROTOCOL		0x49435052
+#define VMWARE_GUESTMSG_COOKIE		BIT(31)
+
+#define VMWARE_MSG_TYPE(n)		((n) << 16)
+#define VMWARE_MSG_OPEN			VMWARE_MSG_TYPE(0U)
+#define VMWARE_MSG_SENDSIZE		VMWARE_MSG_TYPE(1U)
+#define VMWARE_MSG_SENDPAYLOAD		VMWARE_MSG_TYPE(2U)
+#define VMWARE_MSG_CLOSE		VMWARE_MSG_TYPE(6U)
+
+#define VMWARE_LOG_BUF_SIZE		SZ_8K
+#define VMWARE_LOG_ATTEMPTS		3
+#define VMWARE_LOG_PREFIX		"log "
+#define VMWARE_LOG_PREFIX_LEN		(sizeof(VMWARE_LOG_PREFIX) - 1)
+
 struct vmware_steal_time {
 	union {
 		u64 clock;	/* stolen time counter in units of vtsc */
@@ -64,6 +86,12 @@ struct vmware_steal_time {
 	u64 reserved[7];
 };
 
+struct vmware_rpc_channel {
+	u16 id;
+	u32 cookie_high;
+	u32 cookie_low;
+};
+
 static unsigned long vmware_tsc_khz __ro_after_init;
 static u8 vmware_hypercall_mode     __ro_after_init;
 
@@ -142,6 +170,151 @@ static unsigned long vmware_get_tsc_khz(void)
 	return vmware_tsc_khz;
 }
 
+static int vmware_log_open(struct vmware_rpc_channel *channel)
+{
+	u32 info, id;
+
+	vmware_hypercall6(VMWARE_CMD_MESSAGE | VMWARE_MSG_OPEN,
+			  VMWARE_RPCI_PROTOCOL | VMWARE_GUESTMSG_COOKIE, 0,
+			  &info, &id, &channel->cookie_high,
+			  &channel->cookie_low);
+	if (!(info & VMWARE_MSG_STATUS_SUCCESS))
+		return -EIO;
+
+	channel->id = upper_16_bits(id);
+	return 0;
+}
+
+static int vmware_log_close(const struct vmware_rpc_channel *channel)
+{
+	u32 info;
+
+	vmware_hypercall5(VMWARE_CMD_MESSAGE | VMWARE_MSG_CLOSE, 0,
+			  (u32)channel->id << 16, channel->cookie_high,
+			  channel->cookie_low, &info);
+
+	return info & VMWARE_MSG_STATUS_SUCCESS ? 0 : -EIO;
+}
+
+static int vmware_log_send_once(const struct vmware_rpc_channel *channel,
+				const char *buffer, size_t length)
+{
+	u32 info;
+
+	vmware_hypercall5(VMWARE_CMD_MESSAGE | VMWARE_MSG_SENDSIZE, length,
+			  (u32)channel->id << 16, channel->cookie_high,
+			  channel->cookie_low, &info);
+	if (!(info & VMWARE_MSG_STATUS_SUCCESS))
+		return info & VMWARE_MSG_STATUS_CPT ? -EAGAIN : -EIO;
+
+	while (length) {
+		size_t bytes = min_t(size_t, length, sizeof(u32));
+		u32 word = 0;
+
+		memcpy(&word, buffer, bytes);
+		vmware_hypercall5(VMWARE_CMD_MESSAGE | VMWARE_MSG_SENDPAYLOAD,
+				  word, (u32)channel->id << 16,
+				  channel->cookie_high, channel->cookie_low,
+				  &info);
+		if (!(info & VMWARE_MSG_STATUS_SUCCESS))
+			return info & VMWARE_MSG_STATUS_CPT ? -EAGAIN : -EIO;
+
+		buffer += bytes;
+		length -= bytes;
+	}
+
+	return 0;
+}
+
+static int vmware_log_send(const struct vmware_rpc_channel *channel,
+			   const char *buffer, size_t length)
+{
+	int attempt, ret;
+
+	for (attempt = 0; attempt < VMWARE_LOG_ATTEMPTS; attempt++) {
+		ret = vmware_log_send_once(channel, buffer, length);
+		if (ret != -EAGAIN)
+			return ret;
+	}
+
+	return -EAGAIN;
+}
+
+static int vmware_log_rpc(const char *buffer, size_t length)
+{
+	struct vmware_rpc_channel channel;
+	int close_ret, ret;
+
+	ret = vmware_log_open(&channel);
+	if (ret)
+		return ret;
+
+	ret = vmware_log_send(&channel, buffer, length);
+	close_ret = vmware_log_close(&channel);
+	if (!ret && close_ret)
+		ret = close_ret;
+
+	return ret;
+}
+
+static struct page *vmware_panic_page;
+static bool vmware_record_panic_msg;
+
+static int vmware_panic_log_notify(struct notifier_block *nb,
+				   unsigned long action, void *data)
+{
+	struct kmsg_dump_iter iter;
+	char *buffer = page_address(vmware_panic_page);
+	size_t length = 0;
+
+	if (!READ_ONCE(vmware_record_panic_msg))
+		return NOTIFY_DONE;
+
+	memcpy(buffer, VMWARE_LOG_PREFIX, VMWARE_LOG_PREFIX_LEN);
+	kmsg_dump_rewind(&iter);
+	(void)kmsg_dump_get_buffer(&iter, true,
+				   buffer + VMWARE_LOG_PREFIX_LEN,
+				   VMWARE_LOG_BUF_SIZE - VMWARE_LOG_PREFIX_LEN,
+				   &length);
+	(void)vmware_log_rpc(buffer, length + VMWARE_LOG_PREFIX_LEN);
+
+	return NOTIFY_DONE;
+}
+
+static struct notifier_block vmware_panic_log_nb = {
+	.notifier_call = vmware_panic_log_notify,
+};
+
+static int __init vmware_panic_log_init(void)
+{
+	int ret;
+
+	if (!IS_ENABLED(CONFIG_PRINTK) ||
+	    !hypervisor_is_type(X86_HYPER_VMWARE))
+		return 0;
+
+	vmware_record_panic_msg =
+		!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT);
+
+	vmware_panic_page = alloc_pages(GFP_KERNEL,
+					get_order(VMWARE_LOG_BUF_SIZE));
+	if (!vmware_panic_page) {
+		pr_err("failed to allocate panic log buffer\n");
+		return 0;
+	}
+
+	ret = atomic_notifier_chain_register(&panic_pre_kdump_list,
+					     &vmware_panic_log_nb);
+	if (ret) {
+		pr_err("failed to register panic log notifier: %d\n", ret);
+		__free_pages(vmware_panic_page, get_order(VMWARE_LOG_BUF_SIZE));
+		vmware_panic_page = NULL;
+	}
+
+	return 0;
+}
+early_initcall(vmware_panic_log_init);
+
 #ifdef CONFIG_PARAVIRT
 static struct cyc2ns_data vmware_cyc2ns __ro_after_init;
 static bool vmw_sched_clock __initdata = true;

[PATCH v2 5/6] x86/vmware: Add the vmware_record_panic_msg sysctl

[Zack Rusin]

Allow administrators to disable host log transfer on ordinary guests or
opt in on encrypted guests. Expose the boolean recording policy through
kernel.vmware_record_panic_msg using proc_dobool(), as Joel suggested.
Zero disables recording, nonzero integers enable it, and reads return
0 or 1.

Register the sysctl only after the logger is ready. A registration failure
leaves the internal default in force. Document that disabling the old
post-notifier ordering does not disable this independent logger.

Link: https://lore.kernel.org/r/4e73yd2ofpdzl6rptzvd74no3hnyfblknoq7wzxlw7f3zjbz7c@srfabx6lsusj
Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>
---
v2: use bool/proc_dobool as Joel requested; document direct crash entry.
Maaz's v1 Reviewed-by is omitted for renewed review of the changed sysctl.

 Documentation/admin-guide/sysctl/kernel.rst | 20 ++++++++++++++++++++
 arch/x86/kernel/cpu/vmware.c                | 15 +++++++++++++++
 2 files changed, 35 insertions(+)

diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index ffea61d448eb..be13c143b8bb 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -1690,6 +1690,26 @@ entry will default to 2 instead of 0.
 = =============================================================
 
 
+vmware_record_panic_msg
+======================
+
+Controls whether panic or fatal-oops kmsg data is written to the host's
+``vmware.log`` before kdump. This setting does not control the separate
+VMware guest-crash event. Writing zero disables recording; writing a
+nonzero integer enables it. Reads return 0 or 1.
+
+= ==============================================================
+0 Do not write kmsg data to ``vmware.log``. This is the default
+  for encrypted guests.
+1 Write kmsg data to ``vmware.log``. This is the default for
+  ordinary guests.
+= ==============================================================
+
+``crash_kexec_post_notifiers=0`` alone does not disable this logger.
+``panic_pre_kdump_postpone=1`` skips early logging but leaves late panic
+logging eligible; set this sysctl to 0 to disable both.
+
+
 warn_limit
 ==========
 
diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
index ccf8dc84b30e..4b0e5b084c2c 100644
--- a/arch/x86/kernel/cpu/vmware.c
+++ b/arch/x86/kernel/cpu/vmware.c
@@ -33,6 +33,7 @@
 #include <linux/reboot.h>
 #include <linux/sizes.h>
 #include <linux/static_call.h>
+#include <linux/sysctl.h>
 #include <linux/wordpart.h>
 #include <linux/sched/cputime.h>
 #include <asm/div64.h>
@@ -260,6 +261,16 @@ static int vmware_log_rpc(const char *buffer, size_t length)
 static struct page *vmware_panic_page;
 static bool vmware_record_panic_msg;
 
+static const struct ctl_table vmware_panic_sysctls[] = {
+	{
+		.procname	= "vmware_record_panic_msg",
+		.data		= &vmware_record_panic_msg,
+		.maxlen		= sizeof(vmware_record_panic_msg),
+		.mode		= 0644,
+		.proc_handler	= proc_dobool,
+	},
+};
+
 static int vmware_panic_log_notify(struct notifier_block *nb,
 				   unsigned long action, void *data)
 {
@@ -311,6 +322,10 @@ static int __init vmware_panic_log_init(void)
 		vmware_panic_page = NULL;
 	}
 
+	if (vmware_panic_page && IS_ENABLED(CONFIG_SYSCTL) &&
+	    !register_sysctl("kernel", vmware_panic_sysctls))
+		pr_err("failed to register panic log sysctl\n");
+
 	return 0;
 }
 early_initcall(vmware_panic_log_init);

[PATCH v2 6/6] x86/vmware: Report guest crashes after kmsg dumpers

[Zack Rusin]

VMware accepts a structured guest-crash event, but a host configured with
backdoor.coredumpOnCrash may terminate the VM before the hypercall
returns. Keep this event separate from the pre-kdump log sender.

Register a panic-only kmsg dumper with late_initcall_sync(), after built-in
dumpers registered by then. Skip the event while a crash kernel is loaded
so the host cannot terminate the guest before kdump saves its vmcore.

Issue the event at most once. With CONFIG_PRINTK=n, use a late INT_MIN
panic notifier. An unexpected dumper registration failure with printk
enabled leaves reporting disabled rather than overtaking working dumpers.

Link: https://lore.kernel.org/r/20260309235250.2611115-4-alexey.makhalov@broadcom.com
Co-developed-by: Brennan Lamoreaux <brennan.lamoreaux@broadcom.com>
Signed-off-by: Brennan Lamoreaux <brennan.lamoreaux@broadcom.com>
Co-developed-by: Alexey Makhalov <alexey.makhalov@broadcom.com>
Signed-off-by: Alexey Makhalov <alexey.makhalov@broadcom.com>
Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>
---
v2: keep the terminating event separate from the pre-kdump logger.
Maaz's v1 Reviewed-by is omitted for renewed review of the new ordering.

 arch/x86/include/asm/vmware.h |  1 +
 arch/x86/kernel/cpu/vmware.c  | 62 +++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+)

diff --git a/arch/x86/include/asm/vmware.h b/arch/x86/include/asm/vmware.h
index 598d4cd448ec..ea2ce5272a00 100644
--- a/arch/x86/include/asm/vmware.h
+++ b/arch/x86/include/asm/vmware.h
@@ -61,6 +61,7 @@
 #define VMWARE_CMD_GETHZ		45
 #define VMWARE_CMD_GETVCPU_INFO		68
 #define VMWARE_CMD_STEALCLOCK		91
+#define VMWARE_CMD_REPORTGUESTCRASH	102
 /*
  * Hypercall command mask:
  *   bits [6:0] command, range [0, 127]
diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
index 4b0e5b084c2c..70553da771c9 100644
--- a/arch/x86/kernel/cpu/vmware.c
+++ b/arch/x86/kernel/cpu/vmware.c
@@ -22,9 +22,12 @@
  */
 
 #include <linux/dmi.h>
+#include <linux/atomic.h>
+#include <linux/crash_core.h>
 #include <linux/init.h>
 #include <linux/export.h>
 #include <linux/kmsg_dump.h>
+#include <linux/limits.h>
 #include <linux/mm.h>
 #include <linux/panic_notifier.h>
 #include <linux/clocksource.h>
@@ -330,6 +333,65 @@ static int __init vmware_panic_log_init(void)
 }
 early_initcall(vmware_panic_log_init);
 
+static atomic_t vmware_crash_reported = ATOMIC_INIT(0);
+
+static void vmware_report_guest_crash(void)
+{
+	if (kexec_crash_loaded())
+		return;
+	if (atomic_xchg(&vmware_crash_reported, 1))
+		return;
+
+	vmware_hypercall1(VMWARE_CMD_REPORTGUESTCRASH, 0);
+}
+
+static void vmware_crash_report_dump(struct kmsg_dumper *dumper,
+				     struct kmsg_dump_detail *detail)
+{
+	vmware_report_guest_crash();
+}
+
+static struct kmsg_dumper vmware_crash_report_dumper = {
+	.dump = vmware_crash_report_dump,
+	.max_reason = KMSG_DUMP_PANIC,
+};
+
+static int vmware_crash_report_notify(struct notifier_block *nb,
+				      unsigned long action, void *data)
+{
+	vmware_report_guest_crash();
+	return NOTIFY_DONE;
+}
+
+static struct notifier_block vmware_crash_report_nb = {
+	.notifier_call = vmware_crash_report_notify,
+	.priority = INT_MIN,
+};
+
+static int __init vmware_crash_report_init(void)
+{
+	int ret;
+
+	if (!hypervisor_is_type(X86_HYPER_VMWARE))
+		return 0;
+
+	ret = kmsg_dump_register(&vmware_crash_report_dumper);
+	if (!ret)
+		return 0;
+	if (IS_ENABLED(CONFIG_PRINTK)) {
+		pr_err("failed to register crash report dumper: %d\n", ret);
+		return 0;
+	}
+
+	ret = atomic_notifier_chain_register(&panic_notifier_list,
+					     &vmware_crash_report_nb);
+	if (ret)
+		pr_err("failed to register crash report notifier: %d\n", ret);
+
+	return 0;
+}
+late_initcall_sync(vmware_crash_report_init);
+
 #ifdef CONFIG_PARAVIRT
 static struct cyc2ns_data vmware_cyc2ns __ro_after_init;
 static bool vmw_sched_clock __initdata = true;

Re: [PATCH v2 6/6] x86/vmware: Report guest crashes after kmsg dumpers

[sashiko-bot]

> VMware accepts a structured guest-crash event, but a host configured with
> backdoor.coredumpOnCrash may terminate the VM before the hypercall
> returns. Keep this event separate from the pre-kdump log sender.
> 
> Register a panic-only kmsg dumper with late_initcall_sync(), after built-in
> dumpers registered by then. Skip the event while a crash kernel is loaded
> so the host cannot terminate the guest before kdump saves its vmcore.
> 
> Issue the event at most once. With CONFIG_PRINTK=n, use a late INT_MIN
> panic notifier. An unexpected dumper registration failure with printk
> enabled leaves reporting disabled rather than overtaking working dumpers.
> 
> Link: https://lore.kernel.org/r/20260309235250.2611115-4-alexey.makhalov@broadcom.com
> Co-developed-by: Brennan Lamoreaux <brennan.lamoreaux@broadcom.com>
> Signed-off-by: Brennan Lamoreaux <brennan.lamoreaux@broadcom.com>
> Co-developed-by: Alexey Makhalov <alexey.makhalov@broadcom.com>
> Signed-off-by: Alexey Makhalov <alexey.makhalov@broadcom.com>
> Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1790014793.git.zack.rusin@broadcom.com?part=6

Re: [PATCH v2 1/6] panic: Add a notifier chain for pre-kdump callbacks

[sashiko-bot]

> Selected crash callbacks should not require running every panic notifier
> before kdump. Add a separate atomic chain with an at-most-once dispatch
> helper, and invoke it before panic reaches the kmsg dumpers.
> 
> The helper permits another call from the crash path without repeating
> callbacks or recursing into them. Document that callbacks may run in NMI
> context, with other CPUs running and the kexec lock held.
> 
> Link: https://lore.kernel.org/r/aquttpMVKX8e6zGB@pathway.suse.cz
> Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1790014793.git.zack.rusin@broadcom.com?part=1

Re: [PATCH v2 3/6] panic: Allow postponing pre-kdump notifiers

[sashiko-bot]

> Provide an escape hatch for debugging kdump failures that may involve
> pre-kdump callbacks. Add panic_pre_kdump_postpone to skip their early
> invocation in crash core without consuming the once-only guard.
> 
> When enabled, the chain runs before kdump only when
> crash_kexec_post_notifiers is also enabled. Leave the late panic call
> unchanged so callbacks remain eligible if that path is reached.
> Skipping the callbacks may prevent hypervisor crash handling.
> 
> Suggested-by: Guilherme G. Piccoli <gpiccoli@igalia.com>
> Link: https://lore.kernel.org/r/bc0bdd14-5e6a-0676-cd04-14fb0bb93cd5@igalia.com
> Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1790014793.git.zack.rusin@broadcom.com?part=3

Re: [PATCH v2 5/6] x86/vmware: Add the vmware_record_panic_msg sysctl

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] doc: rST title underline too short for vmware_record_panic_msg
--

commit c541f42ddd949c322e4b6facb84fc17737ed184a
Author: Zack Rusin <zack.rusin@broadcom.com>

x86/vmware: Add the vmware_record_panic_msg sysctl

This commit adds a new sysctl kernel.vmware_record_panic_msg to control
whether panic or fatal-oops data is written to the host log before kdump.
It allows administrators to configure this logging behavior depending on
guest encryption, and defers sysctl registration until the logger is ready.

Link: https://lore.kernel.org/r/4e73yd2ofpdzl6rptzvd74no3hnyfblknoq7wzxlw7f3zjbz7c@srfabx6lsusj

> diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
> index ffea61d448ebb..be13c143b8bb0 100644
> --- a/Documentation/admin-guide/sysctl/kernel.rst
> +++ b/Documentation/admin-guide/sysctl/kernel.rst

[ ... ]

> @@ -1690,6 +1690,26 @@ entry will default to 2 instead of 0.
>  = =============================================================
>  
>  
> +vmware_record_panic_msg
> +======================

[Severity: Low]
Is the underline for this section title one character too short?

The title vmware_record_panic_msg is 23 characters long, but the underline
is only 22 characters long. In reStructuredText, the underline must be at
least as long as the text it underlines, so this regression will trigger a
warning during Sphinx documentation builds.

> +
> +Controls whether panic or fatal-oops kmsg data is written to the host's
> +``vmware.log`` before kdump. This setting does not control the separate

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1790014793.git.zack.rusin@broadcom.com?part=5

Re: [PATCH v2 4/6] x86/vmware: Add a bounded pre-kdump log sender

[sashiko-bot]

> Preserve the newest kernel log tail in the host's vmware.log, which can
> outlive a guest with no working userspace or persistent storage.
> 
> Allocate an 8 KiB buffer during early init and register on
> panic_pre_kdump_list. A 4 KiB buffer can omit useful stack-trace
> context. Use a fixed byte budget so transfer work does not scale
> with PAGE_SIZE.
> 
> Copy the newest records, prefix them with "log ", and send one RPC over
> the low-bandwidth register interface. Bound the transfer to 8 KiB and
> at most three checkpoint attempts; zero-fill a partial payload word and
> close the channel after every successful open.
> 
> The callback allocates nothing and emits no printk messages. Enable it
> for ordinary guests, leaving encrypted guests disabled until explicitly
> enabled by the sysctl. Skip initialization without printk support, and
> leave logging disabled after allocation or registration failure.
> 
> Suggested-by: Michael Kelley <mhklinux@outlook.com>
> Link: https://lore.kernel.org/r/SN6PR02MB41574A11D309A0530C63E0FED4842@SN6PR02MB4157.namprd02.prod.outlook.com
> Link: https://lore.kernel.org/r/20260309235250.2611115-3-alexey.makhalov@broadcom.com
> Co-developed-by: Bo Gan <bo.gan@broadcom.com>
> Signed-off-by: Bo Gan <bo.gan@broadcom.com>
> Co-developed-by: Alexey Makhalov <alexey.makhalov@broadcom.com>
> Signed-off-by: Alexey Makhalov <alexey.makhalov@broadcom.com>
> Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1790014793.git.zack.rusin@broadcom.com?part=4

Re: [PATCH v2 2/6] crash: Notify pre-kdump callbacks before switching kernels

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] panic: Pre-kdump callbacks always receive NULL message in default config
--

commit 8884313d127ae4b9734baebea7b4ba8dd25b38a5
Author: Zack Rusin <zack.rusin@broadcom.com>

crash: Notify pre-kdump callbacks before switching kernels

Fatal x86 oopses can call crash_kexec() without reaching panic(). Run the
pre-kdump chain from __crash_kexec() as well, after finding a loaded image
under the kexec lock and capturing the original registers.

Link: https://lore.kernel.org/r/aquttpMVKX8e6zGB@pathway.suse.cz

>     the shared guard skips callbacks already invoked by panic().

[Severity: Medium]
Is this statement accurate?

During a standard panic-triggered kdump with crash_kexec_post_notifiers=0
(the default configuration), vpanic() calls __crash_kexec(NULL) before it
formats the panic message and calls its own explicit
panic_notify_pre_kdump(buf).

Because __crash_kexec(NULL) runs first, won't the new
panic_notify_pre_kdump(NULL) call consume the once-per-boot atomic guard,
rather than being skipped by it?

> diff --git a/kernel/crash_core.c b/kernel/crash_core.c
> index 2b36aa9fade06..5d9fe9e00f39f 100644
> --- a/kernel/crash_core.c
> +++ b/kernel/crash_core.c
[ ... ]
> @@ -139,6 +140,7 @@ void __noclone __crash_kexec(struct pt_regs *regs)
>  			struct pt_regs fixed_regs;
>  
>  			crash_setup_regs(&fixed_regs, regs);
> +			panic_notify_pre_kdump(NULL);

[Severity: Medium]
Will this cause pre-kdump notifiers to receive a NULL message pointer instead
of the expected panic message?

If a system panic occurs and kexec is configured with the default setting of
crash_kexec_post_notifiers=0, vpanic() triggers __crash_kexec(NULL) before
reaching its own explicit panic_notify_pre_kdump(buf):

kernel/panic.c:vpanic() {
    ...
    if (!_crash_kexec_post_notifiers)
        __crash_kexec(NULL);
    ...
    panic_notify_pre_kdump(buf);
}

When this happens, __crash_kexec() will invoke panic_notify_pre_kdump() with
a NULL message pointer. This permanently consumes the once-per-boot guard,
and any callback relying on the documented msg parameter will experience
data loss or potential NULL dereferences, violating the API contract that
promises the panic message during a panic.

>  			crash_save_vmcoreinfo();
>  			machine_crash_shutdown(&fixed_regs);
>  			crash_cma_clear_pending_dma();

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1790014793.git.zack.rusin@broadcom.com?part=2

Re: [PATCH v2 2/6] crash: Notify pre-kdump callbacks before switching kernels

[Zack Rusin]

[-- Attachment #1: Type: text/plain, Size: 3589 bytes --]

On Mon, Sep 28, 2026 at 2:09 PM <sashiko-bot@kernel.org> wrote:
>
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [Medium] panic: Pre-kdump callbacks always receive NULL message in default config

fwiw, this report seems invalid to me.

> commit 8884313d127ae4b9734baebea7b4ba8dd25b38a5
> Author: Zack Rusin <zack.rusin@broadcom.com>
>
> crash: Notify pre-kdump callbacks before switching kernels
>
> Fatal x86 oopses can call crash_kexec() without reaching panic(). Run the
> pre-kdump chain from __crash_kexec() as well, after finding a loaded image
> under the kexec lock and capturing the original registers.
>
> Link: https://lore.kernel.org/r/aquttpMVKX8e6zGB@pathway.suse.cz
>
> >     the shared guard skips callbacks already invoked by panic().
>
> [Severity: Medium]
> Is this statement accurate?
>
> During a standard panic-triggered kdump with crash_kexec_post_notifiers=0
> (the default configuration), vpanic() calls __crash_kexec(NULL) before it
> formats the panic message and calls its own explicit
> panic_notify_pre_kdump(buf).
>
> Because __crash_kexec(NULL) runs first, won't the new
> panic_notify_pre_kdump(NULL) call consume the once-per-boot atomic guard,
> rather than being skipped by it?

Yes, and that's intended. By default the crash-core call runs first
and the late call in vpanic() is skipped. With
crash_kexec_post_notifiers=1 the order is reversed. vpanic() runs the
list with buf, and the crash-core call is skipped. The changelog
sentence only describes the second case.

> > diff --git a/kernel/crash_core.c b/kernel/crash_core.c
> > index 2b36aa9fade06..5d9fe9e00f39f 100644
> > --- a/kernel/crash_core.c
> > +++ b/kernel/crash_core.c
> [ ... ]
> > @@ -139,6 +140,7 @@ void __noclone __crash_kexec(struct pt_regs *regs)
> >                       struct pt_regs fixed_regs;
> >
> >                       crash_setup_regs(&fixed_regs, regs);
> > +                     panic_notify_pre_kdump(NULL);
>
> [Severity: Medium]
> Will this cause pre-kdump notifiers to receive a NULL message pointer instead
> of the expected panic message?
>
> If a system panic occurs and kexec is configured with the default setting of
> crash_kexec_post_notifiers=0, vpanic() triggers __crash_kexec(NULL) before
> reaching its own explicit panic_notify_pre_kdump(buf):
>
> kernel/panic.c:vpanic() {
>     ...
>     if (!_crash_kexec_post_notifiers)
>         __crash_kexec(NULL);
>     ...
>     panic_notify_pre_kdump(buf);
> }
>
> When this happens, __crash_kexec() will invoke panic_notify_pre_kdump() with
> a NULL message pointer. This permanently consumes the once-per-boot guard,
> and any callback relying on the documented msg parameter will experience
> data loss or potential NULL dereferences, violating the API contract that
> promises the panic message during a panic.

afaict Sashiko gets tripped up here by the doc from the previous
email. Nothing in this series dereferences the msg. The only callback
is the VMware logger in patch 4, which ignores the argument. It reads
the printk buffer, where vpanic() has already written the panic line.

If there's an interest in fixing this false-positive in code for
Sashiko (or if there are other issues that would make me respin v3)
I'd just change the doc added in the first change in this series like
this:
- * @msg: Panic message, or NULL for direct crash-kexec entry
+ * @msg: Optional panic message; callbacks must tolerate NULL
which, I think, would fix Sashiko here.

z

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5414 bytes --]

Re: [PATCH v2 0/6] panic, x86/vmware: Preserve crash logs before kdump

[Bradley Morgan]

On 28 September 2026 18:55:34 BST, Zack Rusin <zack.rusin@broadcom.com>
wrote:
>A VMware guest may have no persistent storage or working userspace after
>a crash. Preserve its newest kernel log tail in the host's vmware.log
>before entering kdump, without enabling every panic notifier by default.
>
>Following Petr's suggestion, add one generic panic_pre_kdump_list with
>VMware as its first client. A set-once helper covers panic() and direct
>crash-kexec entry. Fatal oopses can reach kdump without calling panic(),
>so patch 2 adds that call after capturing the original registers under
>the existing kexec lock. The late panic call follows sys_info() and
>precedes the kmsg dumpers.
>
>Patch 3 adds Guilherme's suggested escape hatch for debugging kdump
>failures involving early callbacks. With panic_pre_kdump_postpone set,
>the list runs before kdump only when crash_kexec_post_notifiers is also
>set. This can omit hypervisor crash handling as well as diagnostics.
>The late panic call remains eligible. This policy is a separate patch;
>I can omit patch 3 if it is not wanted.
>
>The logger preallocates an 8 KiB buffer and bounds its register-based
>RPC transfer and retries. Michael suggested increasing the buffer
>because 4 KiB can omit useful stack-trace context. Ordinary guests
>enable logging by default; encrypted guests must opt in. Use
>bool/proc_dobool for the sysctl as Joel requested. The potentially
>terminating REPORTGUESTCRASH event remains a separate late dumper,
>suppressed whenever a crash image is loaded. The v1 assignment to
>crash_kexec_post_notifiers is dropped.
>
>Tested on VMware Workstation and ESXi with ordinary guests, and on
>ESXi with SEV-SNP and TDX guests.
>
>v1:
>https://lore.kernel.org/r/cover.1788414671.git.zack.rusin@broadcom.com
>
>Zack Rusin (6):
>  panic: Add a notifier chain for pre-kdump callbacks
>  crash: Notify pre-kdump callbacks before switching kernels
>  panic: Allow postponing pre-kdump notifiers
>  x86/vmware: Add a bounded pre-kdump log sender
>  x86/vmware: Add the vmware_record_panic_msg sysctl
>  x86/vmware: Report guest crashes after kmsg dumpers
>
> .../admin-guide/kernel-parameters.txt         |  14 +
> Documentation/admin-guide/sysctl/kernel.rst   |  20 ++
> arch/x86/include/asm/vmware.h                 |   2 +
> arch/x86/kernel/cpu/vmware.c                  | 250 ++++++++++++++++++
> include/linux/panic_notifier.h                |   4 +
> kernel/crash_core.c                           |   3 +
> kernel/panic.c                                |  28 ++
> 7 files changed, 321 insertions(+)
>

All the panic changes and crash_core changes looks good to me!

Reviewed-by: Bradley Morgan <brads@mainlining.org>

Tested basic panic calls with Power10, in courtesy of osuosl:

Tested-by: Bradley Morgan <brads@mainlining.org> # Power10
 


>
>base-commit: fd73f4a6659897191fa0d40695fe370925dd3780
>
>
>From mboxrd@z Thu Jan  1 00:00:00 1970
>Received: from mail-pj1-f100.google.com (mail-pj1-f100.google.com [209.85.216.100])
>	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
>	(No client certificate requested)
>	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83AF64E3243
>	for <linux-doc@vger.kernel.org>; Mon, 28 Sep 2026 17:56:08 +0000 (UTC)
>Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.100
>ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
>	t=1790618169; cv=none; b=b4BojE7kyzKnMqCq3yQBsgTjfamCuHeSPBiM3mTeE4v6X5jd8GsgUy8HFjfb6vnu2FsurVAocpFTMaJi8sVvzC7bBtKNDJpFDvZCNqdO4XHY7wznbp2hC1z0LbTBYGMbRHtCk4LvgNFrpWDHzU9DrGZf+1d8PQJtrrUWMR3fWB8=
>ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org;
>	s=arc-20240116; t=1790618169; c=relaxed/simple;
>	bh=qHuvSTbzKTQ99mPQxhPgubTCtOl/pceEwTRUqIC9LiU=;
>	h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=FTXEgGo9/aZOjjBWf/3/aJE9bzE1CX9s/OSrb2hi9L/KTuxvuIv8F7B18Mj6GgYbZIaXobZhtwozeHdFf9pWY4NmLIXIxMlQiB06cvRkp4UYghm/h3KZ3r3B3EPB6W4Zm/qFIUpvsrOcMQBhLX0ulM1Nh7jAkRnR5+Dr4TccEps=
>ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass
>(p=reject dis=none) header.from=broadcom.com; spf=fail
>smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com
>header.i=@broadcom.com header.b=AMSyqTvg; arc=none
>smtp.client-ip=209.85.216.100
>Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=broadcom.com
>Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com
>Authentication-Results: smtp.subspace.kernel.org;
>	dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="AMSyqTvg"
>Received: by mail-pj1-f100.google.com with SMTP id 98e67ed59e1d1-39647aa9d52so87593a91.0
>        for <linux-doc@vger.kernel.org>; Mon, 28 Sep 2026 10:56:08 -0700 (PDT)
>X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
>        d=1e100.net; s=20260707; t=1790618168; x=1791222968;
>        h=content-transfer-encoding:mime-version:message-id:date:subject:cc
>         :to:from:dkim-signature:x-gm-gg:x-gm-message-state:from:to:cc
>         :subject:date:message-id:reply-to:content-type;
>        bh=KmmJxySPwG38CbqK+KUDBFApZhpWarF4JpRDp51PqIc=;
>        b=Jpd/MIj5ewElR34jOfbzVz6y0phWWOuuetH2AMAqSfkw9wo88gnxd8iiGSrwGXb/K8
>         LxiFGW7Gk1/qfXeNKTtiT3pDkujiBk8rYb2hZ9gxVSorRYSYgPAO2EL16XTvbxLgG6YK
>         3FeqkS0hVSU1tdYNfgmZnbEIc2b/RyU4JI14RYidz/e/0W5GCoeMbf3A5MEjESY87AVb
>         Nmwd7ohydW1mz6nApfHFyC6Dnyk8weLtbSVES1jsgb3+ivuTwzujcnvHLoxLcTveo38M
>         /QCQT4LKda0z3bTRNmSUIPrz9XLziXq3W4D3R+fZgS7Ae1bSJihAy/CQ8TsXISh1sea3
>         /Y5g==
>X-Forwarded-Encrypted: i=1; AKwUvBz1cco3gfH83tEf5iTZWgGV7lvOGVAz8sAyGJ4O8VTok46+5cEwqveCnVBIA8QHt65rA8D83p9YKyk=@vger.kernel.org
>X-Gm-Message-State: AFq9FYJJ9fgQVETjiY5fbRpzunCNQ3yHlKk8Ilzj81eoAlnuoS+Dn0Ka
>	McHcsKEBsXysl0ErYrd3UsPTH5PvX2dq9oZLqcmv4M1H2ZZYdezUd/B+B5ApLoXVYyVqwwaKkn1
>	WhxVmBuVCVdWdMiAn3Vi6H3YMgDwkYBQIrAbekUmKYvA8WvWr8eMC191fp1XiUxJ6rhpIfeSsVp
>	NTAsl3Yon3avlK5XQfKHI5gtnaNCTFriKugUOmHPEPXyu8F41SwzaOR5uWzC+zjBz4Vodbmgspr
>	ROsqbvKBTG0
>X-Gm-Gg: AYBFou2P8ZN7jxBSC+9O6FmDIlcBFoArokAtaD0KOTfPh/dbNfovO55NTXigG/qTx0i
>	zW8kFxJ9y8047OYcs7z6GR/iFk/FwAFBB5zgqFHnosJpMNBIX2v+4Kwpu9gjKkmULOJP4mIajIu
>	dJfq/+pAXglXwFTJ+9jxVi8wPktojt/Kb99zzSZdC2pJZ1qGv0VNdQR3aD5YFUJSBFcy3BkabjL
>	sk1IZGxFfeKEGoAO9jrL5qKeIm3f6Ql5otkpaV9x52hh1iOyQ0Xd1UFFrUB+dYPtR6W5bYD0hSO
>	5TbTBiupR7Fk/TU3eH1D3LCTwxJdUmWXGODfhDogaAZUtfuQraIYelR1Lue5OSP5fGKHMw5FPyo
>	iFEAWZ+eQ1MJmXkr4N/LcVUr5vNjYToRFimPtfJQ430PQig2taOVcTvxn7B2m7O6JMTg8qajAW3
>	a/HyZmBnzcwByWASw8uS+bX2LuCwxtV9OO
>X-Received: by 2002:a17:90b:28c6:b0:3a0:34a4:187e with SMTP id 98e67ed59e1d1-3a49c120f78mr7983a91.15.1790618167517;
>        Mon, 28 Sep 2026 10:56:07 -0700 (PDT)
>Received: from smtp-us-east1-p01-i01-si01.dlp.protect.broadcom.com (address-144-49-247-18.dlp.protect.broadcom.com. [144.49.247.18])
>        by smtp-relay.gmail.com with ESMTPS id 98e67ed59e1d1-3a498e35c74sm34226a91.3.2026.09.28.10.56.00
>        for <linux-doc@vger.kernel.org>
>        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
>        Mon, 28 Sep 2026 10:56:07 -0700 (PDT)
>X-Relaying-Domain: broadcom.com
>X-CFilter-Loop: Reflected
>Received: by mail-dy1-f200.google.com with SMTP id 5a478bee46e88-3282d5302ffso156936eec.1
>        for <linux-doc@vger.kernel.org>; Mon, 28 Sep 2026 10:56:00 -0700 (PDT)
>DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
>        d=broadcom.com; s=google; t=1790618159; x=1791222959; darn=vger.kernel.org;
>        h=content-transfer-encoding:mime-version:message-id:date:subject:cc
>         :to:from:from:to:cc:subject:date:message-id:reply-to:content-type;
>        bh=KmmJxySPwG38CbqK+KUDBFApZhpWarF4JpRDp51PqIc=;
>        b=AMSyqTvg7YOAaOo4sIAY0pcRYwHSW0cH09GjKbqzO4apiMMUre94O+4j1U12+P2RKc
>         thNFz4BDgQu1JWPb2Prmfj0x/3+gp1f5TNhwJdRBccsEH9Kg8d61iKzddQPkLXzvvA2Y
>         czPVjFz5/L4SBh9PoEC0hHpQAKBZzbU0jsBOo=
>X-Forwarded-Encrypted: i=1; AKwUvBzwJDvlo2QB15OMSgAG9uvXCCuZffODLbFe+C46K/WtKOc+COVpbcP72cSKjfHL1r07WxDFnjfUXvo=@vger.kernel.org
>X-Received: by 2002:a05:7022:ff49:b0:13c:e1e2:fc86 with SMTP id a92af1059eb24-14b2d993e47mr201594c88.4.1790618159182;
>        Mon, 28 Sep 2026 10:55:59 -0700 (PDT)
>X-Received: by 2002:a05:7022:ff49:b0:13c:e1e2:fc86 with SMTP id a92af1059eb24-14b2d993e47mr201547c88.4.1790618158548;
>        Mon, 28 Sep 2026 10:55:58 -0700 (PDT)
>Received: from vertex.localdomain ([192.19.144.250])
>        by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3421dbb2971sm26076069eec.10.2026.09.28.10.55.53
>        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
>        Mon, 28 Sep 2026 10:55:57 -0700 (PDT)
>From: Zack Rusin <zack.rusin@broadcom.com>
>To: Andrew Morton <akpm@linux-foundation.org>,
>	Petr Mladek <pmladek@suse.com>,
>	Baoquan He <baoquan.he@linux.dev>,
>	Mike Rapoport <rppt@kernel.org>,
>	Pasha Tatashin <pasha.tatashin@soleen.com>,
>	Pratyush Yadav <pratyush@kernel.org>,
>	Dave Young <ruirui.yang@linux.dev>
>Cc: Borislav Petkov <bp@alien8.de>,
>	Ajay Kaher <ajay.kaher@broadcom.com>,
>	Alexey Makhalov <alexey.makhalov@broadcom.com>,
>	x86@kernel.org,
>	Joel Granados <joel.granados@kernel.org>,
>	Thomas Gleixner <tglx@kernel.org>,
>	Ingo Molnar <mingo@redhat.com>,
>	Dave Hansen <dave.hansen@linux.intel.com>,
>	"H. Peter Anvin" <hpa@zytor.com>,
>	virtualization@lists.linux.dev,
>	bcm-kernel-feedback-list@broadcom.com,
>	linux-kernel@vger.kernel.org,
>	John Ogness <john.ogness@linutronix.de>,
>	Steven Rostedt <rostedt@goodmis.org>,
>	Sergey Senozhatsky <senozhatsky@chromium.org>,
>	Kees Cook <kees@kernel.org>,
>	Jonathan Corbet <corbet@lwn.net>,
>	Bo Gan <bo.gan@broadcom.com>,
>	Brennan Lamoreaux <brennan.lamoreaux@broadcom.com>,
>	kexec@lists.infradead.org,
>	linux-doc@vger.kernel.org,
>	"Guilherme G. Piccoli" <gpiccoli@igalia.com>,
>	Maaz Mombasawala <maaz.mombasawala@broadcom.com>,
>	Shuah Khan <skhan@linuxfoundation.org>,
>	Randy Dunlap <rdunlap@infradead.org>,
>	Stephen Brennan <stephen.s.brennan@oracle.com>,
>	Michael Kelley <mhklinux@outlook.com>,
>	Ian Forbes <ian.forbes@broadcom.com>
>Subject: [PATCH v2 0/6] panic, x86/vmware: Preserve crash logs before kdump
>Date: Mon, 28 Sep 2026 13:55:34 -0400
>Message-ID: <cover.1790014793.git.zack.rusin@broadcom.com>
>X-Mailer: git-send-email 2.53.0
>Precedence: bulk
>X-Mailing-List: linux-doc@vger.kernel.org
>List-Id: <linux-doc.vger.kernel.org>
>List-Subscribe: <mailto:linux-doc+subscribe@vger.kernel.org>
>List-Unsubscribe: <mailto:linux-doc+unsubscribe@vger.kernel.org>
>MIME-Version: 1.0
>Content-Transfer-Encoding: 8bit
>X-DetectorID-Processed: b00c1d49-9d2e-4205-b15f-d015386d3d5e
>
>A VMware guest may have no persistent storage or working userspace after
>a crash. Preserve its newest kernel log tail in the host's vmware.log
>before entering kdump, without enabling every panic notifier by default.
>
>Following Petr's suggestion, add one generic panic_pre_kdump_list with
>VMware as its first client. A set-once helper covers panic() and direct
>crash-kexec entry. Fatal oopses can reach kdump without calling panic(),
>so patch 2 adds that call after capturing the original registers under
>the existing kexec lock. The late panic call follows sys_info() and
>precedes the kmsg dumpers.
>
>Patch 3 adds Guilherme's suggested escape hatch for debugging kdump
>failures involving early callbacks. With panic_pre_kdump_postpone set,
>the list runs before kdump only when crash_kexec_post_notifiers is also
>set. This can omit hypervisor crash handling as well as diagnostics.
>The late panic call remains eligible. This policy is a separate patch;
>I can omit patch 3 if it is not wanted.
>
>The logger preallocates an 8 KiB buffer and bounds its register-based
>RPC transfer and retries. Michael suggested increasing the buffer
>because 4 KiB can omit useful stack-trace context. Ordinary guests
>enable logging by default; encrypted guests must opt in. Use
>bool/proc_dobool for the sysctl as Joel requested. The potentially
>terminating REPORTGUESTCRASH event remains a separate late dumper,
>suppressed whenever a crash image is loaded. The v1 assignment to
>crash_kexec_post_notifiers is dropped.
>
>Tested on VMware Workstation and ESXi with ordinary guests, and on
>ESXi with SEV-SNP and TDX guests.
>
>v1:
>https://lore.kernel.org/r/cover.1788414671.git.zack.rusin@broadcom.com
>
>Zack Rusin (6):
>  panic: Add a notifier chain for pre-kdump callbacks
>  crash: Notify pre-kdump callbacks before switching kernels
>  panic: Allow postponing pre-kdump notifiers
>  x86/vmware: Add a bounded pre-kdump log sender
>  x86/vmware: Add the vmware_record_panic_msg sysctl
>  x86/vmware: Report guest crashes after kmsg dumpers
>
> .../admin-guide/kernel-parameters.txt         |  14 +
> Documentation/admin-guide/sysctl/kernel.rst   |  20 ++
> arch/x86/include/asm/vmware.h                 |   2 +
> arch/x86/kernel/cpu/vmware.c                  | 250 ++++++++++++++++++
> include/linux/panic_notifier.h                |   4 +
> kernel/crash_core.c                           |   3 +
> kernel/panic.c                                |  28 ++
> 7 files changed, 321 insertions(+)
>
>
>base-commit: fd73f4a6659897191fa0d40695fe370925dd3780
>
>


--- Thanks!
"I'm not a very positive person" - Linus torvalds

Re: [PATCH v2 0/6] panic, x86/vmware: Preserve crash logs before kdump

[Zack Rusin]

[-- Attachment #1: Type: text/plain, Size: 3216 bytes --]

On Mon, Sep 28, 2026 at 1:40 PM Bradley Morgan <brads@mainlining.org> wrote:
>
> On 28 September 2026 18:55:34 BST, Zack Rusin <zack.rusin@broadcom.com>
> wrote:
> >A VMware guest may have no persistent storage or working userspace after
> >a crash. Preserve its newest kernel log tail in the host's vmware.log
> >before entering kdump, without enabling every panic notifier by default.
> >
> >Following Petr's suggestion, add one generic panic_pre_kdump_list with
> >VMware as its first client. A set-once helper covers panic() and direct
> >crash-kexec entry. Fatal oopses can reach kdump without calling panic(),
> >so patch 2 adds that call after capturing the original registers under
> >the existing kexec lock. The late panic call follows sys_info() and
> >precedes the kmsg dumpers.
> >
> >Patch 3 adds Guilherme's suggested escape hatch for debugging kdump
> >failures involving early callbacks. With panic_pre_kdump_postpone set,
> >the list runs before kdump only when crash_kexec_post_notifiers is also
> >set. This can omit hypervisor crash handling as well as diagnostics.
> >The late panic call remains eligible. This policy is a separate patch;
> >I can omit patch 3 if it is not wanted.
> >
> >The logger preallocates an 8 KiB buffer and bounds its register-based
> >RPC transfer and retries. Michael suggested increasing the buffer
> >because 4 KiB can omit useful stack-trace context. Ordinary guests
> >enable logging by default; encrypted guests must opt in. Use
> >bool/proc_dobool for the sysctl as Joel requested. The potentially
> >terminating REPORTGUESTCRASH event remains a separate late dumper,
> >suppressed whenever a crash image is loaded. The v1 assignment to
> >crash_kexec_post_notifiers is dropped.
> >
> >Tested on VMware Workstation and ESXi with ordinary guests, and on
> >ESXi with SEV-SNP and TDX guests.
> >
> >v1:
> >https://lore.kernel.org/r/cover.1788414671.git.zack.rusin@broadcom.com
> >
> >Zack Rusin (6):
> >  panic: Add a notifier chain for pre-kdump callbacks
> >  crash: Notify pre-kdump callbacks before switching kernels
> >  panic: Allow postponing pre-kdump notifiers
> >  x86/vmware: Add a bounded pre-kdump log sender
> >  x86/vmware: Add the vmware_record_panic_msg sysctl
> >  x86/vmware: Report guest crashes after kmsg dumpers
> >
> > .../admin-guide/kernel-parameters.txt         |  14 +
> > Documentation/admin-guide/sysctl/kernel.rst   |  20 ++
> > arch/x86/include/asm/vmware.h                 |   2 +
> > arch/x86/kernel/cpu/vmware.c                  | 250 ++++++++++++++++++
> > include/linux/panic_notifier.h                |   4 +
> > kernel/crash_core.c                           |   3 +
> > kernel/panic.c                                |  28 ++
> > 7 files changed, 321 insertions(+)
> >
>
> All the panic changes and crash_core changes looks good to me!
>
> Reviewed-by: Bradley Morgan <brads@mainlining.org>
>
> Tested basic panic calls with Power10, in courtesy of osuosl:
>
> Tested-by: Bradley Morgan <brads@mainlining.org> # Power10

Thank you for testing!

Petr, do you have any thoughts? I think this version addresses all of
your concerns/suggestions from v1.

z

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5414 bytes --]

Re: [PATCH v2 2/6] crash: Notify pre-kdump callbacks before switching kernels

[John Ogness]

On 2026-09-28, Zack Rusin <zack.rusin@broadcom.com> wrote:
>> > @@ -139,6 +140,7 @@ void __noclone __crash_kexec(struct pt_regs *regs)
>> >                       struct pt_regs fixed_regs;
>> >
>> >                       crash_setup_regs(&fixed_regs, regs);
>> > +                     panic_notify_pre_kdump(NULL);
>>
>> [Severity: Medium]
>> Will this cause pre-kdump notifiers to receive a NULL message pointer instead
>> of the expected panic message?
>>
>> If a system panic occurs and kexec is configured with the default setting of
>> crash_kexec_post_notifiers=0, vpanic() triggers __crash_kexec(NULL) before
>> reaching its own explicit panic_notify_pre_kdump(buf):
>>
>> kernel/panic.c:vpanic() {
>>     ...
>>     if (!_crash_kexec_post_notifiers)
>>         __crash_kexec(NULL);
>>     ...
>>     panic_notify_pre_kdump(buf);
>> }
>>
>> When this happens, __crash_kexec() will invoke panic_notify_pre_kdump() with
>> a NULL message pointer. This permanently consumes the once-per-boot guard,
>> and any callback relying on the documented msg parameter will experience
>> data loss or potential NULL dereferences, violating the API contract that
>> promises the panic message during a panic.
>
> afaict Sashiko gets tripped up here by the doc from the previous
> email. Nothing in this series dereferences the msg. The only callback
> is the VMware logger in patch 4, which ignores the argument. It reads
> the printk buffer, where vpanic() has already written the panic line.

If you are not using the argument, why are you providing it in your new
API? Just drop the argument. If any future pre_kdump panic notifiers
need additional information, they can adjust the API appropriately.

John Ogness

Re: [PATCH v2 1/6] panic: Add a notifier chain for pre-kdump callbacks

[Guilherme G. Piccoli]

On 28/09/2026 14:55, Zack Rusin wrote:
> Selected crash callbacks should not require running every panic notifier
> before kdump. Add a separate atomic chain with an at-most-once dispatch
> helper, and invoke it before panic reaches the kmsg dumpers.
> 
> The helper permits another call from the crash path without repeating
> callbacks or recursing into them. Document that callbacks may run in NMI
> context, with other CPUs running and the kexec lock held.
> 
> Link: https://lore.kernel.org/r/aquttpMVKX8e6zGB@pathway.suse.cz
> Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>
> ---
> New in v2: separate pre-kdump chain, with a set-once dispatch guard.
> 
>  include/linux/panic_notifier.h |  3 +++
>  kernel/panic.c                 | 26 ++++++++++++++++++++++++++
>  2 files changed, 29 insertions(+)
> 
> diff --git a/include/linux/panic_notifier.h b/include/linux/panic_notifier.h
> index 41e32483d7a7..3ad390764a41 100644
> --- a/include/linux/panic_notifier.h
> +++ b/include/linux/panic_notifier.h
> @@ -6,6 +6,9 @@
>  #include <linux/types.h>
>  
>  extern struct atomic_notifier_head panic_notifier_list;
> +extern struct atomic_notifier_head panic_pre_kdump_list;
> +
> +void panic_notify_pre_kdump(char *msg);
>  
>  extern bool crash_kexec_post_notifiers;
>  
> diff --git a/kernel/panic.c b/kernel/panic.c
> index 213725b612aa..d78e90abb192 100644
> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -82,6 +82,10 @@ ATOMIC_NOTIFIER_HEAD(panic_notifier_list);
>  
>  EXPORT_SYMBOL(panic_notifier_list);
>  
> +struct atomic_notifier_head panic_pre_kdump_list =
> +	ATOMIC_NOTIFIER_INIT(panic_pre_kdump_list);
> +EXPORT_SYMBOL_GPL(panic_pre_kdump_list);
> +
>  static void panic_print_deprecated(void)
>  {
>  	pr_info_once("Kernel: The 'panic_print' parameter is now deprecated. Please use 'panic_sys_info' and 'panic_console_replay' instead.\n");
> @@ -567,6 +571,27 @@ static void panic_other_cpus_shutdown(bool crash_kexec)
>  		crash_smp_send_stop();
>  }
>  
> +/**
> + * panic_notify_pre_kdump - invoke selected callbacks before kdump
> + * @msg: Panic message, or NULL for direct crash-kexec entry
> + *
> + * Dispatch at most once per boot, including recursive entry. Callbacks run
> + * with interrupts disabled, possibly in NMI context, before other CPUs have
> + * necessarily stopped. The kexec lock may be held. They must not sleep,
> + * allocate, wait for other CPUs, or call into kexec management operations.
> + * The message is read-only and may only be used during the callback.
> + */
> +void panic_notify_pre_kdump(char *msg)
> +{
> +	static atomic_t started = ATOMIC_INIT(0);
> +	unsigned long flags;
> +
> +	local_irq_save(flags);
> +	if (!atomic_xchg(&started, 1))
> +		atomic_notifier_call_chain(&panic_pre_kdump_list, 0, msg);
> +	local_irq_restore(flags);
> +}
> +
>  /**
>   * vpanic - halt the system
>   * @fmt: The text string to print
> @@ -682,6 +707,7 @@ void vpanic(const char *fmt, va_list args)
>  
>  	sys_info(panic_print);
>  
> +	panic_notify_pre_kdump(buf);
>  	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);
>  
>  	/*

Hi Zack, thanks for trying to improve this conundrum called panic
notifiers heh

I confess at first sight I got a bit confused, and only after reading
patch 1 and 2 twice, things made sense to me: you're hooking
__crash_kexec() directly!

So, in my head, 2 things that would make sense upfront: merge patches #1
and #2, and comment a bit about it in the code.


But more than this, IIUC (and please correct me if I'm wrong): in case
__crash_kexec() is called earlier, THAT call invokes
panic_notify_pre_kdump(). Now, if that call doesn't happen (for example,
'cause crash_kexec_post_notifiers=Y), then panic_notify_pre_kdump() is
called from vpanic(), *after* the regular panic notifiers!!? Is it
intentional?

If it is, why even call it here, if __crash_exec() will be called again,
in next line, in such case of crash_kexec_post_notifiers=Y? It could be
easily done outside of `if(kexec_crash_image)` [at __crash_kexec()].

Is it really correct/expected that such "early" notifiers will end-up
being the latest notifiers if crash_kexec_post_notifiers=Y? Maybe this
is aimed at collecting also the potential extra info dumped in some
regular panic notifier?
Thanks,


Guilherme

Re: [PATCH v2 3/6] panic: Allow postponing pre-kdump notifiers

[Guilherme G. Piccoli]

On 28/09/2026 14:55, Zack Rusin wrote:
> Provide an escape hatch for debugging kdump failures that may involve
> pre-kdump callbacks. Add panic_pre_kdump_postpone to skip their early
> invocation in crash core without consuming the once-only guard.
> 
> When enabled, the chain runs before kdump only when
> crash_kexec_post_notifiers is also enabled. Leave the late panic call
> unchanged so callbacks remain eligible if that path is reached.
> Skipping the callbacks may prevent hypervisor crash handling.
> 
> Suggested-by: Guilherme G. Piccoli <gpiccoli@igalia.com>
> Link: https://lore.kernel.org/r/bc0bdd14-5e6a-0676-cd04-14fb0bb93cd5@igalia.com
> Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>
> ---
> New in v2: optional postponement policy suggested by Guilherme.
> 
>  Documentation/admin-guide/kernel-parameters.txt | 14 ++++++++++++++
>  include/linux/panic_notifier.h                  |  1 +
>  kernel/crash_core.c                             |  3 ++-
>  kernel/panic.c                                  |  2 ++
>  4 files changed, 19 insertions(+), 1 deletion(-)
> 

Thanks for implementing this! It makes sense to me...though I have some
"conflicting" comments in patch #2 heh

Cheers,


Guilherme

Re: [PATCH v2 1/6] panic: Add a notifier chain for pre-kdump callbacks

[Petr Mladek]

On Mon 2026-10-19 19:05:21, Guilherme G. Piccoli wrote:
> On 28/09/2026 14:55, Zack Rusin wrote:
> > Selected crash callbacks should not require running every panic notifier
> > before kdump. Add a separate atomic chain with an at-most-once dispatch
> > helper, and invoke it before panic reaches the kmsg dumpers.
> > 
> > The helper permits another call from the crash path without repeating
> > callbacks or recursing into them. Document that callbacks may run in NMI
> > context, with other CPUs running and the kexec lock held.
> > 
> > Link: https://lore.kernel.org/r/aquttpMVKX8e6zGB@pathway.suse.cz
> > Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>
> > ---
> > New in v2: separate pre-kdump chain, with a set-once dispatch guard.
> > 
> >  include/linux/panic_notifier.h |  3 +++
> >  kernel/panic.c                 | 26 ++++++++++++++++++++++++++
> >  2 files changed, 29 insertions(+)
> > 
> > diff --git a/include/linux/panic_notifier.h b/include/linux/panic_notifier.h
> > index 41e32483d7a7..3ad390764a41 100644
> > --- a/include/linux/panic_notifier.h
> > +++ b/include/linux/panic_notifier.h
> > @@ -6,6 +6,9 @@
> >  #include <linux/types.h>
> >  
> >  extern struct atomic_notifier_head panic_notifier_list;
> > +extern struct atomic_notifier_head panic_pre_kdump_list;
> > +
> > +void panic_notify_pre_kdump(char *msg);
> >  
> >  extern bool crash_kexec_post_notifiers;
> >  
> > diff --git a/kernel/panic.c b/kernel/panic.c
> > index 213725b612aa..d78e90abb192 100644
> > --- a/kernel/panic.c
> > +++ b/kernel/panic.c
> > @@ -82,6 +82,10 @@ ATOMIC_NOTIFIER_HEAD(panic_notifier_list);
> >  
> >  EXPORT_SYMBOL(panic_notifier_list);
> >  
> > +struct atomic_notifier_head panic_pre_kdump_list =
> > +	ATOMIC_NOTIFIER_INIT(panic_pre_kdump_list);
> > +EXPORT_SYMBOL_GPL(panic_pre_kdump_list);
> > +
> >  static void panic_print_deprecated(void)
> >  {
> >  	pr_info_once("Kernel: The 'panic_print' parameter is now deprecated. Please use 'panic_sys_info' and 'panic_console_replay' instead.\n");
> > @@ -567,6 +571,27 @@ static void panic_other_cpus_shutdown(bool crash_kexec)
> >  		crash_smp_send_stop();
> >  }
> >  
> > +/**
> > + * panic_notify_pre_kdump - invoke selected callbacks before kdump
> > + * @msg: Panic message, or NULL for direct crash-kexec entry
> > + *
> > + * Dispatch at most once per boot, including recursive entry. Callbacks run
> > + * with interrupts disabled, possibly in NMI context, before other CPUs have
> > + * necessarily stopped. The kexec lock may be held. They must not sleep,
> > + * allocate, wait for other CPUs, or call into kexec management operations.
> > + * The message is read-only and may only be used during the callback.
> > + */
> > +void panic_notify_pre_kdump(char *msg)
> > +{
> > +	static atomic_t started = ATOMIC_INIT(0);
> > +	unsigned long flags;
> > +
> > +	local_irq_save(flags);
> > +	if (!atomic_xchg(&started, 1))

It is far from clear why this is needed. It would deserve a comment in
the code and explanation in the commit message.

Well, I am not even sure whether this synchronization is enough.
If __crash_exec() and vpanic() are running in parallel on two CPUS
then one path will skip the notifiers but it will not wait until
they are finished.

But I think that the problem is the explicitly called __crash_exec()
is not synchronized against vpanic() in the first place. IMHO,
it is an existing problem which probably should get fixed.

> > +		atomic_notifier_call_chain(&panic_pre_kdump_list, 0, msg);
> > +	local_irq_restore(flags);
> > +}
> > +
> >  /**
> >   * vpanic - halt the system
> >   * @fmt: The text string to print
> > @@ -682,6 +707,7 @@ void vpanic(const char *fmt, va_list args)
> >  
> >  	sys_info(panic_print);
> >  
> > +	panic_notify_pre_kdump(buf);
> >  	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);
> >  
> >  	/*
> 
> Hi Zack, thanks for trying to improve this conundrum called panic
> notifiers heh
> 
> I confess at first sight I got a bit confused, and only after reading
> patch 1 and 2 twice, things made sense to me: you're hooking
> __crash_kexec() directly!
> 
> So, in my head, 2 things that would make sense upfront: merge patches #1
> and #2, and comment a bit about it in the code.

+1

But honestly, the semantic and ordering of all the variables and
functions is very complicated. It is very hard to follow and
understand all the rules.

I am going to think about it. But I'll first comment the 2nd
and 3rd patch.

Best Regards,
Petr

Re: [PATCH v2 2/6] crash: Notify pre-kdump callbacks before switching kernels

[Petr Mladek]

On Mon 2026-09-28 13:55:36, Zack Rusin wrote:
> Fatal x86 oopses can call crash_kexec() without reaching panic(). Run the
> pre-kdump chain from __crash_kexec() as well, after finding a loaded image
> under the kexec lock and capturing the original registers.
> 
> This covers direct crash entry without moving all panic notifiers ahead
> of kdump. A missing image or failed trylock leaves callbacks uncalled;
> the shared guard skips callbacks already invoked by panic().
> 
> Link: https://lore.kernel.org/r/aquttpMVKX8e6zGB@pathway.suse.cz
> Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>
> ---
> New in v2: cover direct crash-kexec entry in a separate core patch.
> 
>  kernel/crash_core.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/kernel/crash_core.c b/kernel/crash_core.c
> index 2b36aa9fade0..5d9fe9e00f39 100644
> --- a/kernel/crash_core.c
> +++ b/kernel/crash_core.c
> @@ -23,6 +23,7 @@
>  #include <linux/objtool.h>
>  #include <linux/delay.h>
>  #include <linux/panic.h>
> +#include <linux/panic_notifier.h>
>  
>  #include <asm/page.h>
>  #include <asm/sections.h>
> @@ -139,6 +140,7 @@ void __noclone __crash_kexec(struct pt_regs *regs)
>  			struct pt_regs fixed_regs;
>  
>  			crash_setup_regs(&fixed_regs, regs);
> +			panic_notify_pre_kdump(NULL);

I understand why you did this. But it makes the code really hard to
follow: 

  + This code path will be called also by the 1st __crash_kexec() in
    vpanic() when "_crash_kexec_post_notifiers == false". It is the
    default.

  + The panic_notify_pre_kdump(buf) in vpanic(), added by 1st patch,
    is called when "_crash_kexec_post_notifiers == true" or when
    __crash_kexec() returns from some reason.

None of this is obvious.

I am going to think more about it.

Best Regards,
Petr

>  			crash_save_vmcoreinfo();
>  			machine_crash_shutdown(&fixed_regs);
>  			crash_cma_clear_pending_dma();

Re: [PATCH v2 3/6] panic: Allow postponing pre-kdump notifiers

[Petr Mladek]

On Mon 2026-09-28 13:55:37, Zack Rusin wrote:
> Provide an escape hatch for debugging kdump failures that may involve
> pre-kdump callbacks. Add panic_pre_kdump_postpone to skip their early
> invocation in crash core without consuming the once-only guard.
> 
> When enabled, the chain runs before kdump only when
> crash_kexec_post_notifiers is also enabled. Leave the late panic call
> unchanged so callbacks remain eligible if that path is reached.
> Skipping the callbacks may prevent hypervisor crash handling.
> 
> Suggested-by: Guilherme G. Piccoli <gpiccoli@igalia.com>
> Link: https://lore.kernel.org/r/bc0bdd14-5e6a-0676-cd04-14fb0bb93cd5@igalia.com
> Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>
> ---
> New in v2: optional postponement policy suggested by Guilherme.
> 
>  Documentation/admin-guide/kernel-parameters.txt | 14 ++++++++++++++
>  include/linux/panic_notifier.h                  |  1 +
>  kernel/crash_core.c                             |  3 ++-
>  kernel/panic.c                                  |  2 ++
>  4 files changed, 19 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 33cd30996e47..3d722df30838 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -4904,6 +4904,20 @@ Kernel parameters
>  			This option should only be used for systems with the above
>  			constraints as it might cause the panic operation to be less reliable.
>  
> +	panic_pre_kdump_postpone=
> +			[KNL] Defer pre-kdump notifiers to the late panic path.
> +			They run before kdump only when
> +			crash_kexec_post_notifiers is also enabled. If panic
> +			reaches the late path without entering kdump, they
> +			can still run there.
> +			Intended for debugging kexec/crash-dump failures.
> +			Skipping these callbacks may prevent correct crash
> +			handling by the hypervisor.
> +			Format: <bool>
> +			Default: 0
> +			Writable at runtime via
> +			/sys/module/kernel/parameters/panic_pre_kdump_postpone.
> +
>  	panic_print=	Bitmask for printing system info when panic happens.
>  			User can chose combination of the following bits:
>  
> diff --git a/include/linux/panic_notifier.h b/include/linux/panic_notifier.h
> index 3ad390764a41..655ee5c77181 100644
> --- a/include/linux/panic_notifier.h
> +++ b/include/linux/panic_notifier.h
> @@ -11,5 +11,6 @@ extern struct atomic_notifier_head panic_pre_kdump_list;
>  void panic_notify_pre_kdump(char *msg);
>  
>  extern bool crash_kexec_post_notifiers;
> +extern bool panic_pre_kdump_postpone;
>  
>  #endif	/* _LINUX_PANIC_NOTIFIERS_H */
> diff --git a/kernel/crash_core.c b/kernel/crash_core.c
> index 5d9fe9e00f39..d6b2293dd0fe 100644
> --- a/kernel/crash_core.c
> +++ b/kernel/crash_core.c
> @@ -140,7 +140,8 @@ void __noclone __crash_kexec(struct pt_regs *regs)
>  			struct pt_regs fixed_regs;
>  
>  			crash_setup_regs(&fixed_regs, regs);
> -			panic_notify_pre_kdump(NULL);
> +			if (!READ_ONCE(panic_pre_kdump_postpone))
> +				panic_notify_pre_kdump(NULL);

Honestly, it looks a bit confusing to me.

The name "postpone" suggests that they will be called later.
But they likely will never be called because the crash kernel
will take over the control and eventually reboot the system.
The panic() path will never run.

Also there is:

void oops_end(unsigned long flags, struct pt_regs *regs, int signr)
{
	if (regs && kexec_should_crash(current))
		crash_kexec(regs);

where

int kexec_should_crash(struct task_struct *p)
{
	/*
	 * If crash_kexec_post_notifiers is enabled, don't run
	 * crash_kexec() here yet, which must be run after panic
	 * notifiers in panic().
	 */
	if (crash_kexec_post_notifiers)
		return 0;

So, when: "crash_kexec_post_notifiers == true" then
kexec_should_crash() returns "0" then
crash_exec() is not called. The system might still
call panic() but remember "crash_kexec_post_notifiers == true"
so the notifiers are called before __crash_kexec().
So panic_notify_pre_kdump() will still get called
before __crash_exec().

Is this intended, please?
What is use case, please?

Maybe, it would make sense to add "panic_pre_kdump_disable"
which would completely disable the notifiers when they cause
problems.

Best Regards,
Petr

Re: [PATCH v2 1/6] panic: Add a notifier chain for pre-kdump callbacks

[Petr Mladek]

On Tue 2026-10-20 14:01:43, Petr Mladek wrote:
> On Mon 2026-10-19 19:05:21, Guilherme G. Piccoli wrote:
> > On 28/09/2026 14:55, Zack Rusin wrote:
> > > Selected crash callbacks should not require running every panic notifier
> > > before kdump. Add a separate atomic chain with an at-most-once dispatch
> > > helper, and invoke it before panic reaches the kmsg dumpers.
> > > 
> > > The helper permits another call from the crash path without repeating
> > > callbacks or recursing into them. Document that callbacks may run in NMI
> > > context, with other CPUs running and the kexec lock held.
> > > 
> > > Link: https://lore.kernel.org/r/aquttpMVKX8e6zGB@pathway.suse.cz
> > > Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>
> > > ---
> > > New in v2: separate pre-kdump chain, with a set-once dispatch guard.
> > > 
> > >  include/linux/panic_notifier.h |  3 +++
> > >  kernel/panic.c                 | 26 ++++++++++++++++++++++++++
> > >  2 files changed, 29 insertions(+)
> > > 
> > > diff --git a/include/linux/panic_notifier.h b/include/linux/panic_notifier.h
> > > index 41e32483d7a7..3ad390764a41 100644
> > > --- a/include/linux/panic_notifier.h
> > > +++ b/include/linux/panic_notifier.h
> > > @@ -6,6 +6,9 @@
> > >  #include <linux/types.h>
> > >  
> > >  extern struct atomic_notifier_head panic_notifier_list;
> > > +extern struct atomic_notifier_head panic_pre_kdump_list;
> > > +
> > > +void panic_notify_pre_kdump(char *msg);
> > >  
> > >  extern bool crash_kexec_post_notifiers;
> > >  
> > > diff --git a/kernel/panic.c b/kernel/panic.c
> > > index 213725b612aa..d78e90abb192 100644
> > > --- a/kernel/panic.c
> > > +++ b/kernel/panic.c
> > > @@ -82,6 +82,10 @@ ATOMIC_NOTIFIER_HEAD(panic_notifier_list);
> > >  
> > >  EXPORT_SYMBOL(panic_notifier_list);
> > >  
> > > +struct atomic_notifier_head panic_pre_kdump_list =
> > > +	ATOMIC_NOTIFIER_INIT(panic_pre_kdump_list);
> > > +EXPORT_SYMBOL_GPL(panic_pre_kdump_list);
> > > +
> > >  static void panic_print_deprecated(void)
> > >  {
> > >  	pr_info_once("Kernel: The 'panic_print' parameter is now deprecated. Please use 'panic_sys_info' and 'panic_console_replay' instead.\n");
> > > @@ -567,6 +571,27 @@ static void panic_other_cpus_shutdown(bool crash_kexec)
> > >  		crash_smp_send_stop();
> > >  }
> > >  
> > > +/**
> > > + * panic_notify_pre_kdump - invoke selected callbacks before kdump
> > > + * @msg: Panic message, or NULL for direct crash-kexec entry
> > > + *
> > > + * Dispatch at most once per boot, including recursive entry. Callbacks run
> > > + * with interrupts disabled, possibly in NMI context, before other CPUs have
> > > + * necessarily stopped. The kexec lock may be held. They must not sleep,
> > > + * allocate, wait for other CPUs, or call into kexec management operations.
> > > + * The message is read-only and may only be used during the callback.
> > > + */
> > > +void panic_notify_pre_kdump(char *msg)
> > > +{
> > > +	static atomic_t started = ATOMIC_INIT(0);
> > > +	unsigned long flags;
> > > +
> > > +	local_irq_save(flags);
> > > +	if (!atomic_xchg(&started, 1))
> 
> It is far from clear why this is needed. It would deserve a comment in
> the code and explanation in the commit message.
> 
> Well, I am not even sure whether this synchronization is enough.
> If __crash_exec() and vpanic() are running in parallel on two CPUS
> then one path will skip the notifiers but it will not wait until
> they are finished.
> 
> But I think that the problem is the explicitly called __crash_exec()
> is not synchronized against vpanic() in the first place. IMHO,
> it is an existing problem which probably should get fixed.

It seems that __crash_exec() should be called only when panic_cpu
is set. This is why crash_exec() does:

__bpf_kfunc void crash_kexec(struct pt_regs *regs)
{
	if (panic_try_start()) {
		/* This is the 1st CPU which comes here, so go ahead. */
		__crash_kexec(regs);

The only exception seems to be (found by gemini) in Hyper-V code:

static void __noreturn hv_crash_handle(void)
{
	hv_crash_restore_tss();
	hv_crash_clear_kernpt();

	/* we are now fully in devirtualized normal kernel mode */
	__crash_kexec(NULL);

	hv_panic_timeout_reboot();
}

I belive that it is a bug and it should do:

	/* Only one CPU is allowed to process with kdump and panic. */
	if (!panic_try_start())
		return;

But it is possible that it is serialized another way. Anyway, even
Hyper-V must make sure that __crash_kexec() is called only on
one CPU.

Result: We should not need any extra synchronization for
	the panic_notify_pre_kdump().

	We might just need to prevent calling them twice if we
	decide to call this function in __crash_exec() as well.
	A simple bool variable should be enough. But I am still
	not sure if it is a good design.

Continue thinking...

Best Regards,
Petr

Re: [PATCH v2 1/6] panic: Add a notifier chain for pre-kdump callbacks

[Petr Mladek]

On Tue 2026-10-20 14:01:43, Petr Mladek wrote:
> On Mon 2026-10-19 19:05:21, Guilherme G. Piccoli wrote:
> > On 28/09/2026 14:55, Zack Rusin wrote:
> > > Selected crash callbacks should not require running every panic notifier
> > > before kdump. Add a separate atomic chain with an at-most-once dispatch
> > > helper, and invoke it before panic reaches the kmsg dumpers.
> > > 
> > > The helper permits another call from the crash path without repeating
> > > callbacks or recursing into them. Document that callbacks may run in NMI
> > > context, with other CPUs running and the kexec lock held.
> > > 
> > > Link: https://lore.kernel.org/r/aquttpMVKX8e6zGB@pathway.suse.cz
> > > Signed-off-by: Zack Rusin <zack.rusin@broadcom.com>
> > > ---
> > > New in v2: separate pre-kdump chain, with a set-once dispatch guard.
> > > 
> > >  include/linux/panic_notifier.h |  3 +++
> > >  kernel/panic.c                 | 26 ++++++++++++++++++++++++++
> > >  2 files changed, 29 insertions(+)
> > > 
> > > diff --git a/include/linux/panic_notifier.h b/include/linux/panic_notifier.h
> > > index 41e32483d7a7..3ad390764a41 100644
> > > --- a/include/linux/panic_notifier.h
> > > +++ b/include/linux/panic_notifier.h
> > > @@ -6,6 +6,9 @@
> > >  #include <linux/types.h>
> > >  
> > >  extern struct atomic_notifier_head panic_notifier_list;
> > > +extern struct atomic_notifier_head panic_pre_kdump_list;
> > > +
> > > +void panic_notify_pre_kdump(char *msg);
> > >  
> > >  extern bool crash_kexec_post_notifiers;
> > >  
> > > diff --git a/kernel/panic.c b/kernel/panic.c
> > > index 213725b612aa..d78e90abb192 100644
> > > --- a/kernel/panic.c
> > > +++ b/kernel/panic.c
> > > @@ -82,6 +82,10 @@ ATOMIC_NOTIFIER_HEAD(panic_notifier_list);
> > >  
> > >  EXPORT_SYMBOL(panic_notifier_list);
> > >  
> > > +struct atomic_notifier_head panic_pre_kdump_list =
> > > +	ATOMIC_NOTIFIER_INIT(panic_pre_kdump_list);
> > > +EXPORT_SYMBOL_GPL(panic_pre_kdump_list);
> > > +
> > >  static void panic_print_deprecated(void)
> > >  {
> > >  	pr_info_once("Kernel: The 'panic_print' parameter is now deprecated. Please use 'panic_sys_info' and 'panic_console_replay' instead.\n");
> > > @@ -567,6 +571,27 @@ static void panic_other_cpus_shutdown(bool crash_kexec)
> > >  		crash_smp_send_stop();
> > >  }
> > >  
> > > +/**
> > > + * panic_notify_pre_kdump - invoke selected callbacks before kdump
> > > + * @msg: Panic message, or NULL for direct crash-kexec entry
> > > + *
> > > + * Dispatch at most once per boot, including recursive entry. Callbacks run
> > > + * with interrupts disabled, possibly in NMI context, before other CPUs have
> > > + * necessarily stopped. The kexec lock may be held. They must not sleep,
> > > + * allocate, wait for other CPUs, or call into kexec management operations.
> > > + * The message is read-only and may only be used during the callback.
> > > + */
> > > +void panic_notify_pre_kdump(char *msg)
> > > +{
> > > +	static atomic_t started = ATOMIC_INIT(0);
> > > +	unsigned long flags;
> > > +
> > > +	local_irq_save(flags);
> > > +	if (!atomic_xchg(&started, 1))
> 
> It is far from clear why this is needed. It would deserve a comment in
> the code and explanation in the commit message.
> 
> Well, I am not even sure whether this synchronization is enough.
> If __crash_exec() and vpanic() are running in parallel on two CPUS
> then one path will skip the notifiers but it will not wait until
> they are finished.
> 
> But I think that the problem is the explicitly called __crash_exec()
> is not synchronized against vpanic() in the first place. IMHO,
> it is an existing problem which probably should get fixed.
> 
> > > +		atomic_notifier_call_chain(&panic_pre_kdump_list, 0, msg);
> > > +	local_irq_restore(flags);
> > > +}
> > > +
> > >  /**
> > >   * vpanic - halt the system
> > >   * @fmt: The text string to print
> > > @@ -682,6 +707,7 @@ void vpanic(const char *fmt, va_list args)
> > >  
> > >  	sys_info(panic_print);
> > >  
> > > +	panic_notify_pre_kdump(buf);
> > >  	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);
> > >  
> > >  	/*
> > 
> > Hi Zack, thanks for trying to improve this conundrum called panic
> > notifiers heh
> > 
> > I confess at first sight I got a bit confused, and only after reading
> > patch 1 and 2 twice, things made sense to me: you're hooking
> > __crash_kexec() directly!
> > 
> > So, in my head, 2 things that would make sense upfront: merge patches #1
> > and #2, and comment a bit about it in the code.
> 
> +1
> 
> But honestly, the semantic and ordering of all the variables and
> functions is very complicated. It is very hard to follow and
> understand all the rules.

Warning: You might skip the first part about my confusion.
	Maybe, I am the only one who is a bit lost this way.

	My motivation is to do not repeat the mess which I saw in
	the printk registration code. People added many features
	there for over two decades. Each change made sense on its own.
	But, for me, it was really hard to understand and distinguish
	what were the wanted and unwanted effects.


I trying to understans why I am confused by the panic notifiers code.
There are several aspects:

1. I was involved in many mail threads where we discussed calling:

	notifiers  before/after  kdump

   but the variable "crash_kexec_post_notifiers" defines calling:

	kdump   before/after  notifiers

   It actually matches the code because __crash_kexec() is called
   twice in vpanic() before and after the notifiers.

   Sigh, it looks like a trivial mental switch. But it is not
   that trivial when we though about splitting the notifiers
   into more lists and discussed where to call them.

   The discussion always was where to call notifiers and not where to
   call kdump.

   And we are also discussing about disabling notifiers. Not about
   disabling kdump.


2. We call the new list "panic_pre_kdump" and this patchset calls
   it in both vpanic() and __crash_kexec() code paths.

   The name "pre_kdump" suggests that it is needed only when kdump
   is called. But we want to call it in panic() even when
   kdump is not called.


3. The new "pre_kdump" notifiers are called before stopping CPUs.
   While the rest is called in the more safe environment
   with stopped CPUs.


4. I am a bit confused by this patch set because it allows
   dumping printk log before kdump. It does not fit with
   my understanding of having "pre_kdump" list only for
   the strictly needed operations.


5. There is the idea of splitting the remainig notifiers into
   _safe and _unsafe lists, see
   https://lore.kernel.org/r/atcjhe4329sYHWvZ@pathway.suse.cz

   It would complicate the design even more. But it might help
   with the endless war whether "crash_kexec_post_notifiers"
   should be "true" or "false" by a platform-specific default.

   I wonder how all this fits together.


Top-level look:
===============

Sigh, one problem is that the notifiers have different purpose:

   1. Notify hypervisor (platform). This looks like a must-to-have.

   2. Platform specific dump of printk messages. This is
      nice-to-have. But note that we also have kmsg_dump API
      for this, see kmsg_dump_desc().

   3. Print extra info. Also this is nice-to-have.

      Some is optional when kdump is created because it might be
      read from there. Some can't be found in crash dump, e.g.
      dumping lowlevel HW state.

      In addition, there is sys_info() info API which allows to
      dump state of various subsystems.

   4. Some notifiers are smp-safe and some just a best effort
      and more safe after disabling other CPUs.

      In addition, I guess that it is better to call crash_kexec()
      without stopping other CPUs because stopping is not reliable
      without NMI. And it would create extra noise in the backtrace.
      And ???


Now, different people have different preferences but mainly:

   + the more safe kdump the better
   + the more safe reboot the better
   + the more info the better


Now, the question is how to satisfy the various requirements
and keep the design sane. We need a compromise but where.

Continue thinking...

Best Regards,
Petr

Re: [PATCH v2 1/6] panic: Add a notifier chain for pre-kdump callbacks

[Petr Mladek]

On Tue 2026-10-20 16:30:53, Petr Mladek wrote:
> On Tue 2026-10-20 14:01:43, Petr Mladek wrote:
> > On Mon 2026-10-19 19:05:21, Guilherme G. Piccoli wrote:
> > > On 28/09/2026 14:55, Zack Rusin wrote:
> > > > Selected crash callbacks should not require running every panic notifier
> > > > before kdump. Add a separate atomic chain with an at-most-once dispatch
> > > > helper, and invoke it before panic reaches the kmsg dumpers.
> > > > 
> > > > The helper permits another call from the crash path without repeating
> > > > callbacks or recursing into them. Document that callbacks may run in NMI
> > > > context, with other CPUs running and the kexec lock held.
> > > > 
> > > > --- a/kernel/panic.c
> > > > +++ b/kernel/panic.c
> > > > @@ -682,6 +707,7 @@ void vpanic(const char *fmt, va_list args)
> > > >  
> > > >  	sys_info(panic_print);
> > > >  
> > > > +	panic_notify_pre_kdump(buf);
> > > >  	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);
> > > >  
> > > >  	/*
> > > 
> > > Hi Zack, thanks for trying to improve this conundrum called panic
> > > notifiers heh
> > > 
> > > I confess at first sight I got a bit confused, and only after reading
> > > patch 1 and 2 twice, things made sense to me: you're hooking
> > > __crash_kexec() directly!
> > > 
> > > So, in my head, 2 things that would make sense upfront: merge patches #1
> > > and #2, and comment a bit about it in the code.
> > 
> > +1
> > 
> > But honestly, the semantic and ordering of all the variables and
> > functions is very complicated. It is very hard to follow and
> > understand all the rules.
> 
> I trying to understans why I am confused by the panic notifiers code.
> There are several aspects:
> 
> 1. I was involved in many mail threads where we discussed calling:
> 
> 	notifiers  before/after  kdump
> 
>    but the variable "crash_kexec_post_notifiers" defines calling:
> 
> 	kdump   before/after  notifiers
> 
>    It actually matches the code because __crash_kexec() is called
>    twice in vpanic() before and after the notifiers.
> 
>    Sigh, it looks like a trivial mental switch. But it is not
>    that trivial when we though about splitting the notifiers
>    into more lists and discussed where to call them.
> 
>    The discussion always was where to call notifiers and not where to
>    call kdump.
> 
>    And we are also discussing about disabling notifiers. Not about
>    disabling kdump.
> 
> 
> 2. We call the new list "panic_pre_kdump" and this patchset calls
>    it in both vpanic() and __crash_kexec() code paths.
> 
>    The name "pre_kdump" suggests that it is needed only when kdump
>    is called. But we want to call it in panic() even when
>    kdump is not called.
> 
> 
> 3. The new "pre_kdump" notifiers are called before stopping CPUs.
>    While the rest is called in the more safe environment
>    with stopped CPUs.
> 
> 
> 4. I am a bit confused by this patch set because it allows
>    dumping printk log before kdump. It does not fit with
>    my understanding of having "pre_kdump" list only for
>    the strictly needed operations.
> 
> 
> 5. There is the idea of splitting the remainig notifiers into
>    _safe and _unsafe lists, see
>    https://lore.kernel.org/r/atcjhe4329sYHWvZ@pathway.suse.cz
> 
>    It would complicate the design even more. But it might help
>    with the endless war whether "crash_kexec_post_notifiers"
>    should be "true" or "false" by a platform-specific default.
> 
>    I wonder how all this fits together.
> 
> 
> Top-level look:
> ===============
> 
> Sigh, one problem is that the notifiers have different purpose:
> 
>    1. Notify hypervisor (platform). This looks like a must-to-have.
> 
>    2. Platform specific dump of printk messages. This is
>       nice-to-have. But note that we also have kmsg_dump API
>       for this, see kmsg_dump_desc().
> 
>    3. Print extra info. Also this is nice-to-have.
> 
>       Some is optional when kdump is created because it might be
>       read from there. Some can't be found in crash dump, e.g.
>       dumping lowlevel HW state.
> 
>       In addition, there is sys_info() info API which allows to
>       dump state of various subsystems.
> 
>    4. Some notifiers are smp-safe and some just a best effort
>       and more safe after disabling other CPUs.
> 
>       In addition, I guess that it is better to call crash_kexec()
>       without stopping other CPUs because stopping is not reliable
>       without NMI. And it would create extra noise in the backtrace.
>       And ???
> 
> 
> Now, different people have different preferences but mainly:
> 
>    + the more safe kdump the better
>    + the more safe reboot the better
>    + the more info the better
> 
> 
> Now, the question is how to satisfy the various requirements
> and keep the design sane. We need a compromise but where.

With the above in mind, I think about two approaches. I have the
following constrains in mind:

   + keep it simple (2-3 lists max)

   + If people configure kdump then they want to use it.
     Post crash_kexec() paths are just a fallback.

   + If people configure panic_sys_info + crash dump then they
     want it even in crash dump.

   + If people configure any kmsg dump and configure kdump
     then they want it before crash_kexec().

   + If a platform wants to do something then it wants to do
     it before __crash_kexec().

   + If something cause problems then fix it. But allow to disable
     it for debugging purposes.

One very important aspect is when other cpus should get disabled.
I guess that most people would want to call __crash_kexec() before
disabling other CPU. Even though, I am not completely sure why.
But some notifiers might be hard to do smp_safe.

The disabling of other CPUs seems to be the most critical operation
which would split notifiers into two groups.

Approach A: Two list: smp_safe vs. smp_unsafe
=============================================

We could create two list:

   + panic_notifiers_smp_safe_list
   + panic_notifiers_smp_unsafe_list

Where the safe list would be called before __crash_kexec().
And the other after CPUs get disabled.

In the long term, I would get rid of the controversial
"crash_kexec_post_notifiers" options. Instead, I would add:

    + disable_panic_notifiers_smp_safe
    + disable_panic_notifiers_smp_unsafe
    + disable_cpus_before_kdump

Then we might call:

	panic_notify_smp_safe();

	if (disable_cpus_before_kdump) {
		crash_smp_send_stop();
		panic_notify_smp_unsafe();
	}

	__crash_kexec();

	if (!disable_cpus_before_kdump) {
		crash_smp_send_stop();
		panic_notify_smp_unsafe();
	}

Where the panic_notify* functions would get implemented as:

	void panic_notify_smp_safe(void)
	{
		if (disable_panic_notifiers_smp_safe)
			return;

		...
	}


Approach B: Three lists: platform, smp_safe, and smp_unsafe()
=============================================================

It is similar to approach A. We would just add an extra:

     + panic_notifiers_platform_list
     + disable_panic_notifiers_platform_list

This list would be used for smp-safe notifiers which are
required be some platform. It would be the notifiers which
are crucial. Which are currently the reason to set
"_crash_kexec_post_notifiers".

It might be called also when __crash_kexec() is called
outside panic().

It would allow to separate the most crucial stuff from
the rest. It might help when it might be enough to
disable the less crucial stuff for debugging kdump
issues.


Conclusion:
===========

Of course, there are many details which might need to be sorted,
for example:

  + how to put this together with the existing list
    until all notifiers are migrated.

  + which list should be called when (__)crash_kexec() is
    called explicitly outside vpanic().

This patchset, adding vmware_log_notifier, might implement just
the panic_notifiers_platform_list. The rest might be
added by another volunteer.

How does this sounds, please?

Best Regards,
Petr