[RFC PATCH 0/4] panic: a pre kdump notifier list for hypervisor upcalls

[RFC PATCH 0/4] panic: a pre kdump notifier list for hypervisor upcalls

[Bradley Morgan]

When a crash kernel is loaded, panic() jumps to it before the panic
notifiers run, unless crash_kexec_post_notifiers is set. So the
hypervisor or firmware never finds out the guest panicked. Hyper-V
doesn't get the crash registers, gsmi drops its firmware log entry,
pvpanic stays silent, SEV-SNP skips its firmware and IOMMU shutdown.
Whatever's watching the machine, the host, the BMC, fleet, sees a
clean reboot instead of a crash.

The only way around it today is crash_kexec_post_notifiers, but that
runs the whole legacy notifier list before the kdump. That list has
slow callbacks in it, IPMI being the obvious one since it talks to a
BMC, so turning it on slows down every crash dump on the box. Hyper-V
turns it on anyway and eats the cost for one callback. SNP did the
same.

This adds a separate list that runs before the crash kexec no matter
what. Callbacks on it have to follow a contract: no locks, no
allocation, no sleeping, other CPUs may still be running, and it has
to tolerate being entered again if the panic path itself panics. The
list head is private and you register through a function, because
that's how you make people follow the contract. Priorities order it
like any notifier chain. It lives in its own file because it's not
panic code, just a notifier chain vpanic() calls once, and the custom
walk it needs doesn't belong in the generic notifier API either. A
separate file is also the only thing a MAINTAINERS entry can scope a
reviewer to.

Running anything before the crash kexec is a reliability trade. If an
upcall hangs or faults and costs the dump, you want to know which one
did it, so the notify path walks the chain itself and prints each
callback's name on the console before calling it. Return values are
ignored. A nested panic picks up the walk after the callback that was
running, instead of re-faulting on it.

This is the minimum: the list, the hook in the panic path, one driver
converted so there's a real user, and the MAINTAINERS entry. pvpanic
is the first one. Its callback is a self contained upcall that already
takes its lock as a trylock, so it fits the contract as is. The rest
(Hyper-V, Xen, gsmi, SNP) come in a follow on series. gsmi needs a
rework to a trylock because its old deadlock guard assumed the other
CPUs were stopped, which isn't true on this list. Once SNP's notifier
is on the new list, the crash_kexec_post_notifiers forcing in
snp_rmptable_init() goes away and SNP gets the early crash kexec back.
Hyper-V keeps its forcing for now because its kmsg dump pass also needs
to run before kdump. The register report just doesn't depend on it
anymore.

IPMI stays out. Its panic handling assumes the other CPUs have been
stopped, and poking a BMC before the crash kexec would slow down every
kdump on any box with a BMC. It stays on the legacy list, where kdump
skips it like before.

The legacy list, its position in the panic path and
crash_kexec_post_notifiers itself are untouched, so the remaining
registrants see zero change and nothing is renamed.

This is on purpose. Piccoli's 2022 series tried to classify every
panic notifier and the list split didn't go in. This does the one piece
that fixes the kdump versus hypervisor conflict and stops.

One alternative is splitting the legacy list by priority and calling the
high priority half before the crash kexec. That can't do what this list
does. The walk prints each callback's name on the console before it runs,
and it ignores return values. atomic_notifier_call_chain can't do
either. It honors NOTIFY_STOP_MASK, so one callback can halt the pass
and lose every upcall after it. The walk also resumes past the faulting
callback on a nested panic and runs under RCU instead of the atomic
spinlock. Getting this behavior under a priority split means pushing a
custom walk into the generic notifier API, which is more invasive than
a file.

Build tested on arm64.

Bradley Morgan (4):
  panic: add a pre kdump notifier list
  panic: run the pre kdump list before crash kexec
  misc/pvpanic: notify the host on the pre kdump list
  MAINTAINERS: add an entry for the pre kdump notifier list

 Documentation/admin-guide/kernel-parameters.txt |  3 ++
 MAINTAINERS                                     |  6 +++
 drivers/misc/pvpanic/pvpanic.c                  |  5 +-
 include/linux/panic_notifier.h                  | 18 +++++++
 kernel/Makefile                                 |  2 +-
 kernel/panic.c                                  |  7 +++
 kernel/panic_notifiers.c                        | 65 +++++++++++++++++++++++++
 7 files changed, 102 insertions(+), 4 deletions(-)
 create mode 100644 kernel/panic_notifiers.c

--
2.53.0

[RFC PATCH 1/4] panic: add a pre kdump notifier list

[Bradley Morgan]

When a crash kernel is loaded, panic() jumps to it before the panic
notifiers run. A hypervisor or firmware therefore never learns that
the guest panicked: the Hyper-V host never receives the crash
registers, a Google gsmi firmware log entry is dropped, a pvpanic
device never signals the host, and an AMD SEV-SNP guest skips the
firmware and IOMMU shutdown. Fleet alerting, availability tracking,
and crash logging all miss the event.

The only escape hatch today is crash_kexec_post_notifiers, but it runs
the entire legacy notifier list before the kdump. That list carries
slow callbacks, IPMI being the obvious one because it talks to a BMC,
so enabling it delays every crash dump on the machine. Hyper-V forces
it on anyway and penalizes every kdump for the sake of one upcall;
this series also removes the forcing that SEV-SNP added for the same
reason, giving SNP hosts the normal early crash kexec back.

Add a small dedicated list that always runs before crash kexec. The
list head stays private and registration goes through a function,
because the callbacks on this list have a strict contract and an API
is how you enforce one. The list lives in its own file, not in panic.c,
so the contract can carry a MAINTAINERS entry and the existing panic
code does not have to move.

Priorities order the list like any atomic notifier chain. The header
documents that they are reserved for callbacks that must run last, for
example because they may not return or because they tear down a
transport that other callbacks still need.

The notify path walks the chain by hand and names each callback on the
console before invoking it. Running anything before the crash kexec is
a reliability trade, if an upcall hangs or faults and costs us the
crash dump, the console identifies the culprit instead of leaving a
machine that silently never rebooted into the kdump kernel. Callback
return values are deliberately ignored, and a nested panic resumes the
walk after the callback that was running instead of refaulting on it.

Signed-off-by: Bradley Morgan <include@grrlz.net>
---
 include/linux/panic_notifier.h | 18 ++++++++++
 kernel/Makefile                |  2 +-
 kernel/panic_notifiers.c       | 65 ++++++++++++++++++++++++++++++++++
 3 files changed, 84 insertions(+), 1 deletion(-)
 create mode 100644 kernel/panic_notifiers.c

diff --git a/include/linux/panic_notifier.h b/include/linux/panic_notifier.h
index 41e32483d7a7..1e99de3a8a6a 100644
--- a/include/linux/panic_notifier.h
+++ b/include/linux/panic_notifier.h
@@ -7,6 +7,24 @@
 
 extern struct atomic_notifier_head panic_notifier_list;
 
+/*
+ * The pre kdump list runs on every panic, before a potential crash
+ * kexec, so a hypervisor or firmware learns of the panic even when a
+ * crash kernel is loaded. A callback:
+ *
+ *  - must not take locks, allocate memory or sleep
+ *  - must not assume other CPUs have been stopped
+ *  - must tolerate being entered again if the panic path panics
+ *
+ * One that cannot belongs on the legacy panic_notifier_list instead.
+ * Reserve low priorities for callbacks that must run last, for
+ * example because they do not return. Return values are ignored: the
+ * whole list always runs.
+ */
+int panic_notifier_register_pre_kdump(struct notifier_block *nb);
+int panic_notifier_unregister_pre_kdump(struct notifier_block *nb);
+void panic_pre_kdump_notify(const char *buf);
+
 extern bool crash_kexec_post_notifiers;
 
 #endif	/* _LINUX_PANIC_NOTIFIERS_H */
diff --git a/kernel/Makefile b/kernel/Makefile
index 1e1a31673577..7add98a5acc2 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -3,7 +3,7 @@
 # Makefile for the linux kernel.
 #
 
-obj-y     = fork.o exec_domain.o exec_state.o panic.o \
+obj-y     = fork.o exec_domain.o exec_state.o panic.o panic_notifiers.o \
 	    cpu.o exit.o softirq.o resource.o \
 	    sysctl.o capability.o ptrace.o user.o \
 	    signal.o sys.o umh.o workqueue.o pid.o task_work.o \
diff --git a/kernel/panic_notifiers.c b/kernel/panic_notifiers.c
new file mode 100644
index 000000000000..31f15f84662e
--- /dev/null
+++ b/kernel/panic_notifiers.c
@@ -0,0 +1,65 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* The pre kdump panic notifier list. See include/linux/panic_notifier.h. */
+#include <linux/export.h>
+#include <linux/kernel.h>
+#include <linux/notifier.h>
+#include <linux/panic_notifier.h>
+#include <linux/printk.h>
+#include <linux/rcupdate.h>
+
+static ATOMIC_NOTIFIER_HEAD(panic_pre_kdump_list);
+
+/* The callback must satisfy the contract in include/linux/panic_notifier.h. */
+int panic_notifier_register_pre_kdump(struct notifier_block *nb)
+{
+	return atomic_notifier_chain_register(&panic_pre_kdump_list, nb);
+}
+EXPORT_SYMBOL_GPL(panic_notifier_register_pre_kdump);
+
+/* Synchronizes RCU and may block; must not be called from the panic path. */
+int panic_notifier_unregister_pre_kdump(struct notifier_block *nb)
+{
+	return atomic_notifier_chain_unregister(&panic_pre_kdump_list, nb);
+}
+EXPORT_SYMBOL_GPL(panic_notifier_unregister_pre_kdump);
+
+/*
+ * Called from panic() only, right before a potential crash kexec. Callbacks
+ * are named on the console first and their return values ignored.
+ */
+void panic_pre_kdump_notify(const char *buf)
+{
+	static struct notifier_block *cursor;
+	struct notifier_block *nb, *resume_after;
+
+	/* No locking: panic_cpu serializes; a nested panic just runs again. */
+	resume_after = cursor;
+
+	rcu_read_lock();
+	for (nb = rcu_dereference(panic_pre_kdump_list.head); nb;
+	     nb = rcu_dereference(nb->next)) {
+		/*
+		 * On a nested panic, skip past the callback that was
+		 * running when it hit: that one is the likely culprit.
+		 */
+		if (resume_after) {
+			if (nb == resume_after)
+				resume_after = NULL;
+			continue;
+		}
+
+#ifdef CONFIG_DEBUG_NOTIFIERS
+		if (unlikely(!func_ptr_is_kernel_text(nb->notifier_call))) {
+			WARN(1, "Invalid pre kdump notifier!");
+			continue;
+		}
+#endif
+		cursor = nb;
+		pr_emerg("pre kdump notifier: calling %ps\n",
+			 nb->notifier_call);
+		/* The zero action mirrors the legacy panic_notifier_list. */
+		nb->notifier_call(nb, 0, (void *)buf);
+	}
+	rcu_read_unlock();
+	cursor = NULL;
+}
-- 
2.53.0

[RFC PATCH 2/4] panic: run the pre kdump list before crash kexec

[Bradley Morgan]

Call the new list unconditionally in vpanic, right before the
early crash kexec. The legacy list, crash_kexec_post_notifiers
and the cpu shutdown sequence are untouched. Also note in the
kernel parameters entry that the option has no effect on the
pre kdump list.

Signed-off-by: Bradley Morgan <include@grrlz.net>
---
 Documentation/admin-guide/kernel-parameters.txt | 3 +++
 kernel/panic.c                                  | 7 +++++++
 2 files changed, 10 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index b5493a7f8f22..a77ecf3bd564 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -1042,6 +1042,9 @@ Kernel parameters
 			or extra data dumped by panic_print. Note that some
 			configurations enable this option unconditionally,
 			like Hyper-V, PowerPC (fadump) and AMD SEV-SNP.
+			This option does not affect the pre kdump notifier
+			list (see include/linux/panic_notifier.h), which
+			always runs before the jump to the kdump kernel.
 
 	crashkernel=size[KMG][@offset[KMG]]
 			[KNL,EARLY] Using kexec, Linux can switch to a 'crash kernel'
diff --git a/kernel/panic.c b/kernel/panic.c
index 94ce7a94f118..f9ad0250da67 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -664,6 +664,13 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	kgdb_panic(buf);
 
+	/*
+	 * Tell the hypervisor or firmware about the panic. This runs on
+	 * every panic, before a potential crash kexec, so the upcall is
+	 * not lost when a crash kernel is loaded.
+	 */
+	panic_pre_kdump_notify(buf);
+
 	/*
 	 * If we have crashed and we have a crash kernel loaded let it handle
 	 * everything else.
-- 
2.53.0

[RFC PATCH 3/4] misc/pvpanic: notify the host on the pre kdump list

[Bradley Morgan]

Writing the panic event to the pvpanic device is a self contained
upcall, and pvpanic_send_event() already takes pvpanic_lock with a
trylock, so the callback satisfies the contract as is.

The INT_MAX priority is kept so the host hears about the panic
before anything else on the list runs.

Signed-off-by: Bradley Morgan <include@grrlz.net>
---
 drivers/misc/pvpanic/pvpanic.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/drivers/misc/pvpanic/pvpanic.c b/drivers/misc/pvpanic/pvpanic.c
index 17c0eb549463..c409968228e4 100644
--- a/drivers/misc/pvpanic/pvpanic.c
+++ b/drivers/misc/pvpanic/pvpanic.c
@@ -218,7 +218,7 @@ static int pvpanic_init(void)
 	INIT_LIST_HEAD(&pvpanic_list);
 	spin_lock_init(&pvpanic_lock);
 
-	atomic_notifier_chain_register(&panic_notifier_list, &pvpanic_panic_nb);
+	panic_notifier_register_pre_kdump(&pvpanic_panic_nb);
 
 	return 0;
 }
@@ -226,7 +226,6 @@ module_init(pvpanic_init);
 
 static void pvpanic_exit(void)
 {
-	atomic_notifier_chain_unregister(&panic_notifier_list, &pvpanic_panic_nb);
-
+	panic_notifier_unregister_pre_kdump(&pvpanic_panic_nb);
 }
 module_exit(pvpanic_exit);
-- 
2.53.0

[RFC PATCH 4/4] MAINTAINERS: add an entry for the pre kdump notifier list

[Bradley Morgan]

Scoped to the new file and the header so patches touching the
callback contract get an extra reviewer.

Signed-off-by: Bradley Morgan <include@grrlz.net>
---
 MAINTAINERS | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 895a87b571c3..44b6318ed692 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -20453,6 +20453,12 @@ L:	platform-driver-x86@vger.kernel.org
 S:	Maintained
 F:	drivers/platform/x86/panasonic-laptop.c
 
+PANIC PRE KDUMP NOTIFIER LIST
+M:	Bradley Morgan <include@grrlz.net>
+S:	Maintained
+F:	include/linux/panic_notifier.h
+F:	kernel/panic_notifiers.c
+
 PARALLAX PING IIO SENSOR DRIVER
 M:	Andreas Klinger <ak@it-klinger.de>
 L:	linux-iio@vger.kernel.org
-- 
2.53.0