[RFC PATCH v2 0/5] panic, crash_core, sys_info: Introduce crash_kexec_in_memory_sys_info

[RFC PATCH v2 0/5] panic, crash_core, sys_info: Introduce crash_kexec_in_memory_sys_info

[Aaron Tomlin]

In modern fleet environments, capturing a full multi-gigabyte vmcore via
kdump can occasionally fail or be intentionally skipped due to storage,
network, or encryption constraints. In these scenarios, the extracted
dmesg buffer (e.g. vmcore-dmesg.txt) is often the sole artefact
available for post-mortem triage.

Currently, appending supplemental system state (such as memory usage, held
locks, active timers, task states, or ftrace buffers) via panic_print prior
to kdump requires enabling crash_kexec_post_notifiers. Doing so exposes the
crash path to registered panic notifiers (vastly increasing the risk of
hanging or deadlocking) and forces panic_print output to be flushed
synchronously to physical consoles. On a slow serial console or IPMI
Serial-over-LAN (e.g. 115200 baud), flushing comprehensive sys_info()
output (e.g. tasks or stacktraces) can exceed 10 seconds, allowing the
hardware timer to expire and reset the system before kdump can initialise.
Furthermore, on large systems, high-volume telemetry can wrap the printk
ring buffer and overwrite the initial Oops, BUG, or panic stack trace.

This series introduces the "crash_kexec_in_memory_sys_info" kernel
command-line parameter to decouple in-memory telemetry collection from
both panic notifiers and physical console flushing when kdump is active:

    1.  Fixes duplicate and dropped all-CPU backtraces when SYS_INFO_ALL_BT
        is configured in panic_print across early SMP shutdown and
        __sys_info().

    2.  Avoids spinning indefinitely on zone->lock in show_free_areas()
        during NMI, Oops, or panic contexts by falling back to non-blocking
        trylock acquisitions and skipping contended zones.

    3.  Avoids blocking on timekeeping seqcounts and hrtimer_cpu_base locks
        in sysrq_timer_list_show() and print_active_timers() during NMI,
        Oops, or panic contexts.

    4.  Adds printk console flush suppression across nbcon, legacy
        consoles, and klogd, alongside ring buffer tail pinning so
        pre-crash records can still be recycled while protecting the
        initial crash output from being overwritten by subsequent telemetry
        or ftrace dumps.

    5.  Introduces crash_kexec_in_memory_sys_info and invokes
        panic_in_memory_sys_info() from __crash_kexec() prior to
        machine_crash_shutdown() when crash_kexec_post_notifiers is not
        enabled. It also marks the start of crash diagnostics across
        bust_spinlocks(), __report_bug(), oops_enter(), and vpanic(), and
        reorders __sys_info() so compact, high-signal dumps (SYS_INFO_MEM,
        SYS_INFO_LOCKS, SYS_INFO_TIMERS, SYS_INFO_BLOCKED_TASKS) execute
        before high-volume dumps (SYS_INFO_ALL_BT, SYS_INFO_TASKS,
        SYS_INFO_FTRACE).

Changes since v1:

    - Split the single patch into a 5-patch series separating prerequisite
      panic all-CPU backtrace fixes, crash-time lock safety in show_mem()
      and timer_list, printk/tracing infrastructure, and the crash_kexec
      integration

    - Moved in-memory sys_info() collection from vpanic() into
      __crash_kexec() via panic_in_memory_sys_info() with local interrupts
      disabled so all crash_kexec() entry points (including direct Oops/die
      paths prior to vpanic()) are covered, and kept console flush
      suppression and tail pinning active through machine_crash_shutdown()
      and machine_kexec() unless machine_kexec() returns

    - Replaced unconditional ring buffer tail freezing with crash-start
      sequence pinning (printk_mark_panic_start(),
      printk_clear_panic_start(), and printk_is_tail_full()), allowing
      pre-crash records to be recycled while preserving the initial
      Oops/panic trace, finalising the last record when full, and stopping
      ftrace_dump_one() before consuming unprinted trace entries

    - Extended printk_suppress_console_flush() to cover nbcon atomic
      flushes and kthreads, legacy console_flush_one_record() and kthreads,
      and wake_up_klogd(), and dropped unnecessary EXPORT_SYMBOL_GPL
      exports

    - Added crash-context trylock and NMI-safe timekeeping handling in
      show_free_areas(), sysrq_timer_list_show(), and
      print_active_timers(), and fixed duplicate/dropped all-CPU backtraces
      when SYS_INFO_ALL_BT is used during panic

 - Link to v1: https://lore.kernel.org/lkml/20260916201546.661384-1-atomlin@atomlin.com/

Aaron Tomlin (5):
  panic, sys_info: Fix duplicate and dropped all-CPU backtraces in panic
  mm/show_mem: Avoid blocking on zone->lock during Oops or panic
  timers: Avoid blocking on hrtimer_cpu_base lock during Oops or panic
  printk, tracing: Support console flush suppression and tail pinning
  panic, crash_core, sys_info: Introduce crash_kexec_in_memory_sys_info

 .../admin-guide/kernel-parameters.txt         |  11 ++
 include/linux/panic.h                         |   2 +
 include/linux/printk.h                        |  27 +++
 include/linux/sched.h                         |   3 +
 include/linux/sys_info.h                      |   4 +
 kernel/crash_core.c                           |   6 +
 kernel/exit.c                                 |  10 ++
 kernel/fork.c                                 |   3 +
 kernel/panic.c                                |  87 ++++++++-
 kernel/printk/internal.h                      |   4 +
 kernel/printk/nbcon.c                         |  11 +-
 kernel/printk/printk.c                        |  29 ++-
 kernel/printk/printk_ringbuffer.c             | 168 +++++++++++++++++-
 kernel/time/timer_list.c                      |  13 +-
 kernel/trace/trace.c                          |  20 ++-
 lib/bug.c                                     |  18 +-
 lib/bust_spinlocks.c                          |  22 ++-
 lib/sys_info.c                                |  33 ++--
 mm/show_mem.c                                 |  15 +-
 19 files changed, 447 insertions(+), 39 deletions(-)

-- 
2.55.0

[RFC PATCH v2 1/5] panic, sys_info: Fix duplicate and dropped all-CPU backtraces in panic

[Aaron Tomlin]

When sys_info() was factored out of kernel/panic.c by
commit b76e89e50fc3 ("panic: generalize panic_print's function to show
sys info"), __sys_info() gained an unconditional call to
trigger_all_cpu_backtrace() when SYS_INFO_ALL_BT is set, while
panic_other_cpus_shutdown() retained its own call to
panic_trigger_all_cpu_backtrace().

This causes two problems during a kernel panic:

    1.  vpanic() invokes panic_trigger_all_cpu_backtrace() inside
        panic_other_cpus_shutdown() before stopping secondary CPUs, and
        then invokes trigger_all_cpu_backtrace() a second time inside
        sys_info(panic_print) after secondary CPUs have already been
        shut down, stalling for up to 10 seconds waiting for offline
        CPUs to respond to NMIs. Furthermore, if SYS_INFO_ALL_BT is
        configured via the kernel.kernel_sys_info sysctl rather than
        panic_print, panic_other_cpus_shutdown() misses it prior to CPU
        shutdown.

    2.  If sys_info(SYS_INFO_ALL_BT) is invoked on the panicking CPU
        prior to panic_other_cpus_shutdown(), calling
        trigger_all_cpu_backtrace() directly bypasses
        panic_trigger_all_cpu_backtrace() and leaves
        panic_triggering_all_cpu_backtrace unset. Because panic_cpu is
        already active, vprintk_emit() silently drops all NMI backtrace
        messages originating from non-panic CPUs.

Export panic_trigger_all_cpu_backtrace() with a panic_on_this_cpu() and
one-shot guard so all-CPU backtraces are only emitted once before
secondary CPUs are stopped, consult sys_info_get_mask(panic_print) in
panic_other_cpus_shutdown(), record prior Oops stack dumps in
panic_try_start(), and route SYS_INFO_ALL_BT in __sys_info()
through panic_trigger_all_cpu_backtrace() on the panicking CPU.

Fixes: b76e89e50fc3 ("panic: generalize panic_print's function to show sys info")
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 include/linux/panic.h    |  1 +
 include/linux/sys_info.h |  1 +
 kernel/panic.c           | 31 +++++++++++++++++++++++++++----
 lib/sys_info.c           | 17 ++++++++++++++---
 4 files changed, 43 insertions(+), 7 deletions(-)

diff --git a/include/linux/panic.h b/include/linux/panic.h
index 17e61b61c45f..3b56f873f1e2 100644
--- a/include/linux/panic.h
+++ b/include/linux/panic.h
@@ -57,6 +57,7 @@ void panic_reset(void);
 bool panic_in_progress(void);
 bool panic_on_this_cpu(void);
 bool panic_on_other_cpu(void);
+void panic_trigger_all_cpu_backtrace(void);
 
 /*
  * Only to be used by arch init code. If the user over-wrote the default
diff --git a/include/linux/sys_info.h b/include/linux/sys_info.h
index a5bc3ea3d44b..e7699b82aa49 100644
--- a/include/linux/sys_info.h
+++ b/include/linux/sys_info.h
@@ -18,6 +18,7 @@
 #define SYS_INFO_BLOCKED_TASKS		0x00000080
 
 void sys_info(unsigned long si_mask);
+unsigned long sys_info_get_mask(unsigned long si_mask);
 unsigned long sys_info_parse_param(char *str);
 
 #ifdef CONFIG_SYSCTL
diff --git a/kernel/panic.c b/kernel/panic.c
index 574ee5a12f92..4cb26914b2af 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -70,6 +70,7 @@ static bool panic_console_replay;
 
 bool panic_triggering_all_cpu_backtrace;
 static bool panic_this_cpu_backtrace_printed;
+static bool panic_all_cpu_backtrace_printed;
 
 int panic_timeout = CONFIG_PANIC_TIMEOUT;
 EXPORT_SYMBOL_GPL(panic_timeout);
@@ -451,13 +452,21 @@ bool panic_try_start(void)
 	old_cpu = PANIC_CPU_INVALID;
 	this_cpu = raw_smp_processor_id();
 
-	return atomic_try_cmpxchg(&panic_cpu, &old_cpu, this_cpu);
+	if (!atomic_try_cmpxchg(&panic_cpu, &old_cpu, this_cpu))
+		return false;
+
+	if (atomic_read(&panic_redirect_cpu) == PANIC_CPU_INVALID &&
+	    (test_taint(TAINT_DIE) || oops_in_progress))
+		panic_this_cpu_backtrace_printed = true;
+
+	return true;
 }
 EXPORT_SYMBOL(panic_try_start);
 
 void panic_reset(void)
 {
-	atomic_set(&panic_cpu, PANIC_CPU_INVALID);
+	panic_this_cpu_backtrace_printed = false;
+	atomic_set_release(&panic_cpu, PANIC_CPU_INVALID);
 }
 EXPORT_SYMBOL(panic_reset);
 
@@ -538,8 +547,21 @@ void check_panic_on_warn(const char *origin)
 		      origin, limit);
 }
 
-static void panic_trigger_all_cpu_backtrace(void)
+/**
+ * panic_trigger_all_cpu_backtrace - trigger NMI backtraces on all CPUs once
+ *
+ * Temporarily permit non-panic CPUs to emit messages into the printk ring
+ * buffer while capturing all-CPU backtraces during a panic. Subsequent calls
+ * are ignored so backtraces are not re-triggered after secondary CPUs have
+ * been shut down.
+ */
+void panic_trigger_all_cpu_backtrace(void)
 {
+	if (!panic_on_this_cpu() || panic_all_cpu_backtrace_printed)
+		return;
+
+	panic_all_cpu_backtrace_printed = true;
+
 	/* Temporary allow non-panic CPUs to write their backtraces. */
 	panic_triggering_all_cpu_backtrace = true;
 
@@ -558,8 +580,9 @@ static void panic_trigger_all_cpu_backtrace(void)
  */
 static void panic_other_cpus_shutdown(bool crash_kexec)
 {
-	if (panic_print & SYS_INFO_ALL_BT)
+	if (sys_info_get_mask(panic_print) & SYS_INFO_ALL_BT)
 		panic_trigger_all_cpu_backtrace();
+	panic_all_cpu_backtrace_printed = true;
 
 	/*
 	 * Note that smp_send_stop() is the usual SMP shutdown function,
diff --git a/lib/sys_info.c b/lib/sys_info.c
index f32a06ec9ed4..496893eb776b 100644
--- a/lib/sys_info.c
+++ b/lib/sys_info.c
@@ -7,6 +7,7 @@
 #include <linux/kernel.h>
 #include <linux/ftrace.h>
 #include <linux/nmi.h>
+#include <linux/panic.h>
 #include <linux/sched/debug.h>
 #include <linux/string.h>
 #include <linux/sysctl.h>
@@ -153,14 +154,24 @@ static void __sys_info(unsigned long si_mask)
 	if (si_mask & SYS_INFO_FTRACE)
 		ftrace_dump(DUMP_ALL);
 
-	if (si_mask & SYS_INFO_ALL_BT)
-		trigger_all_cpu_backtrace();
+	if (si_mask & SYS_INFO_ALL_BT) {
+		if (panic_on_this_cpu())
+			panic_trigger_all_cpu_backtrace();
+		else if (!panic_in_progress())
+			trigger_all_cpu_backtrace();
+	}
 
 	if (si_mask & SYS_INFO_BLOCKED_TASKS)
 		show_state_filter(TASK_UNINTERRUPTIBLE);
 }
 
+unsigned long sys_info_get_mask(unsigned long si_mask)
+{
+	return (si_mask & ~SYS_INFO_PANIC_CONSOLE_REPLAY) ? :
+		READ_ONCE(kernel_si_mask);
+}
+
 void sys_info(unsigned long si_mask)
 {
-	__sys_info(si_mask ? : kernel_si_mask);
+	__sys_info(sys_info_get_mask(si_mask));
 }
-- 
2.55.0

[RFC PATCH v2 2/5] mm/show_mem: Avoid blocking on zone->lock during Oops or panic

[Aaron Tomlin]

When show_free_areas() iterates over populated memory zones, it
unconditionally acquires zone->lock via spin_lock_irqsave(). If
show_mem() is invoked from an Oops or panic context while the crashing
CPU or a wedged remote CPU already holds zone->lock, or from an NMI or
hardirq context under CONFIG_PREEMPT_RT where spinlock_t is a sleeping
rt_mutex, acquiring zone->lock can self-deadlock or fault before
post-mortem recovery (such as kdump) can proceed.

Check can_spin_trylock() and use spin_trylock_irqsave() when
!can_spin_trylock(), in_nmi(), oops_in_progress, or panic_in_progress()
is active, and emit the zone header after releasing zone->lock,
logging a contention notice and skipping the locked zone rather than
spinning indefinitely.

Fixes: d999bd9392de ("panic: add options to print system info when panic happens")
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 mm/show_mem.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/mm/show_mem.c b/mm/show_mem.c
index b938cbcd774a..77098077523a 100644
--- a/mm/show_mem.c
+++ b/mm/show_mem.c
@@ -367,10 +367,17 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
 			continue;
 		if (show_mem_node_skip(filter, zone_to_nid(zone), nodemask))
 			continue;
-		show_node(zone);
-		printk(KERN_CONT "%s: ", zone->name);
 
-		spin_lock_irqsave(&zone->lock, flags);
+		if (!can_spin_trylock() || in_nmi() ||
+		    oops_in_progress || panic_in_progress()) {
+			if (!can_spin_trylock() ||
+			    !spin_trylock_irqsave(&zone->lock, flags)) {
+				pr_warn("zone %s lock contended\n", zone->name);
+				continue;
+			}
+		} else {
+			spin_lock_irqsave(&zone->lock, flags);
+		}
 		for (order = 0; order < NR_PAGE_ORDERS; order++) {
 			struct free_area *area = &zone->free_area[order];
 			int type;
@@ -385,6 +392,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
 			}
 		}
 		spin_unlock_irqrestore(&zone->lock, flags);
+		show_node(zone);
+		printk(KERN_CONT "%s: ", zone->name);
 		for (order = 0; order < NR_PAGE_ORDERS; order++) {
 			printk(KERN_CONT "%lu*%lukB ",
 			       nr[order], K(1UL) << order);
-- 
2.55.0

[RFC PATCH v2 3/5] timers: Avoid blocking on hrtimer_cpu_base lock during Oops or panic

[Aaron Tomlin]

When sysrq_timer_list_show() dumps active high-resolution timers across
all online CPUs, it calls ktime_get() (which spins on tk_core.seq) and
print_active_timers() unconditionally acquires each CPU's
base->cpu_base->lock via raw_spin_lock_irqsave(). If invoked from an
Oops, NMI, or panic context while the crashing CPU or a wedged remote
CPU is inside a timekeeping write section or already holds
base->cpu_base->lock, sysrq_timer_list_show() will spin indefinitely
and prevent panic recovery or kexec from completing.

Use the NMI-safe ktime_get_mono_fast_ns() helper in
sysrq_timer_list_show() and raw_spin_trylock_irqsave() in
print_active_timers() when in_nmi(), oops_in_progress, or
panic_in_progress() is active, noting any contended lock and returning cleanly instead of
deadlocking.

Fixes: d999bd9392de ("panic: add options to print system info when panic happens")
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 kernel/time/timer_list.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/kernel/time/timer_list.c b/kernel/time/timer_list.c
index 0406bf4488e9..4973650c4ae5 100644
--- a/kernel/time/timer_list.c
+++ b/kernel/time/timer_list.c
@@ -68,7 +68,15 @@ static void print_active_timers(struct seq_file *m, struct hrtimer_clock_base *b
 
 	touch_nmi_watchdog();
 
-	raw_spin_lock_irqsave(&base->cpu_base->lock, flags);
+	if (in_nmi() || oops_in_progress || panic_in_progress()) {
+		if ((!IS_ENABLED(CONFIG_SMP) && in_nmi()) ||
+		    !raw_spin_trylock_irqsave(&base->cpu_base->lock, flags)) {
+			SEQ_printf(m, "  [cpu_base lock contended]\n");
+			return;
+		}
+	} else {
+		raw_spin_lock_irqsave(&base->cpu_base->lock, flags);
+	}
 
 	curr = timerqueue_linked_first(&base->active);
 	/*
@@ -262,7 +270,8 @@ static inline void timer_list_header(struct seq_file *m, ktime_t now)
 
 void sysrq_timer_list_show(void)
 {
-	ktime_t now = ktime_get();
+	ktime_t now = (in_nmi() || oops_in_progress || panic_in_progress()) ?
+		      ns_to_ktime(ktime_get_mono_fast_ns()) : ktime_get();
 	int cpu;
 
 	timer_list_header(NULL, now);
-- 
2.55.0

[RFC PATCH v2 4/5] printk, tracing: Support console flush suppression and tail pinning

[Aaron Tomlin]

When collecting high-volume diagnostic output into the printk ring
buffer during a kernel crash prior to kexec, synchronous console
emission can stall the panicking CPU for tens of seconds on slow serial
or IPMI Serial-over-LAN links, risking external hardware watchdog
resets. At the same time, uninhibited ring buffer rotation can wrap and
overwrite the initial Oops, faulting registers, and primary call trace.

Introduce two scoped mechanisms for in-memory crash telemetry
collection:

    1.  Console flush suppression

        Provide printk_suppress_console_flush(bool) to clear the console
        flush mask in printk_get_console_flush_type(), guard legacy
        record emission and kthread wakeups, and suppress
        wake_up_klogd() in vprintk_emit(). Records written via
        vprintk_store() are stored in the in-memory ring buffer while
        bypassing synchronous hardware console emission and kthread
        wakeups.

    2.  Ring buffer tail pinning

        Provide printk_mark_panic_start(), printk_clear_panic_start(),
        printk_freeze_tail(bool), and printk_is_tail_full(). Recording
        the next reserved sequence number at the start of a crash (or
        clamping to prb_first_valid_seq(prb) if the crash output itself
        already wrapped the buffer) allows data_make_reusable() and
        desc_push_tail() to continue recycling older pre-crash records
        while refusing to advance the tail past the pinned crash
        sequence number. Once the watermark is reached, prb_reserve(),
        prb_reserve_in_last(), and ftrace_dump_one() finalize the last
        record and stop reserving descriptors or consuming trace entries
        so the primary crash context and remaining trace buffer remain
        intact.

Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 include/linux/printk.h            |  27 +++++
 kernel/printk/internal.h          |   4 +
 kernel/printk/nbcon.c             |  11 +-
 kernel/printk/printk.c            |  29 +++++-
 kernel/printk/printk_ringbuffer.c | 168 ++++++++++++++++++++++++++++--
 kernel/trace/trace.c              |  20 +++-
 6 files changed, 244 insertions(+), 15 deletions(-)

diff --git a/include/linux/printk.h b/include/linux/printk.h
index f594c1266bfd..17806c515fe6 100644
--- a/include/linux/printk.h
+++ b/include/linux/printk.h
@@ -205,6 +205,11 @@ extern asmlinkage void dump_stack(void) __cold;
 void printk_trigger_flush(void);
 void console_try_replay_all(void);
 void printk_legacy_allow_panic_sync(void);
+void printk_suppress_console_flush(bool suppress);
+bool printk_mark_panic_start(void);
+void printk_clear_panic_start(void);
+void printk_freeze_tail(bool freeze);
+bool printk_is_tail_full(void);
 extern bool nbcon_device_try_acquire(struct console *con);
 extern void nbcon_device_release(struct console *con);
 void nbcon_atomic_flush_unsafe(void);
@@ -309,6 +314,28 @@ static inline void printk_legacy_allow_panic_sync(void)
 {
 }
 
+static inline void printk_suppress_console_flush(bool suppress)
+{
+}
+
+static inline bool printk_mark_panic_start(void)
+{
+	return false;
+}
+
+static inline void printk_clear_panic_start(void)
+{
+}
+
+static inline void printk_freeze_tail(bool freeze)
+{
+}
+
+static inline bool printk_is_tail_full(void)
+{
+	return false;
+}
+
 static inline bool nbcon_device_try_acquire(struct console *con)
 {
 	return false;
diff --git a/kernel/printk/internal.h b/kernel/printk/internal.h
index 85fbf1801cbe..72f0a5d154ec 100644
--- a/kernel/printk/internal.h
+++ b/kernel/printk/internal.h
@@ -186,6 +186,7 @@ struct console_flush_type {
 };
 
 extern bool console_irqwork_blocked;
+extern bool console_flush_suppressed;
 
 /*
  * Identify which console flushing methods should be used in the context of
@@ -195,6 +196,9 @@ static inline void printk_get_console_flush_type(struct console_flush_type *ft)
 {
 	memset(ft, 0, sizeof(*ft));
 
+	if (unlikely(READ_ONCE(console_flush_suppressed)))
+		return;
+
 	switch (nbcon_get_default_prio()) {
 	case NBCON_PRIO_NORMAL:
 		if (have_nbcon_console && !have_boot_console) {
diff --git a/kernel/printk/nbcon.c b/kernel/printk/nbcon.c
index 5f8afaca8866..36db1a8abe94 100644
--- a/kernel/printk/nbcon.c
+++ b/kernel/printk/nbcon.c
@@ -1194,7 +1194,8 @@ static bool nbcon_kthread_should_wakeup(struct console *con, struct nbcon_contex
 	 * ownership in the middle of a message.
 	 */
 	if (unlikely(atomic_read(&nbcon_cpu_emergency_cnt)) ||
-	    unlikely(panic_in_progress()))
+	    unlikely(panic_in_progress()) ||
+	    unlikely(READ_ONCE(console_flush_suppressed)))
 		return false;
 
 	cookie = console_srcu_read_lock();
@@ -1253,7 +1254,8 @@ static int nbcon_kthread_func(void *__console)
 		 * mode. See nbcon_kthread_should_wakeup() for more details.
 		 */
 		if (unlikely(atomic_read(&nbcon_cpu_emergency_cnt)) ||
-		    unlikely(panic_in_progress()))
+		    unlikely(panic_in_progress()) ||
+		    unlikely(READ_ONCE(console_flush_suppressed)))
 			goto wait_for_event;
 
 		backlog = false;
@@ -1642,7 +1644,7 @@ static void nbcon_atomic_flush_pending_con(struct console *con, u64 stop_seq)
 	 * is not available do it.
 	 */
 	printk_get_console_flush_type(&ft);
-	if (!ft.nbcon_offload &&
+	if (ft.nbcon_atomic && !ft.nbcon_offload &&
 	    prb_read_valid(prb, nbcon_seq_read(con), NULL)) {
 		stop_seq = prb_next_reserve_seq(prb);
 		goto again;
@@ -1659,6 +1661,9 @@ static void __nbcon_atomic_flush_pending(u64 stop_seq)
 	struct console *con;
 	int cookie;
 
+	if (unlikely(READ_ONCE(console_flush_suppressed)))
+		return;
+
 	cookie = console_srcu_read_lock();
 	for_each_console_srcu(con) {
 		short flags = console_srcu_read_flags(con);
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 9a205f94bfac..cf60bcd5acfb 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -467,6 +467,9 @@ bool legacy_allow_panic_sync;
 /* Avoid using irq_work when suspending. */
 bool console_irqwork_blocked;
 
+/* Suppress synchronous console flushing during in-memory panic dumps. */
+bool console_flush_suppressed;
+
 #ifdef CONFIG_PRINTK
 DECLARE_WAIT_QUEUE_HEAD(log_wait);
 static DECLARE_WAIT_QUEUE_HEAD(legacy_wait);
@@ -2386,6 +2389,23 @@ int vprintk_store(int facility, int level,
 	return ret;
 }
 
+/**
+ * printk_suppress_console_flush - suppress console flushing in printk()
+ * @suppress: true to suppress console emission and klogd wakeups, false to
+ *            restore normal flushing behaviour
+ *
+ * Context: Any context.
+ */
+void printk_suppress_console_flush(bool suppress)
+{
+	WRITE_ONCE(console_flush_suppressed, suppress);
+	if (!suppress) {
+		printk_trigger_flush();
+		if (!console_irqwork_blocked)
+			wake_up_klogd();
+	}
+}
+
 /*
  * This acts as a one-way switch to allow legacy consoles to print from
  * the printk() caller context on a panic CPU. It also attempts to flush
@@ -2482,7 +2502,8 @@ asmlinkage int vprintk_emit(int facility, int level,
 
 	if (ft.legacy_offload)
 		defer_console_output();
-	else if (!console_irqwork_blocked)
+	else if (!console_irqwork_blocked &&
+		 !unlikely(READ_ONCE(console_flush_suppressed)))
 		wake_up_klogd();
 
 	return printed_len;
@@ -3243,6 +3264,9 @@ static bool console_flush_one_record(bool do_cond_resched, u64 *next_seq, bool *
 
 	*try_again = false;
 
+	if (unlikely(READ_ONCE(console_flush_suppressed)))
+		return false;
+
 	printk_get_console_flush_type(&ft);
 
 	cookie = console_srcu_read_lock();
@@ -3665,6 +3689,9 @@ static bool legacy_kthread_should_wakeup(void)
 	if (kthread_should_stop())
 		return true;
 
+	if (unlikely(READ_ONCE(console_flush_suppressed)))
+		return false;
+
 	printk_get_console_flush_type(&ft);
 
 	cookie = console_srcu_read_lock();
diff --git a/kernel/printk/printk_ringbuffer.c b/kernel/printk/printk_ringbuffer.c
index d38e7c2f6887..60c8bd2f6490 100644
--- a/kernel/printk/printk_ringbuffer.c
+++ b/kernel/printk/printk_ringbuffer.c
@@ -377,6 +377,90 @@ static struct prb_data_block *to_block(struct prb_data_ring *data_ring,
 	return (void *)&data_ring->data[DATA_INDEX(data_ring, begin_lpos)];
 }
 
+static atomic64_t printk_panic_seq = ATOMIC64_INIT(U64_MAX);
+static bool printk_tail_frozen;
+static bool printk_tail_full;
+
+/**
+ * printk_mark_panic_start - record the ring buffer sequence at crash start
+ *
+ * Pin the next reserved sequence number at the beginning of a top-level Oops,
+ * fatal warning, or panic so subsequent tail freezing can recycle older
+ * pre-crash records without overwriting the primary crash output.
+ *
+ * Context: Any context.
+ * Return: true if this call newly pinned the crash start sequence from
+ *         U64_MAX, false if a crash start sequence was already active.
+ */
+bool printk_mark_panic_start(void)
+{
+	s64 panic_seq = atomic64_read(&printk_panic_seq);
+	u64 first_seq, next_seq;
+	bool pinned;
+
+	do {
+		first_seq = prb_first_valid_seq(prb);
+		if ((u64)panic_seq == U64_MAX) {
+			next_seq = prb_next_reserve_seq(prb);
+			pinned = true;
+		} else if ((u64)panic_seq < first_seq) {
+			next_seq = first_seq;
+			pinned = false;
+		} else {
+			return false;
+		}
+	} while (!atomic64_try_cmpxchg(&printk_panic_seq, &panic_seq, next_seq));
+
+	return pinned;
+}
+
+/**
+ * printk_clear_panic_start - clear a previously recorded crash start sequence
+ *
+ * Clear the pinned sequence number when a warning or Oops completes without
+ * triggering a kernel panic.
+ *
+ * Context: Any context.
+ */
+void printk_clear_panic_start(void)
+{
+	atomic64_set(&printk_panic_seq, U64_MAX);
+}
+
+/**
+ * printk_freeze_tail - prevent ring buffer tail from passing the crash start
+ * @freeze: true to stop tail advancement at the pinned crash sequence number,
+ *          false to restore normal ring buffer rotation
+ *
+ * Context: Any context.
+ */
+void printk_freeze_tail(bool freeze)
+{
+	if (freeze) {
+		printk_mark_panic_start();
+		WRITE_ONCE(printk_tail_full, false);
+		WRITE_ONCE(printk_tail_frozen, true);
+	} else {
+		WRITE_ONCE(printk_tail_frozen, false);
+		WRITE_ONCE(printk_tail_full, false);
+		atomic64_set(&printk_panic_seq, U64_MAX);
+	}
+}
+EXPORT_SYMBOL_GPL(printk_freeze_tail);
+
+/**
+ * printk_is_tail_full - check if the frozen ring buffer tail hit its limit
+ *
+ * Context: Any context.
+ * Return: true if tail freezing is active and the ring buffer has reached the
+ *         pinned crash sequence number, false otherwise.
+ */
+bool printk_is_tail_full(void)
+{
+	return unlikely(READ_ONCE(printk_tail_frozen) &&
+			READ_ONCE(printk_tail_full));
+}
+
 /*
  * Increase the data size to account for data block meta data plus any
  * padding so that the adjacent data block is aligned on the ID size.
@@ -591,6 +675,9 @@ static bool data_make_reusable(struct printk_ringbuffer *rb,
 	struct prb_desc desc;
 	struct prb_data_blk_lpos *blk_lpos = &desc.text_blk_lpos;
 	unsigned long id;
+	u64 seq;
+
+	*lpos_out = lpos_begin;
 
 	/* Loop until @lpos_begin has advanced to or beyond @lpos_end. */
 	while (need_more_space(data_ring, lpos_begin, lpos_end)) {
@@ -607,12 +694,20 @@ static bool data_make_reusable(struct printk_ringbuffer *rb,
 		id = blk->id; /* LMM(data_make_reusable:A) */
 
 		d_state = desc_read(desc_ring, id, &desc,
-				    NULL, NULL); /* LMM(data_make_reusable:B) */
+				    &seq, NULL); /* LMM(data_make_reusable:B) */
 
 		switch (d_state) {
 		case desc_miss:
+			return false;
 		case desc_reserved:
 		case desc_committed:
+			if (unlikely(READ_ONCE(printk_tail_frozen))) {
+				*lpos_out = lpos_begin;
+				/* Pairs with data_push_tail:D. */
+				smp_rmb();
+				if (lpos_begin == atomic_long_read(&data_ring->tail_lpos))
+					WRITE_ONCE(printk_tail_full, true);
+			}
 			return false;
 		case desc_finalized:
 			/*
@@ -621,6 +716,15 @@ static bool data_make_reusable(struct printk_ringbuffer *rb,
 			 */
 			if (blk_lpos->begin != lpos_begin)
 				return false;
+			if (unlikely(READ_ONCE(printk_tail_frozen)) &&
+			    seq >= (u64)atomic64_read(&printk_panic_seq)) {
+				*lpos_out = lpos_begin;
+				/* Pairs with data_push_tail:D. */
+				smp_rmb();
+				if (lpos_begin == atomic_long_read(&data_ring->tail_lpos))
+					WRITE_ONCE(printk_tail_full, true);
+				return false;
+			}
 			desc_make_reusable(desc_ring, id);
 			break;
 		case desc_reusable:
@@ -690,6 +794,12 @@ static bool data_push_tail(struct printk_ringbuffer *rb, unsigned long lpos)
 		 * data blocks before @lpos.
 		 */
 		if (!data_make_reusable(rb, tail_lpos, lpos, &next_lpos)) {
+			if (next_lpos != tail_lpos) {
+				if (atomic_long_try_cmpxchg(&data_ring->tail_lpos,
+							    &tail_lpos, next_lpos))
+					tail_lpos = next_lpos;
+				continue;
+			}
 			/*
 			 * 1. Guarantee the block ID loaded in
 			 *    data_make_reusable() is performed before
@@ -787,8 +897,9 @@ static bool desc_push_tail(struct printk_ringbuffer *rb,
 	struct prb_desc_ring *desc_ring = &rb->desc_ring;
 	enum desc_state d_state;
 	struct prb_desc desc;
+	u64 seq;
 
-	d_state = desc_read(desc_ring, tail_id, &desc, NULL, NULL);
+	d_state = desc_read(desc_ring, tail_id, &desc, &seq, NULL);
 
 	switch (d_state) {
 	case desc_miss:
@@ -813,6 +924,14 @@ static bool desc_push_tail(struct printk_ringbuffer *rb,
 	case desc_committed:
 		return false;
 	case desc_finalized:
+		if (unlikely(READ_ONCE(printk_tail_frozen)) &&
+		    seq >= (u64)atomic64_read(&printk_panic_seq)) {
+			/* Pairs with desc_push_tail:B. */
+			smp_rmb();
+			if (atomic_long_read(&desc_ring->tail_id) == tail_id)
+				WRITE_ONCE(printk_tail_full, true);
+			return false;
+		}
 		desc_make_reusable(desc_ring, tail_id);
 		break;
 	case desc_reusable:
@@ -881,8 +1000,11 @@ static bool desc_push_tail(struct printk_ringbuffer *rb,
 		 * not in an allowed tail state. But if the tail has since
 		 * been moved by another CPU, then it does not matter.
 		 */
-		if (atomic_long_read(&desc_ring->tail_id) == tail_id) /* LMM(desc_push_tail:D) */
+		if (atomic_long_read(&desc_ring->tail_id) == tail_id) { /* LMM(desc_push_tail:D) */
+			if (unlikely(READ_ONCE(printk_tail_frozen)))
+				WRITE_ONCE(printk_tail_full, true);
 			return false;
+		}
 	}
 
 	return true;
@@ -1428,6 +1550,9 @@ bool prb_reserve_in_last(struct prb_reserved_entry *e, struct printk_ringbuffer
 	struct prb_desc *d;
 	unsigned long id;
 
+	if (printk_is_tail_full())
+		goto fail_reopen;
+
 	local_irq_save(e->irqflags);
 
 	/* Transition the newest descriptor back to the reserved state. */
@@ -1505,8 +1630,11 @@ bool prb_reserve_in_last(struct prb_reserved_entry *e, struct printk_ringbuffer
 
 	return true;
 fail:
-	prb_commit(e);
-	/* prb_commit() re-enabled interrupts. */
+	if (printk_is_tail_full())
+		prb_final_commit(e);
+	else
+		prb_commit(e);
+	/* prb_commit() / prb_final_commit() re-enabled interrupts. */
 fail_reopen:
 	/* Make it clear to the caller that the re-reserve failed. */
 	memset(r, 0, sizeof(*r));
@@ -1664,6 +1792,12 @@ bool prb_reserve(struct prb_reserved_entry *e, struct printk_ringbuffer *rb,
 	if (!data_check_size(&rb->text_data_ring, r->text_buf_size))
 		goto fail;
 
+	if (printk_is_tail_full()) {
+		desc_make_final(rb, atomic_long_read(&desc_ring->head_id));
+		atomic_long_inc(&rb->fail);
+		goto fail;
+	}
+
 	/*
 	 * Descriptors in the reserved state act as blockers to all further
 	 * reservations once the desc_ring has fully wrapped. Disable
@@ -1672,7 +1806,24 @@ bool prb_reserve(struct prb_reserved_entry *e, struct printk_ringbuffer *rb,
 	 */
 	local_irq_save(e->irqflags);
 
+	if (unlikely(READ_ONCE(printk_tail_frozen)) && r->text_buf_size) {
+		struct prb_data_ring *data_ring = &rb->text_data_ring;
+		unsigned int size = to_blk_size(r->text_buf_size);
+		unsigned long begin_lpos = atomic_long_read(&data_ring->head_lpos);
+		unsigned long next_lpos = get_next_lpos(data_ring, begin_lpos, size);
+
+		if (!data_push_tail(rb, next_lpos - DATA_SIZE(data_ring))) {
+			if (printk_is_tail_full())
+				desc_make_final(rb, atomic_long_read(&desc_ring->head_id));
+			atomic_long_inc(&rb->fail);
+			local_irq_restore(e->irqflags);
+			goto fail;
+		}
+	}
+
 	if (!desc_reserve(rb, &id)) {
+		if (printk_is_tail_full())
+			desc_make_final(rb, atomic_long_read(&desc_ring->head_id));
 		/* Descriptor reservation failures are tracked. */
 		atomic_long_inc(&rb->fail);
 		local_irq_restore(e->irqflags);
@@ -1725,8 +1876,11 @@ bool prb_reserve(struct prb_reserved_entry *e, struct printk_ringbuffer *rb,
 	r->text_buf = data_alloc(rb, r->text_buf_size, &d->text_blk_lpos, id);
 	/* If text data allocation fails, a data-less record is committed. */
 	if (r->text_buf_size && !r->text_buf) {
-		prb_commit(e);
-		/* prb_commit() re-enabled interrupts. */
+		if (printk_is_tail_full())
+			prb_final_commit(e);
+		else
+			prb_commit(e);
+		/* prb_commit() / prb_final_commit() re-enabled interrupts. */
 		goto fail;
 	}
 
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index e8c4efa8107b..35a56f04e9b3 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -9519,6 +9519,9 @@ static void ftrace_dump_one(struct trace_array *tr, enum ftrace_dump_mode dump_m
 	else
 		iter.cpu_file = RING_BUFFER_ALL_CPUS;
 
+	if (printk_is_tail_full())
+		goto out_enable;
+
 	if (tr == &global_trace)
 		printk(KERN_TRACE "Dumping ftrace buffer:\n");
 	else
@@ -9537,10 +9540,13 @@ static void ftrace_dump_one(struct trace_array *tr, enum ftrace_dump_mode dump_m
 	 * and then release the locks again.
 	 */
 
-	while (!trace_empty(&iter)) {
+	while (!trace_empty(&iter) && !printk_is_tail_full()) {
 
-		if (!cnt)
+		if (!cnt) {
 			printk(KERN_TRACE "---------------------------------\n");
+			if (printk_is_tail_full())
+				break;
+		}
 
 		cnt++;
 
@@ -9551,19 +9557,25 @@ static void ftrace_dump_one(struct trace_array *tr, enum ftrace_dump_mode dump_m
 			int ret;
 
 			ret = print_trace_line(&iter);
+			trace_printk_seq(&iter.seq);
+			if (printk_is_tail_full())
+				break;
+
 			if (ret != TRACE_TYPE_NO_CONSUME)
 				trace_consume(&iter);
-
-			trace_printk_seq(&iter.seq);
 		}
 		touch_nmi_watchdog();
 	}
 
+	if (printk_is_tail_full())
+		goto out_enable;
+
 	if (!cnt)
 		printk(KERN_TRACE "   (ftrace buffer empty)\n");
 	else
 		printk(KERN_TRACE "---------------------------------\n");
 
+out_enable:
 	tr->trace_flags |= old_userobj;
 
 	tracer_tracing_enable(tr);
-- 
2.55.0

[RFC PATCH v2 5/5] panic, crash_core, sys_info: Introduce crash_kexec_in_memory_sys_info

[Aaron Tomlin]

When investigating kernel panics, capturing post-mortem diagnostic
telemetry (e.g. memory zone metrics, lock states, active timers, and
blocked tasks) is vital for root-cause analysis.

While crash_kexec_post_notifiers allows executing panic notifiers and
sys_info() before jumping to the kdump kernel, it is frequently avoided
in production environments due to the risk of hardware watchdog timeouts
induced by synchronous console emission or fragile third-party panic
notifier callbacks. Furthermore, if dump target storage is exhausted and
saving the full vmcore fails, utilities such as makedumpfile still
extract vmcore-dmesg.txt first, making the in-memory printk ring buffer
the sole surviving diagnostic artefact.

Introduce the crash_kexec_in_memory_sys_info boot parameter:

    1.  In-memory pre-kdump collection in __crash_kexec()

        When crash_kexec_in_memory_sys_info is enabled and a crash image
        is loaded, __crash_kexec() invokes panic_in_memory_sys_info()
        after saving crash registers and vmcoreinfo and prior to
        machine_crash_shutdown(). This captures sys_info() directly into
        the printk ring buffer in RAM with local interrupts disabled,
        console flushing suppressed, and the ring buffer tail pinned at
        the start of the crash output (recorded in bust_spinlocks(),
        __report_bug(), oops_enter(), warning paths, and vpanic(), and
        cleared on non-fatal warning completion or task exit in
        make_task_dead()). Because some architectures enter
        make_task_dead() with unbalanced bust_spinlocks(1) calls while
        others balance them in die(), track per-task nesting in
        task_struct->in_oops (using a u8 packed into an existing
        alignment hole, as normal traps and page faults nest at most
        twice and even recursive faults inside die() only reach single
        digits before recursion guards or stack exhaustion halt the CPU)
        so make_task_dead() unwinds only the dying task's unclosed
        increments. Similarly, track __warn_printk() nesting and pin
        ownership in task_struct->in_warn_printk and
        task_struct->in_warn_pinned across the __WARN_FLAGS() trap.
        Suppression and tail freezing remain active across
        machine_crash_shutdown() and machine_kexec(), only being
        released if machine_kexec() fails and returns.

    2.  Execution sequence reordering and defaults

        Reorder __sys_info() so compact, high-signal subsystems (memory,
        locks, timers, and blocked tasks) are collected first, leaving
        high-volume dumps (all-CPU backtraces, full task lists, and
        ftrace) for last. Default to SYS_INFO_IN_MEMORY_DEFAULT when
        neither panic_print nor kernel.kernel_sys_info is configured,
        and avoid duplicate sys_info() execution if machine_kexec()
        fails and returns to vpanic().

Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 .../admin-guide/kernel-parameters.txt         | 11 ++++
 include/linux/panic.h                         |  1 +
 include/linux/sched.h                         |  3 +
 include/linux/sys_info.h                      |  3 +
 kernel/crash_core.c                           |  6 ++
 kernel/exit.c                                 | 10 ++++
 kernel/fork.c                                 |  3 +
 kernel/panic.c                                | 56 ++++++++++++++++++-
 lib/bug.c                                     | 18 +++++-
 lib/bust_spinlocks.c                          | 22 +++++++-
 lib/sys_info.c                                | 20 +++----
 11 files changed, 139 insertions(+), 14 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0b3c140babd9..f21ff4a9254f 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -1041,6 +1041,17 @@ Kernel parameters
 			Default is enabled if CONFIG_HOTPLUG_PARALLEL=y. Otherwise
 			the parameter has no effect.
 
+	crash_kexec_in_memory_sys_info
+			[KNL] Collect diagnostic system telemetry (memory,
+			locks, timers, blocked tasks) into the
+			printk ring buffer entirely in RAM before jumping to
+			the kdump kernel. Unlike crash_kexec_post_notifiers,
+			this option bypasses slow hardware console emission and
+			avoids running external panic notifiers. The printk ring
+			buffer tail is pinned at the start of the initial Oops
+			or panic message during collection to prevent telemetry
+			output from overwriting the primary crash context.
+
 	crash_kexec_post_notifiers
 			Only jump to kdump kernel after running the panic
 			notifiers and dumping kmsg. This option increases
diff --git a/include/linux/panic.h b/include/linux/panic.h
index 3b56f873f1e2..9e3362934967 100644
--- a/include/linux/panic.h
+++ b/include/linux/panic.h
@@ -58,6 +58,7 @@ bool panic_in_progress(void);
 bool panic_on_this_cpu(void);
 bool panic_on_other_cpu(void);
 void panic_trigger_all_cpu_backtrace(void);
+void panic_in_memory_sys_info(void);
 
 /*
  * Only to be used by arch init code. If the user over-wrote the default
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 1c93da75226e..fa7a57aa50f8 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1071,6 +1071,9 @@ struct task_struct {
 	unsigned                        in_thrashing:1;
 #endif
 	unsigned			in_nf_duplicate:1;
+	unsigned			in_warn_pinned:1;
+	u8				in_oops;
+	u8				in_warn_printk;
 #ifdef CONFIG_PREEMPT_RT
 	struct netdev_xmit		net_xmit;
 #endif
diff --git a/include/linux/sys_info.h b/include/linux/sys_info.h
index e7699b82aa49..8dfe8544a7f2 100644
--- a/include/linux/sys_info.h
+++ b/include/linux/sys_info.h
@@ -17,6 +17,9 @@
 #define SYS_INFO_ALL_BT			0x00000040
 #define SYS_INFO_BLOCKED_TASKS		0x00000080
 
+#define SYS_INFO_IN_MEMORY_DEFAULT	(SYS_INFO_MEM | SYS_INFO_LOCKS | \
+					 SYS_INFO_TIMERS | SYS_INFO_BLOCKED_TASKS)
+
 void sys_info(unsigned long si_mask);
 unsigned long sys_info_get_mask(unsigned long si_mask);
 unsigned long sys_info_parse_param(char *str);
diff --git a/kernel/crash_core.c b/kernel/crash_core.c
index d0bd2d0cf899..b3bcb2c9040a 100644
--- a/kernel/crash_core.c
+++ b/kernel/crash_core.c
@@ -140,9 +140,15 @@ void __noclone __crash_kexec(struct pt_regs *regs)
 
 			crash_setup_regs(&fixed_regs, regs);
 			crash_save_vmcoreinfo();
+			if (!crash_kexec_post_notifiers)
+				panic_in_memory_sys_info();
 			machine_crash_shutdown(&fixed_regs);
 			crash_cma_clear_pending_dma();
 			machine_kexec(kexec_crash_image);
+			if (!crash_kexec_post_notifiers) {
+				printk_freeze_tail(false);
+				printk_suppress_console_flush(false);
+			}
 		}
 		kexec_unlock();
 	}
diff --git a/kernel/exit.c b/kernel/exit.c
index d5e7bc2af804..efe3c4677123 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -71,6 +71,8 @@
 #include <linux/unwind_deferred.h>
 #include <linux/uaccess.h>
 #include <linux/pidfs.h>
+#include <linux/panic.h>
+#include <linux/printk.h>
 
 #include <uapi/linux/wait.h>
 
@@ -1069,6 +1071,14 @@ void __noreturn make_task_dead(int signr)
 	if (atomic_inc_return(&oops_count) >= limit && limit)
 		panic("Oopsed too often (kernel.oops_limit is %d)", limit);
 
+	while (tsk->in_oops)
+		bust_spinlocks(0);
+
+	if (!READ_ONCE(oops_in_progress) && !panic_in_progress() &&
+	    ((tsk->flags & PF_EXITING) || !is_global_init(tsk) ||
+	     atomic_read(&tsk->signal->live) > 1))
+		printk_clear_panic_start();
+
 	/*
 	 * We're taking recursive faults here in make_task_dead. Safest is to just
 	 * leave this task alone and wait for reboot.
diff --git a/kernel/fork.c b/kernel/fork.c
index 8b108ed37152..95df3001a390 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -1009,6 +1009,9 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
 #ifdef CONFIG_X86_BUS_LOCK_DETECT
 	tsk->reported_split_lock = 0;
 #endif
+	tsk->in_warn_pinned = 0;
+	tsk->in_oops = 0;
+	tsk->in_warn_printk = 0;
 
 #ifdef CONFIG_SCHED_MM_CID
 	tsk->mm_cid.cid = MM_CID_UNSET;
diff --git a/kernel/panic.c b/kernel/panic.c
index 4cb26914b2af..77e5ef137d15 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -62,6 +62,7 @@ static int pause_on_oops;
 static int pause_on_oops_flag;
 static DEFINE_SPINLOCK(pause_on_oops_lock);
 bool crash_kexec_post_notifiers;
+static bool crash_kexec_in_memory_sys_info;
 int panic_on_warn __read_mostly;
 unsigned long panic_on_taint;
 bool panic_on_taint_nousertaint = false;
@@ -71,6 +72,7 @@ static bool panic_console_replay;
 bool panic_triggering_all_cpu_backtrace;
 static bool panic_this_cpu_backtrace_printed;
 static bool panic_all_cpu_backtrace_printed;
+static bool panic_sys_info_dumped;
 
 int panic_timeout = CONFIG_PANIC_TIMEOUT;
 EXPORT_SYMBOL_GPL(panic_timeout);
@@ -573,6 +575,31 @@ void panic_trigger_all_cpu_backtrace(void)
 	panic_triggering_all_cpu_backtrace = false;
 }
 
+/**
+ * panic_in_memory_sys_info - collect sys_info() into the printk ring buffer
+ *
+ * When crash_kexec_in_memory_sys_info is enabled, record system telemetry
+ * purely into the in-memory printk ring buffer before booting the capture
+ * kernel, suppressing synchronous console emission and pinning the ring
+ * buffer tail at the start of the primary crash output.
+ */
+void panic_in_memory_sys_info(void)
+{
+	unsigned long si_mask;
+
+	if (!crash_kexec_in_memory_sys_info || panic_sys_info_dumped)
+		return;
+
+	panic_sys_info_dumped = true;
+	si_mask = sys_info_get_mask(panic_print) ? : SYS_INFO_IN_MEMORY_DEFAULT;
+
+	local_irq_disable();
+	/* Populate log_buf in RAM without stalling on slow consoles. */
+	printk_suppress_console_flush(true);
+	printk_freeze_tail(true);
+	sys_info(si_mask);
+}
+
 /*
  * Helper that triggers the NMI backtrace (if set in panic_print)
  * and then performs the secondary CPUs shutdown - we cannot have
@@ -670,6 +697,7 @@ void vpanic(const char *fmt, va_list args)
 	if (len && buf[len - 1] == '\n')
 		buf[len - 1] = '\0';
 
+	printk_mark_panic_start();
 	pr_emerg("Kernel panic - not syncing: %s\n", buf);
 	/*
 	 * Avoid nested stack-dumping if a panic occurs during oops processing
@@ -713,7 +741,8 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	sys_info(panic_print);
+	if (!panic_sys_info_dumped)
+		sys_info(panic_print);
 
 	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);
 
@@ -1067,6 +1096,8 @@ void oops_enter(void)
 	/* can't trust the integrity of the kernel anymore: */
 	debug_locks_off();
 	do_oops_enter_exit();
+	if (!panic_in_progress())
+		printk_mark_panic_start();
 
 	if (sysctl_oops_all_cpu_backtrace)
 		trigger_all_cpu_backtrace();
@@ -1097,10 +1128,16 @@ struct warn_args {
 void __warn(const char *file, int line, void *caller, unsigned taint,
 	    struct pt_regs *regs, struct warn_args *args)
 {
+	bool clear_panic_start = false;
+
 	nbcon_cpu_emergency_enter();
 
 	disable_trace_on_warning();
 
+	if (!READ_ONCE(oops_in_progress) && !panic_in_progress() &&
+	    printk_mark_panic_start())
+		clear_panic_start = true;
+
 	if (file) {
 		pr_warn("WARNING: %s:%d at %pS, CPU#%d: %s/%d\n",
 			file, line, caller,
@@ -1137,6 +1174,9 @@ void __warn(const char *file, int line, void *caller, unsigned taint,
 	/* Just a warning, don't kill lockdep. */
 	add_taint(taint, LOCKDEP_STILL_OK);
 
+	if (clear_panic_start && !READ_ONCE(oops_in_progress) && !panic_in_progress())
+		printk_clear_panic_start();
+
 	nbcon_cpu_emergency_exit();
 }
 
@@ -1146,6 +1186,7 @@ void warn_slowpath_fmt(const char *file, int line, unsigned taint,
 		       const char *fmt, ...)
 {
 	bool rcu = warn_rcu_enter();
+	bool clear_panic_start = false;
 	struct warn_args args;
 
 	if (kunit_is_suppressed_warning(true)) {
@@ -1153,11 +1194,16 @@ void warn_slowpath_fmt(const char *file, int line, unsigned taint,
 		return;
 	}
 
+	if (!READ_ONCE(oops_in_progress) && !panic_in_progress())
+		clear_panic_start = printk_mark_panic_start();
+
 	pr_warn(CUT_HERE);
 
 	if (!fmt) {
 		__warn(file, line, __builtin_return_address(0), taint,
 		       NULL, NULL);
+		if (clear_panic_start && !READ_ONCE(oops_in_progress) && !panic_in_progress())
+			printk_clear_panic_start();
 		warn_rcu_exit(rcu);
 		return;
 	}
@@ -1166,6 +1212,8 @@ void warn_slowpath_fmt(const char *file, int line, unsigned taint,
 	va_start(args.args, fmt);
 	__warn(file, line, __builtin_return_address(0), taint, NULL, &args);
 	va_end(args.args);
+	if (clear_panic_start && !READ_ONCE(oops_in_progress) && !panic_in_progress())
+		printk_clear_panic_start();
 	warn_rcu_exit(rcu);
 }
 EXPORT_SYMBOL(warn_slowpath_fmt);
@@ -1180,6 +1228,11 @@ void __warn_printk(const char *fmt, ...)
 		return;
 	}
 
+	current->in_warn_printk++;
+	if (!READ_ONCE(oops_in_progress) && !panic_in_progress() &&
+	    printk_mark_panic_start())
+		current->in_warn_pinned = 1;
+
 	pr_warn(CUT_HERE);
 
 	va_start(args, fmt);
@@ -1240,6 +1293,7 @@ core_param(panic, panic_timeout, int, 0644);
 core_param(pause_on_oops, pause_on_oops, int, 0644);
 core_param(panic_on_warn, panic_on_warn, int, 0644);
 core_param(crash_kexec_post_notifiers, crash_kexec_post_notifiers, bool, 0644);
+core_param(crash_kexec_in_memory_sys_info, crash_kexec_in_memory_sys_info, bool, 0644);
 core_param(panic_console_replay, panic_console_replay, bool, 0644);
 
 static int panic_print_set(const char *val, const struct kernel_param *kp)
diff --git a/lib/bug.c b/lib/bug.c
index 7c1c2c27f58e..c6e02225ce95 100644
--- a/lib/bug.c
+++ b/lib/bug.c
@@ -197,6 +197,7 @@ void __warn_printf(const char *fmt, struct pt_regs *regs)
 
 static enum bug_trap_type __report_bug(struct bug_entry *bug, unsigned long bugaddr, struct pt_regs *regs)
 {
+	bool clear_panic_start = false;
 	bool warning, once, done, no_cut, has_args;
 	const char *file, *fmt;
 	unsigned int line;
@@ -219,12 +220,22 @@ static enum bug_trap_type __report_bug(struct bug_entry *bug, unsigned long buga
 	no_cut   = bug->flags & BUGFLAG_NO_CUT_HERE;
 	has_args = bug->flags & BUGFLAG_ARGS;
 
+	if (warning && no_cut && current->in_warn_printk &&
+	    --current->in_warn_printk == 0) {
+		clear_panic_start = current->in_warn_pinned;
+		current->in_warn_pinned = 0;
+	}
+
 	/*
 	 * Before the once logic so suppressed warnings do not consume
 	 * the single-fire budget of WARN_ON_ONCE().
 	 */
-	if (warning && kunit_is_suppressed_warning(true))
+	if (warning && kunit_is_suppressed_warning(true)) {
+		if (clear_panic_start && !READ_ONCE(oops_in_progress) &&
+		    !panic_in_progress())
+			printk_clear_panic_start();
 		return BUG_TRAP_TYPE_WARN;
+	}
 
 	disable_trace_on_warning();
 
@@ -245,6 +256,8 @@ static enum bug_trap_type __report_bug(struct bug_entry *bug, unsigned long buga
 	 * extra debugging message it writes before triggering the handler.
 	 */
 	if (!no_cut) {
+		if (!READ_ONCE(oops_in_progress) && !panic_in_progress())
+			clear_panic_start = printk_mark_panic_start();
 		pr_info(CUT_HERE);
 		__warn_printf(fmt, has_args ? regs : NULL);
 	}
@@ -253,6 +266,9 @@ static enum bug_trap_type __report_bug(struct bug_entry *bug, unsigned long buga
 		/* this is a WARN_ON rather than BUG/BUG_ON */
 		__warn(file, line, (void *)bugaddr, BUG_GET_TAINT(bug), regs,
 		       NULL);
+		if (clear_panic_start && !READ_ONCE(oops_in_progress) &&
+		    !panic_in_progress())
+			printk_clear_panic_start();
 		return BUG_TRAP_TYPE_WARN;
 	}
 
diff --git a/lib/bust_spinlocks.c b/lib/bust_spinlocks.c
index bfd53972a4d8..a8be7c129cda 100644
--- a/lib/bust_spinlocks.c
+++ b/lib/bust_spinlocks.c
@@ -9,8 +9,12 @@
  * and panic() information from reaching the user.
  */
 
+#include <linux/atomic.h>
 #include <linux/kernel.h>
+#include <linux/panic.h>
+#include <linux/preempt.h>
 #include <linux/printk.h>
+#include <linux/sched.h>
 #include <linux/spinlock.h>
 #include <linux/tty.h>
 #include <linux/wait.h>
@@ -19,11 +23,25 @@
 
 void bust_spinlocks(int yes)
 {
+	int old = READ_ONCE(oops_in_progress);
+
 	if (yes) {
-		++oops_in_progress;
+		if (in_task())
+			current->in_oops++;
+		do {
+		} while (!try_cmpxchg(&oops_in_progress, &old, old + 1));
+		if (!old && !panic_in_progress())
+			printk_mark_panic_start();
 	} else {
+		if (in_task() && current->in_oops)
+			current->in_oops--;
 		console_unblank();
-		if (--oops_in_progress == 0)
+		old = READ_ONCE(oops_in_progress);
+		do {
+			if (unlikely(old <= 0))
+				return;
+		} while (!try_cmpxchg(&oops_in_progress, &old, old - 1));
+		if (old == 1)
 			wake_up_klogd();
 	}
 }
diff --git a/lib/sys_info.c b/lib/sys_info.c
index 496893eb776b..5320369455d1 100644
--- a/lib/sys_info.c
+++ b/lib/sys_info.c
@@ -139,20 +139,17 @@ subsys_initcall(sys_info_sysctl_init);
 
 static void __sys_info(unsigned long si_mask)
 {
-	if (si_mask & SYS_INFO_TASKS)
-		show_state();
-
 	if (si_mask & SYS_INFO_MEM)
 		show_mem();
 
-	if (si_mask & SYS_INFO_TIMERS)
-		sysrq_timer_list_show();
-
 	if (si_mask & SYS_INFO_LOCKS)
 		debug_show_all_locks();
 
-	if (si_mask & SYS_INFO_FTRACE)
-		ftrace_dump(DUMP_ALL);
+	if (si_mask & SYS_INFO_TIMERS)
+		sysrq_timer_list_show();
+
+	if (si_mask & SYS_INFO_BLOCKED_TASKS)
+		show_state_filter(TASK_UNINTERRUPTIBLE);
 
 	if (si_mask & SYS_INFO_ALL_BT) {
 		if (panic_on_this_cpu())
@@ -161,8 +158,11 @@ static void __sys_info(unsigned long si_mask)
 			trigger_all_cpu_backtrace();
 	}
 
-	if (si_mask & SYS_INFO_BLOCKED_TASKS)
-		show_state_filter(TASK_UNINTERRUPTIBLE);
+	if (si_mask & SYS_INFO_TASKS)
+		show_state();
+
+	if (si_mask & SYS_INFO_FTRACE)
+		ftrace_dump(DUMP_ALL);
 }
 
 unsigned long sys_info_get_mask(unsigned long si_mask)
-- 
2.55.0

Re: [RFC PATCH v2 4/5] printk, tracing: Support console flush suppression and tail pinning

[Steven Rostedt]

On Mon,  2 Nov 2026 21:44:52 -0500
Aaron Tomlin <atomlin@atomlin.com> wrote:

> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> index e8c4efa8107b..35a56f04e9b3 100644
> --- a/kernel/trace/trace.c
> +++ b/kernel/trace/trace.c
> @@ -9519,6 +9519,9 @@ static void ftrace_dump_one(struct trace_array *tr, enum ftrace_dump_mode dump_m
>  	else
>  		iter.cpu_file = RING_BUFFER_ALL_CPUS;
>  
> +	if (printk_is_tail_full())
> +		goto out_enable;
> +
>  	if (tr == &global_trace)
>  		printk(KERN_TRACE "Dumping ftrace buffer:\n");
>  	else
> @@ -9537,10 +9540,13 @@ static void ftrace_dump_one(struct trace_array *tr, enum ftrace_dump_mode dump_m
>  	 * and then release the locks again.
>  	 */
>  
> -	while (!trace_empty(&iter)) {
> +	while (!trace_empty(&iter) && !printk_is_tail_full()) {

Wait? Is this to prevent overwriting the printk ring buffer? If so, this
is wrong here!

In your change log, you mention not losing the initial oops or registers.
But if someone is using ftrace_dump_on_oops, that's not what they are
after. They want to know what happened that lead to the crash. That output
is at the *end* of the trace.

By stopping the output of the ftrace buffer early, you just lost the
information you are looking for.

-- Steve


>  
> -		if (!cnt)
> +		if (!cnt) {
>  			printk(KERN_TRACE "---------------------------------\n");
> +			if (printk_is_tail_full())
> +				break;
> +		}
>  
>  		cnt++;
>  
> @@ -9551,19 +9557,25 @@ static void ftrace_dump_one(struct trace_array *tr, enum ftrace_dump_mode dump_m
>  			int ret;
>  
>  			ret = print_trace_line(&iter);
> +			trace_printk_seq(&iter.seq);
> +			if (printk_is_tail_full())
> +				break;
> +
>  			if (ret != TRACE_TYPE_NO_CONSUME)
>  				trace_consume(&iter);
> -
> -			trace_printk_seq(&iter.seq);
>  		}
>  		touch_nmi_watchdog();
>  	}
>  
> +	if (printk_is_tail_full())
> +		goto out_enable;
> +
>  	if (!cnt)
>  		printk(KERN_TRACE "   (ftrace buffer empty)\n");
>  	else
>  		printk(KERN_TRACE "---------------------------------\n");
>  
> +out_enable:
>  	tr->trace_flags |= old_userobj;
>  
>  	tracer_tracing_enable(tr);

Re: [RFC PATCH v2 4/5] printk, tracing: Support console flush suppression and tail pinning

[Aaron Tomlin]

On Tue, Nov 03, 2026 at 09:55:12AM -0500, Steven Rostedt wrote:
> On Mon,  2 Nov 2026 21:44:52 -0500
> Aaron Tomlin <atomlin@atomlin.com> wrote:
> 
> > diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> > index e8c4efa8107b..35a56f04e9b3 100644
> > --- a/kernel/trace/trace.c
> > +++ b/kernel/trace/trace.c
> > @@ -9519,6 +9519,9 @@ static void ftrace_dump_one(struct trace_array *tr, enum ftrace_dump_mode dump_m
> >  	else
> >  		iter.cpu_file = RING_BUFFER_ALL_CPUS;
> >  
> > +	if (printk_is_tail_full())
> > +		goto out_enable;
> > +
> >  	if (tr == &global_trace)
> >  		printk(KERN_TRACE "Dumping ftrace buffer:\n");
> >  	else
> > @@ -9537,10 +9540,13 @@ static void ftrace_dump_one(struct trace_array *tr, enum ftrace_dump_mode dump_m
> >  	 * and then release the locks again.
> >  	 */
> >  
> > -	while (!trace_empty(&iter)) {
> > +	while (!trace_empty(&iter) && !printk_is_tail_full()) {
> 
> Wait? Is this to prevent overwriting the printk ring buffer? If so, this
> is wrong here!

Oops!

> In your change log, you mention not losing the initial oops or registers.
> But if someone is using ftrace_dump_on_oops, that's not what they are
> after. They want to know what happened that lead to the crash. That output
> is at the *end* of the trace.
> 
> By stopping the output of the ftrace buffer early, you just lost the
> information you are looking for.

Agreed. Because ftrace_dump_one() iterates from oldest to newest, stopping
once the pinned printk ring buffer fills up retains only the oldest trace
entries and discards the end of the trace leading up to the crash.

If a user explicitly enables ftrace via panic_sys_info=ftrace alongside
crash_kexec_in_memory_sys_info, freezing the printk ring buffer tail
defeats the purpose of dumping the trace buffer for the exact reason you
described.

I will drop the kernel/trace/trace.c changes altogether and skip
printk_freeze_tail(true) in panic_in_memory_sys_info() when SYS_INFO_FTRACE
is requested. That keeps console flush suppression active so the dump still
completes at RAM speed prior to kexec, while allowing the printk ring
buffer to wrap so the end of the trace is preserved.

Thank you,
-- 
Aaron Tomlin