* [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand
@ 2026-10-24 23:06 Sam Crossley
2026-10-24 23:06 ` [PATCH RFC 1/2] printk: add per-console replay helper Sam Crossley
` (2 more replies)
0 siblings, 3 replies; 15+ messages in thread
From: Sam Crossley @ 2026-10-24 23:06 UTC (permalink / raw)
To: Thomas Gleixner, Breno Leitao, Petr Mladek, John Ogness,
Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski,
Jonathan Corbet, Paolo Abeni, Randy Dunlap, Sergey Senozhatsky,
Shuah Khan, Simon Horman, Steven Rostedt
Cc: linux-kernel, netdev, linux-doc, Sam Crossley
leitao and tglx had discussed this at LPC, sending RFC as suggested
by Breno Leitao.
Netconsole only forwards records while an enabled target and its
transport are available. A target enabled after boot, or one that was
unreachable during a network outage, leaves a gap in the remote log that
cannot currently be recovered.
This adds a write-only "replay" attribute to the netconsole configfs
subsystem that rewinds the netconsole consoles so the retained ring is
re-sent. Patch 1 adds the printk helper; patch 2 is its only caller.
The replay is best effort. console_try_replay() uses console_trylock(),
so unrelated console activity can silently skip the replay.
Testing: verified in a VM and on a physical host (the latter rebased on
a 7.1 kernel) that records emitted before any target was enabled are
re-sent on trigger.
---
Sam Crossley (2):
printk: add per-console replay helper
netconsole: add configfs trigger to replay printk buffer
Documentation/networking/netconsole.rst | 14 ++++++-
drivers/net/netconsole.c | 25 ++++++++++++
include/linux/printk.h | 6 +++
kernel/printk/printk.c | 67 ++++++++++++++++++++++++---------
4 files changed, 94 insertions(+), 18 deletions(-)
---
base-commit: 00513fc3d6c467b1ef3c101a2943cb92de25c49a
change-id: 20260926-netcons-replay-upstream-189eec0b4c79
Best regards,
--
Sam Crossley <samuelcrossley@gmail.com>
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH RFC 1/2] printk: add per-console replay helper
2026-10-24 23:06 [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand Sam Crossley
@ 2026-10-24 23:06 ` Sam Crossley
2026-10-24 23:06 ` [PATCH RFC 2/2] netconsole: add configfs trigger to replay printk buffer Sam Crossley
2026-10-26 9:34 ` [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand John Ogness
2 siblings, 0 replies; 15+ messages in thread
From: Sam Crossley @ 2026-10-24 23:06 UTC (permalink / raw)
To: Thomas Gleixner, Breno Leitao, Petr Mladek, John Ogness,
Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski,
Jonathan Corbet, Paolo Abeni, Randy Dunlap, Sergey Senozhatsky,
Shuah Khan, Simon Horman, Steven Rostedt
Cc: linux-kernel, netdev, linux-doc, Sam Crossley
console_try_replay_all() rewinds every registered console, which is too
broad for a caller that needs to resend records through one console.
Add console_try_replay() for that, sharing the locking and flush logic so
existing callers are unaffected. A NULL console is rejected, since the
shared helper reads it as "every console".
Like the existing all-console operation this is best effort: it does
nothing if the console lock cannot be acquired immediately. That lock is
global, so unrelated console activity can cause a replay to be skipped.
Records have not necessarily been printed by the time it returns either --
consoles served by a printer thread are only woken here.
Signed-off-by: Sam Crossley <samuelcrossley@gmail.com>
---
include/linux/printk.h | 6 +++++
kernel/printk/printk.c | 67 +++++++++++++++++++++++++++++++++++++-------------
2 files changed, 56 insertions(+), 17 deletions(-)
diff --git a/include/linux/printk.h b/include/linux/printk.h
index f594c1266bfd4..9cd06a67e7c83 100644
--- a/include/linux/printk.h
+++ b/include/linux/printk.h
@@ -203,6 +203,7 @@ void show_regs_print_info(const char *log_lvl);
extern asmlinkage void dump_stack_lvl(const char *log_lvl) __cold;
extern asmlinkage void dump_stack(void) __cold;
void printk_trigger_flush(void);
+void console_try_replay(struct console *con);
void console_try_replay_all(void);
void printk_legacy_allow_panic_sync(void);
extern bool nbcon_device_try_acquire(struct console *con);
@@ -301,6 +302,11 @@ static inline void dump_stack(void)
static inline void printk_trigger_flush(void)
{
}
+
+static inline void console_try_replay(struct console *con)
+{
+}
+
static inline void console_try_replay_all(void)
{
}
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 9a205f94bfac7..59e6df8a57de2 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -3494,13 +3494,14 @@ void console_unblank(void)
}
/*
- * Rewind all consoles to the oldest available record.
+ * Rewind a console, or all consoles if @target is NULL, to the oldest
+ * available record.
*
* IMPORTANT: The function is safe only when called under
* console_lock(). It is not enforced because
* it is used as a best effort in panic().
*/
-static void __console_rewind_all(void)
+static void __console_rewind(struct console *target)
{
struct console *c;
short flags;
@@ -3511,6 +3512,9 @@ static void __console_rewind_all(void)
cookie = console_srcu_read_lock();
for_each_console_srcu(c) {
+ if (target && c != target)
+ continue;
+
flags = console_srcu_read_flags(c);
if (flags & CON_NBCON) {
@@ -3557,7 +3561,7 @@ void console_flush_on_panic(enum con_flush_mode mode)
console_may_schedule = 0;
if (mode == CONSOLE_REPLAY_ALL)
- __console_rewind_all();
+ __console_rewind(NULL);
printk_get_console_flush_type(&ft);
if (ft.nbcon_atomic)
@@ -5047,6 +5051,48 @@ void kmsg_dump_rewind(struct kmsg_dump_iter *iter)
}
EXPORT_SYMBOL_GPL(kmsg_dump_rewind);
+static void __console_try_replay(struct console *con)
+{
+ struct console_flush_type ft;
+
+ printk_get_console_flush_type(&ft);
+ /*
+ * The console lock is global, so the replay is silently skipped when
+ * an unrelated console holds it. A trylock is what keeps this
+ * callable from any context.
+ */
+ if (!console_trylock())
+ return;
+
+ __console_rewind(con);
+ if (ft.nbcon_atomic)
+ nbcon_atomic_flush_pending();
+ if (ft.nbcon_offload)
+ nbcon_kthreads_wake();
+ if (ft.legacy_offload)
+ defer_console_output();
+ /* Consoles are flushed as part of console_unlock(). */
+ console_unlock();
+}
+
+/**
+ * console_try_replay - try to replay kernel log on a console
+ * @con: The console to replay the available records on
+ *
+ * Try to obtain the console lock and replay all available records in the
+ * printk buffer on @con. Does nothing if @con is not registered.
+ *
+ * Context: Any, except for NMI.
+ */
+void console_try_replay(struct console *con)
+{
+ if (WARN_ON_ONCE(!con))
+ return;
+
+ __console_try_replay(con);
+}
+EXPORT_SYMBOL_GPL(console_try_replay);
+
/**
* console_try_replay_all - try to replay kernel log on consoles
*
@@ -5058,20 +5104,7 @@ EXPORT_SYMBOL_GPL(kmsg_dump_rewind);
*/
void console_try_replay_all(void)
{
- struct console_flush_type ft;
-
- printk_get_console_flush_type(&ft);
- if (console_trylock()) {
- __console_rewind_all();
- if (ft.nbcon_atomic)
- nbcon_atomic_flush_pending();
- if (ft.nbcon_offload)
- nbcon_kthreads_wake();
- if (ft.legacy_offload)
- defer_console_output();
- /* Consoles are flushed as part of console_unlock(). */
- console_unlock();
- }
+ __console_try_replay(NULL);
}
#endif
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH RFC 2/2] netconsole: add configfs trigger to replay printk buffer
2026-10-24 23:06 [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand Sam Crossley
2026-10-24 23:06 ` [PATCH RFC 1/2] printk: add per-console replay helper Sam Crossley
@ 2026-10-24 23:06 ` Sam Crossley
2026-10-26 9:34 ` [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand John Ogness
2 siblings, 0 replies; 15+ messages in thread
From: Sam Crossley @ 2026-10-24 23:06 UTC (permalink / raw)
To: Thomas Gleixner, Breno Leitao, Petr Mladek, John Ogness,
Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski,
Jonathan Corbet, Paolo Abeni, Randy Dunlap, Sergey Senozhatsky,
Shuah Khan, Simon Horman, Steven Rostedt
Cc: linux-kernel, netdev, linux-doc, Sam Crossley
Netconsole only forwards records while an enabled target and its transport
are available, so enabling a target after startup, or recovering from an
outage, leaves a gap in the remote log.
Add a write-only replay attribute at /sys/kernel/config/netconsole.
Writing 1 rewinds the basic and extended netconsole consoles to the oldest
retained record and queues the records for transmission. Other registered
consoles are unaffected. Records keep their original printk timestamp and,
in extended format, their sequence number, so a receiver can deduplicate
what it already had.
Replay is best effort and asynchronous.
Note that "replay" is now an attribute in the netconsole configfs
directory, so it can no longer be used as a target name.
Signed-off-by: Sam Crossley <samuelcrossley@gmail.com>
---
Documentation/networking/netconsole.rst | 14 +++++++++++++-
drivers/net/netconsole.c | 25 +++++++++++++++++++++++++
2 files changed, 38 insertions(+), 1 deletion(-)
diff --git a/Documentation/networking/netconsole.rst b/Documentation/networking/netconsole.rst
index 4ab5d7b05cf10..2e489dd64ba42 100644
--- a/Documentation/networking/netconsole.rst
+++ b/Documentation/networking/netconsole.rst
@@ -111,7 +111,8 @@ netconsole module (or kernel, if netconsole is built-in).
Some examples follow (where configfs is mounted at the /sys/kernel/config
mountpoint).
-To add a remote logging target (target names can be arbitrary)::
+To add a remote logging target (target names can be arbitrary, except
+"replay", which is used by the attribute below)::
cd /sys/kernel/config/netconsole/
mkdir target1
@@ -125,6 +126,17 @@ To remove a target::
rmdir /sys/kernel/config/netconsole/othertarget/
+To replay the records retained in the kernel printk ring buffer to the enabled
+netconsole targets::
+
+ echo 1 > /sys/kernel/config/netconsole/replay
+
+As during normal operation, records above the current console loglevel are not
+sent. The replay is best effort and asynchronous: the write returns once the
+consoles have been rewound, before the records have been transmitted, and a
+console whose lock cannot be acquired is skipped. Only netconsole output is
+replayed; other registered consoles are unaffected.
+
The interface exposes these parameters of a netconsole target to userspace:
=============== ================================= ============
diff --git a/drivers/net/netconsole.c b/drivers/net/netconsole.c
index b358e5c367351..f527a5ee45662 100644
--- a/drivers/net/netconsole.c
+++ b/drivers/net/netconsole.c
@@ -1830,8 +1830,33 @@ static const struct configfs_group_operations netconsole_subsys_group_ops = {
.drop_item = drop_netconsole_target,
};
+static ssize_t replay_store(struct config_item *item, const char *buf,
+ size_t count)
+{
+ bool replay;
+ int ret;
+
+ ret = kstrtobool(buf, &replay);
+ if (ret)
+ return ret;
+ if (!replay)
+ return -EINVAL;
+
+ console_try_replay(&netconsole);
+ console_try_replay(&netconsole_ext);
+
+ return count;
+}
+CONFIGFS_ATTR_WO(, replay);
+
+static struct configfs_attribute *netconsole_subsys_attrs[] = {
+ &attr_replay,
+ NULL,
+};
+
static const struct config_item_type netconsole_subsys_type = {
.ct_group_ops = &netconsole_subsys_group_ops,
+ .ct_attrs = netconsole_subsys_attrs,
.ct_owner = THIS_MODULE,
};
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand
2026-10-24 23:06 [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand Sam Crossley
2026-10-24 23:06 ` [PATCH RFC 1/2] printk: add per-console replay helper Sam Crossley
2026-10-24 23:06 ` [PATCH RFC 2/2] netconsole: add configfs trigger to replay printk buffer Sam Crossley
@ 2026-10-26 9:34 ` John Ogness
2026-10-26 11:44 ` Breno Leitao
2 siblings, 1 reply; 15+ messages in thread
From: John Ogness @ 2026-10-26 9:34 UTC (permalink / raw)
To: Sam Crossley, Thomas Gleixner, Breno Leitao, Petr Mladek,
Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski,
Jonathan Corbet, Paolo Abeni, Randy Dunlap, Sergey Senozhatsky,
Shuah Khan, Simon Horman, Steven Rostedt
Cc: linux-kernel, netdev, linux-doc, Sam Crossley
On 2026-10-24, Sam Crossley <samuelcrossley@gmail.com> wrote:
> Netconsole only forwards records while an enabled target and its
> transport are available. A target enabled after boot, or one that was
> unreachable during a network outage, leaves a gap in the remote log that
> cannot currently be recovered.
This situation is no different than connecting a UART sometime after
boot, or having unplugged a serial cable temporarily during runtime.
> This adds a write-only "replay" attribute to the netconsole configfs
> subsystem that rewinds the netconsole consoles so the retained ring is
> re-sent.
I would prefer that we find a more general interface for this. One that
can apply to all consoles, not just netconsole.
We have a couple console features planned or in discussion: per-console
loglevels [0] and per-console unsafe dumping/ordering [1]. Both of these
are per-console, meaning they also involve the netconsole.
I think we need the netconsole to support a general console interface
rather than introducing netconsole-only knobs (particularly when such
features could be valid for any console).
In the per-console loglevel series Chris Down introduces [2]
/sys/class/console/ to sysfs. This is where we would put all the
upcoming per-console knobs. I would expect a replay trigger would fit
well there.
John Ogness
[0] https://lore.kernel.org/lkml/cover.1764272407.git.chris@chrisdown.name/
[1] https://lore.kernel.org/lkml/87bj8kkfwk.fsf@jogness.linutronix.de/
[2] https://lore.kernel.org/lkml/84b8d91693e56f6026271bc40553bcb6657ab24d.1764272407.git.chris@chrisdown.name/
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand
2026-10-26 9:34 ` [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand John Ogness
@ 2026-10-26 11:44 ` Breno Leitao
2026-10-26 13:09 ` John Ogness
2026-10-26 14:39 ` Chris Down
0 siblings, 2 replies; 15+ messages in thread
From: Breno Leitao @ 2026-10-26 11:44 UTC (permalink / raw)
To: John Ogness, chris
Cc: Sam Crossley, Thomas Gleixner, Petr Mladek, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Jonathan Corbet,
Paolo Abeni, Randy Dunlap, Sergey Senozhatsky, Shuah Khan,
Simon Horman, Steven Rostedt, linux-kernel, netdev, linux-doc
Hello John,
On Mon, Oct 26, 2026 at 10:40:14AM +0106, John Ogness wrote:
>
> In the per-console loglevel series Chris Down introduces [2]
> /sys/class/console/ to sysfs. This is where we would put all the
> upcoming per-console knobs. I would expect a replay trigger would fit
> well there.
That makes sense from a design perspective. Thanks for the direction.
So the next step would be to add a basic /sys/class/console/, similar to
[2], and put a `replay` entry in it.
That would be about three patches:
1. Create /sys/class/console/, similar to [2] but without any loglevel
handling. cdown can build on it when respinning the loglevel series.
2. Add the per-console replay helper, similar to [3].
3. Add the first entry (/sys/class/console/replay), which triggers the
helper from patch 2.
Is that what you have in mind?
Thanks,
--breno
> [2] https://lore.kernel.org/lkml/84b8d91693e56f6026271bc40553bcb6657ab24d.1764272407.git.chris@chrisdown.name/
[3] https://lore.kernel.org/all/20261024-netcons-replay-upstream-v1-1-9613f38496e5@gmail.com/
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand
2026-10-26 11:44 ` Breno Leitao
@ 2026-10-26 13:09 ` John Ogness
2026-10-26 14:58 ` Petr Mladek
2026-10-26 14:39 ` Chris Down
1 sibling, 1 reply; 15+ messages in thread
From: John Ogness @ 2026-10-26 13:09 UTC (permalink / raw)
To: Breno Leitao, chris
Cc: Sam Crossley, Thomas Gleixner, Petr Mladek, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Jonathan Corbet,
Paolo Abeni, Randy Dunlap, Sergey Senozhatsky, Shuah Khan,
Simon Horman, Steven Rostedt, linux-kernel, netdev, linux-doc
On 2026-10-26, Breno Leitao <leitao@debian.org> wrote:
>> In the per-console loglevel series Chris Down introduces [2]
>> /sys/class/console/ to sysfs. This is where we would put all the
>> upcoming per-console knobs. I would expect a replay trigger would fit
>> well there.
>
> That makes sense from a design perspective. Thanks for the direction.
>
> So the next step would be to add a basic /sys/class/console/, similar to
> [2], and put a `replay` entry in it.
>
>
> That would be about three patches:
>
> 1. Create /sys/class/console/, similar to [2] but without any loglevel
> handling. cdown can build on it when respinning the loglevel series.
> 2. Add the per-console replay helper, similar to [3].
> 3. Add the first entry (/sys/class/console/replay), which triggers the
> helper from patch 2.
>
> Is that what you have in mind?
Yes. But perhaps someone else wants to provide some feedback about this?
John
>> [2] https://lore.kernel.org/lkml/84b8d91693e56f6026271bc40553bcb6657ab24d.1764272407.git.chris@chrisdown.name/
>
> [3] https://lore.kernel.org/all/20261024-netcons-replay-upstream-v1-1-9613f38496e5@gmail.com/
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand
2026-10-26 11:44 ` Breno Leitao
2026-10-26 13:09 ` John Ogness
@ 2026-10-26 14:39 ` Chris Down
2026-10-26 15:32 ` Breno Leitao
1 sibling, 1 reply; 15+ messages in thread
From: Chris Down @ 2026-10-26 14:39 UTC (permalink / raw)
To: Breno Leitao
Cc: John Ogness, Sam Crossley, Thomas Gleixner, Petr Mladek,
Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski,
Jonathan Corbet, Paolo Abeni, Randy Dunlap, Sergey Senozhatsky,
Shuah Khan, Simon Horman, Steven Rostedt, linux-kernel, netdev,
linux-doc
Hi Breno,
Breno Leitao writes:
>So the next step would be to add a basic /sys/class/console/, similar to
>[2], and put a `replay` entry in it.
Feel free to incorporate an abridged version of my patch with my From: in your
series if you like and we can take whichever one lands first.
(That also reminds me, Petr, let me know when it's a good time for a resend.
I've been holding off since there's still quite a bit of activity on the other
cleanup patchset)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand
2026-10-26 13:09 ` John Ogness
@ 2026-10-26 14:58 ` Petr Mladek
2026-10-26 16:33 ` John Ogness
0 siblings, 1 reply; 15+ messages in thread
From: Petr Mladek @ 2026-10-26 14:58 UTC (permalink / raw)
To: John Ogness
Cc: Breno Leitao, chris, Sam Crossley, Thomas Gleixner, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Jonathan Corbet,
Paolo Abeni, Randy Dunlap, Sergey Senozhatsky, Shuah Khan,
Simon Horman, Steven Rostedt, linux-kernel, netdev, linux-doc
On Mon 2026-10-26 14:15:49, John Ogness wrote:
> On 2026-10-26, Breno Leitao <leitao@debian.org> wrote:
> >> In the per-console loglevel series Chris Down introduces [2]
> >> /sys/class/console/ to sysfs. This is where we would put all the
> >> upcoming per-console knobs. I would expect a replay trigger would fit
> >> well there.
> >
> > That makes sense from a design perspective. Thanks for the direction.
> >
> > So the next step would be to add a basic /sys/class/console/, similar to
> > [2], and put a `revplay` entry in it.
> >
> >
> > That would be about three patches:
> >
> > 1. Create /sys/class/console/, similar to [2] but without any loglevel
> > handling. cdown can build on it when respinning the loglevel series.
> > 2. Add the per-console replay helper, similar to [3].
> > 3. Add the first entry (/sys/class/console/replay), which triggers the
> > helper from patch 2.
> >
> > Is that what you have in mind?
Sounds good to me.
> Yes. But perhaps someone else wants to provide some feedback about this?
The "replay" approach looks like a big hammer. Many messages might get
duplicated, especially when the outage was small.
A better solution might be to do not increment con->nbcon_seq
when the transmission failed. And try it later again.
con->write_thread() currently does not return anything. I believe that
it is partly a historical limit. Two problems come to my mind:
1. There used to be a global "console_seq" counter which was common
for all console drivers. But we have a per-console counter since
the commit a699449bb13b70b8bd1 ("printk: refactor and rework
printing logic").
2. There has never been a try-later logic. But is should be easier
to add it now. We have the kthreads. And even the legacy loop
is able to break when there is not progress.
Best regards,
Petr
> >> [2] https://lore.kernel.org/lkml/84b8d91693e56f6026271bc40553bcb6657ab24d.1764272407.git.chris@chrisdown.name/gy
> >
> > [3] https://lore.kernel.org/all/20261024-netcons-replay-upstream-v1-1-9613f38496e5@gmail.com/
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand
2026-10-26 14:39 ` Chris Down
@ 2026-10-26 15:32 ` Breno Leitao
0 siblings, 0 replies; 15+ messages in thread
From: Breno Leitao @ 2026-10-26 15:32 UTC (permalink / raw)
To: Chris Down
Cc: John Ogness, Sam Crossley, Thomas Gleixner, Petr Mladek,
Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski,
Jonathan Corbet, Paolo Abeni, Randy Dunlap, Sergey Senozhatsky,
Shuah Khan, Simon Horman, Steven Rostedt, linux-kernel, netdev,
linux-doc
Hello Chris,
On Mon, Oct 26, 2026 at 10:39:34PM +0800, Chris Down wrote:
> Breno Leitao writes:
> > So the next step would be to add a basic /sys/class/console/, similar to
> > [2], and put a `replay` entry in it.
>
> Feel free to incorporate an abridged version of my patch with my From: in
> your series if you like and we can take whichever one lands first.
Thanks Chris.
Your patch does a bit more than the series needs: it also adds the
loglevel entries to sysfs. Would you be OK if I trimmed those out for
now, kept only the basic sysfs support, and preserved your authorship?
--breno
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand
2026-10-26 14:58 ` Petr Mladek
@ 2026-10-26 16:33 ` John Ogness
2026-10-26 17:49 ` Breno Leitao
2026-10-27 12:58 ` Petr Mladek
0 siblings, 2 replies; 15+ messages in thread
From: John Ogness @ 2026-10-26 16:33 UTC (permalink / raw)
To: Petr Mladek
Cc: Breno Leitao, chris, Sam Crossley, Thomas Gleixner, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Jonathan Corbet,
Paolo Abeni, Randy Dunlap, Sergey Senozhatsky, Shuah Khan,
Simon Horman, Steven Rostedt, linux-kernel, netdev, linux-doc
On 2026-10-26, Petr Mladek <pmladek@suse.com> wrote:
> The "replay" approach looks like a big hammer. Many messages might get
> duplicated, especially when the outage was small.
>
> A better solution might be to do not increment con->nbcon_seq
> when the transmission failed. And try it later again.
Note that there was a time during the rework that we used to allow the
console to return failure but decided it was too complicated [0].
The kthreads are setup to stay in their printing loop until the backlog
is clear. Having a nice=-20 task busy-loop would be bad.
> con->write_thread() currently does not return anything. I believe that
> it is partly a historical limit. Two problems come to my mind:
>
> 1. There used to be a global "console_seq" counter which was common
> for all console drivers. But we have a per-console counter since
> the commit a699449bb13b70b8bd1 ("printk: refactor and rework
> printing logic").
>
> 2. There has never been a try-later logic. But is should be easier
> to add it now. We have the kthreads. And even the legacy loop
> is able to break when there is not progress.
I suppose for only the write_thread() callback we could implement some
sort of throttled retry if the console driver reports failure. But this
is assuming console drivers can even detect if it successfully sent or
not. The netconsole certainly has no method to _know_ if the UDP packet
was received. And UARTs (asside from flow-control) also have no way to
confirm that the other side received the data.
Breno and Sam simply want the user to trigger a replay of the full
log. Essentially a sysrq-R but per-console. I think that is a reasonable
request and certainly much easier than attempting some sort of
transmission guarantee.
John
[0] https://lore.kernel.org/lkml/ZdTCZqhZww8_WgSU@alley
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand
2026-10-26 16:33 ` John Ogness
@ 2026-10-26 17:49 ` Breno Leitao
2026-10-27 13:47 ` Petr Mladek
2026-10-27 12:58 ` Petr Mladek
1 sibling, 1 reply; 15+ messages in thread
From: Breno Leitao @ 2026-10-26 17:49 UTC (permalink / raw)
To: John Ogness
Cc: Petr Mladek, chris, Sam Crossley, Thomas Gleixner, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Jonathan Corbet,
Paolo Abeni, Randy Dunlap, Sergey Senozhatsky, Shuah Khan,
Simon Horman, Steven Rostedt, linux-kernel, netdev, linux-doc
On Mon, Oct 26, 2026 at 05:39:54PM +0106, John Ogness wrote:
> On 2026-10-26, Petr Mladek <pmladek@suse.com> wrote:
> > The "replay" approach looks like a big hammer. Many messages might get
> > duplicated, especially when the outage was small.
> >
> > A better solution might be to do not increment con->nbcon_seq
> > when the transmission failed. And try it later again.
>
> Note that there was a time during the rework that we used to allow the
> console to return failure but decided it was too complicated [0].
>
> The kthreads are setup to stay in their printing loop until the backlog
> is clear. Having a nice=-20 task busy-loop would be bad.
>
> > con->write_thread() currently does not return anything. I believe that
> > it is partly a historical limit. Two problems come to my mind:
> >
> > 1. There used to be a global "console_seq" counter which was common
> > for all console drivers. But we have a per-console counter since
> > the commit a699449bb13b70b8bd1 ("printk: refactor and rework
> > printing logic").
> >
> > 2. There has never been a try-later logic. But is should be easier
> > to add it now. We have the kthreads. And even the legacy loop
> > is able to break when there is not progress.
>
> I suppose for only the write_thread() callback we could implement some
> sort of throttled retry if the console driver reports failure. But this
> is assuming console drivers can even detect if it successfully sent or
> not. The netconsole certainly has no method to _know_ if the UDP packet
> was received. And UARTs (asside from flow-control) also have no way to
> confirm that the other side received the data.
>
> Breno and Sam simply want the user to trigger a replay of the full
> log. Essentially a sysrq-R but per-console. I think that is a reasonable
> request and certainly much easier than attempting some sort of
> transmission guarantee.
A bit of context on the use case from our side.
In an ideal world netconsole is enabled from early boot through the
kernel command line, but in some case this is impossible
(linux as bootloaders - ipxe/sd-stubs/etc).
The other way to setup netconsole is to create a new target at runtime
through configfs. Netconsole then only sends messages from the time the
target is enabled.
For this second case, I would like to replay the whole kernel log on
netconsole, such as previous messages. Something might have happened
before the target was enabled, and we want visibility into that.
That said, the "per-console counter" might work as well, since it would
start as 0, thus, we know that netconsole target would be behind, and
try to replay? (not sure how yet)
That said, one way or another might work for our purposes ?!
Thanks for the proposal and discussions,
--breno
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand
2026-10-26 16:33 ` John Ogness
2026-10-26 17:49 ` Breno Leitao
@ 2026-10-27 12:58 ` Petr Mladek
1 sibling, 0 replies; 15+ messages in thread
From: Petr Mladek @ 2026-10-27 12:58 UTC (permalink / raw)
To: John Ogness
Cc: Breno Leitao, chris, Sam Crossley, Thomas Gleixner, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Jonathan Corbet,
Paolo Abeni, Randy Dunlap, Sergey Senozhatsky, Shuah Khan,
Simon Horman, Steven Rostedt, linux-kernel, netdev, linux-doc
On Mon 2026-10-26 17:39:54, John Ogness wrote:
> On 2026-10-26, Petr Mladek <pmladek@suse.com> wrote:
> > The "replay" approach looks like a big hammer. Many messages might get
> > duplicated, especially when the outage was small.
> >
> > A better solution might be to do not increment con->nbcon_seq
> > when the transmission failed. And try it later again.
>
> Note that there was a time during the rework that we used to allow the
> console to return failure but decided it was too complicated [0].
> [0] https://lore.kernel.org/lkml/ZdTCZqhZww8_WgSU@alley
It seems that the main problem was how to handle the nbcon ownership
when the callback returned an error. But I think that has been later
solved because we needed to re-acquire the ownership for the clean up.
Honestly, I am not completely sure what worms could jump from this
can. But I think that adding the error handling should be doable.
Well, I am not sure that it is worth the effort for this use-case.
> The kthreads are setup to stay in their printing loop until the backlog
> is clear. Having a nice=-20 task busy-loop would be bad.
My idea was to put the kthread into in interruptible sleep. It could
be woken again by a timer and by the next printk() call.
> > con->write_thread() currently does not return anything. I believe that
> > it is partly a historical limit. Two problems come to my mind:
> >
> > 1. There used to be a global "console_seq" counter which was common
> > for all console drivers. But we have a per-console counter since
> > the commit a699449bb13b70b8bd1 ("printk: refactor and rework
> > printing logic").
> >
> > 2. There has never been a try-later logic. But is should be easier
> > to add it now. We have the kthreads. And even the legacy loop
> > is able to break when there is not progress.
>
> I suppose for only the write_thread() callback we could implement some
> sort of throttled retry if the console driver reports failure. But this
> is assuming console drivers can even detect if it successfully sent or
> not. The netconsole certainly has no method to _know_ if the UDP packet
> was received. And UARTs (asside from flow-control) also have no way to
> confirm that the other side received the data.
Good point. We could detect problems only on the sender side
which might need not be enough.
> Breno and Sam simply want the user to trigger a replay of the full
> log. Essentially a sysrq-R but per-console. I think that is a reasonable
> request and certainly much easier than attempting some sort of
> transmission guarantee.
Yes. I am not against the interface. I just thought about a better
solution.
Best Regards,
Petr
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand
2026-10-26 17:49 ` Breno Leitao
@ 2026-10-27 13:47 ` Petr Mladek
2026-10-27 15:54 ` Breno Leitao
0 siblings, 1 reply; 15+ messages in thread
From: Petr Mladek @ 2026-10-27 13:47 UTC (permalink / raw)
To: Breno Leitao
Cc: John Ogness, chris, Sam Crossley, Thomas Gleixner, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Jonathan Corbet,
Paolo Abeni, Randy Dunlap, Sergey Senozhatsky, Shuah Khan,
Simon Horman, Steven Rostedt, linux-kernel, netdev, linux-doc
On Mon 2026-10-26 10:49:32, Breno Leitao wrote:
> A bit of context on the use case from our side.
>
> In an ideal world netconsole is enabled from early boot through the
> kernel command line, but in some case this is impossible
> (linux as bootloaders - ipxe/sd-stubs/etc).
>
> The other way to setup netconsole is to create a new target at runtime
> through configfs. Netconsole then only sends messages from the time the
> target is enabled.
>
> For this second case, I would like to replay the whole kernel log on
> netconsole, such as previous messages. Something might have happened
> before the target was enabled, and we want visibility into that.
Just to be sure that I understand it correctly.
Is the netconsole registered before it gets functional, please?
Is it possible to detect that the netconsole is not yet functional
and when it could start working?
I ask because there is the flag CON_PRINTBUFFER which should cause
re-playing all messages on the console during registration.
A solution might be to delay the console registration until
the netwok target is really working.
Another solution might be to temporary block/disable the console
until it gets operational. We already have console_suspend()/resume()
API. But we might need something new to distinguish it from
the system suspend/resume handling.
> That said, the "per-console counter" might work as well, since it would
> start as 0, thus, we know that netconsole target would be behind, and
> try to replay? (not sure how yet)
This is what the console_try_replay_all() does. The only difference
is that it resets the counter to the oldest available message,
see __console_rewind_all(). It is just a small optimization
when the ring buffer is already full and the oldest messages
are gone.
> That said, one way or another might work for our purposes ?!
Best Regards,
Petr
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand
2026-10-27 13:47 ` Petr Mladek
@ 2026-10-27 15:54 ` Breno Leitao
2026-10-27 17:45 ` Petr Mladek
0 siblings, 1 reply; 15+ messages in thread
From: Breno Leitao @ 2026-10-27 15:54 UTC (permalink / raw)
To: Petr Mladek
Cc: John Ogness, chris, Sam Crossley, Thomas Gleixner, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Jonathan Corbet,
Paolo Abeni, Randy Dunlap, Sergey Senozhatsky, Shuah Khan,
Simon Horman, Steven Rostedt, linux-kernel, netdev, linux-doc
On Tue, Oct 27, 2026 at 02:47:22PM +0100, Petr Mladek wrote:
> On Mon 2026-10-26 10:49:32, Breno Leitao wrote:
> > A bit of context on the use case from our side.
> >
> > In an ideal world netconsole is enabled from early boot through the
> > kernel command line, but in some case this is impossible
> > (linux as bootloaders - ipxe/sd-stubs/etc).
> >
> > The other way to setup netconsole is to create a new target at runtime
> > through configfs. Netconsole then only sends messages from the time the
> > target is enabled.
> >
> > For this second case, I would like to replay the whole kernel log on
> > netconsole, such as previous messages. Something might have happened
> > before the target was enabled, and we want visibility into that.
>
> Just to be sure that I understand it correctly.
>
> Is the netconsole registered before it gets functional, please?
Correct. The console is registered at module load time, in
init_netconsole(), before any target is created.
In other words, a single console is registered at module load, and the
netconsole targets are then handled under that same console.
> Is it possible to detect that the netconsole is not yet functional
> and when it could start working?
>
> I ask because there is the flag CON_PRINTBUFFER which should cause
> re-playing all messages on the console during registration.
>
> A solution might be to delay the console registration until
> the netwok target is really working.
Yes, we can register the console when the target is enabled. However, I
think the user will need to choose whether the console is registered
with CON_PRINTBUFFER or not, since in some cases they only want messages
from the moment the target is enabled.
So we would probably need one console per netconsole target?
> Another solution might be to temporary block/disable the console
> until it gets operational. We already have console_suspend()/resume()
> API. But we might need something new to distinguish it from
> the system suspend/resume handling.
In that case, what happens to the console while it is suspended?
> > That said, the "per-console counter" might work as well, since it would
> > start as 0, thus, we know that netconsole target would be behind, and
> > try to replay? (not sure how yet)
>
> This is what the console_try_replay_all() does. The only difference
> is that it resets the counter to the oldest available message,
> see __console_rewind_all(). It is just a small optimization
> when the ring buffer is already full and the oldest messages
> are gone.
Right, but that would apply to all consoles, while in our case we only
want to replay the log on a specific console (netconsole).
Thanks for the suggestions,
--breno
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand
2026-10-27 15:54 ` Breno Leitao
@ 2026-10-27 17:45 ` Petr Mladek
0 siblings, 0 replies; 15+ messages in thread
From: Petr Mladek @ 2026-10-27 17:45 UTC (permalink / raw)
To: Breno Leitao
Cc: John Ogness, chris, Sam Crossley, Thomas Gleixner, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Jonathan Corbet,
Paolo Abeni, Randy Dunlap, Sergey Senozhatsky, Shuah Khan,
Simon Horman, Steven Rostedt, linux-kernel, netdev, linux-doc
On Tue 2026-10-27 08:54:12, Breno Leitao wrote:
> On Tue, Oct 27, 2026 at 02:47:22PM +0100, Petr Mladek wrote:
> > On Mon 2026-10-26 10:49:32, Breno Leitao wrote:
> > > A bit of context on the use case from our side.
> > >
> > > In an ideal world netconsole is enabled from early boot through the
> > > kernel command line, but in some case this is impossible
> > > (linux as bootloaders - ipxe/sd-stubs/etc).
> > >
> > > The other way to setup netconsole is to create a new target at runtime
> > > through configfs. Netconsole then only sends messages from the time the
> > > target is enabled.
> > >
> > > For this second case, I would like to replay the whole kernel log on
> > > netconsole, such as previous messages. Something might have happened
> > > before the target was enabled, and we want visibility into that.
> >
> > Just to be sure that I understand it correctly.
> >
> > Is the netconsole registered before it gets functional, please?
>
> Correct. The console is registered at module load time, in
> init_netconsole(), before any target is created.
>
> In other words, a single console is registered at module load, and the
> netconsole targets are then handled under that same console.
I see.
> > Is it possible to detect that the netconsole is not yet functional
> > and when it could start working?
> >
> > I ask because there is the flag CON_PRINTBUFFER which should cause
> > re-playing all messages on the console during registration.
> >
> > A solution might be to delay the console registration until
> > the netwok target is really working.
>
> Yes, we can register the console when the target is enabled. However, I
> think the user will need to choose whether the console is registered
> with CON_PRINTBUFFER or not, since in some cases they only want messages
> from the moment the target is enabled.
It is more complicated then. IMHO, this patchset does not work
as expected. The 1st patch does:
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -3494,13 +3494,14 @@ void console_unblank(void)
}
/*
- * Rewind all consoles to the oldest available record.
+ * Rewind a console, or all consoles if @target is NULL, to the oldest
+ * available record.
*
* IMPORTANT: The function is safe only when called under
* console_lock(). It is not enforced because
* it is used as a best effort in panic().
*/
-static void __console_rewind_all(void)
+static void __console_rewind(struct console *target)
{
struct console *c;
short flags;
@@ -3511,6 +3512,9 @@ static void __console_rewind_all(void)
cookie = console_srcu_read_lock();
for_each_console_srcu(c) {
+ if (target && c != target)
+ continue;
+
flags = console_srcu_read_flags(c);
if (flags & CON_NBCON) {
Note that the filtering is done by struct console *target.
And it resets con->nbcon_seq for this structure.
But the "target" in netconsole_write() is struct netconsole_target *nt.
By other words, the messages will be replayed on all netconsole
targets associated with the given struct console. But you want
to replay it only on a particular netconsole_target.
> So we would probably need one console per netconsole target?
This might be the right solution. But it would require some
support in the register_console() and maybe in
__add_preferred_console() machinery.
The console registration code is a historical mess.
I am trying to clean this up a bit, see
https://lore.kernel.org/all/20260929120546.328616-1-pmladek@suse.com/
But it just a beginning. The code will be hairy even with this patchset.
The following problems come to my mind:
1. The information about preferred consoles is stored into
a static array console_cmdline[MAX_CMDLINECONSOLES].
It is currently limited to 8 entries. I am not sure
how much netconsole targets people use.
We might need to create one entry for each netconsole
target.
Alternative solution might be to implement con->match()
callback which would be able to share a single
console_cmdline entry for all netconsole targets.
Note that the ->match() callback was added to match
real console driver with a boot console driver.
But it should be usable to match a netconsole target
as well.
2. AFAIK, all console drivers are currently using a statically
defined struct console. But we might need to allocate
one for each netconsole target dynamically. It might
require revisiting/adding a code for handling the
lifetime of the structure.
But it should be doable.
> > Another solution might be to temporary block/disable the console
> > until it gets operational. We already have console_suspend()/resume()
> > API. But we might need something new to distinguish it from
> > the system suspend/resume handling.
>
> In that case, what happens to the console while it is suspended?
The suspended console gets stalled and the related con->nbcon_seq
stays where it is. The pending messages are flushed when the console
gets resumed.
> > > That said, the "per-console counter" might work as well, since it would
> > > start as 0, thus, we know that netconsole target would be behind, and
> > > try to replay? (not sure how yet)
> >
> > This is what the console_try_replay_all() does. The only difference
> > is that it resets the counter to the oldest available message,
> > see __console_rewind_all(). It is just a small optimization
> > when the ring buffer is already full and the oldest messages
> > are gone.
>
> Right, but that would apply to all consoles, while in our case we only
> want to replay the log on a specific console (netconsole).
As explained above. It is even more complicated because there
might be more netconsole_target pointers for the single
struct console netconsole.
I suggest to investigate the way with the custom ->match()
callback and dynamically allocated per-netconsole target
struct console. It would allow to have per-netconsole-target
con->nbcon_seq counter and keep the netconsole target details
in netconsole code.
You might want to wait for John's opinion before you spend
to much time on this. It is possible that I miss some
obvious problem.
Best Regards,
Petr
PS: Feel free to ask for more details. Just note that there is a public
holiday here tomorrow. I will not be able to react before Thursday.
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2026-10-27 17:45 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-24 23:06 [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand Sam Crossley
2026-10-24 23:06 ` [PATCH RFC 1/2] printk: add per-console replay helper Sam Crossley
2026-10-24 23:06 ` [PATCH RFC 2/2] netconsole: add configfs trigger to replay printk buffer Sam Crossley
2026-10-26 9:34 ` [PATCH RFC 0/2] netconsole: allow replaying the printk buffer on demand John Ogness
2026-10-26 11:44 ` Breno Leitao
2026-10-26 13:09 ` John Ogness
2026-10-26 14:58 ` Petr Mladek
2026-10-26 16:33 ` John Ogness
2026-10-26 17:49 ` Breno Leitao
2026-10-27 13:47 ` Petr Mladek
2026-10-27 15:54 ` Breno Leitao
2026-10-27 17:45 ` Petr Mladek
2026-10-27 12:58 ` Petr Mladek
2026-10-26 14:39 ` Chris Down
2026-10-26 15:32 ` Breno Leitao
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox