[PATCH net-next 0/4] (no cover subject)

[PATCH net-next 0/4] (no cover subject)

[Breno Leitao]

This patch series introduces a new configfs attribute that enables sending
messages directly through netconsole without going through the kernel's logging
infrastructure.

This feature allows users to send custom messages, alerts, or status updates
directly to netconsole receivers by writing to
/sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel
buffers, and sending msgs to the serial, which could be slow.

At Meta this is currently used in two cases right now (through printk by
now):

  a) When a new workload enters or leave the machine.
  b) From time to time, as a "ping" to make sure the netconsole/machine
  is alive.

The implementation reuses the existing message transmission functions
(send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended
message formats.

Regarding code organization, this version uses forward declarations for
send_msg_udp() and send_ext_msg_udp() functions rather than relocating them
within the file. While forward declarations do add a small amount of
redundancy, they avoid the larger churn that would result from moving entire
function definitions.

---
Breno Leitao (4):
      netconsole: extract message fragmentation into send_msg_udp()
      netconsole: Add configfs attribute for direct message sending
      selftests/netconsole: Switch to configfs send_msg interface
      Documentation: netconsole: Document send_msg configfs attribute

 Documentation/networking/netconsole.rst            | 40 +++++++++++++++
 drivers/net/netconsole.c                           | 59 ++++++++++++++++++----
 .../selftests/drivers/net/netcons_sysdata.sh       |  2 +-
 3 files changed, 91 insertions(+), 10 deletions(-)
---
base-commit: ab084f0b8d6d2ee4b1c6a28f39a2a7430bdfa7f0
change-id: 20251127-netconsole_send_msg-89813956dc23

Best regards,
--  
Breno Leitao <leitao@debian.org>

[PATCH net-next 1/4] netconsole: extract message fragmentation into send_msg_udp()

[Breno Leitao]

Extract the message fragmentation logic from write_msg() into a
dedicated send_msg_udp() function. This improves code readability
and prepares for future enhancements.

The new send_msg_udp() function handles splitting messages that
exceed MAX_PRINT_CHUNK into smaller fragments and sending them
sequentially. This function is placed before send_ext_msg_udp()
to maintain a logical ordering of related functions.

No functional changes - this is purely a refactoring commit.

Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Petr Mladek <pmladek@suse.com>
---
 drivers/net/netconsole.c | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)

diff --git a/drivers/net/netconsole.c b/drivers/net/netconsole.c
index 9cb4dfc242f5..dc3bd7c9b049 100644
--- a/drivers/net/netconsole.c
+++ b/drivers/net/netconsole.c
@@ -1725,12 +1725,24 @@ static void write_ext_msg(struct console *con, const char *msg,
 	spin_unlock_irqrestore(&target_list_lock, flags);
 }
 
+static void send_msg_udp(struct netconsole_target *nt, const char *msg,
+			 unsigned int len)
+{
+	const char *tmp = msg;
+	int frag, left = len;
+
+	while (left > 0) {
+		frag = min(left, MAX_PRINT_CHUNK);
+		send_udp(nt, tmp, frag);
+		tmp += frag;
+		left -= frag;
+	}
+}
+
 static void write_msg(struct console *con, const char *msg, unsigned int len)
 {
-	int frag, left;
 	unsigned long flags;
 	struct netconsole_target *nt;
-	const char *tmp;
 
 	if (oops_only && !oops_in_progress)
 		return;
@@ -1747,13 +1759,7 @@ static void write_msg(struct console *con, const char *msg, unsigned int len)
 			 * at least one target if we die inside here, instead
 			 * of unnecessarily keeping all targets in lock-step.
 			 */
-			tmp = msg;
-			for (left = len; left;) {
-				frag = min(left, MAX_PRINT_CHUNK);
-				send_udp(nt, tmp, frag);
-				tmp += frag;
-				left -= frag;
-			}
+			send_msg_udp(nt, msg, len);
 		}
 	}
 	spin_unlock_irqrestore(&target_list_lock, flags);

-- 
2.47.3

Re: [PATCH net-next 1/4] netconsole: extract message fragmentation into send_msg_udp()

[Simon Horman]

On Fri, Nov 28, 2025 at 06:20:46AM -0800, Breno Leitao wrote:
> Extract the message fragmentation logic from write_msg() into a
> dedicated send_msg_udp() function. This improves code readability
> and prepares for future enhancements.
> 
> The new send_msg_udp() function handles splitting messages that
> exceed MAX_PRINT_CHUNK into smaller fragments and sending them
> sequentially. This function is placed before send_ext_msg_udp()
> to maintain a logical ordering of related functions.
> 
> No functional changes - this is purely a refactoring commit.
> 
> Signed-off-by: Breno Leitao <leitao@debian.org>
> Reviewed-by: Petr Mladek <pmladek@suse.com>

Reviewed-by: Simon Horman <horms@kernel.org>

[PATCH net-next 2/4] netconsole: Add configfs attribute for direct message sending

[Breno Leitao]

Add a new write-only configfs attribute "send_msg" to netconsole targets
that allows sending arbitrary messages directly through netconsole.

This feature enables users to send custom messages through netconsole
without having to go through the kernel logging infrastructure. Messages
can be sent by simply writing to:
/sys/kernel/config/netconsole/<target>/send_msg

The implementation:
- Checks if the target is enabled before sending
- Verifies the network interface is running
- Handles both basic and extended message formats
- Fragments large messages when needed for basic targets
- Reuses existing send_msg_udp() and send_ext_msg_udp() functions

Unfortunately this patch has two forward declaration, which is not
ideal, but, moving send_msg_udp() functions earlier would cause too many
changes, and this could be done in an idependent patch.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 drivers/net/netconsole.c | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/drivers/net/netconsole.c b/drivers/net/netconsole.c
index dc3bd7c9b049..245ed2584bbb 100644
--- a/drivers/net/netconsole.c
+++ b/drivers/net/netconsole.c
@@ -1194,6 +1194,39 @@ static const struct config_item_type userdata_type = {
 	.ct_owner	= THIS_MODULE,
 };
 
+/* Forward declarations */
+static void send_ext_msg_udp(struct netconsole_target *nt, const char *msg,
+			     int msg_len);
+static void send_msg_udp(struct netconsole_target *nt, const char *msg,
+			 unsigned int len);
+
+static ssize_t send_msg_store(struct config_item *item, const char *buf,
+			      size_t count)
+{
+	struct netconsole_target *nt = to_target(item);
+	ssize_t ret = -EINVAL;
+	unsigned long flags;
+
+	mutex_lock(&dynamic_netconsole_mutex);
+	if (!nt->enabled)
+		goto out;
+
+	if (!netif_running(nt->np.dev))
+		goto out;
+
+	spin_lock_irqsave(&target_list_lock, flags);
+	if (nt->extended)
+		send_ext_msg_udp(nt, buf, count);
+	else
+		send_msg_udp(nt, buf, count);
+	spin_unlock_irqrestore(&target_list_lock, flags);
+
+	ret = count;
+out:
+	mutex_unlock(&dynamic_netconsole_mutex);
+
+	return ret;
+}
 CONFIGFS_ATTR(, enabled);
 CONFIGFS_ATTR(, extended);
 CONFIGFS_ATTR(, dev_name);
@@ -1205,6 +1238,7 @@ CONFIGFS_ATTR_RO(, local_mac);
 CONFIGFS_ATTR(, remote_mac);
 CONFIGFS_ATTR(, release);
 CONFIGFS_ATTR_RO(, transmit_errors);
+CONFIGFS_ATTR_WO(, send_msg);
 
 static struct configfs_attribute *netconsole_target_attrs[] = {
 	&attr_enabled,
@@ -1218,6 +1252,7 @@ static struct configfs_attribute *netconsole_target_attrs[] = {
 	&attr_local_mac,
 	&attr_remote_mac,
 	&attr_transmit_errors,
+	&attr_send_msg,
 	NULL,
 };
 

-- 
2.47.3

Re: [PATCH net-next 2/4] netconsole: Add configfs attribute for direct message sending

[Simon Horman]

On Fri, Nov 28, 2025 at 06:20:47AM -0800, Breno Leitao wrote:
> Add a new write-only configfs attribute "send_msg" to netconsole targets
> that allows sending arbitrary messages directly through netconsole.
> 
> This feature enables users to send custom messages through netconsole
> without having to go through the kernel logging infrastructure. Messages
> can be sent by simply writing to:
> /sys/kernel/config/netconsole/<target>/send_msg
> 
> The implementation:
> - Checks if the target is enabled before sending
> - Verifies the network interface is running
> - Handles both basic and extended message formats
> - Fragments large messages when needed for basic targets
> - Reuses existing send_msg_udp() and send_ext_msg_udp() functions
> 
> Unfortunately this patch has two forward declaration, which is not
> ideal, but, moving send_msg_udp() functions earlier would cause too many
> changes, and this could be done in an idependent patch.
> 
> Signed-off-by: Breno Leitao <leitao@debian.org>

Reviewed-by: Simon Horman <horms@kernel.org>

[PATCH net-next 3/4] selftests/netconsole: Switch to configfs send_msg interface

[Breno Leitao]

Update the netcons_sysdata test to use the configfs send_msg
attribute for message injection instead of /dev/kmsg. This
validates the new direct message sending functionality that
bypasses the kernel's printk infrastructure.

Only move this test to the new mechanism, given the traditional printk()
flow continues to be default path, and the one that should be mostly
tested.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 tools/testing/selftests/drivers/net/netcons_sysdata.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/drivers/net/netcons_sysdata.sh b/tools/testing/selftests/drivers/net/netcons_sysdata.sh
index baf69031089e..27df730dc8f3 100755
--- a/tools/testing/selftests/drivers/net/netcons_sysdata.sh
+++ b/tools/testing/selftests/drivers/net/netcons_sysdata.sh
@@ -195,7 +195,7 @@ function runtest {
 	# Wait for socat to start and listen to the port.
 	wait_local_port_listen "${NAMESPACE}" "${PORT}" udp
 	# Send the message
-	taskset -c "${CPU}" echo "${MSG}: ${TARGET}" > /dev/kmsg
+	taskset -c "${CPU}" echo "${MSG}: ${TARGET}" > "$NETCONS_PATH"/send_msg
 	# Wait until socat saves the file to disk
 	busywait "${BUSYWAIT_TIMEOUT}" test -s "${OUTPUT_FILE}"
 }

-- 
2.47.3

Re: [PATCH net-next 3/4] selftests/netconsole: Switch to configfs send_msg interface

[Simon Horman]

On Fri, Nov 28, 2025 at 06:20:48AM -0800, Breno Leitao wrote:
> Update the netcons_sysdata test to use the configfs send_msg
> attribute for message injection instead of /dev/kmsg. This
> validates the new direct message sending functionality that
> bypasses the kernel's printk infrastructure.
> 
> Only move this test to the new mechanism, given the traditional printk()
> flow continues to be default path, and the one that should be mostly
> tested.
> 
> Signed-off-by: Breno Leitao <leitao@debian.org>

Reviewed-by: Simon Horman <horms@kernel.org>

[PATCH net-next 4/4] Documentation: netconsole: Document send_msg configfs attribute

[Breno Leitao]

Add documentation for the new send_msg configfs attribute that allows
sending custom messages directly through netconsole targets.

The documentation covers:
- How to use the send_msg attribute
- Key features and requirements
- Use cases for direct message sending
- Example of periodic health check implementation

This feature enables userspace applications to inject custom messages
into the netconsole stream without going through the kernel's printk
infrastructure, which is useful for application monitoring, testing,
and debugging purposes.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 Documentation/networking/netconsole.rst | 40 +++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/Documentation/networking/netconsole.rst b/Documentation/networking/netconsole.rst
index 4ab5d7b05cf1..229d5fe9a3b3 100644
--- a/Documentation/networking/netconsole.rst
+++ b/Documentation/networking/netconsole.rst
@@ -139,6 +139,7 @@ The interface exposes these parameters of a netconsole target to userspace:
 	local_mac	Local interface's MAC address		(read-only)
 	remote_mac	Remote agent's MAC address		(read-write)
 	transmit_errors	Number of packet send errors		(read-only)
+	send_msg	Send custom messages directly		(write-only)
 	=============== =================================       ============
 
 The "enabled" attribute is also used to control whether the parameters of
@@ -158,6 +159,45 @@ You can also update the local interface dynamically. This is especially
 useful if you want to use interfaces that have newly come up (and may not
 have existed when netconsole was loaded / initialized).
 
+Direct Message Sending
+----------------------
+
+The `send_msg` attribute allows sending custom messages directly through a
+netconsole target without going through the kernel's printk infrastructure.
+This is a write-only attribute that can be used to send arbitrary text to
+the configured remote logging agent.
+
+To send a message directly::
+
+ echo "Custom status message" > /sys/kernel/config/netconsole/target1/send_msg
+
+Key features:
+
+* Messages can be sent only when the target is enabled
+* The network interface must be up and running
+* For extended targets, messages are sent with the extended header format
+* For non-extended targets, messages are fragmented if they exceed the
+  maximum chunk size
+* Messages bypass the kernel log buffer entirely
+
+This is useful for:
+
+* Sending application-level alerts or status updates
+* Injecting custom markers or delimiters into the log stream
+* Sending diagnostic information from userspace scripts
+* Testing netconsole connectivity without generating kernel messages
+
+Example use case - sending periodic health checks::
+
+ while true; do
+   echo "[$(date)] System health: OK" > /sys/kernel/config/netconsole/target1/send_msg
+   sleep 60
+ done
+
+.. note::
+   The `send_msg` attribute requires the target to be enabled. Unlike other
+   parameters, you do not need to disable the target to use this attribute.
+
 Netconsole targets defined at boot time (or module load time) with the
 `netconsole=` param are assigned the name `cmdline<index>`.  For example, the
 first target in the parameter is named `cmdline0`.  You can control and modify

-- 
2.47.3

Re: [PATCH net-next 4/4] Documentation: netconsole: Document send_msg configfs attribute

[Simon Horman]

On Fri, Nov 28, 2025 at 06:20:49AM -0800, Breno Leitao wrote:
> Add documentation for the new send_msg configfs attribute that allows
> sending custom messages directly through netconsole targets.
> 
> The documentation covers:
> - How to use the send_msg attribute
> - Key features and requirements
> - Use cases for direct message sending
> - Example of periodic health check implementation
> 
> This feature enables userspace applications to inject custom messages
> into the netconsole stream without going through the kernel's printk
> infrastructure, which is useful for application monitoring, testing,
> and debugging purposes.
> 
> Signed-off-by: Breno Leitao <leitao@debian.org>

Reviewed-by: Simon Horman <horms@kernel.org>

Re: [PATCH net-next 0/4] (no cover subject)

[Simon Horman]

On Fri, Nov 28, 2025 at 06:20:45AM -0800, Breno Leitao wrote:
> This patch series introduces a new configfs attribute that enables sending
> messages directly through netconsole without going through the kernel's logging
> infrastructure.

Minor nit:

Hi Breno,

The patchset looks good to me.
But the cover-letter (this email) has no subject.

Re: [PATCH net-next 0/4] (no cover subject)

[Jakub Kicinski]

On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:
> This patch series introduces a new configfs attribute that enables sending
> messages directly through netconsole without going through the kernel's logging
> infrastructure.
> 
> This feature allows users to send custom messages, alerts, or status updates
> directly to netconsole receivers by writing to
> /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel
> buffers, and sending msgs to the serial, which could be slow.
> 
> At Meta this is currently used in two cases right now (through printk by
> now):
> 
>   a) When a new workload enters or leave the machine.
>   b) From time to time, as a "ping" to make sure the netconsole/machine
>   is alive.
> 
> The implementation reuses the existing message transmission functions
> (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended
> message formats.
> 
> Regarding code organization, this version uses forward declarations for
> send_msg_udp() and send_ext_msg_udp() functions rather than relocating them
> within the file. While forward declarations do add a small amount of
> redundancy, they avoid the larger churn that would result from moving entire
> function definitions.

The two questions we need to address here are :
 - why is the message important in the off-host message stream but not
   important in local dmesg stream. You mention "serial, which could be
   slow" - we need more details here.
 - why do we need the kernel API, netcons is just a UDP message, which
   is easy enough to send from user space. A little bit more detail
   about the advantages would be good to have.
The 2nd point is trivial, the first one is what really gives me pause.
Why do we not care about the logs on host? If the serial is very slow
presumably it impacts a lot of things, certainly boot speed, so...
perhaps it should be configured to only log messages at a high level?

Re: [PATCH net-next 0/4] (no cover subject)

[Breno Leitao]

Hello Jakub,

On Mon, Dec 01, 2025 at 04:36:22PM -0800, Jakub Kicinski wrote:
> On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:
> > This patch series introduces a new configfs attribute that enables sending
> > messages directly through netconsole without going through the kernel's logging
> > infrastructure.
> > 
> > This feature allows users to send custom messages, alerts, or status updates
> > directly to netconsole receivers by writing to
> > /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel
> > buffers, and sending msgs to the serial, which could be slow.
> > 
> > At Meta this is currently used in two cases right now (through printk by
> > now):
> > 
> >   a) When a new workload enters or leave the machine.
> >   b) From time to time, as a "ping" to make sure the netconsole/machine
> >   is alive.
> > 
> > The implementation reuses the existing message transmission functions
> > (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended
> > message formats.
> > 
> > Regarding code organization, this version uses forward declarations for
> > send_msg_udp() and send_ext_msg_udp() functions rather than relocating them
> > within the file. While forward declarations do add a small amount of
> > redundancy, they avoid the larger churn that would result from moving entire
> > function definitions.
> 
> The two questions we need to address here are :
>  - why is the message important in the off-host message stream but not
>    important in local dmesg stream. You mention "serial, which could be
>    slow" - we need more details here.

Thanks for the questions, and I would like to share my view of the world. The
way I see and use netconsole at my company (Meta) is a "kernel message"
on steroids, where it provides more information about the system than
what is available in kernel log buffers (dmesg)

These netconsole messages already have extra data, which provides
information to each message, such as:

 * scheduler configuration (for sched_ext contenxt)
 * THP memory configuration
 * Job/workload running
 * CPU id
 * task->curr name
 * etc
 
So, netconsole already sends extra information today that is not visible
on kernel console (dmesg), and this has proved to be super useful, so
useful that 16 entries are not enough and Gustavo need to do a dynamic
allocation instead of limiting it to 16.

On top of that, printk() has a similar mechanism where extra data is not
printed to the console. printk buffers has a dictionary of structured
data attached to the message that is not printed to the screen, but,
sent through netconsole.

This feature (in this patchset) is just one step ahead, giving some more
power to netconsole, where extra information could be sent beyond what
is in dmesg.

>  - why do we need the kernel API, netcons is just a UDP message, which
>    is easy enough to send from user space. A little bit more detail
>    about the advantages would be good to have.

The primary advantage is leveraging the existing configured netconsole
infrastructure. At Meta, for example, we have a "continuous ping"
mechanism configured by our Configuration Management software that
simply runs 'echo "ping" > /dev/kmsg'.

A userspace solution would require deploying a binary to millons of
machines,  parsing /sys/kernel/configfs/netconsole/cmdline0/configs
and sends packets directly.

While certainly feasible, it's less convenient than using the
existing infrastructure (though I may just be looking for the easier
path here).

> The 2nd point is trivial, the first one is what really gives me pause.
> Why do we not care about the logs on host? If the serial is very slow
> presumably it impacts a lot of things, certainly boot speed, so...

This is spot-on - slow serial definitely impacts things like boot speed.

See my constant complains here, about slow boot

	https://lore.kernel.org/all/aGVn%2FSnOvwWewkOW@gmail.com/

And the something similar in reboot/kexec path:

	https://lore.kernel.org/all/sqwajvt7utnt463tzxgwu2yctyn5m6bjwrslsnupfexeml6hkd@v6sqmpbu3vvu/

> perhaps it should be configured to only log messages at a high level?

Chris is actually working on per-console log levels to solve exactly
this problem, so we could filter serial console messages while keeping
everything in other consoles (aka netconsole):

	https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/

That work has been in progress for years though, and I'm not sure
when/if it'll land upstream. But if it does, we'd be able to have
different log levels per console and then use your suggested approach.

Thanks for the review, and feel free to yell at me if I am missing the
point,
--breno

Re: [PATCH net-next 0/4] (no cover subject)

[Jakub Kicinski]

On Tue, 2 Dec 2025 02:18:44 -0800 Breno Leitao wrote:
> On Mon, Dec 01, 2025 at 04:36:22PM -0800, Jakub Kicinski wrote:
> > On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:  
> > > This patch series introduces a new configfs attribute that enables sending
> > > messages directly through netconsole without going through the kernel's logging
> > > infrastructure.
> > > 
> > > This feature allows users to send custom messages, alerts, or status updates
> > > directly to netconsole receivers by writing to
> > > /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel
> > > buffers, and sending msgs to the serial, which could be slow.
> > > 
> > > At Meta this is currently used in two cases right now (through printk by
> > > now):
> > > 
> > >   a) When a new workload enters or leave the machine.
> > >   b) From time to time, as a "ping" to make sure the netconsole/machine
> > >   is alive.
> > > 
> > > The implementation reuses the existing message transmission functions
> > > (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended
> > > message formats.
> > > 
> > > Regarding code organization, this version uses forward declarations for
> > > send_msg_udp() and send_ext_msg_udp() functions rather than relocating them
> > > within the file. While forward declarations do add a small amount of
> > > redundancy, they avoid the larger churn that would result from moving entire
> > > function definitions.  
> > 
> > The two questions we need to address here are :
> >  - why is the message important in the off-host message stream but not
> >    important in local dmesg stream. You mention "serial, which could be
> >    slow" - we need more details here.  
> 
> Thanks for the questions, and I would like to share my view of the world. The
> way I see and use netconsole at my company (Meta) is a "kernel message"
> on steroids, where it provides more information about the system than
> what is available in kernel log buffers (dmesg)
> 
> These netconsole messages already have extra data, which provides
> information to each message, such as:
> 
>  * scheduler configuration (for sched_ext contenxt)
>  * THP memory configuration
>  * Job/workload running
>  * CPU id
>  * task->curr name
>  * etc
>  
> So, netconsole already sends extra information today that is not visible
> on kernel console (dmesg), and this has proved to be super useful, so
> useful that 16 entries are not enough and Gustavo need to do a dynamic
> allocation instead of limiting it to 16.
> 
> On top of that, printk() has a similar mechanism where extra data is not
> printed to the console. printk buffers has a dictionary of structured
> data attached to the message that is not printed to the screen, but,
> sent through netconsole.
> 
> This feature (in this patchset) is just one step ahead, giving some more
> power to netconsole, where extra information could be sent beyond what
> is in dmesg.

Having extra metadata makes sense, since the interpretation happens in
a different environment. But here we're talking about having extra
messages, not extra metadata.

> >  - why do we need the kernel API, netcons is just a UDP message, which
> >    is easy enough to send from user space. A little bit more detail
> >    about the advantages would be good to have.  
> 
> The primary advantage is leveraging the existing configured netconsole
> infrastructure. At Meta, for example, we have a "continuous ping"
> mechanism configured by our Configuration Management software that
> simply runs 'echo "ping" > /dev/kmsg'.
> 
> A userspace solution would require deploying a binary to millons of
> machines,  parsing /sys/kernel/configfs/netconsole/cmdline0/configs
> and sends packets directly.
> 
> While certainly feasible, it's less convenient than using the
> existing infrastructure (though I may just be looking for the easier
> path here).

If this was your objective, instead of having a uAPI for sending
arbitrary message you should be adding some "keepalive" timer / empty
message sender... With the patches are posted you still need something
to run the echo.

> > The 2nd point is trivial, the first one is what really gives me pause.
> > Why do we not care about the logs on host? If the serial is very slow
> > presumably it impacts a lot of things, certainly boot speed, so...  
> 
> This is spot-on - slow serial definitely impacts things like boot speed.
> 
> See my constant complains here, about slow boot
> 
> 	https://lore.kernel.org/all/aGVn%2FSnOvwWewkOW@gmail.com/
> 
> And the something similar in reboot/kexec path:
> 
> 	https://lore.kernel.org/all/sqwajvt7utnt463tzxgwu2yctyn5m6bjwrslsnupfexeml6hkd@v6sqmpbu3vvu/
> 
> > perhaps it should be configured to only log messages at a high level?  
> 
> Chris is actually working on per-console log levels to solve exactly
> this problem, so we could filter serial console messages while keeping
> everything in other consoles (aka netconsole):
> 
> 	https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/

Excellent! Unless I'm missing more context Chris does seem to be
attacking the problem at a more suitable layer.

> That work has been in progress for years though, and I'm not sure
> when/if it'll land upstream. But if it does, we'd be able to have
> different log levels per console and then use your suggested approach.
> 
> Thanks for the review, and feel free to yell at me if I am missing the
> point,
> --breno
> 

Re: [PATCH net-next 0/4] (no cover subject)

[Petr Mladek]

On Tue 2025-12-02 10:24:42, Jakub Kicinski wrote:
> On Tue, 2 Dec 2025 02:18:44 -0800 Breno Leitao wrote:
> > On Mon, Dec 01, 2025 at 04:36:22PM -0800, Jakub Kicinski wrote:
> > > On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:  
> > > > This patch series introduces a new configfs attribute that enables sending
> > > > messages directly through netconsole without going through the kernel's logging
> > > > infrastructure.
> > > > 
> > > > This feature allows users to send custom messages, alerts, or status updates
> > > > directly to netconsole receivers by writing to
> > > > /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel
> > > > buffers, and sending msgs to the serial, which could be slow.
> > > > 
> > > > At Meta this is currently used in two cases right now (through printk by
> > > > now):
> > > > 
> > > >   a) When a new workload enters or leave the machine.
> > > >   b) From time to time, as a "ping" to make sure the netconsole/machine
> > > >   is alive.
> > > > 
> > > > The implementation reuses the existing message transmission functions
> > > > (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended
> > > > message formats.
> > > > 
> > This feature (in this patchset) is just one step ahead, giving some more
> > power to netconsole, where extra information could be sent beyond what
> > is in dmesg.
> 
> Having extra metadata makes sense, since the interpretation happens in
> a different environment. But here we're talking about having extra
> messages, not extra metadata.
> 
> > > The 2nd point is trivial, the first one is what really gives me pause.
> > > Why do we not care about the logs on host? If the serial is very slow
> > > presumably it impacts a lot of things, certainly boot speed, so...  
> > 
> > This is spot-on - slow serial definitely impacts things like boot speed.
> > 
> > See my constant complains here, about slow boot
> > 
> > 	https://lore.kernel.org/all/aGVn%2FSnOvwWewkOW@gmail.com/
> > 
> > And the something similar in reboot/kexec path:
> > 
> > 	https://lore.kernel.org/all/sqwajvt7utnt463tzxgwu2yctyn5m6bjwrslsnupfexeml6hkd@v6sqmpbu3vvu/
> > 
> > > perhaps it should be configured to only log messages at a high level?  
> > 
> > Chris is actually working on per-console log levels to solve exactly
> > this problem, so we could filter serial console messages while keeping
> > everything in other consoles (aka netconsole):
> > 
> > 	https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
> 
> Excellent! Unless I'm missing more context Chris does seem to be
> attacking the problem at a more suitable layer.

This would help to bypass slow serial consoles. But the extra messages
would still get stored into the kernel ring buffer and passed back
to user space logs, for example journalctl.

I do not have strong opinion whether adding the
/sys/kernel/config/netconsole/<target>/send_msg is a good idea or not.

Re: [PATCH net-next 0/4] (no cover subject)

[Petr Mladek]

Ah, I have sent it prematurely.

On Thu 2025-12-04 11:46:21, Petr Mladek wrote:
> On Tue 2025-12-02 10:24:42, Jakub Kicinski wrote:
> > On Tue, 2 Dec 2025 02:18:44 -0800 Breno Leitao wrote:
> > > On Mon, Dec 01, 2025 at 04:36:22PM -0800, Jakub Kicinski wrote:
> > > > On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:  
> > > > > This patch series introduces a new configfs attribute that enables sending
> > > > > messages directly through netconsole without going through the kernel's logging
> > > > > infrastructure.
> > > > > 
> > > > > This feature allows users to send custom messages, alerts, or status updates
> > > > > directly to netconsole receivers by writing to
> > > > > /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel
> > > > > buffers, and sending msgs to the serial, which could be slow.
> > > > > 
> > > > > At Meta this is currently used in two cases right now (through printk by
> > > > > now):
> > > > > 
> > > > >   a) When a new workload enters or leave the machine.
> > > > >   b) From time to time, as a "ping" to make sure the netconsole/machine
> > > > >   is alive.
> > > > > 
> > > > > The implementation reuses the existing message transmission functions
> > > > > (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended
> > > > > message formats.
> > > > > 
> > > This feature (in this patchset) is just one step ahead, giving some more
> > > power to netconsole, where extra information could be sent beyond what
> > > is in dmesg.
> > 
> > Having extra metadata makes sense, since the interpretation happens in
> > a different environment. But here we're talking about having extra
> > messages, not extra metadata.
> > 
> > > > The 2nd point is trivial, the first one is what really gives me pause.
> > > > Why do we not care about the logs on host? If the serial is very slow
> > > > presumably it impacts a lot of things, certainly boot speed, so...  
> > > 
> > > This is spot-on - slow serial definitely impacts things like boot speed.
> > > 
> > > See my constant complains here, about slow boot
> > > 
> > > 	https://lore.kernel.org/all/aGVn%2FSnOvwWewkOW@gmail.com/
> > > 
> > > And the something similar in reboot/kexec path:
> > > 
> > > 	https://lore.kernel.org/all/sqwajvt7utnt463tzxgwu2yctyn5m6bjwrslsnupfexeml6hkd@v6sqmpbu3vvu/
> > > 
> > > > perhaps it should be configured to only log messages at a high level?  
> > > 
> > > Chris is actually working on per-console log levels to solve exactly
> > > this problem, so we could filter serial console messages while keeping
> > > everything in other consoles (aka netconsole):
> > > 
> > > 	https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
> > 
> > Excellent! Unless I'm missing more context Chris does seem to be
> > attacking the problem at a more suitable layer.
> 
> This would help to bypass slow serial consoles. But the extra messages
> would still get stored into the kernel ring buffer and passed back
> to user space logs, for example journalctl.

It might actually make sense for the "workload enters or leaves" messages.
But I am not sure about the "ping" messages.

> I do not have strong opinion whether adding the
> /sys/kernel/config/netconsole/<target>/send_msg is a good idea or not.

I just wanted to point out that it is not only about slow serial
consoles.

Best Regards,
Petr

Re: [PATCH net-next 0/4] (no cover subject)

[Breno Leitao]

On Thu, Dec 04, 2025 at 11:51:58AM +0100, Petr Mladek wrote:
> > > > > perhaps it should be configured to only log messages at a high level?  
> > > > 
> > > > Chris is actually working on per-console log levels to solve exactly
> > > > this problem, so we could filter serial console messages while keeping
> > > > everything in other consoles (aka netconsole):
> > > > 
> > > > 	https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
> > > 
> > > Excellent! Unless I'm missing more context Chris does seem to be
> > > attacking the problem at a more suitable layer.
> > 
> > This would help to bypass slow serial consoles. But the extra messages
> > would still get stored into the kernel ring buffer and passed back
> > to user space logs, for example journalctl.
> 
> It might actually make sense for the "workload enters or leaves" messages.
> But I am not sure about the "ping" messages.

Agree. Let me back up and explain my "ping" messages better, which
I think might add more information about this topic.

Meta has millions of servers, and all of them must have netconsole
running 100% of the time.

Of course that this is not reality, and problems happen for different
reasons, the ones that interest me here are:

1) Top of the rack switch MAC address changes (mostly associated with
   network hardware (top of the rack switches and gateway) replacement)
    a) Keep in mind that netconsole target has the destination MAC as
       part of its configuration.

2) Netconsole got associated with the wrong network port, which comes in
   two different flavors.
   a) The machine got provisioned wrongly since day one (Most common
      case)
   b) The machine NIC changed and: 
      i) The target doesn't bind correctly anymore (if netconsole
         target is bound by mac address)
      	   * This is easier to detect, given the target will never be
	     enabled.

3) Netconsd (the daemon that listen to netconsole packets) is buggy or
   dead

4) Network failures across the route


Possible Solutions
==================

In order to detect those issues above, I think the best (or only) way is
to send messages from the host, and check if they got received. If not,
raise an alarm (in the common distributed way).

This could be done in very different ways, tho. Such as:

1) Have a binary in each machine:
	a) This binary reads the netconsole target that is configured,
	   and mimics "ping" UDP netconsole packet.

	Pro: 
	     * It doesn't need any kernel change
	Cons:
	     * It needs to reimplement the netconsole logic in userspace
	     * This needs also a widely distributed binary on all
	       machines

2) Send a ping directly to the console
	a) Something as 'echo ping from $hostname" > /dev/kmsg')

	Pro:
		* No kernel changes
	Cons:
		* These debug messages will be sent to journalctl and to
		  the console, polluting both

3) Using per-loglevel patchset.
	a) Same as above, but, setting netconsole loglevel to DEBUG, while
	   all other consoles to INFO.

	Pro:
		* No changes on netconsole
		* Netconsole "buffers" continues to be synchronized with
		  kernel buffers. Everything in the same page, but,
		  netconsole data has one loglevel higher.
		* Sending a message to netconsole-only message is not
		  special at all. It uses the same workflow we have
		  today, through `/dev/kmsg'
	Cons:
		* Needs to change printk/console code (Chris' patch)
		  that is on review for years now. Will it ever get
		  accepted?
		* These "ping" message will be in kernel buffers and
		  journalctl, and are useless in there (!?)
		* It is not possible to send a message to a single
		  netconsole target.

4) send messages only to netconsole (this patchset)
	Pro:
		* It is easy to test netconsole connective (problem above),
		  without kernel buffers/journal pollution
		* It doesn't depend on the per-loglevel patchset
		* Adds flexibility to netconsole targets.
			- only certain netconsole targets receive
			  certain messages
	Cons:
		* Messages sent to netconsole is a superset of messages in the
		  kernel buffer. In other words, "dmesg" and machine
		  logs/journal will not be able to see messages that
		  were sent directly to netconsole.
			- It might be seen as a back channel (!?)
		* Different netconsole targets may receive different
		  messages. Too much flexibility might be bad (!?)

Anyway, options 1 and 2 are available today. In order to make the
problem easier to solve, we are deciding between approach 3) and 4),
which will require kernel changes, either in printk/console or
netconsole.

Sorry for the long email, I just wanted to do the brain dump about my
view of the world, so, we can decide it from a community perspective in
the open.

Thanks for the discussion,
--breno

Re: [PATCH net-next 0/4] (no cover subject)

[Petr Mladek]

On Fri 2025-12-05 02:21:08, Breno Leitao wrote:
> On Thu, Dec 04, 2025 at 11:51:58AM +0100, Petr Mladek wrote:
> > > > > > perhaps it should be configured to only log messages at a high level?  
> > > > > 
> > > > > Chris is actually working on per-console log levels to solve exactly
> > > > > this problem, so we could filter serial console messages while keeping
> > > > > everything in other consoles (aka netconsole):
> > > > > 
> > > > > 	https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
> > > > 
> > > > Excellent! Unless I'm missing more context Chris does seem to be
> > > > attacking the problem at a more suitable layer.
> > > 
> > > This would help to bypass slow serial consoles. But the extra messages
> > > would still get stored into the kernel ring buffer and passed back
> > > to user space logs, for example journalctl.
> > 
> > It might actually make sense for the "workload enters or leaves" messages.
> > But I am not sure about the "ping" messages.
> 
> Agree. Let me back up and explain my "ping" messages better, which
> I think might add more information about this topic.
> 
> Meta has millions of servers, and all of them must have netconsole
> running 100% of the time.
> 
> Of course that this is not reality, and problems happen for different
> reasons, the ones that interest me here are:
> 
> 1) Top of the rack switch MAC address changes (mostly associated with
>    network hardware (top of the rack switches and gateway) replacement)
>     a) Keep in mind that netconsole target has the destination MAC as
>        part of its configuration.
> 
> 2) Netconsole got associated with the wrong network port, which comes in
>    two different flavors.
>    a) The machine got provisioned wrongly since day one (Most common
>       case)
>    b) The machine NIC changed and: 
>       i) The target doesn't bind correctly anymore (if netconsole
>          target is bound by mac address)
>       	   * This is easier to detect, given the target will never be
> 	     enabled.
> 
> 3) Netconsd (the daemon that listen to netconsole packets) is buggy or
>    dead
> 
> 4) Network failures across the route
> 
> 
> Possible Solutions
> ==================
> 
> In order to detect those issues above, I think the best (or only) way is
> to send messages from the host, and check if they got received. If not,
> raise an alarm (in the common distributed way).
> 
> This could be done in very different ways, tho. Such as:
> 
> 1) Have a binary in each machine:
> 	a) This binary reads the netconsole target that is configured,
> 	   and mimics "ping" UDP netconsole packet.
> 
> 	Pro: 
> 	     * It doesn't need any kernel change
> 	Cons:
> 	     * It needs to reimplement the netconsole logic in userspace
> 	     * This needs also a widely distributed binary on all
> 	       machines
> 
> 2) Send a ping directly to the console
> 	a) Something as 'echo ping from $hostname" > /dev/kmsg')
> 
> 	Pro:
> 		* No kernel changes
> 	Cons:
> 		* These debug messages will be sent to journalctl and to
> 		  the console, polluting both
>
> 3) Using per-loglevel patchset.
> 	a) Same as above, but, setting netconsole loglevel to DEBUG, while
> 	   all other consoles to INFO.
> 
> 	Pro:
> 		* No changes on netconsole
> 		* Netconsole "buffers" continues to be synchronized with
> 		  kernel buffers. Everything in the same page, but,
> 		  netconsole data has one loglevel higher.
> 		* Sending a message to netconsole-only message is not
> 		  special at all. It uses the same workflow we have
> 		  today, through `/dev/kmsg'
> 	Cons:
> 		* Needs to change printk/console code (Chris' patch)
> 		  that is on review for years now. Will it ever get
> 		  accepted?
> 		* These "ping" message will be in kernel buffers and
> 		  journalctl, and are useless in there (!?)
> 		* It is not possible to send a message to a single
> 		  netconsole target.

JFYI, I am going to review the last version of the per-console
loglevel patchset later this week. IMHO, we are very close to
get it merged.

BTW: How often do you ping the netconsole, please?

     IMHO, adding a short message once-per-hour might be bearable,
     once-per-minute might be questionable for the kernel buffer
     but still fine for journalctl.

     Also it depends on the size of the kernel buffer and whether
     you use a crash dump. I mean that it might be handy to have
     some useful messages in the kernel buffer when the crash dump
     is generated and used for debugging. Otherwise, the only
     important thing is whether they get stored externally either
     via console or journalctl.


> 4) send messages only to netconsole (this patchset)
> 	Pro:
> 		* It is easy to test netconsole connective (problem above),
> 		  without kernel buffers/journal pollution
> 		* It doesn't depend on the per-loglevel patchset
> 		* Adds flexibility to netconsole targets.
> 			- only certain netconsole targets receive
> 			  certain messages
> 	Cons:
> 		* Messages sent to netconsole is a superset of messages in the
> 		  kernel buffer. In other words, "dmesg" and machine
> 		  logs/journal will not be able to see messages that
> 		  were sent directly to netconsole.
> 			- It might be seen as a back channel (!?)
> 		* Different netconsole targets may receive different
> 		  messages. Too much flexibility might be bad (!?)

I do not have strong opinion about this.

That said, the location /sys/kernel/config/netconsole/<target>/send_msg
looks a bit weird to me. I would rather expect /dev/netconsole_msg
or so. But I do not have strong opinion. It might be an overkill.

How important is it to trigger the ping from userspace, please?
It might be sent by an existing watchdog.

Best Regards,
Petr

Re: [PATCH net-next 0/4] (no cover subject)

[Breno Leitao]

Hello Petr,

On Mon, Dec 08, 2025 at 03:52:37PM +0100, Petr Mladek wrote:
> On Fri 2025-12-05 02:21:08, Breno Leitao wrote:

> JFYI, I am going to review the last version of the per-console
> loglevel patchset later this week. IMHO, we are very close to
> get it merged.
> 
> BTW: How often do you ping the netconsole, please?
>      IMHO, adding a short message once-per-hour might be bearable,
>      once-per-minute might be questionable for the kernel buffer
>      but still fine for journalctl.

It is not very often today, about once a week. This is mostly due to the
pollution of kernel buffers.

Ideally we can set it to multiple times a day, but less than hourly.

> > 4) send messages only to netconsole (this patchset)
> 
> I do not have strong opinion about this.
> 
> That said, the location /sys/kernel/config/netconsole/<target>/send_msg
> looks a bit weird to me. I would rather expect /dev/netconsole_msg
> or so. But I do not have strong opinion. It might be an overkill.
> 
> How important is it to trigger the ping from userspace, please?
> It might be sent by an existing watchdog.

Medium importance, I would say. I am inclined to distributed design,
where each machine detects broken netconsole and report itself. Instead
of something else (a service) finding broken-netconsole hosts in the
"fleet".

Something as:

	1. Machine boots
	2. userspace sends the ping (or knows that the ping was already
	   sent)
	3. the configuration management (chef, ansible, etc) makes sure
	   that the message got received.


So, step number 3 needs to be executed only after step 2. Initiating the
ping (step 2) from userspace is the easiest way, to control when the
machine can go to step 3.

Thanks
--breno

Re: [PATCH net-next 0/4] (no cover subject)

[Jakub Kicinski]

On Fri, 5 Dec 2025 02:21:08 -0800 Breno Leitao wrote:
> 1) Have a binary in each machine:

> 2) Send a ping directly to the console

> 3) Using per-loglevel patchset.

> 4) send messages only to netconsole (this patchset)

I think I was alluding that another option (not saying that it's the
best but IIUC your requirements it'd be the best fit)):

5) Add a keepalive configfs knob, if set to a non-zero value netconsole
will send an empty (?) message at given interval

  Pros:
   - truly does not require a user binary to run periodically, netcons
     would set a timer in the kernel
  Cons:
   - does not provide the arbitrary "console bypass" message
     functionality

Re: [PATCH net-next 0/4] (no cover subject)

[Breno Leitao]

Hello Jakub,

On Tue, Dec 09, 2025 at 04:37:45PM +0900, Jakub Kicinski wrote:
> On Fri, 5 Dec 2025 02:21:08 -0800 Breno Leitao wrote:
> > 1) Have a binary in each machine:
> 
> > 2) Send a ping directly to the console
> 
> > 3) Using per-loglevel patchset.
> 
> > 4) send messages only to netconsole (this patchset)
> 
> I think I was alluding that another option (not saying that it's the
> best but IIUC your requirements it'd be the best fit)):
> 
> 5) Add a keepalive configfs knob, if set to a non-zero value netconsole
> will send an empty (?) message at given interval
> 
>   Pros:
>    - truly does not require a user binary to run periodically, netcons
>      would set a timer in the kernel
>   Cons:
>    - does not provide the arbitrary "console bypass" message
>      functionality

This is a good idea if we change it slightly. What about a "ping"
configfs item that send sit when I touch it?

Something as:

	# echo 1 > /sys/kernel/configs/<target>/ping
	
And it would ping the host with a predefined "ping" message, and nothing
else.

That would work, for my current problem, honestly.

One drawback compared to a more flexible "send_msg" is that I don't have
complete flexibility on the message format. Thus, if I want to pass
extra information such as a Nonce, timestamp, host state, interface
name, health state, it will not be possible, which is fine for now,
given I am NOT planning to use it at this stage.

Thanks for the idea and discussion,
--breno

Re: [PATCH net-next 0/4] (no cover subject)

[Jakub Kicinski]

On Tue, 9 Dec 2025 09:46:51 -0800 Breno Leitao wrote:
> > I think I was alluding that another option (not saying that it's the
> > best but IIUC your requirements it'd be the best fit)):
> > 
> > 5) Add a keepalive configfs knob, if set to a non-zero value netconsole
> > will send an empty (?) message at given interval
> > 
> >   Pros:
> >    - truly does not require a user binary to run periodically, netcons
> >      would set a timer in the kernel
> >   Cons:
> >    - does not provide the arbitrary "console bypass" message
> >      functionality  
> 
> This is a good idea if we change it slightly. What about a "ping"
> configfs item that send sit when I touch it?
> 
> Something as:
> 
> 	# echo 1 > /sys/kernel/configs/<target>/ping
>
> And it would ping the host with a predefined "ping" message, and nothing
> else.
> 
> That would work, for my current problem, honestly.
> 
> One drawback compared to a more flexible "send_msg" is that I don't have
> complete flexibility on the message format. Thus, if I want to pass
> extra information such as a Nonce, timestamp, host state, interface
> name, health state, it will not be possible, which is fine for now,
> given I am NOT planning to use it at this stage.

If you still want to tickle it from user space periodically, I guess
send_msg is more flexible. I think the main advantage of keepalive
would be to remove the need for periodic userspace work.