All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH net-next RFC 0/6] netconsole: Support messages ratelimit-ing
@ 2026-08-18 10:29 Breno Leitao
  2026-08-18 10:29 ` [PATCH net-next RFC 1/6] netconsole: add a per-target message rate limit Breno Leitao
                   ` (5 more replies)
  0 siblings, 6 replies; 9+ messages in thread
From: Breno Leitao @ 2026-08-18 10:29 UTC (permalink / raw)
  To: Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski,
	Paolo Abeni, Simon Horman, Jonathan Corbet, Shuah Khan,
	Shuah Khan
  Cc: paulmck, davej, riel, gustavold, asantostc, netdev, linux-kernel,
	linux-doc, linux-kselftest, samuelcrossley, Breno Leitao,
	kernel-team

netconsole hands every console message to every enabled target, with no
bound on the rate.

At Meta, a few hosts caused some DoS and netconsd was OOM killed in
some regions, fed by 3 billion RCU messages from thousand hosts and by
a swap fault that retries forever, which logged 6 million copies of one
line on a single host in a few hours.

Both floods are being fixed where they are printed, by Paula and me for RCU
Tasks [1][2][3] and by me for swap faul fix [4], but that only ever
covers the flood already found, not new issues that might be happening.

Use the ratelimit in the kernel to ratelimit messages from netconsole.

Give each target a token bucket, consulted once per message so that a
message split into several ncfrag packets is sent whole or dropped
whole.

It starts with a zero interval, which struct ratelimit_state treats as
unlimited, so nothing changes until ratelimit_interval_ms and
ratelimit_burst are written.

What the bucket discards is reported on the wire, so the receiver sees
"netconsole: 45 messages dropped by rate limit" rather than an
unexplained gap. This is what is captured on netconsd, when this patch
is applied:

	7.2.0-rc7-01460-g75848acaf136,13,628,252392647,-;netconsole selftest: netcons_AGujw 1
	7.2.0-rc7-01460-g75848acaf136,13,629,252392782,-;netconsole selftest: netcons_AGujw 2
	....
	7.2.0-rc7-01460-g75848acaf136,4,0,254742176,-;netconsole: 45 messages dropped by rate limit
	7.2.0-rc7-01460-g75848acaf136,13,678,254742125,-;netconsole selftest: netcons_AGujw 1

Crash output is exempt. The bucket is skipped while oops_in_progress is
set, so a limit configured for steady-state logging cannot cost a target
part of an oops or a panic.

[1] https://lore.kernel.org/all/anw8Qw8lfeIykski@gmail.com/
[2] https://lore.kernel.org/all/20260810-rcu_task_shrink_lruvec-v1-1-4d9f7d5251cb@debian.org/
[3] https://lore.kernel.org/all/20260810-rcu_task_shrink_lruvec-v1-1-4d9f7d5251cb@debian.org/
[4] https://lore.kernel.org/all/20260813-swap-v2-0-4a625ccabdae@debian.org/

Signed-off-by: Breno Leitao <leitao@debian.org>
---
Breno Leitao (6):
      netconsole: add a per-target message rate limit
      netconsole: allow configuring the rate limit interval through configfs
      netconsole: allow configuring the rate limit burst through configfs
      netconsole: tell the target when the rate limit drops messages
      docs: netconsole: document rate limit feature
      selftests: netcons: test the per-target rate limit

 Documentation/networking/netconsole.rst            |  43 ++++++
 drivers/net/netconsole.c                           | 145 +++++++++++++++++++
 .../selftests/drivers/net/netconsole/Makefile      |   1 +
 .../drivers/net/netconsole/netcons_ratelimit.sh    | 160 +++++++++++++++++++++
 4 files changed, 349 insertions(+)
---
base-commit: e6a5d573d24cd375e09d24f136523cb3cc85c9d3
change-id: 20260817-netcons_ratelimit-629b04a73c57

Best regards,
--  
Breno Leitao <leitao@debian.org>


^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2026-08-21 12:42 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-18 10:29 [PATCH net-next RFC 0/6] netconsole: Support messages ratelimit-ing Breno Leitao
2026-08-18 10:29 ` [PATCH net-next RFC 1/6] netconsole: add a per-target message rate limit Breno Leitao
2026-08-18 10:29 ` [PATCH net-next RFC 2/6] netconsole: allow configuring the rate limit interval through configfs Breno Leitao
2026-08-18 10:29 ` [PATCH net-next RFC 3/6] netconsole: allow configuring the rate limit burst " Breno Leitao
2026-08-18 10:29 ` [PATCH net-next RFC 4/6] netconsole: tell the target when the rate limit drops messages Breno Leitao
2026-08-20 20:49   ` Gustavo Luiz Duarte
2026-08-21 12:41     ` Breno Leitao
2026-08-18 10:29 ` [PATCH net-next RFC 5/6] docs: netconsole: document rate limit feature Breno Leitao
2026-08-18 10:29 ` [PATCH net-next RFC 6/6] selftests: netcons: test the per-target rate limit Breno Leitao

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.