All of lore.kernel.org
 help / color / mirror / Atom feed
From: Breno Leitao <leitao@debian.org>
To: Andrew Lunn <andrew+netdev@lunn.ch>,
	 "David S. Miller" <davem@davemloft.net>,
	Eric Dumazet <edumazet@google.com>,
	 Jakub Kicinski <kuba@kernel.org>,
	Paolo Abeni <pabeni@redhat.com>,  Simon Horman <horms@kernel.org>,
	Jonathan Corbet <corbet@lwn.net>,
	 Shuah Khan <skhan@linuxfoundation.org>,
	Shuah Khan <shuah@kernel.org>
Cc: Randy Dunlap <rdunlap@infradead.org>,
	paulmck@kernel.org,  gustavold@gmail.com, asantostc@gmail.com,
	netdev@vger.kernel.org,  linux-kernel@vger.kernel.org,
	linux-doc@vger.kernel.org,  linux-kselftest@vger.kernel.org,
	Breno Leitao <leitao@debian.org>,
	 kernel-team@meta.com
Subject: [PATCH net-next v2 0/4] netconsole: Support messages ratelimit-ing
Date: Thu, 10 Sep 2026 06:46:09 -0700	[thread overview]
Message-ID: <20260910-netcons_ratelimit-v2-0-ebf0dd91e26e@debian.org> (raw)

netconsole hands every console message to every enabled target, with no
bound on the rate.

At Meta, a few hosts caused some DoS and netconsd was OOM killed in
some regions, fed by 3 billion RCU messages from thousand hosts and by
a swap fault that retries forever, which logged 6 million copies of one
line on a single host in a few hours.

Both floods are being fixed where they are printed, by Paul and me for RCU
Tasks [1][2] and by me for the swap fault fix [3], but that only ever
covers the flood already found, not new issues that might be happening.

Use the ratelimit in the kernel to ratelimit messages from netconsole.

Give each target a token bucket, consulted once per message so that a
message split into several ncfrag packets is sent whole or dropped
whole.

It starts with a zero interval, which struct ratelimit_state treats as
unlimited, so nothing changes until ratelimit_interval_ms and
ratelimit_burst are written.

What the bucket discards leaves nothing on the wire. An extended target
sees the loss as a gap in the sequence number the header carries:

	7.2.0-rc7-01460-g75848acaf136,13,628,252392647,-;netconsole selftest: netcons_AGujw 1
	7.2.0-rc7-01460-g75848acaf136,13,629,252392782,-;netconsole selftest: netcons_AGujw 2
	7.2.0-rc7-01460-g75848acaf136,13,678,254742125,-;netconsole selftest: netcons_AGujw 1

Crash output is exempt. The bucket is skipped while oops_in_progress is
set, so a limit configured for steady-state logging cannot cost a target
part of an oops or a panic.

[1] https://lore.kernel.org/all/anw8Qw8lfeIykski@gmail.com/
[2] https://lore.kernel.org/all/20260810-rcu_task_shrink_lruvec-v1-1-4d9f7d5251cb@debian.org/
[3] https://lore.kernel.org/all/20260813-swap-v2-0-4a625ccabdae@debian.org/

Signed-off-by: Breno Leitao <leitao@debian.org>
---
Changes in v2:
- Drop the patch that sent a "N messages dropped by rate limit" notice
  to the target. (Gustavo Luiz Duarte).
- Link to v1: https://patch.msgid.link/20260818-netcons_ratelimit-v1-0-8c5d2d17789c@debian.org

---
Breno Leitao (4):
      netconsole: add a per-target message rate limit
      netconsole: allow configuring the rate limit interval through configfs
      netconsole: allow configuring the rate limit burst through configfs
      docs: netconsole: document rate limit feature

 Documentation/networking/netconsole.rst | 38 +++++++++++++
 drivers/net/netconsole.c                | 99 +++++++++++++++++++++++++++++++++
 2 files changed, 137 insertions(+)
---
base-commit: c68a982815dcce5464e3bf2a31ac94f5146c04ca
change-id: 20260817-netcons_ratelimit-629b04a73c57

Best regards,
--  
Breno Leitao <leitao@debian.org>


             reply	other threads:[~2026-09-10 13:46 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-09-10 13:46 Breno Leitao [this message]
2026-09-10 13:46 ` [PATCH net-next v2 1/4] netconsole: add a per-target message rate limit Breno Leitao
2026-09-12  0:06   ` netdev-bot+sashiko
2026-09-10 13:46 ` [PATCH net-next v2 2/4] netconsole: allow configuring the rate limit interval through configfs Breno Leitao
2026-09-12  0:06   ` netdev-bot+sashiko
2026-09-10 13:46 ` [PATCH net-next v2 3/4] netconsole: allow configuring the rate limit burst " Breno Leitao
2026-09-12  0:06   ` netdev-bot+sashiko
2026-09-10 13:46 ` [PATCH net-next v2 4/4] docs: netconsole: document rate limit feature Breno Leitao
2026-09-12  0:06   ` netdev-bot+sashiko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260910-netcons_ratelimit-v2-0-ebf0dd91e26e@debian.org \
    --to=leitao@debian.org \
    --cc=andrew+netdev@lunn.ch \
    --cc=asantostc@gmail.com \
    --cc=corbet@lwn.net \
    --cc=davem@davemloft.net \
    --cc=edumazet@google.com \
    --cc=gustavold@gmail.com \
    --cc=horms@kernel.org \
    --cc=kernel-team@meta.com \
    --cc=kuba@kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=paulmck@kernel.org \
    --cc=rdunlap@infradead.org \
    --cc=shuah@kernel.org \
    --cc=skhan@linuxfoundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.