From: Bjorn Helgaas <helgaas@kernel.org>
To: Jon Pan-Doh <pandoh@google.com>
Cc: "Bjorn Helgaas" <bhelgaas@google.com>,
"Karolina Stolarek" <karolina.stolarek@oracle.com>,
linux-pci@vger.kernel.org,
"Martin Petersen" <martin.petersen@oracle.com>,
"Ben Fuller" <ben.fuller@oracle.com>,
"Drew Walton" <drewwalton@microsoft.com>,
"Anil Agrawal" <anilagrawal@meta.com>,
"Tony Luck" <tony.luck@intel.com>,
"Ilpo Järvinen" <ilpo.jarvinen@linux.intel.com>,
"Sathyanarayanan Kuppuswamy"
<sathyanarayanan.kuppuswamy@linux.intel.com>,
"Lukas Wunner" <lukas@wunner.de>,
"Jonathan Cameron" <Jonathan.Cameron@huawei.com>,
"Terry Bowman" <Terry.bowman@amd.com>
Subject: Re: [PATCH v3 5/8] PCI/AER: Introduce ratelimit for error logs
Date: Wed, 19 Mar 2025 13:47:02 -0500 [thread overview]
Message-ID: <20250319184702.GA1051253@bhelgaas> (raw)
In-Reply-To: <20250319084050.366718-6-pandoh@google.com>
On Wed, Mar 19, 2025 at 01:40:46AM -0700, Jon Pan-Doh wrote:
> Spammy devices can flood kernel logs with AER errors and slow/stall
> execution. Add per-device ratelimits for AER correctable and uncorrectable
> errors that use the kernel defaults (10 per 5s).
>
> Tested using aer-inject[1]. Sent 11 AER errors. Observed 10 errors logged
> while AER stats (cat /sys/bus/pci/devices/<dev>/aer_dev_correctable) show
> true count of 11.
I think this is really on the right track. A few minor comments
below.
> @@ -697,6 +711,7 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info,
> {
> int layer, agent;
> int id = pci_dev_id(dev);
> + struct ratelimit_state *ratelimit;
>
> if (!info->status) {
> pci_err(dev, "PCIe Bus Error: severity=%s, type=Inaccessible, (Unregistered Agent ID)\n",
> @@ -704,6 +719,17 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info,
> goto out;
> }
>
> + if (info->severity == AER_CORRECTABLE)
> + ratelimit = &dev->aer_report->cor_log_ratelimit;
> + else
> + ratelimit = &dev->aer_report->uncor_log_ratelimit;
> +
> + trace_aer_event(dev_name(&dev->dev), (info->status & ~info->mask),
> + info->severity, info->tlp_header_valid, &info->tlp);
> +
> + if (!__ratelimit(ratelimit))
> + return;
- I think the ratelimit lookup and __ratelimit() call should be
together since there's no need for trace_aer_event() to be in the
middle.
- The lookup and __ratelimit() calls are repeated and are probably
worth factoring out into something like this:
static int aer_ratelimit(struct pci_dev *dev, unsigned int severity)
- Previously we *always* called trace_aer_event(), but now we don't
in the !info->status case. Maybe an unintentional change? I
think we should call trace_aer_event() always, or change that in a
separate patch if we need to. This would always have been simpler
if trace_aer_event() had been the very first thing in the
function.
- The !info->status case message is not rate-limited. Seems like
maybe it should be?
> layer = AER_GET_LAYER_ERROR(info->severity, info->status);
> agent = AER_GET_AGENT(info->severity, info->status);
>
> @@ -722,21 +748,33 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info,
> out:
> if (info->id && info->error_dev_num > 1 && info->id == id)
> pci_err(dev, " Error of this Agent is reported first\n");
> -
> - trace_aer_event(dev_name(&dev->dev), (info->status & ~info->mask),
> - info->severity, info->tlp_header_valid, &info->tlp);
> }
next prev parent reply other threads:[~2025-03-19 18:47 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-03-19 8:40 [PATCH v3 0/8] Rate limit AER logs Jon Pan-Doh
2025-03-19 8:40 ` [PATCH v3 1/8] PCI/AER: Check log level once and propagate down Jon Pan-Doh
2025-03-20 0:07 ` Sathyanarayanan Kuppuswamy
2025-03-19 8:40 ` [PATCH v3 2/8] PCI/AER: Make all pci_print_aer() log levels depend on error type Jon Pan-Doh
2025-03-19 9:52 ` Ilpo Järvinen
2025-03-20 2:39 ` Sathyanarayanan Kuppuswamy
2025-03-20 8:27 ` Jon Pan-Doh
2025-03-20 14:23 ` Sathyanarayanan Kuppuswamy
2025-03-20 19:06 ` Jon Pan-Doh
2025-03-19 8:40 ` [PATCH v3 3/8] PCI/AER: Move AER stat collection out of __aer_print_error() Jon Pan-Doh
2025-03-19 18:19 ` Bjorn Helgaas
2025-03-20 8:27 ` Jon Pan-Doh
2025-03-20 3:22 ` Sathyanarayanan Kuppuswamy
2025-03-20 8:29 ` Jon Pan-Doh
2025-03-19 8:40 ` [PATCH v3 4/8] PCI/AER: Rename struct aer_stats to aer_report Jon Pan-Doh
2025-03-20 3:29 ` Sathyanarayanan Kuppuswamy
2025-03-20 8:28 ` Jon Pan-Doh
2025-03-19 8:40 ` [PATCH v3 5/8] PCI/AER: Introduce ratelimit for error logs Jon Pan-Doh
2025-03-19 18:47 ` Bjorn Helgaas [this message]
2025-03-20 8:27 ` Jon Pan-Doh
2025-03-20 18:23 ` Bjorn Helgaas
2025-03-19 8:40 ` [PATCH v3 6/8] PCI/AER: Add ratelimits to PCI AER Documentation Jon Pan-Doh
2025-03-19 8:40 ` [PATCH v3 7/8] PCI/AER: Add sysfs attributes for log ratelimits Jon Pan-Doh
2025-03-19 9:51 ` Ilpo Järvinen
2025-03-20 8:27 ` Jon Pan-Doh
2025-03-19 8:40 ` [PATCH v3 8/8] PCI/AER: Update AER sysfs ABI filename Jon Pan-Doh
2025-03-19 22:29 ` [PATCH v3 0/8] Rate limit AER logs Sathyanarayanan Kuppuswamy
2025-03-19 22:52 ` Jon Pan-Doh
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250319184702.GA1051253@bhelgaas \
--to=helgaas@kernel.org \
--cc=Jonathan.Cameron@huawei.com \
--cc=Terry.bowman@amd.com \
--cc=anilagrawal@meta.com \
--cc=ben.fuller@oracle.com \
--cc=bhelgaas@google.com \
--cc=drewwalton@microsoft.com \
--cc=ilpo.jarvinen@linux.intel.com \
--cc=karolina.stolarek@oracle.com \
--cc=linux-pci@vger.kernel.org \
--cc=lukas@wunner.de \
--cc=martin.petersen@oracle.com \
--cc=pandoh@google.com \
--cc=sathyanarayanan.kuppuswamy@linux.intel.com \
--cc=tony.luck@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox