Linux PCI subsystem development
 help / color / mirror / Atom feed
From: Bjorn Helgaas <helgaas@kernel.org>
To: Jon Pan-Doh <pandoh@google.com>
Cc: "Bjorn Helgaas" <bhelgaas@google.com>,
	"Karolina Stolarek" <karolina.stolarek@oracle.com>,
	linux-pci@vger.kernel.org,
	"Martin Petersen" <martin.petersen@oracle.com>,
	"Ben Fuller" <ben.fuller@oracle.com>,
	"Drew Walton" <drewwalton@microsoft.com>,
	"Anil Agrawal" <anilagrawal@meta.com>,
	"Tony Luck" <tony.luck@intel.com>,
	"Ilpo Järvinen" <ilpo.jarvinen@linux.intel.com>,
	"Sathyanarayanan Kuppuswamy"
	<sathyanarayanan.kuppuswamy@linux.intel.com>,
	"Lukas Wunner" <lukas@wunner.de>,
	"Jonathan Cameron" <Jonathan.Cameron@huawei.com>,
	"Terry Bowman" <Terry.bowman@amd.com>
Subject: Re: [PATCH v3 5/8] PCI/AER: Introduce ratelimit for error logs
Date: Wed, 19 Mar 2025 13:47:02 -0500	[thread overview]
Message-ID: <20250319184702.GA1051253@bhelgaas> (raw)
In-Reply-To: <20250319084050.366718-6-pandoh@google.com>

On Wed, Mar 19, 2025 at 01:40:46AM -0700, Jon Pan-Doh wrote:
> Spammy devices can flood kernel logs with AER errors and slow/stall
> execution. Add per-device ratelimits for AER correctable and uncorrectable
> errors that use the kernel defaults (10 per 5s).
> 
> Tested using aer-inject[1]. Sent 11 AER errors. Observed 10 errors logged
> while AER stats (cat /sys/bus/pci/devices/<dev>/aer_dev_correctable) show
> true count of 11.

I think this is really on the right track.  A few minor comments
below.

> @@ -697,6 +711,7 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info,
>  {
>  	int layer, agent;
>  	int id = pci_dev_id(dev);
> +	struct ratelimit_state *ratelimit;
>  
>  	if (!info->status) {
>  		pci_err(dev, "PCIe Bus Error: severity=%s, type=Inaccessible, (Unregistered Agent ID)\n",
> @@ -704,6 +719,17 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info,
>  		goto out;
>  	}
>  
> +	if (info->severity == AER_CORRECTABLE)
> +		ratelimit = &dev->aer_report->cor_log_ratelimit;
> +	else
> +		ratelimit = &dev->aer_report->uncor_log_ratelimit;
> +
> +	trace_aer_event(dev_name(&dev->dev), (info->status & ~info->mask),
> +			info->severity, info->tlp_header_valid, &info->tlp);
> +
> +	if (!__ratelimit(ratelimit))
> +		return;

  - I think the ratelimit lookup and __ratelimit() call should be
    together since there's no need for trace_aer_event() to be in the
    middle.

  - The lookup and __ratelimit() calls are repeated and are probably
    worth factoring out into something like this:

      static int aer_ratelimit(struct pci_dev *dev, unsigned int severity)

  - Previously we *always* called trace_aer_event(), but now we don't
    in the !info->status case.  Maybe an unintentional change?  I
    think we should call trace_aer_event() always, or change that in a
    separate patch if we need to.  This would always have been simpler
    if trace_aer_event() had been the very first thing in the
    function.

  - The !info->status case message is not rate-limited.  Seems like
    maybe it should be?

>  	layer = AER_GET_LAYER_ERROR(info->severity, info->status);
>  	agent = AER_GET_AGENT(info->severity, info->status);
>  
> @@ -722,21 +748,33 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info,
>  out:
>  	if (info->id && info->error_dev_num > 1 && info->id == id)
>  		pci_err(dev, "  Error of this Agent is reported first\n");
> -
> -	trace_aer_event(dev_name(&dev->dev), (info->status & ~info->mask),
> -			info->severity, info->tlp_header_valid, &info->tlp);
>  }

  reply	other threads:[~2025-03-19 18:47 UTC|newest]

Thread overview: 28+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-03-19  8:40 [PATCH v3 0/8] Rate limit AER logs Jon Pan-Doh
2025-03-19  8:40 ` [PATCH v3 1/8] PCI/AER: Check log level once and propagate down Jon Pan-Doh
2025-03-20  0:07   ` Sathyanarayanan Kuppuswamy
2025-03-19  8:40 ` [PATCH v3 2/8] PCI/AER: Make all pci_print_aer() log levels depend on error type Jon Pan-Doh
2025-03-19  9:52   ` Ilpo Järvinen
2025-03-20  2:39   ` Sathyanarayanan Kuppuswamy
2025-03-20  8:27     ` Jon Pan-Doh
2025-03-20 14:23       ` Sathyanarayanan Kuppuswamy
2025-03-20 19:06         ` Jon Pan-Doh
2025-03-19  8:40 ` [PATCH v3 3/8] PCI/AER: Move AER stat collection out of __aer_print_error() Jon Pan-Doh
2025-03-19 18:19   ` Bjorn Helgaas
2025-03-20  8:27     ` Jon Pan-Doh
2025-03-20  3:22   ` Sathyanarayanan Kuppuswamy
2025-03-20  8:29     ` Jon Pan-Doh
2025-03-19  8:40 ` [PATCH v3 4/8] PCI/AER: Rename struct aer_stats to aer_report Jon Pan-Doh
2025-03-20  3:29   ` Sathyanarayanan Kuppuswamy
2025-03-20  8:28     ` Jon Pan-Doh
2025-03-19  8:40 ` [PATCH v3 5/8] PCI/AER: Introduce ratelimit for error logs Jon Pan-Doh
2025-03-19 18:47   ` Bjorn Helgaas [this message]
2025-03-20  8:27     ` Jon Pan-Doh
2025-03-20 18:23       ` Bjorn Helgaas
2025-03-19  8:40 ` [PATCH v3 6/8] PCI/AER: Add ratelimits to PCI AER Documentation Jon Pan-Doh
2025-03-19  8:40 ` [PATCH v3 7/8] PCI/AER: Add sysfs attributes for log ratelimits Jon Pan-Doh
2025-03-19  9:51   ` Ilpo Järvinen
2025-03-20  8:27     ` Jon Pan-Doh
2025-03-19  8:40 ` [PATCH v3 8/8] PCI/AER: Update AER sysfs ABI filename Jon Pan-Doh
2025-03-19 22:29 ` [PATCH v3 0/8] Rate limit AER logs Sathyanarayanan Kuppuswamy
2025-03-19 22:52   ` Jon Pan-Doh

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20250319184702.GA1051253@bhelgaas \
    --to=helgaas@kernel.org \
    --cc=Jonathan.Cameron@huawei.com \
    --cc=Terry.bowman@amd.com \
    --cc=anilagrawal@meta.com \
    --cc=ben.fuller@oracle.com \
    --cc=bhelgaas@google.com \
    --cc=drewwalton@microsoft.com \
    --cc=ilpo.jarvinen@linux.intel.com \
    --cc=karolina.stolarek@oracle.com \
    --cc=linux-pci@vger.kernel.org \
    --cc=lukas@wunner.de \
    --cc=martin.petersen@oracle.com \
    --cc=pandoh@google.com \
    --cc=sathyanarayanan.kuppuswamy@linux.intel.com \
    --cc=tony.luck@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox