Re: [PATCH 0/8] Rate limit AER logs/IRQs

["Bowman, Terry" <terry.bowman@amd.com>]

Hi Jon,

Can you share the base commit used here? I would like to try the patchset. 

Regards,
Terry

On 1/15/2025 1:42 AM, Jon Pan-Doh wrote:
> Proposal
> ========
> 
> When using native AER, spammy devices can flood kernel logs with AER errors
> and slow/stall execution. Add per-device per-error-severity ratelimits
> for more robust error logging. Allow userspace to configure ratelimits
> via sysfs knobs.
> 
> Motivation
> ==========
> 
> Several OCP members have issues with inconsistent PCIe error handling,
> exacerbated at datacenter scale (myriad of devices).
> OCP HW/Fault Management subproject set out to solve this by
> standardizing industry:
> 
> - PCIe error handling best practices
> - Fault Management/RAS (incl. PCIe errors)
> 
> Exposing PCIe errors/debug info in-band for a userspace daemon (e.g.
> rasdaemon) to collect/pass on to repairability services is part of the
> roadmap.
> 
> Background
> ==========
> 
> AER error spam has been observed many times, both publicly (e.g. [1], [2],
> [3]) and privately. While it usually occurs with correctable errors, it can
> happen with uncorrectable errors (e.g. during new HW bringup). 
> 
> There have been previous attempts to add ratelimits to AER logs ([4],
> [5]). The most recent attempt[5] has many similarities with the proposed
> approach.
> 
> Patch organization
> ==================
> 1-3 AER logging cleanup
> 4-7 Ratelimits and sysfs knobs
> 8   Sysfs cleanup (RFC that breaks existing ABI/can be dropped)
> 
> Outstanding work
> ================
> Cleanup:
> - Consolidate aer_print_error() and pci_print_error() path
> - Elevate log level logic out of print functions[6]
> 
> [1] https://bugzilla.kernel.org/show_bug.cgi?id=215027
> [2] https://bugzilla.kernel.org/show_bug.cgi?id=201517
> [3] https://bugzilla.kernel.org/show_bug.cgi?id=196183
> [4] https://lore.kernel.org/linux-pci/20230606035442.2886343-2-grundler@chromium.org/
> [5] https://lore.kernel.org/linux-pci/cover.1736341506.git.karolina.stolarek@oracle.com/
> [6] https://lore.kernel.org/linux-pci/edd77011aafad4c0654358a26b4e538d0c5a321d.1736341506.git.karolina.stolarek@oracle.com/
> 
> Jon Pan-Doh (8):
>   PCI/AER: Remove aer_print_port_info
>   PCI/AER: Move AER stat collection out of __aer_print_error
>   PCI/AER: Rename struct aer_stats to aer_info
>   PCI/AER: Introduce ratelimit for error logs
>   PCI/AER: Introduce ratelimit for AER IRQs
>   PCI/AER: Add AER sysfs attributes for ratelimits
>   PCI/AER: Update AER sysfs ABI filename
>   PCI/AER: Move AER sysfs attributes into separate directory
> 
>  ...es-aer_stats => sysfs-bus-pci-devices-aer} |  50 +++-
>  Documentation/PCI/pcieaer-howto.rst           |  10 +-
>  drivers/pci/pci-sysfs.c                       |   2 +-
>  drivers/pci/pci.h                             |   2 +-
>  drivers/pci/pcie/aer.c                        | 227 +++++++++++++-----
>  include/linux/pci.h                           |   2 +-
>  6 files changed, 216 insertions(+), 77 deletions(-)
>  rename Documentation/ABI/testing/{sysfs-bus-pci-devices-aer_stats => sysfs-bus-pci-devices-aer} (69%)
>