Re: [PATCH v8 20/20] PCI/AER: Add sysfs attributes for log ratelimits

[Sathyanarayanan Kuppuswamy <sathyanarayanan.kuppuswamy@linux.intel.com>]

Hi,

On 5/22/25 4:21 PM, Bjorn Helgaas wrote:
> From: Jon Pan-Doh <pandoh@google.com>
>
> Allow userspace to read/write log ratelimits per device (including
> enable/disable). Create aer/ sysfs directory to store them and any
> future AER configs.
>
> The new sysfs files are:
>
>    /sys/bus/pci/devices/*/aer/correctable_ratelimit_burst
>    /sys/bus/pci/devices/*/aer/correctable_ratelimit_interval_ms
>    /sys/bus/pci/devices/*/aer/nonfatal_ratelimit_burst
>    /sys/bus/pci/devices/*/aer/nonfatal_ratelimit_interval_ms
>
> The default values are ratelimit_burst=10, ratelimit_interval_ms=5000, so
> if we try to emit more than 10 messages in a 5 second period, some are
> suppressed.
>
> Update AER sysfs ABI filename to reflect the broader scope of AER sysfs
> attributes (e.g. stats and ratelimits).
>
>    Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats ->
>      sysfs-bus-pci-devices-aer
>
> Tested using aer-inject[1]. Configured correctable log ratelimit to 5.
> Sent 6 AER errors. Observed 5 errors logged while AER stats
> (cat /sys/bus/pci/devices/<dev>/aer_dev_correctable) shows 6.
>
> Disabled ratelimiting and sent 6 more AER errors. Observed all 6 errors
> logged and accounted in AER stats (12 total errors).
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/gong.chen/aer-inject.git
>
> [bhelgaas: note fatal errors are not ratelimited, "aer_report" ->
> "aer_info", replace ratelimit_log_enable toggle with *_ratelimit_interval_ms]
>
> Signed-off-by: Karolina Stolarek <karolina.stolarek@oracle.com>
> Signed-off-by: Jon Pan-Doh <pandoh@google.com>
> Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
> Link: https://patch.msgid.link/20250520215047.1350603-18-helgaas@kernel.org
> ---

Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>

>   ...es-aer_stats => sysfs-bus-pci-devices-aer} |  44 ++++++++
>   Documentation/PCI/pcieaer-howto.rst           |   5 +-
>   drivers/pci/pci-sysfs.c                       |   1 +
>   drivers/pci/pci.h                             |   1 +
>   drivers/pci/pcie/aer.c                        | 105 ++++++++++++++++++
>   5 files changed, 155 insertions(+), 1 deletion(-)
>   rename Documentation/ABI/testing/{sysfs-bus-pci-devices-aer_stats => sysfs-bus-pci-devices-aer} (72%)
>
> diff --git a/Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats b/Documentation/ABI/testing/sysfs-bus-pci-devices-aer
> similarity index 72%
> rename from Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats
> rename to Documentation/ABI/testing/sysfs-bus-pci-devices-aer
> index d1f67bb81d5d..5ed284523956 100644
> --- a/Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats
> +++ b/Documentation/ABI/testing/sysfs-bus-pci-devices-aer
> @@ -117,3 +117,47 @@ Date:		July 2018
>   KernelVersion:	4.19.0
>   Contact:	linux-pci@vger.kernel.org, rajatja@google.com
>   Description:	Total number of ERR_NONFATAL messages reported to rootport.
> +
> +PCIe AER ratelimits
> +-------------------
> +
> +These attributes show up under all the devices that are AER capable.
> +They represent configurable ratelimits of logs per error type.
> +
> +See Documentation/PCI/pcieaer-howto.rst for more info on ratelimits.
> +
> +What:		/sys/bus/pci/devices/<dev>/aer/correctable_ratelimit_interval_ms
> +Date:		May 2025
> +KernelVersion:	6.16.0
> +Contact:	linux-pci@vger.kernel.org
> +Description:	Writing 0 disables AER correctable error log ratelimiting.
> +		Writing a positive value sets the ratelimit interval in ms.
> +		Default is DEFAULT_RATELIMIT_INTERVAL (5000 ms).
> +
> +What:		/sys/bus/pci/devices/<dev>/aer/correctable_ratelimit_burst
> +Date:		May 2025
> +KernelVersion:	6.16.0
> +Contact:	linux-pci@vger.kernel.org
> +Description:	Ratelimit burst for correctable error logs. Writing a value
> +		changes the number of errors (burst) allowed per interval
> +		before ratelimiting. Reading gets the current ratelimit
> +		burst. Default is DEFAULT_RATELIMIT_BURST (10).
> +
> +What:		/sys/bus/pci/devices/<dev>/aer/nonfatal_ratelimit_interval_ms
> +Date:		May 2025
> +KernelVersion:	6.16.0
> +Contact:	linux-pci@vger.kernel.org
> +Description:	Writing 0 disables AER non-fatal uncorrectable error log
> +		ratelimiting. Writing a positive value sets the ratelimit
> +		interval in ms. Default is DEFAULT_RATELIMIT_INTERVAL
> +		(5000 ms).
> +
> +What:		/sys/bus/pci/devices/<dev>/aer/nonfatal_ratelimit_burst
> +Date:		May 2025
> +KernelVersion:	6.16.0
> +Contact:	linux-pci@vger.kernel.org
> +Description:	Ratelimit burst for non-fatal uncorrectable error logs.
> +		Writing a value changes the number of errors (burst)
> +		allowed per interval before ratelimiting. Reading gets the
> +		current ratelimit burst. Default is DEFAULT_RATELIMIT_BURST
> +		(10).
> diff --git a/Documentation/PCI/pcieaer-howto.rst b/Documentation/PCI/pcieaer-howto.rst
> index 6fb31516fff1..4b71e2f43ca7 100644
> --- a/Documentation/PCI/pcieaer-howto.rst
> +++ b/Documentation/PCI/pcieaer-howto.rst
> @@ -97,12 +97,15 @@ DPC errors, are not ratelimited.
>   AER uses the default ratelimit of DEFAULT_RATELIMIT_BURST (10 events) over
>   DEFAULT_RATELIMIT_INTERVAL (5 seconds).
>   
> +Ratelimits are exposed in the form of sysfs attributes and configurable.
> +See Documentation/ABI/testing/sysfs-bus-pci-devices-aer.
> +
>   AER Statistics / Counters
>   -------------------------
>   
>   When PCIe AER errors are captured, the counters / statistics are also exposed
>   in the form of sysfs attributes which are documented at
> -Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats
> +Documentation/ABI/testing/sysfs-bus-pci-devices-aer.
>   
>   Developer Guide
>   ===============
> diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
> index c6cda56ca52c..278de99b00ce 100644
> --- a/drivers/pci/pci-sysfs.c
> +++ b/drivers/pci/pci-sysfs.c
> @@ -1805,6 +1805,7 @@ const struct attribute_group *pci_dev_attr_groups[] = {
>   	&pcie_dev_attr_group,
>   #ifdef CONFIG_PCIEAER
>   	&aer_stats_attr_group,
> +	&aer_attr_group,
>   #endif
>   #ifdef CONFIG_PCIEASPM
>   	&aspm_ctrl_attr_group,
> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> index 3023c68fe485..eca2812cfd25 100644
> --- a/drivers/pci/pci.h
> +++ b/drivers/pci/pci.h
> @@ -965,6 +965,7 @@ void pci_no_aer(void);
>   void pci_aer_init(struct pci_dev *dev);
>   void pci_aer_exit(struct pci_dev *dev);
>   extern const struct attribute_group aer_stats_attr_group;
> +extern const struct attribute_group aer_attr_group;
>   void pci_aer_clear_fatal_status(struct pci_dev *dev);
>   int pci_aer_clear_status(struct pci_dev *dev);
>   int pci_aer_raw_clear_status(struct pci_dev *dev);
> diff --git a/drivers/pci/pcie/aer.c b/drivers/pci/pcie/aer.c
> index ebac126144fc..6c331695af58 100644
> --- a/drivers/pci/pcie/aer.c
> +++ b/drivers/pci/pcie/aer.c
> @@ -627,6 +627,111 @@ const struct attribute_group aer_stats_attr_group = {
>   	.is_visible = aer_stats_attrs_are_visible,
>   };
>   
> +/*
> + * Ratelimit interval
> + * <=0: disabled with ratelimit.interval = 0
> + * >0: enabled with ratelimit.interval in ms
> + */
> +#define aer_ratelimit_interval_attr(name, ratelimit)			\
> +	static ssize_t							\
> +	name##_show(struct device *dev, struct device_attribute *attr,	\
> +					 char *buf)			\
> +	{								\
> +		struct pci_dev *pdev = to_pci_dev(dev);			\
> +									\
> +		return sysfs_emit(buf, "%d\n",				\
> +				  pdev->aer_info->ratelimit.interval);	\
> +	}								\
> +									\
> +	static ssize_t							\
> +	name##_store(struct device *dev, struct device_attribute *attr, \
> +		     const char *buf, size_t count) 			\
> +	{								\
> +		struct pci_dev *pdev = to_pci_dev(dev);			\
> +		int interval;						\
> +									\
> +		if (!capable(CAP_SYS_ADMIN))				\
> +			return -EPERM;					\
> +									\
> +		if (kstrtoint(buf, 0, &interval) < 0)			\
> +			return -EINVAL;					\
> +									\
> +		if (interval <= 0)					\
> +			interval = 0;					\
> +		else							\
> +			interval = msecs_to_jiffies(interval); 		\
> +									\
> +		pdev->aer_info->ratelimit.interval = interval;		\
> +									\
> +		return count;						\
> +	}								\
> +	static DEVICE_ATTR_RW(name);
> +
> +#define aer_ratelimit_burst_attr(name, ratelimit)			\
> +	static ssize_t							\
> +	name##_show(struct device *dev, struct device_attribute *attr,	\
> +		    char *buf)						\
> +	{								\
> +		struct pci_dev *pdev = to_pci_dev(dev);			\
> +									\
> +		return sysfs_emit(buf, "%d\n",				\
> +				  pdev->aer_info->ratelimit.burst);	\
> +	}								\
> +									\
> +	static ssize_t							\
> +	name##_store(struct device *dev, struct device_attribute *attr,	\
> +		     const char *buf, size_t count)			\
> +	{								\
> +		struct pci_dev *pdev = to_pci_dev(dev);			\
> +		int burst;						\
> +									\
> +		if (!capable(CAP_SYS_ADMIN))				\
> +			return -EPERM;					\
> +									\
> +		if (kstrtoint(buf, 0, &burst) < 0)			\
> +			return -EINVAL;					\
> +									\
> +		pdev->aer_info->ratelimit.burst = burst;		\
> +									\
> +		return count;						\
> +	}								\
> +	static DEVICE_ATTR_RW(name);
> +
> +#define aer_ratelimit_attrs(name)					\
> +	aer_ratelimit_interval_attr(name##_ratelimit_interval_ms,	\
> +				    name##_ratelimit)			\
> +	aer_ratelimit_burst_attr(name##_ratelimit_burst,		\
> +				 name##_ratelimit)
> +
> +aer_ratelimit_attrs(correctable)
> +aer_ratelimit_attrs(nonfatal)
> +
> +static struct attribute *aer_attrs[] = {
> +	&dev_attr_correctable_ratelimit_interval_ms.attr,
> +	&dev_attr_correctable_ratelimit_burst.attr,
> +	&dev_attr_nonfatal_ratelimit_interval_ms.attr,
> +	&dev_attr_nonfatal_ratelimit_burst.attr,
> +	NULL
> +};
> +
> +static umode_t aer_attrs_are_visible(struct kobject *kobj,
> +				     struct attribute *a, int n)
> +{
> +	struct device *dev = kobj_to_dev(kobj);
> +	struct pci_dev *pdev = to_pci_dev(dev);
> +
> +	if (!pdev->aer_info)
> +		return 0;
> +
> +	return a->mode;
> +}
> +
> +const struct attribute_group aer_attr_group = {
> +	.name = "aer",
> +	.attrs = aer_attrs,
> +	.is_visible = aer_attrs_are_visible,
> +};
> +
>   static void pci_dev_aer_stats_incr(struct pci_dev *pdev,
>   				   struct aer_err_info *info)
>   {

-- 
Sathyanarayanan Kuppuswamy
Linux Kernel Developer