[PATCH v2 0/2] sysfs: add counters for lockups and stalls

[PATCH v2 0/2] sysfs: add counters for lockups and stalls

[Max Kellermann]

Commits 9db89b411170 ("exit: Expose "oops_count" to sysfs") and
8b05aa263361 ("panic: Expose "warn_count" to sysfs") added counters
for oopses and warnings to sysfs, and these two patches do the same
for hard/soft lockups and RCU stalls.

All of these counters are useful for monitoring tools to detect
whether the machine is healthy.  If the kernel has experienced a
lockup or a stall, it's probably due to a kernel bug, and I'd like to
detect that quickly and easily.  There is currently no way to detect
that, other than parsing dmesg.  Or observing indirect effects: such
as certain tasks not responding, but then I need to observe all tasks,
and it may take a while until these effects become visible/measurable.
I'd rather be able to detect the primary cause more quickly, possibly
before everything falls apart.

Max Kellermann (2):
  kernel/watchdog: add /sys/kernel/{hard,soft}lockup_count
  kernel/rcu/tree_stall: add /sys/kernel/rcu_stall_count

 .../ABI/testing/sysfs-kernel-hardlockup_count |  7 +++
 .../ABI/testing/sysfs-kernel-rcu_stall_count  |  6 +++
 .../ABI/testing/sysfs-kernel-softlockup_count |  7 +++
 kernel/rcu/tree_stall.h                       | 26 +++++++++
 kernel/watchdog.c                             | 53 +++++++++++++++++++
 5 files changed, 99 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-hardlockup_count
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-rcu_stall_count
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-softlockup_count

-- 
2.47.2

[PATCH v2 1/2] kernel/watchdog: add /sys/kernel/{hard,soft}lockup_count

[Max Kellermann]

There is /proc/sys/kernel/hung_task_detect_count,
/sys/kernel/warn_count and /sys/kernel/oops_count but there is no
userspace-accessible counter for hard/soft lockups.  Having this is
useful for monitoring tools.

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
v1 -> v2: added documentation; added patch set cover letter with
  justification
---
 .../ABI/testing/sysfs-kernel-hardlockup_count |  7 +++
 .../ABI/testing/sysfs-kernel-softlockup_count |  7 +++
 kernel/watchdog.c                             | 53 +++++++++++++++++++
 3 files changed, 67 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-hardlockup_count
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-softlockup_count

diff --git a/Documentation/ABI/testing/sysfs-kernel-hardlockup_count b/Documentation/ABI/testing/sysfs-kernel-hardlockup_count
new file mode 100644
index 000000000000..dfdd4078b077
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-kernel-hardlockup_count
@@ -0,0 +1,7 @@
+What:		/sys/kernel/hardlockup_count
+Date:		May 2025
+KernelVersion:	6.16
+Contact:	Linux kernel mailing list <linux-kernel@vger.kernel.org>
+Description:
+		Shows how many times the system has detected a hard lockup since last boot.
+		Available only if CONFIG_HARDLOCKUP_DETECTOR is enabled.
diff --git a/Documentation/ABI/testing/sysfs-kernel-softlockup_count b/Documentation/ABI/testing/sysfs-kernel-softlockup_count
new file mode 100644
index 000000000000..337ff5531b5f
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-kernel-softlockup_count
@@ -0,0 +1,7 @@
+What:		/sys/kernel/softlockup_count
+Date:		May 2025
+KernelVersion:	6.16
+Contact:	Linux kernel mailing list <linux-kernel@vger.kernel.org>
+Description:
+		Shows how many times the system has detected a soft lockup since last boot.
+		Available only if CONFIG_SOFTLOCKUP_DETECTOR is enabled.
diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index 9fa2af9dbf2c..09994bfb47af 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -63,6 +63,29 @@ int __read_mostly sysctl_hardlockup_all_cpu_backtrace;
  */
 unsigned int __read_mostly hardlockup_panic =
 			IS_ENABLED(CONFIG_BOOTPARAM_HARDLOCKUP_PANIC);
+
+#ifdef CONFIG_SYSFS
+
+static unsigned int hardlockup_count;
+
+static ssize_t hardlockup_count_show(struct kobject *kobj, struct kobj_attribute *attr,
+				     char *page)
+{
+	return sysfs_emit(page, "%u\n", hardlockup_count);
+}
+
+static struct kobj_attribute hardlockup_count_attr = __ATTR_RO(hardlockup_count);
+
+static __init int kernel_hardlockup_sysfs_init(void)
+{
+	sysfs_add_file_to_group(kernel_kobj, &hardlockup_count_attr.attr, NULL);
+	return 0;
+}
+
+late_initcall(kernel_hardlockup_sysfs_init);
+
+#endif // CONFIG_SYSFS
+
 /*
  * We may not want to enable hard lockup detection by default in all cases,
  * for example when running the kernel as a guest on a hypervisor. In these
@@ -169,6 +192,10 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)
 		unsigned int this_cpu = smp_processor_id();
 		unsigned long flags;
 
+#ifdef CONFIG_SYSFS
+		++hardlockup_count;
+#endif
+
 		/* Only print hardlockups once. */
 		if (per_cpu(watchdog_hardlockup_warned, cpu))
 			return;
@@ -311,6 +338,28 @@ unsigned int __read_mostly softlockup_panic =
 static bool softlockup_initialized __read_mostly;
 static u64 __read_mostly sample_period;
 
+#ifdef CONFIG_SYSFS
+
+static unsigned int softlockup_count;
+
+static ssize_t softlockup_count_show(struct kobject *kobj, struct kobj_attribute *attr,
+				     char *page)
+{
+	return sysfs_emit(page, "%u\n", softlockup_count);
+}
+
+static struct kobj_attribute softlockup_count_attr = __ATTR_RO(softlockup_count);
+
+static __init int kernel_softlockup_sysfs_init(void)
+{
+	sysfs_add_file_to_group(kernel_kobj, &softlockup_count_attr.attr, NULL);
+	return 0;
+}
+
+late_initcall(kernel_softlockup_sysfs_init);
+
+#endif // CONFIG_SYSFS
+
 /* Timestamp taken after the last successful reschedule. */
 static DEFINE_PER_CPU(unsigned long, watchdog_touch_ts);
 /* Timestamp of the last softlockup report. */
@@ -742,6 +791,10 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 	touch_ts = __this_cpu_read(watchdog_touch_ts);
 	duration = is_softlockup(touch_ts, period_ts, now);
 	if (unlikely(duration)) {
+#ifdef CONFIG_SYSFS
+		++softlockup_count;
+#endif
+
 		/*
 		 * Prevent multiple soft-lockup reports if one cpu is already
 		 * engaged in dumping all cpu back traces.
-- 
2.47.2

[PATCH v2 2/2] kernel/rcu/tree_stall: add /sys/kernel/rcu_stall_count

[Max Kellermann]

Exposing a simple counter to userspace for monitoring tools.

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
v1 -> v2: added documentation
---
 .../ABI/testing/sysfs-kernel-rcu_stall_count  |  6 +++++
 kernel/rcu/tree_stall.h                       | 26 +++++++++++++++++++
 2 files changed, 32 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-rcu_stall_count

diff --git a/Documentation/ABI/testing/sysfs-kernel-rcu_stall_count b/Documentation/ABI/testing/sysfs-kernel-rcu_stall_count
new file mode 100644
index 000000000000..a4a97a7f4a4d
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-kernel-rcu_stall_count
@@ -0,0 +1,6 @@
+What:		/sys/kernel/rcu_stall_count
+Date:		May 2025
+KernelVersion:	6.16
+Contact:	Linux kernel mailing list <linux-kernel@vger.kernel.org>
+Description:
+		Shows how many times the system has detected an RCU stall since last boot.
diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
index 925fcdad5dea..158330524795 100644
--- a/kernel/rcu/tree_stall.h
+++ b/kernel/rcu/tree_stall.h
@@ -20,6 +20,28 @@
 int sysctl_panic_on_rcu_stall __read_mostly;
 int sysctl_max_rcu_stall_to_panic __read_mostly;
 
+#ifdef CONFIG_SYSFS
+
+static unsigned int rcu_stall_count;
+
+static ssize_t rcu_stall_count_show(struct kobject *kobj, struct kobj_attribute *attr,
+				    char *page)
+{
+	return sysfs_emit(page, "%u\n", rcu_stall_count);
+}
+
+static struct kobj_attribute rcu_stall_count_attr = __ATTR_RO(rcu_stall_count);
+
+static __init int kernel_rcu_stall_sysfs_init(void)
+{
+	sysfs_add_file_to_group(kernel_kobj, &rcu_stall_count_attr.attr, NULL);
+	return 0;
+}
+
+late_initcall(kernel_rcu_stall_sysfs_init);
+
+#endif // CONFIG_SYSFS
+
 #ifdef CONFIG_PROVE_RCU
 #define RCU_STALL_DELAY_DELTA		(5 * HZ)
 #else
@@ -784,6 +806,10 @@ static void check_cpu_stall(struct rcu_data *rdp)
 		if (kvm_check_and_clear_guest_paused())
 			return;
 
+#ifdef CONFIG_SYSFS
+		++rcu_stall_count;
+#endif
+
 		rcu_stall_notifier_call_chain(RCU_STALL_NOTIFY_NORM, (void *)j - gps);
 		if (READ_ONCE(csd_lock_suppress_rcu_stall) && csd_lock_is_stuck()) {
 			pr_err("INFO: %s detected stall, but suppressed full report due to a stuck CSD-lock.\n", rcu_state.name);
-- 
2.47.2

Re: [PATCH v2 2/2] kernel/rcu/tree_stall: add /sys/kernel/rcu_stall_count

[Sourabh Jain]

Hello Andrew,

On 04/05/25 23:38, Max Kellermann wrote:
> Exposing a simple counter to userspace for monitoring tools.
>
> Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
> ---
> v1 -> v2: added documentation
> ---
>   .../ABI/testing/sysfs-kernel-rcu_stall_count  |  6 +++++
>   kernel/rcu/tree_stall.h                       | 26 +++++++++++++++++++
>   2 files changed, 32 insertions(+)
>   create mode 100644 Documentation/ABI/testing/sysfs-kernel-rcu_stall_count
>
> diff --git a/Documentation/ABI/testing/sysfs-kernel-rcu_stall_count b/Documentation/ABI/testing/sysfs-kernel-rcu_stall_count
> new file mode 100644
> index 000000000000..a4a97a7f4a4d
> --- /dev/null
> +++ b/Documentation/ABI/testing/sysfs-kernel-rcu_stall_count
> @@ -0,0 +1,6 @@
> +What:		/sys/kernel/rcu_stall_count
> +Date:		May 2025
> +KernelVersion:	6.16
> +Contact:	Linux kernel mailing list <linux-kernel@vger.kernel.org>
> +Description:
> +		Shows how many times the system has detected an RCU stall since last boot.
> diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
> index 925fcdad5dea..158330524795 100644
> --- a/kernel/rcu/tree_stall.h
> +++ b/kernel/rcu/tree_stall.h
> @@ -20,6 +20,28 @@
>   int sysctl_panic_on_rcu_stall __read_mostly;
>   int sysctl_max_rcu_stall_to_panic __read_mostly;
>   
> +#ifdef CONFIG_SYSFS
> +
> +static unsigned int rcu_stall_count;
> +
> +static ssize_t rcu_stall_count_show(struct kobject *kobj, struct kobj_attribute *attr,
> +				    char *page)
> +{
> +	return sysfs_emit(page, "%u\n", rcu_stall_count);
> +}
> +
> +static struct kobj_attribute rcu_stall_count_attr = __ATTR_RO(rcu_stall_count);
> +
> +static __init int kernel_rcu_stall_sysfs_init(void)
> +{
> +	sysfs_add_file_to_group(kernel_kobj, &rcu_stall_count_attr.attr, NULL);
> +	return 0;
> +}
> +
> +late_initcall(kernel_rcu_stall_sysfs_init);
> +
> +#endif // CONFIG_SYSFS
> +
>   #ifdef CONFIG_PROVE_RCU
>   #define RCU_STALL_DELAY_DELTA		(5 * HZ)
>   #else
> @@ -784,6 +806,10 @@ static void check_cpu_stall(struct rcu_data *rdp)
>   		if (kvm_check_and_clear_guest_paused())
>   			return;
>   
> +#ifdef CONFIG_SYSFS
> +		++rcu_stall_count;
> +#endif
> +
>   		rcu_stall_notifier_call_chain(RCU_STALL_NOTIFY_NORM, (void *)j - gps);
>   		if (READ_ONCE(csd_lock_suppress_rcu_stall) && csd_lock_is_stuck()) {
>   			pr_err("INFO: %s detected stall, but suppressed full report due to a stuck CSD-lock.\n", rcu_state.name);

It seems like this patch was not applied properly to the upstream tree.

Out of the three hunks in this patch, only the first one is applied; the 
second
and third hunks are missing.

commit 2536c5c7d6ae5e1d844aa21f28b326b5e7f815ef
Author: Max Kellermann <max.kellermann@ionos.com>
Date:   Sun May 4 20:08:31 2025 +0200

     kernel/rcu/tree_stall: add /sys/kernel/rcu_stall_count

     Expose a simple counter to userspace for monitoring tools.


Thanks,
Sourabh Jain

Re: [PATCH v2 2/2] kernel/rcu/tree_stall: add /sys/kernel/rcu_stall_count

[Andrew Morton]

On Tue, 3 Jun 2025 22:09:30 +0530 Sourabh Jain <sourabhjain@linux.ibm.com> wrote:

> Hello Andrew,
> 
> > +#endif
> > +
> >   		rcu_stall_notifier_call_chain(RCU_STALL_NOTIFY_NORM, (void *)j - gps);
> >   		if (READ_ONCE(csd_lock_suppress_rcu_stall) && csd_lock_is_stuck()) {
> >   			pr_err("INFO: %s detected stall, but suppressed full report due to a stuck CSD-lock.\n", rcu_state.name);
> 
> It seems like this patch was not applied properly to the upstream tree.
> 
> Out of the three hunks in this patch, only the first one is applied; the 
> second
> and third hunks are missing.
> 
> commit 2536c5c7d6ae5e1d844aa21f28b326b5e7f815ef
> Author: Max Kellermann <max.kellermann@ionos.com>
> Date:   Sun May 4 20:08:31 2025 +0200
> 
>      kernel/rcu/tree_stall: add /sys/kernel/rcu_stall_count
> 
>      Expose a simple counter to userspace for monitoring tools.

OK.  iirc there was quite a lot of churn and conflicts here :)

Please send a fixup against latest -linus?

Re: [PATCH v2 2/2] kernel/rcu/tree_stall: add /sys/kernel/rcu_stall_count

[Sourabh Jain]

On 04/06/25 05:46, Andrew Morton wrote:
> On Tue, 3 Jun 2025 22:09:30 +0530 Sourabh Jain <sourabhjain@linux.ibm.com> wrote:
>
>> Hello Andrew,
>>
>>> +#endif
>>> +
>>>    		rcu_stall_notifier_call_chain(RCU_STALL_NOTIFY_NORM, (void *)j - gps);
>>>    		if (READ_ONCE(csd_lock_suppress_rcu_stall) && csd_lock_is_stuck()) {
>>>    			pr_err("INFO: %s detected stall, but suppressed full report due to a stuck CSD-lock.\n", rcu_state.name);
>> It seems like this patch was not applied properly to the upstream tree.
>>
>> Out of the three hunks in this patch, only the first one is applied; the
>> second
>> and third hunks are missing.
>>
>> commit 2536c5c7d6ae5e1d844aa21f28b326b5e7f815ef
>> Author: Max Kellermann <max.kellermann@ionos.com>
>> Date:   Sun May 4 20:08:31 2025 +0200
>>
>>       kernel/rcu/tree_stall: add /sys/kernel/rcu_stall_count
>>
>>       Expose a simple counter to userspace for monitoring tools.
> OK.  iirc there was quite a lot of churn and conflicts here :)
>
> Please send a fixup against latest -linus?

Sure, I will wait for a day or two to see if Max is interested in 
sending the fix-up patch. Otherwise, I will send it.

Thanks,
Sourabh Jain