[RFC PATCH 0/1] psi: Introduce in-kernel PSI auto monitor feature

[Pintu Kumar Agarwal <pintu.agarwal@oss.qualcomm.com>]

Hi all,

This RFC introduces an in-kernel PSI auto monitor aimed at improving
root-cause visibility for resource pressure events in Linux systems.

Motivation:

PSI already provides an excellent mechanism to detect CPU, memory and
I/O pressure and includes trigger-based notifications via pollable
interfaces. However, it deliberately avoids attributing pressure to
individual tasks.

In real-world systems, this creates a gap: when a PSI trigger fires,
users still need to determine *which tasks caused the stall* by combining
multiple tools (top, meminfo, vmstat, perf, tracing, etc.), often after
the event has already passed.

This process becomes particularly difficult during:
- transient bursts of pressure
- system boot or early initialization before user space
- PREEMPT_RT or latency-sensitive workloads
- heavily loaded embedded systems where user space is delayed
- small resource-constraints minimal system
- production system where most debugging interface are disabled

Proposal:

This patch introduces an optional in-kernel PSI auto monitor that:
- periodically samples PSI signals
- detects threshold breaches
- captures top contributing tasks at that moment
- emits trace events and kernel logs for analysis

The design goal is **low-latency attribution at the source of truth**,
without relying on user-space daemons or polling loops.

Why in-kernel?

While similar logic can be implemented in user space, there are inherent
limitations:

- scheduling delays under high pressure
- risk of missing short-lived spikes
- dependency on continuous polling or daemons
- difficulty deploying in early boot or minimal environments

In contrast, the in-kernel approach:
- observes PSI signals without scheduling latency
- captures contributors exactly at threshold breach
- works during early boot and degraded system states
- avoids duplicating logic across multiple user-space tools
- easy configurable even in runtime
- captures all sorts of information during same timestamp

Design Highlights:

- Does not modify PSI fast paths
- Optional (CONFIG_PSI_AUTO_MONITOR)
- Runtime configurable thresholds and interval
- Uses existing kernel accounting (task runtime, RSS, I/O stats)
- Provides structured tracepoints for post-processing
- Lightweight and intended for diagnostic use
- Idea is similar to, when OOM occurs dump contending tasks

Reviews and Assistance:

The core idea is mine.
However, I have taken few assistance from AI for review and enhancement.
I have done extensive review and suggestion using ChatGPT and Copilot.
The commit message and this cover letter were also prepared by Copilot.
I have done self-review and corrective actions accordingly.

Experimental Validation:

The feature has been evaluated on multiple ARM64 platforms (Cortex-A53,
A55) across different kernels and storage setups.
Extensive experiments has been carried out with multiple workloads.
Some tools and logs are shared here:
https://github.com/pintuk/KERNEL/tree/master/PSI_WORK

Test scenarios include:
- CPU/memory/IO stress workloads both on eMMC and NAND
- system boot tracing (no external tools)
- mixed workloads (stress-ng, workqueues, user/kernel threads, processes)
- PREEMPT_RT cyclictest correlation with real workloads

Results show:
- consistent identification of top resource contributors
- improved root-cause visibility compared to user-space-only methods
- ability to capture transient hotspots during boot and runtime
- correlation of latency spikes with system pressure

Papers and Reference:

The paper is presented in Open Source Summit India - 2026:
https://ossindia2026.sched.com/event/2KNI4/introducing-in-kernel-psi-auto-monitor-feature-pintu-kumar-agarwal-qualcomm?iframe=yes&w=100%&sidebar=yes&bg=no
https://hosted-files.sched.co/ossindia2026/19/OSS-IND-26-PSI-Auto-Monitor.pdf
The initial idea was also presented in LPC-2024:
https://lpc.events/event/18/contributions/1884/attachments/1439/3069/LPC2024_PIntu_PSI.pdf

Open Questions (RFC):

Feedback is especially appreciated on:

- whether this functionality belongs in-kernel vs user-space
- interface choice (sysfs vs tracefs/debugfs alternatives)
- scoring heuristic (CPU/RSS/IO weighting)
- potential reuse or extension of existing PSI interfaces
- cgroup-aware extensions for future work

Future Work:

- finer-grained PSI window integration
- IRQ pressure support
- cgroup-based attribution
- improved tracing/export interfaces
- optional integration with user-space analysis tools

Thanks for your time, and I’d really appreciate feedback.

Regards,
Pintu Kumar Agarwal

Pintu Kumar Agarwal (1):
  psi: Introduce in-kernel PSI auto monitor feature

 include/trace/events/psi_monitor.h |  53 +++++
 init/Kconfig                       |  16 ++
 kernel/sched/build_utility.c       |   4 +
 kernel/sched/psi_monitor.c         | 307 +++++++++++++++++++++++++++++
 4 files changed, 380 insertions(+)
 create mode 100644 include/trace/events/psi_monitor.h
 create mode 100644 kernel/sched/psi_monitor.c

-- 
2.34.1