[PATCH 0/1] alpha/perf: Enable duration-based sampling throttling

[PATCH 0/1] alpha/perf: Enable duration-based sampling throttling

[Magnus Lindholm]

Hi,

While testing Matt's v7 ProfileMe series [1] on my EV67 AlphaServer ES40,
I found that short-period sampling could make the machine unresponsive.
Once serial logging was enabled, the console showed RCU grace-period
kthread starvation reports.

Count-based throttling was working: at HZ=1024 and the default limit of
100000 samples/s, the observed cadence was approximately 98 interrupts
per throttle cycle. However, Alpha never called perf_sample_event_took(),
so perf could not reduce that limit based on interrupt handling cost.
Setting perf_cpu_time_max_percent to 25 did not provide that feedback.

Keeping period 17 but lowering the limit to 1024 or 4096 samples/s made
the tests pass while still exercising repeated stop/park/restart cycles.
Adding duration feedback also made the original high-rate test pass,
without changing those counter-state operations. The minimal fix measured
average handler costs of 24-26 us against a 2.5 us budget and reduced the
limit to 7168-8192 samples/s.

Reproducer
==========

The reproducer requires Matt's ProfileMe kernel series and matching perf
tools. On the ES40 I used the following one-second workload, built
natively as "busy":

  #include <stdint.h>
  #include <time.h>

  static double now(void)
  {
          struct timespec t;

          clock_gettime(CLOCK_MONOTONIC, &t);
          return t.tv_sec + t.tv_nsec * 1e-9;
  }

  int main(void)
  {
          uint64_t x = 1;
          double end = now() + 1;

          do {
                  for (int i = 0; i < 1000; i++)
                          asm volatile("mulq %0,3,%0" : "+r" (x));
          } while (now() < end);
          return 0;
  }

  cc -O2 -o busy busy.c

As root, with serial capture running:

  dmesg -n 7
  sysctl -w kernel.perf_event_max_sample_rate=100000
  sysctl -w kernel.perf_cpu_time_max_percent=25

The period-10007 control completed successfully:

  taskset -c 0 perf record -B -R \
      -e cpu/instructions,profileme,randomize/ -c 10007 \
      -o control.data -- ./busy

Changing only the period reproduced the failure:

  taskset -c 0 perf record -B -R \
      -e cpu/instructions,profileme,randomize/ -c 17 \
      -o period17.data -- ./busy

The unpatched runs required a manual reset; a userspace timeout did not
reliably recover the system. The commands above are the essential
recording commands from the controlled tests, which also used a timeout
supervisor and saved logs on another machine.

Validation and submission scope
===============================

The minimal fix, without diagnostic instrumentation, passed three
repeats of the high-rate reproducer and the two low-rate controls.
Broader testing covered 21 cases on all three CPUs: randomized ProfileMe
instruction sampling at periods 16, 17 and 31, ProfileMe and Aggregate
cycle sampling, and concurrent ProfileMe and mixed-mode workloads.
All completed, with 1352488 samples, no recorded losses and no new RCU
stalls or kernel faults. This was bounded stress testing, not a long soak.

The missing feedback predates ProfileMe. This patch is therefore based
on mainline and can be reviewed and applied independently of Matt's
series. I cross-built vmlinux, the Alpha boot image and modules on
mainline d48723711bae with the ES40 configuration and GCC 15.0.1. The
only build warning was the existing saved_config COMMON-symbol warning.

The hardware results above used Matt's v7 series with this fix applied;
the mainline-only build has not been boot-tested. RPCC supplies a short,
same-CPU interval because Alpha's tick-based sched_clock() is too coarse
for this measurement. Counter programming and event handling are unchanged.

[1] https://lore.kernel.org/linux-alpha/20261105-alpha-profileme-v7-0-c474e892abb0@gmail.com/

Thanks,
Magnus

Magnus Lindholm (1):
  alpha/perf: Account for time spent handling PMU interrupts

 arch/alpha/kernel/perf_event.c | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)


base-commit: d48723711bae9b8cdc56719c448ba42fef28cbee

[PATCH 1/1] alpha/perf: Account for time spent handling PMU interrupts

[Magnus Lindholm]

Alpha does not report PMU interrupt duration to perf, preventing its
adaptive throttling from reducing an excessively costly sampling rate.

Measure handler time with RPCC and report it to perf_sample_event_took().
Convert cycles using the calibrated frequency selected by time_init();
Alpha's tick-based sched_clock() is too coarse for this measurement.
Wrap the existing handler so all return paths are accounted for, without
changing counter programming or event handling.

Signed-off-by: Magnus Lindholm <linmag7@gmail.com>
---
 arch/alpha/kernel/perf_event.c | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/arch/alpha/kernel/perf_event.c b/arch/alpha/kernel/perf_event.c
index a3eaab094ece..dd3bc70b71d7 100644
--- a/arch/alpha/kernel/perf_event.c
+++ b/arch/alpha/kernel/perf_event.c
@@ -16,8 +16,10 @@
 #include <linux/kdebug.h>
 #include <linux/mutex.h>
 #include <linux/init.h>
+#include <linux/math64.h>
 
 #include <asm/hwrpb.h>
+#include <asm/timex.h>
 #include <linux/atomic.h>
 #include <asm/irq.h>
 #include <asm/irq_regs.h>
@@ -26,6 +28,9 @@
 #include <asm/hw_irq.h>
 
 
+extern unsigned long est_cycle_freq;
+static unsigned long pmu_cycle_freq __read_mostly;
+
 /* The maximum number of PMCs on any Alpha CPU whatsoever. */
 #define MAX_HWEVENTS 3
 #define PMC_NO_INDEX -1
@@ -797,8 +802,8 @@ void perf_event_print_debug(void)
  * Performance Monitoring Interrupt Service Routine called when a PMC
  * overflows.  The PMC that overflowed is passed in la_ptr.
  */
-static void alpha_perf_event_irq_handler(unsigned long la_ptr,
-					struct pt_regs *regs)
+static void alpha_perf_event_irq_inner(unsigned long la_ptr,
+				       struct pt_regs *regs)
 {
 	struct cpu_hw_events *cpuc;
 	struct perf_sample_data data;
@@ -862,6 +867,21 @@ static void alpha_perf_event_irq_handler(unsigned long la_ptr,
 
 
 
+/* sched_clock() is tick-based on Alpha; measure a short same-CPU interval. */
+static void alpha_perf_event_irq_handler(unsigned long la_ptr,
+					 struct pt_regs *regs)
+{
+	u32 start = get_cycles();
+	u32 cycles;
+
+	alpha_perf_event_irq_inner(la_ptr, regs);
+	/* Unsigned subtraction handles a wrap of the low 32-bit counter. */
+	cycles = (u32)get_cycles() - start;
+	if (likely(pmu_cycle_freq))
+		perf_sample_event_took(div64_u64((u64)cycles * NSEC_PER_SEC,
+						 pmu_cycle_freq));
+}
+
 /*
  * Init call to initialise performance events at kernel startup.
  */
@@ -878,6 +898,8 @@ static int __init init_hw_perf_events(void)
 
 	/* Override performance counter IRQ vector */
 
+	/* Match the calibrated frequency selected by time_init(). */
+	pmu_cycle_freq = est_cycle_freq ? est_cycle_freq : hwrpb->cycle_freq;
 	perf_irq = alpha_perf_event_irq_handler;
 
 	/* And set up PMU specification */