Re: perf regression. Was: [PATCH V4 01/16] perf: Fix the throttle logic for a group

["Liang, Kan" <kan.liang@linux.intel.com>]

Hi Alexei,

On 2025-06-01 8:30 p.m., Alexei Starovoitov wrote:
> On Tue, May 20, 2025 at 11:16:29AM -0700, kan.liang@linux.intel.com wrote:
>> From: Kan Liang <kan.liang@linux.intel.com>
>>
>> The current throttle logic doesn't work well with a group, e.g., the
>> following sampling-read case.
>>
>> $ perf record -e "{cycles,cycles}:S" ...
>>
>> $ perf report -D | grep THROTTLE | tail -2
>>             THROTTLE events:        426  ( 9.0%)
>>           UNTHROTTLE events:        425  ( 9.0%)
>>
>> $ perf report -D | grep PERF_RECORD_SAMPLE -a4 | tail -n 5
>> 0 1020120874009167 0x74970 [0x68]: PERF_RECORD_SAMPLE(IP, 0x1):
>> ... sample_read:
>> .... group nr 2
>> ..... id 0000000000000327, value 000000000cbb993a, lost 0
>> ..... id 0000000000000328, value 00000002211c26df, lost 0
>>
>> The second cycles event has a much larger value than the first cycles
>> event in the same group.
>>
>> The current throttle logic in the generic code only logs the THROTTLE
>> event. It relies on the specific driver implementation to disable
>> events. For all ARCHs, the implementation is similar. Only the event is
>> disabled, rather than the group.
>>
>> The logic to disable the group should be generic for all ARCHs. Add the
>> logic in the generic code. The following patch will remove the buggy
>> driver-specific implementation.
>>
>> The throttle only happens when an event is overflowed. Stop the entire
>> group when any event in the group triggers the throttle.
>> The MAX_INTERRUPTS is set to all throttle events.
>>
>> The unthrottled could happen in 3 places.
>> - event/group sched. All events in the group are scheduled one by one.
>>   All of them will be unthrottled eventually. Nothing needs to be
>>   changed.
>> - The perf_adjust_freq_unthr_events for each tick. Needs to restart the
>>   group altogether.
>> - The __perf_event_period(). The whole group needs to be restarted
>>   altogether as well.
>>
>> With the fix,
>> $ sudo perf report -D | grep PERF_RECORD_SAMPLE -a4 | tail -n 5
>> 0 3573470770332 0x12f5f8 [0x70]: PERF_RECORD_SAMPLE(IP, 0x2):
>> ... sample_read:
>> .... group nr 2
>> ..... id 0000000000000a28, value 00000004fd3dfd8f, lost 0
>> ..... id 0000000000000a29, value 00000004fd3dfd8f, lost 0
>>
>> Suggested-by: Peter Zijlstra (Intel) <peterz@infradead.org>
>> Signed-off-by: Kan Liang <kan.liang@linux.intel.com>
>> ---
>>  kernel/events/core.c | 66 ++++++++++++++++++++++++++++++--------------
>>  1 file changed, 46 insertions(+), 20 deletions(-)
> 
> This patch breaks perf hw events somehow.
> 
> After merging this into bpf trees we see random "watchdog: BUG: soft lockup"
> with various stack traces followed up:
> [   78.620749] Sending NMI from CPU 8 to CPUs 0:
> [   76.387722] NMI backtrace for cpu 0
> [   76.387722] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G           O L      6.15.0-10818-ge0f0ee1c31de #1163 PREEMPT
> [   76.387722] Tainted: [O]=OOT_MODULE, [L]=SOFTLOCKUP
> [   76.387722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
> [   76.387722] RIP: 0010:_raw_spin_lock_irqsave+0xc/0x40
> [   76.387722] Call Trace:
> [   76.387722]  <IRQ>
> [   76.387722]  hrtimer_try_to_cancel.part.0+0x24/0xe0
> [   76.387722]  hrtimer_cancel+0x21/0x40
> [   76.387722]  cpu_clock_event_stop+0x64/0x70


The issues should be fixed by the patch.
https://lore.kernel.org/lkml/20250528175832.2999139-1-kan.liang@linux.intel.com/

Could you please give it a try?

Thanks,
Kan


> [   76.387722]  __perf_event_account_interrupt+0xcf/0x140
> [   76.387722]  __perf_event_overflow+0x36/0x340
> [   76.387722]  ? hrtimer_start_range_ns+0x2c1/0x420
> [   76.387722]  ? kvm_sched_clock_read+0x11/0x20
> [   76.387722]  perf_swevent_hrtimer+0xaf/0x100
> [   76.387722]  ? cpu_clock_event_add+0x6e/0x90
> [   76.387722]  ? event_sched_in+0xc3/0x190
> [   76.387722]  ? update_load_avg+0x87/0x3d0
> [   76.387722]  ? _raw_spin_unlock+0xe/0x20
> [   76.387722]  ? sched_balance_update_blocked_averages+0x59b/0x6a0
> [   76.387722]  ? ctx_sched_in+0x184/0x210
> [   76.387722]  ? kvm_sched_clock_read+0x11/0x20
> [   76.387722]  ? sched_clock_cpu+0x55/0x190
> [   76.387722]  ? perf_exclude_event+0x50/0x50
> [   76.387722]  __hrtimer_run_queues+0x111/0x290
> [   76.387722]  hrtimer_interrupt+0xff/0x240
> [   76.387722]  __sysvec_apic_timer_interrupt+0x4f/0x110
> [   76.387722]  sysvec_apic_timer_interrupt+0x6c/0x90
> 
> After reverting:
> commit e800ac51202f ("perf: Only dump the throttle log for the leader")
> commit 9734e25fbf5a ("perf: Fix the throttle logic for a group")
> everything is back to normal.
> 
> There are many ways to reproduce.
> Any test that sets up perf hw event followed up by tests that IPIs all cpus.
> One way:
> selftests/bpf/test_progs -t stacktrace_build_id_nmi
> selftests/bpf/test_progs -t unpriv_bpf_disabled
> 
> Please take a look.
>