perf/amd/ibs: [BUG?] ibs_op sampling silently stops on Zen3: MSR_AMD64_IBSOPCTL latched, no NMI delivered

perf/amd/ibs: [BUG?] ibs_op sampling silently stops on Zen3: MSR_AMD64_IBSOPCTL latched, no NMI delivered

[Liew Rui Yan]

Hi Perf Community,


I found an issue with ibs_op sampling that I can reproduce on my AMD
Zen3 laptop.  Under full CPU load, sampling PMIs stop arriving on each
CPU, one after another.  Reading MSR_AMD64_IBSOPCTL (0xc0011033) while
perf is still running shows the hardware has latched a completed sample
(IBS_OP_ENABLE=1, IBS_OP_VAL=1, IBS_OP_CUR_CNT frozen) and is waiting
for an NMI that never arrives.  The CPU never recovers on its own.

I am really not very familiar with CPU, so I would appreciate any
guidance on whether this is known and what to test next.


Environment
===========

- CPU:    AMD Ryzen 5 5600H with Radeon Graphics (family 25 model 80,
          Zen3/Cezanne), 12 cores, microcode 0xa500012
- Kernel: 7.3.0-rc5-mm-new-damon-1 (self-built from the mm tree)
- perf:   perf version 7.2.11-1
- Also running: stress-ng 0.21.01


How to reproduce
================

Reproducer is here:

    https://github.com/aethernet65535/amd-zen3-nmi-lost-reproducer.git

The essence is a system-wide ibs_op record on top of a full CPU load:

    perf record -a -e ibs_op// -c 100000 -- stress-ng --cpu 0 &

and then watch the per-CPU PMI counters in /proc/interrupts once a
second.  To rule perf's dynamic throttling out as a factor, I set
kernel.perf_event_max_sample_rate=100000 and
kernel.perf_cpu_time_max_percent=0 before starting.

Reproduction is fast and consistent: the first CPU goes silent ~20 s
after perf starts, and all 12 CPUs are silent within ~5 minutes.


Observed behaviour
==================

Found that per-CPU PMI counters (/proc/interrupts) stop incrementing.
From the most recent run, time from perf start until each CPU was
detected silent (zero PMI delta for 3 consecutive 1 s windows):

    CPU 1,5:        +23.7 s
    CPU 2,3:        +31.6 s
    CPU 8:          +62.2 s
    CPU 9:          +68.3 s
    CPU 10:         +96.9 s
    CPU 6:          +166.9 s
    CPU 4:          +192.5 s
    CPU 0:          +198.5 s
    CPU 11:         +220.0 s
    CPU 7:          +271.3 s

The run then continued for another 120 s to confirm if CPU comes back;
none did.

For each CPU at the moment it went silent, I read MSR_AMD64_IBSOPCTL
three times, via rdmsr while perf was still running.  Decoding
IBS_OP_ENABLE (bit 17), IBS_OP_VAL (bit 18) and IBS_OP_CUR_CNT (bits
32-58):

Live control CPU 0 -- IBS_OP_VAL=0, IBS_OP_CUR_CNT keeps moving between
reads:

  read 1: raw=0x231f0002186a  IBS_OP_ENABLE=1 IBS_OP_VAL=0 IBS_OP_CUR_CNT=8991
  read 2: raw=0x10ca50002186a  IBS_OP_ENABLE=1 IBS_OP_VAL=0 IBS_OP_CUR_CNT=68773
  read 3: raw=0x15fa10002186a  IBS_OP_ENABLE=1 IBS_OP_VAL=0 IBS_OP_CUR_CNT=90017

Silent CPU 0 -- three reads are bit-for-bit identical:

  read 1: raw=0xc80006186a  IBS_OP_ENABLE=1 IBS_OP_VAL=1 IBS_OP_CUR_CNT=200
  read 2: raw=0xc80006186a  IBS_OP_ENABLE=1 IBS_OP_VAL=1 IBS_OP_CUR_CNT=200
  read 3: raw=0xc80006186a  IBS_OP_ENABLE=1 IBS_OP_VAL=1 IBS_OP_CUR_CNT=200

perf report --header-only excerpt (trimmed):

    # os release : 7.3.0-rc5-mm-new-damon-1
    # perf version : 7.2.11-1
    # arch : x86_64
    # nrcpus online : 12
    # cpudesc : AMD Ryzen 5 5600H with Radeon Graphics
    # cpuid : AuthenticAMD,25,80,0
    # event : name = ibs_op//, , type = 11 (ibs_op), config = 0,
      { sample_period, sample_freq } = 100000,
      sample_type = IP|TID|TIME|ID|CPU, read_format = ID|LOST
    # pmu mappings: ... ibs_fetch = 10, ibs_op = 11 ...
    # cpu pmu capabilities: max_precise=0


What I think I have ruled out
=============================

- Not perf's dynamic throttling: it is disabled (percent=0) for every
  run.
- Not the kernel stopping the event: IBS_OP_ENABLE stays 1; the event is
  still armed.
- Sampling does work normally for a while before it stops, so the setup
  itself is fine.


One more observation
====================

In earlier testing I noticed there is a workaround -- enable NMI
watchdog.  PMI counters start incrementing again after it is enabled.


Questions
=========

1. Is this known on Zen3/Cezanne?
2. Any recommended next step?

Thank you for your time, and for all your work on perf.

Best regards,
Rui Yan