NMI reason 2d when running perf

NMI reason 2d when running perf

[Jens Axboe]

Hi,

When running perf on my Dell R7525 on a running process, I get a ton of:

[  504.234782] Dazed and confused, but trying to continue
[  504.267843] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.267846] Dazed and confused, but trying to continue
[  504.335975] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.335977] Dazed and confused, but trying to continue
[  504.368031] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.368033] Dazed and confused, but trying to continue
[  504.371037] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.371038] Dazed and confused, but trying to continue
[  504.439165] Uhhuh. NMI received for unknown reason 2d on CPU 48.
[  504.439167] Dazed and confused, but trying to continue

spew in dmesg. The box has 2x7763 CPUS. This seems to be a recent
regression, been using this box for a while and haven't seen this
before. The test being traced is pinned to CPU 48. The box is currently
running:

commit 6015b1aca1a233379625385feb01dd014aca60b5 (origin/master, origin/HEAD)
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date:   Tue Mar 14 19:32:38 2023 -0700

    sched_getaffinity: don't assume 'cpumask_size()' is fully initialized

with the pending block/io_uring branches merged in for testing.

-- 
Jens Axboe

Re: NMI reason 2d when running perf

[Dr. David Alan Gilbert]

* Jens Axboe (axboe@kernel.dk) wrote:
> Hi,
> 
> When running perf on my Dell R7525 on a running process, I get a ton of:
> 
> [  504.234782] Dazed and confused, but trying to continue
> [  504.267843] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [  504.267846] Dazed and confused, but trying to continue
> [  504.335975] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [  504.335977] Dazed and confused, but trying to continue
> [  504.368031] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [  504.368033] Dazed and confused, but trying to continue
> [  504.371037] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [  504.371038] Dazed and confused, but trying to continue
> [  504.439165] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [  504.439167] Dazed and confused, but trying to continue
> 
> spew in dmesg. The box has 2x7763 CPUS. This seems to be a recent
> regression, been using this box for a while and haven't seen this
> before. The test being traced is pinned to CPU 48. The box is currently
> running:

I'm not sure this is new; I've seen this intermittently on AMD boxes
for a few years. Always when running perf or related tools.

Dave

> commit 6015b1aca1a233379625385feb01dd014aca60b5 (origin/master, origin/HEAD)
> Author: Linus Torvalds <torvalds@linux-foundation.org>
> Date:   Tue Mar 14 19:32:38 2023 -0700
> 
>     sched_getaffinity: don't assume 'cpumask_size()' is fully initialized
> 
> with the pending block/io_uring branches merged in for testing.
> 
> -- 
> Jens Axboe
> 
-- 
 -----Open up your eyes, open up your mind, open up your code -------   
/ Dr. David Alan Gilbert    |       Running GNU/Linux       | Happy  \ 
\        dave @ treblig.org |                               | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/

Re: NMI reason 2d when running perf

[Ravi Bangoria]

On 17-Mar-23 7:55 PM, Jens Axboe wrote:
> Hi,
> 
> When running perf on my Dell R7525 on a running process, I get a ton of:
> 
> [  504.234782] Dazed and confused, but trying to continue
> [  504.267843] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [  504.267846] Dazed and confused, but trying to continue
> [  504.335975] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [  504.335977] Dazed and confused, but trying to continue
> [  504.368031] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [  504.368033] Dazed and confused, but trying to continue
> [  504.371037] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [  504.371038] Dazed and confused, but trying to continue
> [  504.439165] Uhhuh. NMI received for unknown reason 2d on CPU 48.
> [  504.439167] Dazed and confused, but trying to continue
> 
> spew in dmesg. The box has 2x7763 CPUS.

Are you using IBS directly or indirectly (perf record, if run as root
without explicit -e event, internally invokes IBS)? fwiw, Zen2 had
similar hw issue where IBS can raise NMI without sample valid bit set
while exiting from CC6 state[1]. But as per my knowledge, it's already
fixed in Zen3.

[1]: https://lore.kernel.org/all/YFDSSxftYw9tCGC6@krava/

> This seems to be a recent
> regression, been using this box for a while and haven't seen this
> before. The test being traced is pinned to CPU 48. The box is currently
> running:
> 
> commit 6015b1aca1a233379625385feb01dd014aca60b5 (origin/master, origin/HEAD)
> Author: Linus Torvalds <torvalds@linux-foundation.org>
> Date:   Tue Mar 14 19:32:38 2023 -0700
> 
>     sched_getaffinity: don't assume 'cpumask_size()' is fully initialized
> 
> with the pending block/io_uring branches merged in for testing.
>