[Bug 222109] New: [amdgpu Picasso 1002:15d8] gfxhub UTCL2 SQC permission fault -> ring gfx timeout when DPM forced high (repro on LTS 6.18 too)

[Bug 222109] New: [amdgpu Picasso 1002:15d8] gfxhub UTCL2 SQC permission fault -> ring gfx timeout when DPM forced high (repro on LTS 6.18 too)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=222109

            Bug ID: 222109
           Summary: [amdgpu Picasso 1002:15d8] gfxhub UTCL2 SQC permission
                    fault -> ring gfx timeout when DPM forced high (repro
                    on LTS 6.18 too)
           Product: Drivers
           Version: 2.5
          Hardware: x86-64
                OS: Linux
            Status: NEW
          Severity: high
          Priority: P3
         Component: Video(DRI - non Intel)
          Assignee: drivers_video-dri@kernel-bugs.osdl.org
          Reporter: aleksey16.11.2000@gmail.com
        Regression: No

Summary:
amdgpu Picasso/Raven2 [1002:15d8] wedges with gfxhub UTCL2 SQC permission fault
followed by ring gfx timeout when DPM forced to high. Deterministic on cold
boot + immediate GUI. Also reproduces on Arch LTS 6.18.55, not
CachyOS-specific. TLP/manual `high` is only the trigger, kernel should not
wedge.

Hardware:
- Laptop QUANTA, BIOS 3.07 09/14/2021
- CPU AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx (Picasso/Raven2)
- GPU 03:00.0 VGA [1002:15d8] rev c2, amdgpu, VBIOS 113-PICASSO-115 2019/04/25,
DCN 1.0, PSP v10, SMU10, VCN 1.15, TMZ enabled
- VRAM 1024M BAR 1024M (0xF400000000-0xF43FFFFFFF), GART 1024M, vm 4-level
262144GB
- RAM 2x4GB Hynix DDR4-2667MT/s @2400MT/s (shared with VRAM/GTT)
- NVMe Toshiba KXG60ZNV256G (SMART PASSED, media errors 0), btrfs 0 errors
- No MCE/MCA, HardwareCorrupted 0, EDAC N/A (no ECC on this APU)

Software (both affected):
- BAD: linux-cachyos 7.2.9-1 (DRM 3.64, DC 3.2.384) AND arch linux-lts
6.18.55-1 (reporter confirms same fault on LTS)
- Mesa 26.2.4-arch3.1 radeonsi raven ACO OpenGL 4.6, vulkan-radeon RADV present
but NOT involved
- linux-firmware 20260916 (picasso_* present), kwin 6.7.5 Wayland, kitty
0.49.2, plasma-desktop 6.7.5, TLP 1.10.2
- Kernel cmdline: BOOT_IMAGE=/@/boot/vmlinuz-linux-cachyos
root=/dev/mapper/vg0-notebook rw rootflags=subvol=@ loglevel=3 quiet splash
- Taint 0. Note: sched_ext lavd present on CachyOS but LTS without it also
fails. Broken BIOS warnings: TSC unstable, ccp psp unable to access device.

Reproducer (99% on cold boot):
1. Set `power_dpm_force_performance_level=high` (e.g. TLP
RADEON_DPM_PERF_LEVEL_ON_AC=high on AC, or `echo high >
/sys/class/drm/card1/device/power_dpm_force_performance_level`). This locks
sclk ~1200MHz / mclk 1200MHz. CPU governor performance + boost 1 in parallel.
2. Boot cold, login to Plasma Wayland, immediately start any GUI GL app (kitty,
kwin compositing itself).
3. Within seconds: 10x gfxhub faults, then ring timeout.
Workaround: `auto` (sclk drops to ~210MHz idle) => 0 faults. If boot in
powersave, wait for DE to settle, then switch to high => usually survives. So
early-first-CS + locked-high is the window.

Expected: high DPM may use more power/heat but must not wedge GPU VM.
Actual: GPU wedged, needs MODE2 reset. Desktop freezes 2-5s+, journal
corruption on hard poweroff (555 unsafe shutdowns total on this drive from a
week of hangs).

Fault log (journal -k -b -1, 23:06:55-23:07:00):
```
amdgpu 0000:03:00.0: [gfxhub0] no-retry page fault (src_id:0 ring:153 vmid:0
pasid:0)
amdgpu 0000:03:00.0:   in page starting at address 0x00000016c4080000 from IH
client 0x1b (UTCL2)
amdgpu 0000:03:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00601431
amdgpu 0000:03:00.0:          Faulty UTCL2 client ID: SQC (data) (0xa)
amdgpu 0000:03:00.0:          MORE_FAULTS: 0x1
amdgpu 0000:03:00.0:          WALKER_ERROR: 0x0
amdgpu 0000:03:00.0:          PERMISSION_FAULTS: 0x3
amdgpu 0000:03:00.0:          MAPPING_ERROR: 0x0
amdgpu 0000:03:00.0:          RW: 0x0
... repeats for 0x16c4090000, 0x16c4020000, 0x16c4030000, 0x16c40e0000,
0x16c40f0000, 0x16c40c0000, 0x16c40d0000, 0x16c4100000, 0x16c4060000
(contiguous ~896KB range)
amdgpu 0000:03:00.0: Dumping IP State / Dumping IP State Completed
amdgpu 0000:03:00.0: [drm] AMDGPU device coredump file has been created
amdgpu 0000:03:00.0: ring gfx timeout, signaled seq=62433, emitted seq=62434
amdgpu 0000:03:00.0:  Process kitty pid 2202 thread kitty:cs0 pid 2236
amdgpu 0000:03:00.0: Starting gfx ring reset
amdgpu 0000:03:00.0: Ring gfx reset failed
amdgpu 0000:03:00.0: GPU reset begin!. Source: 1
amdgpu 0000:03:00.0: MODE2 reset
amdgpu 0000:03:00.0: GPU reset succeeded, trying to resume
amdgpu 0000:03:00.0: GPU reset(1) succeeded!
amdgpu 0000:03:00.0: ring gfx timeout, signaled seq=62434, emitted seq=62437
amdgpu 0000:03:00.0:  Process kwin_wayland pid 1510 thread kwin_wayla:cs0 pid
1522
amdgpu 0000:03:00.0: Starting gfx ring reset
amdgpu 0000:03:00.0: Ring gfx reset succeeded
```
Full `journal-k` and `journal-full` of faulty boot attached. Current `auto`
boot has 0 such lines.

Analysis for developers:
- PERMISSION_FAULTS 0x3 with MAPPING_ERROR 0x0 and RW 0x0 (read): PTE exists
but access denied. Not missing memory. SQC = shader instruction/data cache
fetch path via UTCL2. vmid 0 pasid 0 ring 153. Points to shader BO
use-after-free / wrong permissions / VM-update race in radeonsi+amdgpu, or TMZ
secure mismatch, exposed by timing change under locked-high clocks. Same SQC
0xa class seen widely (RX6700XT, 9070XT, Iris, games) and TCP 0x8 variant in
cosmic-comp#2149, so not Picasso-only.
- TLP is innocent: it only writes `high` to sysfs as configured. Driver must
handle `high`.
- Not proven hardware: no ECC errors possible here, but MCE 0, mem
HardwareCorrupted 0, NVMe media 0, btrfs 0, GPU recovers via MODE2. Shared RAM
still needs memtest86+ for 100% exclusion, will run if requested.
- Confounders: old VBIOS 2019 vs 2025 firmware, PSP LOAD_TA 0x7 (normally
missing RAS TA on Picasso), PSP TMR reserve, broken BIOS (TSC, ccp psp), BIOS
CPU limit 2.1GHz vs 3.7GHz max.

Attachments to follow: journal-k faulty, fault extract, lspci,
versions/uname/cmdline, gpu-init, tlp-stat, pp_dpm. devcoredump was created but
is RAM-only and lost after reboot; reporter can re-capture with udev rule on
next `high` repro if needed. No RADV involvement (OpenGL radeonsi path).

Request: please advise if `amdgpu.vm_update_mode`, `cwsr_enable`, or DPM debug
needed, will test.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 222109] [amdgpu Picasso 1002:15d8] gfxhub UTCL2 SQC permission fault -> ring gfx timeout when DPM forced high (repro on LTS 6.18 too)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=222109

--- Comment #1 from Alex (aleksey16.11.2000@gmail.com) ---
Created attachment 311046
  --> https://bugzilla.kernel.org/attachment.cgi?id=311046&action=edit
journal-k faulty boot (-b -1) with gfxhub SQC faults + ring timeouts

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 222109] [amdgpu Picasso 1002:15d8] gfxhub UTCL2 SQC permission fault -> ring gfx timeout when DPM forced high (repro on LTS 6.18 too)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=222109

--- Comment #2 from Alex (aleksey16.11.2000@gmail.com) ---
Created attachment 311047
  --> https://bugzilla.kernel.org/attachment.cgi?id=311047&action=edit
lspci -nnk (GPU Picasso 1002:15d8 rev c2, amdgpu)

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 222109] [amdgpu Picasso 1002:15d8] gfxhub UTCL2 SQC permission fault -> ring gfx timeout when DPM forced high (repro on LTS 6.18 too)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=222109

--- Comment #3 from Alex (aleksey16.11.2000@gmail.com) ---
Created attachment 311048
  --> https://bugzilla.kernel.org/attachment.cgi?id=311048&action=edit
package versions (kernels 7.2.9 + 6.18.55 LTS, mesa 26.2.4, firmware,
kwin/kitty)

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 222109] [amdgpu Picasso 1002:15d8] gfxhub UTCL2 SQC permission fault -> ring gfx timeout when DPM forced high (repro on LTS 6.18 too)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=222109

--- Comment #4 from Alex (aleksey16.11.2000@gmail.com) ---
Created attachment 311049
  --> https://bugzilla.kernel.org/attachment.cgi?id=311049&action=edit
amdgpu init (VBIOS, VRAM/GART, TMZ, PSP/SMU/VCN, TSC/BIOS warnings)

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 222109] [amdgpu Picasso 1002:15d8] gfxhub UTCL2 SQC permission fault -> ring gfx timeout when DPM forced high (repro on LTS 6.18 too)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=222109

--- Comment #5 from Alex (aleksey16.11.2000@gmail.com) ---
Created attachment 311050
  --> https://bugzilla.kernel.org/attachment.cgi?id=311050&action=edit
tlp-stat -c effective config + tlp-stat-g/p (force_level high vs auto)

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 222109] [amdgpu Picasso 1002:15d8] gfxhub UTCL2 SQC permission fault -> ring gfx timeout when DPM forced high (repro on LTS 6.18 too)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=222109

--- Comment #6 from Alex (aleksey16.11.2000@gmail.com) ---
Created attachment 311051
  --> https://bugzilla.kernel.org/attachment.cgi?id=311051&action=edit
uname/cmdline/BIOS/RAM/pp_dpm/tlp-stat-g-p (auto stable vs high faulty)

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 222109] [amdgpu Picasso 1002:15d8] gfxhub UTCL2 SQC permission fault -> ring gfx timeout when DPM forced high (repro on LTS 6.18 too)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=222109

--- Comment #7 from Alex (aleksey16.11.2000@gmail.com) ---
Additional confirmation from reporter (to use all triage info):

- Reproduces on stock Arch LTS kernel 6.18.55-1 as well, same cold-boot + high
+ immediate-GUI pattern. So NOT CachyOS/sched_ext specific (CachyOS 7.2.9 has
lavd warnings, LTS does not, both wedge).
- No RADV involvement: faulting CS are kitty (OpenGL) and kwin_wayland (OpenGL
radeonsi raven ACO). RADV installed but idle in this repro.
- devcoredump: kernel printed `[drm] AMDGPU device coredump file has been
created` twice in faulty boot, but it is RAM-only and expired after reboot
(nothing in /sys now). Reporter will set a udev rule to auto-save
`/sys/class/drm/card*/device/devcoredump/data` on next `high` repro if you need
it - please tell which artifact you prefer (devcoredump vs umr register dump vs
full dmesg with drm.debug=0x1ff).
- Will run memtest86+ to fully exclude shared DDR4 (VRAM is carved from system
RAM) and `amdgpu.vm_update_mode` / `cwsr_enable` tests on request. Currently
stable on `auto`, so can A/B on demand.
- 3.9MB full journal of faulty boot retained locally (attached trimmed
kernel-only 109KB due to size); can provide compressed full log or split on
request.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 222109] [amdgpu Picasso 1002:15d8] gfxhub UTCL2 SQC permission fault -> ring gfx timeout when DPM forced high (repro on LTS 6.18 too)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=222109

Artem S. Tashkinov (aros@gmx.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |ANSWERED

--- Comment #8 from Artem S. Tashkinov (aros@gmx.com) ---
If this is reproducible in 7.2.10 with the latest firmware, report here:

https://gitlab.freedesktop.org/drm/amd/-/issues

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.