Re: [RFC PATCH 0/3] drm/amdgpu: Render-node second-level trap handler

["Timur Kristóf" <timur.kristof@gmail.com>]

On 2026. augusztus 20., csütörtök 9:01:40 közép-európai nyári idő Srinivasan 
Shanmugam wrote:
> When a GPU shader hits an error or exception, the hardware calls a trap
> handler. AMD GPUs support two levels: a first-level handler that runs
> inside the kernel (via CWSR), and an optional second-level handler that
> runs in userspace. The second-level handler lets a runtime or debugger
> catch shader exceptions without any kernel changes.
> 
> KFD already provides this for compute workloads. Render-node user queues
> use the same first-level CWSR trap infrastructure but had no way to
> install a second-level handler. This series adds that support.

Hello Srini,

Thank you for working on this.
I notice that you mention user queues here. Does that mean that the new trap 
handler will only work with user queues? We would like to see it working with 
kernel queues as well because:

1. User queues are not supported on GFX9, GFX10, GFX10.3
2. User queues are not enabled by default yet in the kernel 
3. RADV currently can't use user queues on an GPU

Thanks & best regards,
Timur


> 
> The second-level handler is a per-process (per-VM) setting. All shader
> queues belonging to the same process share the same hardware VMID, so
> one SET_L2_TRAP call covers every GFX and compute queue for that
> process. This setting belongs in the VM ioctl (DRM_AMDGPU_VM), not in
> the CWSR ioctl, because it is not specific to the first-level handler
> mechanism.
> 
> UAPI backward compatibility
> ---------------------------
> The drm_amdgpu_vm_in struct gains a 32-byte union for the new op data.
> Existing ops (RESERVE/UNRESERVE_VMID) only use the first 8 bytes and
> are unaffected. The DRM framework automatically zero-fills the new fields
> for older userspace programs. No existing Mesa or ROCr code is broken.
> 
> Implementation
> --------------
> On discrete GPUs the CWSR scratch buffer (TMA) may live in VRAM, which
> is accessed through MMIO on some CPU platforms. Direct CPU pointer writes
> are unsafe there. The driver uses struct iosys_map for all TMA writes,
> which picks the right accessor automatically (regular memory or MMIO).
> 
> Installing a new TBA/TMA requires two separate memory writes, which
> creates a race: a shader that traps between the two writes would see a
> mismatched TBA/TMA pair and could crash. To prevent this, the driver
> stops all user queues, flushes the GPU TLB, writes both values, then
> restarts the queues.
> 
> GEM_VA UNMAP and CLEAR operations must always succeed — they cannot
> return an error. If userspace removes a TBA or TMA buffer without first
> calling CLEAR_L2_TRAP, the driver detects the overlap, runs the same
> stop-flush-clear sequence, and lets the unmap proceed silently. Queues
> that lose their trap handler this way are marked invalid and not
> restarted — a queue cannot safely run without a valid trap handler.
> 
> TTMP register layout
> --------------------
> When a shader wave enters the second-level trap handler, these hardware
> registers carry the relevant state:
> 
>   ttmp0/1    faulting shader PC (set by hardware)
>   ttmp14/15  second-level TMA address (set by first-level handler)
>   ttmp2/3    free for the handler to use as temporaries
>   ttmp6[30]  wave stopped flag (set by second-level handler/debugger)
>   ttmp6[29]  saved halt flag (set by second-level handler/debugger)
>   ttmp11[23] debug mode enabled (set by first-level handler)
> 
> Only compilation tested.
> 
> Based on: amd-unified-interface branch
> Base commit: 0f354d789fdc ("drm/amdgpu: Drop vm_manager PASID to VM
>              mapping")
> 
> Cc: Alex Deucher <alexander.deucher@amd.com>
> Cc: Christian König <christian.koenig@amd.com>
> Cc: Lancelot Six <lancelot.six@amd.com>
> Cc: Felix Kuehling <felix.kuehling@amd.com>
> Cc: James Zhu <james.zhu@amd.com>
> Cc: Lijo Lazar <lijo.lazar@amd.com>
> Cc: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
> Cc: Timur Kristóf <timur.kristof@gmail.com>
> Cc: Samuel Pitoiset <hakzsam@gmail.com>
> Cc: Natalie Vock <natalie.vock@gmx.de>
> 
> Srinivasan Shanmugam (3):
>   drm/amdgpu/uapi: Add second-level trap handler ops to VM ioctl
>   drm/amdgpu: Add VM ioctl handlers for second-level trap handler
>   drm/amdgpu: Disable L2 trap handler when its VA range is unmapped
> 
>  drivers/gpu/drm/amd/amdgpu/amdgpu_cwsr.c  | 341 ++++++++++++++++++----
>  drivers/gpu/drm/amd/amdgpu/amdgpu_cwsr.h  |  54 +++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c   |  24 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c   |   5 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c |  15 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_userq.h |   1 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    |  26 ++
>  include/uapi/drm/amdgpu_drm.h             |  80 ++++-
>  8 files changed, 469 insertions(+), 77 deletions(-)
> 
> 
> base-commit: 0f354d789fdcd03147eb77ab2e3ecdd5a67a1fad