Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

["Lazar, Lijo" <lijo.lazar@amd.com>]

On 10-Sep-26 7:36 AM, Lazar, Lijo wrote:
> AMD General
> 
> 
> My question was slightly different context - The first level TMA is 
> fetched based on VMID of execution context .
> 
> If user waves are executed in IB VMID context (even though submitted 
> through kernel queues), do we need to allow user to install handlers for 
> kernel VMIDs?
> 


To clarify -

For other non-zero VMIDs, is there a need to allocate separate BOs like
kq_tba_bo/kq_tma_bo?

Can't we keep just one set of tba/tma bo for first level? User handling 
may always be through second level as the ioctl allows user to install 
only second level ones. If separate handling is required based on queue 
type (kq vs uq), I think trap handler can identify the queue based on 
doorbell offset.

Thanks,
Lijo

> Thanks,
> Lijo
> ------------------------------------------------------------------------
> *From:* Alex Deucher <alexdeucher@gmail.com>
> *Sent:* Thursday, 10 September 2026 00:24:03
> *To:* Lazar, Lijo <Lijo.Lazar@amd.com>
> *Cc:* SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>; Koenig, 
> Christian <Christian.Koenig@amd.com>; Deucher, Alexander 
> <Alexander.Deucher@amd.com>; amd-gfx@lists.freedesktop.org <amd- 
> gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>; 
> Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
> *Subject:* Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first- 
> level trap handler infrastructure
> On Wed, Sep 9, 2026 at 2:05 PM Lazar, Lijo <Lijo.Lazar@amd.com> wrote:
>>
>> AMD General
>>
>>
>> One generic question - I am assuming the overall purpose is to debug a user job submitted to kernel queue. When user IBs are submitted to kernel queue, those IBs carry VMID assigned to user. When wave submitted through such a job encounters a trap, isn't it having the user VMID? If so, when is this kernel queue related TBA/ 
> TMA helpful or selected? If it's only for driver submitted jobs, then 
> this control to user is not required.
> 
> Each fpriv GPUVM will have a copy of the first level trap handler
> mapped at the same GPU virtual address.  If the user requests a second
> level trap handler, their copy of the first level trap handler will be
> updated to point to the provided second level trap handler.
> 
> Alex
> 
>>
>> Thanks,
>> Lijo
>> ________________________________
>> From: SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>
>> Sent: Wednesday, 09 September 2026 18:32:03
>> To: Lazar, Lijo <Lijo.Lazar@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com>
>> Cc: amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>; Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
>> Subject: Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure
>>
>>
>>
>> On 9/9/2026 1:15 PM, Lazar, Lijo wrote:
>>
>>
>>
>> On 05-Sep-26 1:49 PM, Srinivasan Shanmugam wrote:
>>
>> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
>> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
>> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
>> On GFX10 and earlier HWS-based hardware, the driver programs trap
>> registers via SRBM select for KFD queues but no equivalent exists for
>> driver-managed kernel queue VMIDs.
>>
>> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
>> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
>> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
>> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
>> there, so per-VM isolation is handled entirely by page tables without
>> needing to reprogram the register per job or per submission.
>>
>> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
>> (parallel to page table allocation) and mapped read-only into the GPU VM
>> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
>> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
>> The first-level CWSR handler reads this address to chain to the
>> second-level handler when a shader exception fires.
>>
>> This design is:
>>    - Per-VM BO (not device-level) — same model as page tables
>>    - Fixed VA in each VM's address space — same VA, different physical BO
>>    - Read-only from GPU — kernel CPU updates it via CPU mapping
>>    - Treat allocation/free lifecycle identical to page tables
>>
>> Suggested-by: Christian König <christian.koenig@amd.com>
>> Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
>> Cc: Lijo Lazar <lijo.lazar@amd.com>
>> Cc: Timur Kristóf <timur.kristof@gmail.com>
>> Cc: Samuel Pitoiset <hakzsam@gmail.com>
>> Cc: Natalie Vock <natalie.vock@gmx.de>
>> Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
>> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>>   5 files changed, 110 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>> index 3ca187f5ade8..5624a5ab5c62 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>>       void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
>>                            uint32_t status);
>>       uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
>> +    void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
>>   };
>>     struct amdgpu_vmhub {
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>> index 623cac6781be..e913488ca3fa 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>>         amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>>       adev->trap_info = no_free_ptr(trap_info);
>> +    amdgpu_trap_program_kernel_vmids(adev);
>>         return 0;
>>   }
>> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
>>       adev->trap_info = NULL;
>>   }
>>   +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
>> +{
>> +    struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
>> +
>> +    if (!amdgpu_trap_is_enabled(adev))
>> +        return;
>> +    if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
>> +        return;
>> +
>> +    hub->vmhub_funcs->program_kernel_trap_vmids(adev);
>> +}
>> +
>> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>> +                struct amdgpu_vm *vm)
>> +{
>> +    void *cpu_addr;
>> +    uint64_t va;
>> +    int r;
>> +
>> +    dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>> +
>> +    r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
>> +                    AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
>> +                    NULL, &cpu_addr);
>> +    if (r)
>> +        return r;
>> +
>> +    if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
>> +        iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
>> +                      (void __iomem *)cpu_addr);
>> +    else
>> +        iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
>> +
>> +    vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
>> +    if (!vm->kq_tma_va) {
>> +        r = -ENOMEM;
>> +        goto err_free_bo;
>> +    }
>> +
>> +    va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
>>
>>
>> Is this the same address used for mapping of TMA for user queues?
>>
>> No — these are different, non-overlapping addresses in the reserved VA region:
>>
>> AMDGPU_VA_RESERVED_TRAP_UQ_START = TRAP_START − 12 KiB
>> → used for UQ first-level TBA (8 KiB) + TMA (4 KiB)
>> AMDGPU_VA_RESERVED_TRAP_START = SEQ64_START − 64 KiB
>> → used for KQ per-VM TMA (this patch)
>>
>> The UQ region sits immediately below the KQ region in the reserved VA
>> space. No collision between the two mappings in the same VM.
>>
>> Regards, Srini