Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

["Lazar, Lijo" <lijo.lazar@amd.com>]

On 11-Sep-26 7:39 PM, Alex Deucher wrote:
> On Fri, Sep 11, 2026 at 10:07 AM Alex Deucher <alexdeucher@gmail.com> wrote:
>>
>> On Thu, Sep 10, 2026 at 1:23 AM Lazar, Lijo <lijo.lazar@amd.com> wrote:
>>>
>>>
>>>
>>> On 10-Sep-26 7:36 AM, Lazar, Lijo wrote:
>>>> AMD General
>>>>
>>>>
>>>> My question was slightly different context - The first level TMA is
>>>> fetched based on VMID of execution context .
>>>>
>>>> If user waves are executed in IB VMID context (even though submitted
>>>> through kernel queues), do we need to allow user to install handlers for
>>>> kernel VMIDs?
>>>>
>>>
>>>
>>> To clarify -
>>>
>>> For other non-zero VMIDs, is there a need to allocate separate BOs like
>>> kq_tba_bo/kq_tma_bo?
>>>
>>> Can't we keep just one set of tba/tma bo for first level? User handling
>>> may always be through second level as the ioctl allows user to install
>>> only second level ones. If separate handling is required based on queue
>>> type (kq vs uq), I think trap handler can identify the queue based on
>>> doorbell offset.
>>
>> Maybe I'm misunderstanding how the second level registration works.  I
>> thought the pointer to the second level trap handler was changed in
>> the first level trap handler when the user registers the second level
>> trap handler. If so, we need a per VM BO for the first level trap
>> handler.  Then the second level trap handler is either NULL or
>> whatever the user registers for their GPU VM.
>>
> 
> I guess the TBA and TMA could be split and the TBA could be a common
> allocation and the TMA could be per VM.  Was that what you were
> getting at?
> 

I was getting at using a single set of TMA (only one tma bo, not one 
more kq_tma_bo) for first level trap. Later I noticed the thread with 
Srini and I see you asking for the same (i.e, only to program SQ 
registers for non-kfd VMIDs also with the address of the single TMA BO 
allocated and not have one more kq_tma_bo).

Thanks,
Lijo

> Alex
> 
>> Alex
>>
>>>
>>> Thanks,
>>> Lijo
>>>
>>>> Thanks,
>>>> Lijo
>>>> ------------------------------------------------------------------------
>>>> *From:* Alex Deucher <alexdeucher@gmail.com>
>>>> *Sent:* Thursday, 10 September 2026 00:24:03
>>>> *To:* Lazar, Lijo <Lijo.Lazar@amd.com>
>>>> *Cc:* SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>; Koenig,
>>>> Christian <Christian.Koenig@amd.com>; Deucher, Alexander
>>>> <Alexander.Deucher@amd.com>; amd-gfx@lists.freedesktop.org <amd-
>>>> gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>;
>>>> Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
>>>> *Subject:* Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-
>>>> level trap handler infrastructure
>>>> On Wed, Sep 9, 2026 at 2:05 PM Lazar, Lijo <Lijo.Lazar@amd.com> wrote:
>>>>>
>>>>> AMD General
>>>>>
>>>>>
>>>>> One generic question - I am assuming the overall purpose is to debug a user job submitted to kernel queue. When user IBs are submitted to kernel queue, those IBs carry VMID assigned to user. When wave submitted through such a job encounters a trap, isn't it having the user VMID? If so, when is this kernel queue related TBA/
>>>> TMA helpful or selected? If it's only for driver submitted jobs, then
>>>> this control to user is not required.
>>>>
>>>> Each fpriv GPUVM will have a copy of the first level trap handler
>>>> mapped at the same GPU virtual address.  If the user requests a second
>>>> level trap handler, their copy of the first level trap handler will be
>>>> updated to point to the provided second level trap handler.
>>>>
>>>> Alex
>>>>
>>>>>
>>>>> Thanks,
>>>>> Lijo
>>>>> ________________________________
>>>>> From: SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>
>>>>> Sent: Wednesday, 09 September 2026 18:32:03
>>>>> To: Lazar, Lijo <Lijo.Lazar@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com>
>>>>> Cc: amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>; Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
>>>>> Subject: Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure
>>>>>
>>>>>
>>>>>
>>>>> On 9/9/2026 1:15 PM, Lazar, Lijo wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 05-Sep-26 1:49 PM, Srinivasan Shanmugam wrote:
>>>>>
>>>>> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
>>>>> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
>>>>> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
>>>>> On GFX10 and earlier HWS-based hardware, the driver programs trap
>>>>> registers via SRBM select for KFD queues but no equivalent exists for
>>>>> driver-managed kernel queue VMIDs.
>>>>>
>>>>> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
>>>>> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
>>>>> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
>>>>> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
>>>>> there, so per-VM isolation is handled entirely by page tables without
>>>>> needing to reprogram the register per job or per submission.
>>>>>
>>>>> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
>>>>> (parallel to page table allocation) and mapped read-only into the GPU VM
>>>>> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
>>>>> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
>>>>> The first-level CWSR handler reads this address to chain to the
>>>>> second-level handler when a shader exception fires.
>>>>>
>>>>> This design is:
>>>>>     - Per-VM BO (not device-level) — same model as page tables
>>>>>     - Fixed VA in each VM's address space — same VA, different physical BO
>>>>>     - Read-only from GPU — kernel CPU updates it via CPU mapping
>>>>>     - Treat allocation/free lifecycle identical to page tables
>>>>>
>>>>> Suggested-by: Christian König <christian.koenig@amd.com>
>>>>> Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
>>>>> Cc: Lijo Lazar <lijo.lazar@amd.com>
>>>>> Cc: Timur Kristóf <timur.kristof@gmail.com>
>>>>> Cc: Samuel Pitoiset <hakzsam@gmail.com>
>>>>> Cc: Natalie Vock <natalie.vock@gmx.de>
>>>>> Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
>>>>> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
>>>>> ---
>>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
>>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>>>>>    5 files changed, 110 insertions(+)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>> index 3ca187f5ade8..5624a5ab5c62 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>>>>>        void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
>>>>>                             uint32_t status);
>>>>>        uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
>>>>> +    void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
>>>>>    };
>>>>>      struct amdgpu_vmhub {
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>> index 623cac6781be..e913488ca3fa 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>>>>>          amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>>>>>        adev->trap_info = no_free_ptr(trap_info);
>>>>> +    amdgpu_trap_program_kernel_vmids(adev);
>>>>>          return 0;
>>>>>    }
>>>>> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
>>>>>        adev->trap_info = NULL;
>>>>>    }
>>>>>    +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
>>>>> +{
>>>>> +    struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
>>>>> +
>>>>> +    if (!amdgpu_trap_is_enabled(adev))
>>>>> +        return;
>>>>> +    if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
>>>>> +        return;
>>>>> +
>>>>> +    hub->vmhub_funcs->program_kernel_trap_vmids(adev);
>>>>> +}
>>>>> +
>>>>> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>>>>> +                struct amdgpu_vm *vm)
>>>>> +{
>>>>> +    void *cpu_addr;
>>>>> +    uint64_t va;
>>>>> +    int r;
>>>>> +
>>>>> +    dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>>>>> +
>>>>> +    r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
>>>>> +                    AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
>>>>> +                    NULL, &cpu_addr);
>>>>> +    if (r)
>>>>> +        return r;
>>>>> +
>>>>> +    if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
>>>>> +        iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
>>>>> +                      (void __iomem *)cpu_addr);
>>>>> +    else
>>>>> +        iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
>>>>> +
>>>>> +    vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
>>>>> +    if (!vm->kq_tma_va) {
>>>>> +        r = -ENOMEM;
>>>>> +        goto err_free_bo;
>>>>> +    }
>>>>> +
>>>>> +    va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
>>>>>
>>>>>
>>>>> Is this the same address used for mapping of TMA for user queues?
>>>>>
>>>>> No — these are different, non-overlapping addresses in the reserved VA region:
>>>>>
>>>>> AMDGPU_VA_RESERVED_TRAP_UQ_START = TRAP_START − 12 KiB
>>>>> → used for UQ first-level TBA (8 KiB) + TMA (4 KiB)
>>>>> AMDGPU_VA_RESERVED_TRAP_START = SEQ64_START − 64 KiB
>>>>> → used for KQ per-VM TMA (this patch)
>>>>>
>>>>> The UQ region sits immediately below the KQ region in the reserved VA
>>>>> space. No collision between the two mappings in the same VM.
>>>>>
>>>>> Regards, Srini
>>>