[PATCH 0/3] drm/amdgpu: Second-level trap handler for kernel queues

[PATCH 0/3] drm/amdgpu: Second-level trap handler for kernel queues

[Srinivasan Shanmugam]

Background
----------
When a GPU shader crashes or hits an exception, the hardware
automatically jumps to a trap handler. There are two levels:

  First-level  — always active, managed by the kernel (CWSR)
  Second-level — optional, installed by userspace (RADV, ROCm)

The second-level handler lets userspace debuggers catch GPU crashes
and inspect what went wrong.

Problem
-------
The existing second-level trap handler support (merged earlier) only
works for user queues. RADV on GFX9 and GFX10 hardware (Vega, Navi,
Steam Deck) uses kernel queues — not user queues — because user queues
are not supported on those GPUs. This means RADV has no way to install
a second-level trap handler for shader debugging on those platforms.

Root cause: the SQ_SHADER_TMA register for kernel queue VMIDs was
never programmed by the driver, so there was nowhere to store the
second-level handler address.

Solution
--------
This series fixes the problem in three steps:

Patch 1 — Infrastructure
  - Allocate a small per-VM TMA buffer (kq_tma_bo) at VM creation time,
    the same way page tables are allocated.
  - Map it read-only into the GPU VM at a fixed address
    (AMDGPU_VA_RESERVED_TRAP_START).
  - Program SQ_SHADER_TMA for all kernel queue VMIDs to point to this
    fixed address. Because each VM has its own buffer at the same
    address, per-process isolation comes from the page tables — not
    from reprogramming the register per job.

Patch 2 — Hardware register programming (GFX10/11/12)
  - Implement the actual register writes (SQ_SHADER_TBA/TMA) for all
    supported GPU generations via SRBM select.
  - GFX10 uses mm-prefixed registers, GFX11/12 use reg-prefixed.
  - GFX12.1 has multiple XCC instances and iterates over all of them.
  - Called at boot and on GPU resume.

Patch 3 — Wire up SET_L2_TRAP and CLEAR_L2_TRAP
  - When userspace calls SET_L2_TRAP, write the second-level handler
    address into both the user queue TMA and the new kernel queue TMA.
  - When userspace calls CLEAR_L2_TRAP, zero both.
  - No new ioctl needed — RADV uses the same SET_L2_TRAP ioctl that
    user queues already use.

Design decisions
----------------
  - kq_tma_bo is per-VM, not device-level. One BO per process,
    same model as page tables.
  - Fixed VA (AMDGPU_VA_RESERVED_TRAP_START) in every VM's address
    space. Same register value for all VMIDs — isolation via page tables.
  - GPU sees it as read-only. Kernel CPU updates it via CPU mapping
    when SET_L2_TRAP is called.
  - Lifecycle matches page tables: allocated at VM init, freed at
    VM fini.
	
Only compilation tested.

Srinivasan Shanmugam (3):
  drm/amdgpu: Add per-VM kernel queue first-level trap handler
    infrastructure
  drm/amdgpu: Implement kernel VMID SQ_SHADER_TBA/TMA programming for
    GFX10/11/12
  drm/amdgpu: Extend second-level trap handler to kernel queues

 drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h     |   1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c    | 107 ++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h    |   7 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c      |   9 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h      |  13 +++
 drivers/gpu/drm/amd/amdgpu/gfxhub_v11_5_0.c |  38 +++++++
 drivers/gpu/drm/amd/amdgpu/gfxhub_v12_0.c   |  38 +++++++
 drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c   |  36 +++++++
 drivers/gpu/drm/amd/amdgpu/gfxhub_v2_0.c    |  38 +++++++
 drivers/gpu/drm/amd/amdgpu/gfxhub_v3_0.c    |  33 ++++++
 10 files changed, 320 insertions(+)

-- 
2.34.1

[PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Srinivasan Shanmugam]

MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
On GFX10 and earlier HWS-based hardware, the driver programs trap
registers via SRBM select for KFD queues but no equivalent exists for
driver-managed kernel queue VMIDs.

Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
there, so per-VM isolation is handled entirely by page tables without
needing to reprogram the register per job or per submission.

The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
(parallel to page table allocation) and mapped read-only into the GPU VM
at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
The first-level CWSR handler reads this address to chain to the
second-level handler when a shader exception fires.

This design is:
  - Per-VM BO (not device-level) — same model as page tables
  - Fixed VA in each VM's address space — same VA, different physical BO
  - Read-only from GPU — kernel CPU updates it via CPU mapping
  - Treat allocation/free lifecycle identical to page tables

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
Cc: Lijo Lazar <lijo.lazar@amd.com>
Cc: Timur Kristóf <timur.kristof@gmail.com>
Cc: Samuel Pitoiset <hakzsam@gmail.com>
Cc: Natalie Vock <natalie.vock@gmx.de>
Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
 5 files changed, 110 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
index 3ca187f5ade8..5624a5ab5c62 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
@@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
 	void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
 						 uint32_t status);
 	uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
+	void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
 };
 
 struct amdgpu_vmhub {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
index 623cac6781be..e913488ca3fa 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
@@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
 
 	amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
 	adev->trap_info = no_free_ptr(trap_info);
+	amdgpu_trap_program_kernel_vmids(adev);
 
 	return 0;
 }
@@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
 	adev->trap_info = NULL;
 }
 
+void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
+{
+	struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
+
+	if (!amdgpu_trap_is_enabled(adev))
+		return;
+	if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
+		return;
+
+	hub->vmhub_funcs->program_kernel_trap_vmids(adev);
+}
+
+int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
+				struct amdgpu_vm *vm)
+{
+	void *cpu_addr;
+	uint64_t va;
+	int r;
+
+	dma_resv_assert_held(vm->root.bo->tbo.base.resv);
+
+	r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
+				    AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
+				    NULL, &cpu_addr);
+	if (r)
+		return r;
+
+	if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
+		iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
+					  (void __iomem *)cpu_addr);
+	else
+		iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
+
+	vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
+	if (!vm->kq_tma_va) {
+		r = -ENOMEM;
+		goto err_free_bo;
+	}
+
+	va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
+	r = amdgpu_vm_bo_map(adev, vm->kq_tma_va, va, 0,
+			     AMDGPU_GPU_PAGE_SIZE,
+			     AMDGPU_VM_PAGE_READABLE);
+	if (r)
+		goto err_del_va;
+
+	r = amdgpu_vm_bo_update(adev, vm->kq_tma_va, false);
+	if (r)
+		goto err_del_va;
+
+	return 0;
+
+err_del_va:
+	amdgpu_vm_bo_del(adev, vm->kq_tma_va);
+	vm->kq_tma_va = NULL;
+err_free_bo:
+	amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
+	return r;
+}
+
+void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
+				struct amdgpu_vm *vm)
+{
+	uint64_t va;
+
+	if (!vm->kq_tma_bo)
+		return;
+
+	dma_resv_assert_held(vm->root.bo->tbo.base.resv);
+
+	if (vm->kq_tma_va) {
+		va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
+		amdgpu_vm_bo_unmap(adev, vm->kq_tma_va, va);
+		amdgpu_vm_bo_del(adev, vm->kq_tma_va);
+		vm->kq_tma_va = NULL;
+	}
+	amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
+}
+
 static int amdgpu_trap_map_region(struct amdgpu_device *adev,
 				  struct amdgpu_vm *vm,
 				  struct amdgpu_trap_obj *cwsr,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
index 7f83174a4742..9be5035abd1c 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
@@ -155,4 +155,11 @@ int amdgpu_trap_set_trap_debug_flag(struct amdgpu_device *adev,
 				    struct amdgpu_trap_obj *cwsr_obj,
 				    bool enabled);
 
+/* Kernel queue trap handler — per-VM TMA and register programming */
+void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev);
+int  amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
+				 struct amdgpu_vm *vm);
+void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
+				struct amdgpu_vm *vm);
+
 #endif
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index de3ef9ce2234..f5e228d4bfb5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -2640,6 +2640,12 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	if (r)
 		goto error_free_root;
 
+	if (amdgpu_trap_is_enabled(adev)) {
+		r = amdgpu_trap_vm_kq_tma_alloc(adev, vm);
+		if (r)
+			goto error_free_root;
+	}
+
 	r = amdgpu_vm_create_task_info(vm);
 	if (r)
 		dev_dbg(adev->dev, "Failed to create task info for VM\n");
@@ -2774,6 +2780,9 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 		amdgpu_vm_free_mapping(adev, vm, mapping, NULL);
 	}
 
+	if (vm->kq_tma_bo)
+		amdgpu_trap_vm_kq_tma_free(adev, vm);
+
 	amdgpu_vm_pt_free_root(adev, vm);
 	amdgpu_bo_unreserve(root);
 	amdgpu_bo_unref(&root);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index dd825e179979..064f95a83790 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -25,6 +25,7 @@
 #define __AMDGPU_VM_H__
 
 #include <linux/idr.h>
+#include <linux/iosys-map.h>
 #include <linux/kfifo.h>
 #include <linux/rbtree.h>
 #include <drm/gpu_scheduler.h>
@@ -488,6 +489,18 @@ struct amdgpu_vm {
 
 	/* cached fault info */
 	struct amdgpu_vm_fault_info fault_info;
+
+	/*
+	 * Per-VM kernel queue first-level TMA BO.
+	 * Allocated at VM init, freed at VM fini — same lifecycle as page tables.
+	 * Mapped read-only at AMDGPU_VA_RESERVED_TRAP_START in the GPU VM.
+	 * SQ_SHADER_TMA for all kernel VMIDs points to this fixed VA; per-VM
+	 * isolation is via page tables mapping different physical BOs there.
+	 * CPU kernel writes second-level handler address via kq_tma_map.
+	 */
+	struct amdgpu_bo	*kq_tma_bo;
+	struct amdgpu_bo_va	*kq_tma_va;
+	struct iosys_map	 kq_tma_map;
 };
 
 struct amdgpu_vm_manager {
-- 
2.34.1

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Lazar, Lijo]

On 05-Sep-26 1:49 PM, Srinivasan Shanmugam wrote:
> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
> On GFX10 and earlier HWS-based hardware, the driver programs trap
> registers via SRBM select for KFD queues but no equivalent exists for
> driver-managed kernel queue VMIDs.
> 
> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
> there, so per-VM isolation is handled entirely by page tables without
> needing to reprogram the register per job or per submission.
> 
> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
> (parallel to page table allocation) and mapped read-only into the GPU VM
> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
> The first-level CWSR handler reads this address to chain to the
> second-level handler when a shader exception fires.
> 
> This design is:
>    - Per-VM BO (not device-level) — same model as page tables
>    - Fixed VA in each VM's address space — same VA, different physical BO
>    - Read-only from GPU — kernel CPU updates it via CPU mapping
>    - Treat allocation/free lifecycle identical to page tables
> 
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
> Cc: Lijo Lazar <lijo.lazar@amd.com>
> Cc: Timur Kristóf <timur.kristof@gmail.com>
> Cc: Samuel Pitoiset <hakzsam@gmail.com>
> Cc: Natalie Vock <natalie.vock@gmx.de>
> Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>   5 files changed, 110 insertions(+)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> index 3ca187f5ade8..5624a5ab5c62 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>   	void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
>   						 uint32_t status);
>   	uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
> +	void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
>   };
>   
>   struct amdgpu_vmhub {
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> index 623cac6781be..e913488ca3fa 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>   
>   	amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>   	adev->trap_info = no_free_ptr(trap_info);
> +	amdgpu_trap_program_kernel_vmids(adev);
>   
>   	return 0;
>   }
> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
>   	adev->trap_info = NULL;
>   }
>   
> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
> +{
> +	struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
> +
> +	if (!amdgpu_trap_is_enabled(adev))
> +		return;
> +	if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
> +		return;
> +
> +	hub->vmhub_funcs->program_kernel_trap_vmids(adev);
> +}
> +
> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> +				struct amdgpu_vm *vm)
> +{
> +	void *cpu_addr;
> +	uint64_t va;
> +	int r;
> +
> +	dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> +
> +	r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
> +				    AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
> +				    NULL, &cpu_addr);
> +	if (r)
> +		return r;
> +
> +	if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
> +		iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
> +					  (void __iomem *)cpu_addr);
> +	else
> +		iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
> +
> +	vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
> +	if (!vm->kq_tma_va) {
> +		r = -ENOMEM;
> +		goto err_free_bo;
> +	}
> +
> +	va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;

Is this the same address used for mapping of TMA for user queues?

Thanks,
Lijo

> +	r = amdgpu_vm_bo_map(adev, vm->kq_tma_va, va, 0,
> +			     AMDGPU_GPU_PAGE_SIZE,
> +			     AMDGPU_VM_PAGE_READABLE);
> +	if (r)
> +		goto err_del_va;
> +
> +	r = amdgpu_vm_bo_update(adev, vm->kq_tma_va, false);
> +	if (r)
> +		goto err_del_va;
> +
> +	return 0;
> +
> +err_del_va:
> +	amdgpu_vm_bo_del(adev, vm->kq_tma_va);
> +	vm->kq_tma_va = NULL;
> +err_free_bo:
> +	amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
> +	return r;
> +}
> +
> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
> +				struct amdgpu_vm *vm)
> +{
> +	uint64_t va;
> +
> +	if (!vm->kq_tma_bo)
> +		return;
> +
> +	dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> +
> +	if (vm->kq_tma_va) {
> +		va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
> +		amdgpu_vm_bo_unmap(adev, vm->kq_tma_va, va);
> +		amdgpu_vm_bo_del(adev, vm->kq_tma_va);
> +		vm->kq_tma_va = NULL;
> +	}
> +	amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
> +}
> +
>   static int amdgpu_trap_map_region(struct amdgpu_device *adev,
>   				  struct amdgpu_vm *vm,
>   				  struct amdgpu_trap_obj *cwsr,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> index 7f83174a4742..9be5035abd1c 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> @@ -155,4 +155,11 @@ int amdgpu_trap_set_trap_debug_flag(struct amdgpu_device *adev,
>   				    struct amdgpu_trap_obj *cwsr_obj,
>   				    bool enabled);
>   
> +/* Kernel queue trap handler — per-VM TMA and register programming */
> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev);
> +int  amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> +				 struct amdgpu_vm *vm);
> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
> +				struct amdgpu_vm *vm);
> +
>   #endif
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index de3ef9ce2234..f5e228d4bfb5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -2640,6 +2640,12 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   	if (r)
>   		goto error_free_root;
>   
> +	if (amdgpu_trap_is_enabled(adev)) {
> +		r = amdgpu_trap_vm_kq_tma_alloc(adev, vm);
> +		if (r)
> +			goto error_free_root;
> +	}
> +
>   	r = amdgpu_vm_create_task_info(vm);
>   	if (r)
>   		dev_dbg(adev->dev, "Failed to create task info for VM\n");
> @@ -2774,6 +2780,9 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>   		amdgpu_vm_free_mapping(adev, vm, mapping, NULL);
>   	}
>   
> +	if (vm->kq_tma_bo)
> +		amdgpu_trap_vm_kq_tma_free(adev, vm);
> +
>   	amdgpu_vm_pt_free_root(adev, vm);
>   	amdgpu_bo_unreserve(root);
>   	amdgpu_bo_unref(&root);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index dd825e179979..064f95a83790 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -25,6 +25,7 @@
>   #define __AMDGPU_VM_H__
>   
>   #include <linux/idr.h>
> +#include <linux/iosys-map.h>
>   #include <linux/kfifo.h>
>   #include <linux/rbtree.h>
>   #include <drm/gpu_scheduler.h>
> @@ -488,6 +489,18 @@ struct amdgpu_vm {
>   
>   	/* cached fault info */
>   	struct amdgpu_vm_fault_info fault_info;
> +
> +	/*
> +	 * Per-VM kernel queue first-level TMA BO.
> +	 * Allocated at VM init, freed at VM fini — same lifecycle as page tables.
> +	 * Mapped read-only at AMDGPU_VA_RESERVED_TRAP_START in the GPU VM.
> +	 * SQ_SHADER_TMA for all kernel VMIDs points to this fixed VA; per-VM
> +	 * isolation is via page tables mapping different physical BOs there.
> +	 * CPU kernel writes second-level handler address via kq_tma_map.
> +	 */
> +	struct amdgpu_bo	*kq_tma_bo;
> +	struct amdgpu_bo_va	*kq_tma_va;
> +	struct iosys_map	 kq_tma_map;
>   };
>   
>   struct amdgpu_vm_manager {

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[SRINIVASAN SHANMUGAM]

[-- Attachment #1: Type: text/plain, Size: 5820 bytes --]


On 9/9/2026 1:15 PM, Lazar, Lijo wrote:
>
>
> On 05-Sep-26 1:49 PM, Srinivasan Shanmugam wrote:
>> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
>> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
>> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
>> On GFX10 and earlier HWS-based hardware, the driver programs trap
>> registers via SRBM select for KFD queues but no equivalent exists for
>> driver-managed kernel queue VMIDs.
>>
>> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
>> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
>> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
>> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
>> there, so per-VM isolation is handled entirely by page tables without
>> needing to reprogram the register per job or per submission.
>>
>> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
>> (parallel to page table allocation) and mapped read-only into the GPU VM
>> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
>> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
>> The first-level CWSR handler reads this address to chain to the
>> second-level handler when a shader exception fires.
>>
>> This design is:
>>    - Per-VM BO (not device-level) — same model as page tables
>>    - Fixed VA in each VM's address space — same VA, different 
>> physical BO
>>    - Read-only from GPU — kernel CPU updates it via CPU mapping
>>    - Treat allocation/free lifecycle identical to page tables
>>
>> Suggested-by: Christian König <christian.koenig@amd.com>
>> Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
>> Cc: Lijo Lazar <lijo.lazar@amd.com>
>> Cc: Timur Kristóf <timur.kristof@gmail.com>
>> Cc: Samuel Pitoiset <hakzsam@gmail.com>
>> Cc: Natalie Vock <natalie.vock@gmx.de>
>> Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
>> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>>   5 files changed, 110 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h 
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>> index 3ca187f5ade8..5624a5ab5c62 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>>       void (*print_l2_protection_fault_status)(struct amdgpu_device 
>> *adev,
>>                            uint32_t status);
>>       uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t 
>> flush_type);
>> +    void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
>>   };
>>     struct amdgpu_vmhub {
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c 
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>> index 623cac6781be..e913488ca3fa 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>>         amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>>       adev->trap_info = no_free_ptr(trap_info);
>> +    amdgpu_trap_program_kernel_vmids(adev);
>>         return 0;
>>   }
>> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
>>       adev->trap_info = NULL;
>>   }
>>   +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
>> +{
>> +    struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
>> +
>> +    if (!amdgpu_trap_is_enabled(adev))
>> +        return;
>> +    if (!hub->vmhub_funcs || 
>> !hub->vmhub_funcs->program_kernel_trap_vmids)
>> +        return;
>> +
>> +    hub->vmhub_funcs->program_kernel_trap_vmids(adev);
>> +}
>> +
>> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>> +                struct amdgpu_vm *vm)
>> +{
>> +    void *cpu_addr;
>> +    uint64_t va;
>> +    int r;
>> +
>> +    dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>> +
>> +    r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
>> +                    AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
>> +                    NULL, &cpu_addr);
>> +    if (r)
>> +        return r;
>> +
>> +    if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
>> +        iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
>> +                      (void __iomem *)cpu_addr);
>> +    else
>> +        iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
>> +
>> +    vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
>> +    if (!vm->kq_tma_va) {
>> +        r = -ENOMEM;
>> +        goto err_free_bo;
>> +    }
>> +
>> +    va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
>
> Is this the same address used for mapping of TMA for user queues?

No — these are *different, non-overlapping addresses* in the reserved VA 
region:

  * |AMDGPU_VA_RESERVED_TRAP_UQ_START| = TRAP_START − 12 KiB
    → used for UQ first-level TBA (8 KiB) + TMA (4 KiB)
  * |AMDGPU_VA_RESERVED_TRAP_START| = SEQ64_START − 64 KiB
    → used for KQ per-VM TMA (this patch)

The UQ region sits immediately below the KQ region in the reserved VA
space. No collision between the two mappings in the same VM.

Regards, Srini

[-- Attachment #2: Type: text/html, Size: 10239 bytes --]

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Lazar, Lijo]

[-- Attachment #1: Type: text/plain, Size: 6566 bytes --]

AMD General

One generic question - I am assuming the overall purpose is to debug a user job submitted to kernel queue. When user IBs are submitted to kernel queue, those IBs carry VMID assigned to user. When wave submitted through such a job encounters a trap, isn't it having the user VMID? If so, when is this kernel queue related TBA/TMA helpful or selected? If it's only for driver submitted jobs, then this control to user is not required.

Thanks,
Lijo
________________________________
From: SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>
Sent: Wednesday, 09 September 2026 18:32:03
To: Lazar, Lijo <Lijo.Lazar@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com>
Cc: amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>; Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
Subject: Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure



On 9/9/2026 1:15 PM, Lazar, Lijo wrote:


On 05-Sep-26 1:49 PM, Srinivasan Shanmugam wrote:
MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
On GFX10 and earlier HWS-based hardware, the driver programs trap
registers via SRBM select for KFD queues but no equivalent exists for
driver-managed kernel queue VMIDs.

Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
there, so per-VM isolation is handled entirely by page tables without
needing to reprogram the register per job or per submission.

The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
(parallel to page table allocation) and mapped read-only into the GPU VM
at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
The first-level CWSR handler reads this address to chain to the
second-level handler when a shader exception fires.

This design is:
   - Per-VM BO (not device-level) — same model as page tables
   - Fixed VA in each VM's address space — same VA, different physical BO
   - Read-only from GPU — kernel CPU updates it via CPU mapping
   - Treat allocation/free lifecycle identical to page tables

Suggested-by: Christian König <christian.koenig@amd.com><mailto:christian.koenig@amd.com>
Suggested-by: Alexander Deucher <alexander.deucher@amd.com><mailto:alexander.deucher@amd.com>
Cc: Lijo Lazar <lijo.lazar@amd.com><mailto:lijo.lazar@amd.com>
Cc: Timur Kristóf <timur.kristof@gmail.com><mailto:timur.kristof@gmail.com>
Cc: Samuel Pitoiset <hakzsam@gmail.com><mailto:hakzsam@gmail.com>
Cc: Natalie Vock <natalie.vock@gmx.de><mailto:natalie.vock@gmx.de>
Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com><mailto:srinivasan.shanmugam@amd.com>
Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
  5 files changed, 110 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
index 3ca187f5ade8..5624a5ab5c62 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
@@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
      void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
                           uint32_t status);
      uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
+    void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
  };
    struct amdgpu_vmhub {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
index 623cac6781be..e913488ca3fa 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
@@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
        amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
      adev->trap_info = no_free_ptr(trap_info);
+    amdgpu_trap_program_kernel_vmids(adev);
        return 0;
  }
@@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
      adev->trap_info = NULL;
  }
  +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
+{
+    struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
+
+    if (!amdgpu_trap_is_enabled(adev))
+        return;
+    if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
+        return;
+
+    hub->vmhub_funcs->program_kernel_trap_vmids(adev);
+}
+
+int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
+                struct amdgpu_vm *vm)
+{
+    void *cpu_addr;
+    uint64_t va;
+    int r;
+
+    dma_resv_assert_held(vm->root.bo->tbo.base.resv);
+
+    r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
+                    AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
+                    NULL, &cpu_addr);
+    if (r)
+        return r;
+
+    if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
+        iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
+                      (void __iomem *)cpu_addr);
+    else
+        iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
+
+    vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
+    if (!vm->kq_tma_va) {
+        r = -ENOMEM;
+        goto err_free_bo;
+    }
+
+    va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;

Is this the same address used for mapping of TMA for user queues?

No — these are different, non-overlapping addresses in the reserved VA region:

  *   AMDGPU_VA_RESERVED_TRAP_UQ_START = TRAP_START − 12 KiB
→ used for UQ first-level TBA (8 KiB) + TMA (4 KiB)
  *   AMDGPU_VA_RESERVED_TRAP_START = SEQ64_START − 64 KiB
→ used for KQ per-VM TMA (this patch)

The UQ region sits immediately below the KQ region in the reserved VA
space. No collision between the two mappings in the same VM.

Regards, Srini

[-- Attachment #2: Type: text/html, Size: 10787 bytes --]

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Alex Deucher]

On Wed, Sep 9, 2026 at 2:05 PM Lazar, Lijo <Lijo.Lazar@amd.com> wrote:
>
> AMD General
>
>
> One generic question - I am assuming the overall purpose is to debug a user job submitted to kernel queue. When user IBs are submitted to kernel queue, those IBs carry VMID assigned to user. When wave submitted through such a job encounters a trap, isn't it having the user VMID? If so, when is this kernel queue related TBA/TMA helpful or selected? If it's only for driver submitted jobs, then this control to user is not required.

Each fpriv GPUVM will have a copy of the first level trap handler
mapped at the same GPU virtual address.  If the user requests a second
level trap handler, their copy of the first level trap handler will be
updated to point to the provided second level trap handler.

Alex

>
> Thanks,
> Lijo
> ________________________________
> From: SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>
> Sent: Wednesday, 09 September 2026 18:32:03
> To: Lazar, Lijo <Lijo.Lazar@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com>
> Cc: amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>; Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
> Subject: Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure
>
>
>
> On 9/9/2026 1:15 PM, Lazar, Lijo wrote:
>
>
>
> On 05-Sep-26 1:49 PM, Srinivasan Shanmugam wrote:
>
> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
> On GFX10 and earlier HWS-based hardware, the driver programs trap
> registers via SRBM select for KFD queues but no equivalent exists for
> driver-managed kernel queue VMIDs.
>
> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
> there, so per-VM isolation is handled entirely by page tables without
> needing to reprogram the register per job or per submission.
>
> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
> (parallel to page table allocation) and mapped read-only into the GPU VM
> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
> The first-level CWSR handler reads this address to chain to the
> second-level handler when a shader exception fires.
>
> This design is:
>    - Per-VM BO (not device-level) — same model as page tables
>    - Fixed VA in each VM's address space — same VA, different physical BO
>    - Read-only from GPU — kernel CPU updates it via CPU mapping
>    - Treat allocation/free lifecycle identical to page tables
>
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
> Cc: Lijo Lazar <lijo.lazar@amd.com>
> Cc: Timur Kristóf <timur.kristof@gmail.com>
> Cc: Samuel Pitoiset <hakzsam@gmail.com>
> Cc: Natalie Vock <natalie.vock@gmx.de>
> Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>   5 files changed, 110 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> index 3ca187f5ade8..5624a5ab5c62 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>       void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
>                            uint32_t status);
>       uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
> +    void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
>   };
>     struct amdgpu_vmhub {
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> index 623cac6781be..e913488ca3fa 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>         amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>       adev->trap_info = no_free_ptr(trap_info);
> +    amdgpu_trap_program_kernel_vmids(adev);
>         return 0;
>   }
> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
>       adev->trap_info = NULL;
>   }
>   +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
> +{
> +    struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
> +
> +    if (!amdgpu_trap_is_enabled(adev))
> +        return;
> +    if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
> +        return;
> +
> +    hub->vmhub_funcs->program_kernel_trap_vmids(adev);
> +}
> +
> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> +                struct amdgpu_vm *vm)
> +{
> +    void *cpu_addr;
> +    uint64_t va;
> +    int r;
> +
> +    dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> +
> +    r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
> +                    AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
> +                    NULL, &cpu_addr);
> +    if (r)
> +        return r;
> +
> +    if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
> +        iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
> +                      (void __iomem *)cpu_addr);
> +    else
> +        iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
> +
> +    vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
> +    if (!vm->kq_tma_va) {
> +        r = -ENOMEM;
> +        goto err_free_bo;
> +    }
> +
> +    va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
>
>
> Is this the same address used for mapping of TMA for user queues?
>
> No — these are different, non-overlapping addresses in the reserved VA region:
>
> AMDGPU_VA_RESERVED_TRAP_UQ_START = TRAP_START − 12 KiB
> → used for UQ first-level TBA (8 KiB) + TMA (4 KiB)
> AMDGPU_VA_RESERVED_TRAP_START = SEQ64_START − 64 KiB
> → used for KQ per-VM TMA (this patch)
>
> The UQ region sits immediately below the KQ region in the reserved VA
> space. No collision between the two mappings in the same VM.
>
> Regards, Srini

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Lazar, Lijo]

[-- Attachment #1: Type: text/plain, Size: 7872 bytes --]

AMD General

My question was slightly different context - The first level TMA is fetched based on VMID of execution context .

If user waves are executed in IB VMID context (even though submitted through kernel queues), do we need to allow user to install handlers for kernel VMIDs?

Thanks,
Lijo
________________________________
From: Alex Deucher <alexdeucher@gmail.com>
Sent: Thursday, 10 September 2026 00:24:03
To: Lazar, Lijo <Lijo.Lazar@amd.com>
Cc: SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com>; amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>; Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
Subject: Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

On Wed, Sep 9, 2026 at 2:05 PM Lazar, Lijo <Lijo.Lazar@amd.com> wrote:
>
> AMD General
>
>
> One generic question - I am assuming the overall purpose is to debug a user job submitted to kernel queue. When user IBs are submitted to kernel queue, those IBs carry VMID assigned to user. When wave submitted through such a job encounters a trap, isn't it having the user VMID? If so, when is this kernel queue related TBA/TMA helpful or selected? If it's only for driver submitted jobs, then this control to user is not required.

Each fpriv GPUVM will have a copy of the first level trap handler
mapped at the same GPU virtual address.  If the user requests a second
level trap handler, their copy of the first level trap handler will be
updated to point to the provided second level trap handler.

Alex

>
> Thanks,
> Lijo
> ________________________________
> From: SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>
> Sent: Wednesday, 09 September 2026 18:32:03
> To: Lazar, Lijo <Lijo.Lazar@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com>
> Cc: amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>; Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
> Subject: Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure
>
>
>
> On 9/9/2026 1:15 PM, Lazar, Lijo wrote:
>
>
>
> On 05-Sep-26 1:49 PM, Srinivasan Shanmugam wrote:
>
> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
> On GFX10 and earlier HWS-based hardware, the driver programs trap
> registers via SRBM select for KFD queues but no equivalent exists for
> driver-managed kernel queue VMIDs.
>
> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
> there, so per-VM isolation is handled entirely by page tables without
> needing to reprogram the register per job or per submission.
>
> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
> (parallel to page table allocation) and mapped read-only into the GPU VM
> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
> The first-level CWSR handler reads this address to chain to the
> second-level handler when a shader exception fires.
>
> This design is:
>    - Per-VM BO (not device-level) — same model as page tables
>    - Fixed VA in each VM's address space — same VA, different physical BO
>    - Read-only from GPU — kernel CPU updates it via CPU mapping
>    - Treat allocation/free lifecycle identical to page tables
>
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
> Cc: Lijo Lazar <lijo.lazar@amd.com>
> Cc: Timur Kristóf <timur.kristof@gmail.com>
> Cc: Samuel Pitoiset <hakzsam@gmail.com>
> Cc: Natalie Vock <natalie.vock@gmx.de>
> Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>   5 files changed, 110 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> index 3ca187f5ade8..5624a5ab5c62 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>       void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
>                            uint32_t status);
>       uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
> +    void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
>   };
>     struct amdgpu_vmhub {
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> index 623cac6781be..e913488ca3fa 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>         amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>       adev->trap_info = no_free_ptr(trap_info);
> +    amdgpu_trap_program_kernel_vmids(adev);
>         return 0;
>   }
> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
>       adev->trap_info = NULL;
>   }
>   +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
> +{
> +    struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
> +
> +    if (!amdgpu_trap_is_enabled(adev))
> +        return;
> +    if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
> +        return;
> +
> +    hub->vmhub_funcs->program_kernel_trap_vmids(adev);
> +}
> +
> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> +                struct amdgpu_vm *vm)
> +{
> +    void *cpu_addr;
> +    uint64_t va;
> +    int r;
> +
> +    dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> +
> +    r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
> +                    AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
> +                    NULL, &cpu_addr);
> +    if (r)
> +        return r;
> +
> +    if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
> +        iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
> +                      (void __iomem *)cpu_addr);
> +    else
> +        iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
> +
> +    vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
> +    if (!vm->kq_tma_va) {
> +        r = -ENOMEM;
> +        goto err_free_bo;
> +    }
> +
> +    va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
>
>
> Is this the same address used for mapping of TMA for user queues?
>
> No — these are different, non-overlapping addresses in the reserved VA region:
>
> AMDGPU_VA_RESERVED_TRAP_UQ_START = TRAP_START − 12 KiB
> → used for UQ first-level TBA (8 KiB) + TMA (4 KiB)
> AMDGPU_VA_RESERVED_TRAP_START = SEQ64_START − 64 KiB
> → used for KQ per-VM TMA (this patch)
>
> The UQ region sits immediately below the KQ region in the reserved VA
> space. No collision between the two mappings in the same VM.
>
> Regards, Srini

[-- Attachment #2: Type: text/html, Size: 12653 bytes --]

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Lazar, Lijo]

On 10-Sep-26 7:36 AM, Lazar, Lijo wrote:
> AMD General
> 
> 
> My question was slightly different context - The first level TMA is 
> fetched based on VMID of execution context .
> 
> If user waves are executed in IB VMID context (even though submitted 
> through kernel queues), do we need to allow user to install handlers for 
> kernel VMIDs?
> 


To clarify -

For other non-zero VMIDs, is there a need to allocate separate BOs like
kq_tba_bo/kq_tma_bo?

Can't we keep just one set of tba/tma bo for first level? User handling 
may always be through second level as the ioctl allows user to install 
only second level ones. If separate handling is required based on queue 
type (kq vs uq), I think trap handler can identify the queue based on 
doorbell offset.

Thanks,
Lijo

> Thanks,
> Lijo
> ------------------------------------------------------------------------
> *From:* Alex Deucher <alexdeucher@gmail.com>
> *Sent:* Thursday, 10 September 2026 00:24:03
> *To:* Lazar, Lijo <Lijo.Lazar@amd.com>
> *Cc:* SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>; Koenig, 
> Christian <Christian.Koenig@amd.com>; Deucher, Alexander 
> <Alexander.Deucher@amd.com>; amd-gfx@lists.freedesktop.org <amd- 
> gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>; 
> Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
> *Subject:* Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first- 
> level trap handler infrastructure
> On Wed, Sep 9, 2026 at 2:05 PM Lazar, Lijo <Lijo.Lazar@amd.com> wrote:
>>
>> AMD General
>>
>>
>> One generic question - I am assuming the overall purpose is to debug a user job submitted to kernel queue. When user IBs are submitted to kernel queue, those IBs carry VMID assigned to user. When wave submitted through such a job encounters a trap, isn't it having the user VMID? If so, when is this kernel queue related TBA/ 
> TMA helpful or selected? If it's only for driver submitted jobs, then 
> this control to user is not required.
> 
> Each fpriv GPUVM will have a copy of the first level trap handler
> mapped at the same GPU virtual address.  If the user requests a second
> level trap handler, their copy of the first level trap handler will be
> updated to point to the provided second level trap handler.
> 
> Alex
> 
>>
>> Thanks,
>> Lijo
>> ________________________________
>> From: SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>
>> Sent: Wednesday, 09 September 2026 18:32:03
>> To: Lazar, Lijo <Lijo.Lazar@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com>
>> Cc: amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>; Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
>> Subject: Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure
>>
>>
>>
>> On 9/9/2026 1:15 PM, Lazar, Lijo wrote:
>>
>>
>>
>> On 05-Sep-26 1:49 PM, Srinivasan Shanmugam wrote:
>>
>> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
>> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
>> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
>> On GFX10 and earlier HWS-based hardware, the driver programs trap
>> registers via SRBM select for KFD queues but no equivalent exists for
>> driver-managed kernel queue VMIDs.
>>
>> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
>> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
>> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
>> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
>> there, so per-VM isolation is handled entirely by page tables without
>> needing to reprogram the register per job or per submission.
>>
>> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
>> (parallel to page table allocation) and mapped read-only into the GPU VM
>> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
>> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
>> The first-level CWSR handler reads this address to chain to the
>> second-level handler when a shader exception fires.
>>
>> This design is:
>>    - Per-VM BO (not device-level) — same model as page tables
>>    - Fixed VA in each VM's address space — same VA, different physical BO
>>    - Read-only from GPU — kernel CPU updates it via CPU mapping
>>    - Treat allocation/free lifecycle identical to page tables
>>
>> Suggested-by: Christian König <christian.koenig@amd.com>
>> Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
>> Cc: Lijo Lazar <lijo.lazar@amd.com>
>> Cc: Timur Kristóf <timur.kristof@gmail.com>
>> Cc: Samuel Pitoiset <hakzsam@gmail.com>
>> Cc: Natalie Vock <natalie.vock@gmx.de>
>> Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
>> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>>   5 files changed, 110 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>> index 3ca187f5ade8..5624a5ab5c62 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>>       void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
>>                            uint32_t status);
>>       uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
>> +    void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
>>   };
>>     struct amdgpu_vmhub {
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>> index 623cac6781be..e913488ca3fa 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>>         amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>>       adev->trap_info = no_free_ptr(trap_info);
>> +    amdgpu_trap_program_kernel_vmids(adev);
>>         return 0;
>>   }
>> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
>>       adev->trap_info = NULL;
>>   }
>>   +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
>> +{
>> +    struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
>> +
>> +    if (!amdgpu_trap_is_enabled(adev))
>> +        return;
>> +    if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
>> +        return;
>> +
>> +    hub->vmhub_funcs->program_kernel_trap_vmids(adev);
>> +}
>> +
>> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>> +                struct amdgpu_vm *vm)
>> +{
>> +    void *cpu_addr;
>> +    uint64_t va;
>> +    int r;
>> +
>> +    dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>> +
>> +    r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
>> +                    AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
>> +                    NULL, &cpu_addr);
>> +    if (r)
>> +        return r;
>> +
>> +    if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
>> +        iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
>> +                      (void __iomem *)cpu_addr);
>> +    else
>> +        iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
>> +
>> +    vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
>> +    if (!vm->kq_tma_va) {
>> +        r = -ENOMEM;
>> +        goto err_free_bo;
>> +    }
>> +
>> +    va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
>>
>>
>> Is this the same address used for mapping of TMA for user queues?
>>
>> No — these are different, non-overlapping addresses in the reserved VA region:
>>
>> AMDGPU_VA_RESERVED_TRAP_UQ_START = TRAP_START − 12 KiB
>> → used for UQ first-level TBA (8 KiB) + TMA (4 KiB)
>> AMDGPU_VA_RESERVED_TRAP_START = SEQ64_START − 64 KiB
>> → used for KQ per-VM TMA (this patch)
>>
>> The UQ region sits immediately below the KQ region in the reserved VA
>> space. No collision between the two mappings in the same VM.
>>
>> Regards, Srini

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Alex Deucher]

On Thu, Sep 10, 2026 at 1:23 AM Lazar, Lijo <lijo.lazar@amd.com> wrote:
>
>
>
> On 10-Sep-26 7:36 AM, Lazar, Lijo wrote:
> > AMD General
> >
> >
> > My question was slightly different context - The first level TMA is
> > fetched based on VMID of execution context .
> >
> > If user waves are executed in IB VMID context (even though submitted
> > through kernel queues), do we need to allow user to install handlers for
> > kernel VMIDs?
> >
>
>
> To clarify -
>
> For other non-zero VMIDs, is there a need to allocate separate BOs like
> kq_tba_bo/kq_tma_bo?
>
> Can't we keep just one set of tba/tma bo for first level? User handling
> may always be through second level as the ioctl allows user to install
> only second level ones. If separate handling is required based on queue
> type (kq vs uq), I think trap handler can identify the queue based on
> doorbell offset.

Maybe I'm misunderstanding how the second level registration works.  I
thought the pointer to the second level trap handler was changed in
the first level trap handler when the user registers the second level
trap handler. If so, we need a per VM BO for the first level trap
handler.  Then the second level trap handler is either NULL or
whatever the user registers for their GPU VM.

Alex

>
> Thanks,
> Lijo
>
> > Thanks,
> > Lijo
> > ------------------------------------------------------------------------
> > *From:* Alex Deucher <alexdeucher@gmail.com>
> > *Sent:* Thursday, 10 September 2026 00:24:03
> > *To:* Lazar, Lijo <Lijo.Lazar@amd.com>
> > *Cc:* SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>; Koenig,
> > Christian <Christian.Koenig@amd.com>; Deucher, Alexander
> > <Alexander.Deucher@amd.com>; amd-gfx@lists.freedesktop.org <amd-
> > gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>;
> > Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
> > *Subject:* Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-
> > level trap handler infrastructure
> > On Wed, Sep 9, 2026 at 2:05 PM Lazar, Lijo <Lijo.Lazar@amd.com> wrote:
> >>
> >> AMD General
> >>
> >>
> >> One generic question - I am assuming the overall purpose is to debug a user job submitted to kernel queue. When user IBs are submitted to kernel queue, those IBs carry VMID assigned to user. When wave submitted through such a job encounters a trap, isn't it having the user VMID? If so, when is this kernel queue related TBA/
> > TMA helpful or selected? If it's only for driver submitted jobs, then
> > this control to user is not required.
> >
> > Each fpriv GPUVM will have a copy of the first level trap handler
> > mapped at the same GPU virtual address.  If the user requests a second
> > level trap handler, their copy of the first level trap handler will be
> > updated to point to the provided second level trap handler.
> >
> > Alex
> >
> >>
> >> Thanks,
> >> Lijo
> >> ________________________________
> >> From: SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>
> >> Sent: Wednesday, 09 September 2026 18:32:03
> >> To: Lazar, Lijo <Lijo.Lazar@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com>
> >> Cc: amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>; Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
> >> Subject: Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure
> >>
> >>
> >>
> >> On 9/9/2026 1:15 PM, Lazar, Lijo wrote:
> >>
> >>
> >>
> >> On 05-Sep-26 1:49 PM, Srinivasan Shanmugam wrote:
> >>
> >> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
> >> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
> >> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
> >> On GFX10 and earlier HWS-based hardware, the driver programs trap
> >> registers via SRBM select for KFD queues but no equivalent exists for
> >> driver-managed kernel queue VMIDs.
> >>
> >> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
> >> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
> >> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
> >> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
> >> there, so per-VM isolation is handled entirely by page tables without
> >> needing to reprogram the register per job or per submission.
> >>
> >> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
> >> (parallel to page table allocation) and mapped read-only into the GPU VM
> >> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
> >> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
> >> The first-level CWSR handler reads this address to chain to the
> >> second-level handler when a shader exception fires.
> >>
> >> This design is:
> >>    - Per-VM BO (not device-level) — same model as page tables
> >>    - Fixed VA in each VM's address space — same VA, different physical BO
> >>    - Read-only from GPU — kernel CPU updates it via CPU mapping
> >>    - Treat allocation/free lifecycle identical to page tables
> >>
> >> Suggested-by: Christian König <christian.koenig@amd.com>
> >> Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
> >> Cc: Lijo Lazar <lijo.lazar@amd.com>
> >> Cc: Timur Kristóf <timur.kristof@gmail.com>
> >> Cc: Samuel Pitoiset <hakzsam@gmail.com>
> >> Cc: Natalie Vock <natalie.vock@gmx.de>
> >> Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
> >> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
> >> ---
> >>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
> >>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
> >>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
> >>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
> >>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
> >>   5 files changed, 110 insertions(+)
> >>
> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> >> index 3ca187f5ade8..5624a5ab5c62 100644
> >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> >> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
> >>       void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
> >>                            uint32_t status);
> >>       uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
> >> +    void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
> >>   };
> >>     struct amdgpu_vmhub {
> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> >> index 623cac6781be..e913488ca3fa 100644
> >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> >> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
> >>         amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
> >>       adev->trap_info = no_free_ptr(trap_info);
> >> +    amdgpu_trap_program_kernel_vmids(adev);
> >>         return 0;
> >>   }
> >> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
> >>       adev->trap_info = NULL;
> >>   }
> >>   +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
> >> +{
> >> +    struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
> >> +
> >> +    if (!amdgpu_trap_is_enabled(adev))
> >> +        return;
> >> +    if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
> >> +        return;
> >> +
> >> +    hub->vmhub_funcs->program_kernel_trap_vmids(adev);
> >> +}
> >> +
> >> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> >> +                struct amdgpu_vm *vm)
> >> +{
> >> +    void *cpu_addr;
> >> +    uint64_t va;
> >> +    int r;
> >> +
> >> +    dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> >> +
> >> +    r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
> >> +                    AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
> >> +                    NULL, &cpu_addr);
> >> +    if (r)
> >> +        return r;
> >> +
> >> +    if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
> >> +        iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
> >> +                      (void __iomem *)cpu_addr);
> >> +    else
> >> +        iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
> >> +
> >> +    vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
> >> +    if (!vm->kq_tma_va) {
> >> +        r = -ENOMEM;
> >> +        goto err_free_bo;
> >> +    }
> >> +
> >> +    va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
> >>
> >>
> >> Is this the same address used for mapping of TMA for user queues?
> >>
> >> No — these are different, non-overlapping addresses in the reserved VA region:
> >>
> >> AMDGPU_VA_RESERVED_TRAP_UQ_START = TRAP_START − 12 KiB
> >> → used for UQ first-level TBA (8 KiB) + TMA (4 KiB)
> >> AMDGPU_VA_RESERVED_TRAP_START = SEQ64_START − 64 KiB
> >> → used for KQ per-VM TMA (this patch)
> >>
> >> The UQ region sits immediately below the KQ region in the reserved VA
> >> space. No collision between the two mappings in the same VM.
> >>
> >> Regards, Srini
>

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Alex Deucher]

On Fri, Sep 11, 2026 at 10:07 AM Alex Deucher <alexdeucher@gmail.com> wrote:
>
> On Thu, Sep 10, 2026 at 1:23 AM Lazar, Lijo <lijo.lazar@amd.com> wrote:
> >
> >
> >
> > On 10-Sep-26 7:36 AM, Lazar, Lijo wrote:
> > > AMD General
> > >
> > >
> > > My question was slightly different context - The first level TMA is
> > > fetched based on VMID of execution context .
> > >
> > > If user waves are executed in IB VMID context (even though submitted
> > > through kernel queues), do we need to allow user to install handlers for
> > > kernel VMIDs?
> > >
> >
> >
> > To clarify -
> >
> > For other non-zero VMIDs, is there a need to allocate separate BOs like
> > kq_tba_bo/kq_tma_bo?
> >
> > Can't we keep just one set of tba/tma bo for first level? User handling
> > may always be through second level as the ioctl allows user to install
> > only second level ones. If separate handling is required based on queue
> > type (kq vs uq), I think trap handler can identify the queue based on
> > doorbell offset.
>
> Maybe I'm misunderstanding how the second level registration works.  I
> thought the pointer to the second level trap handler was changed in
> the first level trap handler when the user registers the second level
> trap handler. If so, we need a per VM BO for the first level trap
> handler.  Then the second level trap handler is either NULL or
> whatever the user registers for their GPU VM.
>

I guess the TBA and TMA could be split and the TBA could be a common
allocation and the TMA could be per VM.  Was that what you were
getting at?

Alex

> Alex
>
> >
> > Thanks,
> > Lijo
> >
> > > Thanks,
> > > Lijo
> > > ------------------------------------------------------------------------
> > > *From:* Alex Deucher <alexdeucher@gmail.com>
> > > *Sent:* Thursday, 10 September 2026 00:24:03
> > > *To:* Lazar, Lijo <Lijo.Lazar@amd.com>
> > > *Cc:* SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>; Koenig,
> > > Christian <Christian.Koenig@amd.com>; Deucher, Alexander
> > > <Alexander.Deucher@amd.com>; amd-gfx@lists.freedesktop.org <amd-
> > > gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>;
> > > Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
> > > *Subject:* Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-
> > > level trap handler infrastructure
> > > On Wed, Sep 9, 2026 at 2:05 PM Lazar, Lijo <Lijo.Lazar@amd.com> wrote:
> > >>
> > >> AMD General
> > >>
> > >>
> > >> One generic question - I am assuming the overall purpose is to debug a user job submitted to kernel queue. When user IBs are submitted to kernel queue, those IBs carry VMID assigned to user. When wave submitted through such a job encounters a trap, isn't it having the user VMID? If so, when is this kernel queue related TBA/
> > > TMA helpful or selected? If it's only for driver submitted jobs, then
> > > this control to user is not required.
> > >
> > > Each fpriv GPUVM will have a copy of the first level trap handler
> > > mapped at the same GPU virtual address.  If the user requests a second
> > > level trap handler, their copy of the first level trap handler will be
> > > updated to point to the provided second level trap handler.
> > >
> > > Alex
> > >
> > >>
> > >> Thanks,
> > >> Lijo
> > >> ________________________________
> > >> From: SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>
> > >> Sent: Wednesday, 09 September 2026 18:32:03
> > >> To: Lazar, Lijo <Lijo.Lazar@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com>
> > >> Cc: amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>; Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
> > >> Subject: Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure
> > >>
> > >>
> > >>
> > >> On 9/9/2026 1:15 PM, Lazar, Lijo wrote:
> > >>
> > >>
> > >>
> > >> On 05-Sep-26 1:49 PM, Srinivasan Shanmugam wrote:
> > >>
> > >> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
> > >> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
> > >> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
> > >> On GFX10 and earlier HWS-based hardware, the driver programs trap
> > >> registers via SRBM select for KFD queues but no equivalent exists for
> > >> driver-managed kernel queue VMIDs.
> > >>
> > >> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
> > >> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
> > >> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
> > >> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
> > >> there, so per-VM isolation is handled entirely by page tables without
> > >> needing to reprogram the register per job or per submission.
> > >>
> > >> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
> > >> (parallel to page table allocation) and mapped read-only into the GPU VM
> > >> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
> > >> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
> > >> The first-level CWSR handler reads this address to chain to the
> > >> second-level handler when a shader exception fires.
> > >>
> > >> This design is:
> > >>    - Per-VM BO (not device-level) — same model as page tables
> > >>    - Fixed VA in each VM's address space — same VA, different physical BO
> > >>    - Read-only from GPU — kernel CPU updates it via CPU mapping
> > >>    - Treat allocation/free lifecycle identical to page tables
> > >>
> > >> Suggested-by: Christian König <christian.koenig@amd.com>
> > >> Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
> > >> Cc: Lijo Lazar <lijo.lazar@amd.com>
> > >> Cc: Timur Kristóf <timur.kristof@gmail.com>
> > >> Cc: Samuel Pitoiset <hakzsam@gmail.com>
> > >> Cc: Natalie Vock <natalie.vock@gmx.de>
> > >> Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
> > >> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
> > >> ---
> > >>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
> > >>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
> > >>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
> > >>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
> > >>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
> > >>   5 files changed, 110 insertions(+)
> > >>
> > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> > >> index 3ca187f5ade8..5624a5ab5c62 100644
> > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> > >> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
> > >>       void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
> > >>                            uint32_t status);
> > >>       uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
> > >> +    void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
> > >>   };
> > >>     struct amdgpu_vmhub {
> > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> > >> index 623cac6781be..e913488ca3fa 100644
> > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> > >> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
> > >>         amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
> > >>       adev->trap_info = no_free_ptr(trap_info);
> > >> +    amdgpu_trap_program_kernel_vmids(adev);
> > >>         return 0;
> > >>   }
> > >> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
> > >>       adev->trap_info = NULL;
> > >>   }
> > >>   +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
> > >> +{
> > >> +    struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
> > >> +
> > >> +    if (!amdgpu_trap_is_enabled(adev))
> > >> +        return;
> > >> +    if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
> > >> +        return;
> > >> +
> > >> +    hub->vmhub_funcs->program_kernel_trap_vmids(adev);
> > >> +}
> > >> +
> > >> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> > >> +                struct amdgpu_vm *vm)
> > >> +{
> > >> +    void *cpu_addr;
> > >> +    uint64_t va;
> > >> +    int r;
> > >> +
> > >> +    dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> > >> +
> > >> +    r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
> > >> +                    AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
> > >> +                    NULL, &cpu_addr);
> > >> +    if (r)
> > >> +        return r;
> > >> +
> > >> +    if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
> > >> +        iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
> > >> +                      (void __iomem *)cpu_addr);
> > >> +    else
> > >> +        iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
> > >> +
> > >> +    vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
> > >> +    if (!vm->kq_tma_va) {
> > >> +        r = -ENOMEM;
> > >> +        goto err_free_bo;
> > >> +    }
> > >> +
> > >> +    va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
> > >>
> > >>
> > >> Is this the same address used for mapping of TMA for user queues?
> > >>
> > >> No — these are different, non-overlapping addresses in the reserved VA region:
> > >>
> > >> AMDGPU_VA_RESERVED_TRAP_UQ_START = TRAP_START − 12 KiB
> > >> → used for UQ first-level TBA (8 KiB) + TMA (4 KiB)
> > >> AMDGPU_VA_RESERVED_TRAP_START = SEQ64_START − 64 KiB
> > >> → used for KQ per-VM TMA (this patch)
> > >>
> > >> The UQ region sits immediately below the KQ region in the reserved VA
> > >> space. No collision between the two mappings in the same VM.
> > >>
> > >> Regards, Srini
> >

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Lazar, Lijo]

On 11-Sep-26 7:39 PM, Alex Deucher wrote:
> On Fri, Sep 11, 2026 at 10:07 AM Alex Deucher <alexdeucher@gmail.com> wrote:
>>
>> On Thu, Sep 10, 2026 at 1:23 AM Lazar, Lijo <lijo.lazar@amd.com> wrote:
>>>
>>>
>>>
>>> On 10-Sep-26 7:36 AM, Lazar, Lijo wrote:
>>>> AMD General
>>>>
>>>>
>>>> My question was slightly different context - The first level TMA is
>>>> fetched based on VMID of execution context .
>>>>
>>>> If user waves are executed in IB VMID context (even though submitted
>>>> through kernel queues), do we need to allow user to install handlers for
>>>> kernel VMIDs?
>>>>
>>>
>>>
>>> To clarify -
>>>
>>> For other non-zero VMIDs, is there a need to allocate separate BOs like
>>> kq_tba_bo/kq_tma_bo?
>>>
>>> Can't we keep just one set of tba/tma bo for first level? User handling
>>> may always be through second level as the ioctl allows user to install
>>> only second level ones. If separate handling is required based on queue
>>> type (kq vs uq), I think trap handler can identify the queue based on
>>> doorbell offset.
>>
>> Maybe I'm misunderstanding how the second level registration works.  I
>> thought the pointer to the second level trap handler was changed in
>> the first level trap handler when the user registers the second level
>> trap handler. If so, we need a per VM BO for the first level trap
>> handler.  Then the second level trap handler is either NULL or
>> whatever the user registers for their GPU VM.
>>
> 
> I guess the TBA and TMA could be split and the TBA could be a common
> allocation and the TMA could be per VM.  Was that what you were
> getting at?
> 

I was getting at using a single set of TMA (only one tma bo, not one 
more kq_tma_bo) for first level trap. Later I noticed the thread with 
Srini and I see you asking for the same (i.e, only to program SQ 
registers for non-kfd VMIDs also with the address of the single TMA BO 
allocated and not have one more kq_tma_bo).

Thanks,
Lijo

> Alex
> 
>> Alex
>>
>>>
>>> Thanks,
>>> Lijo
>>>
>>>> Thanks,
>>>> Lijo
>>>> ------------------------------------------------------------------------
>>>> *From:* Alex Deucher <alexdeucher@gmail.com>
>>>> *Sent:* Thursday, 10 September 2026 00:24:03
>>>> *To:* Lazar, Lijo <Lijo.Lazar@amd.com>
>>>> *Cc:* SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>; Koenig,
>>>> Christian <Christian.Koenig@amd.com>; Deucher, Alexander
>>>> <Alexander.Deucher@amd.com>; amd-gfx@lists.freedesktop.org <amd-
>>>> gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>;
>>>> Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
>>>> *Subject:* Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-
>>>> level trap handler infrastructure
>>>> On Wed, Sep 9, 2026 at 2:05 PM Lazar, Lijo <Lijo.Lazar@amd.com> wrote:
>>>>>
>>>>> AMD General
>>>>>
>>>>>
>>>>> One generic question - I am assuming the overall purpose is to debug a user job submitted to kernel queue. When user IBs are submitted to kernel queue, those IBs carry VMID assigned to user. When wave submitted through such a job encounters a trap, isn't it having the user VMID? If so, when is this kernel queue related TBA/
>>>> TMA helpful or selected? If it's only for driver submitted jobs, then
>>>> this control to user is not required.
>>>>
>>>> Each fpriv GPUVM will have a copy of the first level trap handler
>>>> mapped at the same GPU virtual address.  If the user requests a second
>>>> level trap handler, their copy of the first level trap handler will be
>>>> updated to point to the provided second level trap handler.
>>>>
>>>> Alex
>>>>
>>>>>
>>>>> Thanks,
>>>>> Lijo
>>>>> ________________________________
>>>>> From: SHANMUGAM, SRINIVASAN <SRINIVASAN.SHANMUGAM@amd.com>
>>>>> Sent: Wednesday, 09 September 2026 18:32:03
>>>>> To: Lazar, Lijo <Lijo.Lazar@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com>
>>>>> Cc: amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>; Timur Kristóf <timur.kristof@gmail.com>; Samuel Pitoiset <hakzsam@gmail.com>; Natalie Vock <natalie.vock@gmx.de>
>>>>> Subject: Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure
>>>>>
>>>>>
>>>>>
>>>>> On 9/9/2026 1:15 PM, Lazar, Lijo wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 05-Sep-26 1:49 PM, Srinivasan Shanmugam wrote:
>>>>>
>>>>> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
>>>>> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
>>>>> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
>>>>> On GFX10 and earlier HWS-based hardware, the driver programs trap
>>>>> registers via SRBM select for KFD queues but no equivalent exists for
>>>>> driver-managed kernel queue VMIDs.
>>>>>
>>>>> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
>>>>> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
>>>>> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
>>>>> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
>>>>> there, so per-VM isolation is handled entirely by page tables without
>>>>> needing to reprogram the register per job or per submission.
>>>>>
>>>>> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
>>>>> (parallel to page table allocation) and mapped read-only into the GPU VM
>>>>> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
>>>>> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
>>>>> The first-level CWSR handler reads this address to chain to the
>>>>> second-level handler when a shader exception fires.
>>>>>
>>>>> This design is:
>>>>>     - Per-VM BO (not device-level) — same model as page tables
>>>>>     - Fixed VA in each VM's address space — same VA, different physical BO
>>>>>     - Read-only from GPU — kernel CPU updates it via CPU mapping
>>>>>     - Treat allocation/free lifecycle identical to page tables
>>>>>
>>>>> Suggested-by: Christian König <christian.koenig@amd.com>
>>>>> Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
>>>>> Cc: Lijo Lazar <lijo.lazar@amd.com>
>>>>> Cc: Timur Kristóf <timur.kristof@gmail.com>
>>>>> Cc: Samuel Pitoiset <hakzsam@gmail.com>
>>>>> Cc: Natalie Vock <natalie.vock@gmx.de>
>>>>> Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
>>>>> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
>>>>> ---
>>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
>>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>>>>>    5 files changed, 110 insertions(+)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>> index 3ca187f5ade8..5624a5ab5c62 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>>>>>        void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
>>>>>                             uint32_t status);
>>>>>        uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
>>>>> +    void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
>>>>>    };
>>>>>      struct amdgpu_vmhub {
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>> index 623cac6781be..e913488ca3fa 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>>>>>          amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>>>>>        adev->trap_info = no_free_ptr(trap_info);
>>>>> +    amdgpu_trap_program_kernel_vmids(adev);
>>>>>          return 0;
>>>>>    }
>>>>> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
>>>>>        adev->trap_info = NULL;
>>>>>    }
>>>>>    +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
>>>>> +{
>>>>> +    struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
>>>>> +
>>>>> +    if (!amdgpu_trap_is_enabled(adev))
>>>>> +        return;
>>>>> +    if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
>>>>> +        return;
>>>>> +
>>>>> +    hub->vmhub_funcs->program_kernel_trap_vmids(adev);
>>>>> +}
>>>>> +
>>>>> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>>>>> +                struct amdgpu_vm *vm)
>>>>> +{
>>>>> +    void *cpu_addr;
>>>>> +    uint64_t va;
>>>>> +    int r;
>>>>> +
>>>>> +    dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>>>>> +
>>>>> +    r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
>>>>> +                    AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
>>>>> +                    NULL, &cpu_addr);
>>>>> +    if (r)
>>>>> +        return r;
>>>>> +
>>>>> +    if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
>>>>> +        iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
>>>>> +                      (void __iomem *)cpu_addr);
>>>>> +    else
>>>>> +        iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
>>>>> +
>>>>> +    vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
>>>>> +    if (!vm->kq_tma_va) {
>>>>> +        r = -ENOMEM;
>>>>> +        goto err_free_bo;
>>>>> +    }
>>>>> +
>>>>> +    va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
>>>>>
>>>>>
>>>>> Is this the same address used for mapping of TMA for user queues?
>>>>>
>>>>> No — these are different, non-overlapping addresses in the reserved VA region:
>>>>>
>>>>> AMDGPU_VA_RESERVED_TRAP_UQ_START = TRAP_START − 12 KiB
>>>>> → used for UQ first-level TBA (8 KiB) + TMA (4 KiB)
>>>>> AMDGPU_VA_RESERVED_TRAP_START = SEQ64_START − 64 KiB
>>>>> → used for KQ per-VM TMA (this patch)
>>>>>
>>>>> The UQ region sits immediately below the KQ region in the reserved VA
>>>>> space. No collision between the two mappings in the same VM.
>>>>>
>>>>> Regards, Srini
>>>

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Alex Deucher]

On Sat, Sep 5, 2026 at 4:55 AM Srinivasan Shanmugam
<srinivasan.shanmugam@amd.com> wrote:
>
> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program

MES doesn't own the kernel queue VMIDs.  The kernel driver does.  MES
assigns VMIDs dynamically for user queues, but for kernel queues, the
kernel assigns the vmids.

> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
> On GFX10 and earlier HWS-based hardware, the driver programs trap
> registers via SRBM select for KFD queues but no equivalent exists for
> driver-managed kernel queue VMIDs.
>
> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
> there, so per-VM isolation is handled entirely by page tables without
> needing to reprogram the register per job or per submission.
>
> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
> (parallel to page table allocation) and mapped read-only into the GPU VM
> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
> The first-level CWSR handler reads this address to chain to the
> second-level handler when a shader exception fires.
>
> This design is:
>   - Per-VM BO (not device-level) — same model as page tables
>   - Fixed VA in each VM's address space — same VA, different physical BO
>   - Read-only from GPU — kernel CPU updates it via CPU mapping
>   - Treat allocation/free lifecycle identical to page tables
>
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
> Cc: Lijo Lazar <lijo.lazar@amd.com>
> Cc: Timur Kristóf <timur.kristof@gmail.com>
> Cc: Samuel Pitoiset <hakzsam@gmail.com>
> Cc: Natalie Vock <natalie.vock@gmx.de>
> Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>  5 files changed, 110 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> index 3ca187f5ade8..5624a5ab5c62 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>         void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
>                                                  uint32_t status);
>         uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
> +       void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
>  };
>
>  struct amdgpu_vmhub {
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> index 623cac6781be..e913488ca3fa 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>
>         amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>         adev->trap_info = no_free_ptr(trap_info);
> +       amdgpu_trap_program_kernel_vmids(adev);
>
>         return 0;
>  }
> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
>         adev->trap_info = NULL;
>  }
>
> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
> +{
> +       struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
> +
> +       if (!amdgpu_trap_is_enabled(adev))
> +               return;
> +       if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
> +               return;
> +
> +       hub->vmhub_funcs->program_kernel_trap_vmids(adev);
> +}
> +
> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> +                               struct amdgpu_vm *vm)
> +{
> +       void *cpu_addr;
> +       uint64_t va;
> +       int r;
> +
> +       dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> +
> +       r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
> +                                   AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
> +                                   NULL, &cpu_addr);
> +       if (r)
> +               return r;
> +
> +       if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
> +               iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
> +                                         (void __iomem *)cpu_addr);
> +       else
> +               iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);

Are you planning to copy the first level trap handler to the buffer
later?  Why not just initialize it here?

Alex

> +
> +       vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
> +       if (!vm->kq_tma_va) {
> +               r = -ENOMEM;
> +               goto err_free_bo;
> +       }
> +
> +       va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
> +       r = amdgpu_vm_bo_map(adev, vm->kq_tma_va, va, 0,
> +                            AMDGPU_GPU_PAGE_SIZE,
> +                            AMDGPU_VM_PAGE_READABLE);
> +       if (r)
> +               goto err_del_va;
> +
> +       r = amdgpu_vm_bo_update(adev, vm->kq_tma_va, false);
> +       if (r)
> +               goto err_del_va;
> +
> +       return 0;
> +
> +err_del_va:
> +       amdgpu_vm_bo_del(adev, vm->kq_tma_va);
> +       vm->kq_tma_va = NULL;
> +err_free_bo:
> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
> +       return r;
> +}
> +
> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
> +                               struct amdgpu_vm *vm)
> +{
> +       uint64_t va;
> +
> +       if (!vm->kq_tma_bo)
> +               return;
> +
> +       dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> +
> +       if (vm->kq_tma_va) {
> +               va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
> +               amdgpu_vm_bo_unmap(adev, vm->kq_tma_va, va);
> +               amdgpu_vm_bo_del(adev, vm->kq_tma_va);
> +               vm->kq_tma_va = NULL;
> +       }
> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
> +}
> +
>  static int amdgpu_trap_map_region(struct amdgpu_device *adev,
>                                   struct amdgpu_vm *vm,
>                                   struct amdgpu_trap_obj *cwsr,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> index 7f83174a4742..9be5035abd1c 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> @@ -155,4 +155,11 @@ int amdgpu_trap_set_trap_debug_flag(struct amdgpu_device *adev,
>                                     struct amdgpu_trap_obj *cwsr_obj,
>                                     bool enabled);
>
> +/* Kernel queue trap handler — per-VM TMA and register programming */
> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev);
> +int  amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> +                                struct amdgpu_vm *vm);
> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
> +                               struct amdgpu_vm *vm);
> +
>  #endif
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index de3ef9ce2234..f5e228d4bfb5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -2640,6 +2640,12 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>         if (r)
>                 goto error_free_root;
>
> +       if (amdgpu_trap_is_enabled(adev)) {
> +               r = amdgpu_trap_vm_kq_tma_alloc(adev, vm);
> +               if (r)
> +                       goto error_free_root;
> +       }
> +
>         r = amdgpu_vm_create_task_info(vm);
>         if (r)
>                 dev_dbg(adev->dev, "Failed to create task info for VM\n");
> @@ -2774,6 +2780,9 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>                 amdgpu_vm_free_mapping(adev, vm, mapping, NULL);
>         }
>
> +       if (vm->kq_tma_bo)
> +               amdgpu_trap_vm_kq_tma_free(adev, vm);
> +
>         amdgpu_vm_pt_free_root(adev, vm);
>         amdgpu_bo_unreserve(root);
>         amdgpu_bo_unref(&root);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index dd825e179979..064f95a83790 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -25,6 +25,7 @@
>  #define __AMDGPU_VM_H__
>
>  #include <linux/idr.h>
> +#include <linux/iosys-map.h>
>  #include <linux/kfifo.h>
>  #include <linux/rbtree.h>
>  #include <drm/gpu_scheduler.h>
> @@ -488,6 +489,18 @@ struct amdgpu_vm {
>
>         /* cached fault info */
>         struct amdgpu_vm_fault_info fault_info;
> +
> +       /*
> +        * Per-VM kernel queue first-level TMA BO.
> +        * Allocated at VM init, freed at VM fini — same lifecycle as page tables.
> +        * Mapped read-only at AMDGPU_VA_RESERVED_TRAP_START in the GPU VM.
> +        * SQ_SHADER_TMA for all kernel VMIDs points to this fixed VA; per-VM
> +        * isolation is via page tables mapping different physical BOs there.
> +        * CPU kernel writes second-level handler address via kq_tma_map.
> +        */
> +       struct amdgpu_bo        *kq_tma_bo;
> +       struct amdgpu_bo_va     *kq_tma_va;
> +       struct iosys_map         kq_tma_map;
>  };
>
>  struct amdgpu_vm_manager {
> --
> 2.34.1
>

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Alex Deucher]

On Wed, Sep 9, 2026 at 3:23 PM Alex Deucher <alexdeucher@gmail.com> wrote:
>
> On Sat, Sep 5, 2026 at 4:55 AM Srinivasan Shanmugam
> <srinivasan.shanmugam@amd.com> wrote:
> >
> > MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
>
> MES doesn't own the kernel queue VMIDs.  The kernel driver does.  MES
> assigns VMIDs dynamically for user queues, but for kernel queues, the
> kernel assigns the vmids.
>
> > SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
> > via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
> > On GFX10 and earlier HWS-based hardware, the driver programs trap
> > registers via SRBM select for KFD queues but no equivalent exists for
> > driver-managed kernel queue VMIDs.
> >
> > Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
> > can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
> > device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
> > address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
> > there, so per-VM isolation is handled entirely by page tables without
> > needing to reprogram the register per job or per submission.
> >
> > The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
> > (parallel to page table allocation) and mapped read-only into the GPU VM
> > at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
> > handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
> > The first-level CWSR handler reads this address to chain to the
> > second-level handler when a shader exception fires.
> >
> > This design is:
> >   - Per-VM BO (not device-level) — same model as page tables
> >   - Fixed VA in each VM's address space — same VA, different physical BO
> >   - Read-only from GPU — kernel CPU updates it via CPU mapping
> >   - Treat allocation/free lifecycle identical to page tables
> >
> > Suggested-by: Christian König <christian.koenig@amd.com>
> > Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
> > Cc: Lijo Lazar <lijo.lazar@amd.com>
> > Cc: Timur Kristóf <timur.kristof@gmail.com>
> > Cc: Samuel Pitoiset <hakzsam@gmail.com>
> > Cc: Natalie Vock <natalie.vock@gmx.de>
> > Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
> > Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
> > ---
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
> >  5 files changed, 110 insertions(+)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> > index 3ca187f5ade8..5624a5ab5c62 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> > @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
> >         void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
> >                                                  uint32_t status);
> >         uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
> > +       void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
> >  };
> >
> >  struct amdgpu_vmhub {
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> > index 623cac6781be..e913488ca3fa 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> > @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
> >
> >         amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
> >         adev->trap_info = no_free_ptr(trap_info);
> > +       amdgpu_trap_program_kernel_vmids(adev);
> >

Why do we call this here?

Alex

> >         return 0;
> >  }
> > @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
> >         adev->trap_info = NULL;
> >  }
> >
> > +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
> > +{
> > +       struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
> > +
> > +       if (!amdgpu_trap_is_enabled(adev))
> > +               return;
> > +       if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
> > +               return;
> > +
> > +       hub->vmhub_funcs->program_kernel_trap_vmids(adev);
> > +}
> > +
> > +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> > +                               struct amdgpu_vm *vm)
> > +{
> > +       void *cpu_addr;
> > +       uint64_t va;
> > +       int r;
> > +
> > +       dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> > +
> > +       r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
> > +                                   AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
> > +                                   NULL, &cpu_addr);
> > +       if (r)
> > +               return r;
> > +
> > +       if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
> > +               iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
> > +                                         (void __iomem *)cpu_addr);
> > +       else
> > +               iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
>
> Are you planning to copy the first level trap handler to the buffer
> later?  Why not just initialize it here?
>
> Alex
>
> > +
> > +       vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
> > +       if (!vm->kq_tma_va) {
> > +               r = -ENOMEM;
> > +               goto err_free_bo;
> > +       }
> > +
> > +       va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
> > +       r = amdgpu_vm_bo_map(adev, vm->kq_tma_va, va, 0,
> > +                            AMDGPU_GPU_PAGE_SIZE,
> > +                            AMDGPU_VM_PAGE_READABLE);
> > +       if (r)
> > +               goto err_del_va;
> > +
> > +       r = amdgpu_vm_bo_update(adev, vm->kq_tma_va, false);
> > +       if (r)
> > +               goto err_del_va;
> > +
> > +       return 0;
> > +
> > +err_del_va:
> > +       amdgpu_vm_bo_del(adev, vm->kq_tma_va);
> > +       vm->kq_tma_va = NULL;
> > +err_free_bo:
> > +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
> > +       return r;
> > +}
> > +
> > +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
> > +                               struct amdgpu_vm *vm)
> > +{
> > +       uint64_t va;
> > +
> > +       if (!vm->kq_tma_bo)
> > +               return;
> > +
> > +       dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> > +
> > +       if (vm->kq_tma_va) {
> > +               va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
> > +               amdgpu_vm_bo_unmap(adev, vm->kq_tma_va, va);
> > +               amdgpu_vm_bo_del(adev, vm->kq_tma_va);
> > +               vm->kq_tma_va = NULL;
> > +       }
> > +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
> > +}
> > +
> >  static int amdgpu_trap_map_region(struct amdgpu_device *adev,
> >                                   struct amdgpu_vm *vm,
> >                                   struct amdgpu_trap_obj *cwsr,
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> > index 7f83174a4742..9be5035abd1c 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> > @@ -155,4 +155,11 @@ int amdgpu_trap_set_trap_debug_flag(struct amdgpu_device *adev,
> >                                     struct amdgpu_trap_obj *cwsr_obj,
> >                                     bool enabled);
> >
> > +/* Kernel queue trap handler — per-VM TMA and register programming */
> > +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev);
> > +int  amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> > +                                struct amdgpu_vm *vm);
> > +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
> > +                               struct amdgpu_vm *vm);
> > +
> >  #endif
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > index de3ef9ce2234..f5e228d4bfb5 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > @@ -2640,6 +2640,12 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
> >         if (r)
> >                 goto error_free_root;
> >
> > +       if (amdgpu_trap_is_enabled(adev)) {
> > +               r = amdgpu_trap_vm_kq_tma_alloc(adev, vm);
> > +               if (r)
> > +                       goto error_free_root;
> > +       }
> > +
> >         r = amdgpu_vm_create_task_info(vm);
> >         if (r)
> >                 dev_dbg(adev->dev, "Failed to create task info for VM\n");
> > @@ -2774,6 +2780,9 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
> >                 amdgpu_vm_free_mapping(adev, vm, mapping, NULL);
> >         }
> >
> > +       if (vm->kq_tma_bo)
> > +               amdgpu_trap_vm_kq_tma_free(adev, vm);
> > +
> >         amdgpu_vm_pt_free_root(adev, vm);
> >         amdgpu_bo_unreserve(root);
> >         amdgpu_bo_unref(&root);
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > index dd825e179979..064f95a83790 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > @@ -25,6 +25,7 @@
> >  #define __AMDGPU_VM_H__
> >
> >  #include <linux/idr.h>
> > +#include <linux/iosys-map.h>
> >  #include <linux/kfifo.h>
> >  #include <linux/rbtree.h>
> >  #include <drm/gpu_scheduler.h>
> > @@ -488,6 +489,18 @@ struct amdgpu_vm {
> >
> >         /* cached fault info */
> >         struct amdgpu_vm_fault_info fault_info;
> > +
> > +       /*
> > +        * Per-VM kernel queue first-level TMA BO.
> > +        * Allocated at VM init, freed at VM fini — same lifecycle as page tables.
> > +        * Mapped read-only at AMDGPU_VA_RESERVED_TRAP_START in the GPU VM.
> > +        * SQ_SHADER_TMA for all kernel VMIDs points to this fixed VA; per-VM
> > +        * isolation is via page tables mapping different physical BOs there.
> > +        * CPU kernel writes second-level handler address via kq_tma_map.
> > +        */
> > +       struct amdgpu_bo        *kq_tma_bo;
> > +       struct amdgpu_bo_va     *kq_tma_va;
> > +       struct iosys_map         kq_tma_map;
> >  };
> >
> >  struct amdgpu_vm_manager {
> > --
> > 2.34.1
> >

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Alex Deucher]

On Sat, Sep 5, 2026 at 4:55 AM Srinivasan Shanmugam
<srinivasan.shanmugam@amd.com> wrote:
>
> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
> On GFX10 and earlier HWS-based hardware, the driver programs trap
> registers via SRBM select for KFD queues but no equivalent exists for
> driver-managed kernel queue VMIDs.
>
> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
> there, so per-VM isolation is handled entirely by page tables without
> needing to reprogram the register per job or per submission.
>
> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
> (parallel to page table allocation) and mapped read-only into the GPU VM
> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
> The first-level CWSR handler reads this address to chain to the
> second-level handler when a shader exception fires.
>
> This design is:
>   - Per-VM BO (not device-level) — same model as page tables
>   - Fixed VA in each VM's address space — same VA, different physical BO
>   - Read-only from GPU — kernel CPU updates it via CPU mapping
>   - Treat allocation/free lifecycle identical to page tables
>
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
> Cc: Lijo Lazar <lijo.lazar@amd.com>
> Cc: Timur Kristóf <timur.kristof@gmail.com>
> Cc: Samuel Pitoiset <hakzsam@gmail.com>
> Cc: Natalie Vock <natalie.vock@gmx.de>
> Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>  5 files changed, 110 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> index 3ca187f5ade8..5624a5ab5c62 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>         void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
>                                                  uint32_t status);
>         uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
> +       void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
>  };
>
>  struct amdgpu_vmhub {
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> index 623cac6781be..e913488ca3fa 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>
>         amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>         adev->trap_info = no_free_ptr(trap_info);
> +       amdgpu_trap_program_kernel_vmids(adev);
>
>         return 0;
>  }
> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
>         adev->trap_info = NULL;
>  }
>
> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
> +{
> +       struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
> +
> +       if (!amdgpu_trap_is_enabled(adev))
> +               return;
> +       if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
> +               return;
> +
> +       hub->vmhub_funcs->program_kernel_trap_vmids(adev);
> +}
> +
> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> +                               struct amdgpu_vm *vm)
> +{
> +       void *cpu_addr;
> +       uint64_t va;
> +       int r;
> +
> +       dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> +
> +       r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
> +                                   AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
> +                                   NULL, &cpu_addr);
> +       if (r)
> +               return r;
> +
> +       if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
> +               iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
> +                                         (void __iomem *)cpu_addr);
> +       else
> +               iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
> +
> +       vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
> +       if (!vm->kq_tma_va) {
> +               r = -ENOMEM;
> +               goto err_free_bo;
> +       }
> +
> +       va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
> +       r = amdgpu_vm_bo_map(adev, vm->kq_tma_va, va, 0,
> +                            AMDGPU_GPU_PAGE_SIZE,
> +                            AMDGPU_VM_PAGE_READABLE);
> +       if (r)
> +               goto err_del_va;
> +
> +       r = amdgpu_vm_bo_update(adev, vm->kq_tma_va, false);
> +       if (r)
> +               goto err_del_va;
> +
> +       return 0;
> +
> +err_del_va:
> +       amdgpu_vm_bo_del(adev, vm->kq_tma_va);
> +       vm->kq_tma_va = NULL;
> +err_free_bo:
> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
> +       return r;
> +}
> +
> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
> +                               struct amdgpu_vm *vm)
> +{
> +       uint64_t va;
> +
> +       if (!vm->kq_tma_bo)
> +               return;
> +
> +       dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> +
> +       if (vm->kq_tma_va) {
> +               va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
> +               amdgpu_vm_bo_unmap(adev, vm->kq_tma_va, va);
> +               amdgpu_vm_bo_del(adev, vm->kq_tma_va);
> +               vm->kq_tma_va = NULL;
> +       }
> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
> +}
> +
>  static int amdgpu_trap_map_region(struct amdgpu_device *adev,
>                                   struct amdgpu_vm *vm,
>                                   struct amdgpu_trap_obj *cwsr,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> index 7f83174a4742..9be5035abd1c 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> @@ -155,4 +155,11 @@ int amdgpu_trap_set_trap_debug_flag(struct amdgpu_device *adev,
>                                     struct amdgpu_trap_obj *cwsr_obj,
>                                     bool enabled);
>
> +/* Kernel queue trap handler — per-VM TMA and register programming */
> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev);
> +int  amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> +                                struct amdgpu_vm *vm);
> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
> +                               struct amdgpu_vm *vm);
> +
>  #endif
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index de3ef9ce2234..f5e228d4bfb5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -2640,6 +2640,12 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>         if (r)
>                 goto error_free_root;
>
> +       if (amdgpu_trap_is_enabled(adev)) {
> +               r = amdgpu_trap_vm_kq_tma_alloc(adev, vm);
> +               if (r)
> +                       goto error_free_root;
> +       }
> +
>         r = amdgpu_vm_create_task_info(vm);
>         if (r)
>                 dev_dbg(adev->dev, "Failed to create task info for VM\n");
> @@ -2774,6 +2780,9 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>                 amdgpu_vm_free_mapping(adev, vm, mapping, NULL);
>         }
>
> +       if (vm->kq_tma_bo)
> +               amdgpu_trap_vm_kq_tma_free(adev, vm);
> +
>         amdgpu_vm_pt_free_root(adev, vm);
>         amdgpu_bo_unreserve(root);
>         amdgpu_bo_unref(&root);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index dd825e179979..064f95a83790 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -25,6 +25,7 @@
>  #define __AMDGPU_VM_H__
>
>  #include <linux/idr.h>
> +#include <linux/iosys-map.h>
>  #include <linux/kfifo.h>
>  #include <linux/rbtree.h>
>  #include <drm/gpu_scheduler.h>
> @@ -488,6 +489,18 @@ struct amdgpu_vm {
>
>         /* cached fault info */
>         struct amdgpu_vm_fault_info fault_info;
> +
> +       /*
> +        * Per-VM kernel queue first-level TMA BO.
> +        * Allocated at VM init, freed at VM fini — same lifecycle as page tables.
> +        * Mapped read-only at AMDGPU_VA_RESERVED_TRAP_START in the GPU VM.
> +        * SQ_SHADER_TMA for all kernel VMIDs points to this fixed VA; per-VM
> +        * isolation is via page tables mapping different physical BOs there.
> +        * CPU kernel writes second-level handler address via kq_tma_map.
> +        */
> +       struct amdgpu_bo        *kq_tma_bo;
> +       struct amdgpu_bo_va     *kq_tma_va;
> +       struct iosys_map         kq_tma_map;

These should be the same for both user and kernel queues.  The only
difference is who manages the vmids (driver vs MES).  They are per
vmid so it doesn't matter whether it's a kernel queue or user queue.

Alex

>  };
>
>  struct amdgpu_vm_manager {
> --
> 2.34.1
>

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Alex Deucher]

On Wed, Sep 9, 2026 at 4:42 PM Alex Deucher <alexdeucher@gmail.com> wrote:
>
> On Sat, Sep 5, 2026 at 4:55 AM Srinivasan Shanmugam
> <srinivasan.shanmugam@amd.com> wrote:
> >
> > MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
> > SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
> > via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
> > On GFX10 and earlier HWS-based hardware, the driver programs trap
> > registers via SRBM select for KFD queues but no equivalent exists for
> > driver-managed kernel queue VMIDs.
> >
> > Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
> > can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
> > device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
> > address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
> > there, so per-VM isolation is handled entirely by page tables without
> > needing to reprogram the register per job or per submission.
> >
> > The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
> > (parallel to page table allocation) and mapped read-only into the GPU VM
> > at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
> > handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
> > The first-level CWSR handler reads this address to chain to the
> > second-level handler when a shader exception fires.
> >
> > This design is:
> >   - Per-VM BO (not device-level) — same model as page tables
> >   - Fixed VA in each VM's address space — same VA, different physical BO
> >   - Read-only from GPU — kernel CPU updates it via CPU mapping
> >   - Treat allocation/free lifecycle identical to page tables
> >
> > Suggested-by: Christian König <christian.koenig@amd.com>
> > Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
> > Cc: Lijo Lazar <lijo.lazar@amd.com>
> > Cc: Timur Kristóf <timur.kristof@gmail.com>
> > Cc: Samuel Pitoiset <hakzsam@gmail.com>
> > Cc: Natalie Vock <natalie.vock@gmx.de>
> > Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
> > Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
> > ---
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
> >  5 files changed, 110 insertions(+)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> > index 3ca187f5ade8..5624a5ab5c62 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> > @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
> >         void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
> >                                                  uint32_t status);
> >         uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
> > +       void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
> >  };
> >
> >  struct amdgpu_vmhub {
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> > index 623cac6781be..e913488ca3fa 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> > @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
> >
> >         amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
> >         adev->trap_info = no_free_ptr(trap_info);
> > +       amdgpu_trap_program_kernel_vmids(adev);
> >
> >         return 0;
> >  }
> > @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
> >         adev->trap_info = NULL;
> >  }
> >
> > +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
> > +{
> > +       struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
> > +
> > +       if (!amdgpu_trap_is_enabled(adev))
> > +               return;
> > +       if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
> > +               return;
> > +
> > +       hub->vmhub_funcs->program_kernel_trap_vmids(adev);
> > +}
> > +
> > +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> > +                               struct amdgpu_vm *vm)
> > +{
> > +       void *cpu_addr;
> > +       uint64_t va;
> > +       int r;
> > +
> > +       dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> > +
> > +       r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
> > +                                   AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
> > +                                   NULL, &cpu_addr);
> > +       if (r)
> > +               return r;
> > +
> > +       if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
> > +               iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
> > +                                         (void __iomem *)cpu_addr);
> > +       else
> > +               iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
> > +
> > +       vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
> > +       if (!vm->kq_tma_va) {
> > +               r = -ENOMEM;
> > +               goto err_free_bo;
> > +       }
> > +
> > +       va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
> > +       r = amdgpu_vm_bo_map(adev, vm->kq_tma_va, va, 0,
> > +                            AMDGPU_GPU_PAGE_SIZE,
> > +                            AMDGPU_VM_PAGE_READABLE);
> > +       if (r)
> > +               goto err_del_va;
> > +
> > +       r = amdgpu_vm_bo_update(adev, vm->kq_tma_va, false);
> > +       if (r)
> > +               goto err_del_va;
> > +
> > +       return 0;
> > +
> > +err_del_va:
> > +       amdgpu_vm_bo_del(adev, vm->kq_tma_va);
> > +       vm->kq_tma_va = NULL;
> > +err_free_bo:
> > +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
> > +       return r;
> > +}
> > +
> > +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
> > +                               struct amdgpu_vm *vm)
> > +{
> > +       uint64_t va;
> > +
> > +       if (!vm->kq_tma_bo)
> > +               return;
> > +
> > +       dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> > +
> > +       if (vm->kq_tma_va) {
> > +               va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
> > +               amdgpu_vm_bo_unmap(adev, vm->kq_tma_va, va);
> > +               amdgpu_vm_bo_del(adev, vm->kq_tma_va);
> > +               vm->kq_tma_va = NULL;
> > +       }
> > +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
> > +}
> > +
> >  static int amdgpu_trap_map_region(struct amdgpu_device *adev,
> >                                   struct amdgpu_vm *vm,
> >                                   struct amdgpu_trap_obj *cwsr,
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> > index 7f83174a4742..9be5035abd1c 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> > @@ -155,4 +155,11 @@ int amdgpu_trap_set_trap_debug_flag(struct amdgpu_device *adev,
> >                                     struct amdgpu_trap_obj *cwsr_obj,
> >                                     bool enabled);
> >
> > +/* Kernel queue trap handler — per-VM TMA and register programming */
> > +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev);
> > +int  amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> > +                                struct amdgpu_vm *vm);
> > +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
> > +                               struct amdgpu_vm *vm);
> > +
> >  #endif
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > index de3ef9ce2234..f5e228d4bfb5 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > @@ -2640,6 +2640,12 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
> >         if (r)
> >                 goto error_free_root;
> >
> > +       if (amdgpu_trap_is_enabled(adev)) {
> > +               r = amdgpu_trap_vm_kq_tma_alloc(adev, vm);
> > +               if (r)
> > +                       goto error_free_root;
> > +       }
> > +
> >         r = amdgpu_vm_create_task_info(vm);
> >         if (r)
> >                 dev_dbg(adev->dev, "Failed to create task info for VM\n");
> > @@ -2774,6 +2780,9 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
> >                 amdgpu_vm_free_mapping(adev, vm, mapping, NULL);
> >         }
> >
> > +       if (vm->kq_tma_bo)
> > +               amdgpu_trap_vm_kq_tma_free(adev, vm);
> > +
> >         amdgpu_vm_pt_free_root(adev, vm);
> >         amdgpu_bo_unreserve(root);
> >         amdgpu_bo_unref(&root);
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > index dd825e179979..064f95a83790 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > @@ -25,6 +25,7 @@
> >  #define __AMDGPU_VM_H__
> >
> >  #include <linux/idr.h>
> > +#include <linux/iosys-map.h>
> >  #include <linux/kfifo.h>
> >  #include <linux/rbtree.h>
> >  #include <drm/gpu_scheduler.h>
> > @@ -488,6 +489,18 @@ struct amdgpu_vm {
> >
> >         /* cached fault info */
> >         struct amdgpu_vm_fault_info fault_info;
> > +
> > +       /*
> > +        * Per-VM kernel queue first-level TMA BO.
> > +        * Allocated at VM init, freed at VM fini — same lifecycle as page tables.
> > +        * Mapped read-only at AMDGPU_VA_RESERVED_TRAP_START in the GPU VM.
> > +        * SQ_SHADER_TMA for all kernel VMIDs points to this fixed VA; per-VM
> > +        * isolation is via page tables mapping different physical BOs there.
> > +        * CPU kernel writes second-level handler address via kq_tma_map.
> > +        */
> > +       struct amdgpu_bo        *kq_tma_bo;
> > +       struct amdgpu_bo_va     *kq_tma_va;
> > +       struct iosys_map         kq_tma_map;
>
> These should be the same for both user and kernel queues.  The only
> difference is who manages the vmids (driver vs MES).  They are per
> vmid so it doesn't matter whether it's a kernel queue or user queue.
>

I would merge these patch sets.  The trap handling is the same for
both kernel queues and user queues.  The only difference for kernel
queues is that the driver has to set the TBA/TMA registers while
MES/KIQ handles it for user queues.  At vm_init time, allocate the
memory for the trap handler, copy the trap handler to the memory and
add the mapping to the GPUVM address space.  Then in gfxhub init,
program the TBA/TMA registers for the kernel managed vmids.  Finally,
add the IOCTL to set/clear the second level trap handler and validate
the user supplied GPU VA.

Alex

> Alex
>
> >  };
> >
> >  struct amdgpu_vm_manager {
> > --
> > 2.34.1
> >

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[SRINIVASAN SHANMUGAM]

[-- Attachment #1: Type: text/plain, Size: 12871 bytes --]


On 9/10/2026 2:20 AM, Alex Deucher wrote:
> On Wed, Sep 9, 2026 at 4:42 PM Alex Deucher<alexdeucher@gmail.com> wrote:
>> On Sat, Sep 5, 2026 at 4:55 AM Srinivasan Shanmugam
>> <srinivasan.shanmugam@amd.com> wrote:
>>> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
>>> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
>>> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
>>> On GFX10 and earlier HWS-based hardware, the driver programs trap
>>> registers via SRBM select for KFD queues but no equivalent exists for
>>> driver-managed kernel queue VMIDs.
>>>
>>> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
>>> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
>>> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
>>> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
>>> there, so per-VM isolation is handled entirely by page tables without
>>> needing to reprogram the register per job or per submission.
>>>
>>> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
>>> (parallel to page table allocation) and mapped read-only into the GPU VM
>>> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
>>> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
>>> The first-level CWSR handler reads this address to chain to the
>>> second-level handler when a shader exception fires.
>>>
>>> This design is:
>>>    - Per-VM BO (not device-level) — same model as page tables
>>>    - Fixed VA in each VM's address space — same VA, different physical BO
>>>    - Read-only from GPU — kernel CPU updates it via CPU mapping
>>>    - Treat allocation/free lifecycle identical to page tables
>>>
>>> Suggested-by: Christian König<christian.koenig@amd.com>
>>> Suggested-by: Alexander Deucher<alexander.deucher@amd.com>
>>> Cc: Lijo Lazar<lijo.lazar@amd.com>
>>> Cc: Timur Kristóf<timur.kristof@gmail.com>
>>> Cc: Samuel Pitoiset<hakzsam@gmail.com>
>>> Cc: Natalie Vock<natalie.vock@gmx.de>
>>> Signed-off-by: Srinivasan Shanmugam<srinivasan.shanmugam@amd.com>
>>> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
>>> ---
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>>>   5 files changed, 110 insertions(+)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>> index 3ca187f5ade8..5624a5ab5c62 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>>>          void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
>>>                                                   uint32_t status);
>>>          uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
>>> +       void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
>>>   };
>>>
>>>   struct amdgpu_vmhub {
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>> index 623cac6781be..e913488ca3fa 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>>>
>>>          amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>>>          adev->trap_info = no_free_ptr(trap_info);
>>> +       amdgpu_trap_program_kernel_vmids(adev);
>>>
>>>          return 0;
>>>   }
>>> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
>>>          adev->trap_info = NULL;
>>>   }
>>>
>>> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
>>> +{
>>> +       struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
>>> +
>>> +       if (!amdgpu_trap_is_enabled(adev))
>>> +               return;
>>> +       if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
>>> +               return;
>>> +
>>> +       hub->vmhub_funcs->program_kernel_trap_vmids(adev);
>>> +}
>>> +
>>> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>>> +                               struct amdgpu_vm *vm)
>>> +{
>>> +       void *cpu_addr;
>>> +       uint64_t va;
>>> +       int r;
>>> +
>>> +       dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>>> +
>>> +       r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
>>> +                                   AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
>>> +                                   NULL, &cpu_addr);
>>> +       if (r)
>>> +               return r;
>>> +
>>> +       if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
>>> +               iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
>>> +                                         (void __iomem *)cpu_addr);
>>> +       else
>>> +               iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
>>> +
>>> +       vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
>>> +       if (!vm->kq_tma_va) {
>>> +               r = -ENOMEM;
>>> +               goto err_free_bo;
>>> +       }
>>> +
>>> +       va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
>>> +       r = amdgpu_vm_bo_map(adev, vm->kq_tma_va, va, 0,
>>> +                            AMDGPU_GPU_PAGE_SIZE,
>>> +                            AMDGPU_VM_PAGE_READABLE);
>>> +       if (r)
>>> +               goto err_del_va;
>>> +
>>> +       r = amdgpu_vm_bo_update(adev, vm->kq_tma_va, false);
>>> +       if (r)
>>> +               goto err_del_va;
>>> +
>>> +       return 0;
>>> +
>>> +err_del_va:
>>> +       amdgpu_vm_bo_del(adev, vm->kq_tma_va);
>>> +       vm->kq_tma_va = NULL;
>>> +err_free_bo:
>>> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
>>> +       return r;
>>> +}
>>> +
>>> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
>>> +                               struct amdgpu_vm *vm)
>>> +{
>>> +       uint64_t va;
>>> +
>>> +       if (!vm->kq_tma_bo)
>>> +               return;
>>> +
>>> +       dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>>> +
>>> +       if (vm->kq_tma_va) {
>>> +               va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
>>> +               amdgpu_vm_bo_unmap(adev, vm->kq_tma_va, va);
>>> +               amdgpu_vm_bo_del(adev, vm->kq_tma_va);
>>> +               vm->kq_tma_va = NULL;
>>> +       }
>>> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
>>> +}
>>> +
>>>   static int amdgpu_trap_map_region(struct amdgpu_device *adev,
>>>                                    struct amdgpu_vm *vm,
>>>                                    struct amdgpu_trap_obj *cwsr,
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
>>> index 7f83174a4742..9be5035abd1c 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
>>> @@ -155,4 +155,11 @@ int amdgpu_trap_set_trap_debug_flag(struct amdgpu_device *adev,
>>>                                      struct amdgpu_trap_obj *cwsr_obj,
>>>                                      bool enabled);
>>>
>>> +/* Kernel queue trap handler — per-VM TMA and register programming */
>>> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev);
>>> +int  amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>>> +                                struct amdgpu_vm *vm);
>>> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
>>> +                               struct amdgpu_vm *vm);
>>> +
>>>   #endif
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> index de3ef9ce2234..f5e228d4bfb5 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> @@ -2640,6 +2640,12 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>>          if (r)
>>>                  goto error_free_root;
>>>
>>> +       if (amdgpu_trap_is_enabled(adev)) {
>>> +               r = amdgpu_trap_vm_kq_tma_alloc(adev, vm);
>>> +               if (r)
>>> +                       goto error_free_root;
>>> +       }
>>> +
>>>          r = amdgpu_vm_create_task_info(vm);
>>>          if (r)
>>>                  dev_dbg(adev->dev, "Failed to create task info for VM\n");
>>> @@ -2774,6 +2780,9 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>>>                  amdgpu_vm_free_mapping(adev, vm, mapping, NULL);
>>>          }
>>>
>>> +       if (vm->kq_tma_bo)
>>> +               amdgpu_trap_vm_kq_tma_free(adev, vm);
>>> +
>>>          amdgpu_vm_pt_free_root(adev, vm);
>>>          amdgpu_bo_unreserve(root);
>>>          amdgpu_bo_unref(&root);
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> index dd825e179979..064f95a83790 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> @@ -25,6 +25,7 @@
>>>   #define __AMDGPU_VM_H__
>>>
>>>   #include <linux/idr.h>
>>> +#include <linux/iosys-map.h>
>>>   #include <linux/kfifo.h>
>>>   #include <linux/rbtree.h>
>>>   #include <drm/gpu_scheduler.h>
>>> @@ -488,6 +489,18 @@ struct amdgpu_vm {
>>>
>>>          /* cached fault info */
>>>          struct amdgpu_vm_fault_info fault_info;
>>> +
>>> +       /*
>>> +        * Per-VM kernel queue first-level TMA BO.
>>> +        * Allocated at VM init, freed at VM fini — same lifecycle as page tables.
>>> +        * Mapped read-only at AMDGPU_VA_RESERVED_TRAP_START in the GPU VM.
>>> +        * SQ_SHADER_TMA for all kernel VMIDs points to this fixed VA; per-VM
>>> +        * isolation is via page tables mapping different physical BOs there.
>>> +        * CPU kernel writes second-level handler address via kq_tma_map.
>>> +        */
>>> +       struct amdgpu_bo        *kq_tma_bo;
>>> +       struct amdgpu_bo_va     *kq_tma_va;
>>> +       struct iosys_map         kq_tma_map;
>> These should be the same for both user and kernel queues.  The only
>> difference is who manages the vmids (driver vs MES).  They are per
>> vmid so it doesn't matter whether it's a kernel queue or user queue.
>>
> I would merge these patch sets.  The trap handling is the same for
> both kernel queues and user queues.  The only difference for kernel
> queues is that the driver has to set the TBA/TMA registers while
> MES/KIQ handles it for user queues.  At vm_init time, allocate the
> memory for the trap handler, copy the trap handler to the memory and
> add the mapping to the GPUVM address space.  Then in gfxhub init,
> program the TBA/TMA registers for the kernel managed vmids.  Finally,
> add the IOCTL to set/clear the second level trap handler and validate
> the user supplied GPU VA.

Hi Alex,

Thank you for the review. I have few questions before proceeding with 
the implementation.

I looked at how KFD handles this today in |kfd_process.c|:

/* KFD writes second-level TBA/TMA into first-level TMA */
iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET,
              uint64_t, tba_addr);
iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET + sizeof(uint64_t),
              uint64_t, tma_addr);

Where |KFD_CWSR_TMA_OFFSET = AMDGPU_GPU_PAGE_SIZE + 2048 = 0x1800|. KFD 
uses physical GPU addresses for TBA/TMA registers, not VM virtual addresses.

Question 1 — Which fixed GPU virtual address for the merged buffer?

KFD uses physical addresses for the TBA/TMA registers. Our design maps 
the buffer into each VM at a fixed virtual address for page table 
isolation. Should the merged per-VM buffer use 
|AMDGPU_VA_RESERVED_TRAP_UQ_START| as the fixed VA? Or a different address?

Question 2 — TMA offset inside the merged buffer — 0x1800 or 0x2000?

KFD places the TMA section at offset |0x1800| from the TBA start 
(|KFD_CWSR_TMA_OFFSET|). The current UQ design uses offset |0x2000| 
(|AMDGPU_TRAP_TBA_MAX_SIZE|). After merging into one per-VM buffer, 
should TMA start at |0x1800| to match KFD's layout?

Question 3 — When should TRAP_EN be set?

When programming |SQ_SHADER_TBA_HI| for kernel VMIDs, should |TRAP_EN| 
be set only after the per-VM buffer is fully mapped? Or is it safe to 
set it at boot time since TMA slots start zeroed (meaning no 
second-level handler installed yet)?

Question 4 — Drain kernel queues before writing second-level handler?

When |SET_L2_TRAP| is called, user queues are evicted and TLB is flushed 
before writing. After the merge, kernel queue VMIDs also read from the 
same TMA. Should kernel queue work be drained as well before writing the 
second-level handler addresses?

Thanks,
Srini

[-- Attachment #2: Type: text/html, Size: 14519 bytes --]

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Lazar, Lijo]

On 10-Sep-26 11:43 AM, SRINIVASAN SHANMUGAM wrote:
> 
> On 9/10/2026 2:20 AM, Alex Deucher wrote:
>> On Wed, Sep 9, 2026 at 4:42 PM Alex Deucher<alexdeucher@gmail.com> wrote:
>>> On Sat, Sep 5, 2026 at 4:55 AM Srinivasan Shanmugam
>>> <srinivasan.shanmugam@amd.com> wrote:
>>>> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
>>>> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
>>>> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
>>>> On GFX10 and earlier HWS-based hardware, the driver programs trap
>>>> registers via SRBM select for KFD queues but no equivalent exists for
>>>> driver-managed kernel queue VMIDs.
>>>>
>>>> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
>>>> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
>>>> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
>>>> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
>>>> there, so per-VM isolation is handled entirely by page tables without
>>>> needing to reprogram the register per job or per submission.
>>>>
>>>> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
>>>> (parallel to page table allocation) and mapped read-only into the GPU VM
>>>> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
>>>> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
>>>> The first-level CWSR handler reads this address to chain to the
>>>> second-level handler when a shader exception fires.
>>>>
>>>> This design is:
>>>>    - Per-VM BO (not device-level) — same model as page tables
>>>>    - Fixed VA in each VM's address space — same VA, different physical BO
>>>>    - Read-only from GPU — kernel CPU updates it via CPU mapping
>>>>    - Treat allocation/free lifecycle identical to page tables
>>>>
>>>> Suggested-by: Christian König<christian.koenig@amd.com>
>>>> Suggested-by: Alexander Deucher<alexander.deucher@amd.com>
>>>> Cc: Lijo Lazar<lijo.lazar@amd.com>
>>>> Cc: Timur Kristóf<timur.kristof@gmail.com>
>>>> Cc: Samuel Pitoiset<hakzsam@gmail.com>
>>>> Cc: Natalie Vock<natalie.vock@gmx.de>
>>>> Signed-off-by: Srinivasan Shanmugam<srinivasan.shanmugam@amd.com>
>>>> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
>>>> ---
>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>>>>   5 files changed, 110 insertions(+)
>>>>
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>> index 3ca187f5ade8..5624a5ab5c62 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>>>>          void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
>>>>                                                   uint32_t status);
>>>>          uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
>>>> +       void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
>>>>   };
>>>>
>>>>   struct amdgpu_vmhub {
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>> index 623cac6781be..e913488ca3fa 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>>>>
>>>>          amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>>>>          adev->trap_info = no_free_ptr(trap_info);
>>>> +       amdgpu_trap_program_kernel_vmids(adev);
>>>>
>>>>          return 0;
>>>>   }
>>>> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
>>>>          adev->trap_info = NULL;
>>>>   }
>>>>
>>>> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
>>>> +{
>>>> +       struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
>>>> +
>>>> +       if (!amdgpu_trap_is_enabled(adev))
>>>> +               return;
>>>> +       if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
>>>> +               return;
>>>> +
>>>> +       hub->vmhub_funcs->program_kernel_trap_vmids(adev);
>>>> +}
>>>> +
>>>> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>>>> +                               struct amdgpu_vm *vm)
>>>> +{
>>>> +       void *cpu_addr;
>>>> +       uint64_t va;
>>>> +       int r;
>>>> +
>>>> +       dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>>>> +
>>>> +       r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
>>>> +                                   AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
>>>> +                                   NULL, &cpu_addr);
>>>> +       if (r)
>>>> +               return r;
>>>> +
>>>> +       if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
>>>> +               iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
>>>> +                                         (void __iomem *)cpu_addr);
>>>> +       else
>>>> +               iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
>>>> +
>>>> +       vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
>>>> +       if (!vm->kq_tma_va) {
>>>> +               r = -ENOMEM;
>>>> +               goto err_free_bo;
>>>> +       }
>>>> +
>>>> +       va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
>>>> +       r = amdgpu_vm_bo_map(adev, vm->kq_tma_va, va, 0,
>>>> +                            AMDGPU_GPU_PAGE_SIZE,
>>>> +                            AMDGPU_VM_PAGE_READABLE);
>>>> +       if (r)
>>>> +               goto err_del_va;
>>>> +
>>>> +       r = amdgpu_vm_bo_update(adev, vm->kq_tma_va, false);
>>>> +       if (r)
>>>> +               goto err_del_va;
>>>> +
>>>> +       return 0;
>>>> +
>>>> +err_del_va:
>>>> +       amdgpu_vm_bo_del(adev, vm->kq_tma_va);
>>>> +       vm->kq_tma_va = NULL;
>>>> +err_free_bo:
>>>> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
>>>> +       return r;
>>>> +}
>>>> +
>>>> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
>>>> +                               struct amdgpu_vm *vm)
>>>> +{
>>>> +       uint64_t va;
>>>> +
>>>> +       if (!vm->kq_tma_bo)
>>>> +               return;
>>>> +
>>>> +       dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>>>> +
>>>> +       if (vm->kq_tma_va) {
>>>> +               va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
>>>> +               amdgpu_vm_bo_unmap(adev, vm->kq_tma_va, va);
>>>> +               amdgpu_vm_bo_del(adev, vm->kq_tma_va);
>>>> +               vm->kq_tma_va = NULL;
>>>> +       }
>>>> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
>>>> +}
>>>> +
>>>>   static int amdgpu_trap_map_region(struct amdgpu_device *adev,
>>>>                                    struct amdgpu_vm *vm,
>>>>                                    struct amdgpu_trap_obj *cwsr,
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
>>>> index 7f83174a4742..9be5035abd1c 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
>>>> @@ -155,4 +155,11 @@ int amdgpu_trap_set_trap_debug_flag(struct amdgpu_device *adev,
>>>>                                      struct amdgpu_trap_obj *cwsr_obj,
>>>>                                      bool enabled);
>>>>
>>>> +/* Kernel queue trap handler — per-VM TMA and register programming */
>>>> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev);
>>>> +int  amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>>>> +                                struct amdgpu_vm *vm);
>>>> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
>>>> +                               struct amdgpu_vm *vm);
>>>> +
>>>>   #endif
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>> index de3ef9ce2234..f5e228d4bfb5 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>> @@ -2640,6 +2640,12 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>>>          if (r)
>>>>                  goto error_free_root;
>>>>
>>>> +       if (amdgpu_trap_is_enabled(adev)) {
>>>> +               r = amdgpu_trap_vm_kq_tma_alloc(adev, vm);
>>>> +               if (r)
>>>> +                       goto error_free_root;
>>>> +       }
>>>> +
>>>>          r = amdgpu_vm_create_task_info(vm);
>>>>          if (r)
>>>>                  dev_dbg(adev->dev, "Failed to create task info for VM\n");
>>>> @@ -2774,6 +2780,9 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>>>>                  amdgpu_vm_free_mapping(adev, vm, mapping, NULL);
>>>>          }
>>>>
>>>> +       if (vm->kq_tma_bo)
>>>> +               amdgpu_trap_vm_kq_tma_free(adev, vm);
>>>> +
>>>>          amdgpu_vm_pt_free_root(adev, vm);
>>>>          amdgpu_bo_unreserve(root);
>>>>          amdgpu_bo_unref(&root);
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>> index dd825e179979..064f95a83790 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>> @@ -25,6 +25,7 @@
>>>>   #define __AMDGPU_VM_H__
>>>>
>>>>   #include <linux/idr.h>
>>>> +#include <linux/iosys-map.h>
>>>>   #include <linux/kfifo.h>
>>>>   #include <linux/rbtree.h>
>>>>   #include <drm/gpu_scheduler.h>
>>>> @@ -488,6 +489,18 @@ struct amdgpu_vm {
>>>>
>>>>          /* cached fault info */
>>>>          struct amdgpu_vm_fault_info fault_info;
>>>> +
>>>> +       /*
>>>> +        * Per-VM kernel queue first-level TMA BO.
>>>> +        * Allocated at VM init, freed at VM fini — same lifecycle as page tables.
>>>> +        * Mapped read-only at AMDGPU_VA_RESERVED_TRAP_START in the GPU VM.
>>>> +        * SQ_SHADER_TMA for all kernel VMIDs points to this fixed VA; per-VM
>>>> +        * isolation is via page tables mapping different physical BOs there.
>>>> +        * CPU kernel writes second-level handler address via kq_tma_map.
>>>> +        */
>>>> +       struct amdgpu_bo        *kq_tma_bo;
>>>> +       struct amdgpu_bo_va     *kq_tma_va;
>>>> +       struct iosys_map         kq_tma_map;
>>> These should be the same for both user and kernel queues.  The only
>>> difference is who manages the vmids (driver vs MES).  They are per
>>> vmid so it doesn't matter whether it's a kernel queue or user queue.
>>>
>> I would merge these patch sets.  The trap handling is the same for
>> both kernel queues and user queues.  The only difference for kernel
>> queues is that the driver has to set the TBA/TMA registers while
>> MES/KIQ handles it for user queues.  At vm_init time, allocate the
>> memory for the trap handler, copy the trap handler to the memory and
>> add the mapping to the GPUVM address space.  Then in gfxhub init,
>> program the TBA/TMA registers for the kernel managed vmids.  Finally,
>> add the IOCTL to set/clear the second level trap handler and validate
>> the user supplied GPU VA.
> 
> Hi Alex,
> 
> Thank you for the review. I have few questions before proceeding with 
> the implementation.
> 
> I looked at how KFD handles this today in |kfd_process.c|:
> 
> /* KFD writes second-level TBA/TMA into first-level TMA */
> iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET,
>               uint64_t, tba_addr);
> iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET + sizeof(uint64_t),
>               uint64_t, tma_addr);
> 
> Where |KFD_CWSR_TMA_OFFSET = AMDGPU_GPU_PAGE_SIZE + 2048 = 0x1800|. KFD 
> uses physical GPU addresses for TBA/TMA registers, not VM virtual addresses.
> 

KFD doesn't use physical adddress, it uses the below fixed virtual address.

pdd->qpd.cwsr_base = AMDGPU_VA_RESERVED_TRAP_START(pdd->dev->adev)

KFD mechanism is to allocate one BO which is mapped to the above virtual 
address. That BO accounts for both TMA and TBA size. TMA is located at 
KFD_CWSR_TMA_OFFSET within that BO.

TBA region carries the first level handler code. It's not allowed to be 
overridden by user. Hence when we moved to new design, TBA is kept 
common for all user VMs.

TMA BO is the place through where user's second level TMA/TBA addresses 
are passed and also the place where stack is saved.

                 iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET,
                              uint64_t, tba_addr);
                 iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET + 
sizeof(uint64_t),
                              uint64_t, tma_addr);

What this code does is to write the second level handler tba/tma 
addresses in the TMA, which is located at KFD_CWSR_TMA_OFFSET within the BO.


TMA region is the only variable thing per VM. First level handler code 
executed is the same for all VMs. So in the new design, we have a first 
level TBA BO that is shared by all VMs, and that carries first level 
trap handler code. The second level TBA is allocated per VM.

I see Alex is saying the same thing which I asked through a different 
thread. There is no need to allocate separate BOs for kernel queues. All 
you need to do is program the fixed TBA/TMA first level virtual 
addresses for non-KFD VMs during init.

Regarding kernel queue draining - my understanding is that the condition 
is no work having that VMID is in progress during the change. So, it may 
not be required to entirely drain kernel queue.

Thanks,
Lijo

> Question 1 — Which fixed GPU virtual address for the merged buffer?
> 
> KFD uses physical addresses for the TBA/TMA registers. Our design maps 
> the buffer into each VM at a fixed virtual address for page table 
> isolation. Should the merged per-VM buffer use | 
> AMDGPU_VA_RESERVED_TRAP_UQ_START| as the fixed VA? Or a different address?
> 
> Question 2 — TMA offset inside the merged buffer — 0x1800 or 0x2000?
> 
> KFD places the TMA section at offset |0x1800| from the TBA start (| 
> KFD_CWSR_TMA_OFFSET|). The current UQ design uses offset |0x2000| (| 
> AMDGPU_TRAP_TBA_MAX_SIZE|). After merging into one per-VM buffer, should 
> TMA start at |0x1800| to match KFD's layout?
> 
> Question 3 — When should TRAP_EN be set?
> 
> When programming |SQ_SHADER_TBA_HI| for kernel VMIDs, should |TRAP_EN| 
> be set only after the per-VM buffer is fully mapped? Or is it safe to 
> set it at boot time since TMA slots start zeroed (meaning no second- 
> level handler installed yet)?
> 
> Question 4 — Drain kernel queues before writing second-level handler?
> 
> When |SET_L2_TRAP| is called, user queues are evicted and TLB is flushed 
> before writing. After the merge, kernel queue VMIDs also read from the 
> same TMA. Should kernel queue work be drained as well before writing the 
> second-level handler addresses?
> 
> Thanks,
> Srini
> 

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[SRINIVASAN SHANMUGAM]

[-- Attachment #1: Type: text/plain, Size: 19457 bytes --]


On 9/10/2026 12:27 PM, Lazar, Lijo wrote:
>
>
> On 10-Sep-26 11:43 AM, SRINIVASAN SHANMUGAM wrote:
>>
>> On 9/10/2026 2:20 AM, Alex Deucher wrote:
>>> On Wed, Sep 9, 2026 at 4:42 PM Alex Deucher<alexdeucher@gmail.com> 
>>> wrote:
>>>> On Sat, Sep 5, 2026 at 4:55 AM Srinivasan Shanmugam
>>>> <srinivasan.shanmugam@amd.com> wrote:
>>>>> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not 
>>>>> program
>>>>> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
>>>>> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler 
>>>>> state.
>>>>> On GFX10 and earlier HWS-based hardware, the driver programs trap
>>>>> registers via SRBM select for KFD queues but no equivalent exists for
>>>>> driver-managed kernel queue VMIDs.
>>>>>
>>>>> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub 
>>>>> version
>>>>> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
>>>>> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
>>>>> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own 
>>>>> kq_tma_bo
>>>>> there, so per-VM isolation is handled entirely by page tables without
>>>>> needing to reprogram the register per job or per submission.
>>>>>
>>>>> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
>>>>> (parallel to page table allocation) and mapped read-only into the 
>>>>> GPU VM
>>>>> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the 
>>>>> second-level
>>>>> handler address into it via kq_tma_map when userspace calls 
>>>>> SET_L2_TRAP.
>>>>> The first-level CWSR handler reads this address to chain to the
>>>>> second-level handler when a shader exception fires.
>>>>>
>>>>> This design is:
>>>>>    - Per-VM BO (not device-level) — same model as page tables
>>>>>    - Fixed VA in each VM's address space — same VA, different 
>>>>> physical BO
>>>>>    - Read-only from GPU — kernel CPU updates it via CPU mapping
>>>>>    - Treat allocation/free lifecycle identical to page tables
>>>>>
>>>>> Suggested-by: Christian König<christian.koenig@amd.com>
>>>>> Suggested-by: Alexander Deucher<alexander.deucher@amd.com>
>>>>> Cc: Lijo Lazar<lijo.lazar@amd.com>
>>>>> Cc: Timur Kristóf<timur.kristof@gmail.com>
>>>>> Cc: Samuel Pitoiset<hakzsam@gmail.com>
>>>>> Cc: Natalie Vock<natalie.vock@gmx.de>
>>>>> Signed-off-by: Srinivasan Shanmugam<srinivasan.shanmugam@amd.com>
>>>>> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
>>>>> ---
>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 
>>>>> ++++++++++++++++++++++++
>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>>>>>   5 files changed, 110 insertions(+)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h 
>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>> index 3ca187f5ade8..5624a5ab5c62 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>>>>>          void (*print_l2_protection_fault_status)(struct 
>>>>> amdgpu_device *adev,
>>>>>                                                   uint32_t status);
>>>>>          uint32_t (*get_invalidate_req)(unsigned int vmid, 
>>>>> uint32_t flush_type);
>>>>> +       void (*program_kernel_trap_vmids)(struct amdgpu_device 
>>>>> *adev);
>>>>>   };
>>>>>
>>>>>   struct amdgpu_vmhub {
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c 
>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>> index 623cac6781be..e913488ca3fa 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>>>>>
>>>>>          amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>>>>>          adev->trap_info = no_free_ptr(trap_info);
>>>>> +       amdgpu_trap_program_kernel_vmids(adev);
>>>>>
>>>>>          return 0;
>>>>>   }
>>>>> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device 
>>>>> *adev)
>>>>>          adev->trap_info = NULL;
>>>>>   }
>>>>>
>>>>> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
>>>>> +{
>>>>> +       struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
>>>>> +
>>>>> +       if (!amdgpu_trap_is_enabled(adev))
>>>>> +               return;
>>>>> +       if (!hub->vmhub_funcs || 
>>>>> !hub->vmhub_funcs->program_kernel_trap_vmids)
>>>>> +               return;
>>>>> +
>>>>> + hub->vmhub_funcs->program_kernel_trap_vmids(adev);
>>>>> +}
>>>>> +
>>>>> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>>>>> +                               struct amdgpu_vm *vm)
>>>>> +{
>>>>> +       void *cpu_addr;
>>>>> +       uint64_t va;
>>>>> +       int r;
>>>>> +
>>>>> + dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>>>>> +
>>>>> +       r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, 
>>>>> PAGE_SIZE,
>>>>> +                                   AMDGPU_GEM_DOMAIN_GTT, 
>>>>> &vm->kq_tma_bo,
>>>>> +                                   NULL, &cpu_addr);
>>>>> +       if (r)
>>>>> +               return r;
>>>>> +
>>>>> +       if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
>>>>> + iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
>>>>> +                                         (void __iomem *)cpu_addr);
>>>>> +       else
>>>>> + iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
>>>>> +
>>>>> +       vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
>>>>> +       if (!vm->kq_tma_va) {
>>>>> +               r = -ENOMEM;
>>>>> +               goto err_free_bo;
>>>>> +       }
>>>>> +
>>>>> +       va = AMDGPU_VA_RESERVED_TRAP_START(adev) & 
>>>>> AMDGPU_GMC_HOLE_MASK;
>>>>> +       r = amdgpu_vm_bo_map(adev, vm->kq_tma_va, va, 0,
>>>>> +                            AMDGPU_GPU_PAGE_SIZE,
>>>>> +                            AMDGPU_VM_PAGE_READABLE);
>>>>> +       if (r)
>>>>> +               goto err_del_va;
>>>>> +
>>>>> +       r = amdgpu_vm_bo_update(adev, vm->kq_tma_va, false);
>>>>> +       if (r)
>>>>> +               goto err_del_va;
>>>>> +
>>>>> +       return 0;
>>>>> +
>>>>> +err_del_va:
>>>>> +       amdgpu_vm_bo_del(adev, vm->kq_tma_va);
>>>>> +       vm->kq_tma_va = NULL;
>>>>> +err_free_bo:
>>>>> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
>>>>> +       return r;
>>>>> +}
>>>>> +
>>>>> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
>>>>> +                               struct amdgpu_vm *vm)
>>>>> +{
>>>>> +       uint64_t va;
>>>>> +
>>>>> +       if (!vm->kq_tma_bo)
>>>>> +               return;
>>>>> +
>>>>> + dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>>>>> +
>>>>> +       if (vm->kq_tma_va) {
>>>>> +               va = AMDGPU_VA_RESERVED_TRAP_START(adev) & 
>>>>> AMDGPU_GMC_HOLE_MASK;
>>>>> +               amdgpu_vm_bo_unmap(adev, vm->kq_tma_va, va);
>>>>> +               amdgpu_vm_bo_del(adev, vm->kq_tma_va);
>>>>> +               vm->kq_tma_va = NULL;
>>>>> +       }
>>>>> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
>>>>> +}
>>>>> +
>>>>>   static int amdgpu_trap_map_region(struct amdgpu_device *adev,
>>>>>                                    struct amdgpu_vm *vm,
>>>>>                                    struct amdgpu_trap_obj *cwsr,
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h 
>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
>>>>> index 7f83174a4742..9be5035abd1c 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
>>>>> @@ -155,4 +155,11 @@ int amdgpu_trap_set_trap_debug_flag(struct 
>>>>> amdgpu_device *adev,
>>>>>                                      struct amdgpu_trap_obj 
>>>>> *cwsr_obj,
>>>>>                                      bool enabled);
>>>>>
>>>>> +/* Kernel queue trap handler — per-VM TMA and register 
>>>>> programming */
>>>>> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev);
>>>>> +int  amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>>>>> +                                struct amdgpu_vm *vm);
>>>>> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
>>>>> +                               struct amdgpu_vm *vm);
>>>>> +
>>>>>   #endif
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>>> index de3ef9ce2234..f5e228d4bfb5 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>>> @@ -2640,6 +2640,12 @@ int amdgpu_vm_init(struct amdgpu_device 
>>>>> *adev, struct amdgpu_vm *vm,
>>>>>          if (r)
>>>>>                  goto error_free_root;
>>>>>
>>>>> +       if (amdgpu_trap_is_enabled(adev)) {
>>>>> +               r = amdgpu_trap_vm_kq_tma_alloc(adev, vm);
>>>>> +               if (r)
>>>>> +                       goto error_free_root;
>>>>> +       }
>>>>> +
>>>>>          r = amdgpu_vm_create_task_info(vm);
>>>>>          if (r)
>>>>>                  dev_dbg(adev->dev, "Failed to create task info 
>>>>> for VM\n");
>>>>> @@ -2774,6 +2780,9 @@ void amdgpu_vm_fini(struct amdgpu_device 
>>>>> *adev, struct amdgpu_vm *vm)
>>>>>                  amdgpu_vm_free_mapping(adev, vm, mapping, NULL);
>>>>>          }
>>>>>
>>>>> +       if (vm->kq_tma_bo)
>>>>> +               amdgpu_trap_vm_kq_tma_free(adev, vm);
>>>>> +
>>>>>          amdgpu_vm_pt_free_root(adev, vm);
>>>>>          amdgpu_bo_unreserve(root);
>>>>>          amdgpu_bo_unref(&root);
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h 
>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>>> index dd825e179979..064f95a83790 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>>> @@ -25,6 +25,7 @@
>>>>>   #define __AMDGPU_VM_H__
>>>>>
>>>>>   #include <linux/idr.h>
>>>>> +#include <linux/iosys-map.h>
>>>>>   #include <linux/kfifo.h>
>>>>>   #include <linux/rbtree.h>
>>>>>   #include <drm/gpu_scheduler.h>
>>>>> @@ -488,6 +489,18 @@ struct amdgpu_vm {
>>>>>
>>>>>          /* cached fault info */
>>>>>          struct amdgpu_vm_fault_info fault_info;
>>>>> +
>>>>> +       /*
>>>>> +        * Per-VM kernel queue first-level TMA BO.
>>>>> +        * Allocated at VM init, freed at VM fini — same lifecycle 
>>>>> as page tables.
>>>>> +        * Mapped read-only at AMDGPU_VA_RESERVED_TRAP_START in 
>>>>> the GPU VM.
>>>>> +        * SQ_SHADER_TMA for all kernel VMIDs points to this fixed 
>>>>> VA; per-VM
>>>>> +        * isolation is via page tables mapping different physical 
>>>>> BOs there.
>>>>> +        * CPU kernel writes second-level handler address via 
>>>>> kq_tma_map.
>>>>> +        */
>>>>> +       struct amdgpu_bo        *kq_tma_bo;
>>>>> +       struct amdgpu_bo_va     *kq_tma_va;
>>>>> +       struct iosys_map         kq_tma_map;
>>>> These should be the same for both user and kernel queues. The only
>>>> difference is who manages the vmids (driver vs MES).  They are per
>>>> vmid so it doesn't matter whether it's a kernel queue or user queue.
>>>>
>>> I would merge these patch sets.  The trap handling is the same for
>>> both kernel queues and user queues.  The only difference for kernel
>>> queues is that the driver has to set the TBA/TMA registers while
>>> MES/KIQ handles it for user queues.  At vm_init time, allocate the
>>> memory for the trap handler, copy the trap handler to the memory and
>>> add the mapping to the GPUVM address space.  Then in gfxhub init,
>>> program the TBA/TMA registers for the kernel managed vmids. Finally,
>>> add the IOCTL to set/clear the second level trap handler and validate
>>> the user supplied GPU VA.
>>
>> Hi Alex,
>>
>> Thank you for the review. I have few questions before proceeding with 
>> the implementation.
>>
>> I looked at how KFD handles this today in |kfd_process.c|:
>>
>> /* KFD writes second-level TBA/TMA into first-level TMA */
>> iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET,
>>               uint64_t, tba_addr);
>> iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET + sizeof(uint64_t),
>>               uint64_t, tma_addr);
>>
>> Where |KFD_CWSR_TMA_OFFSET = AMDGPU_GPU_PAGE_SIZE + 2048 = 0x1800|. 
>> KFD uses physical GPU addresses for TBA/TMA registers, not VM virtual 
>> addresses.
>>
>
> KFD doesn't use physical adddress, it uses the below fixed virtual 
> address.
>
> pdd->qpd.cwsr_base = AMDGPU_VA_RESERVED_TRAP_START(pdd->dev->adev)
>
> KFD mechanism is to allocate one BO which is mapped to the above 
> virtual address. That BO accounts for both TMA and TBA size. TMA is 
> located at KFD_CWSR_TMA_OFFSET within that BO.
>
> TBA region carries the first level handler code. It's not allowed to 
> be overridden by user. Hence when we moved to new design, TBA is kept 
> common for all user VMs.
>
> TMA BO is the place through where user's second level TMA/TBA 
> addresses are passed and also the place where stack is saved.
>
>                 iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET,
>                              uint64_t, tba_addr);
>                 iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET + 
> sizeof(uint64_t),
>                              uint64_t, tma_addr);
>
> What this code does is to write the second level handler tba/tma 
> addresses in the TMA, which is located at KFD_CWSR_TMA_OFFSET within 
> the BO.
>
>
> TMA region is the only variable thing per VM. First level handler code 
> executed is the same for all VMs. So in the new design, we have a 
> first level TBA BO that is shared by all VMs, and that carries first 
> level trap handler code. The second level TBA is allocated per VM.
>
> I see Alex is saying the same thing which I asked through a different 
> thread. There is no need to allocate separate BOs for kernel queues. 
> All you need to do is program the fixed TBA/TMA first level virtual 
> addresses for non-KFD VMs during init.
>
> Regarding kernel queue draining - my understanding is that the 
> condition is no work having that VMID is in progress during the 
> change. So, it may not be required to entirely drain kernel queue.
>
> Thanks,
> Lijo
>
I would like to take a moment to summarize what has been agreed upon and 
share my thoughts on the remaining open points, to ensure we are all 
aligned.

*What is agreed:*

 1. Fixed VA = |AMDGPU_VA_RESERVED_TRAP_START| — same as KFD
 2. TMA starts at offset |0x1800| from BO start — matching
    |KFD_CWSR_TMA_OFFSET|
 3. |SQ_SHADER_TBA/TMA| registers programmed in gfxhub init — kernel
    VMIDs only
 4. UQ and KQ series merged into one — no separate |kq_tba_bo|/|kq_tma_bo|
 5. |SET_L2_TRAP|: evict user queues, ensure no VMID work in progress,
    write TMA, flush TL

*Open Point 1 — Shared TBA or per-VM TBA copy?*

Alex said: /"copy the trap handler to the memory at vm_init"/ → sounds 
like per-VM copy.
Lijo said: /"TBA is kept common for all user VMs"/ → sounds like one 
shared BO.

IMO: The first-level handler code (ISA binary) is identical across all 
VMs for the same hardware. Based on this, one option is to keep a single 
shared device-level TBA BO (|adev->trap_info->isa_bo|) mapped into each 
VM at |AMDGPU_VA_RESERVED_TRAP_START|, and allocate only the TMA BO 
per-VM at |vm_init|. This avoids duplicating the same ISA binary N times 
in memory. Please let me know if a per-VM TBA copy is preferred instead.

*Open Point 2 — What happens to |AMDGPU_VA_RESERVED_TRAP_UQ_START|?*

After merging, |AMDGPU_VA_RESERVED_TRAP_START| covers both UQ and KQ in 
one buffer.

IMO*:* Remove |AMDGPU_VA_RESERVED_TRAP_UQ_START| from the reserved VA 
space entirely since it is no longer needed.

*Open Point 3 — When to set |TRAP_EN|?*

IMO*:* Set |TRAP_EN| at gfxhub init time. Since the per-VM TMA BO is 
always allocated at |vm_init| before any work is submitted, 
|AMDGPU_VA_RESERVED_TRAP_START| is always valid when any shader 
executes. No need to defer.

*Open Point 4 — How to ensure no VMID work is in progress?*

When |SET_L2_TRAP| is called, the TMA slots must be written only when no 
GPU work is actively executing under that VMID. One option is to use 
|amdgpu_vm_wait_idle()| before writing the TMA slots, which waits for 
all pending GPU work in the VM to complete. This is already used in the 
existing trap handler code for the same purpose. Please confirm if this 
is sufficient or if a different mechanism is needed.

Please confirm or correct any of the above

I would greatly appreciate it if anyone has a different perspective on 
any of the above or anything missing to be included or added  — so that 
the design is aligned with everyone.

Regards,

Srini

>> Question 1 — Which fixed GPU virtual address for the merged buffer?
>>
>> KFD uses physical addresses for the TBA/TMA registers. Our design 
>> maps the buffer into each VM at a fixed virtual address for page 
>> table isolation. Should the merged per-VM buffer use | 
>> AMDGPU_VA_RESERVED_TRAP_UQ_START| as the fixed VA? Or a different 
>> address?
>>
>> Question 2 — TMA offset inside the merged buffer — 0x1800 or 0x2000?
>>
>> KFD places the TMA section at offset |0x1800| from the TBA start (| 
>> KFD_CWSR_TMA_OFFSET|). The current UQ design uses offset |0x2000| (| 
>> AMDGPU_TRAP_TBA_MAX_SIZE|). After merging into one per-VM buffer, 
>> should TMA start at |0x1800| to match KFD's layout?
>>
>> Question 3 — When should TRAP_EN be set?
>>
>> When programming |SQ_SHADER_TBA_HI| for kernel VMIDs, should 
>> |TRAP_EN| be set only after the per-VM buffer is fully mapped? Or is 
>> it safe to set it at boot time since TMA slots start zeroed (meaning 
>> no second- level handler installed yet)?
>>
>> Question 4 — Drain kernel queues before writing second-level handler?
>>
>> When |SET_L2_TRAP| is called, user queues are evicted and TLB is 
>> flushed before writing. After the merge, kernel queue VMIDs also read 
>> from the same TMA. Should kernel queue work be drained as well before 
>> writing the second-level handler addresses?
>>
>> Thanks,
>> Srini
>>
>

[-- Attachment #2: Type: text/html, Size: 35672 bytes --]

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Lazar, Lijo]

On 10-Sep-26 1:55 PM, SRINIVASAN SHANMUGAM wrote:
> 
> On 9/10/2026 12:27 PM, Lazar, Lijo wrote:
>>
>>
>> On 10-Sep-26 11:43 AM, SRINIVASAN SHANMUGAM wrote:
>>>
>>> On 9/10/2026 2:20 AM, Alex Deucher wrote:
>>>> On Wed, Sep 9, 2026 at 4:42 PM Alex Deucher<alexdeucher@gmail.com> 
>>>> wrote:
>>>>> On Sat, Sep 5, 2026 at 4:55 AM Srinivasan Shanmugam
>>>>> <srinivasan.shanmugam@amd.com> wrote:
>>>>>> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not 
>>>>>> program
>>>>>> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
>>>>>> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler 
>>>>>> state.
>>>>>> On GFX10 and earlier HWS-based hardware, the driver programs trap
>>>>>> registers via SRBM select for KFD queues but no equivalent exists for
>>>>>> driver-managed kernel queue VMIDs.
>>>>>>
>>>>>> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub 
>>>>>> version
>>>>>> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
>>>>>> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
>>>>>> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own 
>>>>>> kq_tma_bo
>>>>>> there, so per-VM isolation is handled entirely by page tables without
>>>>>> needing to reprogram the register per job or per submission.
>>>>>>
>>>>>> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
>>>>>> (parallel to page table allocation) and mapped read-only into the 
>>>>>> GPU VM
>>>>>> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the 
>>>>>> second-level
>>>>>> handler address into it via kq_tma_map when userspace calls 
>>>>>> SET_L2_TRAP.
>>>>>> The first-level CWSR handler reads this address to chain to the
>>>>>> second-level handler when a shader exception fires.
>>>>>>
>>>>>> This design is:
>>>>>>    - Per-VM BO (not device-level) — same model as page tables
>>>>>>    - Fixed VA in each VM's address space — same VA, different 
>>>>>> physical BO
>>>>>>    - Read-only from GPU — kernel CPU updates it via CPU mapping
>>>>>>    - Treat allocation/free lifecycle identical to page tables
>>>>>>
>>>>>> Suggested-by: Christian König<christian.koenig@amd.com>
>>>>>> Suggested-by: Alexander Deucher<alexander.deucher@amd.com>
>>>>>> Cc: Lijo Lazar<lijo.lazar@amd.com>
>>>>>> Cc: Timur Kristóf<timur.kristof@gmail.com>
>>>>>> Cc: Samuel Pitoiset<hakzsam@gmail.com>
>>>>>> Cc: Natalie Vock<natalie.vock@gmx.de>
>>>>>> Signed-off-by: Srinivasan Shanmugam<srinivasan.shanmugam@amd.com>
>>>>>> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
>>>>>> ---
>>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++ 
>>>>>> ++++++
>>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>>>>>>   5 files changed, 110 insertions(+)
>>>>>>
>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/ 
>>>>>> gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>>> index 3ca187f5ade8..5624a5ab5c62 100644
>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>>> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>>>>>>          void (*print_l2_protection_fault_status)(struct 
>>>>>> amdgpu_device *adev,
>>>>>>                                                   uint32_t status);
>>>>>>          uint32_t (*get_invalidate_req)(unsigned int vmid, 
>>>>>> uint32_t flush_type);
>>>>>> +       void (*program_kernel_trap_vmids)(struct amdgpu_device 
>>>>>> *adev);
>>>>>>   };
>>>>>>
>>>>>>   struct amdgpu_vmhub {
>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/ 
>>>>>> gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>>> index 623cac6781be..e913488ca3fa 100644
>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>>> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>>>>>>
>>>>>>          amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>>>>>>          adev->trap_info = no_free_ptr(trap_info);
>>>>>> +       amdgpu_trap_program_kernel_vmids(adev);
>>>>>>
>>>>>>          return 0;
>>>>>>   }
>>>>>> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device 
>>>>>> *adev)
>>>>>>          adev->trap_info = NULL;
>>>>>>   }
>>>>>>
>>>>>> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
>>>>>> +{
>>>>>> +       struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
>>>>>> +
>>>>>> +       if (!amdgpu_trap_is_enabled(adev))
>>>>>> +               return;
>>>>>> +       if (!hub->vmhub_funcs || !hub->vmhub_funcs- 
>>>>>> >program_kernel_trap_vmids)
>>>>>> +               return;
>>>>>> +
>>>>>> + hub->vmhub_funcs->program_kernel_trap_vmids(adev);
>>>>>> +}
>>>>>> +
>>>>>> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>>>>>> +                               struct amdgpu_vm *vm)
>>>>>> +{
>>>>>> +       void *cpu_addr;
>>>>>> +       uint64_t va;
>>>>>> +       int r;
>>>>>> +
>>>>>> + dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>>>>>> +
>>>>>> +       r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, 
>>>>>> PAGE_SIZE,
>>>>>> +                                   AMDGPU_GEM_DOMAIN_GTT, &vm- 
>>>>>> >kq_tma_bo,
>>>>>> +                                   NULL, &cpu_addr);
>>>>>> +       if (r)
>>>>>> +               return r;
>>>>>> +
>>>>>> +       if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
>>>>>> + iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
>>>>>> +                                         (void __iomem *)cpu_addr);
>>>>>> +       else
>>>>>> + iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
>>>>>> +
>>>>>> +       vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
>>>>>> +       if (!vm->kq_tma_va) {
>>>>>> +               r = -ENOMEM;
>>>>>> +               goto err_free_bo;
>>>>>> +       }
>>>>>> +
>>>>>> +       va = AMDGPU_VA_RESERVED_TRAP_START(adev) & 
>>>>>> AMDGPU_GMC_HOLE_MASK;
>>>>>> +       r = amdgpu_vm_bo_map(adev, vm->kq_tma_va, va, 0,
>>>>>> +                            AMDGPU_GPU_PAGE_SIZE,
>>>>>> +                            AMDGPU_VM_PAGE_READABLE);
>>>>>> +       if (r)
>>>>>> +               goto err_del_va;
>>>>>> +
>>>>>> +       r = amdgpu_vm_bo_update(adev, vm->kq_tma_va, false);
>>>>>> +       if (r)
>>>>>> +               goto err_del_va;
>>>>>> +
>>>>>> +       return 0;
>>>>>> +
>>>>>> +err_del_va:
>>>>>> +       amdgpu_vm_bo_del(adev, vm->kq_tma_va);
>>>>>> +       vm->kq_tma_va = NULL;
>>>>>> +err_free_bo:
>>>>>> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
>>>>>> +       return r;
>>>>>> +}
>>>>>> +
>>>>>> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
>>>>>> +                               struct amdgpu_vm *vm)
>>>>>> +{
>>>>>> +       uint64_t va;
>>>>>> +
>>>>>> +       if (!vm->kq_tma_bo)
>>>>>> +               return;
>>>>>> +
>>>>>> + dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>>>>>> +
>>>>>> +       if (vm->kq_tma_va) {
>>>>>> +               va = AMDGPU_VA_RESERVED_TRAP_START(adev) & 
>>>>>> AMDGPU_GMC_HOLE_MASK;
>>>>>> +               amdgpu_vm_bo_unmap(adev, vm->kq_tma_va, va);
>>>>>> +               amdgpu_vm_bo_del(adev, vm->kq_tma_va);
>>>>>> +               vm->kq_tma_va = NULL;
>>>>>> +       }
>>>>>> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
>>>>>> +}
>>>>>> +
>>>>>>   static int amdgpu_trap_map_region(struct amdgpu_device *adev,
>>>>>>                                    struct amdgpu_vm *vm,
>>>>>>                                    struct amdgpu_trap_obj *cwsr,
>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h b/drivers/ 
>>>>>> gpu/drm/amd/amdgpu/amdgpu_trap.h
>>>>>> index 7f83174a4742..9be5035abd1c 100644
>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
>>>>>> @@ -155,4 +155,11 @@ int amdgpu_trap_set_trap_debug_flag(struct 
>>>>>> amdgpu_device *adev,
>>>>>>                                      struct amdgpu_trap_obj 
>>>>>> *cwsr_obj,
>>>>>>                                      bool enabled);
>>>>>>
>>>>>> +/* Kernel queue trap handler — per-VM TMA and register 
>>>>>> programming */
>>>>>> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev);
>>>>>> +int  amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>>>>>> +                                struct amdgpu_vm *vm);
>>>>>> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
>>>>>> +                               struct amdgpu_vm *vm);
>>>>>> +
>>>>>>   #endif
>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/ 
>>>>>> drm/amd/amdgpu/amdgpu_vm.c
>>>>>> index de3ef9ce2234..f5e228d4bfb5 100644
>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>>>> @@ -2640,6 +2640,12 @@ int amdgpu_vm_init(struct amdgpu_device 
>>>>>> *adev, struct amdgpu_vm *vm,
>>>>>>          if (r)
>>>>>>                  goto error_free_root;
>>>>>>
>>>>>> +       if (amdgpu_trap_is_enabled(adev)) {
>>>>>> +               r = amdgpu_trap_vm_kq_tma_alloc(adev, vm);
>>>>>> +               if (r)
>>>>>> +                       goto error_free_root;
>>>>>> +       }
>>>>>> +
>>>>>>          r = amdgpu_vm_create_task_info(vm);
>>>>>>          if (r)
>>>>>>                  dev_dbg(adev->dev, "Failed to create task info 
>>>>>> for VM\n");
>>>>>> @@ -2774,6 +2780,9 @@ void amdgpu_vm_fini(struct amdgpu_device 
>>>>>> *adev, struct amdgpu_vm *vm)
>>>>>>                  amdgpu_vm_free_mapping(adev, vm, mapping, NULL);
>>>>>>          }
>>>>>>
>>>>>> +       if (vm->kq_tma_bo)
>>>>>> +               amdgpu_trap_vm_kq_tma_free(adev, vm);
>>>>>> +
>>>>>>          amdgpu_vm_pt_free_root(adev, vm);
>>>>>>          amdgpu_bo_unreserve(root);
>>>>>>          amdgpu_bo_unref(&root);
>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/ 
>>>>>> drm/amd/amdgpu/amdgpu_vm.h
>>>>>> index dd825e179979..064f95a83790 100644
>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>>>> @@ -25,6 +25,7 @@
>>>>>>   #define __AMDGPU_VM_H__
>>>>>>
>>>>>>   #include <linux/idr.h>
>>>>>> +#include <linux/iosys-map.h>
>>>>>>   #include <linux/kfifo.h>
>>>>>>   #include <linux/rbtree.h>
>>>>>>   #include <drm/gpu_scheduler.h>
>>>>>> @@ -488,6 +489,18 @@ struct amdgpu_vm {
>>>>>>
>>>>>>          /* cached fault info */
>>>>>>          struct amdgpu_vm_fault_info fault_info;
>>>>>> +
>>>>>> +       /*
>>>>>> +        * Per-VM kernel queue first-level TMA BO.
>>>>>> +        * Allocated at VM init, freed at VM fini — same lifecycle 
>>>>>> as page tables.
>>>>>> +        * Mapped read-only at AMDGPU_VA_RESERVED_TRAP_START in 
>>>>>> the GPU VM.
>>>>>> +        * SQ_SHADER_TMA for all kernel VMIDs points to this fixed 
>>>>>> VA; per-VM
>>>>>> +        * isolation is via page tables mapping different physical 
>>>>>> BOs there.
>>>>>> +        * CPU kernel writes second-level handler address via 
>>>>>> kq_tma_map.
>>>>>> +        */
>>>>>> +       struct amdgpu_bo        *kq_tma_bo;
>>>>>> +       struct amdgpu_bo_va     *kq_tma_va;
>>>>>> +       struct iosys_map         kq_tma_map;
>>>>> These should be the same for both user and kernel queues. The only
>>>>> difference is who manages the vmids (driver vs MES).  They are per
>>>>> vmid so it doesn't matter whether it's a kernel queue or user queue.
>>>>>
>>>> I would merge these patch sets.  The trap handling is the same for
>>>> both kernel queues and user queues.  The only difference for kernel
>>>> queues is that the driver has to set the TBA/TMA registers while
>>>> MES/KIQ handles it for user queues.  At vm_init time, allocate the
>>>> memory for the trap handler, copy the trap handler to the memory and
>>>> add the mapping to the GPUVM address space.  Then in gfxhub init,
>>>> program the TBA/TMA registers for the kernel managed vmids. Finally,
>>>> add the IOCTL to set/clear the second level trap handler and validate
>>>> the user supplied GPU VA.
>>>
>>> Hi Alex,
>>>
>>> Thank you for the review. I have few questions before proceeding with 
>>> the implementation.
>>>
>>> I looked at how KFD handles this today in |kfd_process.c|:
>>>
>>> /* KFD writes second-level TBA/TMA into first-level TMA */
>>> iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET,
>>>               uint64_t, tba_addr);
>>> iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET + sizeof(uint64_t),
>>>               uint64_t, tma_addr);
>>>
>>> Where |KFD_CWSR_TMA_OFFSET = AMDGPU_GPU_PAGE_SIZE + 2048 = 0x1800|. 
>>> KFD uses physical GPU addresses for TBA/TMA registers, not VM virtual 
>>> addresses.
>>>
>>
>> KFD doesn't use physical adddress, it uses the below fixed virtual 
>> address.
>>
>> pdd->qpd.cwsr_base = AMDGPU_VA_RESERVED_TRAP_START(pdd->dev->adev)
>>
>> KFD mechanism is to allocate one BO which is mapped to the above 
>> virtual address. That BO accounts for both TMA and TBA size. TMA is 
>> located at KFD_CWSR_TMA_OFFSET within that BO.
>>
>> TBA region carries the first level handler code. It's not allowed to 
>> be overridden by user. Hence when we moved to new design, TBA is kept 
>> common for all user VMs.
>>
>> TMA BO is the place through where user's second level TMA/TBA 
>> addresses are passed and also the place where stack is saved.
>>
>>                 iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET,
>>                              uint64_t, tba_addr);
>>                 iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET + 
>> sizeof(uint64_t),
>>                              uint64_t, tma_addr);
>>
>> What this code does is to write the second level handler tba/tma 
>> addresses in the TMA, which is located at KFD_CWSR_TMA_OFFSET within 
>> the BO.
>>
>>
>> TMA region is the only variable thing per VM. First level handler code 
>> executed is the same for all VMs. So in the new design, we have a 
>> first level TBA BO that is shared by all VMs, and that carries first 
>> level trap handler code. The second level TBA is allocated per VM.
>>
>> I see Alex is saying the same thing which I asked through a different 
>> thread. There is no need to allocate separate BOs for kernel queues. 
>> All you need to do is program the fixed TBA/TMA first level virtual 
>> addresses for non-KFD VMs during init.
>>
>> Regarding kernel queue draining - my understanding is that the 
>> condition is no work having that VMID is in progress during the 
>> change. So, it may not be required to entirely drain kernel queue.
>>
>> Thanks,
>> Lijo
>>
> I would like to take a moment to summarize what has been agreed upon and 
> share my thoughts on the remaining open points, to ensure we are all 
> aligned.
> 
> *What is agreed:*
> 
>  1. Fixed VA = |AMDGPU_VA_RESERVED_TRAP_START| — same as KFD
>  2. TMA starts at offset |0x1800| from BO start — matching |
>     KFD_CWSR_TMA_OFFSET|
>  3. |SQ_SHADER_TBA/TMA| registers programmed in gfxhub init — kernel
>     VMIDs only
>  4. UQ and KQ series merged into one — no separate |kq_tba_bo|/|kq_tma_bo|
>  5. |SET_L2_TRAP|: evict user queues, ensure no VMID work in progress,
>     write TMA, flush TL
> 
> *Open Point 1 — Shared TBA or per-VM TBA copy?*
> 
> Alex said: /"copy the trap handler to the memory at vm_init"/ → sounds 
> like per-VM copy.
> Lijo said: /"TBA is kept common for all user VMs"/ → sounds like one 
> shared BO.
> 
> IMO: The first-level handler code (ISA binary) is identical across all 
> VMs for the same hardware. Based on this, one option is to keep a single 
> shared device-level TBA BO (|adev->trap_info->isa_bo|) mapped into each 
> VM at |AMDGPU_VA_RESERVED_TRAP_START|, and allocate only the TMA BO per- 
> VM at |vm_init|. This avoids duplicating the same ISA binary N times in 
> memory. Please let me know if a per-VM TBA copy is preferred instead.

For clarififcation, this is the existing implementation in amdgpu_cwsr.

Thanks,
Lijo

> 
> *Open Point 2 — What happens to |AMDGPU_VA_RESERVED_TRAP_UQ_START|?*
> 
> After merging, |AMDGPU_VA_RESERVED_TRAP_START| covers both UQ and KQ in 
> one buffer.
> 
> IMO*:* Remove |AMDGPU_VA_RESERVED_TRAP_UQ_START| from the reserved VA 
> space entirely since it is no longer needed.
> 
> *Open Point 3 — When to set |TRAP_EN|?*
> 
> IMO*:* Set |TRAP_EN| at gfxhub init time. Since the per-VM TMA BO is 
> always allocated at |vm_init| before any work is submitted, | 
> AMDGPU_VA_RESERVED_TRAP_START| is always valid when any shader executes. 
> No need to defer.
> 
> *Open Point 4 — How to ensure no VMID work is in progress?*
> 
> When |SET_L2_TRAP| is called, the TMA slots must be written only when no 
> GPU work is actively executing under that VMID. One option is to use | 
> amdgpu_vm_wait_idle()| before writing the TMA slots, which waits for all 
> pending GPU work in the VM to complete. This is already used in the 
> existing trap handler code for the same purpose. Please confirm if this 
> is sufficient or if a different mechanism is needed.
> 
> Please confirm or correct any of the above
> 
> I would greatly appreciate it if anyone has a different perspective on 
> any of the above or anything missing to be included or added  — so that 
> the design is aligned with everyone.
> 
> Regards,
> 
> Srini
> 
>>> Question 1 — Which fixed GPU virtual address for the merged buffer?
>>>
>>> KFD uses physical addresses for the TBA/TMA registers. Our design 
>>> maps the buffer into each VM at a fixed virtual address for page 
>>> table isolation. Should the merged per-VM buffer use | 
>>> AMDGPU_VA_RESERVED_TRAP_UQ_START| as the fixed VA? Or a different 
>>> address?
>>>
>>> Question 2 — TMA offset inside the merged buffer — 0x1800 or 0x2000?
>>>
>>> KFD places the TMA section at offset |0x1800| from the TBA start (| 
>>> KFD_CWSR_TMA_OFFSET|). The current UQ design uses offset |0x2000| (| 
>>> AMDGPU_TRAP_TBA_MAX_SIZE|). After merging into one per-VM buffer, 
>>> should TMA start at |0x1800| to match KFD's layout?
>>>
>>> Question 3 — When should TRAP_EN be set?
>>>
>>> When programming |SQ_SHADER_TBA_HI| for kernel VMIDs, should | 
>>> TRAP_EN| be set only after the per-VM buffer is fully mapped? Or is 
>>> it safe to set it at boot time since TMA slots start zeroed (meaning 
>>> no second- level handler installed yet)?
>>>
>>> Question 4 — Drain kernel queues before writing second-level handler?
>>>
>>> When |SET_L2_TRAP| is called, user queues are evicted and TLB is 
>>> flushed before writing. After the merge, kernel queue VMIDs also read 
>>> from the same TMA. Should kernel queue work be drained as well before 
>>> writing the second-level handler addresses?
>>>
>>> Thanks,
>>> Srini
>>>
>>

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Lazar, Lijo]

On 10-Sep-26 1:55 PM, SRINIVASAN SHANMUGAM wrote:
> 
> On 9/10/2026 12:27 PM, Lazar, Lijo wrote:
>>
>>
>> On 10-Sep-26 11:43 AM, SRINIVASAN SHANMUGAM wrote:
>>>
>>> On 9/10/2026 2:20 AM, Alex Deucher wrote:
>>>> On Wed, Sep 9, 2026 at 4:42 PM Alex Deucher<alexdeucher@gmail.com> 
>>>> wrote:
>>>>> On Sat, Sep 5, 2026 at 4:55 AM Srinivasan Shanmugam
>>>>> <srinivasan.shanmugam@amd.com> wrote:
>>>>>> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not 
>>>>>> program
>>>>>> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
>>>>>> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler 
>>>>>> state.
>>>>>> On GFX10 and earlier HWS-based hardware, the driver programs trap
>>>>>> registers via SRBM select for KFD queues but no equivalent exists for
>>>>>> driver-managed kernel queue VMIDs.
>>>>>>
>>>>>> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub 
>>>>>> version
>>>>>> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
>>>>>> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
>>>>>> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own 
>>>>>> kq_tma_bo
>>>>>> there, so per-VM isolation is handled entirely by page tables without
>>>>>> needing to reprogram the register per job or per submission.
>>>>>>
>>>>>> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
>>>>>> (parallel to page table allocation) and mapped read-only into the 
>>>>>> GPU VM
>>>>>> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the 
>>>>>> second-level
>>>>>> handler address into it via kq_tma_map when userspace calls 
>>>>>> SET_L2_TRAP.
>>>>>> The first-level CWSR handler reads this address to chain to the
>>>>>> second-level handler when a shader exception fires.
>>>>>>
>>>>>> This design is:
>>>>>>    - Per-VM BO (not device-level) — same model as page tables
>>>>>>    - Fixed VA in each VM's address space — same VA, different 
>>>>>> physical BO
>>>>>>    - Read-only from GPU — kernel CPU updates it via CPU mapping
>>>>>>    - Treat allocation/free lifecycle identical to page tables
>>>>>>
>>>>>> Suggested-by: Christian König<christian.koenig@amd.com>
>>>>>> Suggested-by: Alexander Deucher<alexander.deucher@amd.com>
>>>>>> Cc: Lijo Lazar<lijo.lazar@amd.com>
>>>>>> Cc: Timur Kristóf<timur.kristof@gmail.com>
>>>>>> Cc: Samuel Pitoiset<hakzsam@gmail.com>
>>>>>> Cc: Natalie Vock<natalie.vock@gmx.de>
>>>>>> Signed-off-by: Srinivasan Shanmugam<srinivasan.shanmugam@amd.com>
>>>>>> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
>>>>>> ---
>>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++ 
>>>>>> ++++++
>>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>>>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>>>>>>   5 files changed, 110 insertions(+)
>>>>>>
>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/ 
>>>>>> gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>>> index 3ca187f5ade8..5624a5ab5c62 100644
>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>>>> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>>>>>>          void (*print_l2_protection_fault_status)(struct 
>>>>>> amdgpu_device *adev,
>>>>>>                                                   uint32_t status);
>>>>>>          uint32_t (*get_invalidate_req)(unsigned int vmid, 
>>>>>> uint32_t flush_type);
>>>>>> +       void (*program_kernel_trap_vmids)(struct amdgpu_device 
>>>>>> *adev);
>>>>>>   };
>>>>>>
>>>>>>   struct amdgpu_vmhub {
>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/ 
>>>>>> gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>>> index 623cac6781be..e913488ca3fa 100644
>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
>>>>>> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>>>>>>
>>>>>>          amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>>>>>>          adev->trap_info = no_free_ptr(trap_info);
>>>>>> +       amdgpu_trap_program_kernel_vmids(adev);
>>>>>>
>>>>>>          return 0;
>>>>>>   }
>>>>>> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device 
>>>>>> *adev)
>>>>>>          adev->trap_info = NULL;
>>>>>>   }
>>>>>>
>>>>>> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
>>>>>> +{
>>>>>> +       struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
>>>>>> +
>>>>>> +       if (!amdgpu_trap_is_enabled(adev))
>>>>>> +               return;
>>>>>> +       if (!hub->vmhub_funcs || !hub->vmhub_funcs- 
>>>>>> >program_kernel_trap_vmids)
>>>>>> +               return;
>>>>>> +
>>>>>> + hub->vmhub_funcs->program_kernel_trap_vmids(adev);
>>>>>> +}
>>>>>> +
>>>>>> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>>>>>> +                               struct amdgpu_vm *vm)
>>>>>> +{
>>>>>> +       void *cpu_addr;
>>>>>> +       uint64_t va;
>>>>>> +       int r;
>>>>>> +
>>>>>> + dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>>>>>> +
>>>>>> +       r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, 
>>>>>> PAGE_SIZE,
>>>>>> +                                   AMDGPU_GEM_DOMAIN_GTT, &vm- 
>>>>>> >kq_tma_bo,
>>>>>> +                                   NULL, &cpu_addr);
>>>>>> +       if (r)
>>>>>> +               return r;
>>>>>> +
>>>>>> +       if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
>>>>>> + iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
>>>>>> +                                         (void __iomem *)cpu_addr);
>>>>>> +       else
>>>>>> + iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
>>>>>> +
>>>>>> +       vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
>>>>>> +       if (!vm->kq_tma_va) {
>>>>>> +               r = -ENOMEM;
>>>>>> +               goto err_free_bo;
>>>>>> +       }
>>>>>> +
>>>>>> +       va = AMDGPU_VA_RESERVED_TRAP_START(adev) & 
>>>>>> AMDGPU_GMC_HOLE_MASK;
>>>>>> +       r = amdgpu_vm_bo_map(adev, vm->kq_tma_va, va, 0,
>>>>>> +                            AMDGPU_GPU_PAGE_SIZE,
>>>>>> +                            AMDGPU_VM_PAGE_READABLE);
>>>>>> +       if (r)
>>>>>> +               goto err_del_va;
>>>>>> +
>>>>>> +       r = amdgpu_vm_bo_update(adev, vm->kq_tma_va, false);
>>>>>> +       if (r)
>>>>>> +               goto err_del_va;
>>>>>> +
>>>>>> +       return 0;
>>>>>> +
>>>>>> +err_del_va:
>>>>>> +       amdgpu_vm_bo_del(adev, vm->kq_tma_va);
>>>>>> +       vm->kq_tma_va = NULL;
>>>>>> +err_free_bo:
>>>>>> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
>>>>>> +       return r;
>>>>>> +}
>>>>>> +
>>>>>> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
>>>>>> +                               struct amdgpu_vm *vm)
>>>>>> +{
>>>>>> +       uint64_t va;
>>>>>> +
>>>>>> +       if (!vm->kq_tma_bo)
>>>>>> +               return;
>>>>>> +
>>>>>> + dma_resv_assert_held(vm->root.bo->tbo.base.resv);
>>>>>> +
>>>>>> +       if (vm->kq_tma_va) {
>>>>>> +               va = AMDGPU_VA_RESERVED_TRAP_START(adev) & 
>>>>>> AMDGPU_GMC_HOLE_MASK;
>>>>>> +               amdgpu_vm_bo_unmap(adev, vm->kq_tma_va, va);
>>>>>> +               amdgpu_vm_bo_del(adev, vm->kq_tma_va);
>>>>>> +               vm->kq_tma_va = NULL;
>>>>>> +       }
>>>>>> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
>>>>>> +}
>>>>>> +
>>>>>>   static int amdgpu_trap_map_region(struct amdgpu_device *adev,
>>>>>>                                    struct amdgpu_vm *vm,
>>>>>>                                    struct amdgpu_trap_obj *cwsr,
>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h b/drivers/ 
>>>>>> gpu/drm/amd/amdgpu/amdgpu_trap.h
>>>>>> index 7f83174a4742..9be5035abd1c 100644
>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
>>>>>> @@ -155,4 +155,11 @@ int amdgpu_trap_set_trap_debug_flag(struct 
>>>>>> amdgpu_device *adev,
>>>>>>                                      struct amdgpu_trap_obj 
>>>>>> *cwsr_obj,
>>>>>>                                      bool enabled);
>>>>>>
>>>>>> +/* Kernel queue trap handler — per-VM TMA and register 
>>>>>> programming */
>>>>>> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev);
>>>>>> +int  amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
>>>>>> +                                struct amdgpu_vm *vm);
>>>>>> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
>>>>>> +                               struct amdgpu_vm *vm);
>>>>>> +
>>>>>>   #endif
>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/ 
>>>>>> drm/amd/amdgpu/amdgpu_vm.c
>>>>>> index de3ef9ce2234..f5e228d4bfb5 100644
>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>>>> @@ -2640,6 +2640,12 @@ int amdgpu_vm_init(struct amdgpu_device 
>>>>>> *adev, struct amdgpu_vm *vm,
>>>>>>          if (r)
>>>>>>                  goto error_free_root;
>>>>>>
>>>>>> +       if (amdgpu_trap_is_enabled(adev)) {
>>>>>> +               r = amdgpu_trap_vm_kq_tma_alloc(adev, vm);
>>>>>> +               if (r)
>>>>>> +                       goto error_free_root;
>>>>>> +       }
>>>>>> +
>>>>>>          r = amdgpu_vm_create_task_info(vm);
>>>>>>          if (r)
>>>>>>                  dev_dbg(adev->dev, "Failed to create task info 
>>>>>> for VM\n");
>>>>>> @@ -2774,6 +2780,9 @@ void amdgpu_vm_fini(struct amdgpu_device 
>>>>>> *adev, struct amdgpu_vm *vm)
>>>>>>                  amdgpu_vm_free_mapping(adev, vm, mapping, NULL);
>>>>>>          }
>>>>>>
>>>>>> +       if (vm->kq_tma_bo)
>>>>>> +               amdgpu_trap_vm_kq_tma_free(adev, vm);
>>>>>> +
>>>>>>          amdgpu_vm_pt_free_root(adev, vm);
>>>>>>          amdgpu_bo_unreserve(root);
>>>>>>          amdgpu_bo_unref(&root);
>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/ 
>>>>>> drm/amd/amdgpu/amdgpu_vm.h
>>>>>> index dd825e179979..064f95a83790 100644
>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>>>> @@ -25,6 +25,7 @@
>>>>>>   #define __AMDGPU_VM_H__
>>>>>>
>>>>>>   #include <linux/idr.h>
>>>>>> +#include <linux/iosys-map.h>
>>>>>>   #include <linux/kfifo.h>
>>>>>>   #include <linux/rbtree.h>
>>>>>>   #include <drm/gpu_scheduler.h>
>>>>>> @@ -488,6 +489,18 @@ struct amdgpu_vm {
>>>>>>
>>>>>>          /* cached fault info */
>>>>>>          struct amdgpu_vm_fault_info fault_info;
>>>>>> +
>>>>>> +       /*
>>>>>> +        * Per-VM kernel queue first-level TMA BO.
>>>>>> +        * Allocated at VM init, freed at VM fini — same lifecycle 
>>>>>> as page tables.
>>>>>> +        * Mapped read-only at AMDGPU_VA_RESERVED_TRAP_START in 
>>>>>> the GPU VM.
>>>>>> +        * SQ_SHADER_TMA for all kernel VMIDs points to this fixed 
>>>>>> VA; per-VM
>>>>>> +        * isolation is via page tables mapping different physical 
>>>>>> BOs there.
>>>>>> +        * CPU kernel writes second-level handler address via 
>>>>>> kq_tma_map.
>>>>>> +        */
>>>>>> +       struct amdgpu_bo        *kq_tma_bo;
>>>>>> +       struct amdgpu_bo_va     *kq_tma_va;
>>>>>> +       struct iosys_map         kq_tma_map;
>>>>> These should be the same for both user and kernel queues. The only
>>>>> difference is who manages the vmids (driver vs MES).  They are per
>>>>> vmid so it doesn't matter whether it's a kernel queue or user queue.
>>>>>
>>>> I would merge these patch sets.  The trap handling is the same for
>>>> both kernel queues and user queues.  The only difference for kernel
>>>> queues is that the driver has to set the TBA/TMA registers while
>>>> MES/KIQ handles it for user queues.  At vm_init time, allocate the
>>>> memory for the trap handler, copy the trap handler to the memory and
>>>> add the mapping to the GPUVM address space.  Then in gfxhub init,
>>>> program the TBA/TMA registers for the kernel managed vmids. Finally,
>>>> add the IOCTL to set/clear the second level trap handler and validate
>>>> the user supplied GPU VA.
>>>
>>> Hi Alex,
>>>
>>> Thank you for the review. I have few questions before proceeding with 
>>> the implementation.
>>>
>>> I looked at how KFD handles this today in |kfd_process.c|:
>>>
>>> /* KFD writes second-level TBA/TMA into first-level TMA */
>>> iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET,
>>>               uint64_t, tba_addr);
>>> iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET + sizeof(uint64_t),
>>>               uint64_t, tma_addr);
>>>
>>> Where |KFD_CWSR_TMA_OFFSET = AMDGPU_GPU_PAGE_SIZE + 2048 = 0x1800|. 
>>> KFD uses physical GPU addresses for TBA/TMA registers, not VM virtual 
>>> addresses.
>>>
>>
>> KFD doesn't use physical adddress, it uses the below fixed virtual 
>> address.
>>
>> pdd->qpd.cwsr_base = AMDGPU_VA_RESERVED_TRAP_START(pdd->dev->adev)
>>
>> KFD mechanism is to allocate one BO which is mapped to the above 
>> virtual address. That BO accounts for both TMA and TBA size. TMA is 
>> located at KFD_CWSR_TMA_OFFSET within that BO.
>>
>> TBA region carries the first level handler code. It's not allowed to 
>> be overridden by user. Hence when we moved to new design, TBA is kept 
>> common for all user VMs.
>>
>> TMA BO is the place through where user's second level TMA/TBA 
>> addresses are passed and also the place where stack is saved.
>>
>>                 iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET,
>>                              uint64_t, tba_addr);
>>                 iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET + 
>> sizeof(uint64_t),
>>                              uint64_t, tma_addr);
>>
>> What this code does is to write the second level handler tba/tma 
>> addresses in the TMA, which is located at KFD_CWSR_TMA_OFFSET within 
>> the BO.
>>
>>
>> TMA region is the only variable thing per VM. First level handler code 
>> executed is the same for all VMs. So in the new design, we have a 
>> first level TBA BO that is shared by all VMs, and that carries first 
>> level trap handler code. The second level TBA is allocated per VM.
>>
>> I see Alex is saying the same thing which I asked through a different 
>> thread. There is no need to allocate separate BOs for kernel queues. 
>> All you need to do is program the fixed TBA/TMA first level virtual 
>> addresses for non-KFD VMs during init.
>>
>> Regarding kernel queue draining - my understanding is that the 
>> condition is no work having that VMID is in progress during the 
>> change. So, it may not be required to entirely drain kernel queue.
>>
>> Thanks,
>> Lijo
>>
> I would like to take a moment to summarize what has been agreed upon and 
> share my thoughts on the remaining open points, to ensure we are all 
> aligned.
> 
> *What is agreed:*
> 
>  1. Fixed VA = |AMDGPU_VA_RESERVED_TRAP_START| — same as KFD
>  2. TMA starts at offset |0x1800| from BO start — matching |
>     KFD_CWSR_TMA_OFFSET|
>  3. |SQ_SHADER_TBA/TMA| registers programmed in gfxhub init — kernel
>     VMIDs only
>  4. UQ and KQ series merged into one — no separate |kq_tba_bo|/|kq_tma_bo|
>  5. |SET_L2_TRAP|: evict user queues, ensure no VMID work in progress,
>     write TMA, flush TL
> 
> *Open Point 1 — Shared TBA or per-VM TBA copy?*
> 
> Alex said: /"copy the trap handler to the memory at vm_init"/ → sounds 
> like per-VM copy.
> Lijo said: /"TBA is kept common for all user VMs"/ → sounds like one 
> shared BO.
> 
> IMO: The first-level handler code (ISA binary) is identical across all 
> VMs for the same hardware. Based on this, one option is to keep a single 
> shared device-level TBA BO (|adev->trap_info->isa_bo|) mapped into each 
> VM at |AMDGPU_VA_RESERVED_TRAP_START|, and allocate only the TMA BO per- 
> VM at |vm_init|. This avoids duplicating the same ISA binary N times in 
> memory. Please let me know if a per-VM TBA copy is preferred instead.

For clarification, this is the existing implementation in amdgpu_cwsr.

Thanks,
Lijo

> 
> *Open Point 2 — What happens to |AMDGPU_VA_RESERVED_TRAP_UQ_START|?*
> 
> After merging, |AMDGPU_VA_RESERVED_TRAP_START| covers both UQ and KQ in 
> one buffer.
> 
> IMO*:* Remove |AMDGPU_VA_RESERVED_TRAP_UQ_START| from the reserved VA 
> space entirely since it is no longer needed.
> 
> *Open Point 3 — When to set |TRAP_EN|?*
> 
> IMO*:* Set |TRAP_EN| at gfxhub init time. Since the per-VM TMA BO is 
> always allocated at |vm_init| before any work is submitted, | 
> AMDGPU_VA_RESERVED_TRAP_START| is always valid when any shader executes. 
> No need to defer.
> 
> *Open Point 4 — How to ensure no VMID work is in progress?*
> 
> When |SET_L2_TRAP| is called, the TMA slots must be written only when no 
> GPU work is actively executing under that VMID. One option is to use | 
> amdgpu_vm_wait_idle()| before writing the TMA slots, which waits for all 
> pending GPU work in the VM to complete. This is already used in the 
> existing trap handler code for the same purpose. Please confirm if this 
> is sufficient or if a different mechanism is needed.
> 
> Please confirm or correct any of the above
> 
> I would greatly appreciate it if anyone has a different perspective on 
> any of the above or anything missing to be included or added  — so that 
> the design is aligned with everyone.
> 
> Regards,
> 
> Srini
> 
>>> Question 1 — Which fixed GPU virtual address for the merged buffer?
>>>
>>> KFD uses physical addresses for the TBA/TMA registers. Our design 
>>> maps the buffer into each VM at a fixed virtual address for page 
>>> table isolation. Should the merged per-VM buffer use | 
>>> AMDGPU_VA_RESERVED_TRAP_UQ_START| as the fixed VA? Or a different 
>>> address?
>>>
>>> Question 2 — TMA offset inside the merged buffer — 0x1800 or 0x2000?
>>>
>>> KFD places the TMA section at offset |0x1800| from the TBA start (| 
>>> KFD_CWSR_TMA_OFFSET|). The current UQ design uses offset |0x2000| (| 
>>> AMDGPU_TRAP_TBA_MAX_SIZE|). After merging into one per-VM buffer, 
>>> should TMA start at |0x1800| to match KFD's layout?
>>>
>>> Question 3 — When should TRAP_EN be set?
>>>
>>> When programming |SQ_SHADER_TBA_HI| for kernel VMIDs, should | 
>>> TRAP_EN| be set only after the per-VM buffer is fully mapped? Or is 
>>> it safe to set it at boot time since TMA slots start zeroed (meaning 
>>> no second- level handler installed yet)?
>>>
>>> Question 4 — Drain kernel queues before writing second-level handler?
>>>
>>> When |SET_L2_TRAP| is called, user queues are evicted and TLB is 
>>> flushed before writing. After the merge, kernel queue VMIDs also read 
>>> from the same TMA. Should kernel queue work be drained as well before 
>>> writing the second-level handler addresses?
>>>
>>> Thanks,
>>> Srini
>>>
>>

Re: [PATCH 1/3] drm/amdgpu: Add per-VM kernel queue first-level trap handler infrastructure

[Alex Deucher]

On Thu, Sep 10, 2026 at 4:25 AM SRINIVASAN SHANMUGAM
<srinivasan.shanmugam@amd.com> wrote:
>
>
> On 9/10/2026 12:27 PM, Lazar, Lijo wrote:
>
>
>
> On 10-Sep-26 11:43 AM, SRINIVASAN SHANMUGAM wrote:
>
>
> On 9/10/2026 2:20 AM, Alex Deucher wrote:
>
> On Wed, Sep 9, 2026 at 4:42 PM Alex Deucher<alexdeucher@gmail.com> wrote:
>
> On Sat, Sep 5, 2026 at 4:55 AM Srinivasan Shanmugam
> <srinivasan.shanmugam@amd.com> wrote:
>
> MES owns kernel queue VMIDs (1..first_kfd_vmid-1) but does not program
> SQ_SHADER_TBA/TMA for them. On GFX11+ hardware MES maps kernel queues
> via ADD_QUEUE with map_legacy_kq=1 but does not set trap handler state.
> On GFX10 and earlier HWS-based hardware, the driver programs trap
> registers via SRBM select for KFD queues but no equivalent exists for
> driver-managed kernel queue VMIDs.
>
> Add a vmhub callback program_kernel_trap_vmids() so each gfxhub version
> can write SQ_SHADER_TBA/TMA for kernel VMIDs. The TBA points to the
> device-level CWSR ISA BO. The TMA is set to the fixed per-VM virtual
> address AMDGPU_VA_RESERVED_TRAP_START — each VM maps its own kq_tma_bo
> there, so per-VM isolation is handled entirely by page tables without
> needing to reprogram the register per job or per submission.
>
> The per-VM kq_tma_bo is a small GTT BO allocated at VM creation time
> (parallel to page table allocation) and mapped read-only into the GPU VM
> at AMDGPU_VA_RESERVED_TRAP_START. The kernel CPU writes the second-level
> handler address into it via kq_tma_map when userspace calls SET_L2_TRAP.
> The first-level CWSR handler reads this address to chain to the
> second-level handler when a shader exception fires.
>
> This design is:
>    - Per-VM BO (not device-level) — same model as page tables
>    - Fixed VA in each VM's address space — same VA, different physical BO
>    - Read-only from GPU — kernel CPU updates it via CPU mapping
>    - Treat allocation/free lifecycle identical to page tables
>
> Suggested-by: Christian König<christian.koenig@amd.com>
> Suggested-by: Alexander Deucher<alexander.deucher@amd.com>
> Cc: Lijo Lazar<lijo.lazar@amd.com>
> Cc: Timur Kristóf<timur.kristof@gmail.com>
> Cc: Samuel Pitoiset<hakzsam@gmail.com>
> Cc: Natalie Vock<natalie.vock@gmx.de>
> Signed-off-by: Srinivasan Shanmugam<srinivasan.shanmugam@amd.com>
> Change-Id: I9ce352157c4aa84099cef926cba61264781e8ad9
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 80 ++++++++++++++++++++++++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h |  7 +++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c   |  9 +++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h   | 13 ++++
>   5 files changed, 110 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> index 3ca187f5ade8..5624a5ab5c62 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> @@ -115,6 +115,7 @@ struct amdgpu_vmhub_funcs {
>          void (*print_l2_protection_fault_status)(struct amdgpu_device *adev,
>                                                   uint32_t status);
>          uint32_t (*get_invalidate_req)(unsigned int vmid, uint32_t flush_type);
> +       void (*program_kernel_trap_vmids)(struct amdgpu_device *adev);
>   };
>
>   struct amdgpu_vmhub {
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> index 623cac6781be..e913488ca3fa 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
> @@ -263,6 +263,7 @@ int amdgpu_trap_init(struct amdgpu_device *adev)
>
>          amdgpu_trap_cwsr_init_save_area_info(adev, trap_info);
>          adev->trap_info = no_free_ptr(trap_info);
> +       amdgpu_trap_program_kernel_vmids(adev);
>
>          return 0;
>   }
> @@ -277,6 +278,85 @@ void amdgpu_trap_fini(struct amdgpu_device *adev)
>          adev->trap_info = NULL;
>   }
>
> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev)
> +{
> +       struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
> +
> +       if (!amdgpu_trap_is_enabled(adev))
> +               return;
> +       if (!hub->vmhub_funcs || !hub->vmhub_funcs->program_kernel_trap_vmids)
> +               return;
> +
> +       hub->vmhub_funcs->program_kernel_trap_vmids(adev);
> +}
> +
> +int amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> +                               struct amdgpu_vm *vm)
> +{
> +       void *cpu_addr;
> +       uint64_t va;
> +       int r;
> +
> +       dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> +
> +       r = amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, PAGE_SIZE,
> +                                   AMDGPU_GEM_DOMAIN_GTT, &vm->kq_tma_bo,
> +                                   NULL, &cpu_addr);
> +       if (r)
> +               return r;
> +
> +       if (vm->kq_tma_bo->kmap.bo_kmap_type & TTM_BO_MAP_IOMEM_MASK)
> +               iosys_map_set_vaddr_iomem(&vm->kq_tma_map,
> +                                         (void __iomem *)cpu_addr);
> +       else
> +               iosys_map_set_vaddr(&vm->kq_tma_map, cpu_addr);
> +
> +       vm->kq_tma_va = amdgpu_vm_bo_add(adev, vm, vm->kq_tma_bo);
> +       if (!vm->kq_tma_va) {
> +               r = -ENOMEM;
> +               goto err_free_bo;
> +       }
> +
> +       va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
> +       r = amdgpu_vm_bo_map(adev, vm->kq_tma_va, va, 0,
> +                            AMDGPU_GPU_PAGE_SIZE,
> +                            AMDGPU_VM_PAGE_READABLE);
> +       if (r)
> +               goto err_del_va;
> +
> +       r = amdgpu_vm_bo_update(adev, vm->kq_tma_va, false);
> +       if (r)
> +               goto err_del_va;
> +
> +       return 0;
> +
> +err_del_va:
> +       amdgpu_vm_bo_del(adev, vm->kq_tma_va);
> +       vm->kq_tma_va = NULL;
> +err_free_bo:
> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
> +       return r;
> +}
> +
> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
> +                               struct amdgpu_vm *vm)
> +{
> +       uint64_t va;
> +
> +       if (!vm->kq_tma_bo)
> +               return;
> +
> +       dma_resv_assert_held(vm->root.bo->tbo.base.resv);
> +
> +       if (vm->kq_tma_va) {
> +               va = AMDGPU_VA_RESERVED_TRAP_START(adev) & AMDGPU_GMC_HOLE_MASK;
> +               amdgpu_vm_bo_unmap(adev, vm->kq_tma_va, va);
> +               amdgpu_vm_bo_del(adev, vm->kq_tma_va);
> +               vm->kq_tma_va = NULL;
> +       }
> +       amdgpu_bo_free_kernel(&vm->kq_tma_bo, NULL, NULL);
> +}
> +
>   static int amdgpu_trap_map_region(struct amdgpu_device *adev,
>                                    struct amdgpu_vm *vm,
>                                    struct amdgpu_trap_obj *cwsr,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> index 7f83174a4742..9be5035abd1c 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h
> @@ -155,4 +155,11 @@ int amdgpu_trap_set_trap_debug_flag(struct amdgpu_device *adev,
>                                      struct amdgpu_trap_obj *cwsr_obj,
>                                      bool enabled);
>
> +/* Kernel queue trap handler — per-VM TMA and register programming */
> +void amdgpu_trap_program_kernel_vmids(struct amdgpu_device *adev);
> +int  amdgpu_trap_vm_kq_tma_alloc(struct amdgpu_device *adev,
> +                                struct amdgpu_vm *vm);
> +void amdgpu_trap_vm_kq_tma_free(struct amdgpu_device *adev,
> +                               struct amdgpu_vm *vm);
> +
>   #endif
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index de3ef9ce2234..f5e228d4bfb5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -2640,6 +2640,12 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>          if (r)
>                  goto error_free_root;
>
> +       if (amdgpu_trap_is_enabled(adev)) {
> +               r = amdgpu_trap_vm_kq_tma_alloc(adev, vm);
> +               if (r)
> +                       goto error_free_root;
> +       }
> +
>          r = amdgpu_vm_create_task_info(vm);
>          if (r)
>                  dev_dbg(adev->dev, "Failed to create task info for VM\n");
> @@ -2774,6 +2780,9 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
>                  amdgpu_vm_free_mapping(adev, vm, mapping, NULL);
>          }
>
> +       if (vm->kq_tma_bo)
> +               amdgpu_trap_vm_kq_tma_free(adev, vm);
> +
>          amdgpu_vm_pt_free_root(adev, vm);
>          amdgpu_bo_unreserve(root);
>          amdgpu_bo_unref(&root);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index dd825e179979..064f95a83790 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -25,6 +25,7 @@
>   #define __AMDGPU_VM_H__
>
>   #include <linux/idr.h>
> +#include <linux/iosys-map.h>
>   #include <linux/kfifo.h>
>   #include <linux/rbtree.h>
>   #include <drm/gpu_scheduler.h>
> @@ -488,6 +489,18 @@ struct amdgpu_vm {
>
>          /* cached fault info */
>          struct amdgpu_vm_fault_info fault_info;
> +
> +       /*
> +        * Per-VM kernel queue first-level TMA BO.
> +        * Allocated at VM init, freed at VM fini — same lifecycle as page tables.
> +        * Mapped read-only at AMDGPU_VA_RESERVED_TRAP_START in the GPU VM.
> +        * SQ_SHADER_TMA for all kernel VMIDs points to this fixed VA; per-VM
> +        * isolation is via page tables mapping different physical BOs there.
> +        * CPU kernel writes second-level handler address via kq_tma_map.
> +        */
> +       struct amdgpu_bo        *kq_tma_bo;
> +       struct amdgpu_bo_va     *kq_tma_va;
> +       struct iosys_map         kq_tma_map;
>
> These should be the same for both user and kernel queues.  The only
> difference is who manages the vmids (driver vs MES).  They are per
> vmid so it doesn't matter whether it's a kernel queue or user queue.
>
> I would merge these patch sets.  The trap handling is the same for
> both kernel queues and user queues.  The only difference for kernel
> queues is that the driver has to set the TBA/TMA registers while
> MES/KIQ handles it for user queues.  At vm_init time, allocate the
> memory for the trap handler, copy the trap handler to the memory and
> add the mapping to the GPUVM address space.  Then in gfxhub init,
> program the TBA/TMA registers for the kernel managed vmids.  Finally,
> add the IOCTL to set/clear the second level trap handler and validate
> the user supplied GPU VA.
>
>
> Hi Alex,
>
> Thank you for the review. I have few questions before proceeding with the implementation.
>
> I looked at how KFD handles this today in |kfd_process.c|:
>
> /* KFD writes second-level TBA/TMA into first-level TMA */
> iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET,
>               uint64_t, tba_addr);
> iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET + sizeof(uint64_t),
>               uint64_t, tma_addr);
>
> Where |KFD_CWSR_TMA_OFFSET = AMDGPU_GPU_PAGE_SIZE + 2048 = 0x1800|. KFD uses physical GPU addresses for TBA/TMA registers, not VM virtual addresses.
>
>
> KFD doesn't use physical adddress, it uses the below fixed virtual address.
>
> pdd->qpd.cwsr_base = AMDGPU_VA_RESERVED_TRAP_START(pdd->dev->adev)
>
> KFD mechanism is to allocate one BO which is mapped to the above virtual address. That BO accounts for both TMA and TBA size. TMA is located at KFD_CWSR_TMA_OFFSET within that BO.
>
> TBA region carries the first level handler code. It's not allowed to be overridden by user. Hence when we moved to new design, TBA is kept common for all user VMs.
>
> TMA BO is the place through where user's second level TMA/TBA addresses are passed and also the place where stack is saved.
>
>                 iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET,
>                              uint64_t, tba_addr);
>                 iosys_map_wr(&qpd->cwsr_map, KFD_CWSR_TMA_OFFSET + sizeof(uint64_t),
>                              uint64_t, tma_addr);
>
> What this code does is to write the second level handler tba/tma addresses in the TMA, which is located at KFD_CWSR_TMA_OFFSET within the BO.
>
>
> TMA region is the only variable thing per VM. First level handler code executed is the same for all VMs. So in the new design, we have a first level TBA BO that is shared by all VMs, and that carries first level trap handler code. The second level TBA is allocated per VM.
>
> I see Alex is saying the same thing which I asked through a different thread. There is no need to allocate separate BOs for kernel queues. All you need to do is program the fixed TBA/TMA first level virtual addresses for non-KFD VMs during init.
>
> Regarding kernel queue draining - my understanding is that the condition is no work having that VMID is in progress during the change. So, it may not be required to entirely drain kernel queue.
>
> Thanks,
> Lijo
>
> I would like to take a moment to summarize what has been agreed upon and share my thoughts on the remaining open points, to ensure we are all aligned.
>
> What is agreed:
>
> Fixed VA = AMDGPU_VA_RESERVED_TRAP_START — same as KFD
> TMA starts at offset 0x1800 from BO start — matching KFD_CWSR_TMA_OFFSET
> SQ_SHADER_TBA/TMA registers programmed in gfxhub init — kernel VMIDs only
> UQ and KQ series merged into one — no separate kq_tba_bo/kq_tma_bo
> SET_L2_TRAP: evict user queues, ensure no VMID work in progress, write TMA, flush TL
>
> Open Point 1 — Shared TBA or per-VM TBA copy?
>
> Alex said: "copy the trap handler to the memory at vm_init" → sounds like per-VM copy.
> Lijo said: "TBA is kept common for all user VMs" → sounds like one shared BO.
>
> IMO: The first-level handler code (ISA binary) is identical across all VMs for the same hardware. Based on this, one option is to keep a single shared device-level TBA BO (adev->trap_info->isa_bo) mapped into each VM at AMDGPU_VA_RESERVED_TRAP_START, and allocate only the TMA BO per-VM at vm_init. This avoids duplicating the same ISA binary N times in memory. Please let me know if a per-VM TBA copy is preferred instead.
>

Yes, I think we can have 1 global TBA and a per VM TMAs.

> Open Point 2 — What happens to AMDGPU_VA_RESERVED_TRAP_UQ_START?
>
> After merging, AMDGPU_VA_RESERVED_TRAP_START covers both UQ and KQ in one buffer.
>
> IMO: Remove AMDGPU_VA_RESERVED_TRAP_UQ_START from the reserved VA space entirely since it is no longer needed.
>

As long as the sizes are correct.  We need to make sure we have the
sizes and the locations for the TBA and TMA correct.

> Open Point 3 — When to set TRAP_EN?
>
> IMO: Set TRAP_EN at gfxhub init time. Since the per-VM TMA BO is always allocated at vm_init before any work is submitted, AMDGPU_VA_RESERVED_TRAP_START is always valid when any shader executes. No need to defer.
>

Yes.

> Open Point 4 — How to ensure no VMID work is in progress?
>
> When SET_L2_TRAP is called, the TMA slots must be written only when no GPU work is actively executing under that VMID. One option is to use amdgpu_vm_wait_idle() before writing the TMA slots, which waits for all pending GPU work in the VM to complete. This is already used in the existing trap handler code for the same purpose. Please confirm if this is sufficient or if a different mechanism is needed.
>

For user queues, we can just unmap the queues.  For kernel queues,
you'd need to wait for the VM to be idle and take the appropriate
locks to ensure that that VM doesn't get used while you are updating
the TMA.

Alex

> Please confirm or correct any of the above
>
> I would greatly appreciate it if anyone has a different perspective on any of the above or anything missing to be included or added  — so that the design is aligned with everyone.
>
> Regards,
>
> Srini
>
> Question 1 — Which fixed GPU virtual address for the merged buffer?
>
> KFD uses physical addresses for the TBA/TMA registers. Our design maps the buffer into each VM at a fixed virtual address for page table isolation. Should the merged per-VM buffer use | AMDGPU_VA_RESERVED_TRAP_UQ_START| as the fixed VA? Or a different address?
>
> Question 2 — TMA offset inside the merged buffer — 0x1800 or 0x2000?
>
> KFD places the TMA section at offset |0x1800| from the TBA start (| KFD_CWSR_TMA_OFFSET|). The current UQ design uses offset |0x2000| (| AMDGPU_TRAP_TBA_MAX_SIZE|). After merging into one per-VM buffer, should TMA start at |0x1800| to match KFD's layout?
>
> Question 3 — When should TRAP_EN be set?
>
> When programming |SQ_SHADER_TBA_HI| for kernel VMIDs, should |TRAP_EN| be set only after the per-VM buffer is fully mapped? Or is it safe to set it at boot time since TMA slots start zeroed (meaning no second- level handler installed yet)?
>
> Question 4 — Drain kernel queues before writing second-level handler?
>
> When |SET_L2_TRAP| is called, user queues are evicted and TLB is flushed before writing. After the merge, kernel queue VMIDs also read from the same TMA. Should kernel queue work be drained as well before writing the second-level handler addresses?
>
> Thanks,
> Srini
>
>

[PATCH 2/3] drm/amdgpu: Implement kernel VMID SQ_SHADER_TBA/TMA programming for GFX10/11/12

[Srinivasan Shanmugam]

Implement the program_kernel_trap_vmids() vmhub callback for all
supported hardware generations. Each implementation programs
SQ_SHADER_TBA/TMA for kernel queue VMIDs (1..first_kfd_vmid-1)
via SRBM select.

TBA points to the device-level CWSR ISA BO — same binary for all VMIDs.
TMA is set to AMDGPU_VA_RESERVED_TRAP_START, the fixed virtual address
where each VM's kq_tma_bo is mapped. The hardware resolves this VA
through the active VMID's page tables, giving each process its own
physical TMA buffer without requiring per-submission register updates.

User queue VMIDs (first_kfd_vmid..15) must not be touched here — they
are programmed by MES via the ADD_QUEUE packet's trap_handler_addr field.

The function is called at two points:
  1. amdgpu_trap_init()     — first boot, after ISA BO is ready
  2. setup_vmid_config()    — GPU resume, after GART is restored

GFX versions covered:
  GFX10   (gfxhub_v2_0):    mm-prefixed registers, single XCC
  GFX11   (gfxhub_v3_0):    reg-prefixed registers, single XCC
  GFX11.5 (gfxhub_v11_5_0): reg-prefixed registers, single XCC
  GFX12   (gfxhub_v12_0):   reg-prefixed registers, single XCC
  GFX12.1 (gfxhub_v12_1):   reg-prefixed registers, multi-XCC

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
Cc: Lijo Lazar <lijo.lazar@amd.com>
Cc: Timur Kristóf <timur.kristof@gmail.com>
Cc: Samuel Pitoiset <hakzsam@gmail.com>
Cc: Natalie Vock <natalie.vock@gmx.de>
Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
Change-Id: I2fa05a8ac96e5fe4b6e5f79dbdf62f32b8dbf96f
---
 drivers/gpu/drm/amd/amdgpu/gfxhub_v11_5_0.c | 38 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/gfxhub_v12_0.c   | 38 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c   | 36 +++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/gfxhub_v2_0.c    | 38 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/gfxhub_v3_0.c    | 33 ++++++++++++++++++
 5 files changed, 183 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v11_5_0.c b/drivers/gpu/drm/amd/amdgpu/gfxhub_v11_5_0.c
index 652eea6eae4a..b6f43c8274d4 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v11_5_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v11_5_0.c
@@ -22,6 +22,7 @@
  */
 
 #include "amdgpu.h"
+#include "amdgpu_trap.h"
 #include "gfxhub_v11_5_0.h"
 
 #include "gc/gc_11_5_0_offset.h"
@@ -290,6 +291,39 @@ static void gfxhub_v11_5_0_disable_identity_aperture(struct amdgpu_device *adev)
 
 }
 
+/*
+ * MES owns kernel VMIDs but does not program trap handler registers.
+ * Program SQ_SHADER_TBA/TMA directly via SRBM select so the first-level
+ * CWSR handler is active for kernel queue VMIDs.
+ */
+static void gfxhub_v11_5_0_program_kernel_trap_vmids(struct amdgpu_device *adev)
+{
+	u64 tba_addr = amdgpu_bo_gpu_offset(adev->trap_info->isa_bo);
+	u64 tma_addr = AMDGPU_VA_RESERVED_TRAP_START(adev);
+	int i;
+
+	if (!adev->gfx.funcs || !adev->gfx.funcs->select_me_pipe_q)
+		return;
+
+	WARN_ON(!IS_ALIGNED(tba_addr, 256));
+
+	mutex_lock(&adev->srbm_mutex);
+	for (i = 1; i < adev->vm_manager.first_kfd_vmid; i++) {
+		amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, i, 0);
+		WREG32_SOC15(GC, 0, regSQ_SHADER_TBA_LO,
+			     lower_32_bits(tba_addr >> 8));
+		WREG32_SOC15(GC, 0, regSQ_SHADER_TBA_HI,
+			     upper_32_bits(tba_addr >> 8) |
+			     (1 << SQ_SHADER_TBA_HI__TRAP_EN__SHIFT));
+		WREG32_SOC15(GC, 0, regSQ_SHADER_TMA_LO,
+			     lower_32_bits(tma_addr >> 8));
+		WREG32_SOC15(GC, 0, regSQ_SHADER_TMA_HI,
+			     upper_32_bits(tma_addr >> 8));
+	}
+	amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, 0, 0);
+	mutex_unlock(&adev->srbm_mutex);
+}
+
 static void gfxhub_v11_5_0_setup_vmid_config(struct amdgpu_device *adev)
 {
 	struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
@@ -337,6 +371,9 @@ static void gfxhub_v11_5_0_setup_vmid_config(struct amdgpu_device *adev)
 	}
 
 	hub->vm_cntx_cntl = tmp;
+
+	if (amdgpu_trap_is_enabled(adev))
+		gfxhub_v11_5_0_program_kernel_trap_vmids(adev);
 }
 
 static void gfxhub_v11_5_0_program_invalidation(struct amdgpu_device *adev)
@@ -459,6 +496,7 @@ static void gfxhub_v11_5_0_set_fault_enable_default(struct amdgpu_device *adev,
 static const struct amdgpu_vmhub_funcs gfxhub_v11_5_0_vmhub_funcs = {
 	.print_l2_protection_fault_status = gfxhub_v11_5_0_print_l2_protection_fault_status,
 	.get_invalidate_req = gfxhub_v11_5_0_get_invalidate_req,
+	.program_kernel_trap_vmids = gfxhub_v11_5_0_program_kernel_trap_vmids,
 };
 
 static void gfxhub_v11_5_0_init(struct amdgpu_device *adev)
diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_0.c
index 6cbf837d50dd..5b2f0f59b895 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_0.c
@@ -22,6 +22,7 @@
  */
 
 #include "amdgpu.h"
+#include "amdgpu_trap.h"
 #include "gfxhub_v12_0.h"
 
 #include "gc/gc_12_0_0_offset.h"
@@ -295,6 +296,39 @@ static void gfxhub_v12_0_disable_identity_aperture(struct amdgpu_device *adev)
 
 }
 
+/*
+ * MES owns kernel VMIDs but does not program trap handler registers.
+ * Program SQ_SHADER_TBA/TMA directly via SRBM select so the first-level
+ * CWSR handler is active for kernel queue VMIDs.
+ */
+static void gfxhub_v12_0_program_kernel_trap_vmids(struct amdgpu_device *adev)
+{
+	u64 tba_addr = amdgpu_bo_gpu_offset(adev->trap_info->isa_bo);
+	u64 tma_addr = AMDGPU_VA_RESERVED_TRAP_START(adev);
+	int i;
+
+	if (!adev->gfx.funcs || !adev->gfx.funcs->select_me_pipe_q)
+		return;
+
+	WARN_ON(!IS_ALIGNED(tba_addr, 256));
+
+	mutex_lock(&adev->srbm_mutex);
+	for (i = 1; i < adev->vm_manager.first_kfd_vmid; i++) {
+		amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, i, 0);
+		WREG32_SOC15(GC, 0, regSQ_SHADER_TBA_LO,
+			     lower_32_bits(tba_addr >> 8));
+		WREG32_SOC15(GC, 0, regSQ_SHADER_TBA_HI,
+			     upper_32_bits(tba_addr >> 8) |
+			     (1 << SQ_SHADER_TBA_HI__TRAP_EN__SHIFT));
+		WREG32_SOC15(GC, 0, regSQ_SHADER_TMA_LO,
+			     lower_32_bits(tma_addr >> 8));
+		WREG32_SOC15(GC, 0, regSQ_SHADER_TMA_HI,
+			     upper_32_bits(tma_addr >> 8));
+	}
+	amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, 0, 0);
+	mutex_unlock(&adev->srbm_mutex);
+}
+
 static void gfxhub_v12_0_setup_vmid_config(struct amdgpu_device *adev)
 {
 	struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
@@ -342,6 +376,9 @@ static void gfxhub_v12_0_setup_vmid_config(struct amdgpu_device *adev)
 	}
 
 	hub->vm_cntx_cntl = tmp;
+
+	if (amdgpu_trap_is_enabled(adev))
+		gfxhub_v12_0_program_kernel_trap_vmids(adev);
 }
 
 static void gfxhub_v12_0_program_invalidation(struct amdgpu_device *adev)
@@ -464,6 +501,7 @@ static void gfxhub_v12_0_set_fault_enable_default(struct amdgpu_device *adev,
 static const struct amdgpu_vmhub_funcs gfxhub_v12_0_vmhub_funcs = {
 	.print_l2_protection_fault_status = gfxhub_v12_0_print_l2_protection_fault_status,
 	.get_invalidate_req = gfxhub_v12_0_get_invalidate_req,
+	.program_kernel_trap_vmids = gfxhub_v12_0_program_kernel_trap_vmids,
 };
 
 static void gfxhub_v12_0_init(struct amdgpu_device *adev)
diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c b/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c
index 4c2fd1e6616e..528917913950 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c
@@ -21,6 +21,7 @@
  *
  */
 #include "amdgpu.h"
+#include "amdgpu_trap.h"
 #include "amdgpu_xcp.h"
 #include "gfxhub_v12_1.h"
 
@@ -406,6 +407,37 @@ static void gfxhub_v12_1_xcc_disable_identity_aperture(struct amdgpu_device *ade
 	}
 }
 
+static void gfxhub_v12_1_program_kernel_trap_vmids(struct amdgpu_device *adev)
+{
+	u64 tba_addr = amdgpu_bo_gpu_offset(adev->trap_info->isa_bo);
+	u64 tma_addr = AMDGPU_VA_RESERVED_TRAP_START(adev);
+	u32 xcc_mask = GENMASK(NUM_XCC(adev->gfx.xcc_mask) - 1, 0);
+	int i, j;
+
+	if (!adev->gfx.funcs || !adev->gfx.funcs->select_me_pipe_q)
+		return;
+
+	WARN_ON(!IS_ALIGNED(tba_addr, 256));
+
+	for_each_inst(j, xcc_mask) {
+		mutex_lock(&adev->srbm_mutex);
+		for (i = 1; i < adev->vm_manager.first_kfd_vmid; i++) {
+			amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, i, j);
+			WREG32_SOC15(GC, GET_INST(GC, j), regSQ_SHADER_TBA_LO,
+				     lower_32_bits(tba_addr >> 8));
+			WREG32_SOC15(GC, GET_INST(GC, j), regSQ_SHADER_TBA_HI,
+				     upper_32_bits(tba_addr >> 8) |
+				     (1 << SQ_SHADER_TBA_HI__TRAP_EN__SHIFT));
+			WREG32_SOC15(GC, GET_INST(GC, j), regSQ_SHADER_TMA_LO,
+				     lower_32_bits(tma_addr >> 8));
+			WREG32_SOC15(GC, GET_INST(GC, j), regSQ_SHADER_TMA_HI,
+				     upper_32_bits(tma_addr >> 8));
+		}
+		amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, 0, j);
+		mutex_unlock(&adev->srbm_mutex);
+	}
+}
+
 static void gfxhub_v12_1_xcc_setup_vmid_config(struct amdgpu_device *adev,
 					       uint32_t xcc_mask)
 {
@@ -468,6 +500,9 @@ static void gfxhub_v12_1_xcc_setup_vmid_config(struct amdgpu_device *adev,
 
 		hub->vm_cntx_cntl = tmp;
 	}
+
+	if (amdgpu_trap_is_enabled(adev))
+		gfxhub_v12_1_program_kernel_trap_vmids(adev);
 }
 
 static void gfxhub_v12_1_xcc_program_invalidation(struct amdgpu_device *adev,
@@ -751,6 +786,7 @@ static void gfxhub_v12_1_print_l2_protection_fault_status(struct amdgpu_device *
 static const struct amdgpu_vmhub_funcs gfxhub_v12_1_vmhub_funcs = {
 	.print_l2_protection_fault_status = gfxhub_v12_1_print_l2_protection_fault_status,
 	.get_invalidate_req = gfxhub_v12_1_get_invalidate_req,
+	.program_kernel_trap_vmids = gfxhub_v12_1_program_kernel_trap_vmids,
 };
 
 static void gfxhub_v12_1_xcc_init(struct amdgpu_device *adev, uint32_t xcc_mask)
diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v2_0.c b/drivers/gpu/drm/amd/amdgpu/gfxhub_v2_0.c
index 9ea593e2c719..47a25df2a0c2 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v2_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v2_0.c
@@ -22,6 +22,7 @@
  */
 
 #include "amdgpu.h"
+#include "amdgpu_trap.h"
 #include "gfxhub_v2_0.h"
 
 #include "gc/gc_10_1_0_offset.h"
@@ -280,6 +281,39 @@ static void gfxhub_v2_0_disable_identity_aperture(struct amdgpu_device *adev)
 
 }
 
+/*
+ * MES owns kernel VMIDs but does not program trap handler registers.
+ * Program SQ_SHADER_TBA/TMA directly via SRBM select so the first-level
+ * CWSR handler is active for kernel queue VMIDs.
+ */
+static void gfxhub_v2_0_program_kernel_trap_vmids(struct amdgpu_device *adev)
+{
+	u64 tba_addr = amdgpu_bo_gpu_offset(adev->trap_info->isa_bo);
+	u64 tma_addr = AMDGPU_VA_RESERVED_TRAP_START(adev);
+	int i;
+
+	if (!adev->gfx.funcs || !adev->gfx.funcs->select_me_pipe_q)
+		return;
+
+	WARN_ON(!IS_ALIGNED(tba_addr, 256));
+
+	mutex_lock(&adev->srbm_mutex);
+	for (i = 1; i < adev->vm_manager.first_kfd_vmid; i++) {
+		amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, i, 0);
+		WREG32_SOC15(GC, 0, mmSQ_SHADER_TBA_LO,
+			     lower_32_bits(tba_addr >> 8));
+		WREG32_SOC15(GC, 0, mmSQ_SHADER_TBA_HI,
+			     upper_32_bits(tba_addr >> 8) |
+			     (1 << SQ_SHADER_TBA_HI__TRAP_EN__SHIFT));
+		WREG32_SOC15(GC, 0, mmSQ_SHADER_TMA_LO,
+			     lower_32_bits(tma_addr >> 8));
+		WREG32_SOC15(GC, 0, mmSQ_SHADER_TMA_HI,
+			     upper_32_bits(tma_addr >> 8));
+	}
+	amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, 0, 0);
+	mutex_unlock(&adev->srbm_mutex);
+}
+
 static void gfxhub_v2_0_setup_vmid_config(struct amdgpu_device *adev)
 {
 	struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
@@ -327,6 +361,9 @@ static void gfxhub_v2_0_setup_vmid_config(struct amdgpu_device *adev)
 	}
 
 	hub->vm_cntx_cntl = tmp;
+
+	if (amdgpu_trap_is_enabled(adev))
+		gfxhub_v2_0_program_kernel_trap_vmids(adev);
 }
 
 static void gfxhub_v2_0_program_invalidation(struct amdgpu_device *adev)
@@ -428,6 +465,7 @@ static void gfxhub_v2_0_set_fault_enable_default(struct amdgpu_device *adev,
 static const struct amdgpu_vmhub_funcs gfxhub_v2_0_vmhub_funcs = {
 	.print_l2_protection_fault_status = gfxhub_v2_0_print_l2_protection_fault_status,
 	.get_invalidate_req = gfxhub_v2_0_get_invalidate_req,
+	.program_kernel_trap_vmids = gfxhub_v2_0_program_kernel_trap_vmids,
 };
 
 static void gfxhub_v2_0_init(struct amdgpu_device *adev)
diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v3_0.c b/drivers/gpu/drm/amd/amdgpu/gfxhub_v3_0.c
index 9e6a6e13dec0..2f59de6e9407 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v3_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v3_0.c
@@ -22,6 +22,7 @@
  */
 
 #include "amdgpu.h"
+#include "amdgpu_trap.h"
 #include "gfxhub_v3_0.h"
 
 #include "gc/gc_11_0_0_offset.h"
@@ -287,6 +288,34 @@ static void gfxhub_v3_0_disable_identity_aperture(struct amdgpu_device *adev)
 
 }
 
+static void gfxhub_v3_0_program_kernel_trap_vmids(struct amdgpu_device *adev)
+{
+	u64 tba_addr = amdgpu_bo_gpu_offset(adev->trap_info->isa_bo);
+	u64 tma_addr = AMDGPU_VA_RESERVED_TRAP_START(adev);
+	int i;
+
+	if (!adev->gfx.funcs || !adev->gfx.funcs->select_me_pipe_q)
+		return;
+
+	WARN_ON(!IS_ALIGNED(tba_addr, 256));
+
+	mutex_lock(&adev->srbm_mutex);
+	for (i = 1; i < adev->vm_manager.first_kfd_vmid; i++) {
+		amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, i, 0);
+		WREG32_SOC15(GC, 0, regSQ_SHADER_TBA_LO,
+			     lower_32_bits(tba_addr >> 8));
+		WREG32_SOC15(GC, 0, regSQ_SHADER_TBA_HI,
+			     upper_32_bits(tba_addr >> 8) |
+			     (1 << SQ_SHADER_TBA_HI__TRAP_EN__SHIFT));
+		WREG32_SOC15(GC, 0, regSQ_SHADER_TMA_LO,
+			     lower_32_bits(tma_addr >> 8));
+		WREG32_SOC15(GC, 0, regSQ_SHADER_TMA_HI,
+			     upper_32_bits(tma_addr >> 8));
+	}
+	amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, 0, 0);
+	mutex_unlock(&adev->srbm_mutex);
+}
+
 static void gfxhub_v3_0_setup_vmid_config(struct amdgpu_device *adev)
 {
 	struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
@@ -334,6 +363,9 @@ static void gfxhub_v3_0_setup_vmid_config(struct amdgpu_device *adev)
 	}
 
 	hub->vm_cntx_cntl = tmp;
+
+	if (amdgpu_trap_is_enabled(adev))
+		gfxhub_v3_0_program_kernel_trap_vmids(adev);
 }
 
 static void gfxhub_v3_0_program_invalidation(struct amdgpu_device *adev)
@@ -456,6 +488,7 @@ static void gfxhub_v3_0_set_fault_enable_default(struct amdgpu_device *adev,
 static const struct amdgpu_vmhub_funcs gfxhub_v3_0_vmhub_funcs = {
 	.print_l2_protection_fault_status = gfxhub_v3_0_print_l2_protection_fault_status,
 	.get_invalidate_req = gfxhub_v3_0_get_invalidate_req,
+	.program_kernel_trap_vmids = gfxhub_v3_0_program_kernel_trap_vmids,
 };
 
 static void gfxhub_v3_0_init(struct amdgpu_device *adev)
-- 
2.34.1

Re: [PATCH 2/3] drm/amdgpu: Implement kernel VMID SQ_SHADER_TBA/TMA programming for GFX10/11/12

[Alex Deucher]

On Sat, Sep 5, 2026 at 5:15 AM Srinivasan Shanmugam
<srinivasan.shanmugam@amd.com> wrote:
>
> Implement the program_kernel_trap_vmids() vmhub callback for all
> supported hardware generations. Each implementation programs
> SQ_SHADER_TBA/TMA for kernel queue VMIDs (1..first_kfd_vmid-1)
> via SRBM select.
>
> TBA points to the device-level CWSR ISA BO — same binary for all VMIDs.
> TMA is set to AMDGPU_VA_RESERVED_TRAP_START, the fixed virtual address
> where each VM's kq_tma_bo is mapped. The hardware resolves this VA
> through the active VMID's page tables, giving each process its own
> physical TMA buffer without requiring per-submission register updates.
>
> User queue VMIDs (first_kfd_vmid..15) must not be touched here — they
> are programmed by MES via the ADD_QUEUE packet's trap_handler_addr field.
>
> The function is called at two points:
>   1. amdgpu_trap_init()     — first boot, after ISA BO is ready
>   2. setup_vmid_config()    — GPU resume, after GART is restored
>
> GFX versions covered:
>   GFX10   (gfxhub_v2_0):    mm-prefixed registers, single XCC
>   GFX11   (gfxhub_v3_0):    reg-prefixed registers, single XCC
>   GFX11.5 (gfxhub_v11_5_0): reg-prefixed registers, single XCC
>   GFX12   (gfxhub_v12_0):   reg-prefixed registers, single XCC
>   GFX12.1 (gfxhub_v12_1):   reg-prefixed registers, multi-XCC
>
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
> Cc: Lijo Lazar <lijo.lazar@amd.com>
> Cc: Timur Kristóf <timur.kristof@gmail.com>
> Cc: Samuel Pitoiset <hakzsam@gmail.com>
> Cc: Natalie Vock <natalie.vock@gmx.de>
> Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
> Change-Id: I2fa05a8ac96e5fe4b6e5f79dbdf62f32b8dbf96f
> ---
>  drivers/gpu/drm/amd/amdgpu/gfxhub_v11_5_0.c | 38 +++++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/gfxhub_v12_0.c   | 38 +++++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c   | 36 +++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/gfxhub_v2_0.c    | 38 +++++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/gfxhub_v3_0.c    | 33 ++++++++++++++++++
>  5 files changed, 183 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v11_5_0.c b/drivers/gpu/drm/amd/amdgpu/gfxhub_v11_5_0.c
> index 652eea6eae4a..b6f43c8274d4 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v11_5_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v11_5_0.c
> @@ -22,6 +22,7 @@
>   */
>
>  #include "amdgpu.h"
> +#include "amdgpu_trap.h"
>  #include "gfxhub_v11_5_0.h"
>
>  #include "gc/gc_11_5_0_offset.h"
> @@ -290,6 +291,39 @@ static void gfxhub_v11_5_0_disable_identity_aperture(struct amdgpu_device *adev)
>
>  }
>
> +/*
> + * MES owns kernel VMIDs but does not program trap handler registers.
> + * Program SQ_SHADER_TBA/TMA directly via SRBM select so the first-level
> + * CWSR handler is active for kernel queue VMIDs.
> + */
> +static void gfxhub_v11_5_0_program_kernel_trap_vmids(struct amdgpu_device *adev)

I'd suggest renaming it to something like
program_kernel_trap_tba_tma() since it's not really programming the
vmids, they just happen to be per vmid.

> +{
> +       u64 tba_addr = amdgpu_bo_gpu_offset(adev->trap_info->isa_bo);
> +       u64 tma_addr = AMDGPU_VA_RESERVED_TRAP_START(adev);
> +       int i;
> +
> +       if (!adev->gfx.funcs || !adev->gfx.funcs->select_me_pipe_q)
> +               return;
> +
> +       WARN_ON(!IS_ALIGNED(tba_addr, 256));
> +
> +       mutex_lock(&adev->srbm_mutex);
> +       for (i = 1; i < adev->vm_manager.first_kfd_vmid; i++) {
> +               amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, i, 0);
> +               WREG32_SOC15(GC, 0, regSQ_SHADER_TBA_LO,
> +                            lower_32_bits(tba_addr >> 8));
> +               WREG32_SOC15(GC, 0, regSQ_SHADER_TBA_HI,
> +                            upper_32_bits(tba_addr >> 8) |
> +                            (1 << SQ_SHADER_TBA_HI__TRAP_EN__SHIFT));

If we set the enable bit, we need to make sure trap handler is valid
for each vmid.

> +               WREG32_SOC15(GC, 0, regSQ_SHADER_TMA_LO,
> +                            lower_32_bits(tma_addr >> 8));
> +               WREG32_SOC15(GC, 0, regSQ_SHADER_TMA_HI,
> +                            upper_32_bits(tma_addr >> 8));
> +       }
> +       amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, 0, 0);
> +       mutex_unlock(&adev->srbm_mutex);
> +}
> +
>  static void gfxhub_v11_5_0_setup_vmid_config(struct amdgpu_device *adev)
>  {
>         struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
> @@ -337,6 +371,9 @@ static void gfxhub_v11_5_0_setup_vmid_config(struct amdgpu_device *adev)
>         }
>
>         hub->vm_cntx_cntl = tmp;
> +
> +       if (amdgpu_trap_is_enabled(adev))
> +               gfxhub_v11_5_0_program_kernel_trap_vmids(adev);
>  }
>
>  static void gfxhub_v11_5_0_program_invalidation(struct amdgpu_device *adev)
> @@ -459,6 +496,7 @@ static void gfxhub_v11_5_0_set_fault_enable_default(struct amdgpu_device *adev,
>  static const struct amdgpu_vmhub_funcs gfxhub_v11_5_0_vmhub_funcs = {
>         .print_l2_protection_fault_status = gfxhub_v11_5_0_print_l2_protection_fault_status,
>         .get_invalidate_req = gfxhub_v11_5_0_get_invalidate_req,
> +       .program_kernel_trap_vmids = gfxhub_v11_5_0_program_kernel_trap_vmids,

What do we need the callback for?

Alex

>  };
>
>  static void gfxhub_v11_5_0_init(struct amdgpu_device *adev)
> diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_0.c
> index 6cbf837d50dd..5b2f0f59b895 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_0.c
> @@ -22,6 +22,7 @@
>   */
>
>  #include "amdgpu.h"
> +#include "amdgpu_trap.h"
>  #include "gfxhub_v12_0.h"
>
>  #include "gc/gc_12_0_0_offset.h"
> @@ -295,6 +296,39 @@ static void gfxhub_v12_0_disable_identity_aperture(struct amdgpu_device *adev)
>
>  }
>
> +/*
> + * MES owns kernel VMIDs but does not program trap handler registers.
> + * Program SQ_SHADER_TBA/TMA directly via SRBM select so the first-level
> + * CWSR handler is active for kernel queue VMIDs.
> + */
> +static void gfxhub_v12_0_program_kernel_trap_vmids(struct amdgpu_device *adev)
> +{
> +       u64 tba_addr = amdgpu_bo_gpu_offset(adev->trap_info->isa_bo);
> +       u64 tma_addr = AMDGPU_VA_RESERVED_TRAP_START(adev);
> +       int i;
> +
> +       if (!adev->gfx.funcs || !adev->gfx.funcs->select_me_pipe_q)
> +               return;
> +
> +       WARN_ON(!IS_ALIGNED(tba_addr, 256));
> +
> +       mutex_lock(&adev->srbm_mutex);
> +       for (i = 1; i < adev->vm_manager.first_kfd_vmid; i++) {
> +               amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, i, 0);
> +               WREG32_SOC15(GC, 0, regSQ_SHADER_TBA_LO,
> +                            lower_32_bits(tba_addr >> 8));
> +               WREG32_SOC15(GC, 0, regSQ_SHADER_TBA_HI,
> +                            upper_32_bits(tba_addr >> 8) |
> +                            (1 << SQ_SHADER_TBA_HI__TRAP_EN__SHIFT));
> +               WREG32_SOC15(GC, 0, regSQ_SHADER_TMA_LO,
> +                            lower_32_bits(tma_addr >> 8));
> +               WREG32_SOC15(GC, 0, regSQ_SHADER_TMA_HI,
> +                            upper_32_bits(tma_addr >> 8));
> +       }
> +       amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, 0, 0);
> +       mutex_unlock(&adev->srbm_mutex);
> +}
> +
>  static void gfxhub_v12_0_setup_vmid_config(struct amdgpu_device *adev)
>  {
>         struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
> @@ -342,6 +376,9 @@ static void gfxhub_v12_0_setup_vmid_config(struct amdgpu_device *adev)
>         }
>
>         hub->vm_cntx_cntl = tmp;
> +
> +       if (amdgpu_trap_is_enabled(adev))
> +               gfxhub_v12_0_program_kernel_trap_vmids(adev);
>  }
>
>  static void gfxhub_v12_0_program_invalidation(struct amdgpu_device *adev)
> @@ -464,6 +501,7 @@ static void gfxhub_v12_0_set_fault_enable_default(struct amdgpu_device *adev,
>  static const struct amdgpu_vmhub_funcs gfxhub_v12_0_vmhub_funcs = {
>         .print_l2_protection_fault_status = gfxhub_v12_0_print_l2_protection_fault_status,
>         .get_invalidate_req = gfxhub_v12_0_get_invalidate_req,
> +       .program_kernel_trap_vmids = gfxhub_v12_0_program_kernel_trap_vmids,
>  };
>
>  static void gfxhub_v12_0_init(struct amdgpu_device *adev)
> diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c b/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c
> index 4c2fd1e6616e..528917913950 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c
> @@ -21,6 +21,7 @@
>   *
>   */
>  #include "amdgpu.h"
> +#include "amdgpu_trap.h"
>  #include "amdgpu_xcp.h"
>  #include "gfxhub_v12_1.h"
>
> @@ -406,6 +407,37 @@ static void gfxhub_v12_1_xcc_disable_identity_aperture(struct amdgpu_device *ade
>         }
>  }
>
> +static void gfxhub_v12_1_program_kernel_trap_vmids(struct amdgpu_device *adev)
> +{
> +       u64 tba_addr = amdgpu_bo_gpu_offset(adev->trap_info->isa_bo);
> +       u64 tma_addr = AMDGPU_VA_RESERVED_TRAP_START(adev);
> +       u32 xcc_mask = GENMASK(NUM_XCC(adev->gfx.xcc_mask) - 1, 0);
> +       int i, j;
> +
> +       if (!adev->gfx.funcs || !adev->gfx.funcs->select_me_pipe_q)
> +               return;
> +
> +       WARN_ON(!IS_ALIGNED(tba_addr, 256));
> +
> +       for_each_inst(j, xcc_mask) {
> +               mutex_lock(&adev->srbm_mutex);
> +               for (i = 1; i < adev->vm_manager.first_kfd_vmid; i++) {
> +                       amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, i, j);
> +                       WREG32_SOC15(GC, GET_INST(GC, j), regSQ_SHADER_TBA_LO,
> +                                    lower_32_bits(tba_addr >> 8));
> +                       WREG32_SOC15(GC, GET_INST(GC, j), regSQ_SHADER_TBA_HI,
> +                                    upper_32_bits(tba_addr >> 8) |
> +                                    (1 << SQ_SHADER_TBA_HI__TRAP_EN__SHIFT));
> +                       WREG32_SOC15(GC, GET_INST(GC, j), regSQ_SHADER_TMA_LO,
> +                                    lower_32_bits(tma_addr >> 8));
> +                       WREG32_SOC15(GC, GET_INST(GC, j), regSQ_SHADER_TMA_HI,
> +                                    upper_32_bits(tma_addr >> 8));
> +               }
> +               amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, 0, j);
> +               mutex_unlock(&adev->srbm_mutex);
> +       }
> +}
> +
>  static void gfxhub_v12_1_xcc_setup_vmid_config(struct amdgpu_device *adev,
>                                                uint32_t xcc_mask)
>  {
> @@ -468,6 +500,9 @@ static void gfxhub_v12_1_xcc_setup_vmid_config(struct amdgpu_device *adev,
>
>                 hub->vm_cntx_cntl = tmp;
>         }
> +
> +       if (amdgpu_trap_is_enabled(adev))
> +               gfxhub_v12_1_program_kernel_trap_vmids(adev);
>  }
>
>  static void gfxhub_v12_1_xcc_program_invalidation(struct amdgpu_device *adev,
> @@ -751,6 +786,7 @@ static void gfxhub_v12_1_print_l2_protection_fault_status(struct amdgpu_device *
>  static const struct amdgpu_vmhub_funcs gfxhub_v12_1_vmhub_funcs = {
>         .print_l2_protection_fault_status = gfxhub_v12_1_print_l2_protection_fault_status,
>         .get_invalidate_req = gfxhub_v12_1_get_invalidate_req,
> +       .program_kernel_trap_vmids = gfxhub_v12_1_program_kernel_trap_vmids,
>  };
>
>  static void gfxhub_v12_1_xcc_init(struct amdgpu_device *adev, uint32_t xcc_mask)
> diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v2_0.c b/drivers/gpu/drm/amd/amdgpu/gfxhub_v2_0.c
> index 9ea593e2c719..47a25df2a0c2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v2_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v2_0.c
> @@ -22,6 +22,7 @@
>   */
>
>  #include "amdgpu.h"
> +#include "amdgpu_trap.h"
>  #include "gfxhub_v2_0.h"
>
>  #include "gc/gc_10_1_0_offset.h"
> @@ -280,6 +281,39 @@ static void gfxhub_v2_0_disable_identity_aperture(struct amdgpu_device *adev)
>
>  }
>
> +/*
> + * MES owns kernel VMIDs but does not program trap handler registers.
> + * Program SQ_SHADER_TBA/TMA directly via SRBM select so the first-level
> + * CWSR handler is active for kernel queue VMIDs.
> + */
> +static void gfxhub_v2_0_program_kernel_trap_vmids(struct amdgpu_device *adev)
> +{
> +       u64 tba_addr = amdgpu_bo_gpu_offset(adev->trap_info->isa_bo);
> +       u64 tma_addr = AMDGPU_VA_RESERVED_TRAP_START(adev);
> +       int i;
> +
> +       if (!adev->gfx.funcs || !adev->gfx.funcs->select_me_pipe_q)
> +               return;
> +
> +       WARN_ON(!IS_ALIGNED(tba_addr, 256));
> +
> +       mutex_lock(&adev->srbm_mutex);
> +       for (i = 1; i < adev->vm_manager.first_kfd_vmid; i++) {
> +               amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, i, 0);
> +               WREG32_SOC15(GC, 0, mmSQ_SHADER_TBA_LO,
> +                            lower_32_bits(tba_addr >> 8));
> +               WREG32_SOC15(GC, 0, mmSQ_SHADER_TBA_HI,
> +                            upper_32_bits(tba_addr >> 8) |
> +                            (1 << SQ_SHADER_TBA_HI__TRAP_EN__SHIFT));
> +               WREG32_SOC15(GC, 0, mmSQ_SHADER_TMA_LO,
> +                            lower_32_bits(tma_addr >> 8));
> +               WREG32_SOC15(GC, 0, mmSQ_SHADER_TMA_HI,
> +                            upper_32_bits(tma_addr >> 8));
> +       }
> +       amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, 0, 0);
> +       mutex_unlock(&adev->srbm_mutex);
> +}
> +
>  static void gfxhub_v2_0_setup_vmid_config(struct amdgpu_device *adev)
>  {
>         struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
> @@ -327,6 +361,9 @@ static void gfxhub_v2_0_setup_vmid_config(struct amdgpu_device *adev)
>         }
>
>         hub->vm_cntx_cntl = tmp;
> +
> +       if (amdgpu_trap_is_enabled(adev))
> +               gfxhub_v2_0_program_kernel_trap_vmids(adev);
>  }
>
>  static void gfxhub_v2_0_program_invalidation(struct amdgpu_device *adev)
> @@ -428,6 +465,7 @@ static void gfxhub_v2_0_set_fault_enable_default(struct amdgpu_device *adev,
>  static const struct amdgpu_vmhub_funcs gfxhub_v2_0_vmhub_funcs = {
>         .print_l2_protection_fault_status = gfxhub_v2_0_print_l2_protection_fault_status,
>         .get_invalidate_req = gfxhub_v2_0_get_invalidate_req,
> +       .program_kernel_trap_vmids = gfxhub_v2_0_program_kernel_trap_vmids,
>  };
>
>  static void gfxhub_v2_0_init(struct amdgpu_device *adev)
> diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v3_0.c b/drivers/gpu/drm/amd/amdgpu/gfxhub_v3_0.c
> index 9e6a6e13dec0..2f59de6e9407 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v3_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v3_0.c
> @@ -22,6 +22,7 @@
>   */
>
>  #include "amdgpu.h"
> +#include "amdgpu_trap.h"
>  #include "gfxhub_v3_0.h"
>
>  #include "gc/gc_11_0_0_offset.h"
> @@ -287,6 +288,34 @@ static void gfxhub_v3_0_disable_identity_aperture(struct amdgpu_device *adev)
>
>  }
>
> +static void gfxhub_v3_0_program_kernel_trap_vmids(struct amdgpu_device *adev)
> +{
> +       u64 tba_addr = amdgpu_bo_gpu_offset(adev->trap_info->isa_bo);
> +       u64 tma_addr = AMDGPU_VA_RESERVED_TRAP_START(adev);
> +       int i;
> +
> +       if (!adev->gfx.funcs || !adev->gfx.funcs->select_me_pipe_q)
> +               return;
> +
> +       WARN_ON(!IS_ALIGNED(tba_addr, 256));
> +
> +       mutex_lock(&adev->srbm_mutex);
> +       for (i = 1; i < adev->vm_manager.first_kfd_vmid; i++) {
> +               amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, i, 0);
> +               WREG32_SOC15(GC, 0, regSQ_SHADER_TBA_LO,
> +                            lower_32_bits(tba_addr >> 8));
> +               WREG32_SOC15(GC, 0, regSQ_SHADER_TBA_HI,
> +                            upper_32_bits(tba_addr >> 8) |
> +                            (1 << SQ_SHADER_TBA_HI__TRAP_EN__SHIFT));
> +               WREG32_SOC15(GC, 0, regSQ_SHADER_TMA_LO,
> +                            lower_32_bits(tma_addr >> 8));
> +               WREG32_SOC15(GC, 0, regSQ_SHADER_TMA_HI,
> +                            upper_32_bits(tma_addr >> 8));
> +       }
> +       amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, 0, 0);
> +       mutex_unlock(&adev->srbm_mutex);
> +}
> +
>  static void gfxhub_v3_0_setup_vmid_config(struct amdgpu_device *adev)
>  {
>         struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
> @@ -334,6 +363,9 @@ static void gfxhub_v3_0_setup_vmid_config(struct amdgpu_device *adev)
>         }
>
>         hub->vm_cntx_cntl = tmp;
> +
> +       if (amdgpu_trap_is_enabled(adev))
> +               gfxhub_v3_0_program_kernel_trap_vmids(adev);
>  }
>
>  static void gfxhub_v3_0_program_invalidation(struct amdgpu_device *adev)
> @@ -456,6 +488,7 @@ static void gfxhub_v3_0_set_fault_enable_default(struct amdgpu_device *adev,
>  static const struct amdgpu_vmhub_funcs gfxhub_v3_0_vmhub_funcs = {
>         .print_l2_protection_fault_status = gfxhub_v3_0_print_l2_protection_fault_status,
>         .get_invalidate_req = gfxhub_v3_0_get_invalidate_req,
> +       .program_kernel_trap_vmids = gfxhub_v3_0_program_kernel_trap_vmids,
>  };
>
>  static void gfxhub_v3_0_init(struct amdgpu_device *adev)
> --
> 2.34.1
>

[PATCH 3/3] drm/amdgpu: Extend second-level trap handler to kernel queues

[Srinivasan Shanmugam]

User queues already support a second-level trap handler registered via
AMDGPU_VM_OP_SET_L2_TRAP. Kernel queues had no equivalent because the
first-level TMA for kernel VMIDs was not writable by the driver.

Now that each VM has a per-VM kq_tma_bo mapped read-only at
AMDGPU_VA_RESERVED_TRAP_START (added in patch 1/3), the driver has a
CPU mapping (kq_tma_map) to write the second-level handler address into
it. The TMA layout mirrors the existing user queue TMA ABI:

  offset 0  (u64): second-level TBA virtual address
  offset 8  (u64): second-level TMA virtual address
  offset 16 (u64): debug flag

When userspace calls SET_L2_TRAP the driver writes both the per-process
user queue TMA (trap_obj->tma_map) and the per-VM kernel queue TMA
(vm->kq_tma_map). When CLEAR_L2_TRAP is called both are zeroed.

The evict -> TLB flush -> write -> restore sequence already used for
user queues covers kernel queues as well since both share the same VMID.
No additional synchronization is required.

After this patch RADV on GFX9/GFX10 hardware (which uses kernel queues
because user queues are not supported there) can install a second-level
trap handler for shader debugging via the same SET_L2_TRAP ioctl.

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Alexander Deucher <alexander.deucher@amd.com>
Cc: Lijo Lazar <lijo.lazar@amd.com>
Cc: Timur Kristóf <timur.kristof@gmail.com>
Cc: Samuel Pitoiset <hakzsam@gmail.com>
Cc: Natalie Vock <natalie.vock@gmx.de>
Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
Change-Id: Ie692cc9e6d8334f4124e3297f0520629140737ff
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 27 ++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
index e913488ca3fa..cf6dcbb903d4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c
@@ -717,6 +717,23 @@ int amdgpu_trap_vm_set_l2_trap(struct amdgpu_device *adev,
 	iosys_map_wr(&trap_obj->tma_map,
 		     AMDGPU_TRAP_L2_TBA_INDEX * sizeof(u64), u64, tba_va);
 
+	/*
+	 * Mirror to the per-VM kernel queue TMA so that kernel queues
+	 * get the same second-level handler. The kq_tma_map CPU mapping
+	 * is valid for the lifetime of the VM; iosys_map_is_null guards
+	 * against VMs where trap is disabled or kq_tma_alloc failed.
+	 */
+	if (!iosys_map_is_null(&vm->kq_tma_map)) {
+		iosys_map_wr(&vm->kq_tma_map,
+			     AMDGPU_TRAP_L2_TBA_INDEX * sizeof(u64), u64, 0);
+		dma_wmb();
+		iosys_map_wr(&vm->kq_tma_map,
+			     AMDGPU_TRAP_L2_TMA_INDEX * sizeof(u64), u64, tma_va);
+		dma_wmb();
+		iosys_map_wr(&vm->kq_tma_map,
+			     AMDGPU_TRAP_L2_TBA_INDEX * sizeof(u64), u64, tba_va);
+	}
+
 	trap_obj->l2_tba_addr = tba_va;
 	trap_obj->l2_tba_size = tba_sz;
 	trap_obj->l2_tma_addr = tma_va;
@@ -749,8 +766,18 @@ int amdgpu_trap_vm_clear_l2_trap(struct amdgpu_device *adev,
 	amdgpu_trap_clear_l2_handler(vm, trap_obj);
 	trap_obj->l2_trap_removed = false;
 
+	/* Clear kernel queue TMA as well */
+	if (!iosys_map_is_null(&vm->kq_tma_map)) {
+		iosys_map_wr(&vm->kq_tma_map,
+			     AMDGPU_TRAP_L2_TBA_INDEX * sizeof(u64), u64, 0);
+		dma_wmb();
+		iosys_map_wr(&vm->kq_tma_map,
+			     AMDGPU_TRAP_L2_TMA_INDEX * sizeof(u64), u64, 0);
+	}
+
 	amdgpu_bo_unreserve(vm->root.bo);
 	amdgpu_userq_restore_all(uq_mgr);
+
 	return 0;
 }
 
-- 
2.34.1