* [RFC PATCH 0/3] drm/amdgpu: make the reserved SPM VMID per-partition
@ 2026-11-17 16:59 James Zhu
2026-11-17 16:59 ` [RFC PATCH 1/3] drm/amdgpu: expose the XCC-masked setup_vm_pt_regs as a gfxhub callback James Zhu
` (3 more replies)
0 siblings, 4 replies; 11+ messages in thread
From: James Zhu @ 2026-11-17 16:59 UTC (permalink / raw)
To: amd-gfx; +Cc: Christian König, Sunil Khatri, James Zhu
The reserved (SPM) VMID handling was designed on single-XCC hardware,
where one GFXHUB is all there is. On multi-XCC parts with partitioning
every XCC has an own GFXHUB and every partition is exposed as an own
render node, and the current code is incoherent in three ways at once:
1. AMDGPU_VM_OP_RESERVE_VMID records the reservation on
AMDGPU_GFXHUB(0) only. The VMID grab path checks
vm->reserved_vmid[ring->vm_hub], and the rings of XCC i use
AMDGPU_GFXHUB(i) - so for any job running on XCC 1..n the
reservation silently does nothing. A debugger relying on a stable
VMID gets one on exactly one XCC of the partition it runs on, or
none at all.
2. The physical programming goes the other way: the setup_vm_pt_regs
gfxhub callback broadcasts the SPM VMID's page table base to every
XCC of the device, including XCCs of partitions the caller cannot
even submit to.
3. The exclusion is device-global: because every reservation contends
for hub 0's id_mgr, one partition's render node blocks every other
partition from reserving, which breaks the isolation the render
nodes are supposed to provide.
None of this is visible on single-XCC hardware, where "hub 0", "all
hubs" and "the caller's hubs" happen to be the same thing. gfx 9.4.3
partition modes are where the three diverge.
This series makes the per-hub infrastructure, which already exists in
the data structures (vm->reserved_vmid[] and id_mgr[] are per-vmhub
arrays), real end to end:
Patch 1 exposes the XCC-masked setup_vm_pt_regs variant, which gfxhub
v1.2 and v12.1 already implement internally, as a gfxhub callback.
Patch 2 makes the hub argument of the binding path mean what it says:
amdgpu_gmc_bind_vmid() programs and flushes only the given hub,
amdgpu_vmid_free_spm() unbinds the hub it frees, amdgpu_vm_spm_update()
binds every hub the VM holds the reservation on, and the ioctl reserves
on every GFXHUB instance. This fixes 1. and 2. while keeping the
device-wide scope.
Patch 3 then derives the hub set from the caller's partition
(fpriv->xcp_id), fixing 3.: each partition reserves, binds and
conflicts only on its own XCCs. Unreserve keeps operating on every hub,
freeing whatever the VM actually holds.
A note on the record granularity: the reservation could instead be
recorded on the partition's first GFXHUB only and treated as a pure
ownership token, closest to the original single-hub design. But the
record plays three roles at once: the ownership token, the set of hubs
whose page table base has to be bound (each XCC translates through its
own hub, so the binding must cover every XCC of the partition either
way), and the per-hub check of the VMID grab and CS paths
(vm->reserved_vmid[ring->vm_hub]). Recording on every hub of the
partition keeps the three consistent by construction, lets
amdgpu_vm_spm_update() derive the binding set from the existing
structures without a new field, and gives jobs on every XCC of the
partition the same reserved-VMID semantics instead of only on the
first one. The VMID number itself is the same everywhere either way:
it is always adev->vm_manager.spm_vmid.
No UAPI change. On devices without partitioning, or on the whole-device
node, the mask degenerates to every XCC and nothing changes; on
single-XCC parts the series is a no-op.
Compile-tested on amd-staging-drm-next with CONFIG_DRM_AMDGPU=m
(amdgpu_vm.o, amdgpu_ids.o, amdgpu_gmc.o, gfxhub_v1_2.o,
gfxhub_v12_1.o build warning-free); checkpatch clean.
James Zhu (3):
drm/amdgpu: expose the XCC-masked setup_vm_pt_regs as a gfxhub
callback
drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance
drm/amdgpu: scope AMDGPU_VM_OP_RESERVE_VMID to the caller's partition
drivers/gpu/drm/amd/amdgpu/amdgpu_gfxhub.h | 2 +
drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 4 ++
drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c | 49 ++++++++++++++++++++-
drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h | 4 ++
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 50 ++++++++++++++++------
drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c | 1 +
drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c | 1 +
7 files changed, 96 insertions(+), 15 deletions(-)
--
2.34.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* [RFC PATCH 1/3] drm/amdgpu: expose the XCC-masked setup_vm_pt_regs as a gfxhub callback
2026-11-17 16:59 [RFC PATCH 0/3] drm/amdgpu: make the reserved SPM VMID per-partition James Zhu
@ 2026-11-17 16:59 ` James Zhu
2026-11-17 16:59 ` [RFC PATCH 2/3] drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance James Zhu
` (2 subsequent siblings)
3 siblings, 0 replies; 11+ messages in thread
From: James Zhu @ 2026-11-17 16:59 UTC (permalink / raw)
To: amd-gfx; +Cc: Christian König, Sunil Khatri, James Zhu
On GFX IPs with multiple XCCs each XCC has an own GFXHUB, but the
setup_vm_pt_regs gfxhub callback programs a VMID's page table base on
every XCC. Callers which bind a VMID on a single VMHUB, like the SPM
VMID binding, have no way to reach one GFXHUB through the callback
interface.
gfxhub v1.2 and v12.1 already implement an XCC-masked variant
internally. Expose it as a new setup_vm_pt_regs_xcc callback. GFXHUBs
of IPs with a single XCC don't implement it, for them the existing
callback already is per-hub.
Signed-off-by: James Zhu <James.Zhu@amd.com>
Assisted-by: Claude:claude-fable-5
---
drivers/gpu/drm/amd/amdgpu/amdgpu_gfxhub.h | 2 ++
drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c | 1 +
drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c | 1 +
3 files changed, 4 insertions(+)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfxhub.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfxhub.h
index c7b44aeb671b..19b961cb12f6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfxhub.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfxhub.h
@@ -28,6 +28,8 @@ struct amdgpu_gfxhub_funcs {
u64 (*get_mc_fb_offset)(struct amdgpu_device *adev);
void (*setup_vm_pt_regs)(struct amdgpu_device *adev, uint32_t vmid,
uint64_t page_table_base);
+ void (*setup_vm_pt_regs_xcc)(struct amdgpu_device *adev, uint32_t vmid,
+ uint64_t page_table_base, uint32_t xcc_mask);
int (*gart_enable)(struct amdgpu_device *adev);
void (*gart_disable)(struct amdgpu_device *adev);
diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c b/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c
index b13967a1c8fb..7bc17721f463 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c
@@ -809,6 +809,7 @@ const struct amdgpu_gfxhub_funcs gfxhub_v12_1_funcs = {
.get_fb_location = gfxhub_v12_1_get_fb_location,
.get_mc_fb_offset = gfxhub_v12_1_get_mc_fb_offset,
.setup_vm_pt_regs = gfxhub_v12_1_setup_vm_pt_regs,
+ .setup_vm_pt_regs_xcc = gfxhub_v12_1_xcc_setup_vm_pt_regs,
.gart_enable = gfxhub_v12_1_gart_enable,
.gart_disable = gfxhub_v12_1_gart_disable,
.set_fault_enable_default = gfxhub_v12_1_set_fault_enable_default,
diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c b/drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c
index e89813590539..a56c072daf15 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c
@@ -634,6 +634,7 @@ static int gfxhub_v1_2_get_xgmi_info(struct amdgpu_device *adev)
const struct amdgpu_gfxhub_funcs gfxhub_v1_2_funcs = {
.get_mc_fb_offset = gfxhub_v1_2_get_mc_fb_offset,
.setup_vm_pt_regs = gfxhub_v1_2_setup_vm_pt_regs,
+ .setup_vm_pt_regs_xcc = gfxhub_v1_2_xcc_setup_vm_pt_regs,
.gart_enable = gfxhub_v1_2_gart_enable,
.gart_disable = gfxhub_v1_2_gart_disable,
.set_fault_enable_default = gfxhub_v1_2_set_fault_enable_default,
--
2.34.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [RFC PATCH 2/3] drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance
2026-11-17 16:59 [RFC PATCH 0/3] drm/amdgpu: make the reserved SPM VMID per-partition James Zhu
2026-11-17 16:59 ` [RFC PATCH 1/3] drm/amdgpu: expose the XCC-masked setup_vm_pt_regs as a gfxhub callback James Zhu
@ 2026-11-17 16:59 ` James Zhu
2026-11-17 17:21 ` sashiko-bot
2026-11-17 16:59 ` [RFC PATCH 3/3] drm/amdgpu: scope AMDGPU_VM_OP_RESERVE_VMID to the caller's partition James Zhu
2026-11-17 18:19 ` [RFC PATCH v2 0/3] drm/amdgpu: make the reserved SPM VMID per-partition James Zhu
3 siblings, 1 reply; 11+ messages in thread
From: James Zhu @ 2026-11-17 16:59 UTC (permalink / raw)
To: amd-gfx; +Cc: Christian König, Sunil Khatri, James Zhu
The SPM VMID reservation made through AMDGPU_VM_OP_RESERVE_VMID is
recorded on AMDGPU_GFXHUB(0) only, while gfxhub v1.2's
setup_vm_pt_regs broadcasts the page table base to every XCC. On
multi-XCC parts the two disagree, in both directions:
- the reservation only takes effect for jobs running on XCC 0, since
the VMID grab path checks vm->reserved_vmid[ring->vm_hub] and the
rings of XCC i use AMDGPU_GFXHUB(i);
- the page table base is physically programmed on all XCCs anyway,
and conflicting reservations are only detected on hub 0's id_mgr.
Make the hub argument real end to end: amdgpu_gmc_bind_vmid() programs
only the given hub, through the XCC-masked gfxhub callback where one
exists, and amdgpu_vmid_free_spm() unbinds the hub it frees instead of
hub 0. amdgpu_vm_spm_update() binds every GFXHUB on which @vm holds
the reservation. The VM ioctl reserves the VMID on every GFXHUB
instance through the new amdgpu_vmid_alloc_spm_gfxhubs() helper, so
the reservation now takes effect, is bound and is flushed on every XCC
the VM's jobs can run on, and a conflicting reservation shows up on
whichever hub two VMs share.
Signed-off-by: James Zhu <James.Zhu@amd.com>
Assisted-by: Claude:claude-fable-5
---
drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 4 ++
drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c | 49 ++++++++++++++++++++++++-
drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h | 4 ++
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 40 +++++++++++++-------
4 files changed, 82 insertions(+), 15 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
index 08d3ff61edcc..704a30ad388a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
@@ -797,6 +797,7 @@ void amdgpu_gmc_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid,
*
* For VMIDs that are bound directly to a VM's page table outside the
* normal per-job VM flush path, e.g. a dedicated out-of-pool VMID.
+ * Only @vmhub is programmed and flushed, other hubs keep their binding.
*/
void amdgpu_gmc_bind_vmid(struct amdgpu_device *adev, unsigned int vmhub,
unsigned int vmid, uint64_t pd_addr)
@@ -805,6 +806,9 @@ void amdgpu_gmc_bind_vmid(struct amdgpu_device *adev, unsigned int vmhub,
if (is_mmhub)
adev->mmhub.funcs->setup_vm_pt_regs(adev, vmid, pd_addr);
+ else if (adev->gfxhub.funcs->setup_vm_pt_regs_xcc)
+ adev->gfxhub.funcs->setup_vm_pt_regs_xcc(adev, vmid, pd_addr,
+ BIT(vmhub - AMDGPU_GFXHUB_START));
else
adev->gfxhub.funcs->setup_vm_pt_regs(adev, vmid, pd_addr);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
index 9c879ac68352..41f3dba530e3 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
@@ -651,12 +651,59 @@ void amdgpu_vmid_free_spm(struct amdgpu_device *adev, struct amdgpu_vm *vm,
id_mgr->reserved_vmid = false;
if (adev->vm_manager.spm_vmid && adev->gfxhub.funcs)
- amdgpu_gmc_bind_vmid(adev, AMDGPU_GFXHUB(0),
+ amdgpu_gmc_bind_vmid(adev, vmhub,
adev->vm_manager.spm_vmid, 0);
}
mutex_unlock(&id_mgr->lock);
}
+/*
+ * amdgpu_vmid_alloc_spm_gfxhubs - reserve the SPM VMID on a set of GFXHUBs
+ * @adev: amdgpu device structure
+ * @vm: the VM to reserve the ID for
+ * @xcc_mask: XCC instances on whose GFXHUBs to reserve
+ *
+ * Reserve the SPM VMID on the GFXHUB of every XCC in @xcc_mask, so that
+ * the reservation covers every hub the VM's jobs can run on.
+ *
+ * Returns: 0 for success, the first per-hub error otherwise. On an error
+ * the reservation is released on every hub in @xcc_mask again.
+ */
+int amdgpu_vmid_alloc_spm_gfxhubs(struct amdgpu_device *adev,
+ struct amdgpu_vm *vm, uint32_t xcc_mask)
+{
+ int i, r;
+
+ for_each_inst(i, xcc_mask) {
+ r = amdgpu_vmid_alloc_spm(adev, vm, AMDGPU_GFXHUB(i));
+ if (r) {
+ amdgpu_vmid_free_spm_gfxhubs(adev, vm, xcc_mask);
+ return r;
+ }
+ }
+
+ return 0;
+}
+
+/*
+ * amdgpu_vmid_free_spm_gfxhubs - undo amdgpu_vmid_alloc_spm_gfxhubs
+ * @adev: amdgpu device structure
+ * @vm: the VM with the reserved ID
+ * @xcc_mask: XCC instances on whose GFXHUBs to free the reservation
+ *
+ * Free the SPM VMID on every GFXHUB in @xcc_mask @vm holds it on, hubs
+ * without a reservation are skipped. The same wait requirement as for
+ * amdgpu_vmid_free_spm() applies.
+ */
+void amdgpu_vmid_free_spm_gfxhubs(struct amdgpu_device *adev,
+ struct amdgpu_vm *vm, uint32_t xcc_mask)
+{
+ int i;
+
+ for_each_inst(i, xcc_mask)
+ amdgpu_vmid_free_spm(adev, vm, AMDGPU_GFXHUB(i));
+}
+
/**
* amdgpu_vmid_reset - reset VMID to zero
*
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h
index c134000eecfd..d2b225d0aa38 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h
@@ -129,6 +129,10 @@ int amdgpu_vmid_alloc_spm(struct amdgpu_device *adev, struct amdgpu_vm *vm,
unsigned vmhub);
void amdgpu_vmid_free_spm(struct amdgpu_device *adev, struct amdgpu_vm *vm,
unsigned vmhub);
+int amdgpu_vmid_alloc_spm_gfxhubs(struct amdgpu_device *adev,
+ struct amdgpu_vm *vm, uint32_t xcc_mask);
+void amdgpu_vmid_free_spm_gfxhubs(struct amdgpu_device *adev,
+ struct amdgpu_vm *vm, uint32_t xcc_mask);
int amdgpu_vmid_grab(struct amdgpu_vm *vm, struct amdgpu_ring *ring,
struct amdgpu_job *job, struct dma_fence **fence);
void amdgpu_vmid_reset(struct amdgpu_device *adev, unsigned vmhub,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 6b4dc0276e27..bd9aa35933dc 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -589,30 +589,36 @@ uint64_t amdgpu_vm_generation(struct amdgpu_device *adev, struct amdgpu_vm *vm)
* The SPM VMID is fixed (adev->vm_manager.spm_vmid), so its page-table-base
* register must track whichever VM currently holds the reservation instead
* of being set per-job like a normal VMID. Bind it to @vm's real root PD
- * address if @vm holds the reservation; otherwise do nothing, since @vm is
- * not the one that could legitimately change the current binding.
+ * address on every GFXHUB @vm holds the reservation on; hubs reserved by
+ * other VMs or not reserved at all are left alone, since @vm is not the
+ * one that could legitimately change their binding.
*
* @vm's root PD must be locked by the caller.
*/
void amdgpu_vm_spm_update(struct amdgpu_device *adev, struct amdgpu_vm *vm)
{
- struct amdgpu_kq_vmid_mgr *id_mgr =
- &adev->vm_manager.id_mgr[AMDGPU_GFXHUB(0)];
uint64_t pd_addr;
+ int i;
if (!adev->vm_manager.spm_vmid || !adev->gfxhub.funcs)
return;
- mutex_lock(&id_mgr->lock);
- if (vm->reserved_vmid[AMDGPU_GFXHUB(0)] !=
- &id_mgr->ids[adev->vm_manager.spm_vmid] || !amdgpu_vm_ready(vm)) {
- mutex_unlock(&id_mgr->lock);
+ if (!amdgpu_vm_ready(vm))
return;
- }
pd_addr = amdgpu_gmc_pd_addr(vm->root.bo);
- amdgpu_gmc_bind_vmid(adev, AMDGPU_GFXHUB(0), adev->vm_manager.spm_vmid, pd_addr);
- mutex_unlock(&id_mgr->lock);
+ for_each_inst(i, adev->gfx.xcc_mask) {
+ struct amdgpu_kq_vmid_mgr *id_mgr =
+ &adev->vm_manager.id_mgr[AMDGPU_GFXHUB(i)];
+
+ mutex_lock(&id_mgr->lock);
+ if (vm->reserved_vmid[AMDGPU_GFXHUB(i)] ==
+ &id_mgr->ids[adev->vm_manager.spm_vmid])
+ amdgpu_gmc_bind_vmid(adev, AMDGPU_GFXHUB(i),
+ adev->vm_manager.spm_vmid,
+ pd_addr);
+ mutex_unlock(&id_mgr->lock);
+ }
}
/**
@@ -3018,7 +3024,8 @@ int amdgpu_vm_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
int r;
/* We only have requirement to reserve vmid from gfxhub */
- r = amdgpu_vmid_alloc_spm(adev, vm, AMDGPU_GFXHUB(0));
+ r = amdgpu_vmid_alloc_spm_gfxhubs(adev, vm,
+ adev->gfx.xcc_mask);
if (!r && !amdgpu_bo_reserve(vm->root.bo, true)) {
amdgpu_vm_spm_update(adev, vm);
amdgpu_bo_unreserve(vm->root.bo);
@@ -3027,14 +3034,19 @@ int amdgpu_vm_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
}
case AMDGPU_VM_OP_UNRESERVE_VMID: {
struct amdgpu_bo *root = vm->root.bo;
+ bool reserved = false;
+ int i;
amdgpu_bo_reserve(root, true);
+ for_each_inst(i, adev->gfx.xcc_mask)
+ reserved |= amdgpu_vmid_uses_reserved(vm,
+ AMDGPU_GFXHUB(i));
/* Wait for the jobs which might still use the reserved VMID */
- if (vm->reserved_vmid[AMDGPU_GFXHUB(0)])
+ if (reserved)
dma_resv_wait_timeout(root->tbo.base.resv,
DMA_RESV_USAGE_READ, false,
MAX_SCHEDULE_TIMEOUT);
- amdgpu_vmid_free_spm(adev, vm, AMDGPU_GFXHUB(0));
+ amdgpu_vmid_free_spm_gfxhubs(adev, vm, adev->gfx.xcc_mask);
amdgpu_bo_unreserve(root);
break;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [RFC PATCH 3/3] drm/amdgpu: scope AMDGPU_VM_OP_RESERVE_VMID to the caller's partition
2026-11-17 16:59 [RFC PATCH 0/3] drm/amdgpu: make the reserved SPM VMID per-partition James Zhu
2026-11-17 16:59 ` [RFC PATCH 1/3] drm/amdgpu: expose the XCC-masked setup_vm_pt_regs as a gfxhub callback James Zhu
2026-11-17 16:59 ` [RFC PATCH 2/3] drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance James Zhu
@ 2026-11-17 16:59 ` James Zhu
2026-11-17 17:18 ` sashiko-bot
2026-11-17 18:19 ` [RFC PATCH v2 0/3] drm/amdgpu: make the reserved SPM VMID per-partition James Zhu
3 siblings, 1 reply; 11+ messages in thread
From: James Zhu @ 2026-11-17 16:59 UTC (permalink / raw)
To: amd-gfx; +Cc: Christian König, Sunil Khatri, James Zhu
On partitioned devices every partition is exposed as an own render
node and fpriv->xcp_id records which partition a drm file belongs to.
But AMDGPU_VM_OP_RESERVE_VMID reserves the VMID on every GFXHUB of the
device: a reservation made on one partition's render node blocks every
other partition from reserving, and rebinds the VMID's page table base
on XCCs the caller can't even submit to.
Derive the GFXHUB set from the caller's partition instead. The
reservation then only covers the XCCs of the caller's partition, and
each partition can hold the reserved VMID on its own hubs
independently, matching the isolation the render nodes promise.
Unreserve keeps operating on every hub, freeing whatever the VM
actually holds.
Without partitioning, or on the whole-device node, the mask stays
adev->gfx.xcc_mask and nothing changes.
Signed-off-by: James Zhu <James.Zhu@amd.com>
Assisted-by: Claude:claude-fable-5
---
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index bd9aa35933dc..d37af0dbd691 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -3021,11 +3021,21 @@ int amdgpu_vm_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
switch (args->in.op) {
case AMDGPU_VM_OP_RESERVE_VMID: {
+ uint32_t xcc_mask = adev->gfx.xcc_mask;
int r;
+ /*
+ * Scope the reservation to the hardware partition behind the
+ * caller's render node, so other partitions can hold the
+ * reserved VMID on their own XCCs independently.
+ */
+ if (adev->xcp_mgr && fpriv->xcp_id != AMDGPU_XCP_NO_PARTITION)
+ amdgpu_xcp_get_inst_details(
+ &adev->xcp_mgr->xcp[fpriv->xcp_id],
+ AMDGPU_XCP_GFX, &xcc_mask);
+
/* We only have requirement to reserve vmid from gfxhub */
- r = amdgpu_vmid_alloc_spm_gfxhubs(adev, vm,
- adev->gfx.xcc_mask);
+ r = amdgpu_vmid_alloc_spm_gfxhubs(adev, vm, xcc_mask);
if (!r && !amdgpu_bo_reserve(vm->root.bo, true)) {
amdgpu_vm_spm_update(adev, vm);
amdgpu_bo_unreserve(vm->root.bo);
--
2.34.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [RFC PATCH 3/3] drm/amdgpu: scope AMDGPU_VM_OP_RESERVE_VMID to the caller's partition
2026-11-17 16:59 ` [RFC PATCH 3/3] drm/amdgpu: scope AMDGPU_VM_OP_RESERVE_VMID to the caller's partition James Zhu
@ 2026-11-17 17:18 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-11-17 17:18 UTC (permalink / raw)
To: James Zhu, amd-gfx
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] drm/amdgpu: skipped wait in amdgpu_vm_fini() for partition SPM
VMID
- [Medium] drm/amdgpu: unchecked amdgpu_xcp_get_inst_details() in
amdgpu_vm_ioctl()
--
commit a5f43653ba23f302ebd963bef5053a4d46508cd1
Author: James Zhu <James.Zhu@amd.com>
drm/amdgpu: scope AMDGPU_VM_OP_RESERVE_VMID to the caller's partition
This commit scopes AMDGPU_VM_OP_RESERVE_VMID to the GFXHUB instances of
the caller's compute partition when fpriv->xcp_id is set.
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index e1691145da903..8098fbb542c82 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -3021,11 +3021,21 @@ int amdgpu_vm_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
>
> switch (args->in.op) {
> case AMDGPU_VM_OP_RESERVE_VMID: {
> + uint32_t xcc_mask = adev->gfx.xcc_mask;
> int r;
>
> + /*
> + * Scope the reservation to the hardware partition behind the
> + * caller's render node, so other partitions can hold the
> + * reserved VMID on their own XCCs independently.
> + */
> + if (adev->xcp_mgr && fpriv->xcp_id != AMDGPU_XCP_NO_PARTITION)
> + amdgpu_xcp_get_inst_details(
> + &adev->xcp_mgr->xcp[fpriv->xcp_id],
> + AMDGPU_XCP_GFX, &xcc_mask);
[Severity: Medium]
Should the return value of amdgpu_xcp_get_inst_details() be checked here?
If a runtime compute partition switch reduces the number of partitions
while a DRM file opened on a higher-numbered partition remains open,
xcp[fpriv->xcp_id].valid becomes false and amdgpu_xcp_get_inst_details()
returns -EINVAL without updating xcc_mask. The reservation then silently
falls back to adev->gfx.xcc_mask and reserves the SPM VMID across every
GFXHUB on the device.
> +
> /* We only have requirement to reserve vmid from gfxhub */
> - r = amdgpu_vmid_alloc_spm_gfxhubs(adev, vm,
> - adev->gfx.xcc_mask);
> + r = amdgpu_vmid_alloc_spm_gfxhubs(adev, vm, xcc_mask);
[Severity: High]
When a process on a non-zero partition (fpriv->xcp_id > 0) closes its DRM
file without calling AMDGPU_VM_OP_UNRESERVE_VMID, does amdgpu_vm_fini()
skip waiting for in-flight jobs before unbinding and freeing the reserved
SPM VMID?
For fpriv->xcp_id > 0, xcc_mask only includes that partition's XCCs, so
vm->reserved_vmid[AMDGPU_GFXHUB(0)] stays NULL. Unlike
AMDGPU_VM_OP_UNRESERVE_VMID, amdgpu_vm_fini() only checks GFXHUB(0) before
waiting and then frees the SPM VMID on all hubs:
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c:amdgpu_vm_fini() {
...
/* Wait for the jobs which might still use the reserved VMID */
if (vm->reserved_vmid[AMDGPU_GFXHUB(0)])
dma_resv_wait_timeout(root->tbo.base.resv, DMA_RESV_USAGE_READ,
false, MAX_SCHEDULE_TIMEOUT);
...
for (i = 0; i < AMDGPU_MAX_VMHUBS; i++) {
amdgpu_vmid_free_spm(adev, vm, i);
}
...
}
amdgpu_vmid_free_spm() then unbinds spm_vmid to address 0 and clears
id_mgr->reserved_vmid while GPU jobs on that partition are still running
under spm_vmid, causing in-flight work to page-fault and letting another
process rebind the VMID concurrently. Checking all GFXHUBs in
adev->gfx.xcc_mask in amdgpu_vm_fini(), matching
AMDGPU_VM_OP_UNRESERVE_VMID, would avoid this regression.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261117165952.3558762-1-James.Zhu@amd.com?part=3
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC PATCH 2/3] drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance
2026-11-17 16:59 ` [RFC PATCH 2/3] drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance James Zhu
@ 2026-11-17 17:21 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-11-17 17:21 UTC (permalink / raw)
To: James Zhu, amd-gfx
Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] drm/amdgpu: physical xcc_mask used as logical index in
amdgpu_vm_spm_update()
- [High] drm/amdgpu: amdgpu_vm_fini() only checks GFXHUB(0) before
waiting on jobs
- [Medium] drm/amdgpu: unlocked allocation and unsafe rollback in
amdgpu_vmid_alloc_spm_gfxhubs()
--
commit 0df4e49e9ba90dc0c43a497872e17c100a8d4894
Author: James Zhu <James.Zhu@amd.com>
drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance
This commit updates SPM VMID reservation, binding, and unbinding to operate
per GFXHUB instance across XCCs rather than only on AMDGPU_GFXHUB(0).
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
> index 9c879ac683520..41f3dba530e38 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
[ ... ]
> +int amdgpu_vmid_alloc_spm_gfxhubs(struct amdgpu_device *adev,
> + struct amdgpu_vm *vm, uint32_t xcc_mask)
> +{
> + int i, r;
> +
> + for_each_inst(i, xcc_mask) {
> + r = amdgpu_vmid_alloc_spm(adev, vm, AMDGPU_GFXHUB(i));
> + if (r) {
> + amdgpu_vmid_free_spm_gfxhubs(adev, vm, xcc_mask);
> + return r;
> + }
> + }
> +
> + return 0;
> +}
[Severity: Medium]
If amdgpu_vmid_alloc_spm() fails on a later hub, can rolling back with
amdgpu_vmid_free_spm_gfxhubs(adev, vm, xcc_mask) unbind an active SPM VMID
to 0 while GPU jobs are still using it?
amdgpu_vmid_alloc_spm() returns 0 when vm->reserved_vmid[vmhub] is already
set, and AMDGPU_VM_OP_RESERVE_VMID calls amdgpu_vmid_alloc_spm_gfxhubs()
before reserving vm->root.bo. If vm already held a reservation on an
earlier hub, or a concurrent job submission grabs a newly published
vm->reserved_vmid[AMDGPU_GFXHUB(i)] before a subsequent hub fails with
-ENOENT, the rollback frees and unbinds those earlier hubs to 0 without
waiting via dma_resv_wait_timeout().
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index 94b1cfe05e3e0..e1691145da903 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -589,30 +589,36 @@ uint64_t amdgpu_vm_generation(struct amdgpu_device *adev, struct amdgpu_vm *vm)
> * The SPM VMID is fixed (adev->vm_manager.spm_vmid), so its page-table-base
> * register must track whichever VM currently holds the reservation instead
> * of being set per-job like a normal VMID. Bind it to @vm's real root PD
> - * address if @vm holds the reservation; otherwise do nothing, since @vm is
> - * not the one that could legitimately change the current binding.
> + * address on every GFXHUB @vm holds the reservation on; hubs reserved by
> + * other VMs or not reserved at all are left alone, since @vm is not the
> + * one that could legitimately change their binding.
> *
> * @vm's root PD must be locked by the caller.
> */
> void amdgpu_vm_spm_update(struct amdgpu_device *adev, struct amdgpu_vm *vm)
> {
> - struct amdgpu_kq_vmid_mgr *id_mgr =
> - &adev->vm_manager.id_mgr[AMDGPU_GFXHUB(0)];
> uint64_t pd_addr;
> + int i;
>
> if (!adev->vm_manager.spm_vmid || !adev->gfxhub.funcs)
> return;
>
> - mutex_lock(&id_mgr->lock);
> - if (vm->reserved_vmid[AMDGPU_GFXHUB(0)] !=
> - &id_mgr->ids[adev->vm_manager.spm_vmid] || !amdgpu_vm_ready(vm)) {
> - mutex_unlock(&id_mgr->lock);
> + if (!amdgpu_vm_ready(vm))
> return;
> - }
>
> pd_addr = amdgpu_gmc_pd_addr(vm->root.bo);
> - amdgpu_gmc_bind_vmid(adev, AMDGPU_GFXHUB(0), adev->vm_manager.spm_vmid, pd_addr);
> - mutex_unlock(&id_mgr->lock);
> + for_each_inst(i, adev->gfx.xcc_mask) {
> + struct amdgpu_kq_vmid_mgr *id_mgr =
> + &adev->vm_manager.id_mgr[AMDGPU_GFXHUB(i)];
> +
> + mutex_lock(&id_mgr->lock);
> + if (vm->reserved_vmid[AMDGPU_GFXHUB(i)] ==
> + &id_mgr->ids[adev->vm_manager.spm_vmid])
> + amdgpu_gmc_bind_vmid(adev, AMDGPU_GFXHUB(i),
> + adev->vm_manager.spm_vmid,
> + pd_addr);
> + mutex_unlock(&id_mgr->lock);
> + }
> }
[Severity: High]
Can iterating over adev->gfx.xcc_mask here and in amdgpu_vm_ioctl() index
adev->reg_offset[GC_HWIP] out of bounds when lower-numbered physical XCCs
are harvested?
adev->gfx.xcc_mask is a bitmask of physical XCC instances, whereas
AMDGPU_GFXHUB(i), adev->vmhub[], ring->vm_hub, and setup_vm_pt_regs_xcc()
all use contiguous logical XCC indices 0 .. NUM_XCC(adev->gfx.xcc_mask) - 1
and translate them via GET_INST(GC, i):
drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c:gfxhub_v1_2_xcc_setup_vm_pt_regs() {
for_each_inst(i, xcc_mask) {
hub = &adev->vmhub[AMDGPU_GFXHUB(i)];
WREG32_SOC15_OFFSET(GC, GET_INST(GC, i),
regVM_CONTEXT0_PAGE_TABLE_BASE_ADDR_LO32,
hub->ctx_addr_distance * vmid,
lower_32_bits(page_table_base));
...
}
When a lower physical XCC is harvested, amdgpu_populate_ip_map() sets
adev->ip_map.dev_inst[GC_HWIP][i] to -1 for i >= NUM_XCC(adev->gfx.xcc_mask).
Passing a physical bit position i >= NUM_XCC(adev->gfx.xcc_mask) through
amdgpu_gmc_bind_vmid() causes GET_INST(GC, i) to return 0xffffffff, reading
adev->reg_offset[GC_HWIP][0xffffffff] out of bounds and flushing an
uninitialized adev->vmhub[AMDGPU_GFXHUB(i)], while skipping the logical
GFXHUBs at the harvested bit positions. Using
GENMASK(NUM_XCC(adev->gfx.xcc_mask) - 1, 0) avoids mixing physical and
logical XCC masks.
[ ... ]
> @@ -3027,14 +3034,19 @@ int amdgpu_vm_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
> }
> case AMDGPU_VM_OP_UNRESERVE_VMID: {
> struct amdgpu_bo *root = vm->root.bo;
> + bool reserved = false;
> + int i;
>
> amdgpu_bo_reserve(root, true);
> + for_each_inst(i, adev->gfx.xcc_mask)
> + reserved |= amdgpu_vmid_uses_reserved(vm,
> + AMDGPU_GFXHUB(i));
> /* Wait for the jobs which might still use the reserved VMID */
> - if (vm->reserved_vmid[AMDGPU_GFXHUB(0)])
> + if (reserved)
> dma_resv_wait_timeout(root->tbo.base.resv,
> DMA_RESV_USAGE_READ, false,
> MAX_SCHEDULE_TIMEOUT);
> - amdgpu_vmid_free_spm(adev, vm, AMDGPU_GFXHUB(0));
> + amdgpu_vmid_free_spm_gfxhubs(adev, vm, adev->gfx.xcc_mask);
> amdgpu_bo_unreserve(root);
> break;
> }
[Severity: High]
Should amdgpu_vm_fini() also check all GFXHUBs before waiting on
root->tbo.base.resv, matching this change in AMDGPU_VM_OP_UNRESERVE_VMID?
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c:amdgpu_vm_fini() {
...
/* Wait for the jobs which might still use the reserved VMID */
if (vm->reserved_vmid[AMDGPU_GFXHUB(0)])
dma_resv_wait_timeout(root->tbo.base.resv, DMA_RESV_USAGE_READ,
false, MAX_SCHEDULE_TIMEOUT);
...
for (i = 0; i < AMDGPU_MAX_VMHUBS; i++) {
amdgpu_vmid_free_spm(adev, vm, i);
}
...
}
If a VM holds an SPM reservation on a non-zero GFXHUB while
vm->reserved_vmid[AMDGPU_GFXHUB(0)] is NULL (for example when reserved on a
subset of GFXHUBs via amdgpu_vmid_alloc_spm_gfxhubs()), closing the DRM file
skips dma_resv_wait_timeout(), frees the root PD in amdgpu_vm_pt_free_root(),
and unbinds the SPM VMID to 0 while jobs using that VMID are still running.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261117165952.3558762-1-James.Zhu@amd.com?part=2
^ permalink raw reply [flat|nested] 11+ messages in thread
* [RFC PATCH v2 0/3] drm/amdgpu: make the reserved SPM VMID per-partition
2026-11-17 16:59 [RFC PATCH 0/3] drm/amdgpu: make the reserved SPM VMID per-partition James Zhu
` (2 preceding siblings ...)
2026-11-17 16:59 ` [RFC PATCH 3/3] drm/amdgpu: scope AMDGPU_VM_OP_RESERVE_VMID to the caller's partition James Zhu
@ 2026-11-17 18:19 ` James Zhu
2026-11-17 18:19 ` [RFC PATCH v2 1/3] drm/amdgpu: expose the XCC-masked setup_vm_pt_regs as a gfxhub callback James Zhu
` (2 more replies)
3 siblings, 3 replies; 11+ messages in thread
From: James Zhu @ 2026-11-17 18:19 UTC (permalink / raw)
To: amd-gfx; +Cc: Christian König, Sunil Khatri, James Zhu
The reserved (SPM) VMID handling was designed on single-XCC hardware,
where one GFXHUB is all there is. On multi-XCC parts with partitioning
every XCC has an own GFXHUB and every partition is exposed as an own
render node, and the current code is incoherent in three ways at once:
1. AMDGPU_VM_OP_RESERVE_VMID records the reservation on
AMDGPU_GFXHUB(0) only. The VMID grab path checks
vm->reserved_vmid[ring->vm_hub], and the rings of XCC i use
AMDGPU_GFXHUB(i) - so for any job running on XCC 1..n the
reservation silently does nothing. A debugger relying on a stable
VMID gets one on exactly one XCC of the partition it runs on, or
none at all.
2. The physical programming goes the other way: the setup_vm_pt_regs
gfxhub callback broadcasts the SPM VMID's page table base to every
XCC of the device, including XCCs of partitions the caller cannot
even submit to.
3. The exclusion is device-global: because every reservation contends
for hub 0's id_mgr, one partition's render node blocks every other
partition from reserving, which breaks the isolation the render
nodes are supposed to provide.
None of this is visible on single-XCC hardware, where "hub 0", "all
hubs" and "the caller's hubs" happen to be the same thing. gfx 9.4.3
partition modes are where the three diverge.
This series makes the per-hub infrastructure, which already exists in
the data structures (vm->reserved_vmid[] and id_mgr[] are per-vmhub
arrays), real end to end:
Patch 1 exposes the XCC-masked setup_vm_pt_regs variant, which gfxhub
v1.2 and v12.1 already implement internally, as a gfxhub callback.
Patch 2 makes the hub argument of the binding path mean what it says:
amdgpu_gmc_bind_vmid() programs and flushes only the given hub,
amdgpu_vmid_free_spm() unbinds the hub it frees, amdgpu_vm_spm_update()
binds every hub the VM holds the reservation on, and the ioctl reserves
on every GFXHUB instance. This fixes 1. and 2. while keeping the
device-wide scope.
Patch 3 then derives the hub set from the caller's partition
(fpriv->xcp_id), fixing 3.: each partition reserves, binds and
conflicts only on its own XCCs. Unreserve keeps operating on every hub,
freeing whatever the VM actually holds.
A note on the record granularity: the reservation could instead be
recorded on the partition's first GFXHUB only and treated as a pure
ownership token, closest to the original single-hub design. But the
record plays three roles at once: the ownership token, the set of hubs
whose page table base has to be bound (each XCC translates through its
own hub, so the binding must cover every XCC of the partition either
way), and the per-hub check of the VMID grab and CS paths
(vm->reserved_vmid[ring->vm_hub]). Recording on every hub of the
partition keeps the three consistent by construction, lets
amdgpu_vm_spm_update() derive the binding set from the existing
structures without a new field, and gives jobs on every XCC of the
partition the same reserved-VMID semantics instead of only on the
first one. The VMID number itself is the same everywhere either way:
it is always adev->vm_manager.spm_vmid.
No UAPI change. On devices without partitioning, or on the whole-device
node, the mask degenerates to every XCC and nothing changes; on
single-XCC parts the series is a no-op.
Compile-tested on amd-staging-drm-next with CONFIG_DRM_AMDGPU=m;
checkpatch clean.
v2: address the Sashiko AI review findings, all of which checked out:
- Iterate the GFXHUBs with a logical XCC mask
(GENMASK(NUM_XCC(adev->gfx.xcc_mask) - 1, 0)) instead of the physical
adev->gfx.xcc_mask, which can be sparse when an XCC is harvested and
would send out-of-range logical indices into GET_INST()
(amdgpu_vm_spm_update(), both ioctl ops). The partition path already
used the logical xcp inst_mask.
- amdgpu_vm_fini() now checks every VMHUB, not just GFXHUB(0), before
waiting for jobs that may still use a reserved VMID; with patch 3 a
partition > 0 holds no reservation on hub 0 and the wait was skipped.
- amdgpu_vmid_alloc_spm_gfxhubs() rolls back only the hubs it newly
reserved, and waits on the root PD reservation first, honoring
amdgpu_vmid_free_spm()'s wait requirement; previously the rollback
could free pre-held hubs and unbind a VMID a just-submitted job uses.
- The return of amdgpu_xcp_get_inst_details() is checked; a stale
fpriv->xcp_id now fails the ioctl instead of silently reserving the
whole device.
James Zhu (3):
drm/amdgpu: expose the XCC-masked setup_vm_pt_regs as a gfxhub
callback
drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance
drm/amdgpu: scope AMDGPU_VM_OP_RESERVE_VMID to the caller's partition
drivers/gpu/drm/amd/amdgpu/amdgpu_gfxhub.h | 2 +
drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 4 ++
drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c | 68 ++++++++++++++++++++-
drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h | 4 ++
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 70 ++++++++++++++++------
drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c | 1 +
drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c | 1 +
7 files changed, 132 insertions(+), 18 deletions(-)
--
2.34.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* [RFC PATCH v2 1/3] drm/amdgpu: expose the XCC-masked setup_vm_pt_regs as a gfxhub callback
2026-11-17 18:19 ` [RFC PATCH v2 0/3] drm/amdgpu: make the reserved SPM VMID per-partition James Zhu
@ 2026-11-17 18:19 ` James Zhu
2026-11-17 18:19 ` [RFC PATCH v2 2/3] drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance James Zhu
2026-11-17 18:19 ` [RFC PATCH v2 3/3] drm/amdgpu: scope AMDGPU_VM_OP_RESERVE_VMID to the caller's partition James Zhu
2 siblings, 0 replies; 11+ messages in thread
From: James Zhu @ 2026-11-17 18:19 UTC (permalink / raw)
To: amd-gfx; +Cc: Christian König, Sunil Khatri, James Zhu
On GFX IPs with multiple XCCs each XCC has an own GFXHUB, but the
setup_vm_pt_regs gfxhub callback programs a VMID's page table base on
every XCC. Callers which bind a VMID on a single VMHUB, like the SPM
VMID binding, have no way to reach one GFXHUB through the callback
interface.
gfxhub v1.2 and v12.1 already implement an XCC-masked variant
internally. Expose it as a new setup_vm_pt_regs_xcc callback. GFXHUBs
of IPs with a single XCC don't implement it, for them the existing
callback already is per-hub.
Signed-off-by: James Zhu <James.Zhu@amd.com>
Assisted-by: Claude:claude-fable-5
---
drivers/gpu/drm/amd/amdgpu/amdgpu_gfxhub.h | 2 ++
drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c | 1 +
drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c | 1 +
3 files changed, 4 insertions(+)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfxhub.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfxhub.h
index c7b44aeb671b..19b961cb12f6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfxhub.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfxhub.h
@@ -28,6 +28,8 @@ struct amdgpu_gfxhub_funcs {
u64 (*get_mc_fb_offset)(struct amdgpu_device *adev);
void (*setup_vm_pt_regs)(struct amdgpu_device *adev, uint32_t vmid,
uint64_t page_table_base);
+ void (*setup_vm_pt_regs_xcc)(struct amdgpu_device *adev, uint32_t vmid,
+ uint64_t page_table_base, uint32_t xcc_mask);
int (*gart_enable)(struct amdgpu_device *adev);
void (*gart_disable)(struct amdgpu_device *adev);
diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c b/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c
index b13967a1c8fb..7bc17721f463 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c
@@ -809,6 +809,7 @@ const struct amdgpu_gfxhub_funcs gfxhub_v12_1_funcs = {
.get_fb_location = gfxhub_v12_1_get_fb_location,
.get_mc_fb_offset = gfxhub_v12_1_get_mc_fb_offset,
.setup_vm_pt_regs = gfxhub_v12_1_setup_vm_pt_regs,
+ .setup_vm_pt_regs_xcc = gfxhub_v12_1_xcc_setup_vm_pt_regs,
.gart_enable = gfxhub_v12_1_gart_enable,
.gart_disable = gfxhub_v12_1_gart_disable,
.set_fault_enable_default = gfxhub_v12_1_set_fault_enable_default,
diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c b/drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c
index e89813590539..a56c072daf15 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c
@@ -634,6 +634,7 @@ static int gfxhub_v1_2_get_xgmi_info(struct amdgpu_device *adev)
const struct amdgpu_gfxhub_funcs gfxhub_v1_2_funcs = {
.get_mc_fb_offset = gfxhub_v1_2_get_mc_fb_offset,
.setup_vm_pt_regs = gfxhub_v1_2_setup_vm_pt_regs,
+ .setup_vm_pt_regs_xcc = gfxhub_v1_2_xcc_setup_vm_pt_regs,
.gart_enable = gfxhub_v1_2_gart_enable,
.gart_disable = gfxhub_v1_2_gart_disable,
.set_fault_enable_default = gfxhub_v1_2_set_fault_enable_default,
--
2.34.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [RFC PATCH v2 2/3] drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance
2026-11-17 18:19 ` [RFC PATCH v2 0/3] drm/amdgpu: make the reserved SPM VMID per-partition James Zhu
2026-11-17 18:19 ` [RFC PATCH v2 1/3] drm/amdgpu: expose the XCC-masked setup_vm_pt_regs as a gfxhub callback James Zhu
@ 2026-11-17 18:19 ` James Zhu
2026-11-17 18:47 ` sashiko-bot
2026-11-17 18:19 ` [RFC PATCH v2 3/3] drm/amdgpu: scope AMDGPU_VM_OP_RESERVE_VMID to the caller's partition James Zhu
2 siblings, 1 reply; 11+ messages in thread
From: James Zhu @ 2026-11-17 18:19 UTC (permalink / raw)
To: amd-gfx; +Cc: Christian König, Sunil Khatri, James Zhu
The SPM VMID reservation made through AMDGPU_VM_OP_RESERVE_VMID is
recorded on AMDGPU_GFXHUB(0) only, while gfxhub v1.2's
setup_vm_pt_regs broadcasts the page table base to every XCC. On
multi-XCC parts the two disagree, in both directions:
- the reservation only takes effect for jobs running on XCC 0, since
the VMID grab path checks vm->reserved_vmid[ring->vm_hub] and the
rings of XCC i use AMDGPU_GFXHUB(i);
- the page table base is physically programmed on all XCCs anyway,
and conflicting reservations are only detected on hub 0's id_mgr.
Make the hub argument real end to end: amdgpu_gmc_bind_vmid() programs
only the given hub, through the XCC-masked gfxhub callback where one
exists, and amdgpu_vmid_free_spm() unbinds the hub it frees instead of
hub 0. amdgpu_vm_spm_update() binds every GFXHUB on which @vm holds
the reservation. The VM ioctl reserves the VMID on every GFXHUB
instance through the new amdgpu_vmid_alloc_spm_gfxhubs() helper, so
the reservation now takes effect, is bound and is flushed on every XCC
the VM's jobs can run on, and a conflicting reservation shows up on
whichever hub two VMs share.
Signed-off-by: James Zhu <James.Zhu@amd.com>
Assisted-by: Claude:claude-fable-5
---
drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 4 ++
drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c | 68 ++++++++++++++++++++++++-
drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h | 4 ++
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 56 +++++++++++++-------
4 files changed, 114 insertions(+), 18 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
index 08d3ff61edcc..704a30ad388a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
@@ -797,6 +797,7 @@ void amdgpu_gmc_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid,
*
* For VMIDs that are bound directly to a VM's page table outside the
* normal per-job VM flush path, e.g. a dedicated out-of-pool VMID.
+ * Only @vmhub is programmed and flushed, other hubs keep their binding.
*/
void amdgpu_gmc_bind_vmid(struct amdgpu_device *adev, unsigned int vmhub,
unsigned int vmid, uint64_t pd_addr)
@@ -805,6 +806,9 @@ void amdgpu_gmc_bind_vmid(struct amdgpu_device *adev, unsigned int vmhub,
if (is_mmhub)
adev->mmhub.funcs->setup_vm_pt_regs(adev, vmid, pd_addr);
+ else if (adev->gfxhub.funcs->setup_vm_pt_regs_xcc)
+ adev->gfxhub.funcs->setup_vm_pt_regs_xcc(adev, vmid, pd_addr,
+ BIT(vmhub - AMDGPU_GFXHUB_START));
else
adev->gfxhub.funcs->setup_vm_pt_regs(adev, vmid, pd_addr);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
index 9c879ac68352..a4403e5cc344 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
@@ -651,12 +651,78 @@ void amdgpu_vmid_free_spm(struct amdgpu_device *adev, struct amdgpu_vm *vm,
id_mgr->reserved_vmid = false;
if (adev->vm_manager.spm_vmid && adev->gfxhub.funcs)
- amdgpu_gmc_bind_vmid(adev, AMDGPU_GFXHUB(0),
+ amdgpu_gmc_bind_vmid(adev, vmhub,
adev->vm_manager.spm_vmid, 0);
}
mutex_unlock(&id_mgr->lock);
}
+/*
+ * amdgpu_vmid_alloc_spm_gfxhubs - reserve the SPM VMID on a set of GFXHUBs
+ * @adev: amdgpu device structure
+ * @vm: the VM to reserve the ID for
+ * @xcc_mask: XCC instances on whose GFXHUBs to reserve
+ *
+ * Reserve the SPM VMID on the GFXHUB of every XCC in @xcc_mask, so that
+ * the reservation covers every hub the VM's jobs can run on.
+ *
+ * Returns: 0 for success, the first per-hub error otherwise. On an error
+ * only the hubs this call reserved are released again, after waiting for
+ * the jobs which might have grabbed the reserved VMID in the meantime;
+ * hubs @vm already held before the call stay reserved.
+ */
+int amdgpu_vmid_alloc_spm_gfxhubs(struct amdgpu_device *adev,
+ struct amdgpu_vm *vm, uint32_t xcc_mask)
+{
+ uint32_t allocated = 0;
+ int i, r;
+
+ for_each_inst(i, xcc_mask) {
+ bool held = amdgpu_vmid_uses_reserved(vm, AMDGPU_GFXHUB(i));
+
+ r = amdgpu_vmid_alloc_spm(adev, vm, AMDGPU_GFXHUB(i));
+ if (r)
+ goto error_free;
+ if (!held)
+ allocated |= BIT(i);
+ }
+
+ return 0;
+
+error_free:
+ /*
+ * A job may already have grabbed a freshly reserved VMID, give it
+ * the wait amdgpu_vmid_free_spm() demands before the rollback
+ * unbinds the VMID again.
+ */
+ if (allocated) {
+ dma_resv_wait_timeout(vm->root.bo->tbo.base.resv,
+ DMA_RESV_USAGE_READ, false,
+ MAX_SCHEDULE_TIMEOUT);
+ amdgpu_vmid_free_spm_gfxhubs(adev, vm, allocated);
+ }
+ return r;
+}
+
+/*
+ * amdgpu_vmid_free_spm_gfxhubs - undo amdgpu_vmid_alloc_spm_gfxhubs
+ * @adev: amdgpu device structure
+ * @vm: the VM with the reserved ID
+ * @xcc_mask: XCC instances on whose GFXHUBs to free the reservation
+ *
+ * Free the SPM VMID on every GFXHUB in @xcc_mask @vm holds it on, hubs
+ * without a reservation are skipped. The same wait requirement as for
+ * amdgpu_vmid_free_spm() applies.
+ */
+void amdgpu_vmid_free_spm_gfxhubs(struct amdgpu_device *adev,
+ struct amdgpu_vm *vm, uint32_t xcc_mask)
+{
+ int i;
+
+ for_each_inst(i, xcc_mask)
+ amdgpu_vmid_free_spm(adev, vm, AMDGPU_GFXHUB(i));
+}
+
/**
* amdgpu_vmid_reset - reset VMID to zero
*
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h
index c134000eecfd..d2b225d0aa38 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h
@@ -129,6 +129,10 @@ int amdgpu_vmid_alloc_spm(struct amdgpu_device *adev, struct amdgpu_vm *vm,
unsigned vmhub);
void amdgpu_vmid_free_spm(struct amdgpu_device *adev, struct amdgpu_vm *vm,
unsigned vmhub);
+int amdgpu_vmid_alloc_spm_gfxhubs(struct amdgpu_device *adev,
+ struct amdgpu_vm *vm, uint32_t xcc_mask);
+void amdgpu_vmid_free_spm_gfxhubs(struct amdgpu_device *adev,
+ struct amdgpu_vm *vm, uint32_t xcc_mask);
int amdgpu_vmid_grab(struct amdgpu_vm *vm, struct amdgpu_ring *ring,
struct amdgpu_job *job, struct dma_fence **fence);
void amdgpu_vmid_reset(struct amdgpu_device *adev, unsigned vmhub,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 6b4dc0276e27..a9c23759d058 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -589,30 +589,38 @@ uint64_t amdgpu_vm_generation(struct amdgpu_device *adev, struct amdgpu_vm *vm)
* The SPM VMID is fixed (adev->vm_manager.spm_vmid), so its page-table-base
* register must track whichever VM currently holds the reservation instead
* of being set per-job like a normal VMID. Bind it to @vm's real root PD
- * address if @vm holds the reservation; otherwise do nothing, since @vm is
- * not the one that could legitimately change the current binding.
+ * address on every GFXHUB @vm holds the reservation on; hubs reserved by
+ * other VMs or not reserved at all are left alone, since @vm is not the
+ * one that could legitimately change their binding.
*
* @vm's root PD must be locked by the caller.
*/
void amdgpu_vm_spm_update(struct amdgpu_device *adev, struct amdgpu_vm *vm)
{
- struct amdgpu_kq_vmid_mgr *id_mgr =
- &adev->vm_manager.id_mgr[AMDGPU_GFXHUB(0)];
+ /* the GFXHUBs are indexed by logical XCC, adev->gfx.xcc_mask is physical */
+ uint32_t gfxhub_mask = GENMASK(NUM_XCC(adev->gfx.xcc_mask) - 1, 0);
uint64_t pd_addr;
+ int i;
if (!adev->vm_manager.spm_vmid || !adev->gfxhub.funcs)
return;
- mutex_lock(&id_mgr->lock);
- if (vm->reserved_vmid[AMDGPU_GFXHUB(0)] !=
- &id_mgr->ids[adev->vm_manager.spm_vmid] || !amdgpu_vm_ready(vm)) {
- mutex_unlock(&id_mgr->lock);
+ if (!amdgpu_vm_ready(vm))
return;
- }
pd_addr = amdgpu_gmc_pd_addr(vm->root.bo);
- amdgpu_gmc_bind_vmid(adev, AMDGPU_GFXHUB(0), adev->vm_manager.spm_vmid, pd_addr);
- mutex_unlock(&id_mgr->lock);
+ for_each_inst(i, gfxhub_mask) {
+ struct amdgpu_kq_vmid_mgr *id_mgr =
+ &adev->vm_manager.id_mgr[AMDGPU_GFXHUB(i)];
+
+ mutex_lock(&id_mgr->lock);
+ if (vm->reserved_vmid[AMDGPU_GFXHUB(i)] ==
+ &id_mgr->ids[adev->vm_manager.spm_vmid])
+ amdgpu_gmc_bind_vmid(adev, AMDGPU_GFXHUB(i),
+ adev->vm_manager.spm_vmid,
+ pd_addr);
+ mutex_unlock(&id_mgr->lock);
+ }
}
/**
@@ -2871,9 +2879,14 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
root = amdgpu_bo_ref(vm->root.bo);
amdgpu_bo_reserve(root, true);
/* Wait for the jobs which might still use the reserved VMID */
- if (vm->reserved_vmid[AMDGPU_GFXHUB(0)])
- dma_resv_wait_timeout(root->tbo.base.resv, DMA_RESV_USAGE_READ,
- false, MAX_SCHEDULE_TIMEOUT);
+ for (i = 0; i < AMDGPU_MAX_VMHUBS; i++) {
+ if (vm->reserved_vmid[i]) {
+ dma_resv_wait_timeout(root->tbo.base.resv,
+ DMA_RESV_USAGE_READ,
+ false, MAX_SCHEDULE_TIMEOUT);
+ break;
+ }
+ }
dma_fence_wait(vm->last_unlocked, false);
dma_fence_put(vm->last_unlocked);
dma_fence_wait(vm->last_seq_flush, false);
@@ -3015,10 +3028,12 @@ int amdgpu_vm_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
switch (args->in.op) {
case AMDGPU_VM_OP_RESERVE_VMID: {
+ /* logical XCCs, the GFXHUB index space; xcc_mask is physical */
+ uint32_t gfxhub_mask = GENMASK(NUM_XCC(adev->gfx.xcc_mask) - 1, 0);
int r;
/* We only have requirement to reserve vmid from gfxhub */
- r = amdgpu_vmid_alloc_spm(adev, vm, AMDGPU_GFXHUB(0));
+ r = amdgpu_vmid_alloc_spm_gfxhubs(adev, vm, gfxhub_mask);
if (!r && !amdgpu_bo_reserve(vm->root.bo, true)) {
amdgpu_vm_spm_update(adev, vm);
amdgpu_bo_unreserve(vm->root.bo);
@@ -3026,15 +3041,22 @@ int amdgpu_vm_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
return r;
}
case AMDGPU_VM_OP_UNRESERVE_VMID: {
+ /* logical XCCs, the GFXHUB index space; xcc_mask is physical */
+ uint32_t gfxhub_mask = GENMASK(NUM_XCC(adev->gfx.xcc_mask) - 1, 0);
struct amdgpu_bo *root = vm->root.bo;
+ bool reserved = false;
+ int i;
amdgpu_bo_reserve(root, true);
+ for_each_inst(i, gfxhub_mask)
+ reserved |= amdgpu_vmid_uses_reserved(vm,
+ AMDGPU_GFXHUB(i));
/* Wait for the jobs which might still use the reserved VMID */
- if (vm->reserved_vmid[AMDGPU_GFXHUB(0)])
+ if (reserved)
dma_resv_wait_timeout(root->tbo.base.resv,
DMA_RESV_USAGE_READ, false,
MAX_SCHEDULE_TIMEOUT);
- amdgpu_vmid_free_spm(adev, vm, AMDGPU_GFXHUB(0));
+ amdgpu_vmid_free_spm_gfxhubs(adev, vm, gfxhub_mask);
amdgpu_bo_unreserve(root);
break;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [RFC PATCH v2 3/3] drm/amdgpu: scope AMDGPU_VM_OP_RESERVE_VMID to the caller's partition
2026-11-17 18:19 ` [RFC PATCH v2 0/3] drm/amdgpu: make the reserved SPM VMID per-partition James Zhu
2026-11-17 18:19 ` [RFC PATCH v2 1/3] drm/amdgpu: expose the XCC-masked setup_vm_pt_regs as a gfxhub callback James Zhu
2026-11-17 18:19 ` [RFC PATCH v2 2/3] drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance James Zhu
@ 2026-11-17 18:19 ` James Zhu
2 siblings, 0 replies; 11+ messages in thread
From: James Zhu @ 2026-11-17 18:19 UTC (permalink / raw)
To: amd-gfx; +Cc: Christian König, Sunil Khatri, James Zhu
On partitioned devices every partition is exposed as an own render
node and fpriv->xcp_id records which partition a drm file belongs to.
But AMDGPU_VM_OP_RESERVE_VMID reserves the VMID on every GFXHUB of the
device: a reservation made on one partition's render node blocks every
other partition from reserving, and rebinds the VMID's page table base
on XCCs the caller can't even submit to.
Derive the GFXHUB set from the caller's partition instead. The
reservation then only covers the XCCs of the caller's partition, and
each partition can hold the reserved VMID on its own hubs
independently, matching the isolation the render nodes promise.
Unreserve keeps operating on every hub, freeing whatever the VM
actually holds.
Without partitioning, or on the whole-device node, the mask stays
adev->gfx.xcc_mask and nothing changes.
Signed-off-by: James Zhu <James.Zhu@amd.com>
Assisted-by: Claude:claude-fable-5
---
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index a9c23759d058..403807cb615d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -3032,6 +3032,20 @@ int amdgpu_vm_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
uint32_t gfxhub_mask = GENMASK(NUM_XCC(adev->gfx.xcc_mask) - 1, 0);
int r;
+ /*
+ * Scope the reservation to the hardware partition behind the
+ * caller's render node, so other partitions can hold the
+ * reserved VMID on their own XCCs independently. The
+ * partition's inst_mask is logical like the GFXHUB indices.
+ */
+ if (adev->xcp_mgr && fpriv->xcp_id != AMDGPU_XCP_NO_PARTITION) {
+ r = amdgpu_xcp_get_inst_details(
+ &adev->xcp_mgr->xcp[fpriv->xcp_id],
+ AMDGPU_XCP_GFX, &gfxhub_mask);
+ if (r)
+ return r;
+ }
+
/* We only have requirement to reserve vmid from gfxhub */
r = amdgpu_vmid_alloc_spm_gfxhubs(adev, vm, gfxhub_mask);
if (!r && !amdgpu_bo_reserve(vm->root.bo, true)) {
--
2.34.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [RFC PATCH v2 2/3] drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance
2026-11-17 18:19 ` [RFC PATCH v2 2/3] drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance James Zhu
@ 2026-11-17 18:47 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-11-17 18:47 UTC (permalink / raw)
To: James Zhu, amd-gfx
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] drm/amdgpu: missing root PD lock in
amdgpu_vmid_alloc_spm_gfxhubs()
--
commit 4216e97db26fd6fc889673c452d2434887bc13af
Author: James Zhu <James.Zhu@amd.com>
drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance
This commit reserves, binds, and unbinds the SPM VMID on every GFXHUB
instance rather than only on AMDGPU_GFXHUB(0).
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
> index 9c879ac683520..a4403e5cc3443 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
> @@ -651,12 +651,78 @@ void amdgpu_vmid_free_spm(struct amdgpu_device *adev, struct amdgpu_vm *vm,
> id_mgr->reserved_vmid = false;
>
> if (adev->vm_manager.spm_vmid && adev->gfxhub.funcs)
> - amdgpu_gmc_bind_vmid(adev, AMDGPU_GFXHUB(0),
> + amdgpu_gmc_bind_vmid(adev, vmhub,
> adev->vm_manager.spm_vmid, 0);
> }
> mutex_unlock(&id_mgr->lock);
> }
>
> +/*
> + * amdgpu_vmid_alloc_spm_gfxhubs - reserve the SPM VMID on a set of GFXHUBs
> + * @adev: amdgpu device structure
> + * @vm: the VM to reserve the ID for
> + * @xcc_mask: XCC instances on whose GFXHUBs to reserve
> + *
> + * Reserve the SPM VMID on the GFXHUB of every XCC in @xcc_mask, so that
> + * the reservation covers every hub the VM's jobs can run on.
> + *
> + * Returns: 0 for success, the first per-hub error otherwise. On an error
> + * only the hubs this call reserved are released again, after waiting for
> + * the jobs which might have grabbed the reserved VMID in the meantime;
> + * hubs @vm already held before the call stay reserved.
> + */
> +int amdgpu_vmid_alloc_spm_gfxhubs(struct amdgpu_device *adev,
> + struct amdgpu_vm *vm, uint32_t xcc_mask)
> +{
> + uint32_t allocated = 0;
> + int i, r;
> +
> + for_each_inst(i, xcc_mask) {
> + bool held = amdgpu_vmid_uses_reserved(vm, AMDGPU_GFXHUB(i));
> +
> + r = amdgpu_vmid_alloc_spm(adev, vm, AMDGPU_GFXHUB(i));
> + if (r)
> + goto error_free;
> + if (!held)
> + allocated |= BIT(i);
> + }
> +
> + return 0;
> +
> +error_free:
> + /*
> + * A job may already have grabbed a freshly reserved VMID, give it
> + * the wait amdgpu_vmid_free_spm() demands before the rollback
> + * unbinds the VMID again.
> + */
> + if (allocated) {
> + dma_resv_wait_timeout(vm->root.bo->tbo.base.resv,
> + DMA_RESV_USAGE_READ, false,
> + MAX_SCHEDULE_TIMEOUT);
> + amdgpu_vmid_free_spm_gfxhubs(adev, vm, allocated);
> + }
> + return r;
> +}
[ ... ]
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index 94b1cfe05e3e0..f64020162a4bf 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
[ ... ]
> @@ -3015,10 +3028,12 @@ int amdgpu_vm_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
>
> switch (args->in.op) {
> case AMDGPU_VM_OP_RESERVE_VMID: {
> + /* logical XCCs, the GFXHUB index space; xcc_mask is physical */
> + uint32_t gfxhub_mask = GENMASK(NUM_XCC(adev->gfx.xcc_mask) - 1, 0);
> int r;
>
> /* We only have requirement to reserve vmid from gfxhub */
> - r = amdgpu_vmid_alloc_spm(adev, vm, AMDGPU_GFXHUB(0));
> + r = amdgpu_vmid_alloc_spm_gfxhubs(adev, vm, gfxhub_mask);
> if (!r && !amdgpu_bo_reserve(vm->root.bo, true)) {
> amdgpu_vm_spm_update(adev, vm);
> amdgpu_bo_unreserve(vm->root.bo);
> }
> return r;
> }
[Severity: Medium]
Can the error rollback in amdgpu_vmid_alloc_spm_gfxhubs() unbind a newly
reserved SPM VMID while a concurrent CS job is still using it?
Unlike AMDGPU_VM_OP_UNRESERVE_VMID, AMDGPU_VM_OP_RESERVE_VMID calls
amdgpu_vmid_alloc_spm_gfxhubs() without holding the vm->root.bo
reservation lock:
CPU0 CPU1
amdgpu_vm_ioctl(RESERVE_VMID)
amdgpu_vmid_alloc_spm_gfxhubs()
amdgpu_vmid_alloc_spm(hub 0)
vm->reserved_vmid[hub 0] = id
amdgpu_vmid_alloc_spm(hub 1)
-> -ENOENT
dma_resv_wait_timeout(root)
amdgpu_cs_ioctl()
/* reserves root, adds fence */
amdgpu_job_prepare_job()
amdgpu_vmid_grab()
/* grabs reserved VMID on hub 0 */
amdgpu_vmid_free_spm(hub 0)
amdgpu_gmc_bind_vmid(..., 0) <-- unbinds VMID under active job
Because vm->root.bo is not reserved across the wait and free, a command
submission on the same VM can attach a new fence and grab the reserved
VMID on hub 0 after dma_resv_wait_timeout() returns and right before
amdgpu_vmid_free_spm() unbinds its page-table-base register to 0, causing
GPU page faults for the in-flight job.
Running without vm->root.bo reserved also leaves held =
amdgpu_vmid_uses_reserved() unsynchronized: amdgpu_vmid_alloc_spm()
returns 0 when vm->reserved_vmid[vmhub] is already set, so two concurrent
AMDGPU_VM_OP_RESERVE_VMID calls on the same VM can both observe held ==
false on hub 0 and set allocated |= BIT(0), causing a failure on a later
hub in one thread to free the reservation held by the other. Similarly,
concurrent RESERVE_VMID and UNRESERVE_VMID calls can interleave across
hubs, and if amdgpu_bo_reserve(vm->root.bo, true) fails after
amdgpu_vmid_alloc_spm_gfxhubs() succeeds, amdgpu_vm_spm_update() is
skipped while returning 0.
Reserving vm->root.bo in AMDGPU_VM_OP_RESERVE_VMID before calling
amdgpu_vmid_alloc_spm_gfxhubs() and holding it across
amdgpu_vm_spm_update() (matching AMDGPU_VM_OP_UNRESERVE_VMID) would
serialize per-VM reservation state and prevent new CS jobs during rollback.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261117181941.3968996-1-James.Zhu@amd.com?part=2
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2026-11-17 18:48 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-17 16:59 [RFC PATCH 0/3] drm/amdgpu: make the reserved SPM VMID per-partition James Zhu
2026-11-17 16:59 ` [RFC PATCH 1/3] drm/amdgpu: expose the XCC-masked setup_vm_pt_regs as a gfxhub callback James Zhu
2026-11-17 16:59 ` [RFC PATCH 2/3] drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance James Zhu
2026-11-17 17:21 ` sashiko-bot
2026-11-17 16:59 ` [RFC PATCH 3/3] drm/amdgpu: scope AMDGPU_VM_OP_RESERVE_VMID to the caller's partition James Zhu
2026-11-17 17:18 ` sashiko-bot
2026-11-17 18:19 ` [RFC PATCH v2 0/3] drm/amdgpu: make the reserved SPM VMID per-partition James Zhu
2026-11-17 18:19 ` [RFC PATCH v2 1/3] drm/amdgpu: expose the XCC-masked setup_vm_pt_regs as a gfxhub callback James Zhu
2026-11-17 18:19 ` [RFC PATCH v2 2/3] drm/amdgpu: reserve and bind the SPM VMID on every GFXHUB instance James Zhu
2026-11-17 18:47 ` sashiko-bot
2026-11-17 18:19 ` [RFC PATCH v2 3/3] drm/amdgpu: scope AMDGPU_VM_OP_RESERVE_VMID to the caller's partition James Zhu
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.