* [PATCH 01/11] drm/amdgpu: add parameter to disable kernel queues
2025-03-05 20:47 [PATCH 00/11] Add disable kernel queue support Alex Deucher
@ 2025-03-05 20:47 ` Alex Deucher
2025-03-06 7:37 ` Khatri, Sunil
2025-03-05 20:47 ` [PATCH 02/11] drm/amdgpu: add ring flag for no user submissions Alex Deucher
` (9 subsequent siblings)
10 siblings, 1 reply; 30+ messages in thread
From: Alex Deucher @ 2025-03-05 20:47 UTC (permalink / raw)
To: amd-gfx; +Cc: Alex Deucher
On chips that support user queues, setting this option
will disable kernel queues to be used to validate
user queues without kernel queues.
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 +
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++
2 files changed, 10 insertions(+)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 87062c1adcdf7..45437a8f29d3b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -270,6 +270,7 @@ extern int amdgpu_user_partt_mode;
extern int amdgpu_agp;
extern int amdgpu_wbrf;
+extern int amdgpu_disable_kq;
#define AMDGPU_VM_MAX_NUM_CTX 4096
#define AMDGPU_SG_THRESHOLD (256*1024*1024)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index b161daa900198..42a7619592ab9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -237,6 +237,7 @@ int amdgpu_agp = -1; /* auto */
int amdgpu_wbrf = -1;
int amdgpu_damage_clips = -1; /* auto */
int amdgpu_umsch_mm_fwlog;
+int amdgpu_disable_kq = -1;
DECLARE_DYNDBG_CLASSMAP(drm_debug_classes, DD_CLASS_TYPE_DISJOINT_BITS, 0,
"DRM_UT_CORE",
@@ -1083,6 +1084,14 @@ MODULE_PARM_DESC(wbrf,
"Enable Wifi RFI interference mitigation (0 = disabled, 1 = enabled, -1 = auto(default)");
module_param_named(wbrf, amdgpu_wbrf, int, 0444);
+/**
+ * DOC: disable_kq (int)
+ * Disable kernel queues on systems that support user queues.
+ * (0 = kernel queues enabled, 1 = kernel queues disabled, -1 = auto (default setting))
+ */
+MODULE_PARM_DESC(disable_kq, "Disable kernel queues (-1 = auto (default), 0 = enable KQ, 1 = disable KQ)");
+module_param_named(disable_kq, amdgpu_disable_kq, int, 0444);
+
/* These devices are not supported by amdgpu.
* They are supported by the mach64, r128, radeon drivers
*/
--
2.48.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH 01/11] drm/amdgpu: add parameter to disable kernel queues
2025-03-05 20:47 ` [PATCH 01/11] drm/amdgpu: add parameter to disable kernel queues Alex Deucher
@ 2025-03-06 7:37 ` Khatri, Sunil
0 siblings, 0 replies; 30+ messages in thread
From: Khatri, Sunil @ 2025-03-06 7:37 UTC (permalink / raw)
To: Alex Deucher, amd-gfx
Reviewed-by: Sunil Khatri <sunil.khatri@amd.com>
On 3/6/2025 2:17 AM, Alex Deucher wrote:
> On chips that support user queues, setting this option
> will disable kernel queues to be used to validate
> user queues without kernel queues.
>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 +
> drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++
> 2 files changed, 10 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> index 87062c1adcdf7..45437a8f29d3b 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> @@ -270,6 +270,7 @@ extern int amdgpu_user_partt_mode;
> extern int amdgpu_agp;
>
> extern int amdgpu_wbrf;
> +extern int amdgpu_disable_kq;
>
> #define AMDGPU_VM_MAX_NUM_CTX 4096
> #define AMDGPU_SG_THRESHOLD (256*1024*1024)
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index b161daa900198..42a7619592ab9 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -237,6 +237,7 @@ int amdgpu_agp = -1; /* auto */
> int amdgpu_wbrf = -1;
> int amdgpu_damage_clips = -1; /* auto */
> int amdgpu_umsch_mm_fwlog;
> +int amdgpu_disable_kq = -1;
>
> DECLARE_DYNDBG_CLASSMAP(drm_debug_classes, DD_CLASS_TYPE_DISJOINT_BITS, 0,
> "DRM_UT_CORE",
> @@ -1083,6 +1084,14 @@ MODULE_PARM_DESC(wbrf,
> "Enable Wifi RFI interference mitigation (0 = disabled, 1 = enabled, -1 = auto(default)");
> module_param_named(wbrf, amdgpu_wbrf, int, 0444);
>
> +/**
> + * DOC: disable_kq (int)
> + * Disable kernel queues on systems that support user queues.
> + * (0 = kernel queues enabled, 1 = kernel queues disabled, -1 = auto (default setting))
> + */
> +MODULE_PARM_DESC(disable_kq, "Disable kernel queues (-1 = auto (default), 0 = enable KQ, 1 = disable KQ)");
> +module_param_named(disable_kq, amdgpu_disable_kq, int, 0444);
> +
> /* These devices are not supported by amdgpu.
> * They are supported by the mach64, r128, radeon drivers
> */
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 02/11] drm/amdgpu: add ring flag for no user submissions
2025-03-05 20:47 [PATCH 00/11] Add disable kernel queue support Alex Deucher
2025-03-05 20:47 ` [PATCH 01/11] drm/amdgpu: add parameter to disable kernel queues Alex Deucher
@ 2025-03-05 20:47 ` Alex Deucher
2025-03-06 7:48 ` Khatri, Sunil
2025-03-05 20:47 ` [PATCH 03/11] drm/amdgpu/gfx: add generic handling for disable_kq Alex Deucher
` (8 subsequent siblings)
10 siblings, 1 reply; 30+ messages in thread
From: Alex Deucher @ 2025-03-05 20:47 UTC (permalink / raw)
To: amd-gfx; +Cc: Alex Deucher
This would be set by IPs which only accept submissions
from the kernel, not userspace, such as when kernel
queues are disabled. Don't expose the rings to userspace
and reject any submissions in the CS IOCTL.
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 4 ++++
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 30 ++++++++++++++++--------
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h | 2 +-
3 files changed, 25 insertions(+), 11 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
index 5df21529b3b13..5cc18034b75df 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
@@ -349,6 +349,10 @@ static int amdgpu_cs_p2_ib(struct amdgpu_cs_parser *p,
ring = amdgpu_job_ring(job);
ib = &job->ibs[job->num_ibs++];
+ /* submissions to kernel queus are disabled */
+ if (ring->no_user_submission)
+ return -EINVAL;
+
/* MM engine doesn't support user fences */
if (p->uf_bo && ring->funcs->no_user_fence)
return -EINVAL;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
index cd6eb7a3bc58a..3b7dfd56ccd0e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
@@ -408,7 +408,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
case AMDGPU_HW_IP_GFX:
type = AMD_IP_BLOCK_TYPE_GFX;
for (i = 0; i < adev->gfx.num_gfx_rings; i++)
- if (adev->gfx.gfx_ring[i].sched.ready)
+ if (adev->gfx.gfx_ring[i].sched.ready &&
+ !adev->gfx.gfx_ring[i].no_user_submission)
++num_rings;
ib_start_alignment = 32;
ib_size_alignment = 32;
@@ -416,7 +417,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
case AMDGPU_HW_IP_COMPUTE:
type = AMD_IP_BLOCK_TYPE_GFX;
for (i = 0; i < adev->gfx.num_compute_rings; i++)
- if (adev->gfx.compute_ring[i].sched.ready)
+ if (adev->gfx.compute_ring[i].sched.ready &&
+ !adev->gfx.compute_ring[i].no_user_submission)
++num_rings;
ib_start_alignment = 32;
ib_size_alignment = 32;
@@ -424,7 +426,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
case AMDGPU_HW_IP_DMA:
type = AMD_IP_BLOCK_TYPE_SDMA;
for (i = 0; i < adev->sdma.num_instances; i++)
- if (adev->sdma.instance[i].ring.sched.ready)
+ if (adev->sdma.instance[i].ring.sched.ready &&
+ !adev->gfx.gfx_ring[i].no_user_submission)
++num_rings;
ib_start_alignment = 256;
ib_size_alignment = 4;
@@ -435,7 +438,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
if (adev->uvd.harvest_config & (1 << i))
continue;
- if (adev->uvd.inst[i].ring.sched.ready)
+ if (adev->uvd.inst[i].ring.sched.ready &&
+ !adev->uvd.inst[i].ring.no_user_submission)
++num_rings;
}
ib_start_alignment = 256;
@@ -444,7 +448,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
case AMDGPU_HW_IP_VCE:
type = AMD_IP_BLOCK_TYPE_VCE;
for (i = 0; i < adev->vce.num_rings; i++)
- if (adev->vce.ring[i].sched.ready)
+ if (adev->vce.ring[i].sched.ready &&
+ !adev->vce.ring[i].no_user_submission)
++num_rings;
ib_start_alignment = 256;
ib_size_alignment = 4;
@@ -456,7 +461,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
continue;
for (j = 0; j < adev->uvd.num_enc_rings; j++)
- if (adev->uvd.inst[i].ring_enc[j].sched.ready)
+ if (adev->uvd.inst[i].ring_enc[j].sched.ready &&
+ !adev->uvd.inst[i].ring_enc[j].no_user_submission)
++num_rings;
}
ib_start_alignment = 256;
@@ -468,7 +474,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
if (adev->vcn.harvest_config & (1 << i))
continue;
- if (adev->vcn.inst[i].ring_dec.sched.ready)
+ if (adev->vcn.inst[i].ring_dec.sched.ready &&
+ !adev->vcn.inst[i].ring_dec.no_user_submission)
++num_rings;
}
ib_start_alignment = 256;
@@ -481,7 +488,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
continue;
for (j = 0; j < adev->vcn.inst[i].num_enc_rings; j++)
- if (adev->vcn.inst[i].ring_enc[j].sched.ready)
+ if (adev->vcn.inst[i].ring_enc[j].sched.ready &&
+ !adev->vcn.inst[i].ring_enc[j].no_user_submission)
++num_rings;
}
ib_start_alignment = 256;
@@ -496,7 +504,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
continue;
for (j = 0; j < adev->jpeg.num_jpeg_rings; j++)
- if (adev->jpeg.inst[i].ring_dec[j].sched.ready)
+ if (adev->jpeg.inst[i].ring_dec[j].sched.ready &&
+ !adev->jpeg.inst[i].ring_dec[j].no_user_submission)
++num_rings;
}
ib_start_alignment = 256;
@@ -504,7 +513,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
break;
case AMDGPU_HW_IP_VPE:
type = AMD_IP_BLOCK_TYPE_VPE;
- if (adev->vpe.ring.sched.ready)
+ if (adev->vpe.ring.sched.ready &&
+ !adev->vpe.ring.no_user_submission)
++num_rings;
ib_start_alignment = 256;
ib_size_alignment = 4;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
index b4fd1e17205e9..4a97afcb38b78 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
@@ -297,6 +297,7 @@ struct amdgpu_ring {
struct dma_fence *vmid_wait;
bool has_compute_vm_bug;
bool no_scheduler;
+ bool no_user_submission;
int hw_prio;
unsigned num_hw_submission;
atomic_t *sched_score;
@@ -310,7 +311,6 @@ struct amdgpu_ring {
unsigned int entry_index;
/* store the cached rptr to restore after reset */
uint64_t cached_rptr;
-
};
#define amdgpu_ring_parse_cs(r, p, job, ib) ((r)->funcs->parse_cs((p), (job), (ib)))
--
2.48.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH 02/11] drm/amdgpu: add ring flag for no user submissions
2025-03-05 20:47 ` [PATCH 02/11] drm/amdgpu: add ring flag for no user submissions Alex Deucher
@ 2025-03-06 7:48 ` Khatri, Sunil
2025-03-06 16:00 ` Alex Deucher
0 siblings, 1 reply; 30+ messages in thread
From: Khatri, Sunil @ 2025-03-06 7:48 UTC (permalink / raw)
To: Alex Deucher, amd-gfx
On 3/6/2025 2:17 AM, Alex Deucher wrote:
> This would be set by IPs which only accept submissions
> from the kernel, not userspace, such as when kernel
> queues are disabled. Don't expose the rings to userspace
> and reject any submissions in the CS IOCTL.
>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 4 ++++
> drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 30 ++++++++++++++++--------
> drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h | 2 +-
> 3 files changed, 25 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
> index 5df21529b3b13..5cc18034b75df 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
> @@ -349,6 +349,10 @@ static int amdgpu_cs_p2_ib(struct amdgpu_cs_parser *p,
> ring = amdgpu_job_ring(job);
> ib = &job->ibs[job->num_ibs++];
>
> + /* submissions to kernel queus are disabled */
> + if (ring->no_user_submission)
> + return -EINVAL;
> +
Alex, dont we need to disable submission for other type of chunks, from
function amdgpu_cs_pass2 ? chunk type of bo list and fences are
submitted by users and we do want to handle that too.
If that's not needed then lgtm.
> /* MM engine doesn't support user fences */
> if (p->uf_bo && ring->funcs->no_user_fence)
> return -EINVAL;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> index cd6eb7a3bc58a..3b7dfd56ccd0e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> @@ -408,7 +408,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> case AMDGPU_HW_IP_GFX:
> type = AMD_IP_BLOCK_TYPE_GFX;
> for (i = 0; i < adev->gfx.num_gfx_rings; i++)
> - if (adev->gfx.gfx_ring[i].sched.ready)
> + if (adev->gfx.gfx_ring[i].sched.ready &&
> + !adev->gfx.gfx_ring[i].no_user_submission)
> ++num_rings;
> ib_start_alignment = 32;
> ib_size_alignment = 32;
> @@ -416,7 +417,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> case AMDGPU_HW_IP_COMPUTE:
> type = AMD_IP_BLOCK_TYPE_GFX;
> for (i = 0; i < adev->gfx.num_compute_rings; i++)
> - if (adev->gfx.compute_ring[i].sched.ready)
> + if (adev->gfx.compute_ring[i].sched.ready &&
> + !adev->gfx.compute_ring[i].no_user_submission)
> ++num_rings;
> ib_start_alignment = 32;
> ib_size_alignment = 32;
> @@ -424,7 +426,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> case AMDGPU_HW_IP_DMA:
> type = AMD_IP_BLOCK_TYPE_SDMA;
> for (i = 0; i < adev->sdma.num_instances; i++)
> - if (adev->sdma.instance[i].ring.sched.ready)
> + if (adev->sdma.instance[i].ring.sched.ready &&
> + !adev->gfx.gfx_ring[i].no_user_submission)
> ++num_rings;
> ib_start_alignment = 256;
> ib_size_alignment = 4;
> @@ -435,7 +438,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> if (adev->uvd.harvest_config & (1 << i))
> continue;
>
> - if (adev->uvd.inst[i].ring.sched.ready)
> + if (adev->uvd.inst[i].ring.sched.ready &&
> + !adev->uvd.inst[i].ring.no_user_submission)
> ++num_rings;
> }
> ib_start_alignment = 256;
> @@ -444,7 +448,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> case AMDGPU_HW_IP_VCE:
> type = AMD_IP_BLOCK_TYPE_VCE;
> for (i = 0; i < adev->vce.num_rings; i++)
> - if (adev->vce.ring[i].sched.ready)
> + if (adev->vce.ring[i].sched.ready &&
> + !adev->vce.ring[i].no_user_submission)
> ++num_rings;
> ib_start_alignment = 256;
> ib_size_alignment = 4;
> @@ -456,7 +461,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> continue;
>
> for (j = 0; j < adev->uvd.num_enc_rings; j++)
> - if (adev->uvd.inst[i].ring_enc[j].sched.ready)
> + if (adev->uvd.inst[i].ring_enc[j].sched.ready &&
> + !adev->uvd.inst[i].ring_enc[j].no_user_submission)
> ++num_rings;
> }
> ib_start_alignment = 256;
> @@ -468,7 +474,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> if (adev->vcn.harvest_config & (1 << i))
> continue;
>
> - if (adev->vcn.inst[i].ring_dec.sched.ready)
> + if (adev->vcn.inst[i].ring_dec.sched.ready &&
> + !adev->vcn.inst[i].ring_dec.no_user_submission)
> ++num_rings;
> }
> ib_start_alignment = 256;
> @@ -481,7 +488,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> continue;
>
> for (j = 0; j < adev->vcn.inst[i].num_enc_rings; j++)
> - if (adev->vcn.inst[i].ring_enc[j].sched.ready)
> + if (adev->vcn.inst[i].ring_enc[j].sched.ready &&
> + !adev->vcn.inst[i].ring_enc[j].no_user_submission)
> ++num_rings;
> }
> ib_start_alignment = 256;
> @@ -496,7 +504,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> continue;
>
> for (j = 0; j < adev->jpeg.num_jpeg_rings; j++)
> - if (adev->jpeg.inst[i].ring_dec[j].sched.ready)
> + if (adev->jpeg.inst[i].ring_dec[j].sched.ready &&
> + !adev->jpeg.inst[i].ring_dec[j].no_user_submission)
> ++num_rings;
> }
> ib_start_alignment = 256;
> @@ -504,7 +513,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> break;
> case AMDGPU_HW_IP_VPE:
> type = AMD_IP_BLOCK_TYPE_VPE;
> - if (adev->vpe.ring.sched.ready)
> + if (adev->vpe.ring.sched.ready &&
> + !adev->vpe.ring.no_user_submission)
> ++num_rings;
> ib_start_alignment = 256;
> ib_size_alignment = 4;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> index b4fd1e17205e9..4a97afcb38b78 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> @@ -297,6 +297,7 @@ struct amdgpu_ring {
> struct dma_fence *vmid_wait;
> bool has_compute_vm_bug;
> bool no_scheduler;
> + bool no_user_submission;
> int hw_prio;
> unsigned num_hw_submission;
> atomic_t *sched_score;
> @@ -310,7 +311,6 @@ struct amdgpu_ring {
> unsigned int entry_index;
> /* store the cached rptr to restore after reset */
> uint64_t cached_rptr;
> -
> };
>
> #define amdgpu_ring_parse_cs(r, p, job, ib) ((r)->funcs->parse_cs((p), (job), (ib)))
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [PATCH 02/11] drm/amdgpu: add ring flag for no user submissions
2025-03-06 7:48 ` Khatri, Sunil
@ 2025-03-06 16:00 ` Alex Deucher
0 siblings, 0 replies; 30+ messages in thread
From: Alex Deucher @ 2025-03-06 16:00 UTC (permalink / raw)
To: Khatri, Sunil; +Cc: Alex Deucher, amd-gfx
On Thu, Mar 6, 2025 at 2:48 AM Khatri, Sunil <sukhatri@amd.com> wrote:
>
>
> On 3/6/2025 2:17 AM, Alex Deucher wrote:
> > This would be set by IPs which only accept submissions
> > from the kernel, not userspace, such as when kernel
> > queues are disabled. Don't expose the rings to userspace
> > and reject any submissions in the CS IOCTL.
> >
> > Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> > ---
> > drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 4 ++++
> > drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 30 ++++++++++++++++--------
> > drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h | 2 +-
> > 3 files changed, 25 insertions(+), 11 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
> > index 5df21529b3b13..5cc18034b75df 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
> > @@ -349,6 +349,10 @@ static int amdgpu_cs_p2_ib(struct amdgpu_cs_parser *p,
> > ring = amdgpu_job_ring(job);
> > ib = &job->ibs[job->num_ibs++];
> >
> > + /* submissions to kernel queus are disabled */
> > + if (ring->no_user_submission)
> > + return -EINVAL;
> > +
> Alex, dont we need to disable submission for other type of chunks, from
> function amdgpu_cs_pass2 ? chunk type of bo list and fences are
> submitted by users and we do want to handle that too.
> If that's not needed then lgtm.
That is the only chunk type which directly involves the kernel rings.
Alex
> > /* MM engine doesn't support user fences */
> > if (p->uf_bo && ring->funcs->no_user_fence)
> > return -EINVAL;
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> > index cd6eb7a3bc58a..3b7dfd56ccd0e 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> > @@ -408,7 +408,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> > case AMDGPU_HW_IP_GFX:
> > type = AMD_IP_BLOCK_TYPE_GFX;
> > for (i = 0; i < adev->gfx.num_gfx_rings; i++)
> > - if (adev->gfx.gfx_ring[i].sched.ready)
> > + if (adev->gfx.gfx_ring[i].sched.ready &&
> > + !adev->gfx.gfx_ring[i].no_user_submission)
> > ++num_rings;
> > ib_start_alignment = 32;
> > ib_size_alignment = 32;
> > @@ -416,7 +417,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> > case AMDGPU_HW_IP_COMPUTE:
> > type = AMD_IP_BLOCK_TYPE_GFX;
> > for (i = 0; i < adev->gfx.num_compute_rings; i++)
> > - if (adev->gfx.compute_ring[i].sched.ready)
> > + if (adev->gfx.compute_ring[i].sched.ready &&
> > + !adev->gfx.compute_ring[i].no_user_submission)
> > ++num_rings;
> > ib_start_alignment = 32;
> > ib_size_alignment = 32;
> > @@ -424,7 +426,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> > case AMDGPU_HW_IP_DMA:
> > type = AMD_IP_BLOCK_TYPE_SDMA;
> > for (i = 0; i < adev->sdma.num_instances; i++)
> > - if (adev->sdma.instance[i].ring.sched.ready)
> > + if (adev->sdma.instance[i].ring.sched.ready &&
> > + !adev->gfx.gfx_ring[i].no_user_submission)
> > ++num_rings;
> > ib_start_alignment = 256;
> > ib_size_alignment = 4;
> > @@ -435,7 +438,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> > if (adev->uvd.harvest_config & (1 << i))
> > continue;
> >
> > - if (adev->uvd.inst[i].ring.sched.ready)
> > + if (adev->uvd.inst[i].ring.sched.ready &&
> > + !adev->uvd.inst[i].ring.no_user_submission)
> > ++num_rings;
> > }
> > ib_start_alignment = 256;
> > @@ -444,7 +448,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> > case AMDGPU_HW_IP_VCE:
> > type = AMD_IP_BLOCK_TYPE_VCE;
> > for (i = 0; i < adev->vce.num_rings; i++)
> > - if (adev->vce.ring[i].sched.ready)
> > + if (adev->vce.ring[i].sched.ready &&
> > + !adev->vce.ring[i].no_user_submission)
> > ++num_rings;
> > ib_start_alignment = 256;
> > ib_size_alignment = 4;
> > @@ -456,7 +461,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> > continue;
> >
> > for (j = 0; j < adev->uvd.num_enc_rings; j++)
> > - if (adev->uvd.inst[i].ring_enc[j].sched.ready)
> > + if (adev->uvd.inst[i].ring_enc[j].sched.ready &&
> > + !adev->uvd.inst[i].ring_enc[j].no_user_submission)
> > ++num_rings;
> > }
> > ib_start_alignment = 256;
> > @@ -468,7 +474,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> > if (adev->vcn.harvest_config & (1 << i))
> > continue;
> >
> > - if (adev->vcn.inst[i].ring_dec.sched.ready)
> > + if (adev->vcn.inst[i].ring_dec.sched.ready &&
> > + !adev->vcn.inst[i].ring_dec.no_user_submission)
> > ++num_rings;
> > }
> > ib_start_alignment = 256;
> > @@ -481,7 +488,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> > continue;
> >
> > for (j = 0; j < adev->vcn.inst[i].num_enc_rings; j++)
> > - if (adev->vcn.inst[i].ring_enc[j].sched.ready)
> > + if (adev->vcn.inst[i].ring_enc[j].sched.ready &&
> > + !adev->vcn.inst[i].ring_enc[j].no_user_submission)
> > ++num_rings;
> > }
> > ib_start_alignment = 256;
> > @@ -496,7 +504,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> > continue;
> >
> > for (j = 0; j < adev->jpeg.num_jpeg_rings; j++)
> > - if (adev->jpeg.inst[i].ring_dec[j].sched.ready)
> > + if (adev->jpeg.inst[i].ring_dec[j].sched.ready &&
> > + !adev->jpeg.inst[i].ring_dec[j].no_user_submission)
> > ++num_rings;
> > }
> > ib_start_alignment = 256;
> > @@ -504,7 +513,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
> > break;
> > case AMDGPU_HW_IP_VPE:
> > type = AMD_IP_BLOCK_TYPE_VPE;
> > - if (adev->vpe.ring.sched.ready)
> > + if (adev->vpe.ring.sched.ready &&
> > + !adev->vpe.ring.no_user_submission)
> > ++num_rings;
> > ib_start_alignment = 256;
> > ib_size_alignment = 4;
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> > index b4fd1e17205e9..4a97afcb38b78 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> > @@ -297,6 +297,7 @@ struct amdgpu_ring {
> > struct dma_fence *vmid_wait;
> > bool has_compute_vm_bug;
> > bool no_scheduler;
> > + bool no_user_submission;
> > int hw_prio;
> > unsigned num_hw_submission;
> > atomic_t *sched_score;
> > @@ -310,7 +311,6 @@ struct amdgpu_ring {
> > unsigned int entry_index;
> > /* store the cached rptr to restore after reset */
> > uint64_t cached_rptr;
> > -
> > };
> >
> > #define amdgpu_ring_parse_cs(r, p, job, ib) ((r)->funcs->parse_cs((p), (job), (ib)))
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 03/11] drm/amdgpu/gfx: add generic handling for disable_kq
2025-03-05 20:47 [PATCH 00/11] Add disable kernel queue support Alex Deucher
2025-03-05 20:47 ` [PATCH 01/11] drm/amdgpu: add parameter to disable kernel queues Alex Deucher
2025-03-05 20:47 ` [PATCH 02/11] drm/amdgpu: add ring flag for no user submissions Alex Deucher
@ 2025-03-05 20:47 ` Alex Deucher
2025-03-06 1:06 ` Felix Kuehling
2025-03-05 20:47 ` [PATCH 04/11] drm/amdgpu/mes: centralize gfx_hqd mask management Alex Deucher
` (7 subsequent siblings)
10 siblings, 1 reply; 30+ messages in thread
From: Alex Deucher @ 2025-03-05 20:47 UTC (permalink / raw)
To: amd-gfx; +Cc: Alex Deucher
Add proper checks for disable_kq functionality in
gfx helper functions. Add special logic for families
that require the clear state setup.
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c | 92 +++++++++++++++++--------
drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h | 2 +
2 files changed, 67 insertions(+), 27 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
index a194bf3347cbc..af3f8b62f6fd5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
@@ -371,6 +371,18 @@ int amdgpu_gfx_kiq_init(struct amdgpu_device *adev,
return 0;
}
+static bool amdgpu_gfx_disable_gfx_kq(struct amdgpu_device *adev)
+{
+ if (adev->gfx.disable_kq) {
+ /* GFX11 needs the GFX ring for clear buffer */
+ if (amdgpu_ip_version(adev, GC_HWIP, 0) <= IP_VERSION(12, 0, 0))
+ return false;
+ else
+ return true;
+ }
+ return false;
+}
+
/* create MQD for each compute/gfx queue */
int amdgpu_gfx_mqd_sw_init(struct amdgpu_device *adev,
unsigned int mqd_size, int xcc_id)
@@ -379,6 +391,7 @@ int amdgpu_gfx_mqd_sw_init(struct amdgpu_device *adev,
struct amdgpu_kiq *kiq = &adev->gfx.kiq[xcc_id];
struct amdgpu_ring *ring = &kiq->ring;
u32 domain = AMDGPU_GEM_DOMAIN_GTT;
+ bool disable_kq_gfx = amdgpu_gfx_disable_gfx_kq(adev);
#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
/* Only enable on gfx10 and 11 for now to avoid changing behavior on older chips */
@@ -413,7 +426,8 @@ int amdgpu_gfx_mqd_sw_init(struct amdgpu_device *adev,
}
}
- if (adev->asic_type >= CHIP_NAVI10 && amdgpu_async_gfx_ring) {
+ if (adev->asic_type >= CHIP_NAVI10 && amdgpu_async_gfx_ring &&
+ !disable_kq_gfx) {
/* create MQD for each KGQ */
for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
ring = &adev->gfx.gfx_ring[i];
@@ -437,25 +451,28 @@ int amdgpu_gfx_mqd_sw_init(struct amdgpu_device *adev,
}
}
- /* create MQD for each KCQ */
- for (i = 0; i < adev->gfx.num_compute_rings; i++) {
- j = i + xcc_id * adev->gfx.num_compute_rings;
- ring = &adev->gfx.compute_ring[j];
- if (!ring->mqd_obj) {
- r = amdgpu_bo_create_kernel(adev, mqd_size, PAGE_SIZE,
- domain, &ring->mqd_obj,
- &ring->mqd_gpu_addr, &ring->mqd_ptr);
- if (r) {
- dev_warn(adev->dev, "failed to create ring mqd bo (%d)", r);
- return r;
- }
+ if (!adev->gfx.disable_kq) {
+ /* create MQD for each KCQ */
+ for (i = 0; i < adev->gfx.num_compute_rings; i++) {
+ j = i + xcc_id * adev->gfx.num_compute_rings;
+ ring = &adev->gfx.compute_ring[j];
+ if (!ring->mqd_obj) {
+ r = amdgpu_bo_create_kernel(adev, mqd_size, PAGE_SIZE,
+ domain, &ring->mqd_obj,
+ &ring->mqd_gpu_addr, &ring->mqd_ptr);
+ if (r) {
+ dev_warn(adev->dev, "failed to create ring mqd bo (%d)", r);
+ return r;
+ }
- ring->mqd_size = mqd_size;
- /* prepare MQD backup */
- adev->gfx.mec.mqd_backup[j] = kzalloc(mqd_size, GFP_KERNEL);
- if (!adev->gfx.mec.mqd_backup[j]) {
- dev_warn(adev->dev, "no memory to create MQD backup for ring %s\n", ring->name);
- return -ENOMEM;
+ ring->mqd_size = mqd_size;
+ /* prepare MQD backup */
+ adev->gfx.mec.mqd_backup[j] = kzalloc(mqd_size, GFP_KERNEL);
+ if (!adev->gfx.mec.mqd_backup[j]) {
+ dev_warn(adev->dev, "no memory to create MQD backup for ring %s\n",
+ ring->name);
+ return -ENOMEM;
+ }
}
}
}
@@ -468,8 +485,10 @@ void amdgpu_gfx_mqd_sw_fini(struct amdgpu_device *adev, int xcc_id)
struct amdgpu_ring *ring = NULL;
int i, j;
struct amdgpu_kiq *kiq = &adev->gfx.kiq[xcc_id];
+ bool disable_kq_gfx = amdgpu_gfx_disable_gfx_kq(adev);
- if (adev->asic_type >= CHIP_NAVI10 && amdgpu_async_gfx_ring) {
+ if (adev->asic_type >= CHIP_NAVI10 && amdgpu_async_gfx_ring &&
+ !disable_kq_gfx) {
for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
ring = &adev->gfx.gfx_ring[i];
kfree(adev->gfx.me.mqd_backup[i]);
@@ -479,13 +498,15 @@ void amdgpu_gfx_mqd_sw_fini(struct amdgpu_device *adev, int xcc_id)
}
}
- for (i = 0; i < adev->gfx.num_compute_rings; i++) {
- j = i + xcc_id * adev->gfx.num_compute_rings;
- ring = &adev->gfx.compute_ring[j];
- kfree(adev->gfx.mec.mqd_backup[j]);
- amdgpu_bo_free_kernel(&ring->mqd_obj,
- &ring->mqd_gpu_addr,
- &ring->mqd_ptr);
+ if (!adev->gfx.disable_kq) {
+ for (i = 0; i < adev->gfx.num_compute_rings; i++) {
+ j = i + xcc_id * adev->gfx.num_compute_rings;
+ ring = &adev->gfx.compute_ring[j];
+ kfree(adev->gfx.mec.mqd_backup[j]);
+ amdgpu_bo_free_kernel(&ring->mqd_obj,
+ &ring->mqd_gpu_addr,
+ &ring->mqd_ptr);
+ }
}
ring = &kiq->ring;
@@ -502,6 +523,9 @@ int amdgpu_gfx_disable_kcq(struct amdgpu_device *adev, int xcc_id)
int i, r = 0;
int j;
+ if (adev->gfx.disable_kq)
+ return 0;
+
if (adev->enable_mes) {
for (i = 0; i < adev->gfx.num_compute_rings; i++) {
j = i + xcc_id * adev->gfx.num_compute_rings;
@@ -547,11 +571,15 @@ int amdgpu_gfx_disable_kcq(struct amdgpu_device *adev, int xcc_id)
int amdgpu_gfx_disable_kgq(struct amdgpu_device *adev, int xcc_id)
{
+ bool disable_kq_gfx = amdgpu_gfx_disable_gfx_kq(adev);
struct amdgpu_kiq *kiq = &adev->gfx.kiq[xcc_id];
struct amdgpu_ring *kiq_ring = &kiq->ring;
int i, r = 0;
int j;
+ if (disable_kq_gfx)
+ return 0;
+
if (adev->enable_mes) {
if (amdgpu_gfx_is_master_xcc(adev, xcc_id)) {
for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
@@ -657,6 +685,9 @@ int amdgpu_gfx_enable_kcq(struct amdgpu_device *adev, int xcc_id)
uint64_t queue_mask = 0;
int r, i, j;
+ if (adev->gfx.disable_kq)
+ return 0;
+
if (adev->mes.enable_legacy_queue_map)
return amdgpu_gfx_mes_enable_kcq(adev, xcc_id);
@@ -716,10 +747,14 @@ int amdgpu_gfx_enable_kcq(struct amdgpu_device *adev, int xcc_id)
int amdgpu_gfx_enable_kgq(struct amdgpu_device *adev, int xcc_id)
{
+ bool disable_kq_gfx = amdgpu_gfx_disable_gfx_kq(adev);
struct amdgpu_kiq *kiq = &adev->gfx.kiq[xcc_id];
struct amdgpu_ring *kiq_ring = &kiq->ring;
int r, i, j;
+ if (disable_kq_gfx)
+ return 0;
+
if (!kiq->pmf || !kiq->pmf->kiq_map_queues)
return -EINVAL;
@@ -1544,6 +1579,9 @@ static ssize_t amdgpu_gfx_set_run_cleaner_shader(struct device *dev,
if (adev->in_suspend && !adev->in_runpm)
return -EPERM;
+ if (adev->gfx.disable_kq)
+ return -ENOTSUPP;
+
ret = kstrtol(buf, 0, &value);
if (ret)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
index ddf4533614bac..8fa68a4ac34f1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
@@ -483,6 +483,8 @@ struct amdgpu_gfx {
atomic_t total_submission_cnt;
struct delayed_work idle_work;
+
+ bool disable_kq;
};
struct amdgpu_gfx_ras_reg_entry {
--
2.48.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH 03/11] drm/amdgpu/gfx: add generic handling for disable_kq
2025-03-05 20:47 ` [PATCH 03/11] drm/amdgpu/gfx: add generic handling for disable_kq Alex Deucher
@ 2025-03-06 1:06 ` Felix Kuehling
2025-03-06 9:57 ` Khatri, Sunil
0 siblings, 1 reply; 30+ messages in thread
From: Felix Kuehling @ 2025-03-06 1:06 UTC (permalink / raw)
To: amd-gfx, Deucher, Alexander
On 2025-03-05 15:47, Alex Deucher wrote:
> Add proper checks for disable_kq functionality in
> gfx helper functions. Add special logic for families
> that require the clear state setup.
>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c | 92 +++++++++++++++++--------
> drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h | 2 +
> 2 files changed, 67 insertions(+), 27 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
> index a194bf3347cbc..af3f8b62f6fd5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
> @@ -371,6 +371,18 @@ int amdgpu_gfx_kiq_init(struct amdgpu_device *adev,
> return 0;
> }
>
> +static bool amdgpu_gfx_disable_gfx_kq(struct amdgpu_device *adev)
> +{
> + if (adev->gfx.disable_kq) {
> + /* GFX11 needs the GFX ring for clear buffer */
> + if (amdgpu_ip_version(adev, GC_HWIP, 0) <= IP_VERSION(12, 0, 0))
Should this be < instead of <=?
Regards,
Felix
> + return false;
> + else
> + return true;
> + }
> + return false;
> +}
> +
> /* create MQD for each compute/gfx queue */
> int amdgpu_gfx_mqd_sw_init(struct amdgpu_device *adev,
> unsigned int mqd_size, int xcc_id)
> @@ -379,6 +391,7 @@ int amdgpu_gfx_mqd_sw_init(struct amdgpu_device *adev,
> struct amdgpu_kiq *kiq = &adev->gfx.kiq[xcc_id];
> struct amdgpu_ring *ring = &kiq->ring;
> u32 domain = AMDGPU_GEM_DOMAIN_GTT;
> + bool disable_kq_gfx = amdgpu_gfx_disable_gfx_kq(adev);
>
> #if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
> /* Only enable on gfx10 and 11 for now to avoid changing behavior on older chips */
> @@ -413,7 +426,8 @@ int amdgpu_gfx_mqd_sw_init(struct amdgpu_device *adev,
> }
> }
>
> - if (adev->asic_type >= CHIP_NAVI10 && amdgpu_async_gfx_ring) {
> + if (adev->asic_type >= CHIP_NAVI10 && amdgpu_async_gfx_ring &&
> + !disable_kq_gfx) {
> /* create MQD for each KGQ */
> for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
> ring = &adev->gfx.gfx_ring[i];
> @@ -437,25 +451,28 @@ int amdgpu_gfx_mqd_sw_init(struct amdgpu_device *adev,
> }
> }
>
> - /* create MQD for each KCQ */
> - for (i = 0; i < adev->gfx.num_compute_rings; i++) {
> - j = i + xcc_id * adev->gfx.num_compute_rings;
> - ring = &adev->gfx.compute_ring[j];
> - if (!ring->mqd_obj) {
> - r = amdgpu_bo_create_kernel(adev, mqd_size, PAGE_SIZE,
> - domain, &ring->mqd_obj,
> - &ring->mqd_gpu_addr, &ring->mqd_ptr);
> - if (r) {
> - dev_warn(adev->dev, "failed to create ring mqd bo (%d)", r);
> - return r;
> - }
> + if (!adev->gfx.disable_kq) {
Maybe just set adev->gfx.num_compute_rings to 0 somewhere, then you
don't need this condition.
> + /* create MQD for each KCQ */
> + for (i = 0; i < adev->gfx.num_compute_rings; i++) {
> + j = i + xcc_id * adev->gfx.num_compute_rings;
> + ring = &adev->gfx.compute_ring[j];
> + if (!ring->mqd_obj) {
> + r = amdgpu_bo_create_kernel(adev, mqd_size, PAGE_SIZE,
> + domain, &ring->mqd_obj,
> + &ring->mqd_gpu_addr, &ring->mqd_ptr);
> + if (r) {
> + dev_warn(adev->dev, "failed to create ring mqd bo (%d)", r);
> + return r;
> + }
>
> - ring->mqd_size = mqd_size;
> - /* prepare MQD backup */
> - adev->gfx.mec.mqd_backup[j] = kzalloc(mqd_size, GFP_KERNEL);
> - if (!adev->gfx.mec.mqd_backup[j]) {
> - dev_warn(adev->dev, "no memory to create MQD backup for ring %s\n", ring->name);
> - return -ENOMEM;
> + ring->mqd_size = mqd_size;
> + /* prepare MQD backup */
> + adev->gfx.mec.mqd_backup[j] = kzalloc(mqd_size, GFP_KERNEL);
> + if (!adev->gfx.mec.mqd_backup[j]) {
> + dev_warn(adev->dev, "no memory to create MQD backup for ring %s\n",
> + ring->name);
> + return -ENOMEM;
> + }
> }
> }
> }
> @@ -468,8 +485,10 @@ void amdgpu_gfx_mqd_sw_fini(struct amdgpu_device *adev, int xcc_id)
> struct amdgpu_ring *ring = NULL;
> int i, j;
> struct amdgpu_kiq *kiq = &adev->gfx.kiq[xcc_id];
> + bool disable_kq_gfx = amdgpu_gfx_disable_gfx_kq(adev);
>
> - if (adev->asic_type >= CHIP_NAVI10 && amdgpu_async_gfx_ring) {
> + if (adev->asic_type >= CHIP_NAVI10 && amdgpu_async_gfx_ring &&
> + !disable_kq_gfx) {
> for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
> ring = &adev->gfx.gfx_ring[i];
> kfree(adev->gfx.me.mqd_backup[i]);
> @@ -479,13 +498,15 @@ void amdgpu_gfx_mqd_sw_fini(struct amdgpu_device *adev, int xcc_id)
> }
> }
>
> - for (i = 0; i < adev->gfx.num_compute_rings; i++) {
> - j = i + xcc_id * adev->gfx.num_compute_rings;
> - ring = &adev->gfx.compute_ring[j];
> - kfree(adev->gfx.mec.mqd_backup[j]);
> - amdgpu_bo_free_kernel(&ring->mqd_obj,
> - &ring->mqd_gpu_addr,
> - &ring->mqd_ptr);
> + if (!adev->gfx.disable_kq) {
Same as above.
> + for (i = 0; i < adev->gfx.num_compute_rings; i++) {
> + j = i + xcc_id * adev->gfx.num_compute_rings;
> + ring = &adev->gfx.compute_ring[j];
> + kfree(adev->gfx.mec.mqd_backup[j]);
> + amdgpu_bo_free_kernel(&ring->mqd_obj,
> + &ring->mqd_gpu_addr,
> + &ring->mqd_ptr);
> + }
> }
>
> ring = &kiq->ring;
> @@ -502,6 +523,9 @@ int amdgpu_gfx_disable_kcq(struct amdgpu_device *adev, int xcc_id)
> int i, r = 0;
> int j;
>
> + if (adev->gfx.disable_kq)
Same as above.
> + return 0;
> +
> if (adev->enable_mes) {
> for (i = 0; i < adev->gfx.num_compute_rings; i++) {
> j = i + xcc_id * adev->gfx.num_compute_rings;
> @@ -547,11 +571,15 @@ int amdgpu_gfx_disable_kcq(struct amdgpu_device *adev, int xcc_id)
>
> int amdgpu_gfx_disable_kgq(struct amdgpu_device *adev, int xcc_id)
> {
> + bool disable_kq_gfx = amdgpu_gfx_disable_gfx_kq(adev);
> struct amdgpu_kiq *kiq = &adev->gfx.kiq[xcc_id];
> struct amdgpu_ring *kiq_ring = &kiq->ring;
> int i, r = 0;
> int j;
>
> + if (disable_kq_gfx)
> + return 0;
Maybe just set adev->gfx.num_gfx_rings to 0 somewhere, then you don't
need this condition.
Regards,
Felix
> +
> if (adev->enable_mes) {
> if (amdgpu_gfx_is_master_xcc(adev, xcc_id)) {
> for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
> @@ -657,6 +685,9 @@ int amdgpu_gfx_enable_kcq(struct amdgpu_device *adev, int xcc_id)
> uint64_t queue_mask = 0;
> int r, i, j;
>
> + if (adev->gfx.disable_kq)
> + return 0;
> +
> if (adev->mes.enable_legacy_queue_map)
> return amdgpu_gfx_mes_enable_kcq(adev, xcc_id);
>
> @@ -716,10 +747,14 @@ int amdgpu_gfx_enable_kcq(struct amdgpu_device *adev, int xcc_id)
>
> int amdgpu_gfx_enable_kgq(struct amdgpu_device *adev, int xcc_id)
> {
> + bool disable_kq_gfx = amdgpu_gfx_disable_gfx_kq(adev);
> struct amdgpu_kiq *kiq = &adev->gfx.kiq[xcc_id];
> struct amdgpu_ring *kiq_ring = &kiq->ring;
> int r, i, j;
>
> + if (disable_kq_gfx)
> + return 0;
> +
> if (!kiq->pmf || !kiq->pmf->kiq_map_queues)
> return -EINVAL;
>
> @@ -1544,6 +1579,9 @@ static ssize_t amdgpu_gfx_set_run_cleaner_shader(struct device *dev,
> if (adev->in_suspend && !adev->in_runpm)
> return -EPERM;
>
> + if (adev->gfx.disable_kq)
> + return -ENOTSUPP;
> +
> ret = kstrtol(buf, 0, &value);
>
> if (ret)
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
> index ddf4533614bac..8fa68a4ac34f1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
> @@ -483,6 +483,8 @@ struct amdgpu_gfx {
>
> atomic_t total_submission_cnt;
> struct delayed_work idle_work;
> +
> + bool disable_kq;
> };
>
> struct amdgpu_gfx_ras_reg_entry {
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [PATCH 03/11] drm/amdgpu/gfx: add generic handling for disable_kq
2025-03-06 1:06 ` Felix Kuehling
@ 2025-03-06 9:57 ` Khatri, Sunil
0 siblings, 0 replies; 30+ messages in thread
From: Khatri, Sunil @ 2025-03-06 9:57 UTC (permalink / raw)
To: Felix Kuehling, amd-gfx, Deucher, Alexander
On 3/6/2025 6:36 AM, Felix Kuehling wrote:
>
> On 2025-03-05 15:47, Alex Deucher wrote:
>> Add proper checks for disable_kq functionality in
>> gfx helper functions. Add special logic for families
>> that require the clear state setup.
>>
>> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
>> ---
>> drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c | 92 +++++++++++++++++--------
>> drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h | 2 +
>> 2 files changed, 67 insertions(+), 27 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
>> index a194bf3347cbc..af3f8b62f6fd5 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
>> @@ -371,6 +371,18 @@ int amdgpu_gfx_kiq_init(struct amdgpu_device *adev,
>> return 0;
>> }
>> +static bool amdgpu_gfx_disable_gfx_kq(struct amdgpu_device *adev)
>> +{
>> + if (adev->gfx.disable_kq) {
>> + /* GFX11 needs the GFX ring for clear buffer */
>> + if (amdgpu_ip_version(adev, GC_HWIP, 0) <= IP_VERSION(12, 0,
>> 0))
Yes the check has to be < as gfx12 do not need the clear buffer based
on our discussions.
Regards
Sunil
>
> Should this be < instead of <=?
>
> Regards,
> Felix
>
>> + return false;
>> + else
>> + return true;
>> + }
>> + return false;
>> +}
>> +
>> /* create MQD for each compute/gfx queue */
>> int amdgpu_gfx_mqd_sw_init(struct amdgpu_device *adev,
>> unsigned int mqd_size, int xcc_id)
>> @@ -379,6 +391,7 @@ int amdgpu_gfx_mqd_sw_init(struct amdgpu_device
>> *adev,
>> struct amdgpu_kiq *kiq = &adev->gfx.kiq[xcc_id];
>> struct amdgpu_ring *ring = &kiq->ring;
>> u32 domain = AMDGPU_GEM_DOMAIN_GTT;
>> + bool disable_kq_gfx = amdgpu_gfx_disable_gfx_kq(adev);
name of variable and function could be in sync. disable_gfx_kq and
amdgpu_gfx_disable_gfx_kq or change function name according to variable.
Also another suggestion here is better to have one more variable in the
gfx struct or ring and read this amdgpu_gfx_disable_gfx_kq once and use
it in all the places. It does looks confusing
so many similar sounding names.
Regards
Sunil
>> #if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
>> /* Only enable on gfx10 and 11 for now to avoid changing
>> behavior on older chips */
>> @@ -413,7 +426,8 @@ int amdgpu_gfx_mqd_sw_init(struct amdgpu_device
>> *adev,
>> }
>> }
>> - if (adev->asic_type >= CHIP_NAVI10 && amdgpu_async_gfx_ring) {
>> + if (adev->asic_type >= CHIP_NAVI10 && amdgpu_async_gfx_ring &&
>> + !disable_kq_gfx) {
>> /* create MQD for each KGQ */
>> for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
>> ring = &adev->gfx.gfx_ring[i];
>> @@ -437,25 +451,28 @@ int amdgpu_gfx_mqd_sw_init(struct amdgpu_device
>> *adev,
>> }
>> }
>> - /* create MQD for each KCQ */
>> - for (i = 0; i < adev->gfx.num_compute_rings; i++) {
>> - j = i + xcc_id * adev->gfx.num_compute_rings;
>> - ring = &adev->gfx.compute_ring[j];
>> - if (!ring->mqd_obj) {
>> - r = amdgpu_bo_create_kernel(adev, mqd_size, PAGE_SIZE,
>> - domain, &ring->mqd_obj,
>> - &ring->mqd_gpu_addr, &ring->mqd_ptr);
>> - if (r) {
>> - dev_warn(adev->dev, "failed to create ring mqd bo
>> (%d)", r);
>> - return r;
>> - }
>> + if (!adev->gfx.disable_kq) {
>
> Maybe just set adev->gfx.num_compute_rings to 0 somewhere, then you
> don't need this condition.
>
>
>> + /* create MQD for each KCQ */
>> + for (i = 0; i < adev->gfx.num_compute_rings; i++) {
>> + j = i + xcc_id * adev->gfx.num_compute_rings;
>> + ring = &adev->gfx.compute_ring[j];
>> + if (!ring->mqd_obj) {
>> + r = amdgpu_bo_create_kernel(adev, mqd_size, PAGE_SIZE,
>> + domain, &ring->mqd_obj,
>> + &ring->mqd_gpu_addr, &ring->mqd_ptr);
>> + if (r) {
>> + dev_warn(adev->dev, "failed to create ring mqd
>> bo (%d)", r);
>> + return r;
>> + }
>> - ring->mqd_size = mqd_size;
>> - /* prepare MQD backup */
>> - adev->gfx.mec.mqd_backup[j] = kzalloc(mqd_size,
>> GFP_KERNEL);
>> - if (!adev->gfx.mec.mqd_backup[j]) {
>> - dev_warn(adev->dev, "no memory to create MQD backup
>> for ring %s\n", ring->name);
>> - return -ENOMEM;
>> + ring->mqd_size = mqd_size;
>> + /* prepare MQD backup */
>> + adev->gfx.mec.mqd_backup[j] = kzalloc(mqd_size,
>> GFP_KERNEL);
>> + if (!adev->gfx.mec.mqd_backup[j]) {
>> + dev_warn(adev->dev, "no memory to create MQD
>> backup for ring %s\n",
>> + ring->name);
>> + return -ENOMEM;
>> + }
>> }
>> }
>> }
>> @@ -468,8 +485,10 @@ void amdgpu_gfx_mqd_sw_fini(struct amdgpu_device
>> *adev, int xcc_id)
>> struct amdgpu_ring *ring = NULL;
>> int i, j;
>> struct amdgpu_kiq *kiq = &adev->gfx.kiq[xcc_id];
>> + bool disable_kq_gfx = amdgpu_gfx_disable_gfx_kq(adev);
>> - if (adev->asic_type >= CHIP_NAVI10 && amdgpu_async_gfx_ring) {
>> + if (adev->asic_type >= CHIP_NAVI10 && amdgpu_async_gfx_ring &&
>> + !disable_kq_gfx) {
>> for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
>> ring = &adev->gfx.gfx_ring[i];
>> kfree(adev->gfx.me.mqd_backup[i]);
>> @@ -479,13 +498,15 @@ void amdgpu_gfx_mqd_sw_fini(struct
>> amdgpu_device *adev, int xcc_id)
>> }
>> }
>> - for (i = 0; i < adev->gfx.num_compute_rings; i++) {
>> - j = i + xcc_id * adev->gfx.num_compute_rings;
>> - ring = &adev->gfx.compute_ring[j];
>> - kfree(adev->gfx.mec.mqd_backup[j]);
>> - amdgpu_bo_free_kernel(&ring->mqd_obj,
>> - &ring->mqd_gpu_addr,
>> - &ring->mqd_ptr);
>> + if (!adev->gfx.disable_kq) {
>
> Same as above.
>
>
>> + for (i = 0; i < adev->gfx.num_compute_rings; i++) {
>> + j = i + xcc_id * adev->gfx.num_compute_rings;
>> + ring = &adev->gfx.compute_ring[j];
>> + kfree(adev->gfx.mec.mqd_backup[j]);
>> + amdgpu_bo_free_kernel(&ring->mqd_obj,
>> + &ring->mqd_gpu_addr,
>> + &ring->mqd_ptr);
>> + }
>> }
>> ring = &kiq->ring;
>> @@ -502,6 +523,9 @@ int amdgpu_gfx_disable_kcq(struct amdgpu_device
>> *adev, int xcc_id)
>> int i, r = 0;
>> int j;
>> + if (adev->gfx.disable_kq)
>
> Same as above.
>
>
>> + return 0;
>> +
>> if (adev->enable_mes) {
>> for (i = 0; i < adev->gfx.num_compute_rings; i++) {
>> j = i + xcc_id * adev->gfx.num_compute_rings;
>> @@ -547,11 +571,15 @@ int amdgpu_gfx_disable_kcq(struct amdgpu_device
>> *adev, int xcc_id)
>> int amdgpu_gfx_disable_kgq(struct amdgpu_device *adev, int xcc_id)
>> {
>> + bool disable_kq_gfx = amdgpu_gfx_disable_gfx_kq(adev);
>> struct amdgpu_kiq *kiq = &adev->gfx.kiq[xcc_id];
>> struct amdgpu_ring *kiq_ring = &kiq->ring;
>> int i, r = 0;
>> int j;
>> + if (disable_kq_gfx)
>> + return 0;
> Maybe just set adev->gfx.num_gfx_rings to 0 somewhere, then you don't
> need this condition.
>
> Regards,
> Felix
>
>
>> +
>> if (adev->enable_mes) {
>> if (amdgpu_gfx_is_master_xcc(adev, xcc_id)) {
>> for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
>> @@ -657,6 +685,9 @@ int amdgpu_gfx_enable_kcq(struct amdgpu_device
>> *adev, int xcc_id)
>> uint64_t queue_mask = 0;
>> int r, i, j;
>> + if (adev->gfx.disable_kq)
>> + return 0;
>> +
>> if (adev->mes.enable_legacy_queue_map)
>> return amdgpu_gfx_mes_enable_kcq(adev, xcc_id);
>> @@ -716,10 +747,14 @@ int amdgpu_gfx_enable_kcq(struct
>> amdgpu_device *adev, int xcc_id)
>> int amdgpu_gfx_enable_kgq(struct amdgpu_device *adev, int xcc_id)
>> {
>> + bool disable_kq_gfx = amdgpu_gfx_disable_gfx_kq(adev);
>> struct amdgpu_kiq *kiq = &adev->gfx.kiq[xcc_id];
>> struct amdgpu_ring *kiq_ring = &kiq->ring;
>> int r, i, j;
>> + if (disable_kq_gfx)
>> + return 0;
>> +
>> if (!kiq->pmf || !kiq->pmf->kiq_map_queues)
>> return -EINVAL;
>> @@ -1544,6 +1579,9 @@ static ssize_t
>> amdgpu_gfx_set_run_cleaner_shader(struct device *dev,
>> if (adev->in_suspend && !adev->in_runpm)
>> return -EPERM;
>> + if (adev->gfx.disable_kq)
>> + return -ENOTSUPP;
>> +
>> ret = kstrtol(buf, 0, &value);
>> if (ret)
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
>> index ddf4533614bac..8fa68a4ac34f1 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
>> @@ -483,6 +483,8 @@ struct amdgpu_gfx {
>> atomic_t total_submission_cnt;
>> struct delayed_work idle_work;
>> +
>> + bool disable_kq;
>> };
>> struct amdgpu_gfx_ras_reg_entry {
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 04/11] drm/amdgpu/mes: centralize gfx_hqd mask management
2025-03-05 20:47 [PATCH 00/11] Add disable kernel queue support Alex Deucher
` (2 preceding siblings ...)
2025-03-05 20:47 ` [PATCH 03/11] drm/amdgpu/gfx: add generic handling for disable_kq Alex Deucher
@ 2025-03-05 20:47 ` Alex Deucher
2025-03-06 10:20 ` Khatri, Sunil
2025-03-05 20:47 ` [PATCH 05/11] drm/amdgpu/mes: update hqd masks when disable_kq is set Alex Deucher
` (6 subsequent siblings)
10 siblings, 1 reply; 30+ messages in thread
From: Alex Deucher @ 2025-03-05 20:47 UTC (permalink / raw)
To: amd-gfx; +Cc: Alex Deucher
Move it to amdgpu_mes to align with the compute and
sdma hqd masks. No functional change.
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 24 ++++++++++++++++++++++++
drivers/gpu/drm/amd/amdgpu/mes_v11_0.c | 16 +++-------------
drivers/gpu/drm/amd/amdgpu/mes_v12_0.c | 15 +++------------
3 files changed, 30 insertions(+), 25 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
index ca076306adba4..afc2ce344df52 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
@@ -144,6 +144,30 @@ int amdgpu_mes_init(struct amdgpu_device *adev)
adev->mes.vmid_mask_mmhub = 0xffffff00;
adev->mes.vmid_mask_gfxhub = 0xffffff00;
+ if (adev->gfx.num_gfx_rings) {
+ for (i = 0; i < AMDGPU_MES_MAX_GFX_PIPES; i++) {
+ /* use only 1st ME pipe */
+ if (i >= adev->gfx.me.num_pipe_per_me)
+ continue;
+ if (amdgpu_ip_version(adev, GC_HWIP, 0) >=
+ IP_VERSION(12, 0, 0))
+ /*
+ * GFX V12 has only one GFX pipe, but 8 queues in it.
+ * GFX pipe 0 queue 0 is being used by Kernel queue.
+ * Set GFX pipe 0 queue 1-7 for MES scheduling
+ * mask = 1111 1110b
+ */
+ adev->mes.gfx_hqd_mask[i] = 0xFE;
+ else
+ /*
+ * GFX pipe 0 queue 0 is being used by Kernel queue.
+ * Set GFX pipe 0 queue 1 for MES scheduling
+ * mask = 10b
+ */
+ adev->mes.gfx_hqd_mask[i] = 0x2;
+ }
+ }
+
for (i = 0; i < AMDGPU_MES_MAX_COMPUTE_PIPES; i++) {
/* use only 1st MEC pipes */
if (i >= adev->gfx.mec.num_pipe_per_mec)
diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c b/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
index a569d09a1a748..39b45d8b5f049 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
@@ -669,18 +669,6 @@ static int mes_v11_0_misc_op(struct amdgpu_mes *mes,
offsetof(union MESAPI__MISC, api_status));
}
-static void mes_v11_0_set_gfx_hqd_mask(union MESAPI_SET_HW_RESOURCES *pkt)
-{
- /*
- * GFX pipe 0 queue 0 is being used by Kernel queue.
- * Set GFX pipe 0 queue 1 for MES scheduling
- * mask = 10b
- * GFX pipe 1 can't be used for MES due to HW limitation.
- */
- pkt->gfx_hqd_mask[0] = 0x2;
- pkt->gfx_hqd_mask[1] = 0;
-}
-
static int mes_v11_0_set_hw_resources(struct amdgpu_mes *mes)
{
int i;
@@ -705,7 +693,9 @@ static int mes_v11_0_set_hw_resources(struct amdgpu_mes *mes)
mes_set_hw_res_pkt.compute_hqd_mask[i] =
mes->compute_hqd_mask[i];
- mes_v11_0_set_gfx_hqd_mask(&mes_set_hw_res_pkt);
+ for (i = 0; i < MAX_GFX_PIPES; i++)
+ mes_set_hw_res_pkt.gfx_hqd_mask[i] =
+ mes->gfx_hqd_mask[i];
for (i = 0; i < MAX_SDMA_PIPES; i++)
mes_set_hw_res_pkt.sdma_hqd_mask[i] = mes->sdma_hqd_mask[i];
diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v12_0.c b/drivers/gpu/drm/amd/amdgpu/mes_v12_0.c
index 96336652d14c5..519f054bec60d 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_v12_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_v12_0.c
@@ -694,17 +694,6 @@ static int mes_v12_0_set_hw_resources_1(struct amdgpu_mes *mes, int pipe)
offsetof(union MESAPI_SET_HW_RESOURCES_1, api_status));
}
-static void mes_v12_0_set_gfx_hqd_mask(union MESAPI_SET_HW_RESOURCES *pkt)
-{
- /*
- * GFX V12 has only one GFX pipe, but 8 queues in it.
- * GFX pipe 0 queue 0 is being used by Kernel queue.
- * Set GFX pipe 0 queue 1-7 for MES scheduling
- * mask = 1111 1110b
- */
- pkt->gfx_hqd_mask[0] = 0xFE;
-}
-
static int mes_v12_0_set_hw_resources(struct amdgpu_mes *mes, int pipe)
{
int i;
@@ -727,7 +716,9 @@ static int mes_v12_0_set_hw_resources(struct amdgpu_mes *mes, int pipe)
mes_set_hw_res_pkt.compute_hqd_mask[i] =
mes->compute_hqd_mask[i];
- mes_v12_0_set_gfx_hqd_mask(&mes_set_hw_res_pkt);
+ for (i = 0; i < MAX_GFX_PIPES; i++)
+ mes_set_hw_res_pkt.gfx_hqd_mask[i] =
+ mes->gfx_hqd_mask[i];
for (i = 0; i < MAX_SDMA_PIPES; i++)
mes_set_hw_res_pkt.sdma_hqd_mask[i] =
--
2.48.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH 04/11] drm/amdgpu/mes: centralize gfx_hqd mask management
2025-03-05 20:47 ` [PATCH 04/11] drm/amdgpu/mes: centralize gfx_hqd mask management Alex Deucher
@ 2025-03-06 10:20 ` Khatri, Sunil
0 siblings, 0 replies; 30+ messages in thread
From: Khatri, Sunil @ 2025-03-06 10:20 UTC (permalink / raw)
To: Alex Deucher, amd-gfx
On 3/6/2025 2:17 AM, Alex Deucher wrote:
> Move it to amdgpu_mes to align with the compute and
> sdma hqd masks. No functional change.
>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 24 ++++++++++++++++++++++++
> drivers/gpu/drm/amd/amdgpu/mes_v11_0.c | 16 +++-------------
> drivers/gpu/drm/amd/amdgpu/mes_v12_0.c | 15 +++------------
> 3 files changed, 30 insertions(+), 25 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
> index ca076306adba4..afc2ce344df52 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
> @@ -144,6 +144,30 @@ int amdgpu_mes_init(struct amdgpu_device *adev)
> adev->mes.vmid_mask_mmhub = 0xffffff00;
> adev->mes.vmid_mask_gfxhub = 0xffffff00;
>
> + if (adev->gfx.num_gfx_rings) {
when kernel queue is disabled then arent we having gfx.num_gfx_rings ==
0, so this might not run at all ? Hope we taking care of that situation
too ?
> + for (i = 0; i < AMDGPU_MES_MAX_GFX_PIPES; i++) {
> + /* use only 1st ME pipe */
> + if (i >= adev->gfx.me.num_pipe_per_me)
> + continue;
this if condition makes the outside for loop to run just once and due to
which adev->mes.gfx_hqd_mask[1] = 0x0; is never set but based on
previous code we need to set that to 0
as pipe 1 is disabled in hq or here we do not need to set
adev->mes.gfx_hqd_mask at all now ?
> + if (amdgpu_ip_version(adev, GC_HWIP, 0) >=
> + IP_VERSION(12, 0, 0))
> + /*
> + * GFX V12 has only one GFX pipe, but 8 queues in it.
> + * GFX pipe 0 queue 0 is being used by Kernel queue.
> + * Set GFX pipe 0 queue 1-7 for MES scheduling
> + * mask = 1111 1110b
> + */
> + adev->mes.gfx_hqd_mask[i] = 0xFE;
> + else
> + /*
> + * GFX pipe 0 queue 0 is being used by Kernel queue.
> + * Set GFX pipe 0 queue 1 for MES scheduling
> + * mask = 10b
> + */
> + adev->mes.gfx_hqd_mask[i] = 0x2;
> + }
> + }
> +
> for (i = 0; i < AMDGPU_MES_MAX_COMPUTE_PIPES; i++) {
> /* use only 1st MEC pipes */
> if (i >= adev->gfx.mec.num_pipe_per_mec)
> diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c b/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
> index a569d09a1a748..39b45d8b5f049 100644
> --- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
> @@ -669,18 +669,6 @@ static int mes_v11_0_misc_op(struct amdgpu_mes *mes,
> offsetof(union MESAPI__MISC, api_status));
> }
>
> -static void mes_v11_0_set_gfx_hqd_mask(union MESAPI_SET_HW_RESOURCES *pkt)
> -{
> - /*
> - * GFX pipe 0 queue 0 is being used by Kernel queue.
> - * Set GFX pipe 0 queue 1 for MES scheduling
> - * mask = 10b
> - * GFX pipe 1 can't be used for MES due to HW limitation.
> - */
> - pkt->gfx_hqd_mask[0] = 0x2;
> - pkt->gfx_hqd_mask[1] = 0;
> -}
> -
> static int mes_v11_0_set_hw_resources(struct amdgpu_mes *mes)
> {
> int i;
> @@ -705,7 +693,9 @@ static int mes_v11_0_set_hw_resources(struct amdgpu_mes *mes)
> mes_set_hw_res_pkt.compute_hqd_mask[i] =
> mes->compute_hqd_mask[i];
>
> - mes_v11_0_set_gfx_hqd_mask(&mes_set_hw_res_pkt);
> + for (i = 0; i < MAX_GFX_PIPES; i++)
> + mes_set_hw_res_pkt.gfx_hqd_mask[i] =
> + mes->gfx_hqd_mask[i];
>
> for (i = 0; i < MAX_SDMA_PIPES; i++)
> mes_set_hw_res_pkt.sdma_hqd_mask[i] = mes->sdma_hqd_mask[i];
> diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v12_0.c b/drivers/gpu/drm/amd/amdgpu/mes_v12_0.c
> index 96336652d14c5..519f054bec60d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/mes_v12_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/mes_v12_0.c
> @@ -694,17 +694,6 @@ static int mes_v12_0_set_hw_resources_1(struct amdgpu_mes *mes, int pipe)
> offsetof(union MESAPI_SET_HW_RESOURCES_1, api_status));
> }
>
> -static void mes_v12_0_set_gfx_hqd_mask(union MESAPI_SET_HW_RESOURCES *pkt)
> -{
> - /*
> - * GFX V12 has only one GFX pipe, but 8 queues in it.
> - * GFX pipe 0 queue 0 is being used by Kernel queue.
> - * Set GFX pipe 0 queue 1-7 for MES scheduling
> - * mask = 1111 1110b
> - */
> - pkt->gfx_hqd_mask[0] = 0xFE;
> -}
> -
> static int mes_v12_0_set_hw_resources(struct amdgpu_mes *mes, int pipe)
> {
> int i;
> @@ -727,7 +716,9 @@ static int mes_v12_0_set_hw_resources(struct amdgpu_mes *mes, int pipe)
> mes_set_hw_res_pkt.compute_hqd_mask[i] =
> mes->compute_hqd_mask[i];
>
> - mes_v12_0_set_gfx_hqd_mask(&mes_set_hw_res_pkt);
> + for (i = 0; i < MAX_GFX_PIPES; i++)
> + mes_set_hw_res_pkt.gfx_hqd_mask[i] =
> + mes->gfx_hqd_mask[i];
>
> for (i = 0; i < MAX_SDMA_PIPES; i++)
> mes_set_hw_res_pkt.sdma_hqd_mask[i] =
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 05/11] drm/amdgpu/mes: update hqd masks when disable_kq is set
2025-03-05 20:47 [PATCH 00/11] Add disable kernel queue support Alex Deucher
` (3 preceding siblings ...)
2025-03-05 20:47 ` [PATCH 04/11] drm/amdgpu/mes: centralize gfx_hqd mask management Alex Deucher
@ 2025-03-05 20:47 ` Alex Deucher
2025-03-06 10:21 ` Khatri, Sunil
2025-03-05 20:47 ` [PATCH 06/11] drm/amdgpu/mes: make more vmids available when disable_kq=1 Alex Deucher
` (5 subsequent siblings)
10 siblings, 1 reply; 30+ messages in thread
From: Alex Deucher @ 2025-03-05 20:47 UTC (permalink / raw)
To: amd-gfx; +Cc: Alex Deucher, Sunil Khatri
Make all resources available to user queues.
Suggested-by: Sunil Khatri <sunil.khatri@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
index afc2ce344df52..71dce64a0ea32 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
@@ -157,14 +157,14 @@ int amdgpu_mes_init(struct amdgpu_device *adev)
* Set GFX pipe 0 queue 1-7 for MES scheduling
* mask = 1111 1110b
*/
- adev->mes.gfx_hqd_mask[i] = 0xFE;
+ adev->mes.gfx_hqd_mask[i] = adev->gfx.disable_kq ? 0xFF : 0xFE;
else
/*
* GFX pipe 0 queue 0 is being used by Kernel queue.
* Set GFX pipe 0 queue 1 for MES scheduling
* mask = 10b
*/
- adev->mes.gfx_hqd_mask[i] = 0x2;
+ adev->mes.gfx_hqd_mask[i] = adev->gfx.disable_kq ? 0x3 : 0x2;
}
}
@@ -172,7 +172,7 @@ int amdgpu_mes_init(struct amdgpu_device *adev)
/* use only 1st MEC pipes */
if (i >= adev->gfx.mec.num_pipe_per_mec)
continue;
- adev->mes.compute_hqd_mask[i] = 0xc;
+ adev->mes.compute_hqd_mask[i] = adev->gfx.disable_kq ? 0xF : 0xC;
}
for (i = 0; i < AMDGPU_MES_MAX_SDMA_PIPES; i++) {
--
2.48.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH 05/11] drm/amdgpu/mes: update hqd masks when disable_kq is set
2025-03-05 20:47 ` [PATCH 05/11] drm/amdgpu/mes: update hqd masks when disable_kq is set Alex Deucher
@ 2025-03-06 10:21 ` Khatri, Sunil
0 siblings, 0 replies; 30+ messages in thread
From: Khatri, Sunil @ 2025-03-06 10:21 UTC (permalink / raw)
To: Alex Deucher, amd-gfx; +Cc: Sunil Khatri
Reviewed-by: Sunil Khatri <sunil.khatri@amd.com>
On 3/6/2025 2:17 AM, Alex Deucher wrote:
> Make all resources available to user queues.
>
> Suggested-by: Sunil Khatri <sunil.khatri@amd.com>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
> index afc2ce344df52..71dce64a0ea32 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
> @@ -157,14 +157,14 @@ int amdgpu_mes_init(struct amdgpu_device *adev)
> * Set GFX pipe 0 queue 1-7 for MES scheduling
> * mask = 1111 1110b
> */
> - adev->mes.gfx_hqd_mask[i] = 0xFE;
> + adev->mes.gfx_hqd_mask[i] = adev->gfx.disable_kq ? 0xFF : 0xFE;
> else
> /*
> * GFX pipe 0 queue 0 is being used by Kernel queue.
> * Set GFX pipe 0 queue 1 for MES scheduling
> * mask = 10b
> */
> - adev->mes.gfx_hqd_mask[i] = 0x2;
> + adev->mes.gfx_hqd_mask[i] = adev->gfx.disable_kq ? 0x3 : 0x2;
> }
> }
>
> @@ -172,7 +172,7 @@ int amdgpu_mes_init(struct amdgpu_device *adev)
> /* use only 1st MEC pipes */
> if (i >= adev->gfx.mec.num_pipe_per_mec)
> continue;
> - adev->mes.compute_hqd_mask[i] = 0xc;
> + adev->mes.compute_hqd_mask[i] = adev->gfx.disable_kq ? 0xF : 0xC;
> }
>
> for (i = 0; i < AMDGPU_MES_MAX_SDMA_PIPES; i++) {
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 06/11] drm/amdgpu/mes: make more vmids available when disable_kq=1
2025-03-05 20:47 [PATCH 00/11] Add disable kernel queue support Alex Deucher
` (4 preceding siblings ...)
2025-03-05 20:47 ` [PATCH 05/11] drm/amdgpu/mes: update hqd masks when disable_kq is set Alex Deucher
@ 2025-03-05 20:47 ` Alex Deucher
2025-03-06 10:21 ` Khatri, Sunil
2025-03-05 20:47 ` [PATCH 07/11] drm/amdgpu/gfx11: add support for disable_kq Alex Deucher
` (4 subsequent siblings)
10 siblings, 1 reply; 30+ messages in thread
From: Alex Deucher @ 2025-03-05 20:47 UTC (permalink / raw)
To: amd-gfx; +Cc: Alex Deucher
If we don't have kernel queues, the vmids can be used by
the MES for user queues.
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 2 +-
drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 2 +-
drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
index 71dce64a0ea32..e2007759f5b4c 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
@@ -142,7 +142,7 @@ int amdgpu_mes_init(struct amdgpu_device *adev)
adev->mes.total_max_queue = AMDGPU_FENCE_MES_QUEUE_ID_MASK;
adev->mes.vmid_mask_mmhub = 0xffffff00;
- adev->mes.vmid_mask_gfxhub = 0xffffff00;
+ adev->mes.vmid_mask_gfxhub = adev->gfx.disable_kq ? 0xfffffffe : 0xffffff00;
if (adev->gfx.num_gfx_rings) {
for (i = 0; i < AMDGPU_MES_MAX_GFX_PIPES; i++) {
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
index 95d894a231fcf..19a5f196829f3 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
@@ -900,7 +900,7 @@ static int gmc_v10_0_sw_init(struct amdgpu_ip_block *ip_block)
* amdgpu graphics/compute will use VMIDs 1-7
* amdkfd will use VMIDs 8-15
*/
- adev->vm_manager.first_kfd_vmid = 8;
+ adev->vm_manager.first_kfd_vmid = adev->gfx.disable_kq ? 1 : 8;
amdgpu_vm_manager_init(adev);
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
index ea7c32d8380ba..598324e736092 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
@@ -837,7 +837,7 @@ static int gmc_v12_0_sw_init(struct amdgpu_ip_block *ip_block)
* amdgpu graphics/compute will use VMIDs 1-7
* amdkfd will use VMIDs 8-15
*/
- adev->vm_manager.first_kfd_vmid = 8;
+ adev->vm_manager.first_kfd_vmid = adev->gfx.disable_kq ? 1 : 8;
amdgpu_vm_manager_init(adev);
--
2.48.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH 06/11] drm/amdgpu/mes: make more vmids available when disable_kq=1
2025-03-05 20:47 ` [PATCH 06/11] drm/amdgpu/mes: make more vmids available when disable_kq=1 Alex Deucher
@ 2025-03-06 10:21 ` Khatri, Sunil
0 siblings, 0 replies; 30+ messages in thread
From: Khatri, Sunil @ 2025-03-06 10:21 UTC (permalink / raw)
To: Alex Deucher, amd-gfx
Reviewed-by: Sunil Khatri <sunil.khatri@amd.com>
On 3/6/2025 2:17 AM, Alex Deucher wrote:
> If we don't have kernel queues, the vmids can be used by
> the MES for user queues.
>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 2 +-
> drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 2 +-
> drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c | 2 +-
> 3 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
> index 71dce64a0ea32..e2007759f5b4c 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
> @@ -142,7 +142,7 @@ int amdgpu_mes_init(struct amdgpu_device *adev)
>
> adev->mes.total_max_queue = AMDGPU_FENCE_MES_QUEUE_ID_MASK;
> adev->mes.vmid_mask_mmhub = 0xffffff00;
> - adev->mes.vmid_mask_gfxhub = 0xffffff00;
> + adev->mes.vmid_mask_gfxhub = adev->gfx.disable_kq ? 0xfffffffe : 0xffffff00;
>
> if (adev->gfx.num_gfx_rings) {
> for (i = 0; i < AMDGPU_MES_MAX_GFX_PIPES; i++) {
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
> index 95d894a231fcf..19a5f196829f3 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
> @@ -900,7 +900,7 @@ static int gmc_v10_0_sw_init(struct amdgpu_ip_block *ip_block)
> * amdgpu graphics/compute will use VMIDs 1-7
> * amdkfd will use VMIDs 8-15
> */
> - adev->vm_manager.first_kfd_vmid = 8;
> + adev->vm_manager.first_kfd_vmid = adev->gfx.disable_kq ? 1 : 8;
>
> amdgpu_vm_manager_init(adev);
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
> index ea7c32d8380ba..598324e736092 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
> @@ -837,7 +837,7 @@ static int gmc_v12_0_sw_init(struct amdgpu_ip_block *ip_block)
> * amdgpu graphics/compute will use VMIDs 1-7
> * amdkfd will use VMIDs 8-15
> */
> - adev->vm_manager.first_kfd_vmid = 8;
> + adev->vm_manager.first_kfd_vmid = adev->gfx.disable_kq ? 1 : 8;
>
> amdgpu_vm_manager_init(adev);
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 07/11] drm/amdgpu/gfx11: add support for disable_kq
2025-03-05 20:47 [PATCH 00/11] Add disable kernel queue support Alex Deucher
` (5 preceding siblings ...)
2025-03-05 20:47 ` [PATCH 06/11] drm/amdgpu/mes: make more vmids available when disable_kq=1 Alex Deucher
@ 2025-03-05 20:47 ` Alex Deucher
2025-03-06 11:03 ` Khatri, Sunil
2025-03-05 20:47 ` [PATCH 08/11] drm/amdgpu/gfx12: " Alex Deucher
` (3 subsequent siblings)
10 siblings, 1 reply; 30+ messages in thread
From: Alex Deucher @ 2025-03-05 20:47 UTC (permalink / raw)
To: amd-gfx; +Cc: Alex Deucher
Plumb in support for disabling kernel queues in
GFX11. We have to bring up a GFX queue briefly in
order to initialize the clear state. After that
we can disable it.
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c | 77 +++++++++++++++++---------
1 file changed, 51 insertions(+), 26 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
index 3e49c0f399e02..e18100c5faa2a 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
@@ -1128,6 +1128,10 @@ static int gfx_v11_0_gfx_ring_init(struct amdgpu_device *adev, int ring_id,
ring->ring_obj = NULL;
ring->use_doorbell = true;
+ if (adev->gfx.disable_kq) {
+ ring->no_scheduler = true;
+ ring->no_user_submission = true;
+ }
if (!ring_id)
ring->doorbell_index = adev->doorbell_index.gfx_ring0 << 1;
@@ -1709,21 +1713,23 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block)
}
}
- ring_id = 0;
- /* set up the compute queues - allocate horizontally across pipes */
- for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
- for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
- for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {
- if (!amdgpu_gfx_is_mec_queue_enabled(adev, 0, i,
- k, j))
- continue;
+ if (!adev->gfx.disable_kq) {
+ ring_id = 0;
+ /* set up the compute queues - allocate horizontally across pipes */
+ for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
+ for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
+ for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {
+ if (!amdgpu_gfx_is_mec_queue_enabled(adev, 0, i,
+ k, j))
+ continue;
- r = gfx_v11_0_compute_ring_init(adev, ring_id,
- i, k, j);
- if (r)
- return r;
+ r = gfx_v11_0_compute_ring_init(adev, ring_id,
+ i, k, j);
+ if (r)
+ return r;
- ring_id++;
+ ring_id++;
+ }
}
}
}
@@ -1823,8 +1829,10 @@ static int gfx_v11_0_sw_fini(struct amdgpu_ip_block *ip_block)
for (i = 0; i < adev->gfx.num_gfx_rings; i++)
amdgpu_ring_fini(&adev->gfx.gfx_ring[i]);
- for (i = 0; i < adev->gfx.num_compute_rings; i++)
- amdgpu_ring_fini(&adev->gfx.compute_ring[i]);
+ if (!adev->gfx.disable_kq) {
+ for (i = 0; i < adev->gfx.num_compute_rings; i++)
+ amdgpu_ring_fini(&adev->gfx.compute_ring[i]);
+ }
amdgpu_gfx_mqd_sw_fini(adev, 0);
@@ -4489,6 +4497,9 @@ static int gfx_v11_0_kcq_resume(struct amdgpu_device *adev)
if (!amdgpu_async_gfx_ring)
gfx_v11_0_cp_compute_enable(adev, true);
+ if (adev->gfx.disable_kq)
+ return 0;
+
for (i = 0; i < adev->gfx.num_compute_rings; i++) {
ring = &adev->gfx.compute_ring[i];
@@ -4561,18 +4572,29 @@ static int gfx_v11_0_cp_resume(struct amdgpu_device *adev)
return r;
}
- for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
- ring = &adev->gfx.gfx_ring[i];
- r = amdgpu_ring_test_helper(ring);
- if (r)
- return r;
- }
+ if (adev->gfx.disable_kq) {
+ for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
+ ring = &adev->gfx.gfx_ring[i];
+ /* we don't want to set ring->ready */
+ r = amdgpu_ring_test_ring(ring);
+ if (r)
+ return r;
+ }
+ amdgpu_gfx_disable_kgq(adev, 0);
+ } else {
+ for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
+ ring = &adev->gfx.gfx_ring[i];
+ r = amdgpu_ring_test_helper(ring);
+ if (r)
+ return r;
+ }
- for (i = 0; i < adev->gfx.num_compute_rings; i++) {
- ring = &adev->gfx.compute_ring[i];
- r = amdgpu_ring_test_helper(ring);
- if (r)
- return r;
+ for (i = 0; i < adev->gfx.num_compute_rings; i++) {
+ ring = &adev->gfx.compute_ring[i];
+ r = amdgpu_ring_test_helper(ring);
+ if (r)
+ return r;
+ }
}
return 0;
@@ -5111,6 +5133,9 @@ static int gfx_v11_0_early_init(struct amdgpu_ip_block *ip_block)
{
struct amdgpu_device *adev = ip_block->adev;
+ if (amdgpu_disable_kq == 1)
+ adev->gfx.disable_kq = true;
+
adev->gfx.funcs = &gfx_v11_0_gfx_funcs;
adev->gfx.num_gfx_rings = GFX11_NUM_GFX_RINGS;
--
2.48.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH 07/11] drm/amdgpu/gfx11: add support for disable_kq
2025-03-05 20:47 ` [PATCH 07/11] drm/amdgpu/gfx11: add support for disable_kq Alex Deucher
@ 2025-03-06 11:03 ` Khatri, Sunil
0 siblings, 0 replies; 30+ messages in thread
From: Khatri, Sunil @ 2025-03-06 11:03 UTC (permalink / raw)
To: Alex Deucher, amd-gfx
Reviewed-by: Sunil Khatri <sunil.khatri@amd.com>
On 3/6/2025 2:17 AM, Alex Deucher wrote:
> Plumb in support for disabling kernel queues in
> GFX11. We have to bring up a GFX queue briefly in
> order to initialize the clear state. After that
> we can disable it.
>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c | 77 +++++++++++++++++---------
> 1 file changed, 51 insertions(+), 26 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
> index 3e49c0f399e02..e18100c5faa2a 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
> @@ -1128,6 +1128,10 @@ static int gfx_v11_0_gfx_ring_init(struct amdgpu_device *adev, int ring_id,
>
> ring->ring_obj = NULL;
> ring->use_doorbell = true;
> + if (adev->gfx.disable_kq) {
> + ring->no_scheduler = true;
> + ring->no_user_submission = true;
> + }
>
> if (!ring_id)
> ring->doorbell_index = adev->doorbell_index.gfx_ring0 << 1;
> @@ -1709,21 +1713,23 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block)
> }
> }
>
> - ring_id = 0;
> - /* set up the compute queues - allocate horizontally across pipes */
> - for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
> - for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
> - for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {
> - if (!amdgpu_gfx_is_mec_queue_enabled(adev, 0, i,
> - k, j))
> - continue;
> + if (!adev->gfx.disable_kq) {
> + ring_id = 0;
> + /* set up the compute queues - allocate horizontally across pipes */
> + for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
> + for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
> + for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {
> + if (!amdgpu_gfx_is_mec_queue_enabled(adev, 0, i,
> + k, j))
> + continue;
>
> - r = gfx_v11_0_compute_ring_init(adev, ring_id,
> - i, k, j);
> - if (r)
> - return r;
> + r = gfx_v11_0_compute_ring_init(adev, ring_id,
> + i, k, j);
> + if (r)
> + return r;
>
> - ring_id++;
> + ring_id++;
> + }
> }
> }
> }
> @@ -1823,8 +1829,10 @@ static int gfx_v11_0_sw_fini(struct amdgpu_ip_block *ip_block)
>
> for (i = 0; i < adev->gfx.num_gfx_rings; i++)
> amdgpu_ring_fini(&adev->gfx.gfx_ring[i]);
> - for (i = 0; i < adev->gfx.num_compute_rings; i++)
> - amdgpu_ring_fini(&adev->gfx.compute_ring[i]);
> + if (!adev->gfx.disable_kq) {
> + for (i = 0; i < adev->gfx.num_compute_rings; i++)
> + amdgpu_ring_fini(&adev->gfx.compute_ring[i]);
> + }
>
> amdgpu_gfx_mqd_sw_fini(adev, 0);
>
> @@ -4489,6 +4497,9 @@ static int gfx_v11_0_kcq_resume(struct amdgpu_device *adev)
> if (!amdgpu_async_gfx_ring)
> gfx_v11_0_cp_compute_enable(adev, true);
>
> + if (adev->gfx.disable_kq)
> + return 0;
> +
> for (i = 0; i < adev->gfx.num_compute_rings; i++) {
> ring = &adev->gfx.compute_ring[i];
>
> @@ -4561,18 +4572,29 @@ static int gfx_v11_0_cp_resume(struct amdgpu_device *adev)
> return r;
> }
>
> - for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
> - ring = &adev->gfx.gfx_ring[i];
> - r = amdgpu_ring_test_helper(ring);
> - if (r)
> - return r;
> - }
> + if (adev->gfx.disable_kq) {
> + for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
> + ring = &adev->gfx.gfx_ring[i];
> + /* we don't want to set ring->ready */
> + r = amdgpu_ring_test_ring(ring);
> + if (r)
> + return r;
> + }
> + amdgpu_gfx_disable_kgq(adev, 0);
> + } else {
> + for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
> + ring = &adev->gfx.gfx_ring[i];
> + r = amdgpu_ring_test_helper(ring);
> + if (r)
> + return r;
> + }
>
> - for (i = 0; i < adev->gfx.num_compute_rings; i++) {
> - ring = &adev->gfx.compute_ring[i];
> - r = amdgpu_ring_test_helper(ring);
> - if (r)
> - return r;
> + for (i = 0; i < adev->gfx.num_compute_rings; i++) {
> + ring = &adev->gfx.compute_ring[i];
> + r = amdgpu_ring_test_helper(ring);
> + if (r)
> + return r;
> + }
> }
>
> return 0;
> @@ -5111,6 +5133,9 @@ static int gfx_v11_0_early_init(struct amdgpu_ip_block *ip_block)
> {
> struct amdgpu_device *adev = ip_block->adev;
>
> + if (amdgpu_disable_kq == 1)
> + adev->gfx.disable_kq = true;
> +
> adev->gfx.funcs = &gfx_v11_0_gfx_funcs;
>
> adev->gfx.num_gfx_rings = GFX11_NUM_GFX_RINGS;
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 08/11] drm/amdgpu/gfx12: add support for disable_kq
2025-03-05 20:47 [PATCH 00/11] Add disable kernel queue support Alex Deucher
` (6 preceding siblings ...)
2025-03-05 20:47 ` [PATCH 07/11] drm/amdgpu/gfx11: add support for disable_kq Alex Deucher
@ 2025-03-05 20:47 ` Alex Deucher
2025-03-06 11:16 ` Khatri, Sunil
2025-03-05 20:47 ` [PATCH 09/11] drm/amdgpu/sdma: add flag for tracking disable_kq Alex Deucher
` (2 subsequent siblings)
10 siblings, 1 reply; 30+ messages in thread
From: Alex Deucher @ 2025-03-05 20:47 UTC (permalink / raw)
To: amd-gfx; +Cc: Alex Deucher
Plumb in support for disabling kernel queues.
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c | 114 ++++++++++++++-----------
1 file changed, 65 insertions(+), 49 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
index 8a448a20774a8..8fde7b239fdbb 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
@@ -1453,37 +1453,39 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block)
return r;
}
- /* set up the gfx ring */
- for (i = 0; i < adev->gfx.me.num_me; i++) {
- for (j = 0; j < adev->gfx.me.num_queue_per_pipe; j++) {
- for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {
- if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))
- continue;
-
- r = gfx_v12_0_gfx_ring_init(adev, ring_id,
- i, k, j);
- if (r)
- return r;
- ring_id++;
+ if (!adev->gfx.disable_kq) {
+ /* set up the gfx ring */
+ for (i = 0; i < adev->gfx.me.num_me; i++) {
+ for (j = 0; j < adev->gfx.me.num_queue_per_pipe; j++) {
+ for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {
+ if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))
+ continue;
+
+ r = gfx_v12_0_gfx_ring_init(adev, ring_id,
+ i, k, j);
+ if (r)
+ return r;
+ ring_id++;
+ }
}
}
- }
-
- ring_id = 0;
- /* set up the compute queues - allocate horizontally across pipes */
- for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
- for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
- for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {
- if (!amdgpu_gfx_is_mec_queue_enabled(adev,
- 0, i, k, j))
- continue;
-
- r = gfx_v12_0_compute_ring_init(adev, ring_id,
- i, k, j);
- if (r)
- return r;
- ring_id++;
+ ring_id = 0;
+ /* set up the compute queues - allocate horizontally across pipes */
+ for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
+ for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
+ for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {
+ if (!amdgpu_gfx_is_mec_queue_enabled(adev,
+ 0, i, k, j))
+ continue;
+
+ r = gfx_v12_0_compute_ring_init(adev, ring_id,
+ i, k, j);
+ if (r)
+ return r;
+
+ ring_id++;
+ }
}
}
}
@@ -1572,10 +1574,12 @@ static int gfx_v12_0_sw_fini(struct amdgpu_ip_block *ip_block)
int i;
struct amdgpu_device *adev = ip_block->adev;
- for (i = 0; i < adev->gfx.num_gfx_rings; i++)
- amdgpu_ring_fini(&adev->gfx.gfx_ring[i]);
- for (i = 0; i < adev->gfx.num_compute_rings; i++)
- amdgpu_ring_fini(&adev->gfx.compute_ring[i]);
+ if (!adev->gfx.disable_kq) {
+ for (i = 0; i < adev->gfx.num_gfx_rings; i++)
+ amdgpu_ring_fini(&adev->gfx.gfx_ring[i]);
+ for (i = 0; i < adev->gfx.num_compute_rings; i++)
+ amdgpu_ring_fini(&adev->gfx.compute_ring[i]);
+ }
amdgpu_gfx_mqd_sw_fini(adev, 0);
@@ -3418,6 +3422,9 @@ static int gfx_v12_0_kcq_resume(struct amdgpu_device *adev)
if (!amdgpu_async_gfx_ring)
gfx_v12_0_cp_compute_enable(adev, true);
+ if (adev->gfx.disable_kq)
+ return 0;
+
for (i = 0; i < adev->gfx.num_compute_rings; i++) {
ring = &adev->gfx.compute_ring[i];
@@ -3477,28 +3484,34 @@ static int gfx_v12_0_cp_resume(struct amdgpu_device *adev)
if (r)
return r;
- if (!amdgpu_async_gfx_ring) {
- r = gfx_v12_0_cp_gfx_resume(adev);
+ if (adev->gfx.disable_kq) {
+ r = gfx_v12_0_cp_gfx_start(adev);
if (r)
return r;
} else {
- r = gfx_v12_0_cp_async_gfx_ring_resume(adev);
- if (r)
- return r;
- }
+ if (!amdgpu_async_gfx_ring) {
+ r = gfx_v12_0_cp_gfx_resume(adev);
+ if (r)
+ return r;
+ } else {
+ r = gfx_v12_0_cp_async_gfx_ring_resume(adev);
+ if (r)
+ return r;
+ }
- for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
- ring = &adev->gfx.gfx_ring[i];
- r = amdgpu_ring_test_helper(ring);
- if (r)
- return r;
- }
+ for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
+ ring = &adev->gfx.gfx_ring[i];
+ r = amdgpu_ring_test_helper(ring);
+ if (r)
+ return r;
+ }
- for (i = 0; i < adev->gfx.num_compute_rings; i++) {
- ring = &adev->gfx.compute_ring[i];
- r = amdgpu_ring_test_helper(ring);
- if (r)
- return r;
+ for (i = 0; i < adev->gfx.num_compute_rings; i++) {
+ ring = &adev->gfx.compute_ring[i];
+ r = amdgpu_ring_test_helper(ring);
+ if (r)
+ return r;
+ }
}
return 0;
@@ -3791,6 +3804,9 @@ static int gfx_v12_0_early_init(struct amdgpu_ip_block *ip_block)
{
struct amdgpu_device *adev = ip_block->adev;
+ if (amdgpu_disable_kq == 1)
+ adev->gfx.disable_kq = true;
+
adev->gfx.funcs = &gfx_v12_0_gfx_funcs;
adev->gfx.num_gfx_rings = GFX12_NUM_GFX_RINGS;
--
2.48.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH 08/11] drm/amdgpu/gfx12: add support for disable_kq
2025-03-05 20:47 ` [PATCH 08/11] drm/amdgpu/gfx12: " Alex Deucher
@ 2025-03-06 11:16 ` Khatri, Sunil
0 siblings, 0 replies; 30+ messages in thread
From: Khatri, Sunil @ 2025-03-06 11:16 UTC (permalink / raw)
To: Alex Deucher, amd-gfx
Reviewed-by: Sunil Khatri <sunil.khatri@amd.com>
On 3/6/2025 2:17 AM, Alex Deucher wrote:
> Plumb in support for disabling kernel queues.
>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c | 114 ++++++++++++++-----------
> 1 file changed, 65 insertions(+), 49 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
> index 8a448a20774a8..8fde7b239fdbb 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
> @@ -1453,37 +1453,39 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block)
> return r;
> }
>
> - /* set up the gfx ring */
> - for (i = 0; i < adev->gfx.me.num_me; i++) {
> - for (j = 0; j < adev->gfx.me.num_queue_per_pipe; j++) {
> - for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {
> - if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))
> - continue;
> -
> - r = gfx_v12_0_gfx_ring_init(adev, ring_id,
> - i, k, j);
> - if (r)
> - return r;
> - ring_id++;
> + if (!adev->gfx.disable_kq) {
> + /* set up the gfx ring */
> + for (i = 0; i < adev->gfx.me.num_me; i++) {
> + for (j = 0; j < adev->gfx.me.num_queue_per_pipe; j++) {
> + for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {
> + if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))
> + continue;
> +
> + r = gfx_v12_0_gfx_ring_init(adev, ring_id,
> + i, k, j);
> + if (r)
> + return r;
> + ring_id++;
> + }
> }
> }
> - }
> -
> - ring_id = 0;
> - /* set up the compute queues - allocate horizontally across pipes */
> - for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
> - for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
> - for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {
> - if (!amdgpu_gfx_is_mec_queue_enabled(adev,
> - 0, i, k, j))
> - continue;
> -
> - r = gfx_v12_0_compute_ring_init(adev, ring_id,
> - i, k, j);
> - if (r)
> - return r;
>
> - ring_id++;
> + ring_id = 0;
> + /* set up the compute queues - allocate horizontally across pipes */
> + for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
> + for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
> + for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {
> + if (!amdgpu_gfx_is_mec_queue_enabled(adev,
> + 0, i, k, j))
> + continue;
> +
> + r = gfx_v12_0_compute_ring_init(adev, ring_id,
> + i, k, j);
> + if (r)
> + return r;
> +
> + ring_id++;
> + }
> }
> }
> }
> @@ -1572,10 +1574,12 @@ static int gfx_v12_0_sw_fini(struct amdgpu_ip_block *ip_block)
> int i;
> struct amdgpu_device *adev = ip_block->adev;
>
> - for (i = 0; i < adev->gfx.num_gfx_rings; i++)
> - amdgpu_ring_fini(&adev->gfx.gfx_ring[i]);
> - for (i = 0; i < adev->gfx.num_compute_rings; i++)
> - amdgpu_ring_fini(&adev->gfx.compute_ring[i]);
> + if (!adev->gfx.disable_kq) {
> + for (i = 0; i < adev->gfx.num_gfx_rings; i++)
> + amdgpu_ring_fini(&adev->gfx.gfx_ring[i]);
> + for (i = 0; i < adev->gfx.num_compute_rings; i++)
> + amdgpu_ring_fini(&adev->gfx.compute_ring[i]);
> + }
>
> amdgpu_gfx_mqd_sw_fini(adev, 0);
>
> @@ -3418,6 +3422,9 @@ static int gfx_v12_0_kcq_resume(struct amdgpu_device *adev)
> if (!amdgpu_async_gfx_ring)
> gfx_v12_0_cp_compute_enable(adev, true);
>
> + if (adev->gfx.disable_kq)
> + return 0;
> +
> for (i = 0; i < adev->gfx.num_compute_rings; i++) {
> ring = &adev->gfx.compute_ring[i];
>
> @@ -3477,28 +3484,34 @@ static int gfx_v12_0_cp_resume(struct amdgpu_device *adev)
> if (r)
> return r;
>
> - if (!amdgpu_async_gfx_ring) {
> - r = gfx_v12_0_cp_gfx_resume(adev);
> + if (adev->gfx.disable_kq) {
> + r = gfx_v12_0_cp_gfx_start(adev);
> if (r)
> return r;
> } else {
> - r = gfx_v12_0_cp_async_gfx_ring_resume(adev);
> - if (r)
> - return r;
> - }
> + if (!amdgpu_async_gfx_ring) {
> + r = gfx_v12_0_cp_gfx_resume(adev);
> + if (r)
> + return r;
> + } else {
> + r = gfx_v12_0_cp_async_gfx_ring_resume(adev);
> + if (r)
> + return r;
> + }
>
> - for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
> - ring = &adev->gfx.gfx_ring[i];
> - r = amdgpu_ring_test_helper(ring);
> - if (r)
> - return r;
> - }
> + for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
> + ring = &adev->gfx.gfx_ring[i];
> + r = amdgpu_ring_test_helper(ring);
> + if (r)
> + return r;
> + }
>
> - for (i = 0; i < adev->gfx.num_compute_rings; i++) {
> - ring = &adev->gfx.compute_ring[i];
> - r = amdgpu_ring_test_helper(ring);
> - if (r)
> - return r;
> + for (i = 0; i < adev->gfx.num_compute_rings; i++) {
> + ring = &adev->gfx.compute_ring[i];
> + r = amdgpu_ring_test_helper(ring);
> + if (r)
> + return r;
> + }
> }
>
> return 0;
> @@ -3791,6 +3804,9 @@ static int gfx_v12_0_early_init(struct amdgpu_ip_block *ip_block)
> {
> struct amdgpu_device *adev = ip_block->adev;
>
> + if (amdgpu_disable_kq == 1)
> + adev->gfx.disable_kq = true;
> +
> adev->gfx.funcs = &gfx_v12_0_gfx_funcs;
>
> adev->gfx.num_gfx_rings = GFX12_NUM_GFX_RINGS;
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 09/11] drm/amdgpu/sdma: add flag for tracking disable_kq
2025-03-05 20:47 [PATCH 00/11] Add disable kernel queue support Alex Deucher
` (7 preceding siblings ...)
2025-03-05 20:47 ` [PATCH 08/11] drm/amdgpu/gfx12: " Alex Deucher
@ 2025-03-05 20:47 ` Alex Deucher
2025-03-06 11:19 ` Khatri, Sunil
2025-03-05 20:47 ` [PATCH 10/11] drm/amdgpu/sdma6: add support for disable_kq Alex Deucher
2025-03-05 20:47 ` [PATCH 11/11] drm/amdgpu/sdma7: " Alex Deucher
10 siblings, 1 reply; 30+ messages in thread
From: Alex Deucher @ 2025-03-05 20:47 UTC (permalink / raw)
To: amd-gfx; +Cc: Alex Deucher
For SDMA, we still need kernel queues for paging so
they need to be initialized, but we no not want to
accept submissions from userspace when disable_kq
is set.
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h
index 9651693200655..edc856e10337a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h
@@ -129,6 +129,7 @@ struct amdgpu_sdma {
/* track guilty state of GFX and PAGE queues */
bool gfx_guilty;
bool page_guilty;
+ bool no_user_submission;
};
/*
--
2.48.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH 09/11] drm/amdgpu/sdma: add flag for tracking disable_kq
2025-03-05 20:47 ` [PATCH 09/11] drm/amdgpu/sdma: add flag for tracking disable_kq Alex Deucher
@ 2025-03-06 11:19 ` Khatri, Sunil
0 siblings, 0 replies; 30+ messages in thread
From: Khatri, Sunil @ 2025-03-06 11:19 UTC (permalink / raw)
To: Alex Deucher, amd-gfx
Reviewed-by: Sunil Khatri <sunil.khatri@amd.com>
On 3/6/2025 2:17 AM, Alex Deucher wrote:
> For SDMA, we still need kernel queues for paging so
> they need to be initialized, but we no not want to
> accept submissions from userspace when disable_kq
> is set.
>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h
> index 9651693200655..edc856e10337a 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h
> @@ -129,6 +129,7 @@ struct amdgpu_sdma {
> /* track guilty state of GFX and PAGE queues */
> bool gfx_guilty;
> bool page_guilty;
> + bool no_user_submission;
> };
>
> /*
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 10/11] drm/amdgpu/sdma6: add support for disable_kq
2025-03-05 20:47 [PATCH 00/11] Add disable kernel queue support Alex Deucher
` (8 preceding siblings ...)
2025-03-05 20:47 ` [PATCH 09/11] drm/amdgpu/sdma: add flag for tracking disable_kq Alex Deucher
@ 2025-03-05 20:47 ` Alex Deucher
2025-03-06 11:19 ` Khatri, Sunil
2025-03-05 20:47 ` [PATCH 11/11] drm/amdgpu/sdma7: " Alex Deucher
10 siblings, 1 reply; 30+ messages in thread
From: Alex Deucher @ 2025-03-05 20:47 UTC (permalink / raw)
To: amd-gfx; +Cc: Alex Deucher
When the parameter is set, disable user submissions
to kernel queues.
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
index 3aa4fec4d9e4a..bcc72737f8084 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
@@ -1304,6 +1304,9 @@ static int sdma_v6_0_early_init(struct amdgpu_ip_block *ip_block)
struct amdgpu_device *adev = ip_block->adev;
int r;
+ if (amdgpu_disable_kq == 1)
+ adev->sdma.no_user_submission = true;
+
r = amdgpu_sdma_init_microcode(adev, 0, true);
if (r)
return r;
@@ -1338,6 +1341,7 @@ static int sdma_v6_0_sw_init(struct amdgpu_ip_block *ip_block)
ring->ring_obj = NULL;
ring->use_doorbell = true;
ring->me = i;
+ ring->no_user_submission = adev->sdma.no_user_submission;
DRM_DEBUG("SDMA %d use_doorbell being set to: [%s]\n", i,
ring->use_doorbell?"true":"false");
--
2.48.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH 10/11] drm/amdgpu/sdma6: add support for disable_kq
2025-03-05 20:47 ` [PATCH 10/11] drm/amdgpu/sdma6: add support for disable_kq Alex Deucher
@ 2025-03-06 11:19 ` Khatri, Sunil
0 siblings, 0 replies; 30+ messages in thread
From: Khatri, Sunil @ 2025-03-06 11:19 UTC (permalink / raw)
To: Alex Deucher, amd-gfx
Reviewed-by: Sunil Khatri <sunil.khatri@amd.com>
On 3/6/2025 2:17 AM, Alex Deucher wrote:
> When the parameter is set, disable user submissions
> to kernel queues.
>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
> index 3aa4fec4d9e4a..bcc72737f8084 100644
> --- a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
> @@ -1304,6 +1304,9 @@ static int sdma_v6_0_early_init(struct amdgpu_ip_block *ip_block)
> struct amdgpu_device *adev = ip_block->adev;
> int r;
>
> + if (amdgpu_disable_kq == 1)
> + adev->sdma.no_user_submission = true;
> +
> r = amdgpu_sdma_init_microcode(adev, 0, true);
> if (r)
> return r;
> @@ -1338,6 +1341,7 @@ static int sdma_v6_0_sw_init(struct amdgpu_ip_block *ip_block)
> ring->ring_obj = NULL;
> ring->use_doorbell = true;
> ring->me = i;
> + ring->no_user_submission = adev->sdma.no_user_submission;
>
> DRM_DEBUG("SDMA %d use_doorbell being set to: [%s]\n", i,
> ring->use_doorbell?"true":"false");
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 11/11] drm/amdgpu/sdma7: add support for disable_kq
2025-03-05 20:47 [PATCH 00/11] Add disable kernel queue support Alex Deucher
` (9 preceding siblings ...)
2025-03-05 20:47 ` [PATCH 10/11] drm/amdgpu/sdma6: add support for disable_kq Alex Deucher
@ 2025-03-05 20:47 ` Alex Deucher
2025-03-06 11:20 ` Khatri, Sunil
10 siblings, 1 reply; 30+ messages in thread
From: Alex Deucher @ 2025-03-05 20:47 UTC (permalink / raw)
To: amd-gfx; +Cc: Alex Deucher
When the parameter is set, disable user submissions
to kernel queues.
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
index 92a79296708ae..40d45f738c0a8 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
@@ -1316,6 +1316,9 @@ static int sdma_v7_0_early_init(struct amdgpu_ip_block *ip_block)
struct amdgpu_device *adev = ip_block->adev;
int r;
+ if (amdgpu_disable_kq == 1)
+ adev->sdma.no_user_submission = true;
+
r = amdgpu_sdma_init_microcode(adev, 0, true);
if (r) {
DRM_ERROR("Failed to init sdma firmware!\n");
@@ -1351,6 +1354,7 @@ static int sdma_v7_0_sw_init(struct amdgpu_ip_block *ip_block)
ring->ring_obj = NULL;
ring->use_doorbell = true;
ring->me = i;
+ ring->no_user_submission = adev->sdma.no_user_submission;
DRM_DEBUG("SDMA %d use_doorbell being set to: [%s]\n", i,
ring->use_doorbell?"true":"false");
--
2.48.1
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [PATCH 11/11] drm/amdgpu/sdma7: add support for disable_kq
2025-03-05 20:47 ` [PATCH 11/11] drm/amdgpu/sdma7: " Alex Deucher
@ 2025-03-06 11:20 ` Khatri, Sunil
0 siblings, 0 replies; 30+ messages in thread
From: Khatri, Sunil @ 2025-03-06 11:20 UTC (permalink / raw)
To: Alex Deucher, amd-gfx
Reviewed-by: Sunil Khatri <sunil.khatri@amd.com>
On 3/6/2025 2:17 AM, Alex Deucher wrote:
> When the parameter is set, disable user submissions
> to kernel queues.
>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
> index 92a79296708ae..40d45f738c0a8 100644
> --- a/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
> @@ -1316,6 +1316,9 @@ static int sdma_v7_0_early_init(struct amdgpu_ip_block *ip_block)
> struct amdgpu_device *adev = ip_block->adev;
> int r;
>
> + if (amdgpu_disable_kq == 1)
> + adev->sdma.no_user_submission = true;
> +
> r = amdgpu_sdma_init_microcode(adev, 0, true);
> if (r) {
> DRM_ERROR("Failed to init sdma firmware!\n");
> @@ -1351,6 +1354,7 @@ static int sdma_v7_0_sw_init(struct amdgpu_ip_block *ip_block)
> ring->ring_obj = NULL;
> ring->use_doorbell = true;
> ring->me = i;
> + ring->no_user_submission = adev->sdma.no_user_submission;
>
> DRM_DEBUG("SDMA %d use_doorbell being set to: [%s]\n", i,
> ring->use_doorbell?"true":"false");
^ permalink raw reply [flat|nested] 30+ messages in thread