* [PATCH v3] drm/amdgpu: Increase KIQ invalidate_tlbs timeout
@ 2025-03-31 15:44 Jay Cornwall
2025-04-01 19:58 ` Kasiviswanathan, Harish
2025-04-02 7:37 ` Christian König
0 siblings, 2 replies; 4+ messages in thread
From: Jay Cornwall @ 2025-03-31 15:44 UTC (permalink / raw)
To: amd-gfx; +Cc: Jay Cornwall, Kent Russell
KIQ invalidate_tlbs request has been seen to marginally exceed the
configured 100 ms timeout on systems under load.
All other KIQ requests in the driver use a 10 second timeout. Use a
similar timeout implementation on the invalidate_tlbs path.
v2: Poll once before msleep
v3: Fix return value
Signed-off-by: Jay Cornwall <jay.cornwall@amd.com>
Cc: Kent Russell <kent.russell@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 -
drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 19 ++++++++++++++-----
2 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index ffca74a476da..3cdb5f8325aa 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -356,7 +356,6 @@ enum amdgpu_kiq_irq {
AMDGPU_CP_KIQ_IRQ_DRIVER0 = 0,
AMDGPU_CP_KIQ_IRQ_LAST
};
-#define SRIOV_USEC_TIMEOUT 1200000 /* wait 12 * 100ms for SRIOV */
#define MAX_KIQ_REG_WAIT 5000 /* in usecs, 5ms */
#define MAX_KIQ_REG_BAILOUT_INTERVAL 5 /* in msecs, 5ms */
#define MAX_KIQ_REG_TRY 1000
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
index 464625282872..ecb74ccf1d90 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
@@ -699,12 +699,10 @@ int amdgpu_gmc_flush_gpu_tlb_pasid(struct amdgpu_device *adev, uint16_t pasid,
uint32_t flush_type, bool all_hub,
uint32_t inst)
{
- u32 usec_timeout = amdgpu_sriov_vf(adev) ? SRIOV_USEC_TIMEOUT :
- adev->usec_timeout;
struct amdgpu_ring *ring = &adev->gfx.kiq[inst].ring;
struct amdgpu_kiq *kiq = &adev->gfx.kiq[inst];
unsigned int ndw;
- int r;
+ int r, cnt = 0;
uint32_t seq;
/*
@@ -761,10 +759,21 @@ int amdgpu_gmc_flush_gpu_tlb_pasid(struct amdgpu_device *adev, uint16_t pasid,
amdgpu_ring_commit(ring);
spin_unlock(&adev->gfx.kiq[inst].ring_lock);
- if (amdgpu_fence_wait_polling(ring, seq, usec_timeout) < 1) {
+
+ r = amdgpu_fence_wait_polling(ring, seq, MAX_KIQ_REG_WAIT);
+
+ might_sleep();
+ while (r < 1 && cnt++ < MAX_KIQ_REG_TRY &&
+ !amdgpu_reset_pending(adev->reset_domain)) {
+ msleep(MAX_KIQ_REG_BAILOUT_INTERVAL);
+ r = amdgpu_fence_wait_polling(ring, seq, MAX_KIQ_REG_WAIT);
+ }
+
+ if (cnt > MAX_KIQ_REG_TRY) {
dev_err(adev->dev, "timeout waiting for kiq fence\n");
r = -ETIME;
- }
+ } else
+ r = 0;
}
error_unlock_reset:
--
2.34.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* RE: [PATCH v3] drm/amdgpu: Increase KIQ invalidate_tlbs timeout
2025-03-31 15:44 [PATCH v3] drm/amdgpu: Increase KIQ invalidate_tlbs timeout Jay Cornwall
@ 2025-04-01 19:58 ` Kasiviswanathan, Harish
2025-04-02 7:37 ` Christian König
1 sibling, 0 replies; 4+ messages in thread
From: Kasiviswanathan, Harish @ 2025-04-01 19:58 UTC (permalink / raw)
To: Cornwall, Jay, amd-gfx@lists.freedesktop.org; +Cc: Cornwall, Jay, Russell, Kent
[AMD Official Use Only - AMD Internal Distribution Only]
Reviewed-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
-----Original Message-----
From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Jay Cornwall
Sent: Monday, March 31, 2025 11:45 AM
To: amd-gfx@lists.freedesktop.org
Cc: Cornwall, Jay <Jay.Cornwall@amd.com>; Russell, Kent <Kent.Russell@amd.com>
Subject: [PATCH v3] drm/amdgpu: Increase KIQ invalidate_tlbs timeout
KIQ invalidate_tlbs request has been seen to marginally exceed the
configured 100 ms timeout on systems under load.
All other KIQ requests in the driver use a 10 second timeout. Use a
similar timeout implementation on the invalidate_tlbs path.
v2: Poll once before msleep
v3: Fix return value
Signed-off-by: Jay Cornwall <jay.cornwall@amd.com>
Cc: Kent Russell <kent.russell@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 -
drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 19 ++++++++++++++-----
2 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index ffca74a476da..3cdb5f8325aa 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -356,7 +356,6 @@ enum amdgpu_kiq_irq {
AMDGPU_CP_KIQ_IRQ_DRIVER0 = 0,
AMDGPU_CP_KIQ_IRQ_LAST
};
-#define SRIOV_USEC_TIMEOUT 1200000 /* wait 12 * 100ms for SRIOV */
#define MAX_KIQ_REG_WAIT 5000 /* in usecs, 5ms */
#define MAX_KIQ_REG_BAILOUT_INTERVAL 5 /* in msecs, 5ms */
#define MAX_KIQ_REG_TRY 1000
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
index 464625282872..ecb74ccf1d90 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
@@ -699,12 +699,10 @@ int amdgpu_gmc_flush_gpu_tlb_pasid(struct amdgpu_device *adev, uint16_t pasid,
uint32_t flush_type, bool all_hub,
uint32_t inst)
{
- u32 usec_timeout = amdgpu_sriov_vf(adev) ? SRIOV_USEC_TIMEOUT :
- adev->usec_timeout;
struct amdgpu_ring *ring = &adev->gfx.kiq[inst].ring;
struct amdgpu_kiq *kiq = &adev->gfx.kiq[inst];
unsigned int ndw;
- int r;
+ int r, cnt = 0;
uint32_t seq;
/*
@@ -761,10 +759,21 @@ int amdgpu_gmc_flush_gpu_tlb_pasid(struct amdgpu_device *adev, uint16_t pasid,
amdgpu_ring_commit(ring);
spin_unlock(&adev->gfx.kiq[inst].ring_lock);
- if (amdgpu_fence_wait_polling(ring, seq, usec_timeout) < 1) {
+
+ r = amdgpu_fence_wait_polling(ring, seq, MAX_KIQ_REG_WAIT);
+
+ might_sleep();
+ while (r < 1 && cnt++ < MAX_KIQ_REG_TRY &&
+ !amdgpu_reset_pending(adev->reset_domain)) {
+ msleep(MAX_KIQ_REG_BAILOUT_INTERVAL);
+ r = amdgpu_fence_wait_polling(ring, seq, MAX_KIQ_REG_WAIT);
+ }
+
+ if (cnt > MAX_KIQ_REG_TRY) {
dev_err(adev->dev, "timeout waiting for kiq fence\n");
r = -ETIME;
- }
+ } else
+ r = 0;
}
error_unlock_reset:
--
2.34.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH v3] drm/amdgpu: Increase KIQ invalidate_tlbs timeout
2025-03-31 15:44 [PATCH v3] drm/amdgpu: Increase KIQ invalidate_tlbs timeout Jay Cornwall
2025-04-01 19:58 ` Kasiviswanathan, Harish
@ 2025-04-02 7:37 ` Christian König
2025-04-02 16:33 ` Jay Cornwall
1 sibling, 1 reply; 4+ messages in thread
From: Christian König @ 2025-04-02 7:37 UTC (permalink / raw)
To: Jay Cornwall, amd-gfx; +Cc: Kent Russell
Am 31.03.25 um 17:44 schrieb Jay Cornwall:
> KIQ invalidate_tlbs request has been seen to marginally exceed the
> configured 100 ms timeout on systems under load.
>
> All other KIQ requests in the driver use a 10 second timeout. Use a
> similar timeout implementation on the invalidate_tlbs path.
>
> v2: Poll once before msleep
> v3: Fix return value
>
> Signed-off-by: Jay Cornwall <jay.cornwall@amd.com>
> Cc: Kent Russell <kent.russell@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 -
> drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 19 ++++++++++++++-----
> 2 files changed, 14 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> index ffca74a476da..3cdb5f8325aa 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> @@ -356,7 +356,6 @@ enum amdgpu_kiq_irq {
> AMDGPU_CP_KIQ_IRQ_DRIVER0 = 0,
> AMDGPU_CP_KIQ_IRQ_LAST
> };
> -#define SRIOV_USEC_TIMEOUT 1200000 /* wait 12 * 100ms for SRIOV */
> #define MAX_KIQ_REG_WAIT 5000 /* in usecs, 5ms */
> #define MAX_KIQ_REG_BAILOUT_INTERVAL 5 /* in msecs, 5ms */
> #define MAX_KIQ_REG_TRY 1000
Unrelated to this patch here, but defines like those *must* have an AMDGPU_ prefix.
Please fix in a follow up patch.
Regards,
Christian.
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
> index 464625282872..ecb74ccf1d90 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
> @@ -699,12 +699,10 @@ int amdgpu_gmc_flush_gpu_tlb_pasid(struct amdgpu_device *adev, uint16_t pasid,
> uint32_t flush_type, bool all_hub,
> uint32_t inst)
> {
> - u32 usec_timeout = amdgpu_sriov_vf(adev) ? SRIOV_USEC_TIMEOUT :
> - adev->usec_timeout;
> struct amdgpu_ring *ring = &adev->gfx.kiq[inst].ring;
> struct amdgpu_kiq *kiq = &adev->gfx.kiq[inst];
> unsigned int ndw;
> - int r;
> + int r, cnt = 0;
> uint32_t seq;
>
> /*
> @@ -761,10 +759,21 @@ int amdgpu_gmc_flush_gpu_tlb_pasid(struct amdgpu_device *adev, uint16_t pasid,
>
> amdgpu_ring_commit(ring);
> spin_unlock(&adev->gfx.kiq[inst].ring_lock);
> - if (amdgpu_fence_wait_polling(ring, seq, usec_timeout) < 1) {
> +
> + r = amdgpu_fence_wait_polling(ring, seq, MAX_KIQ_REG_WAIT);
> +
> + might_sleep();
> + while (r < 1 && cnt++ < MAX_KIQ_REG_TRY &&
> + !amdgpu_reset_pending(adev->reset_domain)) {
> + msleep(MAX_KIQ_REG_BAILOUT_INTERVAL);
> + r = amdgpu_fence_wait_polling(ring, seq, MAX_KIQ_REG_WAIT);
> + }
> +
> + if (cnt > MAX_KIQ_REG_TRY) {
> dev_err(adev->dev, "timeout waiting for kiq fence\n");
> r = -ETIME;
> - }
> + } else
> + r = 0;
> }
>
> error_unlock_reset:
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v3] drm/amdgpu: Increase KIQ invalidate_tlbs timeout
2025-04-02 7:37 ` Christian König
@ 2025-04-02 16:33 ` Jay Cornwall
0 siblings, 0 replies; 4+ messages in thread
From: Jay Cornwall @ 2025-04-02 16:33 UTC (permalink / raw)
To: Christian König, amd-gfx; +Cc: Kent Russell
On 4/2/2025 02:37, Christian König wrote:
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>> index ffca74a476da..3cdb5f8325aa 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>> @@ -356,7 +356,6 @@ enum amdgpu_kiq_irq {
>> AMDGPU_CP_KIQ_IRQ_DRIVER0 = 0,
>> AMDGPU_CP_KIQ_IRQ_LAST
>> };
>> -#define SRIOV_USEC_TIMEOUT 1200000 /* wait 12 * 100ms for SRIOV */
>> #define MAX_KIQ_REG_WAIT 5000 /* in usecs, 5ms */
>> #define MAX_KIQ_REG_BAILOUT_INTERVAL 5 /* in msecs, 5ms */
>> #define MAX_KIQ_REG_TRY 1000
>
> Unrelated to this patch here, but defines like those *must* have an AMDGPU_ prefix.
>
> Please fix in a follow up patch.
Sure. A deeper problem which has led to these macros is the duplication
of polling logic across several different files.
We could instead move this code into amdgpu_fence_wait_polling. All
clients would then abort early on in_reset or in_interrupt. There are a
couple of users with different timeouts (adev->usec_timeout and a
hard-coded 2100ms) which could be unified or retained with a fixed 5ms
polling interval.
adev->usec_timeout is too low for this particular system under load.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-04-02 16:34 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-03-31 15:44 [PATCH v3] drm/amdgpu: Increase KIQ invalidate_tlbs timeout Jay Cornwall
2025-04-01 19:58 ` Kasiviswanathan, Harish
2025-04-02 7:37 ` Christian König
2025-04-02 16:33 ` Jay Cornwall
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.