[PATCH V5 00/28] Reset improvements for GC10+

[PATCH V5 00/28] Reset improvements for GC10+

[Alex Deucher]

This set improves per queue reset support for GC10+.
When we reset the queue, the queue is lost so we need
to re-emit the unprocessed state from subsequent submissions.
To that end, in order to make sure we actually restore
unprocessed state, we need to enable legacy enforce isolation
so that we can safely re-emit the unprocessed state.  If
we don't multiple jobs can run in parallel and we may not
end up resetting the correct one.  This is similar to how
windows handles queues.  This also gives us correct guilty
tracking for GC.

Tested on GC 10 and 11 chips with a game running and
then running hang tests.  The game pauses when the
hang happens, then continues after the queue reset.

I tried this same approach and GC8 and 9, but it
was not as reliable as soft recovery.  As such, I've dropped
the KGQ reset code for pre-GC10.

The same approach is extended to SDMA and VCN.
They don't need enforce isolation because those engines
are single threaded so they always operate serially.

Rework re-emit to signal the seq number of the bad job and
verify that to verify that the reset worked, then re-emit the
rest of the non-guilty state.  This way we are not waiting on
the rest of the state to complete, and if the subsequent state
also contains a bad job, we'll end up in queue reset again rather
than adapter reset.

v4: Drop explicit padding patches
    Drop new timeout macro
    Rework re-emit sequence
v5: Add a helper for reemit
    Convert VCN, JPEG, SDMA to use new helpers

Alex Deucher (27):
  drm/amdgpu: enable legacy enforce isolation by default
  drm/amdgpu/gfx7: drop reset_kgq
  drm/amdgpu/gfx8: drop reset_kgq
  drm/amdgpu/gfx9: drop reset_kgq
  drm/amdgpu: move force completion into ring resets
  drm/amdgpu: track ring state associated with a job
  drm/amdgpu/gfx10: re-emit unprocessed state on ring reset
  drm/amdgpu/gfx11: re-emit unprocessed state on ring reset
  drm/amdgpu/gfx12: re-emit unprocessed state on ring reset
  drm/amdgpu/gfx9: re-emit unprocessed state on kcq reset
  drm/amdgpu/gfx9.4.3: re-emit unprocessed state on kcq reset
  drm/amdgpu/sdma5: re-emit unprocessed state on ring reset
  drm/amdgpu/sdma5.2: re-emit unprocessed state on ring reset
  drm/amdgpu/sdma6: re-emit unprocessed state on ring reset
  drm/amdgpu/sdma7: re-emit unprocessed state on ring reset
  drm/amdgpu/jpeg2: re-emit unprocessed state on ring reset
  drm/amdgpu/jpeg2.5: re-emit unprocessed state on ring reset
  drm/amdgpu/jpeg3: re-emit unprocessed state on ring reset
  drm/amdgpu/jpeg4: re-emit unprocessed state on ring reset
  drm/amdgpu/jpeg4.0.3: re-emit unprocessed state on ring reset
  drm/amdgpu/jpeg5.0.0: add queue reset
  drm/amdgpu/jpeg5: re-emit unprocessed state on ring reset
  drm/amdgpu/jpeg5.0.1: re-emit unprocessed state on ring reset
  drm/amdgpu/vcn4: re-emit unprocessed state on ring reset
  drm/amdgpu/vcn4.0.3: re-emit unprocessed state on ring reset
  drm/amdgpu/vcn4.0.5: re-emit unprocessed state on ring reset
  drm/amdgpu/vcn5: re-emit unprocessed state on ring reset

Christian König (1):
  drm/amdgpu: rework queue reset scheduler interaction

 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c  | 12 ++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c     |  6 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    | 32 +++++-----
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.h    |  2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c   | 46 ++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h   |  8 +++
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c     | 31 ++--------
 drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c     | 21 +------
 drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c     | 21 +------
 drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c      | 71 ----------------------
 drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c      | 71 ----------------------
 drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c      | 51 +---------------
 drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c    |  6 +-
 drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c     |  3 +-
 drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c     |  3 +-
 drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c     |  3 +-
 drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c     |  3 +-
 drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c   |  3 +-
 drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_0.c   | 12 ++++
 drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c   |  3 +-
 drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c   |  4 ++
 drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c     |  7 ++-
 drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c     |  7 ++-
 drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c     |  6 +-
 drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c     |  6 +-
 drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c      |  2 +-
 drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c    |  3 +-
 drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c    |  2 +-
 drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c    |  2 +-
 30 files changed, 162 insertions(+), 289 deletions(-)

-- 
2.49.0

[PATCH 01/28] drm/amdgpu: enable legacy enforce isolation by default

[Alex Deucher]

Enable legacy enforce isolation (just serialize kernel
GC submissions).  This way we can reset a ring and
only affect the the process currently using that ring.
This mirrors what windows does.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 72e41781afb06..ca882e3d95bb7 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2148,9 +2148,7 @@ static int amdgpu_device_check_arguments(struct amdgpu_device *adev)
 
 	for (i = 0; i < MAX_XCP; i++) {
 		switch (amdgpu_enforce_isolation) {
-		case -1:
 		case 0:
-		default:
 			/* disable */
 			adev->enforce_isolation[i] = AMDGPU_ENFORCE_ISOLATION_DISABLE;
 			break;
@@ -2159,7 +2157,9 @@ static int amdgpu_device_check_arguments(struct amdgpu_device *adev)
 			adev->enforce_isolation[i] =
 				AMDGPU_ENFORCE_ISOLATION_ENABLE;
 			break;
+		case -1:
 		case 2:
+		default:
 			/* enable legacy mode */
 			adev->enforce_isolation[i] =
 				AMDGPU_ENFORCE_ISOLATION_ENABLE_LEGACY;
-- 
2.49.0

[PATCH 02/28] drm/amdgpu/gfx7: drop reset_kgq

[Alex Deucher]

It doesn't work reliably and we have soft recover and
full adapter reset so drop this.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c | 71 ---------------------------
 1 file changed, 71 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c
index da0534ff1271a..2aa323dab34e3 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c
@@ -4884,76 +4884,6 @@ static void gfx_v7_0_emit_mem_sync_compute(struct amdgpu_ring *ring)
 	amdgpu_ring_write(ring, 0x0000000A);	/* poll interval */
 }
 
-static void gfx_v7_0_wait_reg_mem(struct amdgpu_ring *ring, int eng_sel,
-				  int mem_space, int opt, uint32_t addr0,
-				  uint32_t addr1, uint32_t ref, uint32_t mask,
-				  uint32_t inv)
-{
-	amdgpu_ring_write(ring, PACKET3(PACKET3_WAIT_REG_MEM, 5));
-	amdgpu_ring_write(ring,
-			  /* memory (1) or register (0) */
-			  (WAIT_REG_MEM_MEM_SPACE(mem_space) |
-			   WAIT_REG_MEM_OPERATION(opt) | /* wait */
-			   WAIT_REG_MEM_FUNCTION(3) |  /* equal */
-			   WAIT_REG_MEM_ENGINE(eng_sel)));
-
-	if (mem_space)
-		BUG_ON(addr0 & 0x3); /* Dword align */
-	amdgpu_ring_write(ring, addr0);
-	amdgpu_ring_write(ring, addr1);
-	amdgpu_ring_write(ring, ref);
-	amdgpu_ring_write(ring, mask);
-	amdgpu_ring_write(ring, inv); /* poll interval */
-}
-
-static void gfx_v7_0_ring_emit_reg_wait(struct amdgpu_ring *ring, uint32_t reg,
-					uint32_t val, uint32_t mask)
-{
-	gfx_v7_0_wait_reg_mem(ring, 0, 0, 0, reg, 0, val, mask, 0x20);
-}
-
-static int gfx_v7_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)
-{
-	struct amdgpu_device *adev = ring->adev;
-	struct amdgpu_kiq *kiq = &adev->gfx.kiq[0];
-	struct amdgpu_ring *kiq_ring = &kiq->ring;
-	unsigned long flags;
-	u32 tmp;
-	int r;
-
-	if (amdgpu_sriov_vf(adev))
-		return -EINVAL;
-
-	if (!kiq->pmf || !kiq->pmf->kiq_unmap_queues)
-		return -EINVAL;
-
-	spin_lock_irqsave(&kiq->ring_lock, flags);
-
-	if (amdgpu_ring_alloc(kiq_ring, 5)) {
-		spin_unlock_irqrestore(&kiq->ring_lock, flags);
-		return -ENOMEM;
-	}
-
-	tmp = REG_SET_FIELD(0, CP_VMID_RESET, RESET_REQUEST, 1 << vmid);
-	gfx_v7_0_ring_emit_wreg(kiq_ring, mmCP_VMID_RESET, tmp);
-	amdgpu_ring_commit(kiq_ring);
-
-	spin_unlock_irqrestore(&kiq->ring_lock, flags);
-
-	r = amdgpu_ring_test_ring(kiq_ring);
-	if (r)
-		return r;
-
-	if (amdgpu_ring_alloc(ring, 7 + 12 + 5))
-		return -ENOMEM;
-	gfx_v7_0_ring_emit_fence_gfx(ring, ring->fence_drv.gpu_addr,
-				     ring->fence_drv.sync_seq, AMDGPU_FENCE_FLAG_EXEC);
-	gfx_v7_0_ring_emit_reg_wait(ring, mmCP_VMID_RESET, 0, 0xffff);
-	gfx_v7_0_ring_emit_wreg(ring, mmCP_VMID_RESET, 0);
-
-	return amdgpu_ring_test_ring(ring);
-}
-
 static const struct amd_ip_funcs gfx_v7_0_ip_funcs = {
 	.name = "gfx_v7_0",
 	.early_init = gfx_v7_0_early_init,
@@ -5003,7 +4933,6 @@ static const struct amdgpu_ring_funcs gfx_v7_0_ring_funcs_gfx = {
 	.emit_wreg = gfx_v7_0_ring_emit_wreg,
 	.soft_recovery = gfx_v7_0_ring_soft_recovery,
 	.emit_mem_sync = gfx_v7_0_emit_mem_sync,
-	.reset = gfx_v7_0_reset_kgq,
 };
 
 static const struct amdgpu_ring_funcs gfx_v7_0_ring_funcs_compute = {
-- 
2.49.0

[PATCH 03/28] drm/amdgpu/gfx8: drop reset_kgq

[Alex Deucher]

It doesn't work reliably and we have soft recover and
full adapter reset so drop this.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c | 71 ---------------------------
 1 file changed, 71 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
index 5ee2237d8ee8f..68c401ecb3eca 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
@@ -6339,34 +6339,6 @@ static void gfx_v8_0_ring_emit_wreg(struct amdgpu_ring *ring, uint32_t reg,
 	amdgpu_ring_write(ring, val);
 }
 
-static void gfx_v8_0_wait_reg_mem(struct amdgpu_ring *ring, int eng_sel,
-				  int mem_space, int opt, uint32_t addr0,
-				  uint32_t addr1, uint32_t ref, uint32_t mask,
-				  uint32_t inv)
-{
-	amdgpu_ring_write(ring, PACKET3(PACKET3_WAIT_REG_MEM, 5));
-	amdgpu_ring_write(ring,
-			  /* memory (1) or register (0) */
-			  (WAIT_REG_MEM_MEM_SPACE(mem_space) |
-			   WAIT_REG_MEM_OPERATION(opt) | /* wait */
-			   WAIT_REG_MEM_FUNCTION(3) |  /* equal */
-			   WAIT_REG_MEM_ENGINE(eng_sel)));
-
-	if (mem_space)
-		BUG_ON(addr0 & 0x3); /* Dword align */
-	amdgpu_ring_write(ring, addr0);
-	amdgpu_ring_write(ring, addr1);
-	amdgpu_ring_write(ring, ref);
-	amdgpu_ring_write(ring, mask);
-	amdgpu_ring_write(ring, inv); /* poll interval */
-}
-
-static void gfx_v8_0_ring_emit_reg_wait(struct amdgpu_ring *ring, uint32_t reg,
-					uint32_t val, uint32_t mask)
-{
-	gfx_v8_0_wait_reg_mem(ring, 0, 0, 0, reg, 0, val, mask, 0x20);
-}
-
 static void gfx_v8_0_ring_soft_recovery(struct amdgpu_ring *ring, unsigned vmid)
 {
 	struct amdgpu_device *adev = ring->adev;
@@ -6843,48 +6815,6 @@ static void gfx_v8_0_emit_wave_limit(struct amdgpu_ring *ring, bool enable)
 
 }
 
-static int gfx_v8_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)
-{
-	struct amdgpu_device *adev = ring->adev;
-	struct amdgpu_kiq *kiq = &adev->gfx.kiq[0];
-	struct amdgpu_ring *kiq_ring = &kiq->ring;
-	unsigned long flags;
-	u32 tmp;
-	int r;
-
-	if (amdgpu_sriov_vf(adev))
-		return -EINVAL;
-
-	if (!kiq->pmf || !kiq->pmf->kiq_unmap_queues)
-		return -EINVAL;
-
-	spin_lock_irqsave(&kiq->ring_lock, flags);
-
-	if (amdgpu_ring_alloc(kiq_ring, 5)) {
-		spin_unlock_irqrestore(&kiq->ring_lock, flags);
-		return -ENOMEM;
-	}
-
-	tmp = REG_SET_FIELD(0, CP_VMID_RESET, RESET_REQUEST, 1 << vmid);
-	gfx_v8_0_ring_emit_wreg(kiq_ring, mmCP_VMID_RESET, tmp);
-	amdgpu_ring_commit(kiq_ring);
-
-	spin_unlock_irqrestore(&kiq->ring_lock, flags);
-
-	r = amdgpu_ring_test_ring(kiq_ring);
-	if (r)
-		return r;
-
-	if (amdgpu_ring_alloc(ring, 7 + 12 + 5))
-		return -ENOMEM;
-	gfx_v8_0_ring_emit_fence_gfx(ring, ring->fence_drv.gpu_addr,
-				     ring->fence_drv.sync_seq, AMDGPU_FENCE_FLAG_EXEC);
-	gfx_v8_0_ring_emit_reg_wait(ring, mmCP_VMID_RESET, 0, 0xffff);
-	gfx_v8_0_ring_emit_wreg(ring, mmCP_VMID_RESET, 0);
-
-	return amdgpu_ring_test_ring(ring);
-}
-
 static const struct amd_ip_funcs gfx_v8_0_ip_funcs = {
 	.name = "gfx_v8_0",
 	.early_init = gfx_v8_0_early_init,
@@ -6950,7 +6880,6 @@ static const struct amdgpu_ring_funcs gfx_v8_0_ring_funcs_gfx = {
 	.emit_wreg = gfx_v8_0_ring_emit_wreg,
 	.soft_recovery = gfx_v8_0_ring_soft_recovery,
 	.emit_mem_sync = gfx_v8_0_emit_mem_sync,
-	.reset = gfx_v8_0_reset_kgq,
 };
 
 static const struct amdgpu_ring_funcs gfx_v8_0_ring_funcs_compute = {
-- 
2.49.0

[PATCH 04/28] drm/amdgpu/gfx9: drop reset_kgq

[Alex Deucher]

It doesn't work reliably and we have soft recover and
full adapter reset so drop this.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 46 ---------------------------
 1 file changed, 46 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
index d377a7c57d5e1..d50e125fd3e0d 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
@@ -7152,51 +7152,6 @@ static void gfx_v9_ring_insert_nop(struct amdgpu_ring *ring, uint32_t num_nop)
 	amdgpu_ring_insert_nop(ring, num_nop - 1);
 }
 
-static int gfx_v9_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)
-{
-	struct amdgpu_device *adev = ring->adev;
-	struct amdgpu_kiq *kiq = &adev->gfx.kiq[0];
-	struct amdgpu_ring *kiq_ring = &kiq->ring;
-	unsigned long flags;
-	u32 tmp;
-	int r;
-
-	if (amdgpu_sriov_vf(adev))
-		return -EINVAL;
-
-	if (!kiq->pmf || !kiq->pmf->kiq_unmap_queues)
-		return -EINVAL;
-
-	spin_lock_irqsave(&kiq->ring_lock, flags);
-
-	if (amdgpu_ring_alloc(kiq_ring, 5)) {
-		spin_unlock_irqrestore(&kiq->ring_lock, flags);
-		return -ENOMEM;
-	}
-
-	tmp = REG_SET_FIELD(0, CP_VMID_RESET, RESET_REQUEST, 1 << vmid);
-	gfx_v9_0_ring_emit_wreg(kiq_ring,
-				 SOC15_REG_OFFSET(GC, 0, mmCP_VMID_RESET), tmp);
-	amdgpu_ring_commit(kiq_ring);
-
-	spin_unlock_irqrestore(&kiq->ring_lock, flags);
-
-	r = amdgpu_ring_test_ring(kiq_ring);
-	if (r)
-		return r;
-
-	if (amdgpu_ring_alloc(ring, 7 + 7 + 5))
-		return -ENOMEM;
-	gfx_v9_0_ring_emit_fence(ring, ring->fence_drv.gpu_addr,
-				 ring->fence_drv.sync_seq, AMDGPU_FENCE_FLAG_EXEC);
-	gfx_v9_0_ring_emit_reg_wait(ring,
-				    SOC15_REG_OFFSET(GC, 0, mmCP_VMID_RESET), 0, 0xffff);
-	gfx_v9_0_ring_emit_wreg(ring,
-				SOC15_REG_OFFSET(GC, 0, mmCP_VMID_RESET), 0);
-
-	return amdgpu_ring_test_ring(ring);
-}
-
 static int gfx_v9_0_reset_kcq(struct amdgpu_ring *ring,
 			      unsigned int vmid)
 {
@@ -7477,7 +7432,6 @@ static const struct amdgpu_ring_funcs gfx_v9_0_ring_funcs_gfx = {
 	.emit_reg_write_reg_wait = gfx_v9_0_ring_emit_reg_write_reg_wait,
 	.soft_recovery = gfx_v9_0_ring_soft_recovery,
 	.emit_mem_sync = gfx_v9_0_emit_mem_sync,
-	.reset = gfx_v9_0_reset_kgq,
 	.emit_cleaner_shader = gfx_v9_0_ring_emit_cleaner_shader,
 	.begin_use = amdgpu_gfx_enforce_isolation_ring_begin_use,
 	.end_use = amdgpu_gfx_enforce_isolation_ring_end_use,
-- 
2.49.0

[PATCH 05/28] drm/amdgpu: rework queue reset scheduler interaction

[Alex Deucher]

From: Christian König <ckoenig.leichtzumerken@gmail.com>

Stopping the scheduler for queue reset is generally a good idea because
it prevents any worker from touching the ring buffer.

But using amdgpu_fence_driver_force_completion() before restarting it was
a really bad idea because it marked fences as failed while the work was
potentially still running.

Stop doing that and cleanup the comment a bit.

v2: keep amdgpu_fence_driver_force_completion() for non-gfx rings
v3: drop amdgpu_fence_driver_force_completion() for compute ring

Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 27 ++++++++++++++-----------
 1 file changed, 15 insertions(+), 12 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
index acb21fc8b3ce5..0b9086a747c0a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
@@ -136,10 +136,12 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
 	} else if (amdgpu_gpu_recovery && ring->funcs->reset) {
 		bool is_guilty;
 
-		dev_err(adev->dev, "Starting %s ring reset\n", s_job->sched->name);
-		/* stop the scheduler, but don't mess with the
-		 * bad job yet because if ring reset fails
-		 * we'll fall back to full GPU reset.
+		dev_err(adev->dev, "Starting %s ring reset\n",
+			s_job->sched->name);
+
+		/*
+		 * Stop the scheduler to prevent anybody else from touching the
+		 * ring buffer.
 		 */
 		drm_sched_wqueue_stop(&ring->sched);
 
@@ -157,19 +159,20 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
 
 		r = amdgpu_ring_reset(ring, job->vmid);
 		if (!r) {
-			if (amdgpu_ring_sched_ready(ring))
-				drm_sched_stop(&ring->sched, s_job);
 			if (is_guilty) {
 				atomic_inc(&ring->adev->gpu_reset_counter);
-				amdgpu_fence_driver_force_completion(ring);
+				if ((ring->funcs->type != AMDGPU_RING_TYPE_GFX) &&
+				    (ring->funcs->type != AMDGPU_RING_TYPE_COMPUTE))
+					amdgpu_fence_driver_force_completion(ring);
 			}
-			if (amdgpu_ring_sched_ready(ring))
-				drm_sched_start(&ring->sched, 0);
-			dev_err(adev->dev, "Ring %s reset succeeded\n", ring->sched.name);
-			drm_dev_wedged_event(adev_to_drm(adev), DRM_WEDGE_RECOVERY_NONE);
+			drm_sched_wqueue_start(&ring->sched);
+			dev_err(adev->dev, "Ring %s reset succeeded\n",
+				ring->sched.name);
+			drm_dev_wedged_event(adev_to_drm(adev),
+					     DRM_WEDGE_RECOVERY_NONE);
 			goto exit;
 		}
-		dev_err(adev->dev, "Ring %s reset failure\n", ring->sched.name);
+		dev_err(adev->dev, "Ring %s reset failed\n", ring->sched.name);
 	}
 	dma_fence_set_error(&s_job->s_fence->finished, -ETIME);
 
-- 
2.49.0

[PATCH 06/28] drm/amdgpu: move force completion into ring resets

[Alex Deucher]

Move the force completion handling into each ring
reset function so that each engine can determine
whether or not it needs to force completion on the
jobs in the ring.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c  | 6 +-----
 drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c   | 1 +
 drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c   | 1 +
 drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c   | 1 +
 drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c   | 1 +
 drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c | 1 +
 drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c | 1 +
 drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c | 4 ++++
 drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c   | 5 ++++-
 drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c   | 5 ++++-
 drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c   | 1 +
 drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c   | 1 +
 drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c    | 1 +
 drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c  | 5 ++---
 drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c  | 1 +
 drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c  | 1 +
 16 files changed, 26 insertions(+), 10 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
index 0b9086a747c0a..a0fab947143b5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
@@ -159,12 +159,8 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
 
 		r = amdgpu_ring_reset(ring, job->vmid);
 		if (!r) {
-			if (is_guilty) {
+			if (is_guilty)
 				atomic_inc(&ring->adev->gpu_reset_counter);
-				if ((ring->funcs->type != AMDGPU_RING_TYPE_GFX) &&
-				    (ring->funcs->type != AMDGPU_RING_TYPE_COMPUTE))
-					amdgpu_fence_driver_force_completion(ring);
-			}
 			drm_sched_wqueue_start(&ring->sched);
 			dev_err(adev->dev, "Ring %s reset succeeded\n",
 				ring->sched.name);
diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c
index 4cde8a8bcc837..336ca44950dfa 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c
@@ -768,6 +768,7 @@ static int jpeg_v2_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 {
 	jpeg_v2_0_stop(ring->adev);
 	jpeg_v2_0_start(ring->adev);
+	amdgpu_fence_driver_force_completion(ring);
 	return amdgpu_ring_test_helper(ring);
 }
 
diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c
index 8b39e114f3be1..72d739e38a8ae 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c
@@ -647,6 +647,7 @@ static int jpeg_v2_5_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 {
 	jpeg_v2_5_stop_inst(ring->adev, ring->me);
 	jpeg_v2_5_start_inst(ring->adev, ring->me);
+	amdgpu_fence_driver_force_completion(ring);
 	return amdgpu_ring_test_helper(ring);
 }
 
diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c
index 2f8510c2986b9..7974d04c15e92 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c
@@ -559,6 +559,7 @@ static int jpeg_v3_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 {
 	jpeg_v3_0_stop(ring->adev);
 	jpeg_v3_0_start(ring->adev);
+	amdgpu_fence_driver_force_completion(ring);
 	return amdgpu_ring_test_helper(ring);
 }
 
diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c
index f17ec5414fd69..9301d0ff2593b 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c
@@ -727,6 +727,7 @@ static int jpeg_v4_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 
 	jpeg_v4_0_stop(ring->adev);
 	jpeg_v4_0_start(ring->adev);
+	amdgpu_fence_driver_force_completion(ring);
 	return amdgpu_ring_test_helper(ring);
 }
 
diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c
index 79e342d5ab28d..ef5075e111300 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c
@@ -1150,6 +1150,7 @@ static int jpeg_v4_0_3_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 
 	jpeg_v4_0_3_core_stall_reset(ring);
 	jpeg_v4_0_3_start_jrbc(ring);
+	amdgpu_fence_driver_force_completion(ring);
 	return amdgpu_ring_test_helper(ring);
 }
 
diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c
index 3b6f65a256464..858b930d1dd54 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c
@@ -841,6 +841,7 @@ static int jpeg_v5_0_1_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 
 	jpeg_v5_0_1_core_stall_reset(ring);
 	jpeg_v5_0_1_init_jrbc(ring);
+	amdgpu_fence_driver_force_completion(ring);
 	return amdgpu_ring_test_helper(ring);
 }
 
diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c b/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c
index 9c169112a5e7b..eb78bf04a490b 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c
@@ -1670,6 +1670,7 @@ static bool sdma_v4_4_2_page_ring_is_guilty(struct amdgpu_ring *ring)
 static int sdma_v4_4_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
 {
 	struct amdgpu_device *adev = ring->adev;
+	bool is_guilty = ring->funcs->is_guilty(ring);
 	u32 id = GET_INST(SDMA0, ring->me);
 	int r;
 
@@ -1680,6 +1681,9 @@ static int sdma_v4_4_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
 	r = amdgpu_sdma_reset_engine(adev, id);
 	amdgpu_amdkfd_resume(adev, false);
 
+	if (is_guilty)
+		amdgpu_fence_driver_force_completion(ring);
+
 	return r;
 }
 
diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c
index 9505ae96fbecc..43885dc04b632 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c
@@ -1542,8 +1542,11 @@ static int sdma_v5_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
 {
 	struct amdgpu_device *adev = ring->adev;
 	u32 inst_id = ring->me;
+	int r;
 
-	return amdgpu_sdma_reset_engine(adev, inst_id);
+	r = amdgpu_sdma_reset_engine(adev, inst_id);
+	amdgpu_fence_driver_force_completion(ring);
+	return r;
 }
 
 static int sdma_v5_0_stop_queue(struct amdgpu_ring *ring)
diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c b/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c
index a6e612b4a8928..ee94e457ee25d 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c
@@ -1455,8 +1455,11 @@ static int sdma_v5_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
 {
 	struct amdgpu_device *adev = ring->adev;
 	u32 inst_id = ring->me;
+	int r;
 
-	return amdgpu_sdma_reset_engine(adev, inst_id);
+	r = amdgpu_sdma_reset_engine(adev, inst_id);
+	amdgpu_fence_driver_force_completion(ring);
+	return r;
 }
 
 static int sdma_v5_2_stop_queue(struct amdgpu_ring *ring)
diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
index 5a70ae17be04e..86ef0e30a89e8 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
@@ -1559,6 +1559,7 @@ static int sdma_v6_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
 	if (r)
 		return r;
 
+	amdgpu_fence_driver_force_completion(ring);
 	return sdma_v6_0_gfx_resume_instance(adev, i, true);
 }
 
diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
index befe013b11a78..c9993424b3923 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
@@ -823,6 +823,7 @@ static int sdma_v7_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
 	if (r)
 		return r;
 
+	amdgpu_fence_driver_force_completion(ring);
 	return sdma_v7_0_gfx_resume_instance(adev, i, true);
 }
 
diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
index b5071f77f78d2..f1ccc6c7be390 100644
--- a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
@@ -1977,6 +1977,7 @@ static int vcn_v4_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 
 	vcn_v4_0_stop(vinst);
 	vcn_v4_0_start(vinst);
+	amdgpu_fence_driver_force_completion(ring);
 
 	return amdgpu_ring_test_helper(ring);
 }
diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c
index 5a33140f57235..6ef5944584e56 100644
--- a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c
+++ b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c
@@ -1620,9 +1620,8 @@ static int vcn_v4_0_3_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 		adev->vcn.caps |= AMDGPU_VCN_CAPS(RRMT_ENABLED);
 	vcn_v4_0_3_hw_init_inst(vinst);
 	vcn_v4_0_3_start_dpg_mode(vinst, adev->vcn.inst[ring->me].indirect_sram);
-	r = amdgpu_ring_test_helper(ring);
-
-	return r;
+	amdgpu_fence_driver_force_completion(ring);
+	return amdgpu_ring_test_helper(ring);
 }
 
 static const struct amdgpu_ring_funcs vcn_v4_0_3_unified_ring_vm_funcs = {
diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c
index 16ade84facc78..1c13ceb3bed8b 100644
--- a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c
+++ b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c
@@ -1475,6 +1475,7 @@ static int vcn_v4_0_5_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 
 	vcn_v4_0_5_stop(vinst);
 	vcn_v4_0_5_start(vinst);
+	amdgpu_fence_driver_force_completion(ring);
 
 	return amdgpu_ring_test_helper(ring);
 }
diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c b/drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c
index f8e3f0b882da5..a998708bc0745 100644
--- a/drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c
@@ -1202,6 +1202,7 @@ static int vcn_v5_0_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 
 	vcn_v5_0_0_stop(vinst);
 	vcn_v5_0_0_start(vinst);
+	amdgpu_fence_driver_force_completion(ring);
 
 	return amdgpu_ring_test_helper(ring);
 }
-- 
2.49.0

[PATCH 07/28] drm/amdgpu: track ring state associated with a job

[Alex Deucher]

We need to know the wptr and sequence number associated
with a job so that we can re-emit the unprocessed state
after a ring reset.  Pre-allocate storage space for
the ring buffer contents and add a helper to save off
the unprocessed state so that it can be re-emitted
after the queue is reset.

Add a helper that ring reset callbacks can use to verify
that the ring has reset successfully and to reemit any
unprocessed ring contents from subsequent jobs.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c | 12 ++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c    |  6 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c   |  5 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.h   |  2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c  | 46 +++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h  |  8 ++++
 6 files changed, 78 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
index 2f24a6aa13bf6..319548ac58820 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
@@ -764,6 +764,18 @@ void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring)
 	amdgpu_fence_process(ring);
 }
 
+/**
+ * amdgpu_fence_driver_seq_force_completion - force signal of specified sequence
+ *
+ * @ring: fence of the ring to signal
+ *
+ */
+void amdgpu_fence_driver_seq_force_completion(struct amdgpu_ring *ring, u32 seq)
+{
+	amdgpu_fence_write(ring, seq);
+	amdgpu_fence_process(ring);
+}
+
 /*
  * Common fence implementation
  */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
index 802743efa3b39..67df82d50a74a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
@@ -306,6 +306,12 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned int num_ibs,
 
 	amdgpu_ring_ib_end(ring);
 	amdgpu_ring_commit(ring);
+	/* This must be last for resets to work properly
+	 * as we need to save the wptr associated with this
+	 * job.
+	 */
+	if (job)
+		job->ring_wptr = ring->wptr;
 	return 0;
 }
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
index a0fab947143b5..f0f752284b925 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
@@ -91,6 +91,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
 	struct amdgpu_job *job = to_amdgpu_job(s_job);
 	struct amdgpu_task_info *ti;
 	struct amdgpu_device *adev = ring->adev;
+	struct dma_fence *fence = &job->hw_fence;
 	int idx;
 	int r;
 
@@ -154,8 +155,10 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
 		else
 			is_guilty = true;
 
-		if (is_guilty)
+		if (is_guilty) {
+			amdgpu_ring_backup_unprocessed_jobs(ring, job->ring_wptr, fence->seqno);
 			dma_fence_set_error(&s_job->s_fence->finished, -ETIME);
+		}
 
 		r = amdgpu_ring_reset(ring, job->vmid);
 		if (!r) {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
index f2c049129661f..c2ed0edb5179d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
@@ -79,6 +79,8 @@ struct amdgpu_job {
 	/* enforce isolation */
 	bool			enforce_isolation;
 	bool			run_cleaner_shader;
+	/* wptr for the job for resets */
+	uint32_t		ring_wptr;
 
 	uint32_t		num_ibs;
 	struct amdgpu_ib	ibs[];
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
index 426834806fbf2..909b121d432cb 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
@@ -333,6 +333,12 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
 	/*  Initialize cached_rptr to 0 */
 	ring->cached_rptr = 0;
 
+	if (!ring->ring_backup) {
+		ring->ring_backup = kvzalloc(ring->ring_size, GFP_KERNEL);
+		if (!ring->ring_backup)
+			return -ENOMEM;
+	}
+
 	/* Allocate ring buffer */
 	if (ring->ring_obj == NULL) {
 		r = amdgpu_bo_create_kernel(adev, ring->ring_size + ring->funcs->extra_dw, PAGE_SIZE,
@@ -342,6 +348,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
 					    (void **)&ring->ring);
 		if (r) {
 			dev_err(adev->dev, "(%d) ring create failed\n", r);
+			kvfree(ring->ring_backup);
 			return r;
 		}
 		amdgpu_ring_clear_ring(ring);
@@ -385,6 +392,8 @@ void amdgpu_ring_fini(struct amdgpu_ring *ring)
 	amdgpu_bo_free_kernel(&ring->ring_obj,
 			      &ring->gpu_addr,
 			      (void **)&ring->ring);
+	kvfree(ring->ring_backup);
+	ring->ring_backup = NULL;
 
 	dma_fence_put(ring->vmid_wait);
 	ring->vmid_wait = NULL;
@@ -753,3 +762,40 @@ bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring)
 
 	return true;
 }
+
+void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
+					 u64 bad_wptr, u32 bad_seq)
+{
+	unsigned int entries_to_copy = ring->wptr - bad_wptr;
+	unsigned int idx, i;
+
+	for (i = 0; i < entries_to_copy; i++) {
+		idx = (bad_wptr + i) & ring->buf_mask;
+		ring->ring_backup[i] = ring->ring[idx];
+	}
+	ring->ring_backup_entries_to_copy = entries_to_copy;
+	ring->ring_backup_seq = bad_seq;
+}
+
+int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring)
+{
+	unsigned int i;
+	int r;
+
+	/* signal the fence of the bad job */
+	amdgpu_fence_driver_seq_force_completion(ring, ring->ring_backup_seq);
+	/* verify that the ring is functional */
+	r = amdgpu_ring_test_ring(ring);
+	if (r)
+		return r;
+	/* re-emit the unprocessed ring contents */
+	if (ring->ring_backup_entries_to_copy) {
+		if (amdgpu_ring_alloc(ring, ring->ring_backup_entries_to_copy))
+			return -ENOMEM;
+		for (i = 0; i < ring->ring_backup_entries_to_copy; i++)
+			amdgpu_ring_write(ring, ring->ring_backup[i]);
+		amdgpu_ring_commit(ring);
+	}
+
+	return r;
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
index b95b471107692..fd08449eee33f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
@@ -132,6 +132,8 @@ extern const struct drm_sched_backend_ops amdgpu_sched_ops;
 void amdgpu_fence_driver_clear_job_fences(struct amdgpu_ring *ring);
 void amdgpu_fence_driver_set_error(struct amdgpu_ring *ring, int error);
 void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring);
+void amdgpu_fence_driver_seq_force_completion(struct amdgpu_ring *ring,
+					      u32 seq);
 
 int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring);
 int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring,
@@ -268,6 +270,9 @@ struct amdgpu_ring {
 
 	struct amdgpu_bo	*ring_obj;
 	uint32_t		*ring;
+	uint32_t		*ring_backup;
+	uint32_t		ring_backup_seq;
+	unsigned int		ring_backup_entries_to_copy;
 	unsigned		rptr_offs;
 	u64			rptr_gpu_addr;
 	volatile u32		*rptr_cpu_addr;
@@ -534,4 +539,7 @@ int amdgpu_ib_pool_init(struct amdgpu_device *adev);
 void amdgpu_ib_pool_fini(struct amdgpu_device *adev);
 int amdgpu_ib_ring_tests(struct amdgpu_device *adev);
 bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring);
+void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
+					 u64 bad_wptr, u32 bad_seq);
+int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring);
 #endif
-- 
2.49.0

[PATCH 08/28] drm/amdgpu/gfx10: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 31 +++++---------------------
 1 file changed, 5 insertions(+), 26 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c
index 75ea071744eb5..18ca21d719d2a 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c
@@ -9046,21 +9046,6 @@ static void gfx_v10_0_ring_emit_reg_write_reg_wait(struct amdgpu_ring *ring,
 							   ref, mask);
 }
 
-static void gfx_v10_0_ring_soft_recovery(struct amdgpu_ring *ring,
-					 unsigned int vmid)
-{
-	struct amdgpu_device *adev = ring->adev;
-	uint32_t value = 0;
-
-	value = REG_SET_FIELD(value, SQ_CMD, CMD, 0x03);
-	value = REG_SET_FIELD(value, SQ_CMD, MODE, 0x01);
-	value = REG_SET_FIELD(value, SQ_CMD, CHECK_VMID, 1);
-	value = REG_SET_FIELD(value, SQ_CMD, VM_ID, vmid);
-	amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
-	WREG32_SOC15(GC, 0, mmSQ_CMD, value);
-	amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
-}
-
 static void
 gfx_v10_0_set_gfx_eop_interrupt_state(struct amdgpu_device *adev,
 				      uint32_t me, uint32_t pipe,
@@ -9562,10 +9547,8 @@ static int gfx_v10_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)
 				     SOC15_REG_OFFSET(GC, 0, mmCP_VMID_RESET), 0, 0xffffffff);
 	kiq->pmf->kiq_map_queues(kiq_ring, ring);
 	amdgpu_ring_commit(kiq_ring);
-
-	spin_unlock_irqrestore(&kiq->ring_lock, flags);
-
 	r = amdgpu_ring_test_ring(kiq_ring);
+	spin_unlock_irqrestore(&kiq->ring_lock, flags);
 	if (r)
 		return r;
 
@@ -9575,7 +9558,7 @@ static int gfx_v10_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)
 		return r;
 	}
 
-	return amdgpu_ring_test_ring(ring);
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static int gfx_v10_0_reset_kcq(struct amdgpu_ring *ring,
@@ -9603,9 +9586,8 @@ static int gfx_v10_0_reset_kcq(struct amdgpu_ring *ring,
 	kiq->pmf->kiq_unmap_queues(kiq_ring, ring, RESET_QUEUES,
 				   0, 0);
 	amdgpu_ring_commit(kiq_ring);
-	spin_unlock_irqrestore(&kiq->ring_lock, flags);
-
 	r = amdgpu_ring_test_ring(kiq_ring);
+	spin_unlock_irqrestore(&kiq->ring_lock, flags);
 	if (r)
 		return r;
 
@@ -9641,13 +9623,12 @@ static int gfx_v10_0_reset_kcq(struct amdgpu_ring *ring,
 	}
 	kiq->pmf->kiq_map_queues(kiq_ring, ring);
 	amdgpu_ring_commit(kiq_ring);
-	spin_unlock_irqrestore(&kiq->ring_lock, flags);
-
 	r = amdgpu_ring_test_ring(kiq_ring);
+	spin_unlock_irqrestore(&kiq->ring_lock, flags);
 	if (r)
 		return r;
 
-	return amdgpu_ring_test_ring(ring);
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static void gfx_v10_ip_print(struct amdgpu_ip_block *ip_block, struct drm_printer *p)
@@ -9882,7 +9863,6 @@ static const struct amdgpu_ring_funcs gfx_v10_0_ring_funcs_gfx = {
 	.emit_wreg = gfx_v10_0_ring_emit_wreg,
 	.emit_reg_wait = gfx_v10_0_ring_emit_reg_wait,
 	.emit_reg_write_reg_wait = gfx_v10_0_ring_emit_reg_write_reg_wait,
-	.soft_recovery = gfx_v10_0_ring_soft_recovery,
 	.emit_mem_sync = gfx_v10_0_emit_mem_sync,
 	.reset = gfx_v10_0_reset_kgq,
 	.emit_cleaner_shader = gfx_v10_0_ring_emit_cleaner_shader,
@@ -9923,7 +9903,6 @@ static const struct amdgpu_ring_funcs gfx_v10_0_ring_funcs_compute = {
 	.emit_wreg = gfx_v10_0_ring_emit_wreg,
 	.emit_reg_wait = gfx_v10_0_ring_emit_reg_wait,
 	.emit_reg_write_reg_wait = gfx_v10_0_ring_emit_reg_write_reg_wait,
-	.soft_recovery = gfx_v10_0_ring_soft_recovery,
 	.emit_mem_sync = gfx_v10_0_emit_mem_sync,
 	.reset = gfx_v10_0_reset_kcq,
 	.emit_cleaner_shader = gfx_v10_0_ring_emit_cleaner_shader,
-- 
2.49.0

[PATCH 09/28] drm/amdgpu/gfx11: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c | 21 ++-------------------
 1 file changed, 2 insertions(+), 19 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
index afd6d59164bfa..4e0a6dcce91fa 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
@@ -6278,21 +6278,6 @@ static void gfx_v11_0_ring_emit_reg_write_reg_wait(struct amdgpu_ring *ring,
 			       ref, mask, 0x20);
 }
 
-static void gfx_v11_0_ring_soft_recovery(struct amdgpu_ring *ring,
-					 unsigned vmid)
-{
-	struct amdgpu_device *adev = ring->adev;
-	uint32_t value = 0;
-
-	value = REG_SET_FIELD(value, SQ_CMD, CMD, 0x03);
-	value = REG_SET_FIELD(value, SQ_CMD, MODE, 0x01);
-	value = REG_SET_FIELD(value, SQ_CMD, CHECK_VMID, 1);
-	value = REG_SET_FIELD(value, SQ_CMD, VM_ID, vmid);
-	amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
-	WREG32_SOC15(GC, 0, regSQ_CMD, value);
-	amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
-}
-
 static void
 gfx_v11_0_set_gfx_eop_interrupt_state(struct amdgpu_device *adev,
 				      uint32_t me, uint32_t pipe,
@@ -6835,7 +6820,7 @@ static int gfx_v11_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)
 		return r;
 	}
 
-	return amdgpu_ring_test_ring(ring);
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static int gfx_v11_0_reset_compute_pipe(struct amdgpu_ring *ring)
@@ -6995,7 +6980,7 @@ static int gfx_v11_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid)
 		return r;
 	}
 
-	return amdgpu_ring_test_ring(ring);
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static void gfx_v11_ip_print(struct amdgpu_ip_block *ip_block, struct drm_printer *p)
@@ -7231,7 +7216,6 @@ static const struct amdgpu_ring_funcs gfx_v11_0_ring_funcs_gfx = {
 	.emit_wreg = gfx_v11_0_ring_emit_wreg,
 	.emit_reg_wait = gfx_v11_0_ring_emit_reg_wait,
 	.emit_reg_write_reg_wait = gfx_v11_0_ring_emit_reg_write_reg_wait,
-	.soft_recovery = gfx_v11_0_ring_soft_recovery,
 	.emit_mem_sync = gfx_v11_0_emit_mem_sync,
 	.reset = gfx_v11_0_reset_kgq,
 	.emit_cleaner_shader = gfx_v11_0_ring_emit_cleaner_shader,
@@ -7273,7 +7257,6 @@ static const struct amdgpu_ring_funcs gfx_v11_0_ring_funcs_compute = {
 	.emit_wreg = gfx_v11_0_ring_emit_wreg,
 	.emit_reg_wait = gfx_v11_0_ring_emit_reg_wait,
 	.emit_reg_write_reg_wait = gfx_v11_0_ring_emit_reg_write_reg_wait,
-	.soft_recovery = gfx_v11_0_ring_soft_recovery,
 	.emit_mem_sync = gfx_v11_0_emit_mem_sync,
 	.reset = gfx_v11_0_reset_kcq,
 	.emit_cleaner_shader = gfx_v11_0_ring_emit_cleaner_shader,
-- 
2.49.0

[PATCH 10/28] drm/amdgpu/gfx12: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c | 21 ++-------------------
 1 file changed, 2 insertions(+), 19 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
index f09d96bfee16d..50352b966f4ad 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
@@ -4690,21 +4690,6 @@ static void gfx_v12_0_ring_emit_reg_write_reg_wait(struct amdgpu_ring *ring,
 			       ref, mask, 0x20);
 }
 
-static void gfx_v12_0_ring_soft_recovery(struct amdgpu_ring *ring,
-					 unsigned vmid)
-{
-	struct amdgpu_device *adev = ring->adev;
-	uint32_t value = 0;
-
-	value = REG_SET_FIELD(value, SQ_CMD, CMD, 0x03);
-	value = REG_SET_FIELD(value, SQ_CMD, MODE, 0x01);
-	value = REG_SET_FIELD(value, SQ_CMD, CHECK_VMID, 1);
-	value = REG_SET_FIELD(value, SQ_CMD, VM_ID, vmid);
-	amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
-	WREG32_SOC15(GC, 0, regSQ_CMD, value);
-	amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
-}
-
 static void
 gfx_v12_0_set_gfx_eop_interrupt_state(struct amdgpu_device *adev,
 				      uint32_t me, uint32_t pipe,
@@ -5335,7 +5320,7 @@ static int gfx_v12_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)
 		return r;
 	}
 
-	return amdgpu_ring_test_ring(ring);
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static int gfx_v12_0_reset_compute_pipe(struct amdgpu_ring *ring)
@@ -5448,7 +5433,7 @@ static int gfx_v12_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid)
 		return r;
 	}
 
-	return amdgpu_ring_test_ring(ring);
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static void gfx_v12_0_ring_begin_use(struct amdgpu_ring *ring)
@@ -5526,7 +5511,6 @@ static const struct amdgpu_ring_funcs gfx_v12_0_ring_funcs_gfx = {
 	.emit_wreg = gfx_v12_0_ring_emit_wreg,
 	.emit_reg_wait = gfx_v12_0_ring_emit_reg_wait,
 	.emit_reg_write_reg_wait = gfx_v12_0_ring_emit_reg_write_reg_wait,
-	.soft_recovery = gfx_v12_0_ring_soft_recovery,
 	.emit_mem_sync = gfx_v12_0_emit_mem_sync,
 	.reset = gfx_v12_0_reset_kgq,
 	.emit_cleaner_shader = gfx_v12_0_ring_emit_cleaner_shader,
@@ -5565,7 +5549,6 @@ static const struct amdgpu_ring_funcs gfx_v12_0_ring_funcs_compute = {
 	.emit_wreg = gfx_v12_0_ring_emit_wreg,
 	.emit_reg_wait = gfx_v12_0_ring_emit_reg_wait,
 	.emit_reg_write_reg_wait = gfx_v12_0_ring_emit_reg_write_reg_wait,
-	.soft_recovery = gfx_v12_0_ring_soft_recovery,
 	.emit_mem_sync = gfx_v12_0_emit_mem_sync,
 	.reset = gfx_v12_0_reset_kcq,
 	.emit_cleaner_shader = gfx_v12_0_ring_emit_cleaner_shader,
-- 
2.49.0

[PATCH 11/28] drm/amdgpu/gfx9: re-emit unprocessed state on kcq reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
index d50e125fd3e0d..0bf8613b937f1 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
@@ -7216,13 +7216,14 @@ static int gfx_v9_0_reset_kcq(struct amdgpu_ring *ring,
 	}
 	kiq->pmf->kiq_map_queues(kiq_ring, ring);
 	amdgpu_ring_commit(kiq_ring);
-	spin_unlock_irqrestore(&kiq->ring_lock, flags);
 	r = amdgpu_ring_test_ring(kiq_ring);
+	spin_unlock_irqrestore(&kiq->ring_lock, flags);
 	if (r) {
 		DRM_ERROR("fail to remap queue\n");
 		return r;
 	}
-	return amdgpu_ring_test_ring(ring);
+
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static void gfx_v9_ip_print(struct amdgpu_ip_block *ip_block, struct drm_printer *p)
-- 
2.49.0

[PATCH 12/28] drm/amdgpu/gfx9.4.3: re-emit unprocessed state on kcq reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c
index c233edf605694..409cfdfe54ac3 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c
@@ -3612,14 +3612,14 @@ static int gfx_v9_4_3_reset_kcq(struct amdgpu_ring *ring,
 	}
 	kiq->pmf->kiq_map_queues(kiq_ring, ring);
 	amdgpu_ring_commit(kiq_ring);
-	spin_unlock_irqrestore(&kiq->ring_lock, flags);
-
 	r = amdgpu_ring_test_ring(kiq_ring);
+	spin_unlock_irqrestore(&kiq->ring_lock, flags);
 	if (r) {
 		dev_err(adev->dev, "fail to remap queue\n");
 		return r;
 	}
-	return amdgpu_ring_test_ring(ring);
+
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 enum amdgpu_gfx_cp_ras_mem_id {
-- 
2.49.0

[PATCH 13/28] drm/amdgpu/sdma5: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c
index 43885dc04b632..44c0eb8536d7d 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c
@@ -1545,8 +1545,10 @@ static int sdma_v5_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
 	int r;
 
 	r = amdgpu_sdma_reset_engine(adev, inst_id);
-	amdgpu_fence_driver_force_completion(ring);
-	return r;
+	if (r)
+		return r;
+
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static int sdma_v5_0_stop_queue(struct amdgpu_ring *ring)
-- 
2.49.0

[PATCH 14/28] drm/amdgpu/sdma5.2: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c b/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c
index ee94e457ee25d..64ccc2c09fa12 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c
@@ -1458,8 +1458,10 @@ static int sdma_v5_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
 	int r;
 
 	r = amdgpu_sdma_reset_engine(adev, inst_id);
-	amdgpu_fence_driver_force_completion(ring);
-	return r;
+	if (r)
+		return r;
+
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static int sdma_v5_2_stop_queue(struct amdgpu_ring *ring)
-- 
2.49.0

[PATCH 15/28] drm/amdgpu/sdma6: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
index 86ef0e30a89e8..09c4e3b228632 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
@@ -1559,8 +1559,11 @@ static int sdma_v6_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
 	if (r)
 		return r;
 
-	amdgpu_fence_driver_force_completion(ring);
-	return sdma_v6_0_gfx_resume_instance(adev, i, true);
+	r = sdma_v6_0_gfx_resume_instance(adev, i, true);
+	if (r)
+		return r;
+
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static int sdma_v6_0_set_trap_irq_state(struct amdgpu_device *adev,
-- 
2.49.0

[PATCH 16/28] drm/amdgpu/sdma7: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
index c9993424b3923..808fc658dc040 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
@@ -823,8 +823,11 @@ static int sdma_v7_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
 	if (r)
 		return r;
 
-	amdgpu_fence_driver_force_completion(ring);
-	return sdma_v7_0_gfx_resume_instance(adev, i, true);
+	r = sdma_v7_0_gfx_resume_instance(adev, i, true);
+	if (r)
+		return r;
+
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 /**
-- 
2.49.0

[PATCH 17/28] drm/amdgpu/jpeg2: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c
index 336ca44950dfa..aa3f587e2004d 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c
@@ -768,8 +768,8 @@ static int jpeg_v2_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 {
 	jpeg_v2_0_stop(ring->adev);
 	jpeg_v2_0_start(ring->adev);
-	amdgpu_fence_driver_force_completion(ring);
-	return amdgpu_ring_test_helper(ring);
+
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static const struct amd_ip_funcs jpeg_v2_0_ip_funcs = {
-- 
2.49.0

[PATCH 18/28] drm/amdgpu/jpeg2.5: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c
index 72d739e38a8ae..7105c5b995275 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c
@@ -647,8 +647,8 @@ static int jpeg_v2_5_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 {
 	jpeg_v2_5_stop_inst(ring->adev, ring->me);
 	jpeg_v2_5_start_inst(ring->adev, ring->me);
-	amdgpu_fence_driver_force_completion(ring);
-	return amdgpu_ring_test_helper(ring);
+
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static const struct amd_ip_funcs jpeg_v2_5_ip_funcs = {
-- 
2.49.0

[PATCH 19/28] drm/amdgpu/jpeg3: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c
index 7974d04c15e92..382126f3caae9 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c
@@ -559,8 +559,8 @@ static int jpeg_v3_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 {
 	jpeg_v3_0_stop(ring->adev);
 	jpeg_v3_0_start(ring->adev);
-	amdgpu_fence_driver_force_completion(ring);
-	return amdgpu_ring_test_helper(ring);
+
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static const struct amd_ip_funcs jpeg_v3_0_ip_funcs = {
-- 
2.49.0

[PATCH 20/28] drm/amdgpu/jpeg4: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c
index 9301d0ff2593b..4e644b645bd25 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c
@@ -727,8 +727,8 @@ static int jpeg_v4_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 
 	jpeg_v4_0_stop(ring->adev);
 	jpeg_v4_0_start(ring->adev);
-	amdgpu_fence_driver_force_completion(ring);
-	return amdgpu_ring_test_helper(ring);
+
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static const struct amd_ip_funcs jpeg_v4_0_ip_funcs = {
-- 
2.49.0

[PATCH 21/28] drm/amdgpu/jpeg4.0.3: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c
index ef5075e111300..986fcfb2ca726 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c
@@ -1150,8 +1150,8 @@ static int jpeg_v4_0_3_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 
 	jpeg_v4_0_3_core_stall_reset(ring);
 	jpeg_v4_0_3_start_jrbc(ring);
-	amdgpu_fence_driver_force_completion(ring);
-	return amdgpu_ring_test_helper(ring);
+
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static const struct amd_ip_funcs jpeg_v4_0_3_ip_funcs = {
-- 
2.49.0

[PATCH 22/28] drm/amdgpu/jpeg5.0.0: add queue reset

[Alex Deucher]

Add queue reset support for jpeg 5.0.0.

Untested.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_0.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_0.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_0.c
index 31d213ccbe0a8..231639f2ef0d1 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_0.c
@@ -644,6 +644,17 @@ static int jpeg_v5_0_0_process_interrupt(struct amdgpu_device *adev,
 	return 0;
 }
 
+static int jpeg_v5_0_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
+{
+	if (amdgpu_sriov_vf(ring->adev))
+		return -EINVAL;
+
+	jpeg_v5_0_0_stop(ring->adev);
+	jpeg_v5_0_0_start(ring->adev);
+	amdgpu_fence_driver_force_completion(ring);
+	return amdgpu_ring_test_helper(ring);
+}
+
 static const struct amd_ip_funcs jpeg_v5_0_0_ip_funcs = {
 	.name = "jpeg_v5_0_0",
 	.early_init = jpeg_v5_0_0_early_init,
@@ -689,6 +700,7 @@ static const struct amdgpu_ring_funcs jpeg_v5_0_0_dec_ring_vm_funcs = {
 	.emit_wreg = jpeg_v4_0_3_dec_ring_emit_wreg,
 	.emit_reg_wait = jpeg_v4_0_3_dec_ring_emit_reg_wait,
 	.emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper,
+	.reset = jpeg_v5_0_0_ring_reset,
 };
 
 static void jpeg_v5_0_0_set_dec_ring_funcs(struct amdgpu_device *adev)
-- 
2.49.0

[PATCH 23/28] drm/amdgpu/jpeg5: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_0.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_0.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_0.c
index 231639f2ef0d1..89f3cdb8616f5 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_0.c
@@ -651,8 +651,8 @@ static int jpeg_v5_0_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 
 	jpeg_v5_0_0_stop(ring->adev);
 	jpeg_v5_0_0_start(ring->adev);
-	amdgpu_fence_driver_force_completion(ring);
-	return amdgpu_ring_test_helper(ring);
+
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static const struct amd_ip_funcs jpeg_v5_0_0_ip_funcs = {
-- 
2.49.0

[PATCH 24/28] drm/amdgpu/jpeg5.0.1: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c
index 858b930d1dd54..56112a319c375 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c
@@ -841,8 +841,8 @@ static int jpeg_v5_0_1_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 
 	jpeg_v5_0_1_core_stall_reset(ring);
 	jpeg_v5_0_1_init_jrbc(ring);
-	amdgpu_fence_driver_force_completion(ring);
-	return amdgpu_ring_test_helper(ring);
+
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static const struct amd_ip_funcs jpeg_v5_0_1_ip_funcs = {
-- 
2.49.0

[PATCH 25/28] drm/amdgpu/vcn4: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
index f1ccc6c7be390..04b4712a8f2e2 100644
--- a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
@@ -1977,9 +1977,8 @@ static int vcn_v4_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 
 	vcn_v4_0_stop(vinst);
 	vcn_v4_0_start(vinst);
-	amdgpu_fence_driver_force_completion(ring);
 
-	return amdgpu_ring_test_helper(ring);
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = {
-- 
2.49.0

[PATCH 26/28] drm/amdgpu/vcn4.0.3: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c
index 6ef5944584e56..aa0ae2cd65da4 100644
--- a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c
+++ b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c
@@ -1620,8 +1620,8 @@ static int vcn_v4_0_3_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 		adev->vcn.caps |= AMDGPU_VCN_CAPS(RRMT_ENABLED);
 	vcn_v4_0_3_hw_init_inst(vinst);
 	vcn_v4_0_3_start_dpg_mode(vinst, adev->vcn.inst[ring->me].indirect_sram);
-	amdgpu_fence_driver_force_completion(ring);
-	return amdgpu_ring_test_helper(ring);
+
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static const struct amdgpu_ring_funcs vcn_v4_0_3_unified_ring_vm_funcs = {
-- 
2.49.0

[PATCH 27/28] drm/amdgpu/vcn4.0.5: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c
index 1c13ceb3bed8b..311d44edff224 100644
--- a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c
+++ b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c
@@ -1475,9 +1475,8 @@ static int vcn_v4_0_5_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 
 	vcn_v4_0_5_stop(vinst);
 	vcn_v4_0_5_start(vinst);
-	amdgpu_fence_driver_force_completion(ring);
 
-	return amdgpu_ring_test_helper(ring);
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static struct amdgpu_ring_funcs vcn_v4_0_5_unified_ring_vm_funcs = {
-- 
2.49.0

[PATCH 28/28] drm/amdgpu/vcn5: re-emit unprocessed state on ring reset

[Alex Deucher]

Re-emit the unprocessed state after resetting the queue.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c b/drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c
index a998708bc0745..3cc49c82e324a 100644
--- a/drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c
@@ -1202,9 +1202,8 @@ static int vcn_v5_0_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
 
 	vcn_v5_0_0_stop(vinst);
 	vcn_v5_0_0_start(vinst);
-	amdgpu_fence_driver_force_completion(ring);
 
-	return amdgpu_ring_test_helper(ring);
+	return amdgpu_ring_reemit_unprocessed_jobs(ring);
 }
 
 static const struct amdgpu_ring_funcs vcn_v5_0_0_unified_ring_vm_funcs = {
-- 
2.49.0

Re: [PATCH V5 00/28] Reset improvements for GC10+

[Alex Deucher]

On Thu, May 29, 2025 at 4:08 PM Alex Deucher <alexander.deucher@amd.com> wrote:
>
> This set improves per queue reset support for GC10+.
> When we reset the queue, the queue is lost so we need
> to re-emit the unprocessed state from subsequent submissions.
> To that end, in order to make sure we actually restore
> unprocessed state, we need to enable legacy enforce isolation
> so that we can safely re-emit the unprocessed state.  If
> we don't multiple jobs can run in parallel and we may not
> end up resetting the correct one.  This is similar to how
> windows handles queues.  This also gives us correct guilty
> tracking for GC.
>
> Tested on GC 10 and 11 chips with a game running and
> then running hang tests.  The game pauses when the
> hang happens, then continues after the queue reset.
>
> I tried this same approach and GC8 and 9, but it
> was not as reliable as soft recovery.  As such, I've dropped
> the KGQ reset code for pre-GC10.
>
> The same approach is extended to SDMA and VCN.
> They don't need enforce isolation because those engines
> are single threaded so they always operate serially.
>
> Rework re-emit to signal the seq number of the bad job and
> verify that to verify that the reset worked, then re-emit the
> rest of the non-guilty state.  This way we are not waiting on
> the rest of the state to complete, and if the subsequent state
> also contains a bad job, we'll end up in queue reset again rather
> than adapter reset.

git tree available here:
https://gitlab.freedesktop.org/agd5f/linux/-/commits/kq_resets?ref_type=heads

Alex

>
> v4: Drop explicit padding patches
>     Drop new timeout macro
>     Rework re-emit sequence
> v5: Add a helper for reemit
>     Convert VCN, JPEG, SDMA to use new helpers
>
> Alex Deucher (27):
>   drm/amdgpu: enable legacy enforce isolation by default
>   drm/amdgpu/gfx7: drop reset_kgq
>   drm/amdgpu/gfx8: drop reset_kgq
>   drm/amdgpu/gfx9: drop reset_kgq
>   drm/amdgpu: move force completion into ring resets
>   drm/amdgpu: track ring state associated with a job
>   drm/amdgpu/gfx10: re-emit unprocessed state on ring reset
>   drm/amdgpu/gfx11: re-emit unprocessed state on ring reset
>   drm/amdgpu/gfx12: re-emit unprocessed state on ring reset
>   drm/amdgpu/gfx9: re-emit unprocessed state on kcq reset
>   drm/amdgpu/gfx9.4.3: re-emit unprocessed state on kcq reset
>   drm/amdgpu/sdma5: re-emit unprocessed state on ring reset
>   drm/amdgpu/sdma5.2: re-emit unprocessed state on ring reset
>   drm/amdgpu/sdma6: re-emit unprocessed state on ring reset
>   drm/amdgpu/sdma7: re-emit unprocessed state on ring reset
>   drm/amdgpu/jpeg2: re-emit unprocessed state on ring reset
>   drm/amdgpu/jpeg2.5: re-emit unprocessed state on ring reset
>   drm/amdgpu/jpeg3: re-emit unprocessed state on ring reset
>   drm/amdgpu/jpeg4: re-emit unprocessed state on ring reset
>   drm/amdgpu/jpeg4.0.3: re-emit unprocessed state on ring reset
>   drm/amdgpu/jpeg5.0.0: add queue reset
>   drm/amdgpu/jpeg5: re-emit unprocessed state on ring reset
>   drm/amdgpu/jpeg5.0.1: re-emit unprocessed state on ring reset
>   drm/amdgpu/vcn4: re-emit unprocessed state on ring reset
>   drm/amdgpu/vcn4.0.3: re-emit unprocessed state on ring reset
>   drm/amdgpu/vcn4.0.5: re-emit unprocessed state on ring reset
>   drm/amdgpu/vcn5: re-emit unprocessed state on ring reset
>
> Christian König (1):
>   drm/amdgpu: rework queue reset scheduler interaction
>
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  4 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c  | 12 ++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c     |  6 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    | 32 +++++-----
>  drivers/gpu/drm/amd/amdgpu/amdgpu_job.h    |  2 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c   | 46 ++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h   |  8 +++
>  drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c     | 31 ++--------
>  drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c     | 21 +------
>  drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c     | 21 +------
>  drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c      | 71 ----------------------
>  drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c      | 71 ----------------------
>  drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c      | 51 +---------------
>  drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c    |  6 +-
>  drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c     |  3 +-
>  drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c     |  3 +-
>  drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c     |  3 +-
>  drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c     |  3 +-
>  drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c   |  3 +-
>  drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_0.c   | 12 ++++
>  drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c   |  3 +-
>  drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c   |  4 ++
>  drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c     |  7 ++-
>  drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c     |  7 ++-
>  drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c     |  6 +-
>  drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c     |  6 +-
>  drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c      |  2 +-
>  drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c    |  3 +-
>  drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c    |  2 +-
>  drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c    |  2 +-
>  30 files changed, 162 insertions(+), 289 deletions(-)
>
> --
> 2.49.0
>

Re: [PATCH V5 00/28] Reset improvements for GC10+

[Alex Deucher]

On Thu, May 29, 2025 at 4:54 PM Alex Deucher <alexdeucher@gmail.com> wrote:
>
> On Thu, May 29, 2025 at 4:08 PM Alex Deucher <alexander.deucher@amd.com> wrote:
> >
> > This set improves per queue reset support for GC10+.
> > When we reset the queue, the queue is lost so we need
> > to re-emit the unprocessed state from subsequent submissions.
> > To that end, in order to make sure we actually restore
> > unprocessed state, we need to enable legacy enforce isolation
> > so that we can safely re-emit the unprocessed state.  If
> > we don't multiple jobs can run in parallel and we may not
> > end up resetting the correct one.  This is similar to how
> > windows handles queues.  This also gives us correct guilty
> > tracking for GC.
> >
> > Tested on GC 10 and 11 chips with a game running and
> > then running hang tests.  The game pauses when the
> > hang happens, then continues after the queue reset.
> >
> > I tried this same approach and GC8 and 9, but it
> > was not as reliable as soft recovery.  As such, I've dropped
> > the KGQ reset code for pre-GC10.
> >
> > The same approach is extended to SDMA and VCN.
> > They don't need enforce isolation because those engines
> > are single threaded so they always operate serially.
> >
> > Rework re-emit to signal the seq number of the bad job and
> > verify that to verify that the reset worked, then re-emit the
> > rest of the non-guilty state.  This way we are not waiting on
> > the rest of the state to complete, and if the subsequent state
> > also contains a bad job, we'll end up in queue reset again rather
> > than adapter reset.
>
> git tree available here:
> https://gitlab.freedesktop.org/agd5f/linux/-/commits/kq_resets?ref_type=heads

I've pushed several fixes since I last sent this and will continue to
push updates.

Alex

>
> Alex
>
> >
> > v4: Drop explicit padding patches
> >     Drop new timeout macro
> >     Rework re-emit sequence
> > v5: Add a helper for reemit
> >     Convert VCN, JPEG, SDMA to use new helpers
> >
> > Alex Deucher (27):
> >   drm/amdgpu: enable legacy enforce isolation by default
> >   drm/amdgpu/gfx7: drop reset_kgq
> >   drm/amdgpu/gfx8: drop reset_kgq
> >   drm/amdgpu/gfx9: drop reset_kgq
> >   drm/amdgpu: move force completion into ring resets
> >   drm/amdgpu: track ring state associated with a job
> >   drm/amdgpu/gfx10: re-emit unprocessed state on ring reset
> >   drm/amdgpu/gfx11: re-emit unprocessed state on ring reset
> >   drm/amdgpu/gfx12: re-emit unprocessed state on ring reset
> >   drm/amdgpu/gfx9: re-emit unprocessed state on kcq reset
> >   drm/amdgpu/gfx9.4.3: re-emit unprocessed state on kcq reset
> >   drm/amdgpu/sdma5: re-emit unprocessed state on ring reset
> >   drm/amdgpu/sdma5.2: re-emit unprocessed state on ring reset
> >   drm/amdgpu/sdma6: re-emit unprocessed state on ring reset
> >   drm/amdgpu/sdma7: re-emit unprocessed state on ring reset
> >   drm/amdgpu/jpeg2: re-emit unprocessed state on ring reset
> >   drm/amdgpu/jpeg2.5: re-emit unprocessed state on ring reset
> >   drm/amdgpu/jpeg3: re-emit unprocessed state on ring reset
> >   drm/amdgpu/jpeg4: re-emit unprocessed state on ring reset
> >   drm/amdgpu/jpeg4.0.3: re-emit unprocessed state on ring reset
> >   drm/amdgpu/jpeg5.0.0: add queue reset
> >   drm/amdgpu/jpeg5: re-emit unprocessed state on ring reset
> >   drm/amdgpu/jpeg5.0.1: re-emit unprocessed state on ring reset
> >   drm/amdgpu/vcn4: re-emit unprocessed state on ring reset
> >   drm/amdgpu/vcn4.0.3: re-emit unprocessed state on ring reset
> >   drm/amdgpu/vcn4.0.5: re-emit unprocessed state on ring reset
> >   drm/amdgpu/vcn5: re-emit unprocessed state on ring reset
> >
> > Christian König (1):
> >   drm/amdgpu: rework queue reset scheduler interaction
> >
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  4 +-
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c  | 12 ++++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c     |  6 ++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    | 32 +++++-----
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_job.h    |  2 +
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c   | 46 ++++++++++++++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h   |  8 +++
> >  drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c     | 31 ++--------
> >  drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c     | 21 +------
> >  drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c     | 21 +------
> >  drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c      | 71 ----------------------
> >  drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c      | 71 ----------------------
> >  drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c      | 51 +---------------
> >  drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c    |  6 +-
> >  drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c     |  3 +-
> >  drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c     |  3 +-
> >  drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c     |  3 +-
> >  drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c     |  3 +-
> >  drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c   |  3 +-
> >  drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_0.c   | 12 ++++
> >  drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c   |  3 +-
> >  drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c   |  4 ++
> >  drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c     |  7 ++-
> >  drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c     |  7 ++-
> >  drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c     |  6 +-
> >  drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c     |  6 +-
> >  drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c      |  2 +-
> >  drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c    |  3 +-
> >  drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c    |  2 +-
> >  drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c    |  2 +-
> >  30 files changed, 162 insertions(+), 289 deletions(-)
> >
> > --
> > 2.49.0
> >

Re: [PATCH 01/28] drm/amdgpu: enable legacy enforce isolation by default

[Christian König]

On 5/29/25 22:07, Alex Deucher wrote:
> Enable legacy enforce isolation (just serialize kernel
> GC submissions).  This way we can reset a ring and
> only affect the the process currently using that ring.
> This mirrors what windows does.
> 
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

Yeah, that's exactly what I wanted to suggest after seeing that you wanted to do that per GFX IP version.

Reviewed-by: Christian König <christian.koenig@amd.com>

But we might run into complains if people used compute for background jobs for some reason.

Regards,
Christian.

> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 72e41781afb06..ca882e3d95bb7 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -2148,9 +2148,7 @@ static int amdgpu_device_check_arguments(struct amdgpu_device *adev)
>  
>  	for (i = 0; i < MAX_XCP; i++) {
>  		switch (amdgpu_enforce_isolation) {
> -		case -1:
>  		case 0:
> -		default:
>  			/* disable */
>  			adev->enforce_isolation[i] = AMDGPU_ENFORCE_ISOLATION_DISABLE;
>  			break;
> @@ -2159,7 +2157,9 @@ static int amdgpu_device_check_arguments(struct amdgpu_device *adev)
>  			adev->enforce_isolation[i] =
>  				AMDGPU_ENFORCE_ISOLATION_ENABLE;
>  			break;
> +		case -1:
>  		case 2:
> +		default:
>  			/* enable legacy mode */
>  			adev->enforce_isolation[i] =
>  				AMDGPU_ENFORCE_ISOLATION_ENABLE_LEGACY;

Re: [PATCH 02/28] drm/amdgpu/gfx7: drop reset_kgq

[Christian König]

On 5/29/25 22:07, Alex Deucher wrote:
> It doesn't work reliably and we have soft recover and
> full adapter reset so drop this.
> 
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

I was really hoping that this would be more reliable than soft recovery at some point. And from my testing on gfx9 it seems to work rather reliable.

Are you sure we should just drop that?

Regards,
Christian.

> ---
>  drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c | 71 ---------------------------
>  1 file changed, 71 deletions(-)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c
> index da0534ff1271a..2aa323dab34e3 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c
> @@ -4884,76 +4884,6 @@ static void gfx_v7_0_emit_mem_sync_compute(struct amdgpu_ring *ring)
>  	amdgpu_ring_write(ring, 0x0000000A);	/* poll interval */
>  }
>  
> -static void gfx_v7_0_wait_reg_mem(struct amdgpu_ring *ring, int eng_sel,
> -				  int mem_space, int opt, uint32_t addr0,
> -				  uint32_t addr1, uint32_t ref, uint32_t mask,
> -				  uint32_t inv)
> -{
> -	amdgpu_ring_write(ring, PACKET3(PACKET3_WAIT_REG_MEM, 5));
> -	amdgpu_ring_write(ring,
> -			  /* memory (1) or register (0) */
> -			  (WAIT_REG_MEM_MEM_SPACE(mem_space) |
> -			   WAIT_REG_MEM_OPERATION(opt) | /* wait */
> -			   WAIT_REG_MEM_FUNCTION(3) |  /* equal */
> -			   WAIT_REG_MEM_ENGINE(eng_sel)));
> -
> -	if (mem_space)
> -		BUG_ON(addr0 & 0x3); /* Dword align */
> -	amdgpu_ring_write(ring, addr0);
> -	amdgpu_ring_write(ring, addr1);
> -	amdgpu_ring_write(ring, ref);
> -	amdgpu_ring_write(ring, mask);
> -	amdgpu_ring_write(ring, inv); /* poll interval */
> -}
> -
> -static void gfx_v7_0_ring_emit_reg_wait(struct amdgpu_ring *ring, uint32_t reg,
> -					uint32_t val, uint32_t mask)
> -{
> -	gfx_v7_0_wait_reg_mem(ring, 0, 0, 0, reg, 0, val, mask, 0x20);
> -}
> -
> -static int gfx_v7_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)
> -{
> -	struct amdgpu_device *adev = ring->adev;
> -	struct amdgpu_kiq *kiq = &adev->gfx.kiq[0];
> -	struct amdgpu_ring *kiq_ring = &kiq->ring;
> -	unsigned long flags;
> -	u32 tmp;
> -	int r;
> -
> -	if (amdgpu_sriov_vf(adev))
> -		return -EINVAL;
> -
> -	if (!kiq->pmf || !kiq->pmf->kiq_unmap_queues)
> -		return -EINVAL;
> -
> -	spin_lock_irqsave(&kiq->ring_lock, flags);
> -
> -	if (amdgpu_ring_alloc(kiq_ring, 5)) {
> -		spin_unlock_irqrestore(&kiq->ring_lock, flags);
> -		return -ENOMEM;
> -	}
> -
> -	tmp = REG_SET_FIELD(0, CP_VMID_RESET, RESET_REQUEST, 1 << vmid);
> -	gfx_v7_0_ring_emit_wreg(kiq_ring, mmCP_VMID_RESET, tmp);
> -	amdgpu_ring_commit(kiq_ring);
> -
> -	spin_unlock_irqrestore(&kiq->ring_lock, flags);
> -
> -	r = amdgpu_ring_test_ring(kiq_ring);
> -	if (r)
> -		return r;
> -
> -	if (amdgpu_ring_alloc(ring, 7 + 12 + 5))
> -		return -ENOMEM;
> -	gfx_v7_0_ring_emit_fence_gfx(ring, ring->fence_drv.gpu_addr,
> -				     ring->fence_drv.sync_seq, AMDGPU_FENCE_FLAG_EXEC);
> -	gfx_v7_0_ring_emit_reg_wait(ring, mmCP_VMID_RESET, 0, 0xffff);
> -	gfx_v7_0_ring_emit_wreg(ring, mmCP_VMID_RESET, 0);
> -
> -	return amdgpu_ring_test_ring(ring);
> -}
> -
>  static const struct amd_ip_funcs gfx_v7_0_ip_funcs = {
>  	.name = "gfx_v7_0",
>  	.early_init = gfx_v7_0_early_init,
> @@ -5003,7 +4933,6 @@ static const struct amdgpu_ring_funcs gfx_v7_0_ring_funcs_gfx = {
>  	.emit_wreg = gfx_v7_0_ring_emit_wreg,
>  	.soft_recovery = gfx_v7_0_ring_soft_recovery,
>  	.emit_mem_sync = gfx_v7_0_emit_mem_sync,
> -	.reset = gfx_v7_0_reset_kgq,
>  };
>  
>  static const struct amdgpu_ring_funcs gfx_v7_0_ring_funcs_compute = {

Re: [PATCH 06/28] drm/amdgpu: move force completion into ring resets

[Christian König]

On 5/29/25 22:07, Alex Deucher wrote:
> Move the force completion handling into each ring
> reset function so that each engine can determine
> whether or not it needs to force completion on the
> jobs in the ring.
> 
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

Reviewed-by: Christian König <christian.koenig@amd.com>

> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c  | 6 +-----
>  drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c   | 1 +
>  drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c   | 1 +
>  drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c   | 1 +
>  drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c   | 1 +
>  drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c | 1 +
>  drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c | 1 +
>  drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c | 4 ++++
>  drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c   | 5 ++++-
>  drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c   | 5 ++++-
>  drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c   | 1 +
>  drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c   | 1 +
>  drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c    | 1 +
>  drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c  | 5 ++---
>  drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c  | 1 +
>  drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c  | 1 +
>  16 files changed, 26 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> index 0b9086a747c0a..a0fab947143b5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> @@ -159,12 +159,8 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
>  
>  		r = amdgpu_ring_reset(ring, job->vmid);
>  		if (!r) {
> -			if (is_guilty) {
> +			if (is_guilty)
>  				atomic_inc(&ring->adev->gpu_reset_counter);
> -				if ((ring->funcs->type != AMDGPU_RING_TYPE_GFX) &&
> -				    (ring->funcs->type != AMDGPU_RING_TYPE_COMPUTE))
> -					amdgpu_fence_driver_force_completion(ring);
> -			}
>  			drm_sched_wqueue_start(&ring->sched);
>  			dev_err(adev->dev, "Ring %s reset succeeded\n",
>  				ring->sched.name);
> diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c
> index 4cde8a8bcc837..336ca44950dfa 100644
> --- a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c
> @@ -768,6 +768,7 @@ static int jpeg_v2_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
>  {
>  	jpeg_v2_0_stop(ring->adev);
>  	jpeg_v2_0_start(ring->adev);
> +	amdgpu_fence_driver_force_completion(ring);
>  	return amdgpu_ring_test_helper(ring);
>  }
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c
> index 8b39e114f3be1..72d739e38a8ae 100644
> --- a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c
> +++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c
> @@ -647,6 +647,7 @@ static int jpeg_v2_5_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
>  {
>  	jpeg_v2_5_stop_inst(ring->adev, ring->me);
>  	jpeg_v2_5_start_inst(ring->adev, ring->me);
> +	amdgpu_fence_driver_force_completion(ring);
>  	return amdgpu_ring_test_helper(ring);
>  }
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c
> index 2f8510c2986b9..7974d04c15e92 100644
> --- a/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c
> @@ -559,6 +559,7 @@ static int jpeg_v3_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
>  {
>  	jpeg_v3_0_stop(ring->adev);
>  	jpeg_v3_0_start(ring->adev);
> +	amdgpu_fence_driver_force_completion(ring);
>  	return amdgpu_ring_test_helper(ring);
>  }
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c
> index f17ec5414fd69..9301d0ff2593b 100644
> --- a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c
> @@ -727,6 +727,7 @@ static int jpeg_v4_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
>  
>  	jpeg_v4_0_stop(ring->adev);
>  	jpeg_v4_0_start(ring->adev);
> +	amdgpu_fence_driver_force_completion(ring);
>  	return amdgpu_ring_test_helper(ring);
>  }
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c
> index 79e342d5ab28d..ef5075e111300 100644
> --- a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c
> +++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c
> @@ -1150,6 +1150,7 @@ static int jpeg_v4_0_3_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
>  
>  	jpeg_v4_0_3_core_stall_reset(ring);
>  	jpeg_v4_0_3_start_jrbc(ring);
> +	amdgpu_fence_driver_force_completion(ring);
>  	return amdgpu_ring_test_helper(ring);
>  }
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c
> index 3b6f65a256464..858b930d1dd54 100644
> --- a/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c
> +++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c
> @@ -841,6 +841,7 @@ static int jpeg_v5_0_1_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
>  
>  	jpeg_v5_0_1_core_stall_reset(ring);
>  	jpeg_v5_0_1_init_jrbc(ring);
> +	amdgpu_fence_driver_force_completion(ring);
>  	return amdgpu_ring_test_helper(ring);
>  }
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c b/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c
> index 9c169112a5e7b..eb78bf04a490b 100644
> --- a/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c
> +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c
> @@ -1670,6 +1670,7 @@ static bool sdma_v4_4_2_page_ring_is_guilty(struct amdgpu_ring *ring)
>  static int sdma_v4_4_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
>  {
>  	struct amdgpu_device *adev = ring->adev;
> +	bool is_guilty = ring->funcs->is_guilty(ring);
>  	u32 id = GET_INST(SDMA0, ring->me);
>  	int r;
>  
> @@ -1680,6 +1681,9 @@ static int sdma_v4_4_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
>  	r = amdgpu_sdma_reset_engine(adev, id);
>  	amdgpu_amdkfd_resume(adev, false);
>  
> +	if (is_guilty)
> +		amdgpu_fence_driver_force_completion(ring);
> +
>  	return r;
>  }
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c
> index 9505ae96fbecc..43885dc04b632 100644
> --- a/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c
> @@ -1542,8 +1542,11 @@ static int sdma_v5_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
>  {
>  	struct amdgpu_device *adev = ring->adev;
>  	u32 inst_id = ring->me;
> +	int r;
>  
> -	return amdgpu_sdma_reset_engine(adev, inst_id);
> +	r = amdgpu_sdma_reset_engine(adev, inst_id);
> +	amdgpu_fence_driver_force_completion(ring);
> +	return r;
>  }
>  
>  static int sdma_v5_0_stop_queue(struct amdgpu_ring *ring)
> diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c b/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c
> index a6e612b4a8928..ee94e457ee25d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c
> +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c
> @@ -1455,8 +1455,11 @@ static int sdma_v5_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
>  {
>  	struct amdgpu_device *adev = ring->adev;
>  	u32 inst_id = ring->me;
> +	int r;
>  
> -	return amdgpu_sdma_reset_engine(adev, inst_id);
> +	r = amdgpu_sdma_reset_engine(adev, inst_id);
> +	amdgpu_fence_driver_force_completion(ring);
> +	return r;
>  }
>  
>  static int sdma_v5_2_stop_queue(struct amdgpu_ring *ring)
> diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
> index 5a70ae17be04e..86ef0e30a89e8 100644
> --- a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c
> @@ -1559,6 +1559,7 @@ static int sdma_v6_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
>  	if (r)
>  		return r;
>  
> +	amdgpu_fence_driver_force_completion(ring);
>  	return sdma_v6_0_gfx_resume_instance(adev, i, true);
>  }
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
> index befe013b11a78..c9993424b3923 100644
> --- a/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c
> @@ -823,6 +823,7 @@ static int sdma_v7_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
>  	if (r)
>  		return r;
>  
> +	amdgpu_fence_driver_force_completion(ring);
>  	return sdma_v7_0_gfx_resume_instance(adev, i, true);
>  }
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
> index b5071f77f78d2..f1ccc6c7be390 100644
> --- a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
> @@ -1977,6 +1977,7 @@ static int vcn_v4_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
>  
>  	vcn_v4_0_stop(vinst);
>  	vcn_v4_0_start(vinst);
> +	amdgpu_fence_driver_force_completion(ring);
>  
>  	return amdgpu_ring_test_helper(ring);
>  }
> diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c
> index 5a33140f57235..6ef5944584e56 100644
> --- a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c
> +++ b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c
> @@ -1620,9 +1620,8 @@ static int vcn_v4_0_3_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
>  		adev->vcn.caps |= AMDGPU_VCN_CAPS(RRMT_ENABLED);
>  	vcn_v4_0_3_hw_init_inst(vinst);
>  	vcn_v4_0_3_start_dpg_mode(vinst, adev->vcn.inst[ring->me].indirect_sram);
> -	r = amdgpu_ring_test_helper(ring);
> -
> -	return r;
> +	amdgpu_fence_driver_force_completion(ring);
> +	return amdgpu_ring_test_helper(ring);
>  }
>  
>  static const struct amdgpu_ring_funcs vcn_v4_0_3_unified_ring_vm_funcs = {
> diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c
> index 16ade84facc78..1c13ceb3bed8b 100644
> --- a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c
> +++ b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c
> @@ -1475,6 +1475,7 @@ static int vcn_v4_0_5_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
>  
>  	vcn_v4_0_5_stop(vinst);
>  	vcn_v4_0_5_start(vinst);
> +	amdgpu_fence_driver_force_completion(ring);
>  
>  	return amdgpu_ring_test_helper(ring);
>  }
> diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c b/drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c
> index f8e3f0b882da5..a998708bc0745 100644
> --- a/drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c
> @@ -1202,6 +1202,7 @@ static int vcn_v5_0_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
>  
>  	vcn_v5_0_0_stop(vinst);
>  	vcn_v5_0_0_start(vinst);
> +	amdgpu_fence_driver_force_completion(ring);
>  
>  	return amdgpu_ring_test_helper(ring);
>  }

Re: [PATCH 07/28] drm/amdgpu: track ring state associated with a job

[Christian König]

On 5/29/25 22:07, Alex Deucher wrote:
> We need to know the wptr and sequence number associated
> with a job so that we can re-emit the unprocessed state
> after a ring reset.  Pre-allocate storage space for
> the ring buffer contents and add a helper to save off
> the unprocessed state so that it can be re-emitted
> after the queue is reset.
> 
> Add a helper that ring reset callbacks can use to verify
> that the ring has reset successfully and to reemit any
> unprocessed ring contents from subsequent jobs.
> 
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c | 12 ++++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c    |  6 +++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c   |  5 ++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_job.h   |  2 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c  | 46 +++++++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h  |  8 ++++
>  6 files changed, 78 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
> index 2f24a6aa13bf6..319548ac58820 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
> @@ -764,6 +764,18 @@ void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring)
>  	amdgpu_fence_process(ring);
>  }
>  
> +/**
> + * amdgpu_fence_driver_seq_force_completion - force signal of specified sequence
> + *
> + * @ring: fence of the ring to signal
> + *
> + */
> +void amdgpu_fence_driver_seq_force_completion(struct amdgpu_ring *ring, u32 seq)
> +{
> +	amdgpu_fence_write(ring, seq);
> +	amdgpu_fence_process(ring);
> +}
> +
>  /*
>   * Common fence implementation
>   */
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> index 802743efa3b39..67df82d50a74a 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> @@ -306,6 +306,12 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned int num_ibs,
>  
>  	amdgpu_ring_ib_end(ring);
>  	amdgpu_ring_commit(ring);
> +	/* This must be last for resets to work properly
> +	 * as we need to save the wptr associated with this
> +	 * job.
> +	 */
> +	if (job)
> +		job->ring_wptr = ring->wptr;

First of all such state should *absolutely* not be made part of the job. That belongs into the HW fence.

Then we need to handle the case that one application submitted multiple jobs which potentially depend on each other.

I think we should rather put this logic into amdgpu_device_enforce_isolation().

Regards,
Christian.


>  	return 0;
>  }
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> index a0fab947143b5..f0f752284b925 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> @@ -91,6 +91,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
>  	struct amdgpu_job *job = to_amdgpu_job(s_job);
>  	struct amdgpu_task_info *ti;
>  	struct amdgpu_device *adev = ring->adev;
> +	struct dma_fence *fence = &job->hw_fence;
>  	int idx;
>  	int r;
>  
> @@ -154,8 +155,10 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
>  		else
>  			is_guilty = true;
>  
> -		if (is_guilty)
> +		if (is_guilty) {
> +			amdgpu_ring_backup_unprocessed_jobs(ring, job->ring_wptr, fence->seqno);
>  			dma_fence_set_error(&s_job->s_fence->finished, -ETIME);
> +		}
>  
>  		r = amdgpu_ring_reset(ring, job->vmid);
>  		if (!r) {
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> index f2c049129661f..c2ed0edb5179d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> @@ -79,6 +79,8 @@ struct amdgpu_job {
>  	/* enforce isolation */
>  	bool			enforce_isolation;
>  	bool			run_cleaner_shader;
> +	/* wptr for the job for resets */
> +	uint32_t		ring_wptr;
>  
>  	uint32_t		num_ibs;
>  	struct amdgpu_ib	ibs[];
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> index 426834806fbf2..909b121d432cb 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> @@ -333,6 +333,12 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
>  	/*  Initialize cached_rptr to 0 */
>  	ring->cached_rptr = 0;
>  
> +	if (!ring->ring_backup) {
> +		ring->ring_backup = kvzalloc(ring->ring_size, GFP_KERNEL);
> +		if (!ring->ring_backup)
> +			return -ENOMEM;
> +	}
> +
>  	/* Allocate ring buffer */
>  	if (ring->ring_obj == NULL) {
>  		r = amdgpu_bo_create_kernel(adev, ring->ring_size + ring->funcs->extra_dw, PAGE_SIZE,
> @@ -342,6 +348,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
>  					    (void **)&ring->ring);
>  		if (r) {
>  			dev_err(adev->dev, "(%d) ring create failed\n", r);
> +			kvfree(ring->ring_backup);
>  			return r;
>  		}
>  		amdgpu_ring_clear_ring(ring);
> @@ -385,6 +392,8 @@ void amdgpu_ring_fini(struct amdgpu_ring *ring)
>  	amdgpu_bo_free_kernel(&ring->ring_obj,
>  			      &ring->gpu_addr,
>  			      (void **)&ring->ring);
> +	kvfree(ring->ring_backup);
> +	ring->ring_backup = NULL;
>  
>  	dma_fence_put(ring->vmid_wait);
>  	ring->vmid_wait = NULL;
> @@ -753,3 +762,40 @@ bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring)
>  
>  	return true;
>  }
> +
> +void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
> +					 u64 bad_wptr, u32 bad_seq)
> +{
> +	unsigned int entries_to_copy = ring->wptr - bad_wptr;
> +	unsigned int idx, i;
> +
> +	for (i = 0; i < entries_to_copy; i++) {
> +		idx = (bad_wptr + i) & ring->buf_mask;
> +		ring->ring_backup[i] = ring->ring[idx];
> +	}
> +	ring->ring_backup_entries_to_copy = entries_to_copy;
> +	ring->ring_backup_seq = bad_seq;
> +}
> +
> +int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring)
> +{
> +	unsigned int i;
> +	int r;
> +
> +	/* signal the fence of the bad job */
> +	amdgpu_fence_driver_seq_force_completion(ring, ring->ring_backup_seq);
> +	/* verify that the ring is functional */
> +	r = amdgpu_ring_test_ring(ring);
> +	if (r)
> +		return r;
> +	/* re-emit the unprocessed ring contents */
> +	if (ring->ring_backup_entries_to_copy) {
> +		if (amdgpu_ring_alloc(ring, ring->ring_backup_entries_to_copy))
> +			return -ENOMEM;
> +		for (i = 0; i < ring->ring_backup_entries_to_copy; i++)
> +			amdgpu_ring_write(ring, ring->ring_backup[i]);
> +		amdgpu_ring_commit(ring);
> +	}
> +
> +	return r;
> +}
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> index b95b471107692..fd08449eee33f 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> @@ -132,6 +132,8 @@ extern const struct drm_sched_backend_ops amdgpu_sched_ops;
>  void amdgpu_fence_driver_clear_job_fences(struct amdgpu_ring *ring);
>  void amdgpu_fence_driver_set_error(struct amdgpu_ring *ring, int error);
>  void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring);
> +void amdgpu_fence_driver_seq_force_completion(struct amdgpu_ring *ring,
> +					      u32 seq);
>  
>  int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring);
>  int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring,
> @@ -268,6 +270,9 @@ struct amdgpu_ring {
>  
>  	struct amdgpu_bo	*ring_obj;
>  	uint32_t		*ring;
> +	uint32_t		*ring_backup;
> +	uint32_t		ring_backup_seq;
> +	unsigned int		ring_backup_entries_to_copy;
>  	unsigned		rptr_offs;
>  	u64			rptr_gpu_addr;
>  	volatile u32		*rptr_cpu_addr;
> @@ -534,4 +539,7 @@ int amdgpu_ib_pool_init(struct amdgpu_device *adev);
>  void amdgpu_ib_pool_fini(struct amdgpu_device *adev);
>  int amdgpu_ib_ring_tests(struct amdgpu_device *adev);
>  bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring);
> +void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
> +					 u64 bad_wptr, u32 bad_seq);
> +int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring);
>  #endif

Re: [PATCH 02/28] drm/amdgpu/gfx7: drop reset_kgq

[Alex Deucher]

On Mon, Jun 2, 2025 at 8:57 AM Christian König <christian.koenig@amd.com> wrote:
>
> On 5/29/25 22:07, Alex Deucher wrote:
> > It doesn't work reliably and we have soft recover and
> > full adapter reset so drop this.
> >
> > Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
>
> I was really hoping that this would be more reliable than soft recovery at some point. And from my testing on gfx9 it seems to work rather reliable.
>
> Are you sure we should just drop that?

I couldn't get it to work reliably at all on any of the gfx8 or 9
hardware I tried.  We can always revert the patches if anyone wants to
work on it again.

Alex

>
> Regards,
> Christian.
>
> > ---
> >  drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c | 71 ---------------------------
> >  1 file changed, 71 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c
> > index da0534ff1271a..2aa323dab34e3 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c
> > @@ -4884,76 +4884,6 @@ static void gfx_v7_0_emit_mem_sync_compute(struct amdgpu_ring *ring)
> >       amdgpu_ring_write(ring, 0x0000000A);    /* poll interval */
> >  }
> >
> > -static void gfx_v7_0_wait_reg_mem(struct amdgpu_ring *ring, int eng_sel,
> > -                               int mem_space, int opt, uint32_t addr0,
> > -                               uint32_t addr1, uint32_t ref, uint32_t mask,
> > -                               uint32_t inv)
> > -{
> > -     amdgpu_ring_write(ring, PACKET3(PACKET3_WAIT_REG_MEM, 5));
> > -     amdgpu_ring_write(ring,
> > -                       /* memory (1) or register (0) */
> > -                       (WAIT_REG_MEM_MEM_SPACE(mem_space) |
> > -                        WAIT_REG_MEM_OPERATION(opt) | /* wait */
> > -                        WAIT_REG_MEM_FUNCTION(3) |  /* equal */
> > -                        WAIT_REG_MEM_ENGINE(eng_sel)));
> > -
> > -     if (mem_space)
> > -             BUG_ON(addr0 & 0x3); /* Dword align */
> > -     amdgpu_ring_write(ring, addr0);
> > -     amdgpu_ring_write(ring, addr1);
> > -     amdgpu_ring_write(ring, ref);
> > -     amdgpu_ring_write(ring, mask);
> > -     amdgpu_ring_write(ring, inv); /* poll interval */
> > -}
> > -
> > -static void gfx_v7_0_ring_emit_reg_wait(struct amdgpu_ring *ring, uint32_t reg,
> > -                                     uint32_t val, uint32_t mask)
> > -{
> > -     gfx_v7_0_wait_reg_mem(ring, 0, 0, 0, reg, 0, val, mask, 0x20);
> > -}
> > -
> > -static int gfx_v7_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)
> > -{
> > -     struct amdgpu_device *adev = ring->adev;
> > -     struct amdgpu_kiq *kiq = &adev->gfx.kiq[0];
> > -     struct amdgpu_ring *kiq_ring = &kiq->ring;
> > -     unsigned long flags;
> > -     u32 tmp;
> > -     int r;
> > -
> > -     if (amdgpu_sriov_vf(adev))
> > -             return -EINVAL;
> > -
> > -     if (!kiq->pmf || !kiq->pmf->kiq_unmap_queues)
> > -             return -EINVAL;
> > -
> > -     spin_lock_irqsave(&kiq->ring_lock, flags);
> > -
> > -     if (amdgpu_ring_alloc(kiq_ring, 5)) {
> > -             spin_unlock_irqrestore(&kiq->ring_lock, flags);
> > -             return -ENOMEM;
> > -     }
> > -
> > -     tmp = REG_SET_FIELD(0, CP_VMID_RESET, RESET_REQUEST, 1 << vmid);
> > -     gfx_v7_0_ring_emit_wreg(kiq_ring, mmCP_VMID_RESET, tmp);
> > -     amdgpu_ring_commit(kiq_ring);
> > -
> > -     spin_unlock_irqrestore(&kiq->ring_lock, flags);
> > -
> > -     r = amdgpu_ring_test_ring(kiq_ring);
> > -     if (r)
> > -             return r;
> > -
> > -     if (amdgpu_ring_alloc(ring, 7 + 12 + 5))
> > -             return -ENOMEM;
> > -     gfx_v7_0_ring_emit_fence_gfx(ring, ring->fence_drv.gpu_addr,
> > -                                  ring->fence_drv.sync_seq, AMDGPU_FENCE_FLAG_EXEC);
> > -     gfx_v7_0_ring_emit_reg_wait(ring, mmCP_VMID_RESET, 0, 0xffff);
> > -     gfx_v7_0_ring_emit_wreg(ring, mmCP_VMID_RESET, 0);
> > -
> > -     return amdgpu_ring_test_ring(ring);
> > -}
> > -
> >  static const struct amd_ip_funcs gfx_v7_0_ip_funcs = {
> >       .name = "gfx_v7_0",
> >       .early_init = gfx_v7_0_early_init,
> > @@ -5003,7 +4933,6 @@ static const struct amdgpu_ring_funcs gfx_v7_0_ring_funcs_gfx = {
> >       .emit_wreg = gfx_v7_0_ring_emit_wreg,
> >       .soft_recovery = gfx_v7_0_ring_soft_recovery,
> >       .emit_mem_sync = gfx_v7_0_emit_mem_sync,
> > -     .reset = gfx_v7_0_reset_kgq,
> >  };
> >
> >  static const struct amdgpu_ring_funcs gfx_v7_0_ring_funcs_compute = {
>

Re: [PATCH 07/28] drm/amdgpu: track ring state associated with a job

[Alex Deucher]

On Mon, Jun 2, 2025 at 10:36 AM Christian König
<christian.koenig@amd.com> wrote:
>
> On 5/29/25 22:07, Alex Deucher wrote:
> > We need to know the wptr and sequence number associated
> > with a job so that we can re-emit the unprocessed state
> > after a ring reset.  Pre-allocate storage space for
> > the ring buffer contents and add a helper to save off
> > the unprocessed state so that it can be re-emitted
> > after the queue is reset.
> >
> > Add a helper that ring reset callbacks can use to verify
> > that the ring has reset successfully and to reemit any
> > unprocessed ring contents from subsequent jobs.
> >
> > Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> > ---
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c | 12 ++++++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c    |  6 +++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c   |  5 ++-
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_job.h   |  2 +
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c  | 46 +++++++++++++++++++++++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h  |  8 ++++
> >  6 files changed, 78 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
> > index 2f24a6aa13bf6..319548ac58820 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
> > @@ -764,6 +764,18 @@ void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring)
> >       amdgpu_fence_process(ring);
> >  }
> >
> > +/**
> > + * amdgpu_fence_driver_seq_force_completion - force signal of specified sequence
> > + *
> > + * @ring: fence of the ring to signal
> > + *
> > + */
> > +void amdgpu_fence_driver_seq_force_completion(struct amdgpu_ring *ring, u32 seq)
> > +{
> > +     amdgpu_fence_write(ring, seq);
> > +     amdgpu_fence_process(ring);
> > +}
> > +
> >  /*
> >   * Common fence implementation
> >   */
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> > index 802743efa3b39..67df82d50a74a 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> > @@ -306,6 +306,12 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned int num_ibs,
> >
> >       amdgpu_ring_ib_end(ring);
> >       amdgpu_ring_commit(ring);
> > +     /* This must be last for resets to work properly
> > +      * as we need to save the wptr associated with this
> > +      * job.
> > +      */
> > +     if (job)
> > +             job->ring_wptr = ring->wptr;
>
> First of all such state should *absolutely* not be made part of the job. That belongs into the HW fence.

Done.  Updated patches pushed here:
https://gitlab.freedesktop.org/agd5f/linux/-/commits/kq_resets?ref_type=heads

>
> Then we need to handle the case that one application submitted multiple jobs which potentially depend on each other.
>
> I think we should rather put this logic into amdgpu_device_enforce_isolation().

I'm not quite sure I understand what you are proposing.  Is the idea
to track all of the jobs associated with a particular process and then
when we reset a queue, skip all of the ring contents associated with
those jobs and signal and set the error on all of their job fences?
What about cross ring dependencies?

Alex

>
> Regards,
> Christian.
>
>
> >       return 0;
> >  }
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> > index a0fab947143b5..f0f752284b925 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> > @@ -91,6 +91,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
> >       struct amdgpu_job *job = to_amdgpu_job(s_job);
> >       struct amdgpu_task_info *ti;
> >       struct amdgpu_device *adev = ring->adev;
> > +     struct dma_fence *fence = &job->hw_fence;
> >       int idx;
> >       int r;
> >
> > @@ -154,8 +155,10 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
> >               else
> >                       is_guilty = true;
> >
> > -             if (is_guilty)
> > +             if (is_guilty) {
> > +                     amdgpu_ring_backup_unprocessed_jobs(ring, job->ring_wptr, fence->seqno);
> >                       dma_fence_set_error(&s_job->s_fence->finished, -ETIME);
> > +             }
> >
> >               r = amdgpu_ring_reset(ring, job->vmid);
> >               if (!r) {
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> > index f2c049129661f..c2ed0edb5179d 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> > @@ -79,6 +79,8 @@ struct amdgpu_job {
> >       /* enforce isolation */
> >       bool                    enforce_isolation;
> >       bool                    run_cleaner_shader;
> > +     /* wptr for the job for resets */
> > +     uint32_t                ring_wptr;
> >
> >       uint32_t                num_ibs;
> >       struct amdgpu_ib        ibs[];
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> > index 426834806fbf2..909b121d432cb 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> > @@ -333,6 +333,12 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
> >       /*  Initialize cached_rptr to 0 */
> >       ring->cached_rptr = 0;
> >
> > +     if (!ring->ring_backup) {
> > +             ring->ring_backup = kvzalloc(ring->ring_size, GFP_KERNEL);
> > +             if (!ring->ring_backup)
> > +                     return -ENOMEM;
> > +     }
> > +
> >       /* Allocate ring buffer */
> >       if (ring->ring_obj == NULL) {
> >               r = amdgpu_bo_create_kernel(adev, ring->ring_size + ring->funcs->extra_dw, PAGE_SIZE,
> > @@ -342,6 +348,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
> >                                           (void **)&ring->ring);
> >               if (r) {
> >                       dev_err(adev->dev, "(%d) ring create failed\n", r);
> > +                     kvfree(ring->ring_backup);
> >                       return r;
> >               }
> >               amdgpu_ring_clear_ring(ring);
> > @@ -385,6 +392,8 @@ void amdgpu_ring_fini(struct amdgpu_ring *ring)
> >       amdgpu_bo_free_kernel(&ring->ring_obj,
> >                             &ring->gpu_addr,
> >                             (void **)&ring->ring);
> > +     kvfree(ring->ring_backup);
> > +     ring->ring_backup = NULL;
> >
> >       dma_fence_put(ring->vmid_wait);
> >       ring->vmid_wait = NULL;
> > @@ -753,3 +762,40 @@ bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring)
> >
> >       return true;
> >  }
> > +
> > +void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
> > +                                      u64 bad_wptr, u32 bad_seq)
> > +{
> > +     unsigned int entries_to_copy = ring->wptr - bad_wptr;
> > +     unsigned int idx, i;
> > +
> > +     for (i = 0; i < entries_to_copy; i++) {
> > +             idx = (bad_wptr + i) & ring->buf_mask;
> > +             ring->ring_backup[i] = ring->ring[idx];
> > +     }
> > +     ring->ring_backup_entries_to_copy = entries_to_copy;
> > +     ring->ring_backup_seq = bad_seq;
> > +}
> > +
> > +int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring)
> > +{
> > +     unsigned int i;
> > +     int r;
> > +
> > +     /* signal the fence of the bad job */
> > +     amdgpu_fence_driver_seq_force_completion(ring, ring->ring_backup_seq);
> > +     /* verify that the ring is functional */
> > +     r = amdgpu_ring_test_ring(ring);
> > +     if (r)
> > +             return r;
> > +     /* re-emit the unprocessed ring contents */
> > +     if (ring->ring_backup_entries_to_copy) {
> > +             if (amdgpu_ring_alloc(ring, ring->ring_backup_entries_to_copy))
> > +                     return -ENOMEM;
> > +             for (i = 0; i < ring->ring_backup_entries_to_copy; i++)
> > +                     amdgpu_ring_write(ring, ring->ring_backup[i]);
> > +             amdgpu_ring_commit(ring);
> > +     }
> > +
> > +     return r;
> > +}
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> > index b95b471107692..fd08449eee33f 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> > @@ -132,6 +132,8 @@ extern const struct drm_sched_backend_ops amdgpu_sched_ops;
> >  void amdgpu_fence_driver_clear_job_fences(struct amdgpu_ring *ring);
> >  void amdgpu_fence_driver_set_error(struct amdgpu_ring *ring, int error);
> >  void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring);
> > +void amdgpu_fence_driver_seq_force_completion(struct amdgpu_ring *ring,
> > +                                           u32 seq);
> >
> >  int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring);
> >  int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring,
> > @@ -268,6 +270,9 @@ struct amdgpu_ring {
> >
> >       struct amdgpu_bo        *ring_obj;
> >       uint32_t                *ring;
> > +     uint32_t                *ring_backup;
> > +     uint32_t                ring_backup_seq;
> > +     unsigned int            ring_backup_entries_to_copy;
> >       unsigned                rptr_offs;
> >       u64                     rptr_gpu_addr;
> >       volatile u32            *rptr_cpu_addr;
> > @@ -534,4 +539,7 @@ int amdgpu_ib_pool_init(struct amdgpu_device *adev);
> >  void amdgpu_ib_pool_fini(struct amdgpu_device *adev);
> >  int amdgpu_ib_ring_tests(struct amdgpu_device *adev);
> >  bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring);
> > +void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
> > +                                      u64 bad_wptr, u32 bad_seq);
> > +int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring);
> >  #endif
>

Re: [PATCH 07/28] drm/amdgpu: track ring state associated with a job

[Christian König]

On 6/3/25 00:42, Alex Deucher wrote:
> On Mon, Jun 2, 2025 at 10:36 AM Christian König
> <christian.koenig@amd.com> wrote:
>>
>> On 5/29/25 22:07, Alex Deucher wrote:
>>> We need to know the wptr and sequence number associated
>>> with a job so that we can re-emit the unprocessed state
>>> after a ring reset.  Pre-allocate storage space for
>>> the ring buffer contents and add a helper to save off
>>> the unprocessed state so that it can be re-emitted
>>> after the queue is reset.
>>>
>>> Add a helper that ring reset callbacks can use to verify
>>> that the ring has reset successfully and to reemit any
>>> unprocessed ring contents from subsequent jobs.
>>>
>>> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
>>> ---
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c | 12 ++++++
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c    |  6 +++
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c   |  5 ++-
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_job.h   |  2 +
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c  | 46 +++++++++++++++++++++++
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h  |  8 ++++
>>>  6 files changed, 78 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
>>> index 2f24a6aa13bf6..319548ac58820 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
>>> @@ -764,6 +764,18 @@ void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring)
>>>       amdgpu_fence_process(ring);
>>>  }
>>>
>>> +/**
>>> + * amdgpu_fence_driver_seq_force_completion - force signal of specified sequence
>>> + *
>>> + * @ring: fence of the ring to signal
>>> + *
>>> + */
>>> +void amdgpu_fence_driver_seq_force_completion(struct amdgpu_ring *ring, u32 seq)
>>> +{
>>> +     amdgpu_fence_write(ring, seq);
>>> +     amdgpu_fence_process(ring);
>>> +}
>>> +
>>>  /*
>>>   * Common fence implementation
>>>   */
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
>>> index 802743efa3b39..67df82d50a74a 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
>>> @@ -306,6 +306,12 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned int num_ibs,
>>>
>>>       amdgpu_ring_ib_end(ring);
>>>       amdgpu_ring_commit(ring);
>>> +     /* This must be last for resets to work properly
>>> +      * as we need to save the wptr associated with this
>>> +      * job.
>>> +      */
>>> +     if (job)
>>> +             job->ring_wptr = ring->wptr;
>>
>> First of all such state should *absolutely* not be made part of the job. That belongs into the HW fence.
> 
> Done.  Updated patches pushed here:
> https://gitlab.freedesktop.org/agd5f/linux/-/commits/kq_resets?ref_type=heads
> 
>>
>> Then we need to handle the case that one application submitted multiple jobs which potentially depend on each other.
>>
>> I think we should rather put this logic into amdgpu_device_enforce_isolation().
> 
> I'm not quite sure I understand what you are proposing.  Is the idea
> to track all of the jobs associated with a particular process and then
> when we reset a queue, skip all of the ring contents associated with
> those jobs and signal and set the error on all of their job fences?

More or less yes, I think that is what is needed here.

A simple example: Unigine Heaven in window mode on an X server. Each frame usually results in 3 job submissions from unigine, plus one submission from X to copy the result it into the displayed frame.

When we now assume that we can schedule 4 jobs at a time on the ring we get: U1, U2, U3, X1 | U4, U5, U6, X2 | U7, U8, U9, X3....

Let's assume U4 hangs and we initiate a queue reset, in this case we definately need to skip U5 and U6 as well because they belonged to the same context and depend on each other. Only skipping U4 would certainly crash the GPU again.

X2 also dependet on U6, but that submission is from X and totally innocent and rendering garbage for the window content is probably ok considering that the application just crashed.

> What about cross ring dependencies?

For gang submission we would need to do a queue reset for both the gfx and compute queue to get out of this again. But that is probably ok since each queue can timeout on its own.

We also don't need to track the jobs per process, just looking if job->base.sched_fence->finished.context changes should be sufficient.

Regards,
Christian.

> 
> Alex
> 
>>
>> Regards,
>> Christian.
>>
>>
>>>       return 0;
>>>  }
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
>>> index a0fab947143b5..f0f752284b925 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
>>> @@ -91,6 +91,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
>>>       struct amdgpu_job *job = to_amdgpu_job(s_job);
>>>       struct amdgpu_task_info *ti;
>>>       struct amdgpu_device *adev = ring->adev;
>>> +     struct dma_fence *fence = &job->hw_fence;
>>>       int idx;
>>>       int r;
>>>
>>> @@ -154,8 +155,10 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
>>>               else
>>>                       is_guilty = true;
>>>
>>> -             if (is_guilty)
>>> +             if (is_guilty) {
>>> +                     amdgpu_ring_backup_unprocessed_jobs(ring, job->ring_wptr, fence->seqno);
>>>                       dma_fence_set_error(&s_job->s_fence->finished, -ETIME);
>>> +             }
>>>
>>>               r = amdgpu_ring_reset(ring, job->vmid);
>>>               if (!r) {
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
>>> index f2c049129661f..c2ed0edb5179d 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
>>> @@ -79,6 +79,8 @@ struct amdgpu_job {
>>>       /* enforce isolation */
>>>       bool                    enforce_isolation;
>>>       bool                    run_cleaner_shader;
>>> +     /* wptr for the job for resets */
>>> +     uint32_t                ring_wptr;
>>>
>>>       uint32_t                num_ibs;
>>>       struct amdgpu_ib        ibs[];
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
>>> index 426834806fbf2..909b121d432cb 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
>>> @@ -333,6 +333,12 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
>>>       /*  Initialize cached_rptr to 0 */
>>>       ring->cached_rptr = 0;
>>>
>>> +     if (!ring->ring_backup) {
>>> +             ring->ring_backup = kvzalloc(ring->ring_size, GFP_KERNEL);
>>> +             if (!ring->ring_backup)
>>> +                     return -ENOMEM;
>>> +     }
>>> +
>>>       /* Allocate ring buffer */
>>>       if (ring->ring_obj == NULL) {
>>>               r = amdgpu_bo_create_kernel(adev, ring->ring_size + ring->funcs->extra_dw, PAGE_SIZE,
>>> @@ -342,6 +348,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
>>>                                           (void **)&ring->ring);
>>>               if (r) {
>>>                       dev_err(adev->dev, "(%d) ring create failed\n", r);
>>> +                     kvfree(ring->ring_backup);
>>>                       return r;
>>>               }
>>>               amdgpu_ring_clear_ring(ring);
>>> @@ -385,6 +392,8 @@ void amdgpu_ring_fini(struct amdgpu_ring *ring)
>>>       amdgpu_bo_free_kernel(&ring->ring_obj,
>>>                             &ring->gpu_addr,
>>>                             (void **)&ring->ring);
>>> +     kvfree(ring->ring_backup);
>>> +     ring->ring_backup = NULL;
>>>
>>>       dma_fence_put(ring->vmid_wait);
>>>       ring->vmid_wait = NULL;
>>> @@ -753,3 +762,40 @@ bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring)
>>>
>>>       return true;
>>>  }
>>> +
>>> +void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
>>> +                                      u64 bad_wptr, u32 bad_seq)
>>> +{
>>> +     unsigned int entries_to_copy = ring->wptr - bad_wptr;
>>> +     unsigned int idx, i;
>>> +
>>> +     for (i = 0; i < entries_to_copy; i++) {
>>> +             idx = (bad_wptr + i) & ring->buf_mask;
>>> +             ring->ring_backup[i] = ring->ring[idx];
>>> +     }
>>> +     ring->ring_backup_entries_to_copy = entries_to_copy;
>>> +     ring->ring_backup_seq = bad_seq;
>>> +}
>>> +
>>> +int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring)
>>> +{
>>> +     unsigned int i;
>>> +     int r;
>>> +
>>> +     /* signal the fence of the bad job */
>>> +     amdgpu_fence_driver_seq_force_completion(ring, ring->ring_backup_seq);
>>> +     /* verify that the ring is functional */
>>> +     r = amdgpu_ring_test_ring(ring);
>>> +     if (r)
>>> +             return r;
>>> +     /* re-emit the unprocessed ring contents */
>>> +     if (ring->ring_backup_entries_to_copy) {
>>> +             if (amdgpu_ring_alloc(ring, ring->ring_backup_entries_to_copy))
>>> +                     return -ENOMEM;
>>> +             for (i = 0; i < ring->ring_backup_entries_to_copy; i++)
>>> +                     amdgpu_ring_write(ring, ring->ring_backup[i]);
>>> +             amdgpu_ring_commit(ring);
>>> +     }
>>> +
>>> +     return r;
>>> +}
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
>>> index b95b471107692..fd08449eee33f 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
>>> @@ -132,6 +132,8 @@ extern const struct drm_sched_backend_ops amdgpu_sched_ops;
>>>  void amdgpu_fence_driver_clear_job_fences(struct amdgpu_ring *ring);
>>>  void amdgpu_fence_driver_set_error(struct amdgpu_ring *ring, int error);
>>>  void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring);
>>> +void amdgpu_fence_driver_seq_force_completion(struct amdgpu_ring *ring,
>>> +                                           u32 seq);
>>>
>>>  int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring);
>>>  int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring,
>>> @@ -268,6 +270,9 @@ struct amdgpu_ring {
>>>
>>>       struct amdgpu_bo        *ring_obj;
>>>       uint32_t                *ring;
>>> +     uint32_t                *ring_backup;
>>> +     uint32_t                ring_backup_seq;
>>> +     unsigned int            ring_backup_entries_to_copy;
>>>       unsigned                rptr_offs;
>>>       u64                     rptr_gpu_addr;
>>>       volatile u32            *rptr_cpu_addr;
>>> @@ -534,4 +539,7 @@ int amdgpu_ib_pool_init(struct amdgpu_device *adev);
>>>  void amdgpu_ib_pool_fini(struct amdgpu_device *adev);
>>>  int amdgpu_ib_ring_tests(struct amdgpu_device *adev);
>>>  bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring);
>>> +void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
>>> +                                      u64 bad_wptr, u32 bad_seq);
>>> +int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring);
>>>  #endif
>>

Re: [PATCH 07/28] drm/amdgpu: track ring state associated with a job

[Alex Deucher]

On Tue, Jun 3, 2025 at 4:03 AM Christian König <christian.koenig@amd.com> wrote:
>
> On 6/3/25 00:42, Alex Deucher wrote:
> > On Mon, Jun 2, 2025 at 10:36 AM Christian König
> > <christian.koenig@amd.com> wrote:
> >>
> >> On 5/29/25 22:07, Alex Deucher wrote:
> >>> We need to know the wptr and sequence number associated
> >>> with a job so that we can re-emit the unprocessed state
> >>> after a ring reset.  Pre-allocate storage space for
> >>> the ring buffer contents and add a helper to save off
> >>> the unprocessed state so that it can be re-emitted
> >>> after the queue is reset.
> >>>
> >>> Add a helper that ring reset callbacks can use to verify
> >>> that the ring has reset successfully and to reemit any
> >>> unprocessed ring contents from subsequent jobs.
> >>>
> >>> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> >>> ---
> >>>  drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c | 12 ++++++
> >>>  drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c    |  6 +++
> >>>  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c   |  5 ++-
> >>>  drivers/gpu/drm/amd/amdgpu/amdgpu_job.h   |  2 +
> >>>  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c  | 46 +++++++++++++++++++++++
> >>>  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h  |  8 ++++
> >>>  6 files changed, 78 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
> >>> index 2f24a6aa13bf6..319548ac58820 100644
> >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
> >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
> >>> @@ -764,6 +764,18 @@ void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring)
> >>>       amdgpu_fence_process(ring);
> >>>  }
> >>>
> >>> +/**
> >>> + * amdgpu_fence_driver_seq_force_completion - force signal of specified sequence
> >>> + *
> >>> + * @ring: fence of the ring to signal
> >>> + *
> >>> + */
> >>> +void amdgpu_fence_driver_seq_force_completion(struct amdgpu_ring *ring, u32 seq)
> >>> +{
> >>> +     amdgpu_fence_write(ring, seq);
> >>> +     amdgpu_fence_process(ring);
> >>> +}
> >>> +
> >>>  /*
> >>>   * Common fence implementation
> >>>   */
> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> >>> index 802743efa3b39..67df82d50a74a 100644
> >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> >>> @@ -306,6 +306,12 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned int num_ibs,
> >>>
> >>>       amdgpu_ring_ib_end(ring);
> >>>       amdgpu_ring_commit(ring);
> >>> +     /* This must be last for resets to work properly
> >>> +      * as we need to save the wptr associated with this
> >>> +      * job.
> >>> +      */
> >>> +     if (job)
> >>> +             job->ring_wptr = ring->wptr;
> >>
> >> First of all such state should *absolutely* not be made part of the job. That belongs into the HW fence.
> >
> > Done.  Updated patches pushed here:
> > https://gitlab.freedesktop.org/agd5f/linux/-/commits/kq_resets?ref_type=heads
> >
> >>
> >> Then we need to handle the case that one application submitted multiple jobs which potentially depend on each other.
> >>
> >> I think we should rather put this logic into amdgpu_device_enforce_isolation().
> >
> > I'm not quite sure I understand what you are proposing.  Is the idea
> > to track all of the jobs associated with a particular process and then
> > when we reset a queue, skip all of the ring contents associated with
> > those jobs and signal and set the error on all of their job fences?
>
> More or less yes, I think that is what is needed here.
>
> A simple example: Unigine Heaven in window mode on an X server. Each frame usually results in 3 job submissions from unigine, plus one submission from X to copy the result it into the displayed frame.
>
> When we now assume that we can schedule 4 jobs at a time on the ring we get: U1, U2, U3, X1 | U4, U5, U6, X2 | U7, U8, U9, X3....
>
> Let's assume U4 hangs and we initiate a queue reset, in this case we definately need to skip U5 and U6 as well because they belonged to the same context and depend on each other. Only skipping U4 would certainly crash the GPU again.

In that case, we'd just reset the queue again, but I agree it would be
a nicer experience to skip all of the jobs for that app.

>
> X2 also dependet on U6, but that submission is from X and totally innocent and rendering garbage for the window content is probably ok considering that the application just crashed.
>
> > What about cross ring dependencies?
>
> For gang submission we would need to do a queue reset for both the gfx and compute queue to get out of this again. But that is probably ok since each queue can timeout on its own.
>
> We also don't need to track the jobs per process, just looking if job->base.sched_fence->finished.context changes should be sufficient.

Something like this?

        for (i = 0; i <= ring->fence_drv.num_fences_mask; i++) {
                ptr = &ring->fence_drv.fences[i];
            old = rcu_dereference_protected(*ptr, 1);
                if (old && old->ops == &amdgpu_job_fence_ops) {
                        struct amdgpu_job *other_job =
                container_of(old, struct amdgpu_job, hw_fence.base);

                        if (other_job->base.sched_fence->finished.context ==
                            job->base.sched_fence->finished.context) {
                                struct amdgpu_fence *am_fence =
&other_job->hw_fence;
                                // skip the ring contents associated
with this context
                        }
                }
        }

Thanks,

Alex


>
> Regards,
> Christian.
>
> >
> > Alex
> >
> >>
> >> Regards,
> >> Christian.
> >>
> >>
> >>>       return 0;
> >>>  }
> >>>
> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> >>> index a0fab947143b5..f0f752284b925 100644
> >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> >>> @@ -91,6 +91,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
> >>>       struct amdgpu_job *job = to_amdgpu_job(s_job);
> >>>       struct amdgpu_task_info *ti;
> >>>       struct amdgpu_device *adev = ring->adev;
> >>> +     struct dma_fence *fence = &job->hw_fence;
> >>>       int idx;
> >>>       int r;
> >>>
> >>> @@ -154,8 +155,10 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
> >>>               else
> >>>                       is_guilty = true;
> >>>
> >>> -             if (is_guilty)
> >>> +             if (is_guilty) {
> >>> +                     amdgpu_ring_backup_unprocessed_jobs(ring, job->ring_wptr, fence->seqno);
> >>>                       dma_fence_set_error(&s_job->s_fence->finished, -ETIME);
> >>> +             }
> >>>
> >>>               r = amdgpu_ring_reset(ring, job->vmid);
> >>>               if (!r) {
> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> >>> index f2c049129661f..c2ed0edb5179d 100644
> >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> >>> @@ -79,6 +79,8 @@ struct amdgpu_job {
> >>>       /* enforce isolation */
> >>>       bool                    enforce_isolation;
> >>>       bool                    run_cleaner_shader;
> >>> +     /* wptr for the job for resets */
> >>> +     uint32_t                ring_wptr;
> >>>
> >>>       uint32_t                num_ibs;
> >>>       struct amdgpu_ib        ibs[];
> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> >>> index 426834806fbf2..909b121d432cb 100644
> >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> >>> @@ -333,6 +333,12 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
> >>>       /*  Initialize cached_rptr to 0 */
> >>>       ring->cached_rptr = 0;
> >>>
> >>> +     if (!ring->ring_backup) {
> >>> +             ring->ring_backup = kvzalloc(ring->ring_size, GFP_KERNEL);
> >>> +             if (!ring->ring_backup)
> >>> +                     return -ENOMEM;
> >>> +     }
> >>> +
> >>>       /* Allocate ring buffer */
> >>>       if (ring->ring_obj == NULL) {
> >>>               r = amdgpu_bo_create_kernel(adev, ring->ring_size + ring->funcs->extra_dw, PAGE_SIZE,
> >>> @@ -342,6 +348,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
> >>>                                           (void **)&ring->ring);
> >>>               if (r) {
> >>>                       dev_err(adev->dev, "(%d) ring create failed\n", r);
> >>> +                     kvfree(ring->ring_backup);
> >>>                       return r;
> >>>               }
> >>>               amdgpu_ring_clear_ring(ring);
> >>> @@ -385,6 +392,8 @@ void amdgpu_ring_fini(struct amdgpu_ring *ring)
> >>>       amdgpu_bo_free_kernel(&ring->ring_obj,
> >>>                             &ring->gpu_addr,
> >>>                             (void **)&ring->ring);
> >>> +     kvfree(ring->ring_backup);
> >>> +     ring->ring_backup = NULL;
> >>>
> >>>       dma_fence_put(ring->vmid_wait);
> >>>       ring->vmid_wait = NULL;
> >>> @@ -753,3 +762,40 @@ bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring)
> >>>
> >>>       return true;
> >>>  }
> >>> +
> >>> +void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
> >>> +                                      u64 bad_wptr, u32 bad_seq)
> >>> +{
> >>> +     unsigned int entries_to_copy = ring->wptr - bad_wptr;
> >>> +     unsigned int idx, i;
> >>> +
> >>> +     for (i = 0; i < entries_to_copy; i++) {
> >>> +             idx = (bad_wptr + i) & ring->buf_mask;
> >>> +             ring->ring_backup[i] = ring->ring[idx];
> >>> +     }
> >>> +     ring->ring_backup_entries_to_copy = entries_to_copy;
> >>> +     ring->ring_backup_seq = bad_seq;
> >>> +}
> >>> +
> >>> +int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring)
> >>> +{
> >>> +     unsigned int i;
> >>> +     int r;
> >>> +
> >>> +     /* signal the fence of the bad job */
> >>> +     amdgpu_fence_driver_seq_force_completion(ring, ring->ring_backup_seq);
> >>> +     /* verify that the ring is functional */
> >>> +     r = amdgpu_ring_test_ring(ring);
> >>> +     if (r)
> >>> +             return r;
> >>> +     /* re-emit the unprocessed ring contents */
> >>> +     if (ring->ring_backup_entries_to_copy) {
> >>> +             if (amdgpu_ring_alloc(ring, ring->ring_backup_entries_to_copy))
> >>> +                     return -ENOMEM;
> >>> +             for (i = 0; i < ring->ring_backup_entries_to_copy; i++)
> >>> +                     amdgpu_ring_write(ring, ring->ring_backup[i]);
> >>> +             amdgpu_ring_commit(ring);
> >>> +     }
> >>> +
> >>> +     return r;
> >>> +}
> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> >>> index b95b471107692..fd08449eee33f 100644
> >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> >>> @@ -132,6 +132,8 @@ extern const struct drm_sched_backend_ops amdgpu_sched_ops;
> >>>  void amdgpu_fence_driver_clear_job_fences(struct amdgpu_ring *ring);
> >>>  void amdgpu_fence_driver_set_error(struct amdgpu_ring *ring, int error);
> >>>  void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring);
> >>> +void amdgpu_fence_driver_seq_force_completion(struct amdgpu_ring *ring,
> >>> +                                           u32 seq);
> >>>
> >>>  int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring);
> >>>  int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring,
> >>> @@ -268,6 +270,9 @@ struct amdgpu_ring {
> >>>
> >>>       struct amdgpu_bo        *ring_obj;
> >>>       uint32_t                *ring;
> >>> +     uint32_t                *ring_backup;
> >>> +     uint32_t                ring_backup_seq;
> >>> +     unsigned int            ring_backup_entries_to_copy;
> >>>       unsigned                rptr_offs;
> >>>       u64                     rptr_gpu_addr;
> >>>       volatile u32            *rptr_cpu_addr;
> >>> @@ -534,4 +539,7 @@ int amdgpu_ib_pool_init(struct amdgpu_device *adev);
> >>>  void amdgpu_ib_pool_fini(struct amdgpu_device *adev);
> >>>  int amdgpu_ib_ring_tests(struct amdgpu_device *adev);
> >>>  bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring);
> >>> +void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
> >>> +                                      u64 bad_wptr, u32 bad_seq);
> >>> +int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring);
> >>>  #endif
> >>
>

Re: [PATCH 07/28] drm/amdgpu: track ring state associated with a job

[Christian König]

On 6/3/25 16:27, Alex Deucher wrote:
>>> I'm not quite sure I understand what you are proposing.  Is the idea
>>> to track all of the jobs associated with a particular process and then
>>> when we reset a queue, skip all of the ring contents associated with
>>> those jobs and signal and set the error on all of their job fences?
>>
>> More or less yes, I think that is what is needed here.
>>
>> A simple example: Unigine Heaven in window mode on an X server. Each frame usually results in 3 job submissions from unigine, plus one submission from X to copy the result it into the displayed frame.
>>
>> When we now assume that we can schedule 4 jobs at a time on the ring we get: U1, U2, U3, X1 | U4, U5, U6, X2 | U7, U8, U9, X3....
>>
>> Let's assume U4 hangs and we initiate a queue reset, in this case we definately need to skip U5 and U6 as well because they belonged to the same context and depend on each other. Only skipping U4 would certainly crash the GPU again.
> 
> In that case, we'd just reset the queue again, but I agree it would be
> a nicer experience to skip all of the jobs for that app.

Well, that is a must have I think. Otherwise problems only get worse and worse after a while and we can't let submissions run into timeouts over and over again. 

>>
>> X2 also dependet on U6, but that submission is from X and totally innocent and rendering garbage for the window content is probably ok considering that the application just crashed.
>>
>>> What about cross ring dependencies?
>>
>> For gang submission we would need to do a queue reset for both the gfx and compute queue to get out of this again. But that is probably ok since each queue can timeout on its own.
>>
>> We also don't need to track the jobs per process, just looking if job->base.sched_fence->finished.context changes should be sufficient.
> 
> Something like this?
> 
>         for (i = 0; i <= ring->fence_drv.num_fences_mask; i++) {
>                 ptr = &ring->fence_drv.fences[i];
>             old = rcu_dereference_protected(*ptr, 1);
>                 if (old && old->ops == &amdgpu_job_fence_ops) {
>                         struct amdgpu_job *other_job =
>                 container_of(old, struct amdgpu_job, hw_fence.base);
> 
>                         if (other_job->base.sched_fence->finished.context ==
>                             job->base.sched_fence->finished.context) {
>                                 struct amdgpu_fence *am_fence =
> &other_job->hw_fence;
>                                 // skip the ring contents associated
> with this context
>                         }
>                 }
>         }

I would copy job->base.sched_fence->finished.context into the HW fence when it is created, but apart from that yes that is exactly what I had in mind.

Christian.


> 
> Thanks,
> 
> Alex
> 
> 
>>
>> Regards,
>> Christian.
>>
>>>
>>> Alex
>>>
>>>>
>>>> Regards,
>>>> Christian.
>>>>
>>>>
>>>>>       return 0;
>>>>>  }
>>>>>
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
>>>>> index a0fab947143b5..f0f752284b925 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
>>>>> @@ -91,6 +91,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
>>>>>       struct amdgpu_job *job = to_amdgpu_job(s_job);
>>>>>       struct amdgpu_task_info *ti;
>>>>>       struct amdgpu_device *adev = ring->adev;
>>>>> +     struct dma_fence *fence = &job->hw_fence;
>>>>>       int idx;
>>>>>       int r;
>>>>>
>>>>> @@ -154,8 +155,10 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
>>>>>               else
>>>>>                       is_guilty = true;
>>>>>
>>>>> -             if (is_guilty)
>>>>> +             if (is_guilty) {
>>>>> +                     amdgpu_ring_backup_unprocessed_jobs(ring, job->ring_wptr, fence->seqno);
>>>>>                       dma_fence_set_error(&s_job->s_fence->finished, -ETIME);
>>>>> +             }
>>>>>
>>>>>               r = amdgpu_ring_reset(ring, job->vmid);
>>>>>               if (!r) {
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
>>>>> index f2c049129661f..c2ed0edb5179d 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
>>>>> @@ -79,6 +79,8 @@ struct amdgpu_job {
>>>>>       /* enforce isolation */
>>>>>       bool                    enforce_isolation;
>>>>>       bool                    run_cleaner_shader;
>>>>> +     /* wptr for the job for resets */
>>>>> +     uint32_t                ring_wptr;
>>>>>
>>>>>       uint32_t                num_ibs;
>>>>>       struct amdgpu_ib        ibs[];
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
>>>>> index 426834806fbf2..909b121d432cb 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
>>>>> @@ -333,6 +333,12 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
>>>>>       /*  Initialize cached_rptr to 0 */
>>>>>       ring->cached_rptr = 0;
>>>>>
>>>>> +     if (!ring->ring_backup) {
>>>>> +             ring->ring_backup = kvzalloc(ring->ring_size, GFP_KERNEL);
>>>>> +             if (!ring->ring_backup)
>>>>> +                     return -ENOMEM;
>>>>> +     }
>>>>> +
>>>>>       /* Allocate ring buffer */
>>>>>       if (ring->ring_obj == NULL) {
>>>>>               r = amdgpu_bo_create_kernel(adev, ring->ring_size + ring->funcs->extra_dw, PAGE_SIZE,
>>>>> @@ -342,6 +348,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
>>>>>                                           (void **)&ring->ring);
>>>>>               if (r) {
>>>>>                       dev_err(adev->dev, "(%d) ring create failed\n", r);
>>>>> +                     kvfree(ring->ring_backup);
>>>>>                       return r;
>>>>>               }
>>>>>               amdgpu_ring_clear_ring(ring);
>>>>> @@ -385,6 +392,8 @@ void amdgpu_ring_fini(struct amdgpu_ring *ring)
>>>>>       amdgpu_bo_free_kernel(&ring->ring_obj,
>>>>>                             &ring->gpu_addr,
>>>>>                             (void **)&ring->ring);
>>>>> +     kvfree(ring->ring_backup);
>>>>> +     ring->ring_backup = NULL;
>>>>>
>>>>>       dma_fence_put(ring->vmid_wait);
>>>>>       ring->vmid_wait = NULL;
>>>>> @@ -753,3 +762,40 @@ bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring)
>>>>>
>>>>>       return true;
>>>>>  }
>>>>> +
>>>>> +void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
>>>>> +                                      u64 bad_wptr, u32 bad_seq)
>>>>> +{
>>>>> +     unsigned int entries_to_copy = ring->wptr - bad_wptr;
>>>>> +     unsigned int idx, i;
>>>>> +
>>>>> +     for (i = 0; i < entries_to_copy; i++) {
>>>>> +             idx = (bad_wptr + i) & ring->buf_mask;
>>>>> +             ring->ring_backup[i] = ring->ring[idx];
>>>>> +     }
>>>>> +     ring->ring_backup_entries_to_copy = entries_to_copy;
>>>>> +     ring->ring_backup_seq = bad_seq;
>>>>> +}
>>>>> +
>>>>> +int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring)
>>>>> +{
>>>>> +     unsigned int i;
>>>>> +     int r;
>>>>> +
>>>>> +     /* signal the fence of the bad job */
>>>>> +     amdgpu_fence_driver_seq_force_completion(ring, ring->ring_backup_seq);
>>>>> +     /* verify that the ring is functional */
>>>>> +     r = amdgpu_ring_test_ring(ring);
>>>>> +     if (r)
>>>>> +             return r;
>>>>> +     /* re-emit the unprocessed ring contents */
>>>>> +     if (ring->ring_backup_entries_to_copy) {
>>>>> +             if (amdgpu_ring_alloc(ring, ring->ring_backup_entries_to_copy))
>>>>> +                     return -ENOMEM;
>>>>> +             for (i = 0; i < ring->ring_backup_entries_to_copy; i++)
>>>>> +                     amdgpu_ring_write(ring, ring->ring_backup[i]);
>>>>> +             amdgpu_ring_commit(ring);
>>>>> +     }
>>>>> +
>>>>> +     return r;
>>>>> +}
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
>>>>> index b95b471107692..fd08449eee33f 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
>>>>> @@ -132,6 +132,8 @@ extern const struct drm_sched_backend_ops amdgpu_sched_ops;
>>>>>  void amdgpu_fence_driver_clear_job_fences(struct amdgpu_ring *ring);
>>>>>  void amdgpu_fence_driver_set_error(struct amdgpu_ring *ring, int error);
>>>>>  void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring);
>>>>> +void amdgpu_fence_driver_seq_force_completion(struct amdgpu_ring *ring,
>>>>> +                                           u32 seq);
>>>>>
>>>>>  int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring);
>>>>>  int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring,
>>>>> @@ -268,6 +270,9 @@ struct amdgpu_ring {
>>>>>
>>>>>       struct amdgpu_bo        *ring_obj;
>>>>>       uint32_t                *ring;
>>>>> +     uint32_t                *ring_backup;
>>>>> +     uint32_t                ring_backup_seq;
>>>>> +     unsigned int            ring_backup_entries_to_copy;
>>>>>       unsigned                rptr_offs;
>>>>>       u64                     rptr_gpu_addr;
>>>>>       volatile u32            *rptr_cpu_addr;
>>>>> @@ -534,4 +539,7 @@ int amdgpu_ib_pool_init(struct amdgpu_device *adev);
>>>>>  void amdgpu_ib_pool_fini(struct amdgpu_device *adev);
>>>>>  int amdgpu_ib_ring_tests(struct amdgpu_device *adev);
>>>>>  bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring);
>>>>> +void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
>>>>> +                                      u64 bad_wptr, u32 bad_seq);
>>>>> +int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring);
>>>>>  #endif
>>>>
>>

Re: [PATCH 07/28] drm/amdgpu: track ring state associated with a job

[Alex Deucher]

On Tue, Jun 3, 2025 at 11:01 AM Christian König
<christian.koenig@amd.com> wrote:
>
> On 6/3/25 16:27, Alex Deucher wrote:
> >>> I'm not quite sure I understand what you are proposing.  Is the idea
> >>> to track all of the jobs associated with a particular process and then
> >>> when we reset a queue, skip all of the ring contents associated with
> >>> those jobs and signal and set the error on all of their job fences?
> >>
> >> More or less yes, I think that is what is needed here.
> >>
> >> A simple example: Unigine Heaven in window mode on an X server. Each frame usually results in 3 job submissions from unigine, plus one submission from X to copy the result it into the displayed frame.
> >>
> >> When we now assume that we can schedule 4 jobs at a time on the ring we get: U1, U2, U3, X1 | U4, U5, U6, X2 | U7, U8, U9, X3....
> >>
> >> Let's assume U4 hangs and we initiate a queue reset, in this case we definately need to skip U5 and U6 as well because they belonged to the same context and depend on each other. Only skipping U4 would certainly crash the GPU again.
> >
> > In that case, we'd just reset the queue again, but I agree it would be
> > a nicer experience to skip all of the jobs for that app.
>
> Well, that is a must have I think. Otherwise problems only get worse and worse after a while and we can't let submissions run into timeouts over and over again.
>
> >>
> >> X2 also dependet on U6, but that submission is from X and totally innocent and rendering garbage for the window content is probably ok considering that the application just crashed.
> >>
> >>> What about cross ring dependencies?
> >>
> >> For gang submission we would need to do a queue reset for both the gfx and compute queue to get out of this again. But that is probably ok since each queue can timeout on its own.
> >>
> >> We also don't need to track the jobs per process, just looking if job->base.sched_fence->finished.context changes should be sufficient.
> >
> > Something like this?
> >
> >         for (i = 0; i <= ring->fence_drv.num_fences_mask; i++) {
> >                 ptr = &ring->fence_drv.fences[i];
> >             old = rcu_dereference_protected(*ptr, 1);
> >                 if (old && old->ops == &amdgpu_job_fence_ops) {
> >                         struct amdgpu_job *other_job =
> >                 container_of(old, struct amdgpu_job, hw_fence.base);
> >
> >                         if (other_job->base.sched_fence->finished.context ==
> >                             job->base.sched_fence->finished.context) {
> >                                 struct amdgpu_fence *am_fence =
> > &other_job->hw_fence;
> >                                 // skip the ring contents associated
> > with this context
> >                         }
> >                 }
> >         }
>
> I would copy job->base.sched_fence->finished.context into the HW fence when it is created, but apart from that yes that is exactly what I had in mind.

Where and when does finished.context get set?

Alex

>
> Christian.
>
>
> >
> > Thanks,
> >
> > Alex
> >
> >
> >>
> >> Regards,
> >> Christian.
> >>
> >>>
> >>> Alex
> >>>
> >>>>
> >>>> Regards,
> >>>> Christian.
> >>>>
> >>>>
> >>>>>       return 0;
> >>>>>  }
> >>>>>
> >>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> >>>>> index a0fab947143b5..f0f752284b925 100644
> >>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> >>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> >>>>> @@ -91,6 +91,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
> >>>>>       struct amdgpu_job *job = to_amdgpu_job(s_job);
> >>>>>       struct amdgpu_task_info *ti;
> >>>>>       struct amdgpu_device *adev = ring->adev;
> >>>>> +     struct dma_fence *fence = &job->hw_fence;
> >>>>>       int idx;
> >>>>>       int r;
> >>>>>
> >>>>> @@ -154,8 +155,10 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
> >>>>>               else
> >>>>>                       is_guilty = true;
> >>>>>
> >>>>> -             if (is_guilty)
> >>>>> +             if (is_guilty) {
> >>>>> +                     amdgpu_ring_backup_unprocessed_jobs(ring, job->ring_wptr, fence->seqno);
> >>>>>                       dma_fence_set_error(&s_job->s_fence->finished, -ETIME);
> >>>>> +             }
> >>>>>
> >>>>>               r = amdgpu_ring_reset(ring, job->vmid);
> >>>>>               if (!r) {
> >>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> >>>>> index f2c049129661f..c2ed0edb5179d 100644
> >>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> >>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> >>>>> @@ -79,6 +79,8 @@ struct amdgpu_job {
> >>>>>       /* enforce isolation */
> >>>>>       bool                    enforce_isolation;
> >>>>>       bool                    run_cleaner_shader;
> >>>>> +     /* wptr for the job for resets */
> >>>>> +     uint32_t                ring_wptr;
> >>>>>
> >>>>>       uint32_t                num_ibs;
> >>>>>       struct amdgpu_ib        ibs[];
> >>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> >>>>> index 426834806fbf2..909b121d432cb 100644
> >>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> >>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> >>>>> @@ -333,6 +333,12 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
> >>>>>       /*  Initialize cached_rptr to 0 */
> >>>>>       ring->cached_rptr = 0;
> >>>>>
> >>>>> +     if (!ring->ring_backup) {
> >>>>> +             ring->ring_backup = kvzalloc(ring->ring_size, GFP_KERNEL);
> >>>>> +             if (!ring->ring_backup)
> >>>>> +                     return -ENOMEM;
> >>>>> +     }
> >>>>> +
> >>>>>       /* Allocate ring buffer */
> >>>>>       if (ring->ring_obj == NULL) {
> >>>>>               r = amdgpu_bo_create_kernel(adev, ring->ring_size + ring->funcs->extra_dw, PAGE_SIZE,
> >>>>> @@ -342,6 +348,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
> >>>>>                                           (void **)&ring->ring);
> >>>>>               if (r) {
> >>>>>                       dev_err(adev->dev, "(%d) ring create failed\n", r);
> >>>>> +                     kvfree(ring->ring_backup);
> >>>>>                       return r;
> >>>>>               }
> >>>>>               amdgpu_ring_clear_ring(ring);
> >>>>> @@ -385,6 +392,8 @@ void amdgpu_ring_fini(struct amdgpu_ring *ring)
> >>>>>       amdgpu_bo_free_kernel(&ring->ring_obj,
> >>>>>                             &ring->gpu_addr,
> >>>>>                             (void **)&ring->ring);
> >>>>> +     kvfree(ring->ring_backup);
> >>>>> +     ring->ring_backup = NULL;
> >>>>>
> >>>>>       dma_fence_put(ring->vmid_wait);
> >>>>>       ring->vmid_wait = NULL;
> >>>>> @@ -753,3 +762,40 @@ bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring)
> >>>>>
> >>>>>       return true;
> >>>>>  }
> >>>>> +
> >>>>> +void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
> >>>>> +                                      u64 bad_wptr, u32 bad_seq)
> >>>>> +{
> >>>>> +     unsigned int entries_to_copy = ring->wptr - bad_wptr;
> >>>>> +     unsigned int idx, i;
> >>>>> +
> >>>>> +     for (i = 0; i < entries_to_copy; i++) {
> >>>>> +             idx = (bad_wptr + i) & ring->buf_mask;
> >>>>> +             ring->ring_backup[i] = ring->ring[idx];
> >>>>> +     }
> >>>>> +     ring->ring_backup_entries_to_copy = entries_to_copy;
> >>>>> +     ring->ring_backup_seq = bad_seq;
> >>>>> +}
> >>>>> +
> >>>>> +int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring)
> >>>>> +{
> >>>>> +     unsigned int i;
> >>>>> +     int r;
> >>>>> +
> >>>>> +     /* signal the fence of the bad job */
> >>>>> +     amdgpu_fence_driver_seq_force_completion(ring, ring->ring_backup_seq);
> >>>>> +     /* verify that the ring is functional */
> >>>>> +     r = amdgpu_ring_test_ring(ring);
> >>>>> +     if (r)
> >>>>> +             return r;
> >>>>> +     /* re-emit the unprocessed ring contents */
> >>>>> +     if (ring->ring_backup_entries_to_copy) {
> >>>>> +             if (amdgpu_ring_alloc(ring, ring->ring_backup_entries_to_copy))
> >>>>> +                     return -ENOMEM;
> >>>>> +             for (i = 0; i < ring->ring_backup_entries_to_copy; i++)
> >>>>> +                     amdgpu_ring_write(ring, ring->ring_backup[i]);
> >>>>> +             amdgpu_ring_commit(ring);
> >>>>> +     }
> >>>>> +
> >>>>> +     return r;
> >>>>> +}
> >>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> >>>>> index b95b471107692..fd08449eee33f 100644
> >>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> >>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> >>>>> @@ -132,6 +132,8 @@ extern const struct drm_sched_backend_ops amdgpu_sched_ops;
> >>>>>  void amdgpu_fence_driver_clear_job_fences(struct amdgpu_ring *ring);
> >>>>>  void amdgpu_fence_driver_set_error(struct amdgpu_ring *ring, int error);
> >>>>>  void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring);
> >>>>> +void amdgpu_fence_driver_seq_force_completion(struct amdgpu_ring *ring,
> >>>>> +                                           u32 seq);
> >>>>>
> >>>>>  int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring);
> >>>>>  int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring,
> >>>>> @@ -268,6 +270,9 @@ struct amdgpu_ring {
> >>>>>
> >>>>>       struct amdgpu_bo        *ring_obj;
> >>>>>       uint32_t                *ring;
> >>>>> +     uint32_t                *ring_backup;
> >>>>> +     uint32_t                ring_backup_seq;
> >>>>> +     unsigned int            ring_backup_entries_to_copy;
> >>>>>       unsigned                rptr_offs;
> >>>>>       u64                     rptr_gpu_addr;
> >>>>>       volatile u32            *rptr_cpu_addr;
> >>>>> @@ -534,4 +539,7 @@ int amdgpu_ib_pool_init(struct amdgpu_device *adev);
> >>>>>  void amdgpu_ib_pool_fini(struct amdgpu_device *adev);
> >>>>>  int amdgpu_ib_ring_tests(struct amdgpu_device *adev);
> >>>>>  bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring);
> >>>>> +void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
> >>>>> +                                      u64 bad_wptr, u32 bad_seq);
> >>>>> +int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring);
> >>>>>  #endif
> >>>>
> >>
>

Re: [PATCH 07/28] drm/amdgpu: track ring state associated with a job

[Alex Deucher]

On Tue, Jun 3, 2025 at 11:18 AM Alex Deucher <alexdeucher@gmail.com> wrote:
>
> On Tue, Jun 3, 2025 at 11:01 AM Christian König
> <christian.koenig@amd.com> wrote:
> >
> > On 6/3/25 16:27, Alex Deucher wrote:
> > >>> I'm not quite sure I understand what you are proposing.  Is the idea
> > >>> to track all of the jobs associated with a particular process and then
> > >>> when we reset a queue, skip all of the ring contents associated with
> > >>> those jobs and signal and set the error on all of their job fences?
> > >>
> > >> More or less yes, I think that is what is needed here.
> > >>
> > >> A simple example: Unigine Heaven in window mode on an X server. Each frame usually results in 3 job submissions from unigine, plus one submission from X to copy the result it into the displayed frame.
> > >>
> > >> When we now assume that we can schedule 4 jobs at a time on the ring we get: U1, U2, U3, X1 | U4, U5, U6, X2 | U7, U8, U9, X3....
> > >>
> > >> Let's assume U4 hangs and we initiate a queue reset, in this case we definately need to skip U5 and U6 as well because they belonged to the same context and depend on each other. Only skipping U4 would certainly crash the GPU again.
> > >
> > > In that case, we'd just reset the queue again, but I agree it would be
> > > a nicer experience to skip all of the jobs for that app.
> >
> > Well, that is a must have I think. Otherwise problems only get worse and worse after a while and we can't let submissions run into timeouts over and over again.
> >
> > >>
> > >> X2 also dependet on U6, but that submission is from X and totally innocent and rendering garbage for the window content is probably ok considering that the application just crashed.
> > >>
> > >>> What about cross ring dependencies?
> > >>
> > >> For gang submission we would need to do a queue reset for both the gfx and compute queue to get out of this again. But that is probably ok since each queue can timeout on its own.
> > >>
> > >> We also don't need to track the jobs per process, just looking if job->base.sched_fence->finished.context changes should be sufficient.
> > >
> > > Something like this?
> > >
> > >         for (i = 0; i <= ring->fence_drv.num_fences_mask; i++) {
> > >                 ptr = &ring->fence_drv.fences[i];
> > >             old = rcu_dereference_protected(*ptr, 1);
> > >                 if (old && old->ops == &amdgpu_job_fence_ops) {
> > >                         struct amdgpu_job *other_job =
> > >                 container_of(old, struct amdgpu_job, hw_fence.base);
> > >
> > >                         if (other_job->base.sched_fence->finished.context ==
> > >                             job->base.sched_fence->finished.context) {
> > >                                 struct amdgpu_fence *am_fence =
> > > &other_job->hw_fence;
> > >                                 // skip the ring contents associated
> > > with this context
> > >                         }
> > >                 }
> > >         }
> >
> > I would copy job->base.sched_fence->finished.context into the HW fence when it is created, but apart from that yes that is exactly what I had in mind.
>
> Where and when does finished.context get set?

I found it.  It's drm_sched_job_arm().

Alex


>
> Alex
>
> >
> > Christian.
> >
> >
> > >
> > > Thanks,
> > >
> > > Alex
> > >
> > >
> > >>
> > >> Regards,
> > >> Christian.
> > >>
> > >>>
> > >>> Alex
> > >>>
> > >>>>
> > >>>> Regards,
> > >>>> Christian.
> > >>>>
> > >>>>
> > >>>>>       return 0;
> > >>>>>  }
> > >>>>>
> > >>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> > >>>>> index a0fab947143b5..f0f752284b925 100644
> > >>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> > >>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
> > >>>>> @@ -91,6 +91,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
> > >>>>>       struct amdgpu_job *job = to_amdgpu_job(s_job);
> > >>>>>       struct amdgpu_task_info *ti;
> > >>>>>       struct amdgpu_device *adev = ring->adev;
> > >>>>> +     struct dma_fence *fence = &job->hw_fence;
> > >>>>>       int idx;
> > >>>>>       int r;
> > >>>>>
> > >>>>> @@ -154,8 +155,10 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
> > >>>>>               else
> > >>>>>                       is_guilty = true;
> > >>>>>
> > >>>>> -             if (is_guilty)
> > >>>>> +             if (is_guilty) {
> > >>>>> +                     amdgpu_ring_backup_unprocessed_jobs(ring, job->ring_wptr, fence->seqno);
> > >>>>>                       dma_fence_set_error(&s_job->s_fence->finished, -ETIME);
> > >>>>> +             }
> > >>>>>
> > >>>>>               r = amdgpu_ring_reset(ring, job->vmid);
> > >>>>>               if (!r) {
> > >>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> > >>>>> index f2c049129661f..c2ed0edb5179d 100644
> > >>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> > >>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
> > >>>>> @@ -79,6 +79,8 @@ struct amdgpu_job {
> > >>>>>       /* enforce isolation */
> > >>>>>       bool                    enforce_isolation;
> > >>>>>       bool                    run_cleaner_shader;
> > >>>>> +     /* wptr for the job for resets */
> > >>>>> +     uint32_t                ring_wptr;
> > >>>>>
> > >>>>>       uint32_t                num_ibs;
> > >>>>>       struct amdgpu_ib        ibs[];
> > >>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> > >>>>> index 426834806fbf2..909b121d432cb 100644
> > >>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> > >>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> > >>>>> @@ -333,6 +333,12 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
> > >>>>>       /*  Initialize cached_rptr to 0 */
> > >>>>>       ring->cached_rptr = 0;
> > >>>>>
> > >>>>> +     if (!ring->ring_backup) {
> > >>>>> +             ring->ring_backup = kvzalloc(ring->ring_size, GFP_KERNEL);
> > >>>>> +             if (!ring->ring_backup)
> > >>>>> +                     return -ENOMEM;
> > >>>>> +     }
> > >>>>> +
> > >>>>>       /* Allocate ring buffer */
> > >>>>>       if (ring->ring_obj == NULL) {
> > >>>>>               r = amdgpu_bo_create_kernel(adev, ring->ring_size + ring->funcs->extra_dw, PAGE_SIZE,
> > >>>>> @@ -342,6 +348,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
> > >>>>>                                           (void **)&ring->ring);
> > >>>>>               if (r) {
> > >>>>>                       dev_err(adev->dev, "(%d) ring create failed\n", r);
> > >>>>> +                     kvfree(ring->ring_backup);
> > >>>>>                       return r;
> > >>>>>               }
> > >>>>>               amdgpu_ring_clear_ring(ring);
> > >>>>> @@ -385,6 +392,8 @@ void amdgpu_ring_fini(struct amdgpu_ring *ring)
> > >>>>>       amdgpu_bo_free_kernel(&ring->ring_obj,
> > >>>>>                             &ring->gpu_addr,
> > >>>>>                             (void **)&ring->ring);
> > >>>>> +     kvfree(ring->ring_backup);
> > >>>>> +     ring->ring_backup = NULL;
> > >>>>>
> > >>>>>       dma_fence_put(ring->vmid_wait);
> > >>>>>       ring->vmid_wait = NULL;
> > >>>>> @@ -753,3 +762,40 @@ bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring)
> > >>>>>
> > >>>>>       return true;
> > >>>>>  }
> > >>>>> +
> > >>>>> +void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
> > >>>>> +                                      u64 bad_wptr, u32 bad_seq)
> > >>>>> +{
> > >>>>> +     unsigned int entries_to_copy = ring->wptr - bad_wptr;
> > >>>>> +     unsigned int idx, i;
> > >>>>> +
> > >>>>> +     for (i = 0; i < entries_to_copy; i++) {
> > >>>>> +             idx = (bad_wptr + i) & ring->buf_mask;
> > >>>>> +             ring->ring_backup[i] = ring->ring[idx];
> > >>>>> +     }
> > >>>>> +     ring->ring_backup_entries_to_copy = entries_to_copy;
> > >>>>> +     ring->ring_backup_seq = bad_seq;
> > >>>>> +}
> > >>>>> +
> > >>>>> +int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring)
> > >>>>> +{
> > >>>>> +     unsigned int i;
> > >>>>> +     int r;
> > >>>>> +
> > >>>>> +     /* signal the fence of the bad job */
> > >>>>> +     amdgpu_fence_driver_seq_force_completion(ring, ring->ring_backup_seq);
> > >>>>> +     /* verify that the ring is functional */
> > >>>>> +     r = amdgpu_ring_test_ring(ring);
> > >>>>> +     if (r)
> > >>>>> +             return r;
> > >>>>> +     /* re-emit the unprocessed ring contents */
> > >>>>> +     if (ring->ring_backup_entries_to_copy) {
> > >>>>> +             if (amdgpu_ring_alloc(ring, ring->ring_backup_entries_to_copy))
> > >>>>> +                     return -ENOMEM;
> > >>>>> +             for (i = 0; i < ring->ring_backup_entries_to_copy; i++)
> > >>>>> +                     amdgpu_ring_write(ring, ring->ring_backup[i]);
> > >>>>> +             amdgpu_ring_commit(ring);
> > >>>>> +     }
> > >>>>> +
> > >>>>> +     return r;
> > >>>>> +}
> > >>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> > >>>>> index b95b471107692..fd08449eee33f 100644
> > >>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> > >>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
> > >>>>> @@ -132,6 +132,8 @@ extern const struct drm_sched_backend_ops amdgpu_sched_ops;
> > >>>>>  void amdgpu_fence_driver_clear_job_fences(struct amdgpu_ring *ring);
> > >>>>>  void amdgpu_fence_driver_set_error(struct amdgpu_ring *ring, int error);
> > >>>>>  void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring);
> > >>>>> +void amdgpu_fence_driver_seq_force_completion(struct amdgpu_ring *ring,
> > >>>>> +                                           u32 seq);
> > >>>>>
> > >>>>>  int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring);
> > >>>>>  int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring,
> > >>>>> @@ -268,6 +270,9 @@ struct amdgpu_ring {
> > >>>>>
> > >>>>>       struct amdgpu_bo        *ring_obj;
> > >>>>>       uint32_t                *ring;
> > >>>>> +     uint32_t                *ring_backup;
> > >>>>> +     uint32_t                ring_backup_seq;
> > >>>>> +     unsigned int            ring_backup_entries_to_copy;
> > >>>>>       unsigned                rptr_offs;
> > >>>>>       u64                     rptr_gpu_addr;
> > >>>>>       volatile u32            *rptr_cpu_addr;
> > >>>>> @@ -534,4 +539,7 @@ int amdgpu_ib_pool_init(struct amdgpu_device *adev);
> > >>>>>  void amdgpu_ib_pool_fini(struct amdgpu_device *adev);
> > >>>>>  int amdgpu_ib_ring_tests(struct amdgpu_device *adev);
> > >>>>>  bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring);
> > >>>>> +void amdgpu_ring_backup_unprocessed_jobs(struct amdgpu_ring *ring,
> > >>>>> +                                      u64 bad_wptr, u32 bad_seq);
> > >>>>> +int amdgpu_ring_reemit_unprocessed_jobs(struct amdgpu_ring *ring);
> > >>>>>  #endif
> > >>>>
> > >>
> >