[PATCH v2 0/3] drm/panthor: Fix group state reporting

[PATCH v2 0/3] drm/panthor: Fix group state reporting

[Boris Brezillon]

Hello,

What initially started as a simple fix to avoid queueing jobs to a group
that can't be scheduled has turned into a series of three patches
improving the group state reporting. Two of them are actual fixes, the
last one is an improvement to properly report innocence/guiltiness of
a group when a group becomes unusable. With this new
GROUP_STATE_INNOCENT, we can distinguish between
GL_INNOCENT_CONTEXT_RESET and GL_GUILT_CONTEXT_RESET (see this mesa
MR for more details [1]).

Regards,

Boris

[1]https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31868

Boris Brezillon (3):
  drm/panthor: Fail job creation when the group is dead
  drm/panthor: Report group as timedout when we fail to properly suspend
  drm/panthor: Report innocent group kill

 drivers/gpu/drm/panthor/panthor_drv.c   |  2 +-
 drivers/gpu/drm/panthor/panthor_sched.c | 36 ++++++++++++++++++++++---
 include/uapi/drm/panthor_drm.h          |  9 +++++++
 3 files changed, 42 insertions(+), 5 deletions(-)

-- 
2.46.2

[PATCH v2 1/3] drm/panthor: Fail job creation when the group is dead

[Boris Brezillon]

Userspace can use GROUP_SUBMIT errors as a trigger to check the group
state and recreate the group if it became unusable. Make sure we
report an error when the group became unusable.

Changes in v2:
- Add R-bs

Fixes: de8548813824 ("drm/panthor: Add the scheduler logical block")
Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Steven Price <steven.price@arm.com>
Reviewed-by: Liviu Dudau <liviu.dudau@arm.com>
---
 drivers/gpu/drm/panthor/panthor_sched.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/gpu/drm/panthor/panthor_sched.c b/drivers/gpu/drm/panthor/panthor_sched.c
index eb9f6635cc12..eda8fbb276b3 100644
--- a/drivers/gpu/drm/panthor/panthor_sched.c
+++ b/drivers/gpu/drm/panthor/panthor_sched.c
@@ -3688,6 +3688,11 @@ panthor_job_create(struct panthor_file *pfile,
 		goto err_put_job;
 	}
 
+	if (!group_can_run(job->group)) {
+		ret = -EINVAL;
+		goto err_put_job;
+	}
+
 	if (job->queue_idx >= job->group->queue_count ||
 	    !job->group->queues[job->queue_idx]) {
 		ret = -EINVAL;
-- 
2.46.2

[PATCH v2 2/3] drm/panthor: Report group as timedout when we fail to properly suspend

[Boris Brezillon]

If we don't do that, the group is considered usable by userspace, but
all further GROUP_SUBMIT will fail with -EINVAL.

Changes in v2:
- New patch

Fixes: de8548813824 ("drm/panthor: Add the scheduler logical block")
Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
---
 drivers/gpu/drm/panthor/panthor_sched.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/panthor/panthor_sched.c b/drivers/gpu/drm/panthor/panthor_sched.c
index eda8fbb276b3..ef4bec7ff9c7 100644
--- a/drivers/gpu/drm/panthor/panthor_sched.c
+++ b/drivers/gpu/drm/panthor/panthor_sched.c
@@ -602,10 +602,11 @@ struct panthor_group {
 	 * @timedout: True when a timeout occurred on any of the queues owned by
 	 * this group.
 	 *
-	 * Timeouts can be reported by drm_sched or by the FW. In any case, any
-	 * timeout situation is unrecoverable, and the group becomes useless.
-	 * We simply wait for all references to be dropped so we can release the
-	 * group object.
+	 * Timeouts can be reported by drm_sched or by the FW. If a reset is required,
+	 * and the group can't be suspended, this also leads to a timeout. In any case,
+	 * any timeout situation is unrecoverable, and the group becomes useless. We
+	 * simply wait for all references to be dropped so we can release the group
+	 * object.
 	 */
 	bool timedout;
 
@@ -2687,6 +2688,12 @@ void panthor_sched_suspend(struct panthor_device *ptdev)
 		csgs_upd_ctx_init(&upd_ctx);
 		while (slot_mask) {
 			u32 csg_id = ffs(slot_mask) - 1;
+			struct panthor_csg_slot *csg_slot = &sched->csg_slots[csg_id];
+
+			/* We consider group suspension failures as fatal and flag the
+			 * group as unusable by setting timedout=true.
+			 */
+			csg_slot->group->timedout = true;
 
 			csgs_upd_ctx_queue_reqs(ptdev, &upd_ctx, csg_id,
 						CSG_STATE_TERMINATE,
-- 
2.46.2

[PATCH v2 3/3] drm/panthor: Report innocent group kill

[Boris Brezillon]

Groups can be killed during a reset even though they did nothing wrong.
That usually happens when the FW is put in a bad state by other groups,
resulting in group suspension failures when the reset happens.

If we end up in that situation, flag the group innocent and report
innocence through a new DRM_PANTHOR_GROUP_STATE flag.

Bump the minor driver version to reflect the uAPI change.

Changes in v2:
- New patch

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
---
 drivers/gpu/drm/panthor/panthor_drv.c   |  2 +-
 drivers/gpu/drm/panthor/panthor_sched.c | 16 ++++++++++++++++
 include/uapi/drm/panthor_drm.h          |  9 +++++++++
 3 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/panthor/panthor_drv.c b/drivers/gpu/drm/panthor/panthor_drv.c
index ac7e53f6e3f0..f1dff7e0173d 100644
--- a/drivers/gpu/drm/panthor/panthor_drv.c
+++ b/drivers/gpu/drm/panthor/panthor_drv.c
@@ -1507,7 +1507,7 @@ static const struct drm_driver panthor_drm_driver = {
 	.desc = "Panthor DRM driver",
 	.date = "20230801",
 	.major = 1,
-	.minor = 2,
+	.minor = 3,
 
 	.gem_create_object = panthor_gem_create_object,
 	.gem_prime_import_sg_table = drm_gem_shmem_prime_import_sg_table,
diff --git a/drivers/gpu/drm/panthor/panthor_sched.c b/drivers/gpu/drm/panthor/panthor_sched.c
index ef4bec7ff9c7..bb4a8c522fdc 100644
--- a/drivers/gpu/drm/panthor/panthor_sched.c
+++ b/drivers/gpu/drm/panthor/panthor_sched.c
@@ -610,6 +610,16 @@ struct panthor_group {
 	 */
 	bool timedout;
 
+	/**
+	 * @innocent: True when the group becomes unusable because the group suspension
+	 * failed during a reset.
+	 *
+	 * Sometimes the FW was put in a bad state by other groups, causing the group
+	 * suspension happening in the reset path to fail. In that case, we consider the
+	 * group innocent.
+	 */
+	bool innocent;
+
 	/**
 	 * @syncobjs: Pool of per-queue synchronization objects.
 	 *
@@ -2690,6 +2700,12 @@ void panthor_sched_suspend(struct panthor_device *ptdev)
 			u32 csg_id = ffs(slot_mask) - 1;
 			struct panthor_csg_slot *csg_slot = &sched->csg_slots[csg_id];
 
+			/* If the group was still usable before that point, we consider
+			 * it innocent.
+			 */
+			if (group_can_run(csg_slot->group))
+				csg_slot->group->innocent = true;
+
 			/* We consider group suspension failures as fatal and flag the
 			 * group as unusable by setting timedout=true.
 			 */
diff --git a/include/uapi/drm/panthor_drm.h b/include/uapi/drm/panthor_drm.h
index 87c9cb555dd1..b99763cbae48 100644
--- a/include/uapi/drm/panthor_drm.h
+++ b/include/uapi/drm/panthor_drm.h
@@ -923,6 +923,15 @@ enum drm_panthor_group_state_flags {
 	 * When a group ends up with this flag set, no jobs can be submitted to its queues.
 	 */
 	DRM_PANTHOR_GROUP_STATE_FATAL_FAULT = 1 << 1,
+
+	/**
+	 * @DRM_PANTHOR_GROUP_STATE_INNOCENT: Group was killed during a reset caused by other
+	 * groups.
+	 *
+	 * This flag can only be set if DRM_PANTHOR_GROUP_STATE_TIMEDOUT is set and
+	 * DRM_PANTHOR_GROUP_STATE_FATAL_FAULT is not.
+	 */
+	DRM_PANTHOR_GROUP_STATE_INNOCENT = 1 << 2,
 };
 
 /**
-- 
2.46.2

Re: [PATCH v2 2/3] drm/panthor: Report group as timedout when we fail to properly suspend

[Steven Price]

On 28/10/2024 11:48, Boris Brezillon wrote:
> If we don't do that, the group is considered usable by userspace, but
> all further GROUP_SUBMIT will fail with -EINVAL.
> 
> Changes in v2:
> - New patch
> 
> Fixes: de8548813824 ("drm/panthor: Add the scheduler logical block")
> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>

I guess a failure to suspend is close enough to a time out. I think the
only case that might actually matter is dealt with by your next patch.

Reviewed-by: Steven Price <steven.price@arm.com>

> ---
>  drivers/gpu/drm/panthor/panthor_sched.c | 15 +++++++++++----
>  1 file changed, 11 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/gpu/drm/panthor/panthor_sched.c b/drivers/gpu/drm/panthor/panthor_sched.c
> index eda8fbb276b3..ef4bec7ff9c7 100644
> --- a/drivers/gpu/drm/panthor/panthor_sched.c
> +++ b/drivers/gpu/drm/panthor/panthor_sched.c
> @@ -602,10 +602,11 @@ struct panthor_group {
>  	 * @timedout: True when a timeout occurred on any of the queues owned by
>  	 * this group.
>  	 *
> -	 * Timeouts can be reported by drm_sched or by the FW. In any case, any
> -	 * timeout situation is unrecoverable, and the group becomes useless.
> -	 * We simply wait for all references to be dropped so we can release the
> -	 * group object.
> +	 * Timeouts can be reported by drm_sched or by the FW. If a reset is required,
> +	 * and the group can't be suspended, this also leads to a timeout. In any case,
> +	 * any timeout situation is unrecoverable, and the group becomes useless. We
> +	 * simply wait for all references to be dropped so we can release the group
> +	 * object.
>  	 */
>  	bool timedout;
>  
> @@ -2687,6 +2688,12 @@ void panthor_sched_suspend(struct panthor_device *ptdev)
>  		csgs_upd_ctx_init(&upd_ctx);
>  		while (slot_mask) {
>  			u32 csg_id = ffs(slot_mask) - 1;
> +			struct panthor_csg_slot *csg_slot = &sched->csg_slots[csg_id];
> +
> +			/* We consider group suspension failures as fatal and flag the
> +			 * group as unusable by setting timedout=true.
> +			 */
> +			csg_slot->group->timedout = true;
>  
>  			csgs_upd_ctx_queue_reqs(ptdev, &upd_ctx, csg_id,
>  						CSG_STATE_TERMINATE,

Re: [PATCH v2 3/3] drm/panthor: Report innocent group kill

[Steven Price]

On 28/10/2024 11:48, Boris Brezillon wrote:
> Groups can be killed during a reset even though they did nothing wrong.
> That usually happens when the FW is put in a bad state by other groups,
> resulting in group suspension failures when the reset happens.
> 
> If we end up in that situation, flag the group innocent and report
> innocence through a new DRM_PANTHOR_GROUP_STATE flag.

Sadly I don't see any code to return that new flag... Shouldn't there be
a change to panthor_get_get_state()?

> 
> Bump the minor driver version to reflect the uAPI change.
> 
> Changes in v2:
> - New patch
> 
> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> ---
>  drivers/gpu/drm/panthor/panthor_drv.c   |  2 +-
>  drivers/gpu/drm/panthor/panthor_sched.c | 16 ++++++++++++++++
>  include/uapi/drm/panthor_drm.h          |  9 +++++++++
>  3 files changed, 26 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/gpu/drm/panthor/panthor_drv.c b/drivers/gpu/drm/panthor/panthor_drv.c
> index ac7e53f6e3f0..f1dff7e0173d 100644
> --- a/drivers/gpu/drm/panthor/panthor_drv.c
> +++ b/drivers/gpu/drm/panthor/panthor_drv.c
> @@ -1507,7 +1507,7 @@ static const struct drm_driver panthor_drm_driver = {
>  	.desc = "Panthor DRM driver",
>  	.date = "20230801",
>  	.major = 1,
> -	.minor = 2,
> +	.minor = 3,
>  
>  	.gem_create_object = panthor_gem_create_object,
>  	.gem_prime_import_sg_table = drm_gem_shmem_prime_import_sg_table,
> diff --git a/drivers/gpu/drm/panthor/panthor_sched.c b/drivers/gpu/drm/panthor/panthor_sched.c
> index ef4bec7ff9c7..bb4a8c522fdc 100644
> --- a/drivers/gpu/drm/panthor/panthor_sched.c
> +++ b/drivers/gpu/drm/panthor/panthor_sched.c
> @@ -610,6 +610,16 @@ struct panthor_group {
>  	 */
>  	bool timedout;
>  
> +	/**
> +	 * @innocent: True when the group becomes unusable because the group suspension
> +	 * failed during a reset.
> +	 *
> +	 * Sometimes the FW was put in a bad state by other groups, causing the group
> +	 * suspension happening in the reset path to fail. In that case, we consider the
> +	 * group innocent.
> +	 */
> +	bool innocent;
> +
>  	/**
>  	 * @syncobjs: Pool of per-queue synchronization objects.
>  	 *
> @@ -2690,6 +2700,12 @@ void panthor_sched_suspend(struct panthor_device *ptdev)
>  			u32 csg_id = ffs(slot_mask) - 1;
>  			struct panthor_csg_slot *csg_slot = &sched->csg_slots[csg_id];
>  
> +			/* If the group was still usable before that point, we consider
> +			 * it innocent.
> +			 */
> +			if (group_can_run(csg_slot->group))
> +				csg_slot->group->innocent = true;
> +
>  			/* We consider group suspension failures as fatal and flag the
>  			 * group as unusable by setting timedout=true.
>  			 */
> diff --git a/include/uapi/drm/panthor_drm.h b/include/uapi/drm/panthor_drm.h
> index 87c9cb555dd1..b99763cbae48 100644
> --- a/include/uapi/drm/panthor_drm.h
> +++ b/include/uapi/drm/panthor_drm.h
> @@ -923,6 +923,15 @@ enum drm_panthor_group_state_flags {
>  	 * When a group ends up with this flag set, no jobs can be submitted to its queues.
>  	 */
>  	DRM_PANTHOR_GROUP_STATE_FATAL_FAULT = 1 << 1,
> +
> +	/**
> +	 * @DRM_PANTHOR_GROUP_STATE_INNOCENT: Group was killed during a reset caused by other
> +	 * groups.
> +	 *
> +	 * This flag can only be set if DRM_PANTHOR_GROUP_STATE_TIMEDOUT is set and
> +	 * DRM_PANTHOR_GROUP_STATE_FATAL_FAULT is not.
> +	 */
> +	DRM_PANTHOR_GROUP_STATE_INNOCENT = 1 << 2,
>  };
>  
>  /**

Re: [PATCH v2 3/3] drm/panthor: Report innocent group kill

[Boris Brezillon]

On Mon, 28 Oct 2024 15:31:19 +0000
Steven Price <steven.price@arm.com> wrote:

> On 28/10/2024 11:48, Boris Brezillon wrote:
> > Groups can be killed during a reset even though they did nothing wrong.
> > That usually happens when the FW is put in a bad state by other groups,
> > resulting in group suspension failures when the reset happens.
> > 
> > If we end up in that situation, flag the group innocent and report
> > innocence through a new DRM_PANTHOR_GROUP_STATE flag.  
> 
> Sadly I don't see any code to return that new flag... Shouldn't there be
> a change to panthor_get_get_state()?

My bad, I had those changes in a debug commit and somehow overlooked the
following two lines when preparing the final version:

@@ -3570,6 +3586,8 @@ int panthor_group_get_state(struct panthor_file
*pfile, get_state->state |= DRM_PANTHOR_GROUP_STATE_FATAL_FAULT;
                get_state->fatal_queues = group->fatal_queues;
        }
+       if (group->innocent)
+               get_state->state |= DRM_PANTHOR_GROUP_STATE_INNOCENT;
        mutex_unlock(&sched->lock);
 
        group_put(group);

Will send a v3 shortly.

Re: [PATCH v2 2/3] drm/panthor: Report group as timedout when we fail to properly suspend

[Liviu Dudau]

On Mon, Oct 28, 2024 at 12:48:14PM +0100, Boris Brezillon wrote:
> If we don't do that, the group is considered usable by userspace, but
> all further GROUP_SUBMIT will fail with -EINVAL.
> 
> Changes in v2:
> - New patch
> 
> Fixes: de8548813824 ("drm/panthor: Add the scheduler logical block")
> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>

Reviewed-by: Liviu Dudau <liviu.dudau@arm.com>

Best regards,
Liviu

> ---
>  drivers/gpu/drm/panthor/panthor_sched.c | 15 +++++++++++----
>  1 file changed, 11 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/gpu/drm/panthor/panthor_sched.c b/drivers/gpu/drm/panthor/panthor_sched.c
> index eda8fbb276b3..ef4bec7ff9c7 100644
> --- a/drivers/gpu/drm/panthor/panthor_sched.c
> +++ b/drivers/gpu/drm/panthor/panthor_sched.c
> @@ -602,10 +602,11 @@ struct panthor_group {
>  	 * @timedout: True when a timeout occurred on any of the queues owned by
>  	 * this group.
>  	 *
> -	 * Timeouts can be reported by drm_sched or by the FW. In any case, any
> -	 * timeout situation is unrecoverable, and the group becomes useless.
> -	 * We simply wait for all references to be dropped so we can release the
> -	 * group object.
> +	 * Timeouts can be reported by drm_sched or by the FW. If a reset is required,
> +	 * and the group can't be suspended, this also leads to a timeout. In any case,
> +	 * any timeout situation is unrecoverable, and the group becomes useless. We
> +	 * simply wait for all references to be dropped so we can release the group
> +	 * object.
>  	 */
>  	bool timedout;
>  
> @@ -2687,6 +2688,12 @@ void panthor_sched_suspend(struct panthor_device *ptdev)
>  		csgs_upd_ctx_init(&upd_ctx);
>  		while (slot_mask) {
>  			u32 csg_id = ffs(slot_mask) - 1;
> +			struct panthor_csg_slot *csg_slot = &sched->csg_slots[csg_id];
> +
> +			/* We consider group suspension failures as fatal and flag the
> +			 * group as unusable by setting timedout=true.
> +			 */
> +			csg_slot->group->timedout = true;
>  
>  			csgs_upd_ctx_queue_reqs(ptdev, &upd_ctx, csg_id,
>  						CSG_STATE_TERMINATE,
> -- 
> 2.46.2
> 

-- 
====================
| I would like to |
| fix the world,  |
| but they're not |
| giving me the   |
 \ source code!  /
  ---------------
    ¯\_(ツ)_/¯