[PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

[PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

[Liu01 Tong]

During process kill, drm_sched_entity_flush() will kill the vm
entities. The following job submissions of this process will fail, and
the resources of these jobs have not been released, nor have the fences
been signalled, causing tasks to hang.

Fix by not doing job init when the entity is stopped. And when the job
is already submitted, free the job resource if the entity is stopped.

Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com>
Signed-off-by: Lin.Cao <lincao12@amd.com>
---
 drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------
 drivers/gpu/drm/scheduler/sched_main.c   |  5 +++++
 2 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c
index ac678de7fe5e..1e744b2eb2db 100644
--- a/drivers/gpu/drm/scheduler/sched_entity.c
+++ b/drivers/gpu/drm/scheduler/sched_entity.c
@@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
 	bool first;
 	ktime_t submit_ts;
 
+	if (entity->stopped) {
+		DRM_ERROR("Trying to push job to a killed entity\n");
+		INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work);
+		schedule_work(&sched_job->work);
+		return;
+	}
+
 	trace_drm_sched_job(sched_job, entity);
 	atomic_inc(entity->rq->sched->score);
 	WRITE_ONCE(entity->last_user, current->group_leader);
@@ -589,12 +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
 
 		/* Add the entity to the run queue */
 		spin_lock(&entity->lock);
-		if (entity->stopped) {
-			spin_unlock(&entity->lock);
-
-			DRM_ERROR("Trying to push to a killed entity\n");
-			return;
-		}
 
 		rq = entity->rq;
 		sched = rq->sched;
diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
index bfea608a7106..c15b17d9ffe3 100644
--- a/drivers/gpu/drm/scheduler/sched_main.c
+++ b/drivers/gpu/drm/scheduler/sched_main.c
@@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job,
 		return -ENOENT;
 	}
 
+	if (unlikely(entity->stopped)) {
+		pr_err("*ERROR* %s: entity is stopped!\n", __func__);
+		return -EINVAL;
+	}
+
 	if (unlikely(!credits)) {
 		pr_err("*ERROR* %s: credits cannot be 0!\n", __func__);
 		return -EINVAL;
-- 
2.34.1

Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

[Philipp Stanner]

Hi,

title: this patch changes nothing in amdgpu.

Thus, the prefix must be drm/sched: Fix […]


Furthermore, please use scripts/get_maintainer. A few relevant folks
are missing. +Cc Danilo, Matthew


On Mon, 2025-08-11 at 15:20 +0800, Liu01 Tong wrote:
> During process kill, drm_sched_entity_flush() will kill the vm
> entities.
> 

What is a "vm entity"? This seems to be driver-specific language.


>  The following job submissions of this process will fail, and
> the resources of these jobs have not been released, nor have the fences
> been signalled, causing tasks to hang.
> 
> Fix by not doing job init when the entity is stopped. And when the job
> is already submitted, free the job resource if the entity is stopped.

I'm not sure I can fully follow. Can you give more details on why that
bug doesn't always occur?

In general: Why is this something that needs to be fixed in the
scheduler? amdgpu knows when it killed an entity. Why can't it stop
submitting thereafter?

> 
> Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com>
> Signed-off-by: Lin.Cao <lincao12@amd.com>

Two authors? AFAIK should contain a Co-authored-by tag then.

> ---
>  drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------
>  drivers/gpu/drm/scheduler/sched_main.c   |  5 +++++
>  2 files changed, 12 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c
> index ac678de7fe5e..1e744b2eb2db 100644
> --- a/drivers/gpu/drm/scheduler/sched_entity.c
> +++ b/drivers/gpu/drm/scheduler/sched_entity.c
> @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
>  	bool first;
>  	ktime_t submit_ts;
>  
> +	if (entity->stopped) {

This screams "race!" because you're checking for the entity being
stopped now without the lock, as was done before.

That's definitely a no-go because that has caused big trouble and the
past and is still causing trouble right now at other places where the
lock was not taken:

https://lore.kernel.org/dri-devel/20250731093008.45267-2-phasta@kernel.org/


> +		DRM_ERROR("Trying to push job to a killed entity\n");
> +		INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work);
> +		schedule_work(&sched_job->work);
> +		return;
> +	}
> +
>  	trace_drm_sched_job(sched_job, entity);
>  	atomic_inc(entity->rq->sched->score);
>  	WRITE_ONCE(entity->last_user, current->group_leader);
> @@ -589,12 +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
>  
>  		/* Add the entity to the run queue */
>  		spin_lock(&entity->lock);
> -		if (entity->stopped) {
> -			spin_unlock(&entity->lock);
> -
> -			DRM_ERROR("Trying to push to a killed entity\n");
> -			return;
> -		}
>  
>  		rq = entity->rq;
>  		sched = rq->sched;
> diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
> index bfea608a7106..c15b17d9ffe3 100644
> --- a/drivers/gpu/drm/scheduler/sched_main.c
> +++ b/drivers/gpu/drm/scheduler/sched_main.c
> @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job,
>  		return -ENOENT;
>  	}
>  
> +	if (unlikely(entity->stopped)) {
> +		pr_err("*ERROR* %s: entity is stopped!\n", __func__);
> +		return -EINVAL;
> +	}

Same here, racy.


Regards,
Philipp

> +
>  	if (unlikely(!credits)) {
>  		pr_err("*ERROR* %s: credits cannot be 0!\n", __func__);
>  		return -EINVAL;

Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

[Philipp Stanner]

On Mon, 2025-08-11 at 10:18 +0200, Philipp Stanner wrote:
> Hi,
> 
> title: this patch changes nothing in amdgpu.
> 
> Thus, the prefix must be drm/sched: Fix […]
> 
> 
> Furthermore, please use scripts/get_maintainer. A few relevant folks
> are missing. +Cc Danilo, Matthew

Oh, never mind, just overlooked them because the names weren't spelled
out. My bad.

P.

> 
> 
> On Mon, 2025-08-11 at 15:20 +0800, Liu01 Tong wrote:
> > During process kill, drm_sched_entity_flush() will kill the vm
> > entities.
> > 
> 
> What is a "vm entity"? This seems to be driver-specific language.
> 
> 
> >  The following job submissions of this process will fail, and
> > the resources of these jobs have not been released, nor have the fences
> > been signalled, causing tasks to hang.
> > 
> > Fix by not doing job init when the entity is stopped. And when the job
> > is already submitted, free the job resource if the entity is stopped.
> 
> I'm not sure I can fully follow. Can you give more details on why that
> bug doesn't always occur?
> 
> In general: Why is this something that needs to be fixed in the
> scheduler? amdgpu knows when it killed an entity. Why can't it stop
> submitting thereafter?
> 
> > 
> > Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com>
> > Signed-off-by: Lin.Cao <lincao12@amd.com>
> 
> Two authors? AFAIK should contain a Co-authored-by tag then.
> 
> > ---
> >  drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------
> >  drivers/gpu/drm/scheduler/sched_main.c   |  5 +++++
> >  2 files changed, 12 insertions(+), 6 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c
> > index ac678de7fe5e..1e744b2eb2db 100644
> > --- a/drivers/gpu/drm/scheduler/sched_entity.c
> > +++ b/drivers/gpu/drm/scheduler/sched_entity.c
> > @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
> >  	bool first;
> >  	ktime_t submit_ts;
> >  
> > +	if (entity->stopped) {
> 
> This screams "race!" because you're checking for the entity being
> stopped now without the lock, as was done before.
> 
> That's definitely a no-go because that has caused big trouble and the
> past and is still causing trouble right now at other places where the
> lock was not taken:
> 
> https://lore.kernel.org/dri-devel/20250731093008.45267-2-phasta@kernel.org/
> 
> 
> > +		DRM_ERROR("Trying to push job to a killed entity\n");
> > +		INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work);
> > +		schedule_work(&sched_job->work);
> > +		return;
> > +	}
> > +
> >  	trace_drm_sched_job(sched_job, entity);
> >  	atomic_inc(entity->rq->sched->score);
> >  	WRITE_ONCE(entity->last_user, current->group_leader);
> > @@ -589,12 +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
> >  
> >  		/* Add the entity to the run queue */
> >  		spin_lock(&entity->lock);
> > -		if (entity->stopped) {
> > -			spin_unlock(&entity->lock);
> > -
> > -			DRM_ERROR("Trying to push to a killed entity\n");
> > -			return;
> > -		}
> >  
> >  		rq = entity->rq;
> >  		sched = rq->sched;
> > diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
> > index bfea608a7106..c15b17d9ffe3 100644
> > --- a/drivers/gpu/drm/scheduler/sched_main.c
> > +++ b/drivers/gpu/drm/scheduler/sched_main.c
> > @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job,
> >  		return -ENOENT;
> >  	}
> >  
> > +	if (unlikely(entity->stopped)) {
> > +		pr_err("*ERROR* %s: entity is stopped!\n", __func__);
> > +		return -EINVAL;
> > +	}
> 
> Same here, racy.
> 
> 
> Regards,
> Philipp
> 
> > +
> >  	if (unlikely(!credits)) {
> >  		pr_err("*ERROR* %s: credits cannot be 0!\n", __func__);
> >  		return -EINVAL;
> 

Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

[Christian König]

On 11.08.25 09:20, Liu01 Tong wrote:
> During process kill, drm_sched_entity_flush() will kill the vm
> entities. The following job submissions of this process will fail

Well when the process is killed how can it still make job submissions?

Regards,
Christian.

>, and
> the resources of these jobs have not been released, nor have the fences
> been signalled, causing tasks to hang.
> 
> Fix by not doing job init when the entity is stopped. And when the job
> is already submitted, free the job resource if the entity is stopped.
> 
> Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com>
> Signed-off-by: Lin.Cao <lincao12@amd.com>
> ---
>  drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------
>  drivers/gpu/drm/scheduler/sched_main.c   |  5 +++++
>  2 files changed, 12 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c
> index ac678de7fe5e..1e744b2eb2db 100644
> --- a/drivers/gpu/drm/scheduler/sched_entity.c
> +++ b/drivers/gpu/drm/scheduler/sched_entity.c
> @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
>  	bool first;
>  	ktime_t submit_ts;
>  
> +	if (entity->stopped) {
> +		DRM_ERROR("Trying to push job to a killed entity\n");
> +		INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work);
> +		schedule_work(&sched_job->work);
> +		return;
> +	}
> +
>  	trace_drm_sched_job(sched_job, entity);
>  	atomic_inc(entity->rq->sched->score);
>  	WRITE_ONCE(entity->last_user, current->group_leader);
> @@ -589,12 +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
>  
>  		/* Add the entity to the run queue */
>  		spin_lock(&entity->lock);
> -		if (entity->stopped) {
> -			spin_unlock(&entity->lock);
> -
> -			DRM_ERROR("Trying to push to a killed entity\n");
> -			return;
> -		}
>  
>  		rq = entity->rq;
>  		sched = rq->sched;
> diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
> index bfea608a7106..c15b17d9ffe3 100644
> --- a/drivers/gpu/drm/scheduler/sched_main.c
> +++ b/drivers/gpu/drm/scheduler/sched_main.c
> @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job,
>  		return -ENOENT;
>  	}
>  
> +	if (unlikely(entity->stopped)) {
> +		pr_err("*ERROR* %s: entity is stopped!\n", __func__);
> +		return -EINVAL;
> +	}
> +
>  	if (unlikely(!credits)) {
>  		pr_err("*ERROR* %s: credits cannot be 0!\n", __func__);
>  		return -EINVAL;

RE: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

[Liu01, Tong (Esther)]

[AMD Official Use Only - AMD Internal Distribution Only]

Hi Christian,

The real issue is a race condition during process exit after patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1f02f2044bda1db1fd995bc35961ab075fa7b5a2. This patch changed amdgpu_vm_wait_idle to use drm_sched_entity_flush instead of dma_resv_wait_timeout. Here is what happens:

do_exit
    |
    exit_files(tsk) ... amdgpu_flush ... amdgpu_vm_wait_idle ... drm_sched_entity_flush (kills entity)
    ...
    exit_task_work(tsk) ...amdgpu_gem_object_close  ...  amdgpu_vm_clear_freed (tries to submit to killed entity)

The entity gets killed in amdgpu_vm_wait_idle(), but amdgpu_vm_clear_freed() called by exit_task_work() still tries to submit jobs.

Kind regards,
Esther

-----Original Message-----
From: Koenig, Christian <Christian.Koenig@amd.com>
Sent: Monday, August 11, 2025 4:25 PM
To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; dri-devel@lists.freedesktop.org
Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin <lin.cao@amd.com>; cao, lin <lin.cao@amd.com>
Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

On 11.08.25 09:20, Liu01 Tong wrote:
> During process kill, drm_sched_entity_flush() will kill the vm
> entities. The following job submissions of this process will fail

Well when the process is killed how can it still make job submissions?

Regards,
Christian.

>, and
> the resources of these jobs have not been released, nor have the
>fences  been signalled, causing tasks to hang.
>
> Fix by not doing job init when the entity is stopped. And when the job
> is already submitted, free the job resource if the entity is stopped.
>
> Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com>
> Signed-off-by: Lin.Cao <lincao12@amd.com>
> ---
>  drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------
>  drivers/gpu/drm/scheduler/sched_main.c   |  5 +++++
>  2 files changed, 12 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c
> b/drivers/gpu/drm/scheduler/sched_entity.c
> index ac678de7fe5e..1e744b2eb2db 100644
> --- a/drivers/gpu/drm/scheduler/sched_entity.c
> +++ b/drivers/gpu/drm/scheduler/sched_entity.c
> @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
>       bool first;
>       ktime_t submit_ts;
>
> +     if (entity->stopped) {
> +             DRM_ERROR("Trying to push job to a killed entity\n");
> +             INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work);
> +             schedule_work(&sched_job->work);
> +             return;
> +     }
> +
>       trace_drm_sched_job(sched_job, entity);
>       atomic_inc(entity->rq->sched->score);
>       WRITE_ONCE(entity->last_user, current->group_leader); @@ -589,12
> +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job
> *sched_job)
>
>               /* Add the entity to the run queue */
>               spin_lock(&entity->lock);
> -             if (entity->stopped) {
> -                     spin_unlock(&entity->lock);
> -
> -                     DRM_ERROR("Trying to push to a killed entity\n");
> -                     return;
> -             }
>
>               rq = entity->rq;
>               sched = rq->sched;
> diff --git a/drivers/gpu/drm/scheduler/sched_main.c
> b/drivers/gpu/drm/scheduler/sched_main.c
> index bfea608a7106..c15b17d9ffe3 100644
> --- a/drivers/gpu/drm/scheduler/sched_main.c
> +++ b/drivers/gpu/drm/scheduler/sched_main.c
> @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job,
>               return -ENOENT;
>       }
>
> +     if (unlikely(entity->stopped)) {
> +             pr_err("*ERROR* %s: entity is stopped!\n", __func__);
> +             return -EINVAL;
> +     }
> +
>       if (unlikely(!credits)) {
>               pr_err("*ERROR* %s: credits cannot be 0!\n", __func__);
>               return -EINVAL;

Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

[Christian König]

Hi Esther,

but that is harmless and potentially only gives a warning in the system log.

You could adjust amdgpu_vm_ready() if necessary.

Regards,
Christian.

On 11.08.25 11:05, Liu01, Tong (Esther) wrote:
> [AMD Official Use Only - AMD Internal Distribution Only]
> 
> Hi Christian,
> 
> The real issue is a race condition during process exit after patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1f02f2044bda1db1fd995bc35961ab075fa7b5a2. This patch changed amdgpu_vm_wait_idle to use drm_sched_entity_flush instead of dma_resv_wait_timeout. Here is what happens:
> 
> do_exit
>     |
>     exit_files(tsk) ... amdgpu_flush ... amdgpu_vm_wait_idle ... drm_sched_entity_flush (kills entity)
>     ...
>     exit_task_work(tsk) ...amdgpu_gem_object_close  ...  amdgpu_vm_clear_freed (tries to submit to killed entity)
> 
> The entity gets killed in amdgpu_vm_wait_idle(), but amdgpu_vm_clear_freed() called by exit_task_work() still tries to submit jobs.
> 
> Kind regards,
> Esther
> 
> -----Original Message-----
> From: Koenig, Christian <Christian.Koenig@amd.com>
> Sent: Monday, August 11, 2025 4:25 PM
> To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; dri-devel@lists.freedesktop.org
> Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin <lin.cao@amd.com>; cao, lin <lin.cao@amd.com>
> Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill
> 
> On 11.08.25 09:20, Liu01 Tong wrote:
>> During process kill, drm_sched_entity_flush() will kill the vm
>> entities. The following job submissions of this process will fail
> 
> Well when the process is killed how can it still make job submissions?
> 
> Regards,
> Christian.
> 
>> , and
>> the resources of these jobs have not been released, nor have the
>> fences  been signalled, causing tasks to hang.
>>
>> Fix by not doing job init when the entity is stopped. And when the job
>> is already submitted, free the job resource if the entity is stopped.
>>
>> Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com>
>> Signed-off-by: Lin.Cao <lincao12@amd.com>
>> ---
>>  drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------
>>  drivers/gpu/drm/scheduler/sched_main.c   |  5 +++++
>>  2 files changed, 12 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c
>> b/drivers/gpu/drm/scheduler/sched_entity.c
>> index ac678de7fe5e..1e744b2eb2db 100644
>> --- a/drivers/gpu/drm/scheduler/sched_entity.c
>> +++ b/drivers/gpu/drm/scheduler/sched_entity.c
>> @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
>>       bool first;
>>       ktime_t submit_ts;
>>
>> +     if (entity->stopped) {
>> +             DRM_ERROR("Trying to push job to a killed entity\n");
>> +             INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work);
>> +             schedule_work(&sched_job->work);
>> +             return;
>> +     }
>> +
>>       trace_drm_sched_job(sched_job, entity);
>>       atomic_inc(entity->rq->sched->score);
>>       WRITE_ONCE(entity->last_user, current->group_leader); @@ -589,12
>> +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job
>> *sched_job)
>>
>>               /* Add the entity to the run queue */
>>               spin_lock(&entity->lock);
>> -             if (entity->stopped) {
>> -                     spin_unlock(&entity->lock);
>> -
>> -                     DRM_ERROR("Trying to push to a killed entity\n");
>> -                     return;
>> -             }
>>
>>               rq = entity->rq;
>>               sched = rq->sched;
>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c
>> b/drivers/gpu/drm/scheduler/sched_main.c
>> index bfea608a7106..c15b17d9ffe3 100644
>> --- a/drivers/gpu/drm/scheduler/sched_main.c
>> +++ b/drivers/gpu/drm/scheduler/sched_main.c
>> @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job,
>>               return -ENOENT;
>>       }
>>
>> +     if (unlikely(entity->stopped)) {
>> +             pr_err("*ERROR* %s: entity is stopped!\n", __func__);
>> +             return -EINVAL;
>> +     }
>> +
>>       if (unlikely(!credits)) {
>>               pr_err("*ERROR* %s: credits cannot be 0!\n", __func__);
>>               return -EINVAL;
> 

RE: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

[Liu01, Tong (Esther)]

[AMD Official Use Only - AMD Internal Distribution Only]

Hi Christian,

If a job is submitted into a stopped entity, in addition to an error log, it will also cause task to hang and timeout, and subsequently generate a call trace since the fence of the submitted job is not signaled. Moreover, the refcnt of amdgpu will not decrease because process killing fails, resulting in the inability to unload amdgpu.

[Tue Aug  5 11:05:20 2025] [drm:amddrm_sched_entity_push_job [amd_sched]] *ERROR* Trying to push to a killed entity
[Tue Aug  5 11:07:43 2025] INFO: task kworker/u17:0:117 blocked for more than 122 seconds.
[Tue Aug  5 11:07:43 2025]       Tainted: G           OE      6.8.0-45-generic #45-Ubuntu
[Tue Aug  5 11:07:43 2025] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Aug  5 11:07:43 2025] task:kworker/u17:0   state:D stack:0     pid:117   tgid:117   ppid:2      flags:0x00004000
[Tue Aug  5 11:07:43 2025] Workqueue: ttm ttm_bo_delayed_delete [amdttm]
[Tue Aug  5 11:07:43 2025] Call Trace:
[Tue Aug  5 11:07:43 2025]  <TASK>
[Tue Aug  5 11:07:43 2025]  __schedule+0x27c/0x6b0
[Tue Aug  5 11:07:43 2025]  schedule+0x33/0x110
[Tue Aug  5 11:07:43 2025]  schedule_timeout+0x157/0x170
[Tue Aug  5 11:07:43 2025]  dma_fence_default_wait+0x1e1/0x220
[Tue Aug  5 11:07:43 2025]  ? __pfx_dma_fence_default_wait_cb+0x10/0x10
[Tue Aug  5 11:07:43 2025]  dma_fence_wait_timeout+0x116/0x140
[Tue Aug  5 11:07:43 2025]  amddma_resv_wait_timeout+0x7f/0xf0 [amdkcl]
[Tue Aug  5 11:07:43 2025]  ttm_bo_delayed_delete+0x2a/0xc0 [amdttm]
[Tue Aug  5 11:07:43 2025]  process_one_work+0x16f/0x350
[Tue Aug  5 11:07:43 2025]  worker_thread+0x306/0x440
[Tue Aug  5 11:07:43 2025]  ? __pfx_worker_thread+0x10/0x10
[Tue Aug  5 11:07:43 2025]  kthread+0xf2/0x120
[Tue Aug  5 11:07:43 2025]  ? __pfx_kthread+0x10/0x10
[Tue Aug  5 11:07:43 2025]  ret_from_fork+0x47/0x70
[Tue Aug  5 11:07:43 2025]  ? __pfx_kthread+0x10/0x10
[Tue Aug  5 11:07:43 2025]  ret_from_fork_asm+0x1b/0x30
[Tue Aug  5 11:07:43 2025]  </TASK>

Checking vm entity stopped or not in amdgpu_vm_ready() can avoid to submit job to stopped entity. But as I understand it there still has risk of memory leaks and resource leaks since amdgpu_vm_clear_freed() is skipped during killing process. In amdgpu_vm_clear_freed() , it will update page table to remove mappings and free the mapping structures. If this clean up is skipped, the page table entries remain in VRAM pointing to freed buffer object and mapping structures are allocated but not freed. Please correct me if I have any misunderstanding.

Kind regards,
Esther

-----Original Message-----
From: Koenig, Christian <Christian.Koenig@amd.com>
Sent: Monday, August 11, 2025 8:17 PM
To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; dri-devel@lists.freedesktop.org
Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin <lin.cao@amd.com>
Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

Hi Esther,

but that is harmless and potentially only gives a warning in the system log.

You could adjust amdgpu_vm_ready() if necessary.

Regards,
Christian.

On 11.08.25 11:05, Liu01, Tong (Esther) wrote:
> [AMD Official Use Only - AMD Internal Distribution Only]
>
> Hi Christian,
>
> The real issue is a race condition during process exit after patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1f02f2044bda1db1fd995bc35961ab075fa7b5a2. This patch changed amdgpu_vm_wait_idle to use drm_sched_entity_flush instead of dma_resv_wait_timeout. Here is what happens:
>
> do_exit
>     |
>     exit_files(tsk) ... amdgpu_flush ... amdgpu_vm_wait_idle ... drm_sched_entity_flush (kills entity)
>     ...
>     exit_task_work(tsk) ...amdgpu_gem_object_close  ...
> amdgpu_vm_clear_freed (tries to submit to killed entity)
>
> The entity gets killed in amdgpu_vm_wait_idle(), but amdgpu_vm_clear_freed() called by exit_task_work() still tries to submit jobs.
>
> Kind regards,
> Esther
>
> -----Original Message-----
> From: Koenig, Christian <Christian.Koenig@amd.com>
> Sent: Monday, August 11, 2025 4:25 PM
> To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>;
> dri-devel@lists.freedesktop.org
> Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba,
> Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin
> <lin.cao@amd.com>; cao, lin <lin.cao@amd.com>
> Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job
> submission during process kill
>
> On 11.08.25 09:20, Liu01 Tong wrote:
>> During process kill, drm_sched_entity_flush() will kill the vm
>> entities. The following job submissions of this process will fail
>
> Well when the process is killed how can it still make job submissions?
>
> Regards,
> Christian.
>
>> , and
>> the resources of these jobs have not been released, nor have the
>> fences  been signalled, causing tasks to hang.
>>
>> Fix by not doing job init when the entity is stopped. And when the
>> job is already submitted, free the job resource if the entity is stopped.
>>
>> Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com>
>> Signed-off-by: Lin.Cao <lincao12@amd.com>
>> ---
>>  drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------
>>  drivers/gpu/drm/scheduler/sched_main.c   |  5 +++++
>>  2 files changed, 12 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c
>> b/drivers/gpu/drm/scheduler/sched_entity.c
>> index ac678de7fe5e..1e744b2eb2db 100644
>> --- a/drivers/gpu/drm/scheduler/sched_entity.c
>> +++ b/drivers/gpu/drm/scheduler/sched_entity.c
>> @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
>>       bool first;
>>       ktime_t submit_ts;
>>
>> +     if (entity->stopped) {
>> +             DRM_ERROR("Trying to push job to a killed entity\n");
>> +             INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work);
>> +             schedule_work(&sched_job->work);
>> +             return;
>> +     }
>> +
>>       trace_drm_sched_job(sched_job, entity);
>>       atomic_inc(entity->rq->sched->score);
>>       WRITE_ONCE(entity->last_user, current->group_leader); @@
>> -589,12
>> +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job
>> *sched_job)
>>
>>               /* Add the entity to the run queue */
>>               spin_lock(&entity->lock);
>> -             if (entity->stopped) {
>> -                     spin_unlock(&entity->lock);
>> -
>> -                     DRM_ERROR("Trying to push to a killed entity\n");
>> -                     return;
>> -             }
>>
>>               rq = entity->rq;
>>               sched = rq->sched;
>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c
>> b/drivers/gpu/drm/scheduler/sched_main.c
>> index bfea608a7106..c15b17d9ffe3 100644
>> --- a/drivers/gpu/drm/scheduler/sched_main.c
>> +++ b/drivers/gpu/drm/scheduler/sched_main.c
>> @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job,
>>               return -ENOENT;
>>       }
>>
>> +     if (unlikely(entity->stopped)) {
>> +             pr_err("*ERROR* %s: entity is stopped!\n", __func__);
>> +             return -EINVAL;
>> +     }
>> +
>>       if (unlikely(!credits)) {
>>               pr_err("*ERROR* %s: credits cannot be 0!\n", __func__);
>>               return -EINVAL;
>

Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

[Christian König]

On 12.08.25 08:37, Liu01, Tong (Esther) wrote:
> [AMD Official Use Only - AMD Internal Distribution Only]
> 
> Hi Christian,
> 
> If a job is submitted into a stopped entity, in addition to an error log, it will also cause task to hang and timeout

Oh that's really ugly and needs to get fixed.

>, and subsequently generate a call trace since the fence of the submitted job is not signaled. Moreover, the refcnt of amdgpu will not decrease because process killing fails, resulting in the inability to unload amdgpu.
> 
> [Tue Aug  5 11:05:20 2025] [drm:amddrm_sched_entity_push_job [amd_sched]] *ERROR* Trying to push to a killed entity
> [Tue Aug  5 11:07:43 2025] INFO: task kworker/u17:0:117 blocked for more than 122 seconds.
> [Tue Aug  5 11:07:43 2025]       Tainted: G           OE      6.8.0-45-generic #45-Ubuntu
> [Tue Aug  5 11:07:43 2025] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Tue Aug  5 11:07:43 2025] task:kworker/u17:0   state:D stack:0     pid:117   tgid:117   ppid:2      flags:0x00004000
> [Tue Aug  5 11:07:43 2025] Workqueue: ttm ttm_bo_delayed_delete [amdttm]
> [Tue Aug  5 11:07:43 2025] Call Trace:
> [Tue Aug  5 11:07:43 2025]  <TASK>
> [Tue Aug  5 11:07:43 2025]  __schedule+0x27c/0x6b0
> [Tue Aug  5 11:07:43 2025]  schedule+0x33/0x110
> [Tue Aug  5 11:07:43 2025]  schedule_timeout+0x157/0x170
> [Tue Aug  5 11:07:43 2025]  dma_fence_default_wait+0x1e1/0x220
> [Tue Aug  5 11:07:43 2025]  ? __pfx_dma_fence_default_wait_cb+0x10/0x10
> [Tue Aug  5 11:07:43 2025]  dma_fence_wait_timeout+0x116/0x140
> [Tue Aug  5 11:07:43 2025]  amddma_resv_wait_timeout+0x7f/0xf0 [amdkcl]
> [Tue Aug  5 11:07:43 2025]  ttm_bo_delayed_delete+0x2a/0xc0 [amdttm]
> [Tue Aug  5 11:07:43 2025]  process_one_work+0x16f/0x350
> [Tue Aug  5 11:07:43 2025]  worker_thread+0x306/0x440
> [Tue Aug  5 11:07:43 2025]  ? __pfx_worker_thread+0x10/0x10
> [Tue Aug  5 11:07:43 2025]  kthread+0xf2/0x120
> [Tue Aug  5 11:07:43 2025]  ? __pfx_kthread+0x10/0x10
> [Tue Aug  5 11:07:43 2025]  ret_from_fork+0x47/0x70
> [Tue Aug  5 11:07:43 2025]  ? __pfx_kthread+0x10/0x10
> [Tue Aug  5 11:07:43 2025]  ret_from_fork_asm+0x1b/0x30
> [Tue Aug  5 11:07:43 2025]  </TASK>
> 
> Checking vm entity stopped or not in amdgpu_vm_ready() can avoid to submit job to stopped entity. But as I understand it there still has risk of memory leaks and resource leaks since amdgpu_vm_clear_freed() is skipped during killing process. In amdgpu_vm_clear_freed() , it will update page table to remove mappings and free the mapping structures. If this clean up is skipped, the page table entries remain in VRAM pointing to freed buffer object and mapping structures are allocated but not freed. Please correct me if I have any misunderstanding.

No your understanding is correct, but that page tables are not cleared is completely harmless.

The application is killed and can't submit anything any more. We should just make sure that we check amdgpu_vm_ready() in the submit path as well.

Regards,
Christian.

> 
> Kind regards,
> Esther
> 
> -----Original Message-----
> From: Koenig, Christian <Christian.Koenig@amd.com>
> Sent: Monday, August 11, 2025 8:17 PM
> To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; dri-devel@lists.freedesktop.org
> Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin <lin.cao@amd.com>
> Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill
> 
> Hi Esther,
> 
> but that is harmless and potentially only gives a warning in the system log.
> 
> You could adjust amdgpu_vm_ready() if necessary.
> 
> Regards,
> Christian.
> 
> On 11.08.25 11:05, Liu01, Tong (Esther) wrote:
>> [AMD Official Use Only - AMD Internal Distribution Only]
>>
>> Hi Christian,
>>
>> The real issue is a race condition during process exit after patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1f02f2044bda1db1fd995bc35961ab075fa7b5a2. This patch changed amdgpu_vm_wait_idle to use drm_sched_entity_flush instead of dma_resv_wait_timeout. Here is what happens:
>>
>> do_exit
>>     |
>>     exit_files(tsk) ... amdgpu_flush ... amdgpu_vm_wait_idle ... drm_sched_entity_flush (kills entity)
>>     ...
>>     exit_task_work(tsk) ...amdgpu_gem_object_close  ...
>> amdgpu_vm_clear_freed (tries to submit to killed entity)
>>
>> The entity gets killed in amdgpu_vm_wait_idle(), but amdgpu_vm_clear_freed() called by exit_task_work() still tries to submit jobs.
>>
>> Kind regards,
>> Esther
>>
>> -----Original Message-----
>> From: Koenig, Christian <Christian.Koenig@amd.com>
>> Sent: Monday, August 11, 2025 4:25 PM
>> To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>;
>> dri-devel@lists.freedesktop.org
>> Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba,
>> Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin
>> <lin.cao@amd.com>; cao, lin <lin.cao@amd.com>
>> Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job
>> submission during process kill
>>
>> On 11.08.25 09:20, Liu01 Tong wrote:
>>> During process kill, drm_sched_entity_flush() will kill the vm
>>> entities. The following job submissions of this process will fail
>>
>> Well when the process is killed how can it still make job submissions?
>>
>> Regards,
>> Christian.
>>
>>> , and
>>> the resources of these jobs have not been released, nor have the
>>> fences  been signalled, causing tasks to hang.
>>>
>>> Fix by not doing job init when the entity is stopped. And when the
>>> job is already submitted, free the job resource if the entity is stopped.
>>>
>>> Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com>
>>> Signed-off-by: Lin.Cao <lincao12@amd.com>
>>> ---
>>>  drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------
>>>  drivers/gpu/drm/scheduler/sched_main.c   |  5 +++++
>>>  2 files changed, 12 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c
>>> b/drivers/gpu/drm/scheduler/sched_entity.c
>>> index ac678de7fe5e..1e744b2eb2db 100644
>>> --- a/drivers/gpu/drm/scheduler/sched_entity.c
>>> +++ b/drivers/gpu/drm/scheduler/sched_entity.c
>>> @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
>>>       bool first;
>>>       ktime_t submit_ts;
>>>
>>> +     if (entity->stopped) {
>>> +             DRM_ERROR("Trying to push job to a killed entity\n");
>>> +             INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work);
>>> +             schedule_work(&sched_job->work);
>>> +             return;
>>> +     }
>>> +
>>>       trace_drm_sched_job(sched_job, entity);
>>>       atomic_inc(entity->rq->sched->score);
>>>       WRITE_ONCE(entity->last_user, current->group_leader); @@
>>> -589,12
>>> +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job
>>> *sched_job)
>>>
>>>               /* Add the entity to the run queue */
>>>               spin_lock(&entity->lock);
>>> -             if (entity->stopped) {
>>> -                     spin_unlock(&entity->lock);
>>> -
>>> -                     DRM_ERROR("Trying to push to a killed entity\n");
>>> -                     return;
>>> -             }
>>>
>>>               rq = entity->rq;
>>>               sched = rq->sched;
>>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c
>>> b/drivers/gpu/drm/scheduler/sched_main.c
>>> index bfea608a7106..c15b17d9ffe3 100644
>>> --- a/drivers/gpu/drm/scheduler/sched_main.c
>>> +++ b/drivers/gpu/drm/scheduler/sched_main.c
>>> @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job,
>>>               return -ENOENT;
>>>       }
>>>
>>> +     if (unlikely(entity->stopped)) {
>>> +             pr_err("*ERROR* %s: entity is stopped!\n", __func__);
>>> +             return -EINVAL;
>>> +     }
>>> +
>>>       if (unlikely(!credits)) {
>>>               pr_err("*ERROR* %s: credits cannot be 0!\n", __func__);
>>>               return -EINVAL;
>>
> 

[PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

[Liu01 Tong]

During process kill, drm_sched_entity_flush() will kill the vm
entities. The following job submissions of this process will fail, and
the resources of these jobs have not been released, nor have the fences
been signalled, causing tasks to hang and timeout.

Fix by check entity status in amdgpu_vm_ready() and avoid submit jobs to
stopped entity.

Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com>
Signed-off-by: Lin.Cao <lincao12@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 283dd44f04b0..bf42246a3db2 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -654,11 +654,10 @@ int amdgpu_vm_validate(struct amdgpu_device *adev, struct amdgpu_vm *vm,
  * Check if all VM PDs/PTs are ready for updates
  *
  * Returns:
- * True if VM is not evicting.
+ * True if VM is not evicting and all VM entities are not stopped
  */
 bool amdgpu_vm_ready(struct amdgpu_vm *vm)
 {
-	bool empty;
 	bool ret;
 
 	amdgpu_vm_eviction_lock(vm);
@@ -666,10 +665,18 @@ bool amdgpu_vm_ready(struct amdgpu_vm *vm)
 	amdgpu_vm_eviction_unlock(vm);
 
 	spin_lock(&vm->status_lock);
-	empty = list_empty(&vm->evicted);
+	ret &= list_empty(&vm->evicted);
 	spin_unlock(&vm->status_lock);
 
-	return ret && empty;
+	spin_lock(&vm->immediate.lock);
+	ret &= !vm->immediate.stopped;
+	spin_unlock(&vm->immediate.lock);
+
+	spin_lock(&vm->delayed.lock);
+	ret &= !vm->delayed.stopped;
+	spin_unlock(&vm->delayed.lock);
+
+	return ret;
 }
 
 /**
-- 
2.34.1

Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

[Christian König]

On 12.08.25 10:00, Liu01 Tong wrote:
> During process kill, drm_sched_entity_flush() will kill the vm
> entities. The following job submissions of this process will fail, and
> the resources of these jobs have not been released, nor have the fences
> been signalled, causing tasks to hang and timeout.
> 
> Fix by check entity status in amdgpu_vm_ready() and avoid submit jobs to
> stopped entity.

Looks good to me, but to just be on the safe side please add another call to amdgpu_vm_ready() to the function amdgpu_cs_vm_handling().

Right before we start updating the VM, e.g. after the amdgpu_vmid_uses_reserved() check for the gang submission and before the call to amdgpu_vm_clear_freed().

Regards,
Christian.

> 
> Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com>
> Signed-off-by: Lin.Cao <lincao12@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 15 +++++++++++----
>  1 file changed, 11 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index 283dd44f04b0..bf42246a3db2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -654,11 +654,10 @@ int amdgpu_vm_validate(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   * Check if all VM PDs/PTs are ready for updates
>   *
>   * Returns:
> - * True if VM is not evicting.
> + * True if VM is not evicting and all VM entities are not stopped
>   */
>  bool amdgpu_vm_ready(struct amdgpu_vm *vm)
>  {
> -	bool empty;
>  	bool ret;
>  
>  	amdgpu_vm_eviction_lock(vm);
> @@ -666,10 +665,18 @@ bool amdgpu_vm_ready(struct amdgpu_vm *vm)
>  	amdgpu_vm_eviction_unlock(vm);
>  
>  	spin_lock(&vm->status_lock);
> -	empty = list_empty(&vm->evicted);
> +	ret &= list_empty(&vm->evicted);
>  	spin_unlock(&vm->status_lock);
>  
> -	return ret && empty;
> +	spin_lock(&vm->immediate.lock);
> +	ret &= !vm->immediate.stopped;
> +	spin_unlock(&vm->immediate.lock);
> +
> +	spin_lock(&vm->delayed.lock);
> +	ret &= !vm->delayed.stopped;
> +	spin_unlock(&vm->delayed.lock);
> +
> +	return ret;
>  }
>  
>  /**

Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

[Philipp Stanner]

On Tue, 2025-08-12 at 08:58 +0200, Christian König wrote:
> On 12.08.25 08:37, Liu01, Tong (Esther) wrote:
> > [AMD Official Use Only - AMD Internal Distribution Only]
> > 
> > Hi Christian,
> > 
> > If a job is submitted into a stopped entity, in addition to an error log, it will also cause task to hang and timeout
> 
> Oh that's really ugly and needs to get fixed.

And we agree that the proposed fix is to stop the driver from
submitting to killed entities, don't we?

P.

> 
> > , and subsequently generate a call trace since the fence of the submitted job is not signaled. Moreover, the refcnt of amdgpu will not decrease because process killing fails, resulting in the inability to unload amdgpu.
> > 
> > [Tue Aug  5 11:05:20 2025] [drm:amddrm_sched_entity_push_job [amd_sched]] *ERROR* Trying to push to a killed entity
> > [Tue Aug  5 11:07:43 2025] INFO: task kworker/u17:0:117 blocked for more than 122 seconds.
> > [Tue Aug  5 11:07:43 2025]       Tainted: G           OE      6.8.0-45-generic #45-Ubuntu
> > [Tue Aug  5 11:07:43 2025] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [Tue Aug  5 11:07:43 2025] task:kworker/u17:0   state:D stack:0     pid:117   tgid:117   ppid:2      flags:0x00004000
> > [Tue Aug  5 11:07:43 2025] Workqueue: ttm ttm_bo_delayed_delete [amdttm]
> > [Tue Aug  5 11:07:43 2025] Call Trace:
> > [Tue Aug  5 11:07:43 2025]  <TASK>
> > [Tue Aug  5 11:07:43 2025]  __schedule+0x27c/0x6b0
> > [Tue Aug  5 11:07:43 2025]  schedule+0x33/0x110
> > [Tue Aug  5 11:07:43 2025]  schedule_timeout+0x157/0x170
> > [Tue Aug  5 11:07:43 2025]  dma_fence_default_wait+0x1e1/0x220
> > [Tue Aug  5 11:07:43 2025]  ? __pfx_dma_fence_default_wait_cb+0x10/0x10
> > [Tue Aug  5 11:07:43 2025]  dma_fence_wait_timeout+0x116/0x140
> > [Tue Aug  5 11:07:43 2025]  amddma_resv_wait_timeout+0x7f/0xf0 [amdkcl]
> > [Tue Aug  5 11:07:43 2025]  ttm_bo_delayed_delete+0x2a/0xc0 [amdttm]
> > [Tue Aug  5 11:07:43 2025]  process_one_work+0x16f/0x350
> > [Tue Aug  5 11:07:43 2025]  worker_thread+0x306/0x440
> > [Tue Aug  5 11:07:43 2025]  ? __pfx_worker_thread+0x10/0x10
> > [Tue Aug  5 11:07:43 2025]  kthread+0xf2/0x120
> > [Tue Aug  5 11:07:43 2025]  ? __pfx_kthread+0x10/0x10
> > [Tue Aug  5 11:07:43 2025]  ret_from_fork+0x47/0x70
> > [Tue Aug  5 11:07:43 2025]  ? __pfx_kthread+0x10/0x10
> > [Tue Aug  5 11:07:43 2025]  ret_from_fork_asm+0x1b/0x30
> > [Tue Aug  5 11:07:43 2025]  </TASK>
> > 
> > Checking vm entity stopped or not in amdgpu_vm_ready() can avoid to submit job to stopped entity. But as I understand it there still has risk of memory leaks and resource leaks since amdgpu_vm_clear_freed() is skipped during killing process. In amdgpu_vm_clear_freed() , it will update page table to remove mappings and free the mapping structures. If this clean up is skipped, the page table entries remain in VRAM pointing to freed buffer object and mapping structures are allocated but not freed. Please correct me if I have any misunderstanding.
> 
> No your understanding is correct, but that page tables are not cleared is completely harmless.
> 
> The application is killed and can't submit anything any more. We should just make sure that we check amdgpu_vm_ready() in the submit path as well.
> 
> Regards,
> Christian.
> 
> > 
> > Kind regards,
> > Esther
> > 
> > -----Original Message-----
> > From: Koenig, Christian <Christian.Koenig@amd.com>
> > Sent: Monday, August 11, 2025 8:17 PM
> > To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; dri-devel@lists.freedesktop.org
> > Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin <lin.cao@amd.com>
> > Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill
> > 
> > Hi Esther,
> > 
> > but that is harmless and potentially only gives a warning in the system log.
> > 
> > You could adjust amdgpu_vm_ready() if necessary.
> > 
> > Regards,
> > Christian.
> > 
> > On 11.08.25 11:05, Liu01, Tong (Esther) wrote:
> > > [AMD Official Use Only - AMD Internal Distribution Only]
> > > 
> > > Hi Christian,
> > > 
> > > The real issue is a race condition during process exit after patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1f02f2044bda1db1fd995bc35961ab075fa7b5a2. This patch changed amdgpu_vm_wait_idle to use drm_sched_entity_flush instead of dma_resv_wait_timeout. Here is what happens:
> > > 
> > > do_exit
> > >     |
> > >     exit_files(tsk) ... amdgpu_flush ... amdgpu_vm_wait_idle ... drm_sched_entity_flush (kills entity)
> > >     ...
> > >     exit_task_work(tsk) ...amdgpu_gem_object_close  ...
> > > amdgpu_vm_clear_freed (tries to submit to killed entity)
> > > 
> > > The entity gets killed in amdgpu_vm_wait_idle(), but amdgpu_vm_clear_freed() called by exit_task_work() still tries to submit jobs.
> > > 
> > > Kind regards,
> > > Esther
> > > 
> > > -----Original Message-----
> > > From: Koenig, Christian <Christian.Koenig@amd.com>
> > > Sent: Monday, August 11, 2025 4:25 PM
> > > To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>;
> > > dri-devel@lists.freedesktop.org
> > > Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba,
> > > Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin
> > > <lin.cao@amd.com>; cao, lin <lin.cao@amd.com>
> > > Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job
> > > submission during process kill
> > > 
> > > On 11.08.25 09:20, Liu01 Tong wrote:
> > > > During process kill, drm_sched_entity_flush() will kill the vm
> > > > entities. The following job submissions of this process will fail
> > > 
> > > Well when the process is killed how can it still make job submissions?
> > > 
> > > Regards,
> > > Christian.
> > > 
> > > > , and
> > > > the resources of these jobs have not been released, nor have the
> > > > fences  been signalled, causing tasks to hang.
> > > > 
> > > > Fix by not doing job init when the entity is stopped. And when the
> > > > job is already submitted, free the job resource if the entity is stopped.
> > > > 
> > > > Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com>
> > > > Signed-off-by: Lin.Cao <lincao12@amd.com>
> > > > ---
> > > >  drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------
> > > >  drivers/gpu/drm/scheduler/sched_main.c   |  5 +++++
> > > >  2 files changed, 12 insertions(+), 6 deletions(-)
> > > > 
> > > > diff --git a/drivers/gpu/drm/scheduler/sched_entity.c
> > > > b/drivers/gpu/drm/scheduler/sched_entity.c
> > > > index ac678de7fe5e..1e744b2eb2db 100644
> > > > --- a/drivers/gpu/drm/scheduler/sched_entity.c
> > > > +++ b/drivers/gpu/drm/scheduler/sched_entity.c
> > > > @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
> > > >       bool first;
> > > >       ktime_t submit_ts;
> > > > 
> > > > +     if (entity->stopped) {
> > > > +             DRM_ERROR("Trying to push job to a killed entity\n");
> > > > +             INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work);
> > > > +             schedule_work(&sched_job->work);
> > > > +             return;
> > > > +     }
> > > > +
> > > >       trace_drm_sched_job(sched_job, entity);
> > > >       atomic_inc(entity->rq->sched->score);
> > > >       WRITE_ONCE(entity->last_user, current->group_leader); @@
> > > > -589,12
> > > > +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job
> > > > *sched_job)
> > > > 
> > > >               /* Add the entity to the run queue */
> > > >               spin_lock(&entity->lock);
> > > > -             if (entity->stopped) {
> > > > -                     spin_unlock(&entity->lock);
> > > > -
> > > > -                     DRM_ERROR("Trying to push to a killed entity\n");
> > > > -                     return;
> > > > -             }
> > > > 
> > > >               rq = entity->rq;
> > > >               sched = rq->sched;
> > > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c
> > > > b/drivers/gpu/drm/scheduler/sched_main.c
> > > > index bfea608a7106..c15b17d9ffe3 100644
> > > > --- a/drivers/gpu/drm/scheduler/sched_main.c
> > > > +++ b/drivers/gpu/drm/scheduler/sched_main.c
> > > > @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job,
> > > >               return -ENOENT;
> > > >       }
> > > > 
> > > > +     if (unlikely(entity->stopped)) {
> > > > +             pr_err("*ERROR* %s: entity is stopped!\n", __func__);
> > > > +             return -EINVAL;
> > > > +     }
> > > > +
> > > >       if (unlikely(!credits)) {
> > > >               pr_err("*ERROR* %s: credits cannot be 0!\n", __func__);
> > > >               return -EINVAL;
> > > 
> > 
> 

Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill

[Christian König]

On 12.08.25 14:52, Philipp Stanner wrote:
> On Tue, 2025-08-12 at 08:58 +0200, Christian König wrote:
>> On 12.08.25 08:37, Liu01, Tong (Esther) wrote:
>>> [AMD Official Use Only - AMD Internal Distribution Only]
>>>
>>> Hi Christian,
>>>
>>> If a job is submitted into a stopped entity, in addition to an error log, it will also cause task to hang and timeout
>>
>> Oh that's really ugly and needs to get fixed.
> 
> And we agree that the proposed fix is to stop the driver from
> submitting to killed entities, don't we?

Yes, absolutely.

It's basically the cleanup procedure which tries to update page tables of a killed process and is missing a check to not do that.

Regards,
Christian.

> 
> P.
> 
>>
>>> , and subsequently generate a call trace since the fence of the submitted job is not signaled. Moreover, the refcnt of amdgpu will not decrease because process killing fails, resulting in the inability to unload amdgpu.
>>>
>>> [Tue Aug  5 11:05:20 2025] [drm:amddrm_sched_entity_push_job [amd_sched]] *ERROR* Trying to push to a killed entity
>>> [Tue Aug  5 11:07:43 2025] INFO: task kworker/u17:0:117 blocked for more than 122 seconds.
>>> [Tue Aug  5 11:07:43 2025]       Tainted: G           OE      6.8.0-45-generic #45-Ubuntu
>>> [Tue Aug  5 11:07:43 2025] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> [Tue Aug  5 11:07:43 2025] task:kworker/u17:0   state:D stack:0     pid:117   tgid:117   ppid:2      flags:0x00004000
>>> [Tue Aug  5 11:07:43 2025] Workqueue: ttm ttm_bo_delayed_delete [amdttm]
>>> [Tue Aug  5 11:07:43 2025] Call Trace:
>>> [Tue Aug  5 11:07:43 2025]  <TASK>
>>> [Tue Aug  5 11:07:43 2025]  __schedule+0x27c/0x6b0
>>> [Tue Aug  5 11:07:43 2025]  schedule+0x33/0x110
>>> [Tue Aug  5 11:07:43 2025]  schedule_timeout+0x157/0x170
>>> [Tue Aug  5 11:07:43 2025]  dma_fence_default_wait+0x1e1/0x220
>>> [Tue Aug  5 11:07:43 2025]  ? __pfx_dma_fence_default_wait_cb+0x10/0x10
>>> [Tue Aug  5 11:07:43 2025]  dma_fence_wait_timeout+0x116/0x140
>>> [Tue Aug  5 11:07:43 2025]  amddma_resv_wait_timeout+0x7f/0xf0 [amdkcl]
>>> [Tue Aug  5 11:07:43 2025]  ttm_bo_delayed_delete+0x2a/0xc0 [amdttm]
>>> [Tue Aug  5 11:07:43 2025]  process_one_work+0x16f/0x350
>>> [Tue Aug  5 11:07:43 2025]  worker_thread+0x306/0x440
>>> [Tue Aug  5 11:07:43 2025]  ? __pfx_worker_thread+0x10/0x10
>>> [Tue Aug  5 11:07:43 2025]  kthread+0xf2/0x120
>>> [Tue Aug  5 11:07:43 2025]  ? __pfx_kthread+0x10/0x10
>>> [Tue Aug  5 11:07:43 2025]  ret_from_fork+0x47/0x70
>>> [Tue Aug  5 11:07:43 2025]  ? __pfx_kthread+0x10/0x10
>>> [Tue Aug  5 11:07:43 2025]  ret_from_fork_asm+0x1b/0x30
>>> [Tue Aug  5 11:07:43 2025]  </TASK>
>>>
>>> Checking vm entity stopped or not in amdgpu_vm_ready() can avoid to submit job to stopped entity. But as I understand it there still has risk of memory leaks and resource leaks since amdgpu_vm_clear_freed() is skipped during killing process. In amdgpu_vm_clear_freed() , it will update page table to remove mappings and free the mapping structures. If this clean up is skipped, the page table entries remain in VRAM pointing to freed buffer object and mapping structures are allocated but not freed. Please correct me if I have any misunderstanding.
>>
>> No your understanding is correct, but that page tables are not cleared is completely harmless.
>>
>> The application is killed and can't submit anything any more. We should just make sure that we check amdgpu_vm_ready() in the submit path as well.
>>
>> Regards,
>> Christian.
>>
>>>
>>> Kind regards,
>>> Esther
>>>
>>> -----Original Message-----
>>> From: Koenig, Christian <Christian.Koenig@amd.com>
>>> Sent: Monday, August 11, 2025 8:17 PM
>>> To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; dri-devel@lists.freedesktop.org
>>> Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin <lin.cao@amd.com>
>>> Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill
>>>
>>> Hi Esther,
>>>
>>> but that is harmless and potentially only gives a warning in the system log.
>>>
>>> You could adjust amdgpu_vm_ready() if necessary.
>>>
>>> Regards,
>>> Christian.
>>>
>>> On 11.08.25 11:05, Liu01, Tong (Esther) wrote:
>>>> [AMD Official Use Only - AMD Internal Distribution Only]
>>>>
>>>> Hi Christian,
>>>>
>>>> The real issue is a race condition during process exit after patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1f02f2044bda1db1fd995bc35961ab075fa7b5a2. This patch changed amdgpu_vm_wait_idle to use drm_sched_entity_flush instead of dma_resv_wait_timeout. Here is what happens:
>>>>
>>>> do_exit
>>>>     |
>>>>     exit_files(tsk) ... amdgpu_flush ... amdgpu_vm_wait_idle ... drm_sched_entity_flush (kills entity)
>>>>     ...
>>>>     exit_task_work(tsk) ...amdgpu_gem_object_close  ...
>>>> amdgpu_vm_clear_freed (tries to submit to killed entity)
>>>>
>>>> The entity gets killed in amdgpu_vm_wait_idle(), but amdgpu_vm_clear_freed() called by exit_task_work() still tries to submit jobs.
>>>>
>>>> Kind regards,
>>>> Esther
>>>>
>>>> -----Original Message-----
>>>> From: Koenig, Christian <Christian.Koenig@amd.com>
>>>> Sent: Monday, August 11, 2025 4:25 PM
>>>> To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>;
>>>> dri-devel@lists.freedesktop.org
>>>> Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba,
>>>> Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin
>>>> <lin.cao@amd.com>; cao, lin <lin.cao@amd.com>
>>>> Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job
>>>> submission during process kill
>>>>
>>>> On 11.08.25 09:20, Liu01 Tong wrote:
>>>>> During process kill, drm_sched_entity_flush() will kill the vm
>>>>> entities. The following job submissions of this process will fail
>>>>
>>>> Well when the process is killed how can it still make job submissions?
>>>>
>>>> Regards,
>>>> Christian.
>>>>
>>>>> , and
>>>>> the resources of these jobs have not been released, nor have the
>>>>> fences  been signalled, causing tasks to hang.
>>>>>
>>>>> Fix by not doing job init when the entity is stopped. And when the
>>>>> job is already submitted, free the job resource if the entity is stopped.
>>>>>
>>>>> Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com>
>>>>> Signed-off-by: Lin.Cao <lincao12@amd.com>
>>>>> ---
>>>>>  drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------
>>>>>  drivers/gpu/drm/scheduler/sched_main.c   |  5 +++++
>>>>>  2 files changed, 12 insertions(+), 6 deletions(-)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c
>>>>> b/drivers/gpu/drm/scheduler/sched_entity.c
>>>>> index ac678de7fe5e..1e744b2eb2db 100644
>>>>> --- a/drivers/gpu/drm/scheduler/sched_entity.c
>>>>> +++ b/drivers/gpu/drm/scheduler/sched_entity.c
>>>>> @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
>>>>>       bool first;
>>>>>       ktime_t submit_ts;
>>>>>
>>>>> +     if (entity->stopped) {
>>>>> +             DRM_ERROR("Trying to push job to a killed entity\n");
>>>>> +             INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work);
>>>>> +             schedule_work(&sched_job->work);
>>>>> +             return;
>>>>> +     }
>>>>> +
>>>>>       trace_drm_sched_job(sched_job, entity);
>>>>>       atomic_inc(entity->rq->sched->score);
>>>>>       WRITE_ONCE(entity->last_user, current->group_leader); @@
>>>>> -589,12
>>>>> +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job
>>>>> *sched_job)
>>>>>
>>>>>               /* Add the entity to the run queue */
>>>>>               spin_lock(&entity->lock);
>>>>> -             if (entity->stopped) {
>>>>> -                     spin_unlock(&entity->lock);
>>>>> -
>>>>> -                     DRM_ERROR("Trying to push to a killed entity\n");
>>>>> -                     return;
>>>>> -             }
>>>>>
>>>>>               rq = entity->rq;
>>>>>               sched = rq->sched;
>>>>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c
>>>>> b/drivers/gpu/drm/scheduler/sched_main.c
>>>>> index bfea608a7106..c15b17d9ffe3 100644
>>>>> --- a/drivers/gpu/drm/scheduler/sched_main.c
>>>>> +++ b/drivers/gpu/drm/scheduler/sched_main.c
>>>>> @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job,
>>>>>               return -ENOENT;
>>>>>       }
>>>>>
>>>>> +     if (unlikely(entity->stopped)) {
>>>>> +             pr_err("*ERROR* %s: entity is stopped!\n", __func__);
>>>>> +             return -EINVAL;
>>>>> +     }
>>>>> +
>>>>>       if (unlikely(!credits)) {
>>>>>               pr_err("*ERROR* %s: credits cannot be 0!\n", __func__);
>>>>>               return -EINVAL;
>>>>
>>>
>>
>