* [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill @ 2025-08-11 7:20 Liu01 Tong 2025-08-11 8:18 ` Philipp Stanner 2025-08-11 8:25 ` Christian König 0 siblings, 2 replies; 12+ messages in thread From: Liu01 Tong @ 2025-08-11 7:20 UTC (permalink / raw) To: dri-devel Cc: phasta, dakr, matthew.brost, Christian König, gang.ba, matthew.schwartz, lin.cao, Liu01 Tong, Lin . Cao During process kill, drm_sched_entity_flush() will kill the vm entities. The following job submissions of this process will fail, and the resources of these jobs have not been released, nor have the fences been signalled, causing tasks to hang. Fix by not doing job init when the entity is stopped. And when the job is already submitted, free the job resource if the entity is stopped. Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com> Signed-off-by: Lin.Cao <lincao12@amd.com> --- drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------ drivers/gpu/drm/scheduler/sched_main.c | 5 +++++ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index ac678de7fe5e..1e744b2eb2db 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) bool first; ktime_t submit_ts; + if (entity->stopped) { + DRM_ERROR("Trying to push job to a killed entity\n"); + INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work); + schedule_work(&sched_job->work); + return; + } + trace_drm_sched_job(sched_job, entity); atomic_inc(entity->rq->sched->score); WRITE_ONCE(entity->last_user, current->group_leader); @@ -589,12 +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) /* Add the entity to the run queue */ spin_lock(&entity->lock); - if (entity->stopped) { - spin_unlock(&entity->lock); - - DRM_ERROR("Trying to push to a killed entity\n"); - return; - } rq = entity->rq; sched = rq->sched; diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index bfea608a7106..c15b17d9ffe3 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job, return -ENOENT; } + if (unlikely(entity->stopped)) { + pr_err("*ERROR* %s: entity is stopped!\n", __func__); + return -EINVAL; + } + if (unlikely(!credits)) { pr_err("*ERROR* %s: credits cannot be 0!\n", __func__); return -EINVAL; -- 2.34.1 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill 2025-08-11 7:20 [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill Liu01 Tong @ 2025-08-11 8:18 ` Philipp Stanner 2025-08-11 8:20 ` Philipp Stanner 2025-08-11 8:25 ` Christian König 1 sibling, 1 reply; 12+ messages in thread From: Philipp Stanner @ 2025-08-11 8:18 UTC (permalink / raw) To: Liu01 Tong, dri-devel Cc: phasta, dakr, matthew.brost, Christian König, gang.ba, matthew.schwartz, lin.cao, Lin . Cao, dakr, Matthew Brost Hi, title: this patch changes nothing in amdgpu. Thus, the prefix must be drm/sched: Fix […] Furthermore, please use scripts/get_maintainer. A few relevant folks are missing. +Cc Danilo, Matthew On Mon, 2025-08-11 at 15:20 +0800, Liu01 Tong wrote: > During process kill, drm_sched_entity_flush() will kill the vm > entities. > What is a "vm entity"? This seems to be driver-specific language. > The following job submissions of this process will fail, and > the resources of these jobs have not been released, nor have the fences > been signalled, causing tasks to hang. > > Fix by not doing job init when the entity is stopped. And when the job > is already submitted, free the job resource if the entity is stopped. I'm not sure I can fully follow. Can you give more details on why that bug doesn't always occur? In general: Why is this something that needs to be fixed in the scheduler? amdgpu knows when it killed an entity. Why can't it stop submitting thereafter? > > Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com> > Signed-off-by: Lin.Cao <lincao12@amd.com> Two authors? AFAIK should contain a Co-authored-by tag then. > --- > drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------ > drivers/gpu/drm/scheduler/sched_main.c | 5 +++++ > 2 files changed, 12 insertions(+), 6 deletions(-) > > diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c > index ac678de7fe5e..1e744b2eb2db 100644 > --- a/drivers/gpu/drm/scheduler/sched_entity.c > +++ b/drivers/gpu/drm/scheduler/sched_entity.c > @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) > bool first; > ktime_t submit_ts; > > + if (entity->stopped) { This screams "race!" because you're checking for the entity being stopped now without the lock, as was done before. That's definitely a no-go because that has caused big trouble and the past and is still causing trouble right now at other places where the lock was not taken: https://lore.kernel.org/dri-devel/20250731093008.45267-2-phasta@kernel.org/ > + DRM_ERROR("Trying to push job to a killed entity\n"); > + INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work); > + schedule_work(&sched_job->work); > + return; > + } > + > trace_drm_sched_job(sched_job, entity); > atomic_inc(entity->rq->sched->score); > WRITE_ONCE(entity->last_user, current->group_leader); > @@ -589,12 +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) > > /* Add the entity to the run queue */ > spin_lock(&entity->lock); > - if (entity->stopped) { > - spin_unlock(&entity->lock); > - > - DRM_ERROR("Trying to push to a killed entity\n"); > - return; > - } > > rq = entity->rq; > sched = rq->sched; > diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c > index bfea608a7106..c15b17d9ffe3 100644 > --- a/drivers/gpu/drm/scheduler/sched_main.c > +++ b/drivers/gpu/drm/scheduler/sched_main.c > @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job, > return -ENOENT; > } > > + if (unlikely(entity->stopped)) { > + pr_err("*ERROR* %s: entity is stopped!\n", __func__); > + return -EINVAL; > + } Same here, racy. Regards, Philipp > + > if (unlikely(!credits)) { > pr_err("*ERROR* %s: credits cannot be 0!\n", __func__); > return -EINVAL; ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill 2025-08-11 8:18 ` Philipp Stanner @ 2025-08-11 8:20 ` Philipp Stanner 0 siblings, 0 replies; 12+ messages in thread From: Philipp Stanner @ 2025-08-11 8:20 UTC (permalink / raw) To: phasta, Liu01 Tong, dri-devel Cc: dakr, matthew.brost, Christian König, gang.ba, matthew.schwartz, lin.cao, Lin . Cao On Mon, 2025-08-11 at 10:18 +0200, Philipp Stanner wrote: > Hi, > > title: this patch changes nothing in amdgpu. > > Thus, the prefix must be drm/sched: Fix […] > > > Furthermore, please use scripts/get_maintainer. A few relevant folks > are missing. +Cc Danilo, Matthew Oh, never mind, just overlooked them because the names weren't spelled out. My bad. P. > > > On Mon, 2025-08-11 at 15:20 +0800, Liu01 Tong wrote: > > During process kill, drm_sched_entity_flush() will kill the vm > > entities. > > > > What is a "vm entity"? This seems to be driver-specific language. > > > > The following job submissions of this process will fail, and > > the resources of these jobs have not been released, nor have the fences > > been signalled, causing tasks to hang. > > > > Fix by not doing job init when the entity is stopped. And when the job > > is already submitted, free the job resource if the entity is stopped. > > I'm not sure I can fully follow. Can you give more details on why that > bug doesn't always occur? > > In general: Why is this something that needs to be fixed in the > scheduler? amdgpu knows when it killed an entity. Why can't it stop > submitting thereafter? > > > > > Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com> > > Signed-off-by: Lin.Cao <lincao12@amd.com> > > Two authors? AFAIK should contain a Co-authored-by tag then. > > > --- > > drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------ > > drivers/gpu/drm/scheduler/sched_main.c | 5 +++++ > > 2 files changed, 12 insertions(+), 6 deletions(-) > > > > diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c > > index ac678de7fe5e..1e744b2eb2db 100644 > > --- a/drivers/gpu/drm/scheduler/sched_entity.c > > +++ b/drivers/gpu/drm/scheduler/sched_entity.c > > @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) > > bool first; > > ktime_t submit_ts; > > > > + if (entity->stopped) { > > This screams "race!" because you're checking for the entity being > stopped now without the lock, as was done before. > > That's definitely a no-go because that has caused big trouble and the > past and is still causing trouble right now at other places where the > lock was not taken: > > https://lore.kernel.org/dri-devel/20250731093008.45267-2-phasta@kernel.org/ > > > > + DRM_ERROR("Trying to push job to a killed entity\n"); > > + INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work); > > + schedule_work(&sched_job->work); > > + return; > > + } > > + > > trace_drm_sched_job(sched_job, entity); > > atomic_inc(entity->rq->sched->score); > > WRITE_ONCE(entity->last_user, current->group_leader); > > @@ -589,12 +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) > > > > /* Add the entity to the run queue */ > > spin_lock(&entity->lock); > > - if (entity->stopped) { > > - spin_unlock(&entity->lock); > > - > > - DRM_ERROR("Trying to push to a killed entity\n"); > > - return; > > - } > > > > rq = entity->rq; > > sched = rq->sched; > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c > > index bfea608a7106..c15b17d9ffe3 100644 > > --- a/drivers/gpu/drm/scheduler/sched_main.c > > +++ b/drivers/gpu/drm/scheduler/sched_main.c > > @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job, > > return -ENOENT; > > } > > > > + if (unlikely(entity->stopped)) { > > + pr_err("*ERROR* %s: entity is stopped!\n", __func__); > > + return -EINVAL; > > + } > > Same here, racy. > > > Regards, > Philipp > > > + > > if (unlikely(!credits)) { > > pr_err("*ERROR* %s: credits cannot be 0!\n", __func__); > > return -EINVAL; > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill 2025-08-11 7:20 [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill Liu01 Tong 2025-08-11 8:18 ` Philipp Stanner @ 2025-08-11 8:25 ` Christian König 2025-08-11 9:05 ` Liu01, Tong (Esther) 1 sibling, 1 reply; 12+ messages in thread From: Christian König @ 2025-08-11 8:25 UTC (permalink / raw) To: Liu01 Tong, dri-devel Cc: phasta, dakr, matthew.brost, gang.ba, matthew.schwartz, lin.cao, Lin . Cao On 11.08.25 09:20, Liu01 Tong wrote: > During process kill, drm_sched_entity_flush() will kill the vm > entities. The following job submissions of this process will fail Well when the process is killed how can it still make job submissions? Regards, Christian. >, and > the resources of these jobs have not been released, nor have the fences > been signalled, causing tasks to hang. > > Fix by not doing job init when the entity is stopped. And when the job > is already submitted, free the job resource if the entity is stopped. > > Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com> > Signed-off-by: Lin.Cao <lincao12@amd.com> > --- > drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------ > drivers/gpu/drm/scheduler/sched_main.c | 5 +++++ > 2 files changed, 12 insertions(+), 6 deletions(-) > > diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c > index ac678de7fe5e..1e744b2eb2db 100644 > --- a/drivers/gpu/drm/scheduler/sched_entity.c > +++ b/drivers/gpu/drm/scheduler/sched_entity.c > @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) > bool first; > ktime_t submit_ts; > > + if (entity->stopped) { > + DRM_ERROR("Trying to push job to a killed entity\n"); > + INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work); > + schedule_work(&sched_job->work); > + return; > + } > + > trace_drm_sched_job(sched_job, entity); > atomic_inc(entity->rq->sched->score); > WRITE_ONCE(entity->last_user, current->group_leader); > @@ -589,12 +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) > > /* Add the entity to the run queue */ > spin_lock(&entity->lock); > - if (entity->stopped) { > - spin_unlock(&entity->lock); > - > - DRM_ERROR("Trying to push to a killed entity\n"); > - return; > - } > > rq = entity->rq; > sched = rq->sched; > diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c > index bfea608a7106..c15b17d9ffe3 100644 > --- a/drivers/gpu/drm/scheduler/sched_main.c > +++ b/drivers/gpu/drm/scheduler/sched_main.c > @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job, > return -ENOENT; > } > > + if (unlikely(entity->stopped)) { > + pr_err("*ERROR* %s: entity is stopped!\n", __func__); > + return -EINVAL; > + } > + > if (unlikely(!credits)) { > pr_err("*ERROR* %s: credits cannot be 0!\n", __func__); > return -EINVAL; ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill 2025-08-11 8:25 ` Christian König @ 2025-08-11 9:05 ` Liu01, Tong (Esther) 2025-08-11 12:16 ` Christian König 0 siblings, 1 reply; 12+ messages in thread From: Liu01, Tong (Esther) @ 2025-08-11 9:05 UTC (permalink / raw) To: Koenig, Christian, dri-devel@lists.freedesktop.org Cc: phasta@kernel.org, dakr@kernel.org, matthew.brost@intel.com, Ba, Gang, matthew.schwartz@linux.dev, cao, lin, cao, lin [AMD Official Use Only - AMD Internal Distribution Only] Hi Christian, The real issue is a race condition during process exit after patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1f02f2044bda1db1fd995bc35961ab075fa7b5a2. This patch changed amdgpu_vm_wait_idle to use drm_sched_entity_flush instead of dma_resv_wait_timeout. Here is what happens: do_exit | exit_files(tsk) ... amdgpu_flush ... amdgpu_vm_wait_idle ... drm_sched_entity_flush (kills entity) ... exit_task_work(tsk) ...amdgpu_gem_object_close ... amdgpu_vm_clear_freed (tries to submit to killed entity) The entity gets killed in amdgpu_vm_wait_idle(), but amdgpu_vm_clear_freed() called by exit_task_work() still tries to submit jobs. Kind regards, Esther -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com> Sent: Monday, August 11, 2025 4:25 PM To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; dri-devel@lists.freedesktop.org Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin <lin.cao@amd.com>; cao, lin <lin.cao@amd.com> Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill On 11.08.25 09:20, Liu01 Tong wrote: > During process kill, drm_sched_entity_flush() will kill the vm > entities. The following job submissions of this process will fail Well when the process is killed how can it still make job submissions? Regards, Christian. >, and > the resources of these jobs have not been released, nor have the >fences been signalled, causing tasks to hang. > > Fix by not doing job init when the entity is stopped. And when the job > is already submitted, free the job resource if the entity is stopped. > > Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com> > Signed-off-by: Lin.Cao <lincao12@amd.com> > --- > drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------ > drivers/gpu/drm/scheduler/sched_main.c | 5 +++++ > 2 files changed, 12 insertions(+), 6 deletions(-) > > diff --git a/drivers/gpu/drm/scheduler/sched_entity.c > b/drivers/gpu/drm/scheduler/sched_entity.c > index ac678de7fe5e..1e744b2eb2db 100644 > --- a/drivers/gpu/drm/scheduler/sched_entity.c > +++ b/drivers/gpu/drm/scheduler/sched_entity.c > @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) > bool first; > ktime_t submit_ts; > > + if (entity->stopped) { > + DRM_ERROR("Trying to push job to a killed entity\n"); > + INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work); > + schedule_work(&sched_job->work); > + return; > + } > + > trace_drm_sched_job(sched_job, entity); > atomic_inc(entity->rq->sched->score); > WRITE_ONCE(entity->last_user, current->group_leader); @@ -589,12 > +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job > *sched_job) > > /* Add the entity to the run queue */ > spin_lock(&entity->lock); > - if (entity->stopped) { > - spin_unlock(&entity->lock); > - > - DRM_ERROR("Trying to push to a killed entity\n"); > - return; > - } > > rq = entity->rq; > sched = rq->sched; > diff --git a/drivers/gpu/drm/scheduler/sched_main.c > b/drivers/gpu/drm/scheduler/sched_main.c > index bfea608a7106..c15b17d9ffe3 100644 > --- a/drivers/gpu/drm/scheduler/sched_main.c > +++ b/drivers/gpu/drm/scheduler/sched_main.c > @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job, > return -ENOENT; > } > > + if (unlikely(entity->stopped)) { > + pr_err("*ERROR* %s: entity is stopped!\n", __func__); > + return -EINVAL; > + } > + > if (unlikely(!credits)) { > pr_err("*ERROR* %s: credits cannot be 0!\n", __func__); > return -EINVAL; ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill 2025-08-11 9:05 ` Liu01, Tong (Esther) @ 2025-08-11 12:16 ` Christian König 2025-08-12 6:37 ` Liu01, Tong (Esther) 0 siblings, 1 reply; 12+ messages in thread From: Christian König @ 2025-08-11 12:16 UTC (permalink / raw) To: Liu01, Tong (Esther), dri-devel@lists.freedesktop.org Cc: phasta@kernel.org, dakr@kernel.org, matthew.brost@intel.com, Ba, Gang, matthew.schwartz@linux.dev, cao, lin Hi Esther, but that is harmless and potentially only gives a warning in the system log. You could adjust amdgpu_vm_ready() if necessary. Regards, Christian. On 11.08.25 11:05, Liu01, Tong (Esther) wrote: > [AMD Official Use Only - AMD Internal Distribution Only] > > Hi Christian, > > The real issue is a race condition during process exit after patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1f02f2044bda1db1fd995bc35961ab075fa7b5a2. This patch changed amdgpu_vm_wait_idle to use drm_sched_entity_flush instead of dma_resv_wait_timeout. Here is what happens: > > do_exit > | > exit_files(tsk) ... amdgpu_flush ... amdgpu_vm_wait_idle ... drm_sched_entity_flush (kills entity) > ... > exit_task_work(tsk) ...amdgpu_gem_object_close ... amdgpu_vm_clear_freed (tries to submit to killed entity) > > The entity gets killed in amdgpu_vm_wait_idle(), but amdgpu_vm_clear_freed() called by exit_task_work() still tries to submit jobs. > > Kind regards, > Esther > > -----Original Message----- > From: Koenig, Christian <Christian.Koenig@amd.com> > Sent: Monday, August 11, 2025 4:25 PM > To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; dri-devel@lists.freedesktop.org > Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin <lin.cao@amd.com>; cao, lin <lin.cao@amd.com> > Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill > > On 11.08.25 09:20, Liu01 Tong wrote: >> During process kill, drm_sched_entity_flush() will kill the vm >> entities. The following job submissions of this process will fail > > Well when the process is killed how can it still make job submissions? > > Regards, > Christian. > >> , and >> the resources of these jobs have not been released, nor have the >> fences been signalled, causing tasks to hang. >> >> Fix by not doing job init when the entity is stopped. And when the job >> is already submitted, free the job resource if the entity is stopped. >> >> Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com> >> Signed-off-by: Lin.Cao <lincao12@amd.com> >> --- >> drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------ >> drivers/gpu/drm/scheduler/sched_main.c | 5 +++++ >> 2 files changed, 12 insertions(+), 6 deletions(-) >> >> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c >> b/drivers/gpu/drm/scheduler/sched_entity.c >> index ac678de7fe5e..1e744b2eb2db 100644 >> --- a/drivers/gpu/drm/scheduler/sched_entity.c >> +++ b/drivers/gpu/drm/scheduler/sched_entity.c >> @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) >> bool first; >> ktime_t submit_ts; >> >> + if (entity->stopped) { >> + DRM_ERROR("Trying to push job to a killed entity\n"); >> + INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work); >> + schedule_work(&sched_job->work); >> + return; >> + } >> + >> trace_drm_sched_job(sched_job, entity); >> atomic_inc(entity->rq->sched->score); >> WRITE_ONCE(entity->last_user, current->group_leader); @@ -589,12 >> +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job >> *sched_job) >> >> /* Add the entity to the run queue */ >> spin_lock(&entity->lock); >> - if (entity->stopped) { >> - spin_unlock(&entity->lock); >> - >> - DRM_ERROR("Trying to push to a killed entity\n"); >> - return; >> - } >> >> rq = entity->rq; >> sched = rq->sched; >> diff --git a/drivers/gpu/drm/scheduler/sched_main.c >> b/drivers/gpu/drm/scheduler/sched_main.c >> index bfea608a7106..c15b17d9ffe3 100644 >> --- a/drivers/gpu/drm/scheduler/sched_main.c >> +++ b/drivers/gpu/drm/scheduler/sched_main.c >> @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job, >> return -ENOENT; >> } >> >> + if (unlikely(entity->stopped)) { >> + pr_err("*ERROR* %s: entity is stopped!\n", __func__); >> + return -EINVAL; >> + } >> + >> if (unlikely(!credits)) { >> pr_err("*ERROR* %s: credits cannot be 0!\n", __func__); >> return -EINVAL; > ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill 2025-08-11 12:16 ` Christian König @ 2025-08-12 6:37 ` Liu01, Tong (Esther) 2025-08-12 6:58 ` Christian König 0 siblings, 1 reply; 12+ messages in thread From: Liu01, Tong (Esther) @ 2025-08-12 6:37 UTC (permalink / raw) To: Koenig, Christian, dri-devel@lists.freedesktop.org Cc: phasta@kernel.org, dakr@kernel.org, matthew.brost@intel.com, Ba, Gang, matthew.schwartz@linux.dev, cao, lin [AMD Official Use Only - AMD Internal Distribution Only] Hi Christian, If a job is submitted into a stopped entity, in addition to an error log, it will also cause task to hang and timeout, and subsequently generate a call trace since the fence of the submitted job is not signaled. Moreover, the refcnt of amdgpu will not decrease because process killing fails, resulting in the inability to unload amdgpu. [Tue Aug 5 11:05:20 2025] [drm:amddrm_sched_entity_push_job [amd_sched]] *ERROR* Trying to push to a killed entity [Tue Aug 5 11:07:43 2025] INFO: task kworker/u17:0:117 blocked for more than 122 seconds. [Tue Aug 5 11:07:43 2025] Tainted: G OE 6.8.0-45-generic #45-Ubuntu [Tue Aug 5 11:07:43 2025] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Aug 5 11:07:43 2025] task:kworker/u17:0 state:D stack:0 pid:117 tgid:117 ppid:2 flags:0x00004000 [Tue Aug 5 11:07:43 2025] Workqueue: ttm ttm_bo_delayed_delete [amdttm] [Tue Aug 5 11:07:43 2025] Call Trace: [Tue Aug 5 11:07:43 2025] <TASK> [Tue Aug 5 11:07:43 2025] __schedule+0x27c/0x6b0 [Tue Aug 5 11:07:43 2025] schedule+0x33/0x110 [Tue Aug 5 11:07:43 2025] schedule_timeout+0x157/0x170 [Tue Aug 5 11:07:43 2025] dma_fence_default_wait+0x1e1/0x220 [Tue Aug 5 11:07:43 2025] ? __pfx_dma_fence_default_wait_cb+0x10/0x10 [Tue Aug 5 11:07:43 2025] dma_fence_wait_timeout+0x116/0x140 [Tue Aug 5 11:07:43 2025] amddma_resv_wait_timeout+0x7f/0xf0 [amdkcl] [Tue Aug 5 11:07:43 2025] ttm_bo_delayed_delete+0x2a/0xc0 [amdttm] [Tue Aug 5 11:07:43 2025] process_one_work+0x16f/0x350 [Tue Aug 5 11:07:43 2025] worker_thread+0x306/0x440 [Tue Aug 5 11:07:43 2025] ? __pfx_worker_thread+0x10/0x10 [Tue Aug 5 11:07:43 2025] kthread+0xf2/0x120 [Tue Aug 5 11:07:43 2025] ? __pfx_kthread+0x10/0x10 [Tue Aug 5 11:07:43 2025] ret_from_fork+0x47/0x70 [Tue Aug 5 11:07:43 2025] ? __pfx_kthread+0x10/0x10 [Tue Aug 5 11:07:43 2025] ret_from_fork_asm+0x1b/0x30 [Tue Aug 5 11:07:43 2025] </TASK> Checking vm entity stopped or not in amdgpu_vm_ready() can avoid to submit job to stopped entity. But as I understand it there still has risk of memory leaks and resource leaks since amdgpu_vm_clear_freed() is skipped during killing process. In amdgpu_vm_clear_freed() , it will update page table to remove mappings and free the mapping structures. If this clean up is skipped, the page table entries remain in VRAM pointing to freed buffer object and mapping structures are allocated but not freed. Please correct me if I have any misunderstanding. Kind regards, Esther -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com> Sent: Monday, August 11, 2025 8:17 PM To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; dri-devel@lists.freedesktop.org Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin <lin.cao@amd.com> Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill Hi Esther, but that is harmless and potentially only gives a warning in the system log. You could adjust amdgpu_vm_ready() if necessary. Regards, Christian. On 11.08.25 11:05, Liu01, Tong (Esther) wrote: > [AMD Official Use Only - AMD Internal Distribution Only] > > Hi Christian, > > The real issue is a race condition during process exit after patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1f02f2044bda1db1fd995bc35961ab075fa7b5a2. This patch changed amdgpu_vm_wait_idle to use drm_sched_entity_flush instead of dma_resv_wait_timeout. Here is what happens: > > do_exit > | > exit_files(tsk) ... amdgpu_flush ... amdgpu_vm_wait_idle ... drm_sched_entity_flush (kills entity) > ... > exit_task_work(tsk) ...amdgpu_gem_object_close ... > amdgpu_vm_clear_freed (tries to submit to killed entity) > > The entity gets killed in amdgpu_vm_wait_idle(), but amdgpu_vm_clear_freed() called by exit_task_work() still tries to submit jobs. > > Kind regards, > Esther > > -----Original Message----- > From: Koenig, Christian <Christian.Koenig@amd.com> > Sent: Monday, August 11, 2025 4:25 PM > To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; > dri-devel@lists.freedesktop.org > Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, > Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin > <lin.cao@amd.com>; cao, lin <lin.cao@amd.com> > Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job > submission during process kill > > On 11.08.25 09:20, Liu01 Tong wrote: >> During process kill, drm_sched_entity_flush() will kill the vm >> entities. The following job submissions of this process will fail > > Well when the process is killed how can it still make job submissions? > > Regards, > Christian. > >> , and >> the resources of these jobs have not been released, nor have the >> fences been signalled, causing tasks to hang. >> >> Fix by not doing job init when the entity is stopped. And when the >> job is already submitted, free the job resource if the entity is stopped. >> >> Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com> >> Signed-off-by: Lin.Cao <lincao12@amd.com> >> --- >> drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------ >> drivers/gpu/drm/scheduler/sched_main.c | 5 +++++ >> 2 files changed, 12 insertions(+), 6 deletions(-) >> >> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c >> b/drivers/gpu/drm/scheduler/sched_entity.c >> index ac678de7fe5e..1e744b2eb2db 100644 >> --- a/drivers/gpu/drm/scheduler/sched_entity.c >> +++ b/drivers/gpu/drm/scheduler/sched_entity.c >> @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) >> bool first; >> ktime_t submit_ts; >> >> + if (entity->stopped) { >> + DRM_ERROR("Trying to push job to a killed entity\n"); >> + INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work); >> + schedule_work(&sched_job->work); >> + return; >> + } >> + >> trace_drm_sched_job(sched_job, entity); >> atomic_inc(entity->rq->sched->score); >> WRITE_ONCE(entity->last_user, current->group_leader); @@ >> -589,12 >> +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job >> *sched_job) >> >> /* Add the entity to the run queue */ >> spin_lock(&entity->lock); >> - if (entity->stopped) { >> - spin_unlock(&entity->lock); >> - >> - DRM_ERROR("Trying to push to a killed entity\n"); >> - return; >> - } >> >> rq = entity->rq; >> sched = rq->sched; >> diff --git a/drivers/gpu/drm/scheduler/sched_main.c >> b/drivers/gpu/drm/scheduler/sched_main.c >> index bfea608a7106..c15b17d9ffe3 100644 >> --- a/drivers/gpu/drm/scheduler/sched_main.c >> +++ b/drivers/gpu/drm/scheduler/sched_main.c >> @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job, >> return -ENOENT; >> } >> >> + if (unlikely(entity->stopped)) { >> + pr_err("*ERROR* %s: entity is stopped!\n", __func__); >> + return -EINVAL; >> + } >> + >> if (unlikely(!credits)) { >> pr_err("*ERROR* %s: credits cannot be 0!\n", __func__); >> return -EINVAL; > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill 2025-08-12 6:37 ` Liu01, Tong (Esther) @ 2025-08-12 6:58 ` Christian König 2025-08-12 12:52 ` Philipp Stanner 0 siblings, 1 reply; 12+ messages in thread From: Christian König @ 2025-08-12 6:58 UTC (permalink / raw) To: Liu01, Tong (Esther), dri-devel@lists.freedesktop.org Cc: phasta@kernel.org, dakr@kernel.org, matthew.brost@intel.com, Ba, Gang, matthew.schwartz@linux.dev, cao, lin On 12.08.25 08:37, Liu01, Tong (Esther) wrote: > [AMD Official Use Only - AMD Internal Distribution Only] > > Hi Christian, > > If a job is submitted into a stopped entity, in addition to an error log, it will also cause task to hang and timeout Oh that's really ugly and needs to get fixed. >, and subsequently generate a call trace since the fence of the submitted job is not signaled. Moreover, the refcnt of amdgpu will not decrease because process killing fails, resulting in the inability to unload amdgpu. > > [Tue Aug 5 11:05:20 2025] [drm:amddrm_sched_entity_push_job [amd_sched]] *ERROR* Trying to push to a killed entity > [Tue Aug 5 11:07:43 2025] INFO: task kworker/u17:0:117 blocked for more than 122 seconds. > [Tue Aug 5 11:07:43 2025] Tainted: G OE 6.8.0-45-generic #45-Ubuntu > [Tue Aug 5 11:07:43 2025] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [Tue Aug 5 11:07:43 2025] task:kworker/u17:0 state:D stack:0 pid:117 tgid:117 ppid:2 flags:0x00004000 > [Tue Aug 5 11:07:43 2025] Workqueue: ttm ttm_bo_delayed_delete [amdttm] > [Tue Aug 5 11:07:43 2025] Call Trace: > [Tue Aug 5 11:07:43 2025] <TASK> > [Tue Aug 5 11:07:43 2025] __schedule+0x27c/0x6b0 > [Tue Aug 5 11:07:43 2025] schedule+0x33/0x110 > [Tue Aug 5 11:07:43 2025] schedule_timeout+0x157/0x170 > [Tue Aug 5 11:07:43 2025] dma_fence_default_wait+0x1e1/0x220 > [Tue Aug 5 11:07:43 2025] ? __pfx_dma_fence_default_wait_cb+0x10/0x10 > [Tue Aug 5 11:07:43 2025] dma_fence_wait_timeout+0x116/0x140 > [Tue Aug 5 11:07:43 2025] amddma_resv_wait_timeout+0x7f/0xf0 [amdkcl] > [Tue Aug 5 11:07:43 2025] ttm_bo_delayed_delete+0x2a/0xc0 [amdttm] > [Tue Aug 5 11:07:43 2025] process_one_work+0x16f/0x350 > [Tue Aug 5 11:07:43 2025] worker_thread+0x306/0x440 > [Tue Aug 5 11:07:43 2025] ? __pfx_worker_thread+0x10/0x10 > [Tue Aug 5 11:07:43 2025] kthread+0xf2/0x120 > [Tue Aug 5 11:07:43 2025] ? __pfx_kthread+0x10/0x10 > [Tue Aug 5 11:07:43 2025] ret_from_fork+0x47/0x70 > [Tue Aug 5 11:07:43 2025] ? __pfx_kthread+0x10/0x10 > [Tue Aug 5 11:07:43 2025] ret_from_fork_asm+0x1b/0x30 > [Tue Aug 5 11:07:43 2025] </TASK> > > Checking vm entity stopped or not in amdgpu_vm_ready() can avoid to submit job to stopped entity. But as I understand it there still has risk of memory leaks and resource leaks since amdgpu_vm_clear_freed() is skipped during killing process. In amdgpu_vm_clear_freed() , it will update page table to remove mappings and free the mapping structures. If this clean up is skipped, the page table entries remain in VRAM pointing to freed buffer object and mapping structures are allocated but not freed. Please correct me if I have any misunderstanding. No your understanding is correct, but that page tables are not cleared is completely harmless. The application is killed and can't submit anything any more. We should just make sure that we check amdgpu_vm_ready() in the submit path as well. Regards, Christian. > > Kind regards, > Esther > > -----Original Message----- > From: Koenig, Christian <Christian.Koenig@amd.com> > Sent: Monday, August 11, 2025 8:17 PM > To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; dri-devel@lists.freedesktop.org > Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin <lin.cao@amd.com> > Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill > > Hi Esther, > > but that is harmless and potentially only gives a warning in the system log. > > You could adjust amdgpu_vm_ready() if necessary. > > Regards, > Christian. > > On 11.08.25 11:05, Liu01, Tong (Esther) wrote: >> [AMD Official Use Only - AMD Internal Distribution Only] >> >> Hi Christian, >> >> The real issue is a race condition during process exit after patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1f02f2044bda1db1fd995bc35961ab075fa7b5a2. This patch changed amdgpu_vm_wait_idle to use drm_sched_entity_flush instead of dma_resv_wait_timeout. Here is what happens: >> >> do_exit >> | >> exit_files(tsk) ... amdgpu_flush ... amdgpu_vm_wait_idle ... drm_sched_entity_flush (kills entity) >> ... >> exit_task_work(tsk) ...amdgpu_gem_object_close ... >> amdgpu_vm_clear_freed (tries to submit to killed entity) >> >> The entity gets killed in amdgpu_vm_wait_idle(), but amdgpu_vm_clear_freed() called by exit_task_work() still tries to submit jobs. >> >> Kind regards, >> Esther >> >> -----Original Message----- >> From: Koenig, Christian <Christian.Koenig@amd.com> >> Sent: Monday, August 11, 2025 4:25 PM >> To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; >> dri-devel@lists.freedesktop.org >> Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, >> Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin >> <lin.cao@amd.com>; cao, lin <lin.cao@amd.com> >> Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job >> submission during process kill >> >> On 11.08.25 09:20, Liu01 Tong wrote: >>> During process kill, drm_sched_entity_flush() will kill the vm >>> entities. The following job submissions of this process will fail >> >> Well when the process is killed how can it still make job submissions? >> >> Regards, >> Christian. >> >>> , and >>> the resources of these jobs have not been released, nor have the >>> fences been signalled, causing tasks to hang. >>> >>> Fix by not doing job init when the entity is stopped. And when the >>> job is already submitted, free the job resource if the entity is stopped. >>> >>> Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com> >>> Signed-off-by: Lin.Cao <lincao12@amd.com> >>> --- >>> drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------ >>> drivers/gpu/drm/scheduler/sched_main.c | 5 +++++ >>> 2 files changed, 12 insertions(+), 6 deletions(-) >>> >>> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c >>> b/drivers/gpu/drm/scheduler/sched_entity.c >>> index ac678de7fe5e..1e744b2eb2db 100644 >>> --- a/drivers/gpu/drm/scheduler/sched_entity.c >>> +++ b/drivers/gpu/drm/scheduler/sched_entity.c >>> @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) >>> bool first; >>> ktime_t submit_ts; >>> >>> + if (entity->stopped) { >>> + DRM_ERROR("Trying to push job to a killed entity\n"); >>> + INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work); >>> + schedule_work(&sched_job->work); >>> + return; >>> + } >>> + >>> trace_drm_sched_job(sched_job, entity); >>> atomic_inc(entity->rq->sched->score); >>> WRITE_ONCE(entity->last_user, current->group_leader); @@ >>> -589,12 >>> +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job >>> *sched_job) >>> >>> /* Add the entity to the run queue */ >>> spin_lock(&entity->lock); >>> - if (entity->stopped) { >>> - spin_unlock(&entity->lock); >>> - >>> - DRM_ERROR("Trying to push to a killed entity\n"); >>> - return; >>> - } >>> >>> rq = entity->rq; >>> sched = rq->sched; >>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c >>> b/drivers/gpu/drm/scheduler/sched_main.c >>> index bfea608a7106..c15b17d9ffe3 100644 >>> --- a/drivers/gpu/drm/scheduler/sched_main.c >>> +++ b/drivers/gpu/drm/scheduler/sched_main.c >>> @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job, >>> return -ENOENT; >>> } >>> >>> + if (unlikely(entity->stopped)) { >>> + pr_err("*ERROR* %s: entity is stopped!\n", __func__); >>> + return -EINVAL; >>> + } >>> + >>> if (unlikely(!credits)) { >>> pr_err("*ERROR* %s: credits cannot be 0!\n", __func__); >>> return -EINVAL; >> > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill 2025-08-12 6:58 ` Christian König @ 2025-08-12 12:52 ` Philipp Stanner 2025-08-12 13:00 ` Christian König 0 siblings, 1 reply; 12+ messages in thread From: Philipp Stanner @ 2025-08-12 12:52 UTC (permalink / raw) To: Christian König, Liu01, Tong (Esther), dri-devel@lists.freedesktop.org Cc: phasta@kernel.org, dakr@kernel.org, matthew.brost@intel.com, Ba, Gang, matthew.schwartz@linux.dev, cao, lin On Tue, 2025-08-12 at 08:58 +0200, Christian König wrote: > On 12.08.25 08:37, Liu01, Tong (Esther) wrote: > > [AMD Official Use Only - AMD Internal Distribution Only] > > > > Hi Christian, > > > > If a job is submitted into a stopped entity, in addition to an error log, it will also cause task to hang and timeout > > Oh that's really ugly and needs to get fixed. And we agree that the proposed fix is to stop the driver from submitting to killed entities, don't we? P. > > > , and subsequently generate a call trace since the fence of the submitted job is not signaled. Moreover, the refcnt of amdgpu will not decrease because process killing fails, resulting in the inability to unload amdgpu. > > > > [Tue Aug 5 11:05:20 2025] [drm:amddrm_sched_entity_push_job [amd_sched]] *ERROR* Trying to push to a killed entity > > [Tue Aug 5 11:07:43 2025] INFO: task kworker/u17:0:117 blocked for more than 122 seconds. > > [Tue Aug 5 11:07:43 2025] Tainted: G OE 6.8.0-45-generic #45-Ubuntu > > [Tue Aug 5 11:07:43 2025] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > [Tue Aug 5 11:07:43 2025] task:kworker/u17:0 state:D stack:0 pid:117 tgid:117 ppid:2 flags:0x00004000 > > [Tue Aug 5 11:07:43 2025] Workqueue: ttm ttm_bo_delayed_delete [amdttm] > > [Tue Aug 5 11:07:43 2025] Call Trace: > > [Tue Aug 5 11:07:43 2025] <TASK> > > [Tue Aug 5 11:07:43 2025] __schedule+0x27c/0x6b0 > > [Tue Aug 5 11:07:43 2025] schedule+0x33/0x110 > > [Tue Aug 5 11:07:43 2025] schedule_timeout+0x157/0x170 > > [Tue Aug 5 11:07:43 2025] dma_fence_default_wait+0x1e1/0x220 > > [Tue Aug 5 11:07:43 2025] ? __pfx_dma_fence_default_wait_cb+0x10/0x10 > > [Tue Aug 5 11:07:43 2025] dma_fence_wait_timeout+0x116/0x140 > > [Tue Aug 5 11:07:43 2025] amddma_resv_wait_timeout+0x7f/0xf0 [amdkcl] > > [Tue Aug 5 11:07:43 2025] ttm_bo_delayed_delete+0x2a/0xc0 [amdttm] > > [Tue Aug 5 11:07:43 2025] process_one_work+0x16f/0x350 > > [Tue Aug 5 11:07:43 2025] worker_thread+0x306/0x440 > > [Tue Aug 5 11:07:43 2025] ? __pfx_worker_thread+0x10/0x10 > > [Tue Aug 5 11:07:43 2025] kthread+0xf2/0x120 > > [Tue Aug 5 11:07:43 2025] ? __pfx_kthread+0x10/0x10 > > [Tue Aug 5 11:07:43 2025] ret_from_fork+0x47/0x70 > > [Tue Aug 5 11:07:43 2025] ? __pfx_kthread+0x10/0x10 > > [Tue Aug 5 11:07:43 2025] ret_from_fork_asm+0x1b/0x30 > > [Tue Aug 5 11:07:43 2025] </TASK> > > > > Checking vm entity stopped or not in amdgpu_vm_ready() can avoid to submit job to stopped entity. But as I understand it there still has risk of memory leaks and resource leaks since amdgpu_vm_clear_freed() is skipped during killing process. In amdgpu_vm_clear_freed() , it will update page table to remove mappings and free the mapping structures. If this clean up is skipped, the page table entries remain in VRAM pointing to freed buffer object and mapping structures are allocated but not freed. Please correct me if I have any misunderstanding. > > No your understanding is correct, but that page tables are not cleared is completely harmless. > > The application is killed and can't submit anything any more. We should just make sure that we check amdgpu_vm_ready() in the submit path as well. > > Regards, > Christian. > > > > > Kind regards, > > Esther > > > > -----Original Message----- > > From: Koenig, Christian <Christian.Koenig@amd.com> > > Sent: Monday, August 11, 2025 8:17 PM > > To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; dri-devel@lists.freedesktop.org > > Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin <lin.cao@amd.com> > > Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill > > > > Hi Esther, > > > > but that is harmless and potentially only gives a warning in the system log. > > > > You could adjust amdgpu_vm_ready() if necessary. > > > > Regards, > > Christian. > > > > On 11.08.25 11:05, Liu01, Tong (Esther) wrote: > > > [AMD Official Use Only - AMD Internal Distribution Only] > > > > > > Hi Christian, > > > > > > The real issue is a race condition during process exit after patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1f02f2044bda1db1fd995bc35961ab075fa7b5a2. This patch changed amdgpu_vm_wait_idle to use drm_sched_entity_flush instead of dma_resv_wait_timeout. Here is what happens: > > > > > > do_exit > > > | > > > exit_files(tsk) ... amdgpu_flush ... amdgpu_vm_wait_idle ... drm_sched_entity_flush (kills entity) > > > ... > > > exit_task_work(tsk) ...amdgpu_gem_object_close ... > > > amdgpu_vm_clear_freed (tries to submit to killed entity) > > > > > > The entity gets killed in amdgpu_vm_wait_idle(), but amdgpu_vm_clear_freed() called by exit_task_work() still tries to submit jobs. > > > > > > Kind regards, > > > Esther > > > > > > -----Original Message----- > > > From: Koenig, Christian <Christian.Koenig@amd.com> > > > Sent: Monday, August 11, 2025 4:25 PM > > > To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; > > > dri-devel@lists.freedesktop.org > > > Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, > > > Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin > > > <lin.cao@amd.com>; cao, lin <lin.cao@amd.com> > > > Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job > > > submission during process kill > > > > > > On 11.08.25 09:20, Liu01 Tong wrote: > > > > During process kill, drm_sched_entity_flush() will kill the vm > > > > entities. The following job submissions of this process will fail > > > > > > Well when the process is killed how can it still make job submissions? > > > > > > Regards, > > > Christian. > > > > > > > , and > > > > the resources of these jobs have not been released, nor have the > > > > fences been signalled, causing tasks to hang. > > > > > > > > Fix by not doing job init when the entity is stopped. And when the > > > > job is already submitted, free the job resource if the entity is stopped. > > > > > > > > Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com> > > > > Signed-off-by: Lin.Cao <lincao12@amd.com> > > > > --- > > > > drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------ > > > > drivers/gpu/drm/scheduler/sched_main.c | 5 +++++ > > > > 2 files changed, 12 insertions(+), 6 deletions(-) > > > > > > > > diff --git a/drivers/gpu/drm/scheduler/sched_entity.c > > > > b/drivers/gpu/drm/scheduler/sched_entity.c > > > > index ac678de7fe5e..1e744b2eb2db 100644 > > > > --- a/drivers/gpu/drm/scheduler/sched_entity.c > > > > +++ b/drivers/gpu/drm/scheduler/sched_entity.c > > > > @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) > > > > bool first; > > > > ktime_t submit_ts; > > > > > > > > + if (entity->stopped) { > > > > + DRM_ERROR("Trying to push job to a killed entity\n"); > > > > + INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work); > > > > + schedule_work(&sched_job->work); > > > > + return; > > > > + } > > > > + > > > > trace_drm_sched_job(sched_job, entity); > > > > atomic_inc(entity->rq->sched->score); > > > > WRITE_ONCE(entity->last_user, current->group_leader); @@ > > > > -589,12 > > > > +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job > > > > *sched_job) > > > > > > > > /* Add the entity to the run queue */ > > > > spin_lock(&entity->lock); > > > > - if (entity->stopped) { > > > > - spin_unlock(&entity->lock); > > > > - > > > > - DRM_ERROR("Trying to push to a killed entity\n"); > > > > - return; > > > > - } > > > > > > > > rq = entity->rq; > > > > sched = rq->sched; > > > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c > > > > b/drivers/gpu/drm/scheduler/sched_main.c > > > > index bfea608a7106..c15b17d9ffe3 100644 > > > > --- a/drivers/gpu/drm/scheduler/sched_main.c > > > > +++ b/drivers/gpu/drm/scheduler/sched_main.c > > > > @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job, > > > > return -ENOENT; > > > > } > > > > > > > > + if (unlikely(entity->stopped)) { > > > > + pr_err("*ERROR* %s: entity is stopped!\n", __func__); > > > > + return -EINVAL; > > > > + } > > > > + > > > > if (unlikely(!credits)) { > > > > pr_err("*ERROR* %s: credits cannot be 0!\n", __func__); > > > > return -EINVAL; > > > > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill 2025-08-12 12:52 ` Philipp Stanner @ 2025-08-12 13:00 ` Christian König 0 siblings, 0 replies; 12+ messages in thread From: Christian König @ 2025-08-12 13:00 UTC (permalink / raw) To: phasta, Liu01, Tong (Esther), dri-devel@lists.freedesktop.org Cc: dakr@kernel.org, matthew.brost@intel.com, Ba, Gang, matthew.schwartz@linux.dev, cao, lin On 12.08.25 14:52, Philipp Stanner wrote: > On Tue, 2025-08-12 at 08:58 +0200, Christian König wrote: >> On 12.08.25 08:37, Liu01, Tong (Esther) wrote: >>> [AMD Official Use Only - AMD Internal Distribution Only] >>> >>> Hi Christian, >>> >>> If a job is submitted into a stopped entity, in addition to an error log, it will also cause task to hang and timeout >> >> Oh that's really ugly and needs to get fixed. > > And we agree that the proposed fix is to stop the driver from > submitting to killed entities, don't we? Yes, absolutely. It's basically the cleanup procedure which tries to update page tables of a killed process and is missing a check to not do that. Regards, Christian. > > P. > >> >>> , and subsequently generate a call trace since the fence of the submitted job is not signaled. Moreover, the refcnt of amdgpu will not decrease because process killing fails, resulting in the inability to unload amdgpu. >>> >>> [Tue Aug 5 11:05:20 2025] [drm:amddrm_sched_entity_push_job [amd_sched]] *ERROR* Trying to push to a killed entity >>> [Tue Aug 5 11:07:43 2025] INFO: task kworker/u17:0:117 blocked for more than 122 seconds. >>> [Tue Aug 5 11:07:43 2025] Tainted: G OE 6.8.0-45-generic #45-Ubuntu >>> [Tue Aug 5 11:07:43 2025] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>> [Tue Aug 5 11:07:43 2025] task:kworker/u17:0 state:D stack:0 pid:117 tgid:117 ppid:2 flags:0x00004000 >>> [Tue Aug 5 11:07:43 2025] Workqueue: ttm ttm_bo_delayed_delete [amdttm] >>> [Tue Aug 5 11:07:43 2025] Call Trace: >>> [Tue Aug 5 11:07:43 2025] <TASK> >>> [Tue Aug 5 11:07:43 2025] __schedule+0x27c/0x6b0 >>> [Tue Aug 5 11:07:43 2025] schedule+0x33/0x110 >>> [Tue Aug 5 11:07:43 2025] schedule_timeout+0x157/0x170 >>> [Tue Aug 5 11:07:43 2025] dma_fence_default_wait+0x1e1/0x220 >>> [Tue Aug 5 11:07:43 2025] ? __pfx_dma_fence_default_wait_cb+0x10/0x10 >>> [Tue Aug 5 11:07:43 2025] dma_fence_wait_timeout+0x116/0x140 >>> [Tue Aug 5 11:07:43 2025] amddma_resv_wait_timeout+0x7f/0xf0 [amdkcl] >>> [Tue Aug 5 11:07:43 2025] ttm_bo_delayed_delete+0x2a/0xc0 [amdttm] >>> [Tue Aug 5 11:07:43 2025] process_one_work+0x16f/0x350 >>> [Tue Aug 5 11:07:43 2025] worker_thread+0x306/0x440 >>> [Tue Aug 5 11:07:43 2025] ? __pfx_worker_thread+0x10/0x10 >>> [Tue Aug 5 11:07:43 2025] kthread+0xf2/0x120 >>> [Tue Aug 5 11:07:43 2025] ? __pfx_kthread+0x10/0x10 >>> [Tue Aug 5 11:07:43 2025] ret_from_fork+0x47/0x70 >>> [Tue Aug 5 11:07:43 2025] ? __pfx_kthread+0x10/0x10 >>> [Tue Aug 5 11:07:43 2025] ret_from_fork_asm+0x1b/0x30 >>> [Tue Aug 5 11:07:43 2025] </TASK> >>> >>> Checking vm entity stopped or not in amdgpu_vm_ready() can avoid to submit job to stopped entity. But as I understand it there still has risk of memory leaks and resource leaks since amdgpu_vm_clear_freed() is skipped during killing process. In amdgpu_vm_clear_freed() , it will update page table to remove mappings and free the mapping structures. If this clean up is skipped, the page table entries remain in VRAM pointing to freed buffer object and mapping structures are allocated but not freed. Please correct me if I have any misunderstanding. >> >> No your understanding is correct, but that page tables are not cleared is completely harmless. >> >> The application is killed and can't submit anything any more. We should just make sure that we check amdgpu_vm_ready() in the submit path as well. >> >> Regards, >> Christian. >> >>> >>> Kind regards, >>> Esther >>> >>> -----Original Message----- >>> From: Koenig, Christian <Christian.Koenig@amd.com> >>> Sent: Monday, August 11, 2025 8:17 PM >>> To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; dri-devel@lists.freedesktop.org >>> Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin <lin.cao@amd.com> >>> Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill >>> >>> Hi Esther, >>> >>> but that is harmless and potentially only gives a warning in the system log. >>> >>> You could adjust amdgpu_vm_ready() if necessary. >>> >>> Regards, >>> Christian. >>> >>> On 11.08.25 11:05, Liu01, Tong (Esther) wrote: >>>> [AMD Official Use Only - AMD Internal Distribution Only] >>>> >>>> Hi Christian, >>>> >>>> The real issue is a race condition during process exit after patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1f02f2044bda1db1fd995bc35961ab075fa7b5a2. This patch changed amdgpu_vm_wait_idle to use drm_sched_entity_flush instead of dma_resv_wait_timeout. Here is what happens: >>>> >>>> do_exit >>>> | >>>> exit_files(tsk) ... amdgpu_flush ... amdgpu_vm_wait_idle ... drm_sched_entity_flush (kills entity) >>>> ... >>>> exit_task_work(tsk) ...amdgpu_gem_object_close ... >>>> amdgpu_vm_clear_freed (tries to submit to killed entity) >>>> >>>> The entity gets killed in amdgpu_vm_wait_idle(), but amdgpu_vm_clear_freed() called by exit_task_work() still tries to submit jobs. >>>> >>>> Kind regards, >>>> Esther >>>> >>>> -----Original Message----- >>>> From: Koenig, Christian <Christian.Koenig@amd.com> >>>> Sent: Monday, August 11, 2025 4:25 PM >>>> To: Liu01, Tong (Esther) <Tong.Liu01@amd.com>; >>>> dri-devel@lists.freedesktop.org >>>> Cc: phasta@kernel.org; dakr@kernel.org; matthew.brost@intel.com; Ba, >>>> Gang <Gang.Ba@amd.com>; matthew.schwartz@linux.dev; cao, lin >>>> <lin.cao@amd.com>; cao, lin <lin.cao@amd.com> >>>> Subject: Re: [PATCH] drm/amdgpu: fix task hang from failed job >>>> submission during process kill >>>> >>>> On 11.08.25 09:20, Liu01 Tong wrote: >>>>> During process kill, drm_sched_entity_flush() will kill the vm >>>>> entities. The following job submissions of this process will fail >>>> >>>> Well when the process is killed how can it still make job submissions? >>>> >>>> Regards, >>>> Christian. >>>> >>>>> , and >>>>> the resources of these jobs have not been released, nor have the >>>>> fences been signalled, causing tasks to hang. >>>>> >>>>> Fix by not doing job init when the entity is stopped. And when the >>>>> job is already submitted, free the job resource if the entity is stopped. >>>>> >>>>> Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com> >>>>> Signed-off-by: Lin.Cao <lincao12@amd.com> >>>>> --- >>>>> drivers/gpu/drm/scheduler/sched_entity.c | 13 +++++++------ >>>>> drivers/gpu/drm/scheduler/sched_main.c | 5 +++++ >>>>> 2 files changed, 12 insertions(+), 6 deletions(-) >>>>> >>>>> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c >>>>> b/drivers/gpu/drm/scheduler/sched_entity.c >>>>> index ac678de7fe5e..1e744b2eb2db 100644 >>>>> --- a/drivers/gpu/drm/scheduler/sched_entity.c >>>>> +++ b/drivers/gpu/drm/scheduler/sched_entity.c >>>>> @@ -570,6 +570,13 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) >>>>> bool first; >>>>> ktime_t submit_ts; >>>>> >>>>> + if (entity->stopped) { >>>>> + DRM_ERROR("Trying to push job to a killed entity\n"); >>>>> + INIT_WORK(&sched_job->work, drm_sched_entity_kill_jobs_work); >>>>> + schedule_work(&sched_job->work); >>>>> + return; >>>>> + } >>>>> + >>>>> trace_drm_sched_job(sched_job, entity); >>>>> atomic_inc(entity->rq->sched->score); >>>>> WRITE_ONCE(entity->last_user, current->group_leader); @@ >>>>> -589,12 >>>>> +596,6 @@ void drm_sched_entity_push_job(struct drm_sched_job >>>>> *sched_job) >>>>> >>>>> /* Add the entity to the run queue */ >>>>> spin_lock(&entity->lock); >>>>> - if (entity->stopped) { >>>>> - spin_unlock(&entity->lock); >>>>> - >>>>> - DRM_ERROR("Trying to push to a killed entity\n"); >>>>> - return; >>>>> - } >>>>> >>>>> rq = entity->rq; >>>>> sched = rq->sched; >>>>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c >>>>> b/drivers/gpu/drm/scheduler/sched_main.c >>>>> index bfea608a7106..c15b17d9ffe3 100644 >>>>> --- a/drivers/gpu/drm/scheduler/sched_main.c >>>>> +++ b/drivers/gpu/drm/scheduler/sched_main.c >>>>> @@ -795,6 +795,11 @@ int drm_sched_job_init(struct drm_sched_job *job, >>>>> return -ENOENT; >>>>> } >>>>> >>>>> + if (unlikely(entity->stopped)) { >>>>> + pr_err("*ERROR* %s: entity is stopped!\n", __func__); >>>>> + return -EINVAL; >>>>> + } >>>>> + >>>>> if (unlikely(!credits)) { >>>>> pr_err("*ERROR* %s: credits cannot be 0!\n", __func__); >>>>> return -EINVAL; >>>> >>> >> > ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill @ 2025-08-12 8:00 Liu01 Tong 2025-08-12 8:07 ` Christian König 0 siblings, 1 reply; 12+ messages in thread From: Liu01 Tong @ 2025-08-12 8:00 UTC (permalink / raw) To: amd-gfx; +Cc: christian.koenig, gang.ba, Liu01 Tong, Lin . Cao During process kill, drm_sched_entity_flush() will kill the vm entities. The following job submissions of this process will fail, and the resources of these jobs have not been released, nor have the fences been signalled, causing tasks to hang and timeout. Fix by check entity status in amdgpu_vm_ready() and avoid submit jobs to stopped entity. Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com> Signed-off-by: Lin.Cao <lincao12@amd.com> --- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c index 283dd44f04b0..bf42246a3db2 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c @@ -654,11 +654,10 @@ int amdgpu_vm_validate(struct amdgpu_device *adev, struct amdgpu_vm *vm, * Check if all VM PDs/PTs are ready for updates * * Returns: - * True if VM is not evicting. + * True if VM is not evicting and all VM entities are not stopped */ bool amdgpu_vm_ready(struct amdgpu_vm *vm) { - bool empty; bool ret; amdgpu_vm_eviction_lock(vm); @@ -666,10 +665,18 @@ bool amdgpu_vm_ready(struct amdgpu_vm *vm) amdgpu_vm_eviction_unlock(vm); spin_lock(&vm->status_lock); - empty = list_empty(&vm->evicted); + ret &= list_empty(&vm->evicted); spin_unlock(&vm->status_lock); - return ret && empty; + spin_lock(&vm->immediate.lock); + ret &= !vm->immediate.stopped; + spin_unlock(&vm->immediate.lock); + + spin_lock(&vm->delayed.lock); + ret &= !vm->delayed.stopped; + spin_unlock(&vm->delayed.lock); + + return ret; } /** -- 2.34.1 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill 2025-08-12 8:00 Liu01 Tong @ 2025-08-12 8:07 ` Christian König 0 siblings, 0 replies; 12+ messages in thread From: Christian König @ 2025-08-12 8:07 UTC (permalink / raw) To: Liu01 Tong, amd-gfx; +Cc: gang.ba, Lin . Cao On 12.08.25 10:00, Liu01 Tong wrote: > During process kill, drm_sched_entity_flush() will kill the vm > entities. The following job submissions of this process will fail, and > the resources of these jobs have not been released, nor have the fences > been signalled, causing tasks to hang and timeout. > > Fix by check entity status in amdgpu_vm_ready() and avoid submit jobs to > stopped entity. Looks good to me, but to just be on the safe side please add another call to amdgpu_vm_ready() to the function amdgpu_cs_vm_handling(). Right before we start updating the VM, e.g. after the amdgpu_vmid_uses_reserved() check for the gang submission and before the call to amdgpu_vm_clear_freed(). Regards, Christian. > > Signed-off-by: Liu01 Tong <Tong.Liu01@amd.com> > Signed-off-by: Lin.Cao <lincao12@amd.com> > --- > drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 15 +++++++++++---- > 1 file changed, 11 insertions(+), 4 deletions(-) > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c > index 283dd44f04b0..bf42246a3db2 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c > @@ -654,11 +654,10 @@ int amdgpu_vm_validate(struct amdgpu_device *adev, struct amdgpu_vm *vm, > * Check if all VM PDs/PTs are ready for updates > * > * Returns: > - * True if VM is not evicting. > + * True if VM is not evicting and all VM entities are not stopped > */ > bool amdgpu_vm_ready(struct amdgpu_vm *vm) > { > - bool empty; > bool ret; > > amdgpu_vm_eviction_lock(vm); > @@ -666,10 +665,18 @@ bool amdgpu_vm_ready(struct amdgpu_vm *vm) > amdgpu_vm_eviction_unlock(vm); > > spin_lock(&vm->status_lock); > - empty = list_empty(&vm->evicted); > + ret &= list_empty(&vm->evicted); > spin_unlock(&vm->status_lock); > > - return ret && empty; > + spin_lock(&vm->immediate.lock); > + ret &= !vm->immediate.stopped; > + spin_unlock(&vm->immediate.lock); > + > + spin_lock(&vm->delayed.lock); > + ret &= !vm->delayed.stopped; > + spin_unlock(&vm->delayed.lock); > + > + return ret; > } > > /** ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2025-08-12 13:00 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-08-11 7:20 [PATCH] drm/amdgpu: fix task hang from failed job submission during process kill Liu01 Tong 2025-08-11 8:18 ` Philipp Stanner 2025-08-11 8:20 ` Philipp Stanner 2025-08-11 8:25 ` Christian König 2025-08-11 9:05 ` Liu01, Tong (Esther) 2025-08-11 12:16 ` Christian König 2025-08-12 6:37 ` Liu01, Tong (Esther) 2025-08-12 6:58 ` Christian König 2025-08-12 12:52 ` Philipp Stanner 2025-08-12 13:00 ` Christian König -- strict thread matches above, loose matches on Subject: below -- 2025-08-12 8:00 Liu01 Tong 2025-08-12 8:07 ` Christian König
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.