* Re: [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill()
2022-11-23 0:13 [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill() Dmitry Osipenko
@ 2022-11-23 6:40 ` Christian König
2022-11-23 13:22 ` Dmitry Osipenko
` (2 subsequent siblings)
3 siblings, 0 replies; 7+ messages in thread
From: Christian König @ 2022-11-23 6:40 UTC (permalink / raw)
To: Dmitry Osipenko, Luben Tuikov, David Airlie, Daniel Vetter
Cc: dri-devel, linux-kernel
Am 23.11.22 um 01:13 schrieb Dmitry Osipenko:
> The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy()
> while userspace process is exiting or being killed. First time it's invoked
> when sched entity is flushed and second time when entity is released. This
> causes a lockup within wait_for_completion(entity_idle) due to how completion
> API works.
>
> Calling wait_for_completion() more times than complete() was invoked is a
> error condition that causes lockup because completion internally uses
> counter for complete/wait calls. The complete_all() must be used instead
> in such cases.
>
> This patch fixes lockup of Panfrost driver that is reproducible by killing
> any application in a middle of 3d drawing operation.
>
> Fixes: 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")
> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Oh, good point. Reviewed-by: Christian König <christian.koenig@amd.com>
> ---
> drivers/gpu/drm/scheduler/sched_entity.c | 2 +-
> drivers/gpu/drm/scheduler/sched_main.c | 4 ++--
> 2 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c
> index fe09e5be79bd..15d04a0ec623 100644
> --- a/drivers/gpu/drm/scheduler/sched_entity.c
> +++ b/drivers/gpu/drm/scheduler/sched_entity.c
> @@ -81,7 +81,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
> init_completion(&entity->entity_idle);
>
> /* We start in an idle state. */
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
>
> spin_lock_init(&entity->rq_lock);
> spsc_queue_init(&entity->job_queue);
> diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
> index 6ce04c2e90c0..857ec20be9e8 100644
> --- a/drivers/gpu/drm/scheduler/sched_main.c
> +++ b/drivers/gpu/drm/scheduler/sched_main.c
> @@ -1026,7 +1026,7 @@ static int drm_sched_main(void *param)
> sched_job = drm_sched_entity_pop_job(entity);
>
> if (!sched_job) {
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
> continue;
> }
>
> @@ -1037,7 +1037,7 @@ static int drm_sched_main(void *param)
>
> trace_drm_run_job(sched_job, entity);
> fence = sched->ops->run_job(sched_job);
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
> drm_sched_fence_scheduled(s_fence);
>
> if (!IS_ERR_OR_NULL(fence)) {
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill()
2022-11-23 0:13 [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill() Dmitry Osipenko
2022-11-23 6:40 ` Christian König
@ 2022-11-23 13:22 ` Dmitry Osipenko
2022-12-27 19:28 ` Guilherme G. Piccoli
2023-01-02 15:01 ` Dmitry Osipenko
3 siblings, 0 replies; 7+ messages in thread
From: Dmitry Osipenko @ 2022-11-23 13:22 UTC (permalink / raw)
To: Luben Tuikov, Christian König, David Airlie, Daniel Vetter
Cc: dri-devel, linux-kernel
On 11/23/22 03:13, Dmitry Osipenko wrote:
> The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy()
> while userspace process is exiting or being killed. First time it's invoked
> when sched entity is flushed and second time when entity is released. This
> causes a lockup within wait_for_completion(entity_idle) due to how completion
> API works.
>
> Calling wait_for_completion() more times than complete() was invoked is a
> error condition that causes lockup because completion internally uses
> counter for complete/wait calls. The complete_all() must be used instead
> in such cases.
>
> This patch fixes lockup of Panfrost driver that is reproducible by killing
> any application in a middle of 3d drawing operation.
>
> Fixes: 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")
> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
> ---
> drivers/gpu/drm/scheduler/sched_entity.c | 2 +-
> drivers/gpu/drm/scheduler/sched_main.c | 4 ++--
> 2 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c
> index fe09e5be79bd..15d04a0ec623 100644
> --- a/drivers/gpu/drm/scheduler/sched_entity.c
> +++ b/drivers/gpu/drm/scheduler/sched_entity.c
> @@ -81,7 +81,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
> init_completion(&entity->entity_idle);
>
> /* We start in an idle state. */
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
>
> spin_lock_init(&entity->rq_lock);
> spsc_queue_init(&entity->job_queue);
> diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
> index 6ce04c2e90c0..857ec20be9e8 100644
> --- a/drivers/gpu/drm/scheduler/sched_main.c
> +++ b/drivers/gpu/drm/scheduler/sched_main.c
> @@ -1026,7 +1026,7 @@ static int drm_sched_main(void *param)
> sched_job = drm_sched_entity_pop_job(entity);
>
> if (!sched_job) {
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
> continue;
> }
>
> @@ -1037,7 +1037,7 @@ static int drm_sched_main(void *param)
>
> trace_drm_run_job(sched_job, entity);
> fence = sched->ops->run_job(sched_job);
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
> drm_sched_fence_scheduled(s_fence);
>
> if (!IS_ERR_OR_NULL(fence)) {
Applied to drm-misc-next
--
Best regards,
Dmitry
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill()
2022-11-23 0:13 [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill() Dmitry Osipenko
2022-11-23 6:40 ` Christian König
2022-11-23 13:22 ` Dmitry Osipenko
@ 2022-12-27 19:28 ` Guilherme G. Piccoli
2022-12-29 10:15 ` Dmitry Osipenko
2023-01-02 15:01 ` Dmitry Osipenko
3 siblings, 1 reply; 7+ messages in thread
From: Guilherme G. Piccoli @ 2022-12-27 19:28 UTC (permalink / raw)
To: dmitry.osipenko, Christian Koenig
Cc: airlied, Daniel Vetter, Maling list - DRI developers,
linux-kernel, luben.tuikov, Melissa Wen, Guilherme G. Piccoli,
gpiccoli
Hi Dmitry / Christian, thanks for the fix!
(And thanks Melissa for pointing that, saving me lots of time in
research heh)
Is this fix planned to be released on 6.2-rc cycle? I've just tested it
on Steam Deck, and it resolved a lockup observed (since v6.2-rc1) -
exactly the same thing mentioned in the commit message.
FWIW:
Tested-By: Guilherme G. Piccoli <gpiccoli@igalia.com> # Steam Deck
Cheers,
Guilherme
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill()
2022-12-27 19:28 ` Guilherme G. Piccoli
@ 2022-12-29 10:15 ` Dmitry Osipenko
2022-12-29 18:52 ` Guilherme G. Piccoli
0 siblings, 1 reply; 7+ messages in thread
From: Dmitry Osipenko @ 2022-12-29 10:15 UTC (permalink / raw)
To: Guilherme G. Piccoli, Christian Koenig
Cc: airlied, Daniel Vetter, Maling list - DRI developers,
linux-kernel, luben.tuikov, Melissa Wen, Guilherme G. Piccoli
Hi,
On 12/27/22 22:28, Guilherme G. Piccoli wrote:
> Hi Dmitry / Christian, thanks for the fix!
>
> (And thanks Melissa for pointing that, saving me lots of time in
> research heh)
>
> Is this fix planned to be released on 6.2-rc cycle? I've just tested it
> on Steam Deck, and it resolved a lockup observed (since v6.2-rc1) -
> exactly the same thing mentioned in the commit message.
>
> FWIW:
> Tested-By: Guilherme G. Piccoli <gpiccoli@igalia.com> # Steam Deck
I'll push the patch to misc-fixes as soon as it will be rebased on
6.2-rc. Thanks!
Best regards,
Dmitry
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill()
2022-12-29 10:15 ` Dmitry Osipenko
@ 2022-12-29 18:52 ` Guilherme G. Piccoli
0 siblings, 0 replies; 7+ messages in thread
From: Guilherme G. Piccoli @ 2022-12-29 18:52 UTC (permalink / raw)
To: Dmitry Osipenko
Cc: airlied, Daniel Vetter, Maling list - DRI developers,
linux-kernel, luben.tuikov, Melissa Wen, Guilherme G. Piccoli,
Christian Koenig
On 29/12/2022 07:15, Dmitry Osipenko wrote:
> [...]
> I'll push the patch to misc-fixes as soon as it will be rebased on
> 6.2-rc. Thanks!
>
> Best regards,
> Dmitry
>
Thank you Dmitry, much appreciated!
Cheers,
Guilherme
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill()
2022-11-23 0:13 [PATCH v1] drm/scheduler: Fix lockup in drm_sched_entity_kill() Dmitry Osipenko
` (2 preceding siblings ...)
2022-12-27 19:28 ` Guilherme G. Piccoli
@ 2023-01-02 15:01 ` Dmitry Osipenko
3 siblings, 0 replies; 7+ messages in thread
From: Dmitry Osipenko @ 2023-01-02 15:01 UTC (permalink / raw)
To: Luben Tuikov, Christian König, David Airlie, Daniel Vetter,
Guilherme G. Piccoli
Cc: dri-devel, linux-kernel
On 11/23/22 03:13, Dmitry Osipenko wrote:
> The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy()
> while userspace process is exiting or being killed. First time it's invoked
> when sched entity is flushed and second time when entity is released. This
> causes a lockup within wait_for_completion(entity_idle) due to how completion
> API works.
>
> Calling wait_for_completion() more times than complete() was invoked is a
> error condition that causes lockup because completion internally uses
> counter for complete/wait calls. The complete_all() must be used instead
> in such cases.
>
> This patch fixes lockup of Panfrost driver that is reproducible by killing
> any application in a middle of 3d drawing operation.
>
> Fixes: 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")
> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
> ---
> drivers/gpu/drm/scheduler/sched_entity.c | 2 +-
> drivers/gpu/drm/scheduler/sched_main.c | 4 ++--
> 2 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c
> index fe09e5be79bd..15d04a0ec623 100644
> --- a/drivers/gpu/drm/scheduler/sched_entity.c
> +++ b/drivers/gpu/drm/scheduler/sched_entity.c
> @@ -81,7 +81,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
> init_completion(&entity->entity_idle);
>
> /* We start in an idle state. */
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
>
> spin_lock_init(&entity->rq_lock);
> spsc_queue_init(&entity->job_queue);
> diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
> index 6ce04c2e90c0..857ec20be9e8 100644
> --- a/drivers/gpu/drm/scheduler/sched_main.c
> +++ b/drivers/gpu/drm/scheduler/sched_main.c
> @@ -1026,7 +1026,7 @@ static int drm_sched_main(void *param)
> sched_job = drm_sched_entity_pop_job(entity);
>
> if (!sched_job) {
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
> continue;
> }
>
> @@ -1037,7 +1037,7 @@ static int drm_sched_main(void *param)
>
> trace_drm_run_job(sched_job, entity);
> fence = sched->ops->run_job(sched_job);
> - complete(&entity->entity_idle);
> + complete_all(&entity->entity_idle);
> drm_sched_fence_scheduled(s_fence);
>
> if (!IS_ERR_OR_NULL(fence)) {
Applied to drm-misc-next-fixes
--
Best regards,
Dmitry
^ permalink raw reply [flat|nested] 7+ messages in thread