From mboxrd@z Thu Jan 1 00:00:00 1970 From: bugzilla-daemon@freedesktop.org Subject: [Bug 109692] deadlock occurs during GPU reset Date: Thu, 21 Feb 2019 17:23:00 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0351976424==" Return-path: Received: from culpepper.freedesktop.org (culpepper.freedesktop.org [IPv6:2610:10:20:722:a800:ff:fe98:4b55]) by gabe.freedesktop.org (Postfix) with ESMTP id AD2F08919C for ; Thu, 21 Feb 2019 17:23:00 +0000 (UTC) In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: dri-devel@lists.freedesktop.org List-Id: dri-devel@lists.freedesktop.org --===============0351976424== Content-Type: multipart/alternative; boundary="15507697800.dcAeaCa.8678" Content-Transfer-Encoding: 7bit --15507697800.dcAeaCa.8678 Date: Thu, 21 Feb 2019 17:23:00 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://bugs.freedesktop.org/ Auto-Submitted: auto-generated https://bugs.freedesktop.org/show_bug.cgi?id=3D109692 Andrey Grodzovsky changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |christian.koenig@amd.com --- Comment #9 from Andrey Grodzovsky --- Adding Christian to consult on solution to the problem: The deadlock happens because we lock fence->lock from dma_fence_signal foll= owed by sched->job_list_lock from drm_sched_process_job on one hand while on the other do reverse order locking in drm_sched_stop, locking first sched->job_list_lock and then fence->lock when trying to remove the cb from dma_fence_remove_callback. I see 2 possible solutions -=20 1) Change ring_mirror_list to lock-less list (e.g. RCU list), problem is I don't think there is a readily available implementation of doubly linked lockless list, I didn't do to much search though...=20 2) Go back to removing a job from ring_mirror_list in drm_sched_job_finish= - as I see it for this to work we need to add a wait_queue (signal) to drm_sched_job which will signal AFTER job is removed from ring_mirror_list = in drm_sched_job_finish, then in drm_sched_stop when iterating over ring_mirror_list you build a new list of all jobs in progress (for them dma_fence_remove_callback returns false) and then you wait on all theirs wait_queues - then you are sure they removed themselves from ring_mirror_li= st and you can proceed.=20 Specific to AMDGPU - To insure the jobs are not being freed in drm_sched_job_finish while we process them in drm_sched_stop it's not enoug= h to wait to do cancel_delayed_work_sync(&sched->work_tdr) for this scheduler b= ut also for all the other schedulers in the device and even in the hive for XG= MI use case. --=20 You are receiving this mail because: You are the assignee for the bug.= --15507697800.dcAeaCa.8678 Date: Thu, 21 Feb 2019 17:23:00 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://bugs.freedesktop.org/ Auto-Submitted: auto-generated Andrey Grodzovsky changed bug 10969= 2
What Removed Added
CC   christian.koenig@amd.com

Commen= t # 9 on bug 10969= 2 from Andrey Grodzovsky
Adding Christian to consult on solution to the problem:

The deadlock happens because we lock fence->lock from dma_fence_signal f=
ollowed
by sched->job_list_lock from drm_sched_process_job on one hand while on =
the
other do reverse order locking in drm_sched_stop, locking first
sched->job_list_lock and then fence->lock when trying to remove the c=
b from
dma_fence_remove_callback.

I see 2 possible solutions -=20

1) Change ring_mirror_list to lock-less list (e.g. RCU list), problem is I
don't think there is a readily available implementation of doubly linked
lockless list, I didn't do to much search though...=20

2) Go back to removing a job from  ring_mirror_list in drm_sched_job_finish=
 -
as I see it for this to work we need to add a wait_queue (signal) to
drm_sched_job which will signal AFTER job is removed from ring_mirror_list =
in
drm_sched_job_finish, then in drm_sched_stop when iterating over
ring_mirror_list you build a new list of all jobs in progress (for them
dma_fence_remove_callback returns false) and then you wait on all theirs
wait_queues - then you are sure they removed themselves from ring_mirror_li=
st
and you can proceed.=20
Specific to AMDGPU  - To insure the jobs are not being freed in
drm_sched_job_finish while we process them in drm_sched_stop it's not enoug=
h to
wait to do  cancel_delayed_work_sync(&sched->work_tdr) for this sche=
duler but
also for all the other schedulers in the device and even in the hive for XG=
MI
use case.


You are receiving this mail because:
  • You are the assignee for the bug.
= --15507697800.dcAeaCa.8678-- --===============0351976424== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KZHJpLWRldmVs IG1haWxpbmcgbGlzdApkcmktZGV2ZWxAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlz dHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vZHJpLWRldmVs --===============0351976424==--