| Bug ID | 111763 |
|---|---|
| Summary | ring_gfx hangs/freezes on Navi gpus |
| Product | DRI |
| Version | unspecified |
| Hardware | x86-64 (AMD64) |
| OS | Linux (All) |
| Status | NEW |
| Severity | major |
| Priority | not set |
| Component | DRM/AMDgpu |
| Assignee | dri-devel@lists.freedesktop.org |
| Reporter | popovic.marko@protonmail.com |
I'm making this topic as a separate tracking of ring_gfx relat= ed bugs since we should keep https://bugs.freedesktop.org/show_bug.= cgi?id=3D111481 related to sdma0/1 type freezes since those are ones that seem to cause random "O= ut of the blue" hangs on the desktop. There is another type of freeze/hang happening when playing Starcraft II via D9VK. This one doesn't seem to be related to either ngg or dma because I ha= ve them both disabled by AMD_DEBUG=3Dnodma and AMD_DEBUG=3Dnongg and the hangs= occur anyway, on exactly the same place every time. Error logs: sep 17 11:48:24 Marko-PC kernel: [drm:amdgpu_dm_commit_planes.constprop.0 [amdgpu]] *ERROR* Waiting for fences timed out or interrupted! sep 17 11:48:24 Marko-PC kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=3D2361623, emitted seq=3D2361625 sep 17 11:48:24 Marko-PC kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process SC2_x64.exe pid 20236 thread SC2_x64.exe pid 2= 0236 I will try and provide trace files by using renderdoc for described issues. They also happen in native games like Rise of the Tomb Raider and Vulkan et= c. Will provide as much info as possible. Using Kernel 5.3, MESA 19.2 and llvm9.
Not sure if that might help someone else, but I found a workar= ound in my case with DOOM. I was having the same crashes as Marko described with Starcraft = II, I tried the following: - In Steam, I disabled the In Game Steam Overlay - I switched the Graphics API from OpenGL to Vulkan I did not have any crash so far. But I haven't tried to isolate one or the other. Packages: linux 5.3.arch1-1 linux-firmware-agd5f-radeon-navi10 2019.09.13.18.36-1 mesa-git 1:19.3.0_devel.115574.40087ffc5b9-1 vulkan-radeon-git 1:19.3.0_devel.115574.40087ffc5b9-1 libdrm 2.4.99-1 lib32-mesa-git 1:19.3.0_devel.115574.40087ffc5b9-1 lib32-vulkan-radeon-git 1:19.3.0_devel.115574.40087ffc5b9-1 lib32-libdrm 2.4.99-1
Created =
attachment 145465 [details]
output of running sudo umr -R gfx_0.0.0
I am seeing a similar hang in Starcraft II. Unlike Marko, I am= not using d9vk --- instead, I'm using wine-nine. The hang doesn't happen in all games but seems to be particularly frequent in the coop mission "dead of night&q= uot;. Using mesa-git 19.3.0_devel.115092.3f5b541fc8b-1.
I've been getting this too with Minecraft:=20=20 https://bugs.freedesktop.org/show_bug.= cgi?id=3D111669 For my particular case at least, AMD_DEBUG=3Dnodma seems to fix it
(In reply to Doug Ty from comment #5) > I've been getting this too with Minecraft:=20=20 > https://bugs.freedesktop.org/show_bug.= cgi?id=3D111669 >=20 > For my particular case at least, AMD_DEBUG=3Dnodma seems to fix it (In reply to Marko Popovic from com= ment #0) > There is another type of freeze/hang happening w= hen playing Starcraft II via > D9VK. This one doesn't seem to be related to either ngg or dma because= I > have them both disabled by AMD_DEBUG=3Dnodma and AMD_DEBUG=3Dnongg and= the hangs > occur anyway, on exactly the same place every time. You are refering to sdma0 / sdma1 type hang which is tracked here:https://bugs.freedesktop.org/show_bug.= cgi?id=3D111481 For ring_gfx hangs they're quite more reproducible and are not affected by AMD_DEBUG=3Dnodma or AMD_DEBUG=3Dnongg which I already mentioned above in t= he bug description.
(In reply to Marko Popovic from comment #6) > (In reply to Doug Ty from comment #5) > > I've been getting this too with Minecraft:=20=20 > > https://bugs.freedesktop.org/show_bug.= cgi?id=3D111669 > >=20 > > For my particular case at least, AMD_DEBUG=3Dnodma seems to fix it >=20 > You are refering to sdma0 / sdma1 type hang which is tracked > here:https://bugs.freedesktop.org/show_bug.= cgi?id=3D111481 >=20 > For ring_gfx hangs they're quite more reproducible and are not affecte= d by > AMD_DEBUG=3Dnodma or AMD_DEBUG=3Dnongg which I already mentioned above= in the > bug description. Sorry, but this is incorrect. My Minecraft hang is most definitely a ring g= fx hang, *not* sdma. I've posted logs and apitraces in the linked thread if yo= u'd like to check for yourself. I can't explain why nodma isn't working for you, perhaps it doesn't work for game? Have you tried putting it in /etc/environment so it's system-wide? I don't know what to tell you regarding nodma, but my hang is definitely ring= gfx as well.
(In reply to Doug Ty from comment #7) > (In reply to Marko Popovic from comment #6) > > (In reply to Doug Ty from comment #5) > > > I've been getting this too with Minecraft:=20=20 > > > https://bugs.freedesktop.org/show_bug.= cgi?id=3D111669 > > >=20 > > > For my particular case at least, AMD_DEBUG=3Dnodma seems to = fix it > >=20 > > You are refering to sdma0 / sdma1 type hang which is tracked > > here:https://bugs.freedesktop.org/show_bug.= cgi?id=3D111481 > >=20 > > For ring_gfx hangs they're quite more reproducible and are not af= fected by > > AMD_DEBUG=3Dnodma or AMD_DEBUG=3Dnongg which I already mentioned = above in the > > bug description. >=20 > Sorry, but this is incorrect. My Minecraft hang is most definitely a r= ing > gfx hang, *not* sdma. I've posted logs and apitraces in the linked thr= ead if > you'd like to check for yourself. >=20 > I can't explain why nodma isn't working for you, perhaps it doesn't wo= rk for > game? Have you tried putting it in /etc/environment so it's system-wid= e? I > don't know what to tell you regarding nodma, but my hang is definitely= ring > gfx as well. I guess we just have many different types of hangs then... ring_gfx hangs a= re more mysterious than sdma0/1 hangs it seems, since there is no "univer= sal" workaround for them. nodma works for stopping global sdma-type hangs for me, nongg works for stopping the citra-related hang of ring_gfx type, but none = of those 2 variables work for stopping Starcraft II and RoTR ring_gfx-type han= gs for me, so it's really really confusing.
https://cgit.freedesktop.org/m= esa/mesa/commit/?id=3Da2a68d551c1c2a4f13761ffa8f3f6f13fee7a384 This might actually fix the ring_gfx type hangs or even sdma ones at least = for Vulkan API? Not exactly sure but will also be testing the latest MESA builds from Oibaf's PPA in following days and report back on the issue :)
(In reply to Marko Popovic from comment #9) > https://cgit.freedesktop.org/mesa/mesa/commit/ > ?id=3Da2a68d551c1c2a4f13761ffa8f3f6f13fee7a384 >=20 > This might actually fix the ring_gfx type hangs or even sdma ones at l= east > for Vulkan API? Not exactly sure but will also be testing the latest M= ESA > builds from Oibaf's PPA in following days and report back on the issue= :) Sadly, I'm still getting the ring_gfx hangs after a few minutes of playing Trackmania 2.
(In reply to takios+fdbugs from comment #10) > (In reply to Marko Popovic from comment #9) > > https:= //cgit.freedesktop.org/mesa/mesa/commit/ > > ?id=3Da2a68d551c1c2a4f13761ffa8f3f6f13fee7a384 > >=20 > > This might actually fix the ring_gfx type hangs or even sdma ones= at least > > for Vulkan API? Not exactly sure but will also be testing the lat= est MESA > > builds from Oibaf's PPA in following days and report back on the = issue :) >=20 > Sadly, I'm still getting the ring_gfx hangs after a few minutes of pla= ying > Trackmania 2. Oh yes I forgot to add a reply here. It didn't solve any of the hangs for me either.
I am working on Navi10 RX5700 I am facing below issue when i run unigine-heaven benchmark [drm:amdgpu_dm_atomic_commit_tail [amdgpu]] *ERROR* Waiting for fences tim= ed out! [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=3D5075872, emitted seq=3D5075874 [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process heaven_x64 pid 13723 thread heaven_x64:cs0 pid 13741 [drm] GPU recovery disabled. Is any fix for it ?=20 Thanks on advance.
For hangs involving radv the AMD_DEBUG options aren't relevant. You should use RADV_DEBUG instead (probably doesn't support the same values= ). Also opening a bug in https://gitlab.freedesktop.org/mesa/mesa/issues is a good idea since gfx hangs are most likely a driver issue (radv or radeonsi, depending on the API used).
| What | Removed | Added |
|---|---|---|
| Priority | not set | medium |
RX 5700 XT Pop OS 19.10 latest Oibaf mesa not sure what llvm Anomaly 1.5.0 update 3 standalone 64 bit mod for S.T.A.L.K.E.R. Call of Pri= pyat running under wine d3dx11_43->dxvk (winetricks dxvk d3dcompiler_43 d3dx1= 1_43) Oct 30 02:49:30 pop-os kernel: [ 4864.627343] [drm:amdgpu_dm_commit_planes.constprop.0 [amdgpu]] *ERROR* Waiting for fenc= es timed out! Oct 30 02:49:30 pop-os kernel: [ 4869.231450] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=3D2626284, emitted seq=3D2626286 Oct 30 02:49:30 pop-os kernel: [ 4869.231486] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process AnomalyDX11.exe pid 5791 thr= ead AnomalyDX11.exe pid 5791 Oct 30 02:49:30 pop-os kernel: [ 4869.231487] [drm] GPU recovery disabled. Happens at random. Sometimes hangs straight away, sometimes can go over an = hour without crash. Complete crash, no option available besides hard reset. Not = even mouse pointer would move (as with sdma0 hang). I'm sorry if it's not the right place to report this, I'm somewhat new to a= ll of this.
(In reply to wychuchol from comment #14) > RX 5700 XT Pop OS 19.10 latest Oibaf mesa not su= re what llvm > Anomaly 1.5.0 update 3 standalone 64 bit mod for S.T.A.L.K.E.R. Call of > Pripyat running under wine d3dx11_43->dxvk (winetricks dxvk d3dcomp= iler_43 > d3dx11_43) >=20 > Oct 30 02:49:30 pop-os kernel: [ 4864.627343] > [drm:amdgpu_dm_commit_planes.constprop.0 [amdgpu]] *ERROR* Waiting for > fences timed out! > Oct 30 02:49:30 pop-os kernel: [ 4869.231450] [drm:amdgpu_job_timedout > [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=3D2626284, emit= ted > seq=3D2626286 > Oct 30 02:49:30 pop-os kernel: [ 4869.231486] [drm:amdgpu_job_timedout > [amdgpu]] *ERROR* Process information: process AnomalyDX11.exe pid 5791 > thread AnomalyDX11.exe pid 5791 > Oct 30 02:49:30 pop-os kernel: [ 4869.231487] [drm] GPU recovery disab= led. >=20 > Happens at random. Sometimes hangs straight away, sometimes can go ove= r an > hour without crash. Complete crash, no option available besides hard r= eset. > Not even mouse pointer would move (as with sdma0 hang). >=20 > I'm sorry if it's not the right place to report this, I'm somewhat new= to > all of this. Ring gfx type hangs tend to be in Mesa. Report here: https://gitlab.= freedesktop.org/mesa/mesa/issues Also I'm not sure how up to date the Oibaf repo is, but Mesa git landed ACO recently for Navi cards. You can try with RADV_PERFTEST=3Daco environment variable set if your Mesa is new enough, and you might have better luck with hangs.
(In reply to Andrew Sheldon from comment #16) > (In reply to wychuchol from comment #14) > > RX 5700 XT Pop OS 19.10 latest Oibaf mesa not sure what llvm > > Anomaly 1.5.0 update 3 standalone 64 bit mod for S.T.A.L.K.E.R. C= all of > > Pripyat running under wine d3dx11_43->dxvk (winetricks dxvk d3= dcompiler_43 > > d3dx11_43) > >=20 > > Oct 30 02:49:30 pop-os kernel: [ 4864.627343] > > [drm:amdgpu_dm_commit_planes.constprop.0 [amdgpu]] *ERROR* Waitin= g for > > fences timed out! > > Oct 30 02:49:30 pop-os kernel: [ 4869.231450] [drm:amdgpu_job_tim= edout > > [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=3D2626284,= emitted > > seq=3D2626286 > > Oct 30 02:49:30 pop-os kernel: [ 4869.231486] [drm:amdgpu_job_tim= edout > > [amdgpu]] *ERROR* Process information: process AnomalyDX11.exe pi= d 5791 > > thread AnomalyDX11.exe pid 5791 > > Oct 30 02:49:30 pop-os kernel: [ 4869.231487] [drm] GPU recovery = disabled. > >=20 > > Happens at random. Sometimes hangs straight away, sometimes can g= o over an > > hour without crash. Complete crash, no option available besides h= ard reset. > > Not even mouse pointer would move (as with sdma0 hang). > >=20 > > I'm sorry if it's not the right place to report this, I'm somewha= t new to > > all of this. >=20 > Ring gfx type hangs tend to be in Mesa. Report here: > https://gi= tlab.freedesktop.org/mesa/mesa/issues >=20 > Also I'm not sure how up to date the Oibaf repo is, but Mesa git lande= d ACO > recently for Navi cards. You can try with RADV_PERFTEST=3Daco environm= ent > variable set if your Mesa is new enough, and you might have better luc= k with > hangs. Thank you so very much, no way to be sure since they seemed to happen at ra= ndom but I think I'd experience at least 2 or 3 hangs in the time I've tested it= but smooth ride so far. No performance impact either but running this game as I= do I'm supposedly laying most of the calculations on CPU not GPU.
It happened again. This time without a game or anything runnin= g, barely logged in and opened a program and boom. Nov 2 12:42:07 pop-os kernel: [ 1675.883513] [drm:amdgpu_dm_commit_planes.constprop.0 [amdgpu]] *ERROR* Waiting for fenc= es timed out! Nov 2 12:42:07 pop-os kernel: [ 1680.747513] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=3D2714, emitted seq= =3D2716 Nov 2 12:42:07 pop-os kernel: [ 1680.747549] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process Xorg pid 2293 thread Xorg:cs0 pid 2294 Nov 2 12:42:07 pop-os kernel: [ 1680.747551] [drm] GPU recovery disabled. Only cursor moved, no clicks registered, restart achieved with REISUB. I tried registering at https://gitlab.freedesktop.org/mesa/mesa/issues but I'm getting no account confirmation mail so can't post it there.
Perhaps needs another entry started but it's related (since it= didn't happen before I tried RADV_PERFTEST=3Daco and AMD_DEBUG=3D"nongg,nodma")= so I'll post it in case someone has had same issues as me. After some time in Witcher 3 GOTY run with Lutris PC restarts on it's own. I thought something is overheating (I've noticed graphic card memory in PSens= or sometimes reaching 90 so I thought maybe that's what's happening) but I investigated kern.log and this always happened before that autonomous reset: Nov 2 22:01:53 pop-os kernel: [ 979.244964] pcieport 0000:00:01.1: AER: Corrected error received: 0000:01:00.0 Nov 2 22:01:53 pop-os kernel: [ 979.244967] nvme 0000:01:00.0: AER: PCIe = Bus Error: severity=3DCorrected, type=3DData Link Layer, (Transmitter ID) Nov 2 22:01:53 pop-os kernel: [ 979.244968] nvme 0000:01:00.0: AER: dev= ice [1987:5012] error status/mask=3D00001000/00006000 Nov 2 22:01:53 pop-os kernel: [ 979.244968] nvme 0000:01:00.0: AER: [1= 2] Timeout=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 Nov 2 22:01:53 pop-os kernel: [ 979.262629] Emergency Sync complete A solution I found is to add pci=3Dnommconf in /etc/default/grub to the lin= e=20 GRUB_CMDLINE_LINUX_DEFAULT=3D"quiet splash" (so it looks like thi= s: GRUB_CMDLINE_LINUX_DEFAULT=3D"quiet splash pci=3Dnommconf").
Barely started PC, opened palemoon, curse move only hang and t=
hen dozens of
graphical artifacts on screen like square patches of glitches.=20
Nov 3 13:15:10 pop-os kernel: [ 133.998883]
[drm:amdgpu_dm_commit_planes.constprop.0 [amdgpu]] *ERROR* Waiting for fenc=
es
timed out!
Nov 3 13:15:10 pop-os kernel: [ 139.118912] [drm:amdgpu_job_timedout
[amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=3D11145, emitted seq=
=3D11148
Nov 3 13:15:10 pop-os kernel: [ 139.118956] [drm:amdgpu_job_timedout
[amdgpu]] *ERROR* Process information: process gnome-shell pid 2588 thread
gnome-shel:cs0 pid 2606
Nov 3 13:15:10 pop-os kernel: [ 139.118958] [drm] GPU recovery disabled.
Then sometime later I got ring gfx related crash with Witcher 3 which didn't
happen before:
Nov 3 14:08:47 pop-os kernel: [ 3185.175837]
[drm:amdgpu_dm_commit_planes.constprop.0 [amdgpu]] *ERROR* Waiting for fenc=
es
timed out!
Nov 3 14:08:47 pop-os kernel: [ 3190.039750] [drm:amdgpu_job_timedout
[amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=3D1448573, emitted
seq=3D1448575
Nov 3 14:08:47 pop-os kernel: [ 3190.039786] [drm:amdgpu_job_timedout
[amdgpu]] *ERROR* Process information: process witcher3.exe pid 8100 thread
witcher3.exe pid 10168
Nov 3 14:08:47 pop-os kernel: [ 3190.039788] [drm] GPU recovery disabled.<=
/pre>