| Bug ID | 100712 |
|---|---|
| Summary | ring 0 stalled after bytes_moved_threshold reached - Cap Verd= e - HD 7770 |
| Product | DRI |
| Version | DRI git |
| Hardware | Other |
| OS | All |
| Status | NEW |
| Severity | normal |
| Priority | medium |
| Component | DRM/Radeon |
| Assignee | dri-devel@lists.freedesktop.org |
| Reporter | julien.isorce@gmail.com |
Kernel 4.9 from https://cgit.freedesktop.org/~agd5f/linux/log/?h=3Damd-staging-4.9 = and latest mesa. (same result with drm-next-4.12 branch) Same result with kernel 4.8 and mesa 12.0.6. In kernel radeon_object.c::radeon_bo_list_validate, once "bytes_moved = > bytes_moved_threshold" is reached (this is the case for 850 bo in the = same list_for_each_entry loop), I can see that radeon_ib_schedule emits a fence = that it takes more than the radeon.lockup_timeout to be signaled. In radeon_fence_activity, I checked that the "last_emitted" is th= e seq number for this last emited fence. And last_seq is equal to last_emitted-1. Then the next call to ttm_wait_bo blocks (15 * HZ > radeon.lockup_timeou= t) until gpu lockup which leads to a gpu reset. Also it seems the fence is signaled by swapper after more than 10 seconds b= ut it is too late. I requires to reduce the "15" param above to 4 to= see that. Is it normal that radeon_bo_list_validate still tries to move the bo if bytes_moved_threshold is reached ? Indeed ttm_bo_validate is always called = (it blits from vram to vram). Is it also normal that ttm_bo_validate is called with evict flag as true on= ce bytes_moved_threshold is reached ?
Created attachment 130902 [de=
tails]
dmesg_HD7770_kernel_amd-staging-4.9_ring_stalled
Created attachment 130903 [de=
tails]
dmesg_HD7770_kernel_amd-staging-4.9_ring_stalled
| What | Removed | Added |
|---|---|---|
| Attachment #130903 is obsolete | 1 |
Created attachment 130904 [details]
ddebug_dumps_HD7770_kernel_amd-staging-4.9_ring_stalled
(In reply to Julien Isorce from comment #0) > In kernel radeon_object.c::radeon_bo_list_valida= te, once "bytes_moved > > bytes_moved_threshold" is reached (this is the case for 850 bo in= the same > list_for_each_entry loop), I can see that radeon_ib_schedule emits a f= ence > that it takes more than the radeon.lockup_timeout to be signaled. radeon_ib_schedule is called for submitting the command stream from userspa= ce, not for any BO moves directly, right? How did you determine that this hang is directly related to bytes_moved / bytes_moved_threshold? Maybe it's only indirectly related, e.g. due to the threshold preventing a BO from being moved to VRAM despite userspace's preference. > Also it seems the fence is signaled by swapper a= fter more than 10 seconds > but it is too late. I requires to reduce the "15" param abov= e to 4 to see > that. How does "swapper" (what is that exactly?) signal the fence? > Is it normal that radeon_bo_list_validate still = tries to move the bo if > bytes_moved_threshold is reached ? There are circumstances where a BO has to be moved even though the threshol= d is reached. > Indeed ttm_bo_validate is always called ttm_bo_validate must be called for every BO referenced by the command stream from userspace for correct lifetime management of its memory. > (it blits from vram to vram). It might be worth looking into why this happens, though. If domain =3D=3D current_domain =3D=3D RADEON_GEM_DOMAIN_VRAM, I wouldn't expect ttm_bo_vali= date to trigger a blit.
(In reply to Michel D=C3=A4nzer from comment #4) > (In reply to Julien Isorce from comment #0) > > In kernel radeon_object.c::radeon_bo_list_validate, once "by= tes_moved > > > bytes_moved_threshold" is reached (this is the case for 850 = bo in the same > > list_for_each_entry loop), I can see that radeon_ib_schedule emit= s a fence > > that it takes more than the radeon.lockup_timeout to be signaled. >=20 > radeon_ib_schedule is called for submitting the command stream from > userspace, not for any BO moves directly, right? >=20 > How did you determine that this hang is directly related to bytes_move= d / > bytes_moved_threshold? Maybe it's only indirectly related, e.g. due to= the > threshold preventing a BO from being moved to VRAM despite userspace's > preference. > I added a trace and the fence that is not signaled on time is always the one emited by radeon_ib_schedule after that the bytes_moved_threshold is reache= d. But you are right it could be only indirectly related. Here is the sequence I have: ioctl_radeon_cs radeon_bo_list_validate bytes_moved > bytes_moved_threshold(=3D1024*1024ull) 800 bo are not moved from gtt to vram because of that. radeon_cs_ib_vm_chunk radeon_ib_schedule(rdev, &parser->ib, NULL, true); radeon_fence_emit on ring 0 r600_mmio_hdp_flush /ioctl_radeon_cs Then anything calling ttm_bo_wait will block more than the radeon.lockup_timeout because the above fence is not signaled on time. Could it be that something is not flushed properly ? (ref: https://patchwork.k= ernel.org/patch/5807141/ ? tlb_flush ?)=20 Are you saying that some bos are required to be moved from gtt to vram in o= rder for this fence to be signaled ? As you can see above it happens when vram_usage >=3D half_vram so radeon_bo_get_threshold_for_moves returns 1024*1024, which explains why onl= y 1 or 2 bos can be moved from gtt to vram in that case and why all others are forced to stay in gtt. In the same run of radeon_bo_list_validate there are many calls to ttm_bo_validate with both domain and current_domain as VRAM, this is the ca= se for around 400 bo. Maybe this cause delay for this fence to be signaled, providing vram usage is high too. >=20 > > Also it seems the fence is signaled by swapper after more than 10= seconds > > but it is too late. I requires to reduce the "15" param= above to 4 to see > > that. >=20 > How does "swapper" (what is that exactly?) signal the fence?= My wording was wrong sorry, I should have said "the first entity notic= ing that the fence is signaled" by calling radeon_fence_activity. swapper is th= e name for process 0 (idle). I change drm logging to print process name and id: (current->comm, current->pid) >=20 > It might be worth looking into why this happens, though. If domain =3D= =3D > current_domain =3D=3D RADEON_GEM_DOMAIN_VRAM, I wouldn't expect ttm_bo= _validate > to trigger a blit. I will check though I think I get just confused by a previous trace.
Created =
attachment 130947 [details]
dmesg_W9000_with_custom_fence_debug.log
| What | Removed | Added |
|---|---|---|
| Summary | ring 0 stalled after bytes_moved_threshold reached - Cap Ver= de - HD 7770 | ring 0 stalled after bytes_moved_threshold reached - CAPVERD= E/HD7770 - TAHITI/W9000 |
I made 2 apitrace using Zach's test mentioned here https://bugs.freedesktop.org/show_= bug.cgi?id=3D100465#c24 . This test is also good to reproduce this ring 0 stalled issue. 1: apitrace ideal for vram size 2048 (ex: HD7770) https://drive.google.com/file/d/0Bzat_iFKrgCWYzBlZFFLQjgyR= U0/view?usp=3Dsharing 2: apitrace ideal for vram size 6144 (ex: W9000) https://drive.google.com/file/d/0Bzat_iFKrgCWczgzM2FzaVFTU= Xc/view?usp=3Dsharing DISPLAY=3D:0 apitrace replay thrash.trace Also I have attached the log (dmesg_W9000_with_custom_fence_debug.log) I get with my dev branch here https://github.com/CapOM/linux/commits/amd-staging-4.9_add_debug_f= ences where I added traces to debug ring N stalled issues. It prints the backtrace from w= here it waits for the fence and it also prints the backtrace from where it has emited that fence. Also note that setting R600_DEBUG=3Dnowc avoids this ring N stalled (so the endless fence is signaled).
Hack submitted here https://patchwork.kernel.org/patch/9695945/. It contains some info in the commit message and in the replies. For those who want to t= ry, it is easier to just set R600_DEBUG=3Dnowc as said in #7.
| What | Removed | Added |
|---|---|---|
| Resolution | --- | MOVED |
| Status | NEW | RESOLVED |
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this = link to our GitLab instance: https://gitlab.freedesktop.org/drm/amd/issues/793.