Re: [PATCH v6 00/22] Add generic memory shrinker to VirtIO-GPU and Panfrost DRM drivers

Robin Murphy
2022-06-28T12:31:16Z

On 2022-05-27 00:50, Dmitry Osipenko wrote:
> Hello,
>
> This patchset introduces memory shrinker for the VirtIO-GPU DRM driver
> and adds memory purging and eviction support to VirtIO-GPU driver.
>
> The new dma-buf locking convention is introduced here as well.
>
> During OOM, the shrinker will release BOs that are marked as "not needed"
> by userspace using the new madvise IOCTL, it will also evict idling BOs
> to SWAP. The userspace in this case is the Mesa VirGL driver, it will mark
> the cached BOs as "not needed", allowing kernel driver to release memory
> of the cached shmem BOs on lowmem situations, preventing OOM kills.
>
> The Panfrost driver is switched to use generic memory shrinker.

I think we still have some outstanding issues here - Alyssa reported
some weirdness yesterday, so I just tried provoking a low-memory
condition locally with this series applied and a few debug options
enabled, and the results as below were... interesting.

Thanks,
Robin.

----->8-----
[ 68.295951] ======================================================
[ 68.295956] WARNING: possible circular locking dependency detected
[ 68.295963] 5.19.0-rc3+ #400 Not tainted
[ 68.295972] ------------------------------------------------------
[ 68.295977] cc1/295 is trying to acquire lock:
[ 68.295986] ffff000008d7f1a0
(reservation_ww_class_mutex){+.+.}-{3:3}, at: drm_gem_shmem_free+0x7c/0x198
[ 68.296036]
[ 68.296036] but task is already holding lock:
[ 68.296041] ffff80000c14b820 (fs_reclaim){+.+.}-{0:0}, at:
__alloc_pages_slowpath.constprop.0+0x4d8/0x1470
[ 68.296080]
[ 68.296080] which lock already depends on the new lock.
[ 68.296080]
[ 68.296085]
[ 68.296085] the existing dependency chain (in reverse order) is:
[ 68.296090]
[ 68.296090] -> #1 (fs_reclaim){+.+.}-{0:0}:
[ 68.296111] fs_reclaim_acquire+0xb8/0x150
[ 68.296130] dma_resv_lockdep+0x298/0x3fc
[ 68.296148] do_one_initcall+0xe4/0x5f8
[ 68.296163] kernel_init_freeable+0x414/0x49c
[ 68.296180] kernel_init+0x2c/0x148
[ 68.296195] ret_from_fork+0x10/0x20
[ 68.296207]
[ 68.296207] -> #0 (reservation_ww_class_mutex){+.+.}-{3:3}:
[ 68.296229] __lock_acquire+0x1724/0x2398
[ 68.296246] lock_acquire+0x218/0x5b0
[ 68.296260] __ww_mutex_lock.constprop.0+0x158/0x2378
[ 68.296277] ww_mutex_lock+0x7c/0x4d8
[ 68.296291] drm_gem_shmem_free+0x7c/0x198
[ 68.296304] panfrost_gem_free_object+0x118/0x138
[ 68.296318] drm_gem_object_free+0x40/0x68
[ 68.296334] drm_gem_shmem_shrinker_run_objects_scan+0x42c/0x5b8
[ 68.296352] drm_gem_shmem_shrinker_scan_objects+0xa4/0x170
[ 68.296368] do_shrink_slab+0x220/0x808
[ 68.296381] shrink_slab+0x11c/0x408
[ 68.296392] shrink_node+0x6ac/0xb90
[ 68.296403] do_try_to_free_pages+0x1dc/0x8d0
[ 68.296416] try_to_free_pages+0x1ec/0x5b0
[ 68.296429] __alloc_pages_slowpath.constprop.0+0x528/0x1470
[ 68.296444] __alloc_pages+0x4e0/0x5b8
[ 68.296455] __folio_alloc+0x24/0x60
[ 68.296467] vma_alloc_folio+0xb8/0x2f8
[ 68.296483] alloc_zeroed_user_highpage_movable+0x58/0x68
[ 68.296498] __handle_mm_fault+0x918/0x12a8
[ 68.296513] handle_mm_fault+0x130/0x300
[ 68.296527] do_page_fault+0x1d0/0x568
[ 68.296539] do_translation_fault+0xa0/0xb8
[ 68.296551] do_mem_abort+0x68/0xf8
[ 68.296562] el0_da+0x74/0x100
[ 68.296572] el0t_64_sync_handler+0x68/0xc0
[ 68.296585] el0t_64_sync+0x18c/0x190
[ 68.296596]
[ 68.296596] other info that might help us debug this:
[ 68.296596]
[ 68.296601] Possible unsafe locking scenario:
[ 68.296601]
[ 68.296604] CPU0 CPU1
[ 68.296608] ---- ----
[ 68.296612] lock(fs_reclaim);
[ 68.296622]
lock(reservation_ww_class_mutex);
[ 68.296633] lock(fs_reclaim);
[ 68.296644] lock(reservation_ww_class_mutex);
[ 68.296654]
[ 68.296654] *** DEADLOCK ***
[ 68.296654]
[ 68.296658] 3 locks held by cc1/295:
[ 68.296666] #0: ffff00000616e898 (&mm->mmap_lock){++++}-{3:3}, at:
do_page_fault+0x144/0x568
[ 68.296702] #1: ffff80000c14b820 (fs_reclaim){+.+.}-{0:0}, at:
__alloc_pages_slowpath.constprop.0+0x4d8/0x1470
[ 68.296740] #2: ffff80000c1215b0 (shrinker_rwsem){++++}-{3:3}, at:
shrink_slab+0xc0/0x408
[ 68.296774]
[ 68.296774] stack backtrace:
[ 68.296780] CPU: 2 PID: 295 Comm: cc1 Not tainted 5.19.0-rc3+ #400
[ 68.296794] Hardware name: ARM LTD ARM Juno Development Platform/ARM
Juno Development Platform, BIOS EDK II Sep 3 2019
[ 68.296803] Call trace:
[ 68.296808] dump_backtrace+0x1e4/0x1f0
[ 68.296821] show_stack+0x20/0x70
[ 68.296832] dump_stack_lvl+0x8c/0xb8
[ 68.296849] dump_stack+0x1c/0x38
[ 68.296864] print_circular_bug.isra.0+0x284/0x378
[ 68.296881] check_noncircular+0x1d8/0x1f8
[ 68.296896] __lock_acquire+0x1724/0x2398
[ 68.296911] lock_acquire+0x218/0x5b0
[ 68.296926] __ww_mutex_lock.constprop.0+0x158/0x2378
[ 68.296942] ww_mutex_lock+0x7c/0x4d8
[ 68.296956] drm_gem_shmem_free+0x7c/0x198
[ 68.296970] panfrost_gem_free_object+0x118/0x138
[ 68.296984] drm_gem_object_free+0x40/0x68
[ 68.296999] drm_gem_shmem_shrinker_run_objects_scan+0x42c/0x5b8
[ 68.297017] drm_gem_shmem_shrinker_scan_objects+0xa4/0x170
[ 68.297033] do_shrink_slab+0x220/0x808
[ 68.297045] shrink_slab+0x11c/0x408
[ 68.297056] shrink_node+0x6ac/0xb90
[ 68.297068] do_try_to_free_pages+0x1dc/0x8d0
[ 68.297081] try_to_free_pages+0x1ec/0x5b0
[ 68.297094] __alloc_pages_slowpath.constprop.0+0x528/0x1470
[ 68.297110] __alloc_pages+0x4e0/0x5b8
[ 68.297122] __folio_alloc+0x24/0x60
[ 68.297134] vma_alloc_folio+0xb8/0x2f8
[ 68.297148] alloc_zeroed_user_highpage_movable+0x58/0x68
[ 68.297163] __handle_mm_fault+0x918/0x12a8
[ 68.297178] handle_mm_fault+0x130/0x300
[ 68.297193] do_page_fault+0x1d0/0x568
[ 68.297205] do_translation_fault+0xa0/0xb8
[ 68.297218] do_mem_abort+0x68/0xf8
[ 68.297229] el0_da+0x74/0x100
[ 68.297239] el0t_64_sync_handler+0x68/0xc0
[ 68.297252] el0t_64_sync+0x18c/0x190
[ 68.471812] arm-scmi firmware:scmi: timed out in resp(caller:
scmi_power_state_set+0x11c/0x190)
[ 68.501947] arm-scmi firmware:scmi: Message for 119 type 0 is not
expected!
[ 68.939686] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000915e2d34
[ 69.739386] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ac77ac55
[ 70.415329] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ee980c7e
[ 70.987166] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ffb7ff37
[ 71.914939] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000000e92b26e
[ 72.426987] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000c036a911
[ 73.578683] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000001c6fc094
[ 74.090555] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000075d00f9
[ 74.922709] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000005add546
[ 75.434401] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000000154189b
[ 76.394300] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ac77ac55
[ 76.906236] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ee980c7e
[ 79.657234] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000f6d059fb
[ 80.168831] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000061a0f6bf
[ 80.808354] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000071ade02
[ 81.319967] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000b0afea73
[ 81.831574] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000d78f36c2
[ 82.343160] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000000f689397
[ 83.046689] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000412c2a2f
[ 83.558352] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000020e551b3
[ 84.261913] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000009437aace
[ 84.773576] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000001c6fc094
[ 85.317275] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000c036a911
[ 85.829035] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000000e92b26e
[ 86.660555] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ac77ac55
[ 87.172126] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000b940e406
[ 87.875846] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000001c6fc094
[ 88.387443] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000009437aace
[ 89.059175] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000075dadb7f
[ 89.570960] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000005add546
[ 90.146687] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000cba2873c
[ 90.662497] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000a4beb490
[ 95.392748] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000005b5fc4ec
[ 95.904179] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000a17436ee
[ 96.416085] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000003888d2a7
[ 96.927874] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000093e04a98
[ 97.439742] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000c036a911
[ 97.954109] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000084c51113
[ 98.467374] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000664663ce
[ 98.975192] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000060f2d45c
[ 99.487231] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000b29288f8
[ 99.998833] panfrost 2d000000.gpu: gpu sched timeout, js=0,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000002f07ab24
[ 100.510744] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000008c15c751
[ 100.511411]
==================================================================
[ 100.511419] BUG: KASAN: use-after-free in irq_work_single+0xa4/0x110
[ 100.511445] Write of size 4 at addr ffff0000107f5830 by task
glmark2-es2-drm/280
[ 100.511458]
[ 100.511464] CPU: 1 PID: 280 Comm: glmark2-es2-drm Not tainted
5.19.0-rc3+ #400
[ 100.511479] Hardware name: ARM LTD ARM Juno Development Platform/ARM
Juno Development Platform, BIOS EDK II Sep 3 2019
[ 100.511489] Call trace:
[ 100.511494] dump_backtrace+0x1e4/0x1f0
[ 100.511512] show_stack+0x20/0x70
[ 100.511523] dump_stack_lvl+0x8c/0xb8
[ 100.511543] print_report+0x16c/0x668
[ 100.511559] kasan_report+0x80/0x208
[ 100.511574] kasan_check_range+0x100/0x1b8
[ 100.511590] __kasan_check_write+0x34/0x60
[ 100.511607] irq_work_single+0xa4/0x110
[ 100.511619] irq_work_run_list+0x6c/0x88
[ 100.511632] irq_work_run+0x28/0x48
[ 100.511644] ipi_handler+0x254/0x468
[ 100.511664] handle_percpu_devid_irq+0x11c/0x518
[ 100.511681] generic_handle_domain_irq+0x50/0x70
[ 100.511699] gic_handle_irq+0xd4/0x118
[ 100.511711] call_on_irq_stack+0x2c/0x58
[ 100.511725] do_interrupt_handler+0xc0/0xc8
[ 100.511741] el1_interrupt+0x40/0x68
[ 100.511754] el1h_64_irq_handler+0x18/0x28
[ 100.511767] el1h_64_irq+0x64/0x68
[ 100.511778] irq_work_queue+0xc0/0xd8
[ 100.511790] drm_sched_entity_fini+0x2c4/0x3b0
[ 100.511805] drm_sched_entity_destroy+0x2c/0x40
[ 100.511818] panfrost_job_close+0x44/0x1c0
[ 100.511833] panfrost_postclose+0x38/0x60
[ 100.511845] drm_file_free.part.0+0x33c/0x4b8
[ 100.511862] drm_close_helper.isra.0+0xc0/0xd8
[ 100.511877] drm_release+0xe4/0x1e0
[ 100.511891] __fput+0xf8/0x390
[ 100.511904] ____fput+0x18/0x28
[ 100.511917] task_work_run+0xc4/0x1e0
[ 100.511929] do_exit+0x554/0x1168
[ 100.511945] do_group_exit+0x60/0x108
[ 100.511960] __arm64_sys_exit_group+0x34/0x38
[ 100.511977] invoke_syscall+0x64/0x180
[ 100.511993] el0_svc_common.constprop.0+0x13c/0x170
[ 100.512012] do_el0_svc+0x48/0xe8
[ 100.512028] el0_svc+0x5c/0xe0
[ 100.512038] el0t_64_sync_handler+0xb8/0xc0
[ 100.512051] el0t_64_sync+0x18c/0x190
[ 100.512064]
[ 100.512068] Allocated by task 280:
[ 100.512075] kasan_save_stack+0x2c/0x58
[ 100.512091] __kasan_kmalloc+0x90/0xb8
[ 100.512105] kmem_cache_alloc_trace+0x1d4/0x330
[ 100.512118] panfrost_ioctl_submit+0x100/0x630
[ 100.512131] drm_ioctl_kernel+0x160/0x250
[ 100.512147] drm_ioctl+0x36c/0x628
[ 100.512161] __arm64_sys_ioctl+0xd8/0x120
[ 100.512178] invoke_syscall+0x64/0x180
[ 100.512194] el0_svc_common.constprop.0+0x13c/0x170
[ 100.512211] do_el0_svc+0x48/0xe8
[ 100.512226] el0_svc+0x5c/0xe0
[ 100.512236] el0t_64_sync_handler+0xb8/0xc0
[ 100.512248] el0t_64_sync+0x18c/0x190
[ 100.512259]
[ 100.512262] Freed by task 280:
[ 100.512268] kasan_save_stack+0x2c/0x58
[ 100.512283] kasan_set_track+0x2c/0x40
[ 100.512296] kasan_set_free_info+0x28/0x50
[ 100.512312] __kasan_slab_free+0xf0/0x170
[ 100.512326] kfree+0x124/0x418
[ 100.512337] panfrost_job_cleanup+0x1f0/0x298
[ 100.512350] panfrost_job_free+0x80/0xb0
[ 100.512363] drm_sched_entity_kill_jobs_irq_work+0x80/0xa0
[ 100.512377] irq_work_single+0x88/0x110
[ 100.512389] irq_work_run_list+0x6c/0x88
[ 100.512401] irq_work_run+0x28/0x48
[ 100.512413] ipi_handler+0x254/0x468
[ 100.512427] handle_percpu_devid_irq+0x11c/0x518
[ 100.512443] generic_handle_domain_irq+0x50/0x70
[ 100.512460] gic_handle_irq+0xd4/0x118
[ 100.512471]
[ 100.512474] The buggy address belongs to the object at ffff0000107f5800
[ 100.512474] which belongs to the cache kmalloc-512 of size 512
[ 100.512484] The buggy address is located 48 bytes inside of
[ 100.512484] 512-byte region [ffff0000107f5800, ffff0000107f5a00)
[ 100.512497]
[ 100.512500] The buggy address belongs to the physical page:
[ 100.512506] page:000000000a626feb refcount:1 mapcount:0
mapping:0000000000000000 index:0x0 pfn:0x907f4
[ 100.512520] head:000000000a626feb order:2 compound_mapcount:0
compound_pincount:0
[ 100.512530] flags:
0xffff00000010200(slab|head|node=0|zone=0|lastcpupid=0xffff)
[ 100.512556] raw: 0ffff00000010200 fffffc0000076400 dead000000000002
ffff000000002600
[ 100.512569] raw: 0000000000000000 0000000080100010 00000001ffffffff
0000000000000000
[ 100.512577] page dumped because: kasan: bad access detected
[ 100.512582]
[ 100.512585] Memory state around the buggy address:
[ 100.512592] ffff0000107f5700: fc fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc
[ 100.512602] ffff0000107f5780: fc fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc
[ 100.512612] >ffff0000107f5800: fa fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 100.512619] ^
[ 100.512627] ffff0000107f5880: fb fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 100.512636] ffff0000107f5900: fb fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb
[ 100.512643]
==================================================================
[ 101.022573] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000be4b1b31
[ 101.534469] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000a8ff2c8a
[ 101.535981] BUG: sleeping function called from invalid context at
kernel/locking/mutex.c:870
[ 101.535994] in_atomic(): 1, irqs_disabled(): 128, non_block: 0, pid:
280, name: glmark2-es2-drm
[ 101.536006] preempt_count: 10000, expected: 0
[ 101.536012] RCU nest depth: 0, expected: 0
[ 101.536019] INFO: lockdep is turned off.
[ 101.536023] irq event stamp: 1666508
[ 101.536029] hardirqs last enabled at (1666507): [<ffff80000997ed70>]
exit_to_kernel_mode.isra.0+0x40/0x140
[ 101.536056] hardirqs last disabled at (1666508): [<ffff800009985030>]
__schedule+0xb38/0xea8
[ 101.536076] softirqs last enabled at (1664950): [<ffff800008010ac8>]
__do_softirq+0x6b8/0x89c
[ 101.536092] softirqs last disabled at (1664941): [<ffff8000080e4fdc>]
irq_exit_rcu+0x27c/0x2b0
[ 101.536118] CPU: 1 PID: 280 Comm: glmark2-es2-drm Tainted: G B
5.19.0-rc3+ #400
[ 101.536134] Hardware name: ARM LTD ARM Juno Development Platform/ARM
Juno Development Platform, BIOS EDK II Sep 3 2019
[ 101.536143] Call trace:
[ 101.536147] dump_backtrace+0x1e4/0x1f0
[ 101.536161] show_stack+0x20/0x70
[ 101.536171] dump_stack_lvl+0x8c/0xb8
[ 101.536189] dump_stack+0x1c/0x38
[ 101.536204] __might_resched+0x1f0/0x2b0
[ 101.536220] __might_sleep+0x74/0xd0
[ 101.536234] ww_mutex_lock+0x40/0x4d8
[ 101.536249] drm_gem_shmem_free+0x7c/0x198
[ 101.536264] panfrost_gem_free_object+0x118/0x138
[ 101.536278] drm_gem_object_free+0x40/0x68
[ 101.536295] panfrost_job_cleanup+0x1bc/0x298
[ 101.536309] panfrost_job_free+0x80/0xb0
[ 101.536322] drm_sched_entity_kill_jobs_irq_work+0x80/0xa0
[ 101.536337] irq_work_single+0x88/0x110
[ 101.536351] irq_work_run_list+0x6c/0x88
[ 101.536364] irq_work_run+0x28/0x48
[ 101.536375] ipi_handler+0x254/0x468
[ 101.536392] handle_percpu_devid_irq+0x11c/0x518
[ 101.536409] generic_handle_domain_irq+0x50/0x70
[ 101.536428] gic_handle_irq+0xd4/0x118
[ 101.536439] call_on_irq_stack+0x2c/0x58
[ 101.536453] do_interrupt_handler+0xc0/0xc8
[ 101.536468] el1_interrupt+0x40/0x68
[ 101.536479] el1h_64_irq_handler+0x18/0x28
[ 101.536492] el1h_64_irq+0x64/0x68
[ 101.536503] __asan_load8+0x30/0xd0
[ 101.536519] drm_sched_entity_fini+0x1e8/0x3b0
[ 101.536532] drm_sched_entity_destroy+0x2c/0x40
[ 101.536545] panfrost_job_close+0x44/0x1c0
[ 101.536559] panfrost_postclose+0x38/0x60
[ 101.536571] drm_file_free.part.0+0x33c/0x4b8
[ 101.536586] drm_close_helper.isra.0+0xc0/0xd8
[ 101.536601] drm_release+0xe4/0x1e0
[ 101.536615] __fput+0xf8/0x390
[ 101.536628] ____fput+0x18/0x28
[ 101.536640] task_work_run+0xc4/0x1e0
[ 101.536652] do_exit+0x554/0x1168
[ 101.536667] do_group_exit+0x60/0x108
[ 101.536682] __arm64_sys_exit_group+0x34/0x38
[ 101.536698] invoke_syscall+0x64/0x180
[ 101.536714] el0_svc_common.constprop.0+0x13c/0x170
[ 101.536733] do_el0_svc+0x48/0xe8
[ 101.536748] el0_svc+0x5c/0xe0
[ 101.536759] el0t_64_sync_handler+0xb8/0xc0
[ 101.536771] el0t_64_sync+0x18c/0x190
[ 101.541928] ------------[ cut here ]------------
[ 101.541934] kernel BUG at kernel/irq_work.c:235!
[ 101.541944] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[ 101.541961] Modules linked in:
[ 101.541978] CPU: 1 PID: 280 Comm: glmark2-es2-drm Tainted: G B W
5.19.0-rc3+ #400
[ 101.541997] Hardware name: ARM LTD ARM Juno Development Platform/ARM
Juno Development Platform, BIOS EDK II Sep 3 2019
[ 101.542009] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS
BTYPE=--)
[ 101.542027] pc : irq_work_run_list+0x80/0x88
[ 101.542044] lr : irq_work_run+0x34/0x48
[ 101.542060] sp : ffff80000da37eb0
[ 101.542069] x29: ffff80000da37eb0 x28: ffff000006bb0000 x27:
ffff000006bb0008
[ 101.542107] x26: ffff80000da37f20 x25: ffff8000080304d8 x24:
0000000000000001
[ 101.542142] x23: ffff80000abcd008 x22: ffff80000da37ed0 x21:
ffff80001c0de000
[ 101.542177] x20: ffff80000abcd008 x19: ffff80000abdbad0 x18:
0000000000000000
[ 101.542212] x17: 616e202c30383220 x16: 3a646970202c3020 x15:
ffff8000082df9d0
[ 101.542246] x14: ffff800008dfada8 x13: 0000000000000003 x12:
1fffe000018b2a06
[ 101.542280] x11: ffff6000018b2a06 x10: dfff800000000000 x9 :
ffff00000c595033
[ 101.542315] x8 : ffff6000018b2a07 x7 : 0000000000000001 x6 :
00000000000000fb
[ 101.542349] x5 : ffff00000c595030 x4 : 0000000000000000 x3 :
ffff00000c595030
[ 101.542382] x2 : 0000000000000000 x1 : 0000000000000000 x0 :
ffff000026cb9ad0
[ 101.542416] Call trace:
[ 101.542424] irq_work_run_list+0x80/0x88
[ 101.542441] ipi_handler+0x254/0x468
[ 101.542460] handle_percpu_devid_irq+0x11c/0x518
[ 101.542480] generic_handle_domain_irq+0x50/0x70
[ 101.542501] gic_handle_irq+0xd4/0x118
[ 101.542516] call_on_irq_stack+0x2c/0x58
[ 101.542534] do_interrupt_handler+0xc0/0xc8
[ 101.542553] el1_interrupt+0x40/0x68
[ 101.542568] el1h_64_irq_handler+0x18/0x28
[ 101.542584] el1h_64_irq+0x64/0x68
[ 101.542599] __asan_load8+0x30/0xd0
[ 101.542617] drm_sched_entity_fini+0x1e8/0x3b0
[ 101.542634] drm_sched_entity_destroy+0x2c/0x40
[ 101.542651] panfrost_job_close+0x44/0x1c0
[ 101.542669] panfrost_postclose+0x38/0x60
[ 101.542685] drm_file_free.part.0+0x33c/0x4b8
[ 101.542704] drm_close_helper.isra.0+0xc0/0xd8
[ 101.542723] drm_release+0xe4/0x1e0
[ 101.542740] __fput+0xf8/0x390
[ 101.542756] ____fput+0x18/0x28
[ 101.542773] task_work_run+0xc4/0x1e0
[ 101.542788] do_exit+0x554/0x1168
[ 101.542806] do_group_exit+0x60/0x108
[ 101.542825] __arm64_sys_exit_group+0x34/0x38
[ 101.542845] invoke_syscall+0x64/0x180
[ 101.542865] el0_svc_common.constprop.0+0x13c/0x170
[ 101.542887] do_el0_svc+0x48/0xe8
[ 101.542906] el0_svc+0x5c/0xe0
[ 101.542921] el0t_64_sync_handler+0xb8/0xc0
[ 101.542938] el0t_64_sync+0x18c/0x190
[ 101.542960] Code: a94153f3 a8c27bfd d50323bf d65f03c0 (d4210000)
[ 101.542979] ---[ end trace 0000000000000000 ]---
[ 101.678650] Kernel panic - not syncing: Oops - BUG: Fatal exception
in interrupt
[ 102.046301] panfrost 2d000000.gpu: gpu sched timeout, js=1,
config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000001da14c98
[ 103.227334] SMP: stopping secondary CPUs
[ 103.241055] Kernel Offset: disabled
[ 103.254316] CPU features: 0x800,00184810,00001086
[ 103.268904] Memory Limit: 800 MB
[ 103.411625] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal
exception in interrupt ]---