[bug report] fstests generic/774 hang again

[bug report] fstests generic/774 hang again

[Shin'ichiro Kawasaki]

I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB
TCMU fileio devices. Actually I once reported this hang symptom in last October,
and Darrick and John kindly took some actions [1]. Since then, the hang
disappeared and I have not observed the hang almost one year. However, my test
system started reporting the hang again last week.

[1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t

FYI, here I attache the kernel message [2]. I observed the hang last week with
the kernel on the xfs/for-next branch at the commit 0ca15a1a1151 ("xfs: remove
an extra cast in xfs_file_compat_ioctl"). Today, I tried again with the kernel
at current xfs/for-next branch tip 4266ffdfd7cb ("xfs: remove an extra cast in
xfs_file_compat_ioctl"), and observed the hang again. The hang can be recreated
in stable manner by repeating the test case 20 times or so.

I also tried other test cases in atmoicwrites group, and observed no failure.

 generic/765        [not run] write atomic not supported by this block device
 generic/767         11s
 generic/768         13s
 generic/769         13s
 generic/770         32s
 generic/773        [not run] write atomic not supported by this block device
 generic/774        (skipped)
 generic/775         293s
 generic/776        [notrun] write atomic not supported by this block device
 generic/778         48s
 xfs/838            [not run] External volumes not in use, skipped this test
 xfs/839            [not run] XFS error injection requires CONFIG_XFS_DEBUG
 xfs/840            [not run] write atomic not supported by this block device

Action for fix will be appreciated. If I can do anything on my test node, please
let me know.


[2] kernel message during generic/774

[144539.456849][T1634092] run fstests generic/774 at 2026-09-09 16:51:53
[144539.691199][T3820710] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[144543.436776][T3868062] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[144543.648458][T3872722] XFS (sdg): Mounting V5 Filesystem 306fb2c4-6cd3-4cbd-b9ed-953c0bc0d3b8
[144543.674712][T3872722] XFS (sdg): Ending clean mount
[144544.328859][T3872774] XFS (sdg): Unmounting Filesystem 306fb2c4-6cd3-4cbd-b9ed-953c0bc0d3b8
[144545.743530][T3795425] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[144545.964812][T3872823] XFS (sdg): Mounting V5 Filesystem 91ee7431-7139-4d92-bea7-b7802c3f70d2
[144545.990603][T3872823] XFS (sdg): Ending clean mount
[144870.819650][  T172] INFO: task kworker/10:173:3785794 blocked for more than 122 seconds.
[144870.821440][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144870.823252][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144870.825054][  T172] task:kworker/10:173  state:D stack:0     pid:3785794 tgid:3785794 ppid:2      task_flags:0x4248060 flags:0x00080000
[144870.827098][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144870.828170][  T172] Call Trace:
[144870.829209][  T172]  <TASK>
[144870.830154][  T172]  __schedule+0xc86/0x1b10
[144870.831126][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144870.832166][  T172]  ? __pfx___schedule+0x10/0x10
[144870.833250][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.834400][  T172]  ? wq_worker_sleeping+0x213/0x280
[144870.835559][  T172]  ? find_held_lock+0x2b/0x80
[144870.836634][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144870.837699][  T172]  schedule+0xe5/0x2e0
[144870.838721][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.839915][  T172]  schedule_preempt_disabled+0x16/0x30
[144870.841017][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144870.842139][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144870.843236][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144870.844383][  T172]  ? __lock_acquire+0x691/0xd00
[144870.845598][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.846840][  T172]  ? lock_acquire+0x11a/0x140
[144870.848019][  T172]  down_write_nested+0x1c0/0x1f0
[144870.849121][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144870.850284][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144870.852895][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144870.855451][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144870.858004][  T172]  ? __lock_acquire+0x691/0xd00
[144870.859160][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144870.861775][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144870.864257][  T172]  iomap_dio_complete+0x140/0xb20
[144870.865378][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144870.866570][  T172]  iomap_dio_complete_work+0x58/0x90
[144870.867785][  T172]  process_one_work+0x947/0x1760
[144870.868995][  T172]  ? __pfx_process_one_work+0x10/0x10
[144870.870062][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.871227][  T172]  ? lock_is_held_type+0xb0/0x180
[144870.872361][  T172]  worker_thread+0x601/0xff0
[144870.873609][  T172]  ? __pfx_worker_thread+0x10/0x10
[144870.874813][  T172]  kthread+0x361/0x460
[144870.875873][  T172]  ? __pfx_kthread+0x10/0x10
[144870.876881][  T172]  ret_from_fork+0x560/0x860
[144870.878012][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144870.879036][  T172]  ? __switch_to+0x473/0xd50
[144870.880200][  T172]  ? __pfx_kthread+0x10/0x10
[144870.881232][  T172]  ret_from_fork_asm+0x1a/0x30
[144870.882353][  T172]  </TASK>
[144870.883486][  T172] INFO: task kworker/10:173:3785794 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144870.885498][  T172] INFO: task kworker/18:2:3812136 blocked for more than 122 seconds.
[144870.887038][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144870.888467][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144870.890050][  T172] task:kworker/18:2    state:D stack:0     pid:3812136 tgid:3812136 ppid:2      task_flags:0x4248060 flags:0x00080000
[144870.892033][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144870.893266][  T172] Call Trace:
[144870.894349][  T172]  <TASK>
[144870.895228][  T172]  __schedule+0xc86/0x1b10
[144870.896439][  T172]  ? __pfx___schedule+0x10/0x10
[144870.897500][  T172]  ? schedule+0x17e/0x2e0
[144870.898441][  T172]  ? wq_worker_sleeping+0x65/0x280
[144870.899427][  T172]  ? find_held_lock+0x2b/0x80
[144870.900392][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144870.901382][  T172]  schedule+0xe5/0x2e0
[144870.902249][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.903287][  T172]  schedule_preempt_disabled+0x16/0x30
[144870.904511][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144870.905518][  T172]  ? lock_release.part.0+0x1c/0x50
[144870.906482][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144870.907575][  T172]  ? __lock_acquire+0x691/0xd00
[144870.908506][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.939885][  T172]  ? lock_acquire+0x11a/0x140
[144870.940681][  T172]  down_write_nested+0x1c0/0x1f0
[144870.941553][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144870.942530][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144870.944492][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144870.946365][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144870.948153][  T172]  ? __lock_acquire+0x691/0xd00
[144870.949019][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144870.950793][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144870.952653][  T172]  iomap_dio_complete+0x140/0xb20
[144870.953439][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144870.954096][  T172]  iomap_dio_complete_work+0x58/0x90
[144870.954877][  T172]  process_one_work+0x947/0x1760
[144870.955644][  T172]  ? __pfx_process_one_work+0x10/0x10
[144870.956399][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.957121][  T172]  ? lock_is_held_type+0xb0/0x180
[144870.957949][  T172]  worker_thread+0x601/0xff0
[144870.958860][  T172]  ? __pfx_worker_thread+0x10/0x10
[144870.959586][  T172]  kthread+0x361/0x460
[144870.960273][  T172]  ? __pfx_kthread+0x10/0x10
[144870.960891][  T172]  ret_from_fork+0x560/0x860
[144870.961610][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144870.962503][  T172]  ? __switch_to+0x473/0xd50
[144870.963250][  T172]  ? __pfx_kthread+0x10/0x10
[144870.963930][  T172]  ret_from_fork_asm+0x1a/0x30
[144870.964760][  T172]  </TASK>
[144870.965444][  T172] INFO: task kworker/18:2:3812136 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144870.966764][  T172] INFO: task kworker/20:1:3814407 blocked for more than 123 seconds.
[144870.967875][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144870.968920][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144870.970043][  T172] task:kworker/20:1    state:D stack:0     pid:3814407 tgid:3814407 ppid:2      task_flags:0x4248060 flags:0x00080000
[144870.971395][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144870.972198][  T172] Call Trace:
[144870.972772][  T172]  <TASK>
[144870.973279][  T172]  __schedule+0xc86/0x1b10
[144870.973888][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144870.974596][  T172]  ? __pfx___schedule+0x10/0x10
[144870.975254][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.975899][  T172]  ? wq_worker_sleeping+0x213/0x280
[144870.976689][  T172]  ? find_held_lock+0x2b/0x80
[144870.977241][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144870.977916][  T172]  schedule+0xe5/0x2e0
[144870.978586][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.979182][  T172]  schedule_preempt_disabled+0x16/0x30
[144870.979886][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144870.980635][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144870.981287][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144870.982017][  T172]  ? __lock_acquire+0x691/0xd00
[144870.982700][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.983391][  T172]  ? lock_acquire+0x11a/0x140
[144870.984047][  T172]  down_write_nested+0x1c0/0x1f0
[144870.984778][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144870.985569][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144870.987084][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144870.988587][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144870.990093][  T172]  ? lock_is_held_type+0xb0/0x180
[144870.990804][  T172]  ? __lock_acquire+0x691/0xd00
[144870.991435][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144870.992815][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144870.994256][  T172]  iomap_dio_complete+0x140/0xb20
[144870.994897][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144870.995533][  T172]  iomap_dio_complete_work+0x58/0x90
[144870.996229][  T172]  process_one_work+0x947/0x1760
[144870.996919][  T172]  ? __pfx_process_one_work+0x10/0x10
[144870.997519][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.998095][  T172]  ? lock_is_held_type+0xb0/0x180
[144870.998755][  T172]  worker_thread+0x601/0xff0
[144870.999262][  T172]  ? __pfx_worker_thread+0x10/0x10
[144870.999854][  T172]  kthread+0x361/0x460
[144871.000422][  T172]  ? __pfx_kthread+0x10/0x10
[144871.000971][  T172]  ret_from_fork+0x560/0x860
[144871.001596][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.002150][  T172]  ? __switch_to+0x473/0xd50
[144871.002764][  T172]  ? __pfx_kthread+0x10/0x10
[144871.003268][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.003871][  T172]  </TASK>
[144871.006437][  T172] INFO: task kworker/20:1:3814407 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.007488][  T172] INFO: task kworker/16:2:3814415 blocked for more than 123 seconds.
[144871.008373][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.009098][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.009935][  T172] task:kworker/16:2    state:D stack:0     pid:3814415 tgid:3814415 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.010908][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.011529][  T172] Call Trace:
[144871.011921][  T172]  <TASK>
[144871.012338][  T172]  __schedule+0xc86/0x1b10
[144871.012801][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144871.013341][  T172]  ? __pfx___schedule+0x10/0x10
[144871.013912][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.014491][  T172]  ? wq_worker_sleeping+0x213/0x280
[144871.015122][  T172]  ? find_held_lock+0x2b/0x80
[144871.015734][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.016274][  T172]  schedule+0xe5/0x2e0
[144871.016820][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.017430][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.017981][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.018500][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144871.019066][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.019691][  T172]  ? __lock_acquire+0x691/0xd00
[144871.020216][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.020806][  T172]  ? lock_acquire+0x11a/0x140
[144871.021350][  T172]  down_write_nested+0x1c0/0x1f0
[144871.021847][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.022365][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.023499][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.024814][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.026064][  T172]  ? update_load_avg+0xb65/0x2c70
[144871.026726][  T172]  ? __lock_acquire+0x691/0xd00
[144871.027267][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.028538][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.029728][  T172]  iomap_dio_complete+0x140/0xb20
[144871.030224][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.030743][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.031190][  T172]  process_one_work+0x947/0x1760
[144871.031764][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.032294][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.032889][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.033502][  T172]  worker_thread+0x601/0xff0
[144871.034040][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.034640][  T172]  kthread+0x361/0x460
[144871.035071][  T172]  ? __pfx_kthread+0x10/0x10
[144871.035679][  T172]  ret_from_fork+0x560/0x860
[144871.036162][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.036758][  T172]  ? __switch_to+0x473/0xd50
[144871.037334][  T172]  ? __pfx_kthread+0x10/0x10
[144871.037914][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.038394][  T172]  </TASK>
[144871.038867][  T172] INFO: task kworker/16:2:3814415 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.039846][  T172] INFO: task kworker/1:2:3819893 blocked for more than 123 seconds.
[144871.040650][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.041422][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.042136][  T172] task:kworker/1:2     state:D stack:0     pid:3819893 tgid:3819893 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.043104][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.043712][  T172] Call Trace:
[144871.044164][  T172]  <TASK>
[144871.044651][  T172]  __schedule+0xc86/0x1b10
[144871.045172][  T172]  ? __pfx___schedule+0x10/0x10
[144871.045707][  T172]  ? schedule+0x17e/0x2e0
[144871.046192][  T172]  ? wq_worker_sleeping+0x65/0x280
[144871.046741][  T172]  ? find_held_lock+0x2b/0x80
[144871.047186][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.047796][  T172]  schedule+0xe5/0x2e0
[144871.048299][  T172]  xlog_grant_head_wait+0x175/0xac0 [xfs]
[144871.049471][  T172]  xlog_grant_head_check+0x312/0x3f0 [xfs]
[144871.050717][  T172]  ? __pfx_xlog_grant_head_check+0x10/0x10 [xfs]
[144871.051932][  T172]  xfs_log_regrant+0x380/0x7d0 [xfs]
[144871.053059][  T172]  ? __pfx_xfs_log_regrant+0x10/0x10 [xfs]
[144871.054192][  T172]  ? xfs_log_ticket_get+0x1b/0x30 [xfs]
[144871.055259][  T172]  ? xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
[144871.056414][  T172]  xfs_trans_roll+0x2d9/0x420 [xfs]
[144871.057517][  T172]  ? kmem_cache_alloc_noprof+0x45b/0x680
[144871.058100][  T172]  ? xfs_defer_finish_noroll+0x460/0xe70 [xfs]
[144871.059201][  T172]  xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
[144871.060341][  T172]  ? __pfx_xfs_defer_trans_roll+0x10/0x10 [xfs]
[144871.061364][  T172]  ? xfs_defer_relog+0x3e7/0xb50 [xfs]
[144871.062454][  T172]  xfs_defer_finish_noroll+0x460/0xe70 [xfs]
[144871.063491][  T172]  ? __pfx_xfs_reflink_end_cow_extent_locked+0x10/0x10 [xfs]
[144871.064782][  T172]  ? __pfx_xfs_defer_finish_noroll+0x10/0x10 [xfs]
[144871.065859][  T172]  ? lock_acquire+0x11a/0x140
[144871.066413][  T172]  xfs_trans_commit+0xfc/0x180 [xfs]
[144871.067509][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.068046][  T172]  ? __pfx_xfs_trans_commit+0x10/0x10 [xfs]
[144871.069143][  T172]  xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
[144871.070340][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.071419][  T172]  ? __lock_acquire+0x691/0xd00
[144871.089717][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.090837][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.091794][  T172]  iomap_dio_complete+0x140/0xb20
[144871.092194][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.092684][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.093251][  T172]  process_one_work+0x947/0x1760
[144871.093761][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.094406][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.095084][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.095679][  T172]  worker_thread+0x601/0xff0
[144871.096234][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.096820][  T172]  kthread+0x361/0x460
[144871.097360][  T172]  ? __pfx_kthread+0x10/0x10
[144871.098065][  T172]  ret_from_fork+0x560/0x860
[144871.098693][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.099175][  T172]  ? __switch_to+0x473/0xd50
[144871.099709][  T172]  ? __pfx_kthread+0x10/0x10
[144871.100275][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.100783][  T172]  </TASK>
[144871.101203][  T172] INFO: task kworker/20:2:3824669 blocked for more than 123 seconds.
[144871.102109][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.102813][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.103685][  T172] task:kworker/20:2    state:D stack:0     pid:3824669 tgid:3824669 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.104588][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.105360][  T172] Call Trace:
[144871.105868][  T172]  <TASK>
[144871.106339][  T172]  __schedule+0xc86/0x1b10
[144871.106949][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144871.107632][  T172]  ? __pfx___schedule+0x10/0x10
[144871.108170][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.108600][  T172]  ? wq_worker_sleeping+0x213/0x280
[144871.109207][  T172]  ? find_held_lock+0x2b/0x80
[144871.109728][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.110270][  T172]  schedule+0xe5/0x2e0
[144871.110762][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.111328][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.111898][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.112410][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144871.112989][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.113648][  T172]  ? __lock_acquire+0x691/0xd00
[144871.114157][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.114665][  T172]  ? lock_acquire+0x11a/0x140
[144871.115101][  T172]  down_write_nested+0x1c0/0x1f0
[144871.115664][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.116122][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.117105][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.118239][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.119389][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.119781][  T172]  ? __lock_acquire+0x691/0xd00
[144871.120177][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.121294][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.122391][  T172]  iomap_dio_complete+0x140/0xb20
[144871.122770][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.123425][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.124051][  T172]  process_one_work+0x947/0x1760
[144871.124445][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.125017][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.125580][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.126138][  T172]  worker_thread+0x601/0xff0
[144871.126680][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.127126][  T172]  kthread+0x361/0x460
[144871.127762][  T172]  ? __pfx_kthread+0x10/0x10
[144871.128234][  T172]  ret_from_fork+0x560/0x860
[144871.128772][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.129337][  T172]  ? __switch_to+0x473/0xd50
[144871.129842][  T172]  ? __pfx_kthread+0x10/0x10
[144871.130364][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.130892][  T172]  </TASK>
[144871.131250][  T172] INFO: task kworker/20:2:3824669 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.132261][  T172] INFO: task kworker/4:0:3824816 blocked for more than 123 seconds.
[144871.133177][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.133912][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.134827][  T172] task:kworker/4:0     state:D stack:0     pid:3824816 tgid:3824816 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.135724][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.136356][  T172] Call Trace:
[144871.136856][  T172]  <TASK>
[144871.137342][  T172]  __schedule+0xc86/0x1b10
[144871.137818][  T172]  ? __pfx___schedule+0x10/0x10
[144871.138410][  T172]  ? schedule+0x17e/0x2e0
[144871.138810][  T172]  ? wq_worker_sleeping+0x65/0x280
[144871.139356][  T172]  ? find_held_lock+0x2b/0x80
[144871.139817][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.140631][  T172]  schedule+0xe5/0x2e0
[144871.141220][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.141887][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.142473][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.143026][  T172]  ? lock_release.part.0+0x1c/0x50
[144871.143557][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.144101][  T172]  ? __lock_acquire+0x691/0xd00
[144871.144681][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.145057][  T172]  ? lock_acquire+0x11a/0x140
[144871.145709][  T172]  down_write_nested+0x1c0/0x1f0
[144871.146205][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.146885][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.147929][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.148978][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.150061][  T172]  ? __lock_acquire+0x691/0xd00
[144871.150641][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.151842][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.152998][  T172]  iomap_dio_complete+0x140/0xb20
[144871.153548][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.154099][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.154759][  T172]  process_one_work+0x947/0x1760
[144871.155210][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.155675][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.156336][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.156920][  T172]  worker_thread+0x601/0xff0
[144871.157437][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.157951][  T172]  kthread+0x361/0x460
[144871.158466][  T172]  ? __pfx_kthread+0x10/0x10
[144871.159012][  T172]  ret_from_fork+0x560/0x860
[144871.159502][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.160015][  T172]  ? __switch_to+0x473/0xd50
[144871.160539][  T172]  ? __pfx_kthread+0x10/0x10
[144871.161051][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.161685][  T172]  </TASK>
[144871.162077][  T172] INFO: task kworker/4:0:3824816 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.162900][  T172] INFO: task kworker/12:0:3826849 blocked for more than 123 seconds.
[144871.163697][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.164415][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.165277][  T172] task:kworker/12:0    state:D stack:0     pid:3826849 tgid:3826849 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.166221][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.166952][  T172] Call Trace:
[144871.167283][  T172]  <TASK>
[144871.167853][  T172]  __schedule+0xc86/0x1b10
[144871.168439][  T172]  ? __pfx___schedule+0x10/0x10
[144871.168914][  T172]  ? schedule+0x17e/0x2e0
[144871.169451][  T172]  ? wq_worker_sleeping+0x65/0x280
[144871.169927][  T172]  ? find_held_lock+0x2b/0x80
[144871.170288][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.170911][  T172]  schedule+0xe5/0x2e0
[144871.171387][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.171939][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.172496][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.173087][  T172]  ? lock_release.part.0+0x1c/0x50
[144871.173702][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.174226][  T172]  ? __lock_acquire+0x691/0xd00
[144871.174891][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.175466][  T172]  ? lock_acquire+0x11a/0x140
[144871.176164][  T172]  down_write_nested+0x1c0/0x1f0
[144871.176754][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.177531][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.178526][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.179662][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.180880][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.181342][  T172]  ? __lock_acquire+0x691/0xd00
[144871.181872][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.183071][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.184221][  T172]  iomap_dio_complete+0x140/0xb20
[144871.184745][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.185338][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.185864][  T172]  process_one_work+0x947/0x1760
[144871.186361][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.186920][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.187381][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.187924][  T172]  worker_thread+0x601/0xff0
[144871.188591][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.189109][  T172]  kthread+0x361/0x460
[144871.189647][  T172]  ? __pfx_kthread+0x10/0x10
[144871.190131][  T172]  ret_from_fork+0x560/0x860
[144871.190578][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.191172][  T172]  ? __switch_to+0x473/0xd50
[144871.191771][  T172]  ? __pfx_kthread+0x10/0x10
[144871.192247][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.192760][  T172]  </TASK>
[144871.193146][  T172] INFO: task kworker/12:0:3826849 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.202516][  T172] INFO: task kworker/21:1:3827095 blocked for more than 123 seconds.
[144871.203182][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.203953][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.204755][  T172] task:kworker/21:1    state:D stack:0     pid:3827095 tgid:3827095 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.205642][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.206251][  T172] Call Trace:
[144871.206793][  T172]  <TASK>
[144871.207382][  T172]  __schedule+0xc86/0x1b10
[144871.207815][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144871.208402][  T172]  ? __pfx___schedule+0x10/0x10
[144871.208871][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.209404][  T172]  ? wq_worker_sleeping+0x213/0x280
[144871.209925][  T172]  ? find_held_lock+0x2b/0x80
[144871.210492][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.211050][  T172]  schedule+0xe5/0x2e0
[144871.211527][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.212043][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.212786][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.213398][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144871.213975][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.214563][  T172]  ? __lock_acquire+0x691/0xd00
[144871.215009][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.215710][  T172]  ? lock_acquire+0x11a/0x140
[144871.216164][  T172]  down_write_nested+0x1c0/0x1f0
[144871.216688][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.217212][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.218301][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.219411][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.220508][  T172]  ? __lock_acquire+0x691/0xd00
[144871.221057][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.222191][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.223292][  T172]  iomap_dio_complete+0x140/0xb20
[144871.223820][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.224332][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.224792][  T172]  process_one_work+0x947/0x1760
[144871.225354][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.225861][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.226458][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.226982][  T172]  worker_thread+0x601/0xff0
[144871.227454][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.227993][  T172]  ? __kthread_parkme+0xbd/0x210
[144871.228532][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.229057][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.229619][  T172]  kthread+0x361/0x460
[144871.230083][  T172]  ? __pfx_kthread+0x10/0x10
[144871.230578][  T172]  ret_from_fork+0x560/0x860
[144871.231049][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.231633][  T172]  ? __switch_to+0x473/0xd50
[144871.232124][  T172]  ? __pfx_kthread+0x10/0x10
[144871.232544][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.233081][  T172]  </TASK>
[144871.233440][  T172] INFO: task kworker/21:1:3827095 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.234328][  T172] INFO: task kworker/10:0:3827669 blocked for more than 123 seconds.
[144871.235091][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.235858][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.236582][  T172] task:kworker/10:0    state:D stack:0     pid:3827669 tgid:3827669 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.237651][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.238173][  T172] Call Trace:
[144871.238946][  T172]  <TASK>
[144871.239469][  T172]  __schedule+0xc86/0x1b10
[144871.240084][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144871.240642][  T172]  ? __pfx___schedule+0x10/0x10
[144871.241088][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.241694][  T172]  ? wq_worker_sleeping+0x213/0x280
[144871.242326][  T172]  ? find_held_lock+0x2b/0x80
[144871.242813][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.243379][  T172]  schedule+0xe5/0x2e0
[144871.243881][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.244378][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.244920][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.245543][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144871.246066][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.246662][  T172]  ? __lock_acquire+0x691/0xd00
[144871.247131][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.247711][  T172]  ? lock_acquire+0x11a/0x140
[144871.248149][  T172]  down_write_nested+0x1c0/0x1f0
[144871.248724][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.249285][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.250297][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.251346][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.252414][  T172]  ? __lock_acquire+0x691/0xd00
[144871.252983][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.254104][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.255189][  T172]  iomap_dio_complete+0x140/0xb20
[144871.255756][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.256283][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.256814][  T172]  process_one_work+0x947/0x1760
[144871.257211][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.257962][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.258415][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.258907][  T172]  worker_thread+0x601/0xff0
[144871.259454][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.259949][  T172]  kthread+0x361/0x460
[144871.260364][  T172]  ? __pfx_kthread+0x10/0x10
[144871.260889][  T172]  ret_from_fork+0x560/0x860
[144871.261384][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.261926][  T172]  ? __switch_to+0x473/0xd50
[144871.262448][  T172]  ? __pfx_kthread+0x10/0x10
[144871.262968][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.263492][  T172]  </TASK>
[144871.263903][  T172] INFO: task kworker/10:0:3827669 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.264856][  T172] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
[144871.265760][  T172] 
[144871.265760][  T172] Showing all locks held in the system:
[144871.266766][  T172] locks held by pr/ttyS0/16: 2, last CPU#22:
[144871.267407][  T172]  #0: ffffffff8c320658 (console_srcu){....}-{0:0}, at: console_srcu_read_lock+0x49/0x70
[144871.268258][  T172]  #1: ffffffff9c3db638 (&port_lock_key){-.-.}-{3:3}, at: univ8250_console_device_lock+0x63/0xc0
[144871.269176][  T172] locks held by khungtaskd/172: 1, last CPU#20:
[144871.269752][  T172]  #0: ffffffff8c3369e0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[144871.270692][  T172] locks held by kworker/10:173/3785794: 4, on CPU#10:
[144871.271375][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.272341][  T172]  #1: ff11000109407d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.273284][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.274707][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.276050][  T172] locks held by kworker/18:2/3812136: 4, on CPU#18:
[144871.276811][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.277696][  T172]  #1: ff1100018e71fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.278634][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.280090][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.281626][  T172] locks held by kworker/20:1/3814407: 4, on CPU#20:
[144871.282267][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.283196][  T172]  #1: ff11000214267d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.284154][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.285620][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.287063][  T172] locks held by kworker/16:2/3814415: 4, on CPU#16:
[144871.287789][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.288637][  T172]  #1: ff1100019cc67d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.289792][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.291054][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.292462][  T172] locks held by kworker/1:2/3819893: 4, on CPU#1:
[144871.293085][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.294070][  T172]  #1: ff110001246d7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.294991][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.296348][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.297789][  T172] locks held by kworker/20:2/3824669: 4, on CPU#20:
[144871.298349][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.299205][  T172]  #1: ff1100022fc77d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.300225][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.301638][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.303077][  T172] locks held by kworker/4:0/3824816: 4, on CPU#4:
[144871.303744][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.312636][  T172]  #1: ff11000424997d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.313372][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.314586][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.315921][  T172] locks held by kworker/12:0/3826849: 4, on CPU#12:
[144871.316445][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.317222][  T172]  #1: ff110001e71bfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.318124][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.319340][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.320712][  T172] locks held by kworker/21:1/3827095: 4, on CPU#21:
[144871.321339][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.322330][  T172]  #1: ff11000172fefd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.323140][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.324642][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.326042][  T172] locks held by kworker/10:0/3827669: 4, on CPU#10:
[144871.326655][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.327575][  T172]  #1: ff11000122177d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.328366][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.329889][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.331476][  T172] locks held by kworker/17:1/3827670: 4, on CPU#17:
[144871.332105][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.332897][  T172]  #1: ff110001d0307d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.333703][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.334981][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.336354][  T172] locks held by kworker/16:1/3864132: 4, on CPU#16:
[144871.337033][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.337973][  T172]  #1: ff1100023211fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.338641][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.339966][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.341388][  T172] locks held by kworker/8:1/3868486: 4, on CPU#8:
[144871.342196][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.343143][  T172]  #1: ff110001a24d7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.344179][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.345576][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.347006][  T172] locks held by kworker/10:1/3872909: 4, on CPU#10:
[144871.347626][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.348518][  T172]  #1: ff110001606dfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.349516][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.350857][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.352433][  T172] locks held by kworker/17:2/3872910: 4, on CPU#17:
[144871.353061][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.354161][  T172]  #1: ff1100041c1b7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.354964][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.356302][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.357755][  T172] locks held by kworker/23:0/3872911: 4, on CPU#23:
[144871.358527][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.359329][  T172]  #1: ff110004b8c8fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.360223][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.361505][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.363110][  T172] locks held by kworker/7:3/3872919: 4, on CPU#7:
[144871.363803][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.364639][  T172]  #1: ff1100019af97d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.365488][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.366938][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.368364][  T172] locks held by kworker/20:3/3872924: 4, on CPU#20:
[144871.369008][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.369969][  T172]  #1: ff110001aa42fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.371118][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.372518][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.373983][  T172] locks held by kworker/3:2/3872925: 4, on CPU#3:
[144871.374629][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.375515][  T172]  #1: ff11000444b8fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.376358][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.377586][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.378950][  T172] locks held by kworker/16:3/3872926: 4, on CPU#16:
[144871.379685][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.380610][  T172]  #1: ff11000682cdfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.381567][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.382968][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.384263][  T172] locks held by kworker/17:3/3872927: 4, on CPU#17:
[144871.384833][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.385697][  T172]  #1: ff110001d3ca7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.386411][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.387778][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.389139][  T172] locks held by kworker/16:4/3872929: 4, on CPU#16:
[144871.389779][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.390582][  T172]  #1: ff110003cb7efd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.391402][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.392568][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.394019][  T172] locks held by kworker/19:2/3872930: 4, on CPU#19:
[144871.394685][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.395449][  T172]  #1: ff1100021374fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.396394][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.397664][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.399332][  T172] locks held by kworker/18:3/3872935: 3, on CPU#18:
[144871.400131][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.401110][  T172]  #1: ff110002862b7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.402243][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.403502][  T172] locks held by kworker/10:2/3872936: 4, on CPU#10:
[144871.404186][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.405139][  T172]  #1: ff110003d8e8fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.406011][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.407341][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.408729][  T172] locks held by kworker/8:3/3872941: 3, on CPU#8:
[144871.409141][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.410067][  T172]  #1: ff110003df1e7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.410799][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.411989][  T172] locks held by kworker/1:3/3872953: 4, on CPU#1:
[144871.412584][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.413531][  T172]  #1: ff1100043f1bfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.414464][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.424444][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.425808][  T172] locks held by kworker/4:4/3872955: 4, on CPU#4:
[144871.426603][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.427468][  T172]  #1: ff1100014de77d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.428336][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.429615][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.430975][  T172] locks held by fio/3872961: 3, on CPU#2:
[144871.431620][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.433089][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.434279][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.435797][  T172] locks held by fio/3872962: 3, on CPU#23:
[144871.436291][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.437618][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.438845][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.440124][  T172] locks held by fio/3872963: 3, on CPU#23:
[144871.440772][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.442297][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.443694][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.445025][  T172] locks held by fio/3872964: 3, on CPU#14:
[144871.445514][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.446890][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.448097][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.449466][  T172] locks held by fio/3872965: 3, on CPU#12:
[144871.450120][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.451359][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.452681][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.454164][  T172] locks held by fio/3872966: 3, on CPU#16:
[144871.454720][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.455979][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.457331][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.458737][  T172] locks held by fio/3872967: 3, on CPU#4:
[144871.459369][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.460809][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.461990][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.463126][  T172] locks held by fio/3872968: 3, on CPU#1:
[144871.463662][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.464985][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.466188][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.467438][  T172] locks held by fio/3872969: 3, on CPU#6:
[144871.468111][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.469612][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.471012][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.472352][  T172] locks held by fio/3872970: 3, on CPU#4:
[144871.473018][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.474395][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.475759][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.477331][  T172] locks held by fio/3872971: 3, on CPU#14:
[144871.477764][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.479087][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.480334][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.481548][  T172] locks held by fio/3872972: 2, on CPU#10:
[144871.482113][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.483336][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.484625][  T172] locks held by fio/3872973: 3, on CPU#10:
[144871.485350][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.486616][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.487949][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.489152][  T172] locks held by fio/3872974: 2, on CPU#18:
[144871.489572][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.491050][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.492416][  T172] locks held by fio/3872975: 3, on CPU#8:
[144871.493166][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.494401][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.495553][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.496938][  T172] locks held by fio/3872976: 3, on CPU#23:
[144871.497422][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.498613][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.499903][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.501249][  T172] locks held by fio/3872977: 3, on CPU#1:
[144871.501945][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.503238][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.504494][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.505963][  T172] locks held by fio/3872978: 3, on CPU#12:
[144871.506558][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.508041][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.509561][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.510975][  T172] locks held by fio/3872979: 3, on CPU#4:
[144871.511616][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.512986][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.514342][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.515561][  T172] locks held by fio/3872980: 2, on CPU#2:
[144871.516337][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.517588][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.518941][  T172] locks held by fio/3872981: 3, on CPU#5:
[144871.519544][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.520980][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.522436][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.523761][  T172] locks held by fio/3872982: 3, on CPU#7:
[144871.524345][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.525617][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.526995][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.528345][  T172] locks held by fio/3872983: 2, on CPU#18:
[144871.529013][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.530159][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.531658][  T172] locks held by fio/3872984: 3, on CPU#21:
[144871.532344][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.533662][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.535127][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.536354][  T172] locks held by fio/3872985: 3, on CPU#14:
[144871.536893][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.538193][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.539344][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.540565][  T172] locks held by fio/3872986: 3, on CPU#23:
[144871.549064][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.550342][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.551412][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.552644][  T172] locks held by fio/3872987: 3, on CPU#9:
[144871.553252][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.554617][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.555963][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.557289][  T172] locks held by fio/3872988: 2, on CPU#9:
[144871.557983][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.559251][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.560650][  T172] locks held by fio/3872989: 3, on CPU#13:
[144871.561148][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.562620][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.563966][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.565353][  T172] locks held by fio/3872990: 3, on CPU#22:
[144871.566014][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.567292][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.568639][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.569912][  T172] locks held by fio/3872991: 3, on CPU#15:
[144871.570643][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.572074][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.573526][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.574865][  T172] locks held by fio/3872992: 3, on CPU#19:
[144871.575385][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.576702][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.577974][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.579176][  T172] locks held by fio/3872993: 2, on CPU#8:
[144871.579776][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.581296][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.582823][  T172] locks held by fio/3872994: 2, on CPU#10:
[144871.583327][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.584824][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.586338][  T172] locks held by fio/3872995: 2, on CPU#0:
[144871.586746][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.588065][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.589592][  T172] locks held by fio/3872996: 3, on CPU#3:
[144871.590228][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.591624][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.592955][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.594411][  T172] locks held by fio/3872997: 2, on CPU#9:
[144871.594880][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.596102][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.597443][  T172] locks held by fio/3872998: 3, on CPU#11:
[144871.598214][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.599581][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.600877][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.602277][  T172] locks held by fio/3872999: 2, on CPU#15:
[144871.602933][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.604171][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.605502][  T172] locks held by fio/3873000: 2, on CPU#13:
[144871.606086][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.607446][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.608891][  T172] locks held by fio/3873001: 2, on CPU#20:
[144871.609629][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.610941][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.612536][  T172] locks held by fio/3873002: 2, on CPU#21:
[144871.613261][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.614548][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.615849][  T172] locks held by fio/3873003: 2, on CPU#7:
[144871.616466][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.617936][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.619512][  T172] locks held by fio/3873004: 3, on CPU#12:
[144871.620084][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.621559][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.623035][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.624371][  T172] locks held by fio/3873005: 3, on CPU#2:
[144871.624968][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.626326][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.627504][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.628825][  T172] locks held by fio/3873006: 2, on CPU#5:
[144871.629555][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.630924][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.632295][  T172] locks held by fio/3873007: 3, on CPU#22:
[144871.632986][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.634247][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.635569][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.636907][  T172] locks held by fio/3873008: 3, on CPU#16:
[144871.637391][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.638706][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.639974][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.641349][  T172] locks held by kworker/22:9/3873014: 4, on CPU#22:
[144871.641954][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.643046][  T172]  #1: ff1100075a3b7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.643896][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.645235][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.646870][  T172] locks held by kworker/10:3/3873017: 4, on CPU#10:
[144871.647514][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.648341][  T172]  #1: ff110002411cfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.649336][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.650672][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.652162][  T172] locks held by kworker/16:5/3873018: 4, on CPU#16:
[144871.652872][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.653694][  T172]  #1: ff1100019cfd7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.654690][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.656010][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.657334][  T172] locks held by kworker/16:6/3873019: 4, on CPU#16:
[144871.657807][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.658834][  T172]  #1: ff110005ec73fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.659987][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.661147][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.662392][  T172] locks held by kworker/16:7/3873020: 4, on CPU#16:
[144871.662945][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.663830][  T172]  #1: ff110004e58a7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.664843][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.674237][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.675488][  T172] locks held by kworker/16:8/3873021: 4, on CPU#16:
[144871.676270][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.676948][  T172]  #1: ff110001e6617d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.677872][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.679079][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.680423][  T172] locks held by kworker/16:9/3873022: 4, on CPU#16:
[144871.680913][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.681864][  T172]  #1: ff11000156bbfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.682805][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.684009][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.685395][  T172] locks held by kworker/16:10/3873023: 4, on CPU#16:
[144871.685974][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.686863][  T172]  #1: ff1100040bf57d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.687686][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.689001][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.690449][  T172] locks held by kworker/17:5/3873025: 4, on CPU#17:
[144871.691061][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.691966][  T172]  #1: ff1100013957fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.692935][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.694197][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.695674][  T172] locks held by kworker/12:3/3873030: 4, on CPU#12:
[144871.696301][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.697143][  T172]  #1: ff110007b3e6fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.698099][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.699455][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.700840][  T172] 
[144871.701294][  T172] =============================================

Re: [bug report] fstests generic/774 hang again

[John Garry]

On 9/15/26 10:34, Shin'ichiro Kawasaki wrote:
> I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB
> TCMU fileio devices. Actually I once reported this hang symptom in last October,
> and Darrick and John kindly took some actions [1]. Since then, the hang
> disappeared and I have not observed the hang almost one year. However, my test
> system started reporting the hang again last week.
> 
> [1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t

This was the series to resolve, I think:
https://lore.kernel.org/fstests/176279908967.605950.2192923313361120314.stgit@frogsfrogsfrogs/T/#t

As an experiment, can you reduce the file size further, like:

--- a/tests/generic/774
+++ b/tests/generic/774
@@ -29,7 +29,7 @@ aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))")
  fsbsize=$(_get_block_size $SCRATCH_MNT)

  threads=$(_min "$(($(nproc) * 2 * LOAD_FACTOR))" "100")
-filesize=$((aw_bsize * threads * 10))
+filesize=$((aw_bsize * threads))
  depth=$threads
  aw_io_size=$((filesize / threads))
  aw_io_inc=$aw_io_size


> 
> FYI, here I attache the kernel message [2]. 

Is mainline baseline hanging, i.e. v7.3-rc2?

> I observed the hang last week with
> the kernel on the xfs/for-next branch at the commit 0ca15a1a1151 ("xfs: remove
> an extra cast in xfs_file_compat_ioctl"). Today, I tried again with the kernel
> at current xfs/for-next branch tip 4266ffdfd7cb ("xfs: remove an extra cast in
> xfs_file_compat_ioctl"), and observed the hang again. The hang can be recreated
> in stable manner by repeating the test case 20 times or so.
> 
> I also tried other test cases in atmoicwrites group, and observed no failure.
> 
>   generic/765        [not run] write atomic not supported by this block device
>   generic/767         11s
>   generic/768         13s
>   generic/769         13s
>   generic/770         32s
>   generic/773        [not run] write atomic not supported by this block device
>   generic/774        (skipped)
>   generic/775         293s
>   generic/776        [notrun] write atomic not supported by this block device
>   generic/778         48s
>   xfs/838            [not run] External volumes not in use, skipped this test
>   xfs/839            [not run] XFS error injection requires CONFIG_XFS_DEBUG
>   xfs/840            [not run] write atomic not supported by this block device
> 
> Action for fix will be appreciated. If I can do anything on my test node, please
> let me know.
> 
> 
> [2] kernel message during generic/774
> 
> [144539.456849][T1634092] run fstests generic/774 at 2026-09-09 16:51:53
> [144539.691199][T3820710] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
> [144543.436776][T3868062] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
> [144543.648458][T3872722] XFS (sdg): Mounting V5 Filesystem 306fb2c4-6cd3-4cbd-b9ed-953c0bc0d3b8
> [144543.674712][T3872722] XFS (sdg): Ending clean mount
> [144544.328859][T3872774] XFS (sdg): Unmounting Filesystem 306fb2c4-6cd3-4cbd-b9ed-953c0bc0d3b8
> [144545.743530][T3795425] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
> [144545.964812][T3872823] XFS (sdg): Mounting V5 Filesystem 91ee7431-7139-4d92-bea7-b7802c3f70d2
> [144545.990603][T3872823] XFS (sdg): Ending clean mount
> [144870.819650][  T172] INFO: task kworker/10:173:3785794 blocked for more than 122 seconds.
> [144870.821440][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
> [144870.823252][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [144870.825054][  T172] task:kworker/10:173  state:D stack:0     pid:3785794 tgid:3785794 ppid:2      task_flags:0x4248060 flags:0x00080000
> [144870.827098][  T172] Workqueue: dio/sdg iomap_dio_complete_work
> [144870.828170][  T172] Call Trace:
> [144870.829209][  T172]  <TASK>
> [144870.830154][  T172]  __schedule+0xc86/0x1b10
> [144870.831126][  T172]  ? trace_hardirqs_on+0x18/0x1a0
> [144870.832166][  T172]  ? __pfx___schedule+0x10/0x10
> [144870.833250][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144870.834400][  T172]  ? wq_worker_sleeping+0x213/0x280
> [144870.835559][  T172]  ? find_held_lock+0x2b/0x80
> [144870.836634][  T172]  ? __lock_release.isra.0+0x69/0x1a0
> [144870.837699][  T172]  schedule+0xe5/0x2e0
> [144870.838721][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144870.839915][  T172]  schedule_preempt_disabled+0x16/0x30
> [144870.841017][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
> [144870.842139][  T172]  ? __kasan_slab_alloc+0x7f/0x90
> [144870.843236][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
> [144870.844383][  T172]  ? __lock_acquire+0x691/0xd00
> [144870.845598][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144870.846840][  T172]  ? lock_acquire+0x11a/0x140
> [144870.848019][  T172]  down_write_nested+0x1c0/0x1f0
> [144870.849121][  T172]  ? __pfx_down_write_nested+0x10/0x10
> [144870.850284][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
> [144870.852895][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144870.855451][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
> [144870.858004][  T172]  ? __lock_acquire+0x691/0xd00
> [144870.859160][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> [144870.861775][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
> [144870.864257][  T172]  iomap_dio_complete+0x140/0xb20
> [144870.865378][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
> [144870.866570][  T172]  iomap_dio_complete_work+0x58/0x90
> [144870.867785][  T172]  process_one_work+0x947/0x1760
> [144870.868995][  T172]  ? __pfx_process_one_work+0x10/0x10
> [144870.870062][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144870.871227][  T172]  ? lock_is_held_type+0xb0/0x180
> [144870.872361][  T172]  worker_thread+0x601/0xff0
> [144870.873609][  T172]  ? __pfx_worker_thread+0x10/0x10
> [144870.874813][  T172]  kthread+0x361/0x460
> [144870.875873][  T172]  ? __pfx_kthread+0x10/0x10
> [144870.876881][  T172]  ret_from_fork+0x560/0x860
> [144870.878012][  T172]  ? __pfx_ret_from_fork+0x10/0x10
> [144870.879036][  T172]  ? __switch_to+0x473/0xd50
> [144870.880200][  T172]  ? __pfx_kthread+0x10/0x10
> [144870.881232][  T172]  ret_from_fork_asm+0x1a/0x30
> [144870.882353][  T172]  </TASK>
> [144870.883486][  T172] INFO: task kworker/10:173:3785794 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
> [144870.885498][  T172] INFO: task kworker/18:2:3812136 blocked for more than 122 seconds.
> [144870.887038][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
> [144870.888467][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [144870.890050][  T172] task:kworker/18:2    state:D stack:0     pid:3812136 tgid:3812136 ppid:2      task_flags:0x4248060 flags:0x00080000
> [144870.892033][  T172] Workqueue: dio/sdg iomap_dio_complete_work
> [144870.893266][  T172] Call Trace:
> [144870.894349][  T172]  <TASK>
> [144870.895228][  T172]  __schedule+0xc86/0x1b10
> [144870.896439][  T172]  ? __pfx___schedule+0x10/0x10
> [144870.897500][  T172]  ? schedule+0x17e/0x2e0
> [144870.898441][  T172]  ? wq_worker_sleeping+0x65/0x280
> [144870.899427][  T172]  ? find_held_lock+0x2b/0x80
> [144870.900392][  T172]  ? __lock_release.isra.0+0x69/0x1a0
> [144870.901382][  T172]  schedule+0xe5/0x2e0
> [144870.902249][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144870.903287][  T172]  schedule_preempt_disabled+0x16/0x30
> [144870.904511][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
> [144870.905518][  T172]  ? lock_release.part.0+0x1c/0x50
> [144870.906482][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
> [144870.907575][  T172]  ? __lock_acquire+0x691/0xd00
> [144870.908506][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144870.939885][  T172]  ? lock_acquire+0x11a/0x140
> [144870.940681][  T172]  down_write_nested+0x1c0/0x1f0
> [144870.941553][  T172]  ? __pfx_down_write_nested+0x10/0x10
> [144870.942530][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
> [144870.944492][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144870.946365][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
> [144870.948153][  T172]  ? __lock_acquire+0x691/0xd00
> [144870.949019][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> [144870.950793][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
> [144870.952653][  T172]  iomap_dio_complete+0x140/0xb20
> [144870.953439][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
> [144870.954096][  T172]  iomap_dio_complete_work+0x58/0x90
> [144870.954877][  T172]  process_one_work+0x947/0x1760
> [144870.955644][  T172]  ? __pfx_process_one_work+0x10/0x10
> [144870.956399][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144870.957121][  T172]  ? lock_is_held_type+0xb0/0x180
> [144870.957949][  T172]  worker_thread+0x601/0xff0
> [144870.958860][  T172]  ? __pfx_worker_thread+0x10/0x10
> [144870.959586][  T172]  kthread+0x361/0x460
> [144870.960273][  T172]  ? __pfx_kthread+0x10/0x10
> [144870.960891][  T172]  ret_from_fork+0x560/0x860
> [144870.961610][  T172]  ? __pfx_ret_from_fork+0x10/0x10
> [144870.962503][  T172]  ? __switch_to+0x473/0xd50
> [144870.963250][  T172]  ? __pfx_kthread+0x10/0x10
> [144870.963930][  T172]  ret_from_fork_asm+0x1a/0x30
> [144870.964760][  T172]  </TASK>
> [144870.965444][  T172] INFO: task kworker/18:2:3812136 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
> [144870.966764][  T172] INFO: task kworker/20:1:3814407 blocked for more than 123 seconds.
> [144870.967875][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
> [144870.968920][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [144870.970043][  T172] task:kworker/20:1    state:D stack:0     pid:3814407 tgid:3814407 ppid:2      task_flags:0x4248060 flags:0x00080000
> [144870.971395][  T172] Workqueue: dio/sdg iomap_dio_complete_work
> [144870.972198][  T172] Call Trace:
> [144870.972772][  T172]  <TASK>
> [144870.973279][  T172]  __schedule+0xc86/0x1b10
> [144870.973888][  T172]  ? trace_hardirqs_on+0x18/0x1a0
> [144870.974596][  T172]  ? __pfx___schedule+0x10/0x10
> [144870.975254][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144870.975899][  T172]  ? wq_worker_sleeping+0x213/0x280
> [144870.976689][  T172]  ? find_held_lock+0x2b/0x80
> [144870.977241][  T172]  ? __lock_release.isra.0+0x69/0x1a0
> [144870.977916][  T172]  schedule+0xe5/0x2e0
> [144870.978586][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144870.979182][  T172]  schedule_preempt_disabled+0x16/0x30
> [144870.979886][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
> [144870.980635][  T172]  ? __kasan_slab_alloc+0x7f/0x90
> [144870.981287][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
> [144870.982017][  T172]  ? __lock_acquire+0x691/0xd00
> [144870.982700][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144870.983391][  T172]  ? lock_acquire+0x11a/0x140
> [144870.984047][  T172]  down_write_nested+0x1c0/0x1f0
> [144870.984778][  T172]  ? __pfx_down_write_nested+0x10/0x10
> [144870.985569][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
> [144870.987084][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144870.988587][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
> [144870.990093][  T172]  ? lock_is_held_type+0xb0/0x180
> [144870.990804][  T172]  ? __lock_acquire+0x691/0xd00
> [144870.991435][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> [144870.992815][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
> [144870.994256][  T172]  iomap_dio_complete+0x140/0xb20
> [144870.994897][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
> [144870.995533][  T172]  iomap_dio_complete_work+0x58/0x90
> [144870.996229][  T172]  process_one_work+0x947/0x1760
> [144870.996919][  T172]  ? __pfx_process_one_work+0x10/0x10
> [144870.997519][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144870.998095][  T172]  ? lock_is_held_type+0xb0/0x180
> [144870.998755][  T172]  worker_thread+0x601/0xff0
> [144870.999262][  T172]  ? __pfx_worker_thread+0x10/0x10
> [144870.999854][  T172]  kthread+0x361/0x460
> [144871.000422][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.000971][  T172]  ret_from_fork+0x560/0x860
> [144871.001596][  T172]  ? __pfx_ret_from_fork+0x10/0x10
> [144871.002150][  T172]  ? __switch_to+0x473/0xd50
> [144871.002764][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.003268][  T172]  ret_from_fork_asm+0x1a/0x30
> [144871.003871][  T172]  </TASK>
> [144871.006437][  T172] INFO: task kworker/20:1:3814407 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
> [144871.007488][  T172] INFO: task kworker/16:2:3814415 blocked for more than 123 seconds.
> [144871.008373][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
> [144871.009098][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [144871.009935][  T172] task:kworker/16:2    state:D stack:0     pid:3814415 tgid:3814415 ppid:2      task_flags:0x4248060 flags:0x00080000
> [144871.010908][  T172] Workqueue: dio/sdg iomap_dio_complete_work
> [144871.011529][  T172] Call Trace:
> [144871.011921][  T172]  <TASK>
> [144871.012338][  T172]  __schedule+0xc86/0x1b10
> [144871.012801][  T172]  ? trace_hardirqs_on+0x18/0x1a0
> [144871.013341][  T172]  ? __pfx___schedule+0x10/0x10
> [144871.013912][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144871.014491][  T172]  ? wq_worker_sleeping+0x213/0x280
> [144871.015122][  T172]  ? find_held_lock+0x2b/0x80
> [144871.015734][  T172]  ? __lock_release.isra.0+0x69/0x1a0
> [144871.016274][  T172]  schedule+0xe5/0x2e0
> [144871.016820][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144871.017430][  T172]  schedule_preempt_disabled+0x16/0x30
> [144871.017981][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
> [144871.018500][  T172]  ? __kasan_slab_alloc+0x7f/0x90
> [144871.019066][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
> [144871.019691][  T172]  ? __lock_acquire+0x691/0xd00
> [144871.020216][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144871.020806][  T172]  ? lock_acquire+0x11a/0x140
> [144871.021350][  T172]  down_write_nested+0x1c0/0x1f0
> [144871.021847][  T172]  ? __pfx_down_write_nested+0x10/0x10
> [144871.022365][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
> [144871.023499][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.024814][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
> [144871.026064][  T172]  ? update_load_avg+0xb65/0x2c70
> [144871.026726][  T172]  ? __lock_acquire+0x691/0xd00
> [144871.027267][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> [144871.028538][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
> [144871.029728][  T172]  iomap_dio_complete+0x140/0xb20
> [144871.030224][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
> [144871.030743][  T172]  iomap_dio_complete_work+0x58/0x90
> [144871.031190][  T172]  process_one_work+0x947/0x1760
> [144871.031764][  T172]  ? __pfx_process_one_work+0x10/0x10
> [144871.032294][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144871.032889][  T172]  ? lock_is_held_type+0xb0/0x180
> [144871.033502][  T172]  worker_thread+0x601/0xff0
> [144871.034040][  T172]  ? __pfx_worker_thread+0x10/0x10
> [144871.034640][  T172]  kthread+0x361/0x460
> [144871.035071][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.035679][  T172]  ret_from_fork+0x560/0x860
> [144871.036162][  T172]  ? __pfx_ret_from_fork+0x10/0x10
> [144871.036758][  T172]  ? __switch_to+0x473/0xd50
> [144871.037334][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.037914][  T172]  ret_from_fork_asm+0x1a/0x30
> [144871.038394][  T172]  </TASK>
> [144871.038867][  T172] INFO: task kworker/16:2:3814415 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
> [144871.039846][  T172] INFO: task kworker/1:2:3819893 blocked for more than 123 seconds.
> [144871.040650][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
> [144871.041422][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [144871.042136][  T172] task:kworker/1:2     state:D stack:0     pid:3819893 tgid:3819893 ppid:2      task_flags:0x4248060 flags:0x00080000
> [144871.043104][  T172] Workqueue: dio/sdg iomap_dio_complete_work
> [144871.043712][  T172] Call Trace:
> [144871.044164][  T172]  <TASK>
> [144871.044651][  T172]  __schedule+0xc86/0x1b10
> [144871.045172][  T172]  ? __pfx___schedule+0x10/0x10
> [144871.045707][  T172]  ? schedule+0x17e/0x2e0
> [144871.046192][  T172]  ? wq_worker_sleeping+0x65/0x280
> [144871.046741][  T172]  ? find_held_lock+0x2b/0x80
> [144871.047186][  T172]  ? __lock_release.isra.0+0x69/0x1a0
> [144871.047796][  T172]  schedule+0xe5/0x2e0
> [144871.048299][  T172]  xlog_grant_head_wait+0x175/0xac0 [xfs]
> [144871.049471][  T172]  xlog_grant_head_check+0x312/0x3f0 [xfs]
> [144871.050717][  T172]  ? __pfx_xlog_grant_head_check+0x10/0x10 [xfs]
> [144871.051932][  T172]  xfs_log_regrant+0x380/0x7d0 [xfs]
> [144871.053059][  T172]  ? __pfx_xfs_log_regrant+0x10/0x10 [xfs]
> [144871.054192][  T172]  ? xfs_log_ticket_get+0x1b/0x30 [xfs]
> [144871.055259][  T172]  ? xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
> [144871.056414][  T172]  xfs_trans_roll+0x2d9/0x420 [xfs]
> [144871.057517][  T172]  ? kmem_cache_alloc_noprof+0x45b/0x680
> [144871.058100][  T172]  ? xfs_defer_finish_noroll+0x460/0xe70 [xfs]
> [144871.059201][  T172]  xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
> [144871.060341][  T172]  ? __pfx_xfs_defer_trans_roll+0x10/0x10 [xfs]
> [144871.061364][  T172]  ? xfs_defer_relog+0x3e7/0xb50 [xfs]
> [144871.062454][  T172]  xfs_defer_finish_noroll+0x460/0xe70 [xfs]
> [144871.063491][  T172]  ? __pfx_xfs_reflink_end_cow_extent_locked+0x10/0x10 [xfs]
> [144871.064782][  T172]  ? __pfx_xfs_defer_finish_noroll+0x10/0x10 [xfs]
> [144871.065859][  T172]  ? lock_acquire+0x11a/0x140
> [144871.066413][  T172]  xfs_trans_commit+0xfc/0x180 [xfs]
> [144871.067509][  T172]  ? __pfx_down_write_nested+0x10/0x10
> [144871.068046][  T172]  ? __pfx_xfs_trans_commit+0x10/0x10 [xfs]
> [144871.069143][  T172]  xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
> [144871.070340][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
> [144871.071419][  T172]  ? __lock_acquire+0x691/0xd00
> [144871.089717][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> [144871.090837][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
> [144871.091794][  T172]  iomap_dio_complete+0x140/0xb20
> [144871.092194][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
> [144871.092684][  T172]  iomap_dio_complete_work+0x58/0x90
> [144871.093251][  T172]  process_one_work+0x947/0x1760
> [144871.093761][  T172]  ? __pfx_process_one_work+0x10/0x10
> [144871.094406][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144871.095084][  T172]  ? lock_is_held_type+0xb0/0x180
> [144871.095679][  T172]  worker_thread+0x601/0xff0
> [144871.096234][  T172]  ? __pfx_worker_thread+0x10/0x10
> [144871.096820][  T172]  kthread+0x361/0x460
> [144871.097360][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.098065][  T172]  ret_from_fork+0x560/0x860
> [144871.098693][  T172]  ? __pfx_ret_from_fork+0x10/0x10
> [144871.099175][  T172]  ? __switch_to+0x473/0xd50
> [144871.099709][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.100275][  T172]  ret_from_fork_asm+0x1a/0x30
> [144871.100783][  T172]  </TASK>
> [144871.101203][  T172] INFO: task kworker/20:2:3824669 blocked for more than 123 seconds.
> [144871.102109][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
> [144871.102813][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [144871.103685][  T172] task:kworker/20:2    state:D stack:0     pid:3824669 tgid:3824669 ppid:2      task_flags:0x4248060 flags:0x00080000
> [144871.104588][  T172] Workqueue: dio/sdg iomap_dio_complete_work
> [144871.105360][  T172] Call Trace:
> [144871.105868][  T172]  <TASK>
> [144871.106339][  T172]  __schedule+0xc86/0x1b10
> [144871.106949][  T172]  ? trace_hardirqs_on+0x18/0x1a0
> [144871.107632][  T172]  ? __pfx___schedule+0x10/0x10
> [144871.108170][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144871.108600][  T172]  ? wq_worker_sleeping+0x213/0x280
> [144871.109207][  T172]  ? find_held_lock+0x2b/0x80
> [144871.109728][  T172]  ? __lock_release.isra.0+0x69/0x1a0
> [144871.110270][  T172]  schedule+0xe5/0x2e0
> [144871.110762][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144871.111328][  T172]  schedule_preempt_disabled+0x16/0x30
> [144871.111898][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
> [144871.112410][  T172]  ? __kasan_slab_alloc+0x7f/0x90
> [144871.112989][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
> [144871.113648][  T172]  ? __lock_acquire+0x691/0xd00
> [144871.114157][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144871.114665][  T172]  ? lock_acquire+0x11a/0x140
> [144871.115101][  T172]  down_write_nested+0x1c0/0x1f0
> [144871.115664][  T172]  ? __pfx_down_write_nested+0x10/0x10
> [144871.116122][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
> [144871.117105][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.118239][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
> [144871.119389][  T172]  ? lock_is_held_type+0xb0/0x180
> [144871.119781][  T172]  ? __lock_acquire+0x691/0xd00
> [144871.120177][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> [144871.121294][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
> [144871.122391][  T172]  iomap_dio_complete+0x140/0xb20
> [144871.122770][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
> [144871.123425][  T172]  iomap_dio_complete_work+0x58/0x90
> [144871.124051][  T172]  process_one_work+0x947/0x1760
> [144871.124445][  T172]  ? __pfx_process_one_work+0x10/0x10
> [144871.125017][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144871.125580][  T172]  ? lock_is_held_type+0xb0/0x180
> [144871.126138][  T172]  worker_thread+0x601/0xff0
> [144871.126680][  T172]  ? __pfx_worker_thread+0x10/0x10
> [144871.127126][  T172]  kthread+0x361/0x460
> [144871.127762][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.128234][  T172]  ret_from_fork+0x560/0x860
> [144871.128772][  T172]  ? __pfx_ret_from_fork+0x10/0x10
> [144871.129337][  T172]  ? __switch_to+0x473/0xd50
> [144871.129842][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.130364][  T172]  ret_from_fork_asm+0x1a/0x30
> [144871.130892][  T172]  </TASK>
> [144871.131250][  T172] INFO: task kworker/20:2:3824669 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
> [144871.132261][  T172] INFO: task kworker/4:0:3824816 blocked for more than 123 seconds.
> [144871.133177][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
> [144871.133912][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [144871.134827][  T172] task:kworker/4:0     state:D stack:0     pid:3824816 tgid:3824816 ppid:2      task_flags:0x4248060 flags:0x00080000
> [144871.135724][  T172] Workqueue: dio/sdg iomap_dio_complete_work
> [144871.136356][  T172] Call Trace:
> [144871.136856][  T172]  <TASK>
> [144871.137342][  T172]  __schedule+0xc86/0x1b10
> [144871.137818][  T172]  ? __pfx___schedule+0x10/0x10
> [144871.138410][  T172]  ? schedule+0x17e/0x2e0
> [144871.138810][  T172]  ? wq_worker_sleeping+0x65/0x280
> [144871.139356][  T172]  ? find_held_lock+0x2b/0x80
> [144871.139817][  T172]  ? __lock_release.isra.0+0x69/0x1a0
> [144871.140631][  T172]  schedule+0xe5/0x2e0
> [144871.141220][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144871.141887][  T172]  schedule_preempt_disabled+0x16/0x30
> [144871.142473][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
> [144871.143026][  T172]  ? lock_release.part.0+0x1c/0x50
> [144871.143557][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
> [144871.144101][  T172]  ? __lock_acquire+0x691/0xd00
> [144871.144681][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144871.145057][  T172]  ? lock_acquire+0x11a/0x140
> [144871.145709][  T172]  down_write_nested+0x1c0/0x1f0
> [144871.146205][  T172]  ? __pfx_down_write_nested+0x10/0x10
> [144871.146885][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
> [144871.147929][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.148978][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
> [144871.150061][  T172]  ? __lock_acquire+0x691/0xd00
> [144871.150641][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> [144871.151842][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
> [144871.152998][  T172]  iomap_dio_complete+0x140/0xb20
> [144871.153548][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
> [144871.154099][  T172]  iomap_dio_complete_work+0x58/0x90
> [144871.154759][  T172]  process_one_work+0x947/0x1760
> [144871.155210][  T172]  ? __pfx_process_one_work+0x10/0x10
> [144871.155675][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144871.156336][  T172]  ? lock_is_held_type+0xb0/0x180
> [144871.156920][  T172]  worker_thread+0x601/0xff0
> [144871.157437][  T172]  ? __pfx_worker_thread+0x10/0x10
> [144871.157951][  T172]  kthread+0x361/0x460
> [144871.158466][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.159012][  T172]  ret_from_fork+0x560/0x860
> [144871.159502][  T172]  ? __pfx_ret_from_fork+0x10/0x10
> [144871.160015][  T172]  ? __switch_to+0x473/0xd50
> [144871.160539][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.161051][  T172]  ret_from_fork_asm+0x1a/0x30
> [144871.161685][  T172]  </TASK>
> [144871.162077][  T172] INFO: task kworker/4:0:3824816 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
> [144871.162900][  T172] INFO: task kworker/12:0:3826849 blocked for more than 123 seconds.
> [144871.163697][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
> [144871.164415][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [144871.165277][  T172] task:kworker/12:0    state:D stack:0     pid:3826849 tgid:3826849 ppid:2      task_flags:0x4248060 flags:0x00080000
> [144871.166221][  T172] Workqueue: dio/sdg iomap_dio_complete_work
> [144871.166952][  T172] Call Trace:
> [144871.167283][  T172]  <TASK>
> [144871.167853][  T172]  __schedule+0xc86/0x1b10
> [144871.168439][  T172]  ? __pfx___schedule+0x10/0x10
> [144871.168914][  T172]  ? schedule+0x17e/0x2e0
> [144871.169451][  T172]  ? wq_worker_sleeping+0x65/0x280
> [144871.169927][  T172]  ? find_held_lock+0x2b/0x80
> [144871.170288][  T172]  ? __lock_release.isra.0+0x69/0x1a0
> [144871.170911][  T172]  schedule+0xe5/0x2e0
> [144871.171387][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144871.171939][  T172]  schedule_preempt_disabled+0x16/0x30
> [144871.172496][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
> [144871.173087][  T172]  ? lock_release.part.0+0x1c/0x50
> [144871.173702][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
> [144871.174226][  T172]  ? __lock_acquire+0x691/0xd00
> [144871.174891][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144871.175466][  T172]  ? lock_acquire+0x11a/0x140
> [144871.176164][  T172]  down_write_nested+0x1c0/0x1f0
> [144871.176754][  T172]  ? __pfx_down_write_nested+0x10/0x10
> [144871.177531][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
> [144871.178526][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.179662][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
> [144871.180880][  T172]  ? lock_is_held_type+0xb0/0x180
> [144871.181342][  T172]  ? __lock_acquire+0x691/0xd00
> [144871.181872][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> [144871.183071][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
> [144871.184221][  T172]  iomap_dio_complete+0x140/0xb20
> [144871.184745][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
> [144871.185338][  T172]  iomap_dio_complete_work+0x58/0x90
> [144871.185864][  T172]  process_one_work+0x947/0x1760
> [144871.186361][  T172]  ? __pfx_process_one_work+0x10/0x10
> [144871.186920][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144871.187381][  T172]  ? lock_is_held_type+0xb0/0x180
> [144871.187924][  T172]  worker_thread+0x601/0xff0
> [144871.188591][  T172]  ? __pfx_worker_thread+0x10/0x10
> [144871.189109][  T172]  kthread+0x361/0x460
> [144871.189647][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.190131][  T172]  ret_from_fork+0x560/0x860
> [144871.190578][  T172]  ? __pfx_ret_from_fork+0x10/0x10
> [144871.191172][  T172]  ? __switch_to+0x473/0xd50
> [144871.191771][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.192247][  T172]  ret_from_fork_asm+0x1a/0x30
> [144871.192760][  T172]  </TASK>
> [144871.193146][  T172] INFO: task kworker/12:0:3826849 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
> [144871.202516][  T172] INFO: task kworker/21:1:3827095 blocked for more than 123 seconds.
> [144871.203182][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
> [144871.203953][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [144871.204755][  T172] task:kworker/21:1    state:D stack:0     pid:3827095 tgid:3827095 ppid:2      task_flags:0x4248060 flags:0x00080000
> [144871.205642][  T172] Workqueue: dio/sdg iomap_dio_complete_work
> [144871.206251][  T172] Call Trace:
> [144871.206793][  T172]  <TASK>
> [144871.207382][  T172]  __schedule+0xc86/0x1b10
> [144871.207815][  T172]  ? trace_hardirqs_on+0x18/0x1a0
> [144871.208402][  T172]  ? __pfx___schedule+0x10/0x10
> [144871.208871][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144871.209404][  T172]  ? wq_worker_sleeping+0x213/0x280
> [144871.209925][  T172]  ? find_held_lock+0x2b/0x80
> [144871.210492][  T172]  ? __lock_release.isra.0+0x69/0x1a0
> [144871.211050][  T172]  schedule+0xe5/0x2e0
> [144871.211527][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144871.212043][  T172]  schedule_preempt_disabled+0x16/0x30
> [144871.212786][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
> [144871.213398][  T172]  ? __kasan_slab_alloc+0x7f/0x90
> [144871.213975][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
> [144871.214563][  T172]  ? __lock_acquire+0x691/0xd00
> [144871.215009][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144871.215710][  T172]  ? lock_acquire+0x11a/0x140
> [144871.216164][  T172]  down_write_nested+0x1c0/0x1f0
> [144871.216688][  T172]  ? __pfx_down_write_nested+0x10/0x10
> [144871.217212][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
> [144871.218301][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.219411][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
> [144871.220508][  T172]  ? __lock_acquire+0x691/0xd00
> [144871.221057][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> [144871.222191][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
> [144871.223292][  T172]  iomap_dio_complete+0x140/0xb20
> [144871.223820][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
> [144871.224332][  T172]  iomap_dio_complete_work+0x58/0x90
> [144871.224792][  T172]  process_one_work+0x947/0x1760
> [144871.225354][  T172]  ? __pfx_process_one_work+0x10/0x10
> [144871.225861][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144871.226458][  T172]  ? lock_is_held_type+0xb0/0x180
> [144871.226982][  T172]  worker_thread+0x601/0xff0
> [144871.227454][  T172]  ? __pfx_worker_thread+0x10/0x10
> [144871.227993][  T172]  ? __kthread_parkme+0xbd/0x210
> [144871.228532][  T172]  ? __pfx_worker_thread+0x10/0x10
> [144871.229057][  T172]  ? __pfx_worker_thread+0x10/0x10
> [144871.229619][  T172]  kthread+0x361/0x460
> [144871.230083][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.230578][  T172]  ret_from_fork+0x560/0x860
> [144871.231049][  T172]  ? __pfx_ret_from_fork+0x10/0x10
> [144871.231633][  T172]  ? __switch_to+0x473/0xd50
> [144871.232124][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.232544][  T172]  ret_from_fork_asm+0x1a/0x30
> [144871.233081][  T172]  </TASK>
> [144871.233440][  T172] INFO: task kworker/21:1:3827095 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
> [144871.234328][  T172] INFO: task kworker/10:0:3827669 blocked for more than 123 seconds.
> [144871.235091][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
> [144871.235858][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [144871.236582][  T172] task:kworker/10:0    state:D stack:0     pid:3827669 tgid:3827669 ppid:2      task_flags:0x4248060 flags:0x00080000
> [144871.237651][  T172] Workqueue: dio/sdg iomap_dio_complete_work
> [144871.238173][  T172] Call Trace:
> [144871.238946][  T172]  <TASK>
> [144871.239469][  T172]  __schedule+0xc86/0x1b10
> [144871.240084][  T172]  ? trace_hardirqs_on+0x18/0x1a0
> [144871.240642][  T172]  ? __pfx___schedule+0x10/0x10
> [144871.241088][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144871.241694][  T172]  ? wq_worker_sleeping+0x213/0x280
> [144871.242326][  T172]  ? find_held_lock+0x2b/0x80
> [144871.242813][  T172]  ? __lock_release.isra.0+0x69/0x1a0
> [144871.243379][  T172]  schedule+0xe5/0x2e0
> [144871.243881][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
> [144871.244378][  T172]  schedule_preempt_disabled+0x16/0x30
> [144871.244920][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
> [144871.245543][  T172]  ? __kasan_slab_alloc+0x7f/0x90
> [144871.246066][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
> [144871.246662][  T172]  ? __lock_acquire+0x691/0xd00
> [144871.247131][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144871.247711][  T172]  ? lock_acquire+0x11a/0x140
> [144871.248149][  T172]  down_write_nested+0x1c0/0x1f0
> [144871.248724][  T172]  ? __pfx_down_write_nested+0x10/0x10
> [144871.249285][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
> [144871.250297][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.251346][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
> [144871.252414][  T172]  ? __lock_acquire+0x691/0xd00
> [144871.252983][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> [144871.254104][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
> [144871.255189][  T172]  iomap_dio_complete+0x140/0xb20
> [144871.255756][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
> [144871.256283][  T172]  iomap_dio_complete_work+0x58/0x90
> [144871.256814][  T172]  process_one_work+0x947/0x1760
> [144871.257211][  T172]  ? __pfx_process_one_work+0x10/0x10
> [144871.257962][  T172]  ? lock_acquire.part.0+0xc8/0x240
> [144871.258415][  T172]  ? lock_is_held_type+0xb0/0x180
> [144871.258907][  T172]  worker_thread+0x601/0xff0
> [144871.259454][  T172]  ? __pfx_worker_thread+0x10/0x10
> [144871.259949][  T172]  kthread+0x361/0x460
> [144871.260364][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.260889][  T172]  ret_from_fork+0x560/0x860
> [144871.261384][  T172]  ? __pfx_ret_from_fork+0x10/0x10
> [144871.261926][  T172]  ? __switch_to+0x473/0xd50
> [144871.262448][  T172]  ? __pfx_kthread+0x10/0x10
> [144871.262968][  T172]  ret_from_fork_asm+0x1a/0x30
> [144871.263492][  T172]  </TASK>
> [144871.263903][  T172] INFO: task kworker/10:0:3827669 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
> [144871.264856][  T172] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
> [144871.265760][  T172]
> [144871.265760][  T172] Showing all locks held in the system:
> [144871.266766][  T172] locks held by pr/ttyS0/16: 2, last CPU#22:
> [144871.267407][  T172]  #0: ffffffff8c320658 (console_srcu){....}-{0:0}, at: console_srcu_read_lock+0x49/0x70
> [144871.268258][  T172]  #1: ffffffff9c3db638 (&port_lock_key){-.-.}-{3:3}, at: univ8250_console_device_lock+0x63/0xc0
> [144871.269176][  T172] locks held by khungtaskd/172: 1, last CPU#20:
> [144871.269752][  T172]  #0: ffffffff8c3369e0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
> [144871.270692][  T172] locks held by kworker/10:173/3785794: 4, on CPU#10:
> [144871.271375][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.272341][  T172]  #1: ff11000109407d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.273284][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.274707][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.276050][  T172] locks held by kworker/18:2/3812136: 4, on CPU#18:
> [144871.276811][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.277696][  T172]  #1: ff1100018e71fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.278634][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.280090][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.281626][  T172] locks held by kworker/20:1/3814407: 4, on CPU#20:
> [144871.282267][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.283196][  T172]  #1: ff11000214267d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.284154][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.285620][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.287063][  T172] locks held by kworker/16:2/3814415: 4, on CPU#16:
> [144871.287789][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.288637][  T172]  #1: ff1100019cc67d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.289792][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.291054][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.292462][  T172] locks held by kworker/1:2/3819893: 4, on CPU#1:
> [144871.293085][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.294070][  T172]  #1: ff110001246d7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.294991][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.296348][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.297789][  T172] locks held by kworker/20:2/3824669: 4, on CPU#20:
> [144871.298349][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.299205][  T172]  #1: ff1100022fc77d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.300225][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.301638][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.303077][  T172] locks held by kworker/4:0/3824816: 4, on CPU#4:
> [144871.303744][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.312636][  T172]  #1: ff11000424997d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.313372][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.314586][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.315921][  T172] locks held by kworker/12:0/3826849: 4, on CPU#12:
> [144871.316445][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.317222][  T172]  #1: ff110001e71bfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.318124][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.319340][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.320712][  T172] locks held by kworker/21:1/3827095: 4, on CPU#21:
> [144871.321339][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.322330][  T172]  #1: ff11000172fefd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.323140][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.324642][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.326042][  T172] locks held by kworker/10:0/3827669: 4, on CPU#10:
> [144871.326655][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.327575][  T172]  #1: ff11000122177d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.328366][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.329889][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.331476][  T172] locks held by kworker/17:1/3827670: 4, on CPU#17:
> [144871.332105][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.332897][  T172]  #1: ff110001d0307d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.333703][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.334981][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.336354][  T172] locks held by kworker/16:1/3864132: 4, on CPU#16:
> [144871.337033][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.337973][  T172]  #1: ff1100023211fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.338641][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.339966][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.341388][  T172] locks held by kworker/8:1/3868486: 4, on CPU#8:
> [144871.342196][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.343143][  T172]  #1: ff110001a24d7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.344179][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.345576][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.347006][  T172] locks held by kworker/10:1/3872909: 4, on CPU#10:
> [144871.347626][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.348518][  T172]  #1: ff110001606dfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.349516][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.350857][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.352433][  T172] locks held by kworker/17:2/3872910: 4, on CPU#17:
> [144871.353061][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.354161][  T172]  #1: ff1100041c1b7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.354964][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.356302][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.357755][  T172] locks held by kworker/23:0/3872911: 4, on CPU#23:
> [144871.358527][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.359329][  T172]  #1: ff110004b8c8fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.360223][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.361505][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.363110][  T172] locks held by kworker/7:3/3872919: 4, on CPU#7:
> [144871.363803][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.364639][  T172]  #1: ff1100019af97d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.365488][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.366938][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.368364][  T172] locks held by kworker/20:3/3872924: 4, on CPU#20:
> [144871.369008][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.369969][  T172]  #1: ff110001aa42fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.371118][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.372518][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.373983][  T172] locks held by kworker/3:2/3872925: 4, on CPU#3:
> [144871.374629][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.375515][  T172]  #1: ff11000444b8fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.376358][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.377586][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.378950][  T172] locks held by kworker/16:3/3872926: 4, on CPU#16:
> [144871.379685][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.380610][  T172]  #1: ff11000682cdfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.381567][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.382968][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.384263][  T172] locks held by kworker/17:3/3872927: 4, on CPU#17:
> [144871.384833][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.385697][  T172]  #1: ff110001d3ca7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.386411][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.387778][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.389139][  T172] locks held by kworker/16:4/3872929: 4, on CPU#16:
> [144871.389779][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.390582][  T172]  #1: ff110003cb7efd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.391402][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.392568][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.394019][  T172] locks held by kworker/19:2/3872930: 4, on CPU#19:
> [144871.394685][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.395449][  T172]  #1: ff1100021374fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.396394][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.397664][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.399332][  T172] locks held by kworker/18:3/3872935: 3, on CPU#18:
> [144871.400131][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.401110][  T172]  #1: ff110002862b7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.402243][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.403502][  T172] locks held by kworker/10:2/3872936: 4, on CPU#10:
> [144871.404186][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.405139][  T172]  #1: ff110003d8e8fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.406011][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.407341][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.408729][  T172] locks held by kworker/8:3/3872941: 3, on CPU#8:
> [144871.409141][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.410067][  T172]  #1: ff110003df1e7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.410799][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.411989][  T172] locks held by kworker/1:3/3872953: 4, on CPU#1:
> [144871.412584][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.413531][  T172]  #1: ff1100043f1bfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.414464][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.424444][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.425808][  T172] locks held by kworker/4:4/3872955: 4, on CPU#4:
> [144871.426603][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.427468][  T172]  #1: ff1100014de77d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.428336][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.429615][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.430975][  T172] locks held by fio/3872961: 3, on CPU#2:
> [144871.431620][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.433089][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.434279][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
> [144871.435797][  T172] locks held by fio/3872962: 3, on CPU#23:
> [144871.436291][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.437618][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.438845][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
> [144871.440124][  T172] locks held by fio/3872963: 3, on CPU#23:
> [144871.440772][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.442297][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.443694][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
> [144871.445025][  T172] locks held by fio/3872964: 3, on CPU#14:
> [144871.445514][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.446890][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.448097][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
> [144871.449466][  T172] locks held by fio/3872965: 3, on CPU#12:
> [144871.450120][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.451359][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.452681][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.454164][  T172] locks held by fio/3872966: 3, on CPU#16:
> [144871.454720][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.455979][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.457331][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.458737][  T172] locks held by fio/3872967: 3, on CPU#4:
> [144871.459369][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.460809][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.461990][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
> [144871.463126][  T172] locks held by fio/3872968: 3, on CPU#1:
> [144871.463662][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.464985][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.466188][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.467438][  T172] locks held by fio/3872969: 3, on CPU#6:
> [144871.468111][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.469612][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.471012][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.472352][  T172] locks held by fio/3872970: 3, on CPU#4:
> [144871.473018][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.474395][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.475759][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.477331][  T172] locks held by fio/3872971: 3, on CPU#14:
> [144871.477764][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.479087][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.480334][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.481548][  T172] locks held by fio/3872972: 2, on CPU#10:
> [144871.482113][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.483336][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.484625][  T172] locks held by fio/3872973: 3, on CPU#10:
> [144871.485350][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.486616][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.487949][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.489152][  T172] locks held by fio/3872974: 2, on CPU#18:
> [144871.489572][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.491050][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.492416][  T172] locks held by fio/3872975: 3, on CPU#8:
> [144871.493166][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.494401][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.495553][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.496938][  T172] locks held by fio/3872976: 3, on CPU#23:
> [144871.497422][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.498613][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.499903][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
> [144871.501249][  T172] locks held by fio/3872977: 3, on CPU#1:
> [144871.501945][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.503238][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.504494][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.505963][  T172] locks held by fio/3872978: 3, on CPU#12:
> [144871.506558][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.508041][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.509561][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.510975][  T172] locks held by fio/3872979: 3, on CPU#4:
> [144871.511616][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.512986][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.514342][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.515561][  T172] locks held by fio/3872980: 2, on CPU#2:
> [144871.516337][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.517588][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.518941][  T172] locks held by fio/3872981: 3, on CPU#5:
> [144871.519544][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.520980][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.522436][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
> [144871.523761][  T172] locks held by fio/3872982: 3, on CPU#7:
> [144871.524345][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.525617][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.526995][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.528345][  T172] locks held by fio/3872983: 2, on CPU#18:
> [144871.529013][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.530159][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.531658][  T172] locks held by fio/3872984: 3, on CPU#21:
> [144871.532344][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.533662][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.535127][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.536354][  T172] locks held by fio/3872985: 3, on CPU#14:
> [144871.536893][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.538193][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.539344][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.540565][  T172] locks held by fio/3872986: 3, on CPU#23:
> [144871.549064][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.550342][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.551412][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.552644][  T172] locks held by fio/3872987: 3, on CPU#9:
> [144871.553252][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.554617][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.555963][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.557289][  T172] locks held by fio/3872988: 2, on CPU#9:
> [144871.557983][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.559251][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.560650][  T172] locks held by fio/3872989: 3, on CPU#13:
> [144871.561148][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.562620][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.563966][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.565353][  T172] locks held by fio/3872990: 3, on CPU#22:
> [144871.566014][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.567292][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.568639][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.569912][  T172] locks held by fio/3872991: 3, on CPU#15:
> [144871.570643][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.572074][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.573526][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.574865][  T172] locks held by fio/3872992: 3, on CPU#19:
> [144871.575385][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.576702][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.577974][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.579176][  T172] locks held by fio/3872993: 2, on CPU#8:
> [144871.579776][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.581296][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.582823][  T172] locks held by fio/3872994: 2, on CPU#10:
> [144871.583327][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.584824][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.586338][  T172] locks held by fio/3872995: 2, on CPU#0:
> [144871.586746][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.588065][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.589592][  T172] locks held by fio/3872996: 3, on CPU#3:
> [144871.590228][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.591624][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.592955][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.594411][  T172] locks held by fio/3872997: 2, on CPU#9:
> [144871.594880][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.596102][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.597443][  T172] locks held by fio/3872998: 3, on CPU#11:
> [144871.598214][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.599581][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.600877][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.602277][  T172] locks held by fio/3872999: 2, on CPU#15:
> [144871.602933][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.604171][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.605502][  T172] locks held by fio/3873000: 2, on CPU#13:
> [144871.606086][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.607446][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.608891][  T172] locks held by fio/3873001: 2, on CPU#20:
> [144871.609629][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.610941][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.612536][  T172] locks held by fio/3873002: 2, on CPU#21:
> [144871.613261][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.614548][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.615849][  T172] locks held by fio/3873003: 2, on CPU#7:
> [144871.616466][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.617936][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.619512][  T172] locks held by fio/3873004: 3, on CPU#12:
> [144871.620084][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.621559][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.623035][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.624371][  T172] locks held by fio/3873005: 3, on CPU#2:
> [144871.624968][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.626326][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.627504][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.628825][  T172] locks held by fio/3873006: 2, on CPU#5:
> [144871.629555][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.630924][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
> [144871.632295][  T172] locks held by fio/3873007: 3, on CPU#22:
> [144871.632986][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.634247][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.635569][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
> [144871.636907][  T172] locks held by fio/3873008: 3, on CPU#16:
> [144871.637391][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
> [144871.638706][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.639974][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
> [144871.641349][  T172] locks held by kworker/22:9/3873014: 4, on CPU#22:
> [144871.641954][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.643046][  T172]  #1: ff1100075a3b7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.643896][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.645235][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.646870][  T172] locks held by kworker/10:3/3873017: 4, on CPU#10:
> [144871.647514][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.648341][  T172]  #1: ff110002411cfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.649336][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.650672][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.652162][  T172] locks held by kworker/16:5/3873018: 4, on CPU#16:
> [144871.652872][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.653694][  T172]  #1: ff1100019cfd7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.654690][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.656010][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.657334][  T172] locks held by kworker/16:6/3873019: 4, on CPU#16:
> [144871.657807][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.658834][  T172]  #1: ff110005ec73fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.659987][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.661147][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.662392][  T172] locks held by kworker/16:7/3873020: 4, on CPU#16:
> [144871.662945][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.663830][  T172]  #1: ff110004e58a7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.664843][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.674237][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.675488][  T172] locks held by kworker/16:8/3873021: 4, on CPU#16:
> [144871.676270][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.676948][  T172]  #1: ff110001e6617d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.677872][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.679079][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.680423][  T172] locks held by kworker/16:9/3873022: 4, on CPU#16:
> [144871.680913][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.681864][  T172]  #1: ff11000156bbfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.682805][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.684009][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.685395][  T172] locks held by kworker/16:10/3873023: 4, on CPU#16:
> [144871.685974][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.686863][  T172]  #1: ff1100040bf57d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.687686][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.689001][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.690449][  T172] locks held by kworker/17:5/3873025: 4, on CPU#17:
> [144871.691061][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.691966][  T172]  #1: ff1100013957fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.692935][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.694197][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.695674][  T172] locks held by kworker/12:3/3873030: 4, on CPU#12:
> [144871.696301][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
> [144871.697143][  T172]  #1: ff110007b3e6fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
> [144871.698099][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
> [144871.699455][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> [144871.700840][  T172]
> [144871.701294][  T172] =============================================

Re: [bug report] fstests generic/774 hang again

[Shin'ichiro Kawasaki]

On Sep 15, 2026 / 11:00, John Garry wrote:
> On 9/15/26 10:34, Shin'ichiro Kawasaki wrote:
> > I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB
> > TCMU fileio devices. Actually I once reported this hang symptom in last October,
> > and Darrick and John kindly took some actions [1]. Since then, the hang
> > disappeared and I have not observed the hang almost one year. However, my test
> > system started reporting the hang again last week.
> > 
> > [1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t
> 
> This was the series to resolve, I think:
> https://lore.kernel.org/fstests/176279908967.605950.2192923313361120314.stgit@frogsfrogsfrogs/T/#t
> 
> As an experiment, can you reduce the file size further, like:
> 
> --- a/tests/generic/774
> +++ b/tests/generic/774
> @@ -29,7 +29,7 @@ aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))")
>  fsbsize=$(_get_block_size $SCRATCH_MNT)
> 
>  threads=$(_min "$(($(nproc) * 2 * LOAD_FACTOR))" "100")
> -filesize=$((aw_bsize * threads * 10))
> +filesize=$((aw_bsize * threads))
>  depth=$threads
>  aw_io_size=$((filesize / threads))
>  aw_io_inc=$aw_io_size

Thanks for the response. I repeated the test case 100 times with the change
above, and I did not observe the hang. The change looks avoiding the hang.

> 
> > 
> > FYI, here I attache the kernel message [2].
> 
> Is mainline baseline hanging, i.e. v7.3-rc2?

I will try this out tomorrow.

Re: [bug report] fstests generic/774 hang again

[John Garry]

On 9/15/26 12:48, Shin'ichiro Kawasaki wrote:
> On Sep 15, 2026 / 11:00, John Garry wrote:
>> On 9/15/26 10:34, Shin'ichiro Kawasaki wrote:
>>> I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB
>>> TCMU fileio devices. Actually I once reported this hang symptom in last October,
>>> and Darrick and John kindly took some actions [1]. Since then, the hang
>>> disappeared and I have not observed the hang almost one year. However, my test
>>> system started reporting the hang again last week.
>>>
>>> [1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t
>>
>> This was the series to resolve, I think:
>> https://lore.kernel.org/fstests/176279908967.605950.2192923313361120314.stgit@frogsfrogsfrogs/T/#t
>>
>> As an experiment, can you reduce the file size further, like:
>>
>> --- a/tests/generic/774
>> +++ b/tests/generic/774
>> @@ -29,7 +29,7 @@ aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))")
>>   fsbsize=$(_get_block_size $SCRATCH_MNT)
>>
>>   threads=$(_min "$(($(nproc) * 2 * LOAD_FACTOR))" "100")
>> -filesize=$((aw_bsize * threads * 10))
>> +filesize=$((aw_bsize * threads))
>>   depth=$threads
>>   aw_io_size=$((filesize / threads))
>>   aw_io_inc=$aw_io_size
> 
> Thanks for the response. I repeated the test case 100 times with the change
> above, and I did not observe the hang. The change looks avoiding the hang.
> 
>>
>>>
>>> FYI, here I attache the kernel message [2].
>>
>> Is mainline baseline hanging, i.e. v7.3-rc2?
> 
> I will try this out tomorrow.

thanks

About the kernel code, I am wondering if using IOMAP_DIO_FORCE_WAIT for 
CoW-based atomics could help avoid this issue as we seem to be bogged 
down in lock contention.

Re: [bug report] fstests generic/774 hang again

[Darrick J. Wong]

On Tue, Sep 15, 2026 at 03:48:13PM +0100, John Garry wrote:
> On 9/15/26 12:48, Shin'ichiro Kawasaki wrote:
> > On Sep 15, 2026 / 11:00, John Garry wrote:
> > > On 9/15/26 10:34, Shin'ichiro Kawasaki wrote:
> > > > I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB
> > > > TCMU fileio devices. Actually I once reported this hang symptom in last October,
> > > > and Darrick and John kindly took some actions [1]. Since then, the hang
> > > > disappeared and I have not observed the hang almost one year. However, my test
> > > > system started reporting the hang again last week.
> > > > 
> > > > [1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t
> > > 
> > > This was the series to resolve, I think:
> > > https://lore.kernel.org/fstests/176279908967.605950.2192923313361120314.stgit@frogsfrogsfrogs/T/#t
> > > 
> > > As an experiment, can you reduce the file size further, like:
> > > 
> > > --- a/tests/generic/774
> > > +++ b/tests/generic/774
> > > @@ -29,7 +29,7 @@ aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))")
> > >   fsbsize=$(_get_block_size $SCRATCH_MNT)
> > > 
> > >   threads=$(_min "$(($(nproc) * 2 * LOAD_FACTOR))" "100")
> > > -filesize=$((aw_bsize * threads * 10))
> > > +filesize=$((aw_bsize * threads))
> > >   depth=$threads
> > >   aw_io_size=$((filesize / threads))
> > >   aw_io_inc=$aw_io_size
> > 
> > Thanks for the response. I repeated the test case 100 times with the change
> > above, and I did not observe the hang. The change looks avoiding the hang.
> > 
> > > 
> > > > 
> > > > FYI, here I attache the kernel message [2].
> > > 
> > > Is mainline baseline hanging, i.e. v7.3-rc2?
> > 
> > I will try this out tomorrow.
> 
> thanks
> 
> About the kernel code, I am wondering if using IOMAP_DIO_FORCE_WAIT for
> CoW-based atomics could help avoid this issue as we seem to be bogged down
> in lock contention.

Which lock is being contended, anyway?  It looks like the ILOCK?

--D

Re: [bug report] fstests generic/774 hang again

[John Garry]

On 9/15/26 15:50, Darrick J. Wong wrote:
>> thanks
>>
>> About the kernel code, I am wondering if using IOMAP_DIO_FORCE_WAIT for
>> CoW-based atomics could help avoid this issue as we seem to be bogged down
>> in lock contention.
> Which lock is being contended, anyway?  It looks like the ILOCK?

Yeah, I think so Re. ilock. I had some perf data illustrating this from 
last year, which I can't seem to find, so I will re-generate it.

Re: [bug report] fstests generic/774 hang again

[John Garry]

On 15/09/2026 16:41, John Garry wrote:
> On 9/15/26 15:50, Darrick J. Wong wrote:
>>> thanks
>>>
>>> About the kernel code, I am wondering if using IOMAP_DIO_FORCE_WAIT for
>>> CoW-based atomics could help avoid this issue as we seem to be bogged 
>>> down
>>> in lock contention.
>> Which lock is being contended, anyway?  It looks like the ILOCK?
> 
> Yeah, I think so Re. ilock. I had some perf data illustrating this from 
> last year, which I can't seem to find, so I will re-generate it.

Here is perf call graph snippet when running fio with 72x threads 
issuing 4KB atomic writes on 1MB file:

--50.46%--xfs_file_dio_write_atomic
|
|--37.19%--iomap_dio_rw
|          |
|      --37.14%--__iomap_dio_rw
|           |
|       --36.50%--iomap_iter
|            |
|             --36.49%--xfs_atomic_write_cow_iomap_next
|                  |
|                  |--27.34%--xfs_ilock
|                  |          |
|                  |      --27.34%--down_write
|                  |           |
|                  |       --27.28%--rwsem_down_write_slowpath
|                  |            |
|                  |            |--24.65%--osq_lock
|                  |            |
|                  |             --2.35%--rwsem_spin_on_owner
|                  |
|                  |--7.62%--xfs_trans_alloc_inode
|                  |          |
|                  |      --7.59%--xfs_ilock


Notice how much time we spend getting that ilock.

We could try something like this:

----8<------

[PATCH] xfs: serialize CoW-based atomic writes

We have had reports of system hangs when testing Cow-based atomic writes
for large block sizes.

Analysis has shown large contention on the the per-inode ilock,
superficially in creating the mapping in
xfs_atomic_write_cow_iomap_begin().

Serialize atomic writes to reduce this contention. In some cases this 
may reduce performance, but high performance has not been a requirement 
so far for CoW-based atomic writes.

Signed-off-by: John Garry <john.garry@linux.dev>

diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index d8202da15aca..ff48757cf0f0 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -819,10 +819,12 @@ xfs_file_dio_write_atomic(
  	 * HW offload should be faster, so try that first if it is already
  	 * known that the write length is not too large.
  	 */
-	if (ocount > xfs_inode_buftarg(ip)->bt_awu_max)
+	if (ocount > xfs_inode_buftarg(ip)->bt_awu_max) {
  		dops = &xfs_atomic_write_cow_iomap_ops;
-	else
+		dio_flags |= IOMAP_DIO_FORCE_WAIT;
+	} else {
  		dops = &xfs_direct_write_iomap_ops;
+	}

  retry:
  	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
@@ -840,6 +842,8 @@ xfs_file_dio_write_atomic(
  	}

  	trace_xfs_file_direct_write(iocb, from);
+	if (dio_flags & IOMAP_DIO_FORCE_WAIT)
+		inode_dio_wait(VFS_I(ip));
  	if (mapping_stable_writes(iocb->ki_filp->f_mapping))
  		dio_flags |= IOMAP_DIO_BOUNCE;
  	ret = iomap_dio_rw(iocb, from, dops, &xfs_dio_write_ops, dio_flags,
@@ -853,6 +857,7 @@ xfs_file_dio_write_atomic(
  	 */
  	if (ret == -ENOPROTOOPT && dops == &xfs_direct_write_iomap_ops) {
  		xfs_iunlock(ip, iolock);
+		dio_flags |= IOMAP_DIO_FORCE_WAIT;
  		dops = &xfs_atomic_write_cow_iomap_ops;
  		goto retry;
  	}

---->8------

This is far from ideal, but I think that it should stops the hangs - 
having the kernel hang is intolerable.

I also notice that in xfs_atomic_write_cow_iomap_next() we drop the 
ilock for calling xfs_trans_alloc_inode() (which immediately grabs the 
same lock). It would seem that could be improved.

Re: [bug report] fstests generic/774 hang again

[Dave Chinner]

On Wed, Sep 16, 2026 at 11:23:16AM +0100, John Garry wrote:
> On 15/09/2026 16:41, John Garry wrote:
> > On 9/15/26 15:50, Darrick J. Wong wrote:
> > > > thanks
> > > > 
> > > > About the kernel code, I am wondering if using IOMAP_DIO_FORCE_WAIT for
> > > > CoW-based atomics could help avoid this issue as we seem to be
> > > > bogged down
> > > > in lock contention.
> > > Which lock is being contended, anyway?  It looks like the ILOCK?
> > 
> > Yeah, I think so Re. ilock. I had some perf data illustrating this from
> > last year, which I can't seem to find, so I will re-generate it.
> 
> Here is perf call graph snippet when running fio with 72x threads issuing
> 4KB atomic writes on 1MB file:
> 
> --50.46%--xfs_file_dio_write_atomic
> |
> |--37.19%--iomap_dio_rw
> |          |
> |      --37.14%--__iomap_dio_rw
> |           |
> |       --36.50%--iomap_iter
> |            |
> |             --36.49%--xfs_atomic_write_cow_iomap_next
> |                  |
> |                  |--27.34%--xfs_ilock
> |                  |          |
> |                  |      --27.34%--down_write
> |                  |           |
> |                  |       --27.28%--rwsem_down_write_slowpath
> |                  |            |
> |                  |            |--24.65%--osq_lock
> |                  |            |
> |                  |             --2.35%--rwsem_spin_on_owner
> |                  |
> |                  |--7.62%--xfs_trans_alloc_inode
> |                  |          |
> |                  |      --7.59%--xfs_ilock

Where's the other half of the lock contention?

If this is all there is, it's because of concurrent submission of
serialising IO, and xfs_atomic_write_cow_iomap_begin() is the first
place in the IO submission path that we hit a serialising lock. That
indicates a problem with high level IO serialisation, not a problem
with the ILOCK.

> Notice how much time we spend getting that ilock.
> 
> We could try something like this:
> 
> ----8<------
> 
> [PATCH] xfs: serialize CoW-based atomic writes
> 
> We have had reports of system hangs when testing Cow-based atomic writes
> for large block sizes.
> 
> Analysis has shown large contention on the the per-inode ilock,
> superficially in creating the mapping in
> xfs_atomic_write_cow_iomap_begin().
> 
> Serialize atomic writes to reduce this contention. In some cases this may
> reduce performance, but high performance has not been a requirement so far
> for CoW-based atomic writes.
> 
> Signed-off-by: John Garry <john.garry@linux.dev>
> 
> diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
> index d8202da15aca..ff48757cf0f0 100644
> --- a/fs/xfs/xfs_file.c
> +++ b/fs/xfs/xfs_file.c
> @@ -819,10 +819,12 @@ xfs_file_dio_write_atomic(
>  	 * HW offload should be faster, so try that first if it is already
>  	 * known that the write length is not too large.
>  	 */
> -	if (ocount > xfs_inode_buftarg(ip)->bt_awu_max)
> +	if (ocount > xfs_inode_buftarg(ip)->bt_awu_max) {
>  		dops = &xfs_atomic_write_cow_iomap_ops;
> -	else
> +		dio_flags |= IOMAP_DIO_FORCE_WAIT;
> +	} else {
>  		dops = &xfs_direct_write_iomap_ops;
> +	}
> 
>  retry:
>  	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
> @@ -840,6 +842,8 @@ xfs_file_dio_write_atomic(
>  	}
> 
>  	trace_xfs_file_direct_write(iocb, from);
> +	if (dio_flags & IOMAP_DIO_FORCE_WAIT)
> +		inode_dio_wait(VFS_I(ip));

Based on my comment above, this seems like the wrong approach to me.

We serialise concurrent DIO write submission when necessary by
taking the IOLOCK_EXCL instead of IOLOCK_SHARED (e.g. EOF
extension). inode_dio_wait() waits for completion of all DIO (read
and write), but we don't need to wait for completions if all we need
to do is serialise IO submission.

Hence if we know we are going to do a software COW for the write, we
should take IOLOCK_EXCL to serialise submission up high. Waiting
until we are deep into the IO path and serialising on the 
internal metadata ILOCK is much, much harder to do efficiently and
correctly.

i.e. we need to take the ILOCK vs transaction reservation vs unbound
user-controlled concurrency deadlock problem out of
the equation. Using the IOLOCK achieves this because IO submission
blocks on the IOLOCK rather than on the ILOCK whilst holding a
transaction reservation.

> I also notice that in xfs_atomic_write_cow_iomap_next() we drop the ilock
> for calling xfs_trans_alloc_inode() (which immediately grabs the same lock).
> It would seem that could be improved.

Doubt it - holding the ILOCK across transaction reservation
guarantees journal space deadlocks will occur.  That's why the
suboptimal pattern of 'lock - check - unlock - trans_alloc - lock -
check' exists throughout XFS inode modification paths.

FWIW, that's one of the anit-patterns that the patchset I posted
recently begins to address:

https://lore.kernel.org/linux-xfs/20260819001442.1451892-1-dgc@kernel.org/

It creates consistently correct patterns in the IO path by changes all the
individual alloc-lock-check-mod-commit-unlock loops to an "atomic"
alloc-lock, check-mod-roll loop, commit-unlock pattern.

IOWs, multi-part extent modifications are all done under a single
lock/unlock pair, and so appear atomic to all other extent map and
modification operations. This makes many of the nasty IO corner
cases that lead to sub-optimal behaviour (like data corruption or
lock contention in COW workloads due to continual bouncing of the
ILOCK between submission and completion contexts) largely go away...

-Dave.
-- 
Dave Chinner
dgc@kernel.org

Re: [bug report] fstests generic/774 hang again

[John Garry]

On 9/16/26 23:23, Dave Chinner wrote:
> On Wed, Sep 16, 2026 at 11:23:16AM +0100, John Garry wrote:
>> On 15/09/2026 16:41, John Garry wrote:
>>> On 9/15/26 15:50, Darrick J. Wong wrote:
>>>>> thanks
>>>>>
>>>>> About the kernel code, I am wondering if using IOMAP_DIO_FORCE_WAIT for
>>>>> CoW-based atomics could help avoid this issue as we seem to be
>>>>> bogged down
>>>>> in lock contention.
>>>> Which lock is being contended, anyway?  It looks like the ILOCK?
>>>
>>> Yeah, I think so Re. ilock. I had some perf data illustrating this from
>>> last year, which I can't seem to find, so I will re-generate it.
>>
>> Here is perf call graph snippet when running fio with 72x threads issuing
>> 4KB atomic writes on 1MB file:
>>
>> --50.46%--xfs_file_dio_write_atomic
>> |
>> |--37.19%--iomap_dio_rw
>> |          |
>> |      --37.14%--__iomap_dio_rw
>> |           |
>> |       --36.50%--iomap_iter
>> |            |
>> |             --36.49%--xfs_atomic_write_cow_iomap_next
>> |                  |
>> |                  |--27.34%--xfs_ilock
>> |                  |          |
>> |                  |      --27.34%--down_write
>> |                  |           |
>> |                  |       --27.28%--rwsem_down_write_slowpath
>> |                  |            |
>> |                  |            |--24.65%--osq_lock
>> |                  |            |
>> |                  |             --2.35%--rwsem_spin_on_owner
>> |                  |
>> |                  |--7.62%--xfs_trans_alloc_inode
>> |                  |          |
>> |                  |      --7.59%--xfs_ilock
> 
> Where's the other half of the lock contention?

I was expecting xfs_reflink_end_atomic_cow() -> xfs_ilock() to be a 
point of contention as well, but it does not show up. I suppose that it 
because all those threads at completion stage are sleeping.

However, there is this which I omitted:

--13.21%--xfs_file_write_checks
    |
       --13.18%--kiocb_modified
          |
             --13.17%--file_update_time_flags
                |
                   --13.16%--xfs_vn_update_time
                      |
                         --12.99%--xfs_ilock
                            |
                               --12.98%--down_write

> 
> If this is all there is, it's because of concurrent submission of
> serialising IO, and xfs_atomic_write_cow_iomap_begin() is the first
> place in the IO submission path that we hit a serialising lock. That
> indicates a problem with high level IO serialisation, not a problem
> with the ILOCK.
> 
>> Notice how much time we spend getting that ilock.
>>
>> We could try something like this:
>>
>> ----8<------
>>
>> [PATCH] xfs: serialize CoW-based atomic writes
>>
>> We have had reports of system hangs when testing Cow-based atomic writes
>> for large block sizes.
>>
>> Analysis has shown large contention on the the per-inode ilock,
>> superficially in creating the mapping in
>> xfs_atomic_write_cow_iomap_begin().
>>
>> Serialize atomic writes to reduce this contention. In some cases this may
>> reduce performance, but high performance has not been a requirement so far
>> for CoW-based atomic writes.
>>
>> Signed-off-by: John Garry <john.garry@linux.dev>
>>
>> diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
>> index d8202da15aca..ff48757cf0f0 100644
>> --- a/fs/xfs/xfs_file.c
>> +++ b/fs/xfs/xfs_file.c
>> @@ -819,10 +819,12 @@ xfs_file_dio_write_atomic(
>>   	 * HW offload should be faster, so try that first if it is already
>>   	 * known that the write length is not too large.
>>   	 */
>> -	if (ocount > xfs_inode_buftarg(ip)->bt_awu_max)
>> +	if (ocount > xfs_inode_buftarg(ip)->bt_awu_max) {
>>   		dops = &xfs_atomic_write_cow_iomap_ops;
>> -	else
>> +		dio_flags |= IOMAP_DIO_FORCE_WAIT;
>> +	} else {
>>   		dops = &xfs_direct_write_iomap_ops;
>> +	}
>>
>>   retry:
>>   	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
>> @@ -840,6 +842,8 @@ xfs_file_dio_write_atomic(
>>   	}
>>
>>   	trace_xfs_file_direct_write(iocb, from);
>> +	if (dio_flags & IOMAP_DIO_FORCE_WAIT)
>> +		inode_dio_wait(VFS_I(ip));
> 
> Based on my comment above, this seems like the wrong approach to me.
> 
> We serialise concurrent DIO write submission when necessary by
> taking the IOLOCK_EXCL instead of IOLOCK_SHARED (e.g. EOF
> extension). inode_dio_wait() waits for completion of all DIO (read
> and write), but we don't need to wait for completions if all we need
> to do is serialise IO submission.
> 
> Hence if we know we are going to do a software COW for the write, we
> should take IOLOCK_EXCL to serialise submission up high. Waiting
> until we are deep into the IO path and serialising on the
> internal metadata ILOCK is much, much harder to do efficiently and
> correctly.
> 
> i.e. we need to take the ILOCK vs transaction reservation vs unbound
> user-controlled concurrency deadlock problem out of
> the equation. Using the IOLOCK achieves this because IO submission
> blocks on the IOLOCK rather than on the ILOCK whilst holding a
> transaction reservation.
> 
>> I also notice that in xfs_atomic_write_cow_iomap_next() we drop the ilock
>> for calling xfs_trans_alloc_inode() (which immediately grabs the same lock).
>> It would seem that could be improved.
> 
> Doubt it - holding the ILOCK across transaction reservation
> guarantees journal space deadlocks will occur.  That's why the
> suboptimal pattern of 'lock - check - unlock - trans_alloc - lock -
> check' exists throughout XFS inode modification paths.
> 
> FWIW, that's one of the anit-patterns that the patchset I posted
> recently begins to address:
> 
> https://lore.kernel.org/linux-xfs/20260819001442.1451892-1-dgc@kernel.org/
> 
> It creates consistently correct patterns in the IO path by changes all the
> individual alloc-lock-check-mod-commit-unlock loops to an "atomic"
> alloc-lock, check-mod-roll loop, commit-unlock pattern.
> 
> IOWs, multi-part extent modifications are all done under a single
> lock/unlock pair, and so appear atomic to all other extent map and
> modification operations. This makes many of the nasty IO corner
> cases that lead to sub-optimal behaviour (like data corruption or
> lock contention in COW workloads due to continual bouncing of the
> ILOCK between submission and completion contexts) largely go away...
> 

Ack, and unfortunately that series did not get much expert review.

JFYI, I have another log for a similar hang which we see one thread 
deadlock for log space, below.

fio-3.38
Starting 14 processes
[  247.169311] INFO: task kworker/34:1:568 blocked for more than 122 
seconds.
[  247.169355]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.169370] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  247.169372] task:kworker/34:1    state:D stack:23048 pid:568 
tgid:568   ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.169388] Workqueue: dio/sda iomap_dio_complete_work
[  247.169424] Call Trace:
[  247.169427]  <TASK>
[  247.169431]  __schedule+0x1179/0x44c0
[  247.169446]  ? kick_pool_pick+0x24c/0x640
[  247.169458]  ? __pfx___schedule+0x10/0x10
[  247.169464]  ? lock_release+0xc9/0x290
[  247.169471]  schedule+0xd2/0x260
[  247.169476]  schedule_preempt_disabled+0x11/0x20
[  247.169479]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.169486]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.169494]  ? xfs_dec_freecounter+0xec/0x580
[  247.169501]  down_write_nested+0x1b5/0x1e0
[  247.169508]  ? __pfx_down_write_nested+0x10/0x10
[  247.169512]  ? xfs_ilock+0x2a0/0x3d0
[  247.169518]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.169523]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.169527]  ? __lock_acquire+0x607/0x1f70
[  247.169532]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.169539]  iomap_dio_complete+0x136/0xaa0
[  247.169543]  ? process_scheduled_works+0xaee/0x1b30
[  247.169546]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.169552]  iomap_dio_complete_work+0x51/0x80
[  247.169555]  process_scheduled_works+0xc08/0x1b30
[  247.169570]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.169573]  ? lock_acquire+0x185/0x2e0
[  247.169577]  ? lock_is_held_type+0x9e/0x120
[  247.169582]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.169586]  worker_thread+0x4ad/0xb40
[  247.169591]  ? __kthread_parkme+0x99/0x180
[  247.169596]  ? __pfx_worker_thread+0x10/0x10
[  247.169600]  kthread+0x351/0x450
[  247.169603]  ? __pfx_kthread+0x10/0x10
[  247.169606]  ret_from_fork+0x563/0x800
[  247.169618]  ? __pfx_ret_from_fork+0x10/0x10
[  247.169621]  ? __pfx_kthread+0x10/0x10
[  247.169624]  ? __switch_to+0x58e/0xea0
[  247.169630]  ? __pfx_kthread+0x10/0x10
[  247.169632]  ret_from_fork_asm+0x1a/0x30
[  247.169651]  </TASK>
[  247.170111] INFO: task kworker/34:1:568 <writer> blocked on an 
rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.170169] INFO: task kworker/48:12:800 blocked for more than 122 
seconds.
[  247.170172]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.170173] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  247.170174] task:kworker/48:12   state:D stack:26088 pid:800 
tgid:800   ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.170188] Workqueue: dio/sda iomap_dio_complete_work
[  247.170196] Call Trace:
[  247.170198]  <TASK>
[  247.170200]  __schedule+0x1179/0x44c0
[  247.170208]  ? kick_pool_pick+0x24c/0x640
[  247.170216]  ? __pfx___schedule+0x10/0x10
[  247.170220]  ? lock_release+0xc9/0x290
[  247.170227]  schedule+0xd2/0x260
[  247.170231]  schedule_preempt_disabled+0x11/0x20
[  247.170234]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.170240]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.170249]  ? xfs_dec_freecounter+0xec/0x580
[  247.170254]  down_write_nested+0x1b5/0x1e0
[  247.170260]  ? __pfx_down_write_nested+0x10/0x10
[  247.170264]  ? xfs_ilock+0x2a0/0x3d0
[  247.170269]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.170274]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.170278]  ? __lock_acquire+0x607/0x1f70
[  247.170283]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.170289]  iomap_dio_complete+0x136/0xaa0
[  247.170293]  ? process_scheduled_works+0xaee/0x1b30
[  247.170296]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.170302]  iomap_dio_complete_work+0x51/0x80
[  247.170305]  process_scheduled_works+0xc08/0x1b30
[  247.170319]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.170323]  ? lock_acquire+0x185/0x2e0
[  247.170327]  ? lock_is_held_type+0x9e/0x120
[  247.170331]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.170335]  worker_thread+0x4ad/0xb40
[  247.170342]  ? __pfx_worker_thread+0x10/0x10
[  247.170345]  kthread+0x351/0x450
[  247.170348]  ? __pfx_kthread+0x10/0x10
[  247.170351]  ret_from_fork+0x563/0x800
[  247.170358]  ? __pfx_ret_from_fork+0x10/0x10
[  247.170371]  ? __pfx_kthread+0x10/0x10
[  247.170374]  ? __switch_to+0x58e/0xea0
[  247.170378]  ? __pfx_kthread+0x10/0x10
[  247.170381]  ret_from_fork_asm+0x1a/0x30
[  247.170391]  </TASK>
[  247.170688] INFO: task kworker/48:12:800 <writer> blocked on an 
rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.170694] INFO: task kworker/48:13:807 blocked for more than 122 
seconds.
[  247.170696]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.170698] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  247.170699] task:kworker/48:13   state:D stack:26088 pid:807 
tgid:807   ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.170708] Workqueue: dio/sda iomap_dio_complete_work
[  247.170713] Call Trace:
[  247.170715]  <TASK>
[  247.170718]  __schedule+0x1179/0x44c0
[  247.170722]  ? kick_pool_pick+0x24c/0x640
[  247.170729]  ? __pfx___schedule+0x10/0x10
[  247.170733]  ? lock_release+0xc9/0x290
[  247.170739]  schedule+0xd2/0x260
[  247.170742]  schedule_preempt_disabled+0x11/0x20
[  247.170746]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.170752]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.170759]  ? xfs_dec_freecounter+0xec/0x580
[  247.170764]  down_write_nested+0x1b5/0x1e0
[  247.170769]  ? __pfx_down_write_nested+0x10/0x10
[  247.170774]  ? xfs_ilock+0x2a0/0x3d0
[  247.170778]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.170781]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.170786]  ? __lock_acquire+0x607/0x1f70
[  247.170791]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.170796]  iomap_dio_complete+0x136/0xaa0
[  247.170799]  ? process_scheduled_works+0xaee/0x1b30
[  247.170802]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.170807]  iomap_dio_complete_work+0x51/0x80
[  247.170810]  process_scheduled_works+0xc08/0x1b30
[  247.170824]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.170827]  ? lock_acquire+0x185/0x2e0
[  247.170831]  ? lock_is_held_type+0x9e/0x120
[  247.170835]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.170839]  worker_thread+0x4ad/0xb40
[  247.170845]  ? __pfx_worker_thread+0x10/0x10
[  247.170848]  kthread+0x351/0x450
[  247.170850]  ? __pfx_kthread+0x10/0x10
[  247.170853]  ret_from_fork+0x563/0x800
[  247.170858]  ? __pfx_ret_from_fork+0x10/0x10
[  247.170861]  ? __pfx_kthread+0x10/0x10
[  247.170863]  ? __switch_to+0x58e/0xea0
[  247.170867]  ? __pfx_kthread+0x10/0x10
[  247.170870]  ret_from_fork_asm+0x1a/0x30
[  247.170878]  </TASK>
[  247.171134] INFO: task kworker/48:13:807 <writer> blocked on an 
rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.171153] INFO: task kworker/52:22:893 blocked for more than 122 
seconds.
[  247.171155]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.171156] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  247.171157] task:kworker/52:22   state:D stack:26280 pid:893 
tgid:893   ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.171166] Workqueue: dio/sda iomap_dio_complete_work
[  247.171171] Call Trace:
[  247.171172]  <TASK>
[  247.171174]  __schedule+0x1179/0x44c0
[  247.171179]  ? kick_pool_pick+0x24c/0x640
[  247.171186]  ? __pfx___schedule+0x10/0x10
[  247.171189]  ? lock_release+0xc9/0x290
[  247.171195]  schedule+0xd2/0x260
[  247.171198]  schedule_preempt_disabled+0x11/0x20
[  247.171202]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.171208]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.171216]  ? xfs_dec_freecounter+0xec/0x580
[  247.171221]  down_write_nested+0x1b5/0x1e0
[  247.171225]  ? __pfx_down_write_nested+0x10/0x10
[  247.171230]  ? xfs_ilock+0x2a0/0x3d0
[  247.171234]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.171238]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.171242]  ? __lock_acquire+0x607/0x1f70
[  247.171248]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.171253]  iomap_dio_complete+0x136/0xaa0
[  247.171256]  ? process_scheduled_works+0xaee/0x1b30
[  247.171259]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.171264]  iomap_dio_complete_work+0x51/0x80
[  247.171267]  process_scheduled_works+0xc08/0x1b30
[  247.171281]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.171285]  ? lock_acquire+0x185/0x2e0
[  247.171289]  ? lock_is_held_type+0x9e/0x120
[  247.171293]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.171297]  worker_thread+0x4ad/0xb40
[  247.171303]  ? __pfx_worker_thread+0x10/0x10
[  247.171306]  kthread+0x351/0x450
[  247.171308]  ? __pfx_kthread+0x10/0x10
[  247.171311]  ret_from_fork+0x563/0x800
[  247.171315]  ? __pfx_ret_from_fork+0x10/0x10
[  247.171318]  ? __pfx_kthread+0x10/0x10
[  247.171320]  ? __switch_to+0x58e/0xea0
[  247.171324]  ? __pfx_kthread+0x10/0x10
[  247.171327]  ret_from_fork_asm+0x1a/0x30
[  247.171335]  </TASK>
[  247.171539] INFO: task kworker/52:22:893 <writer> blocked on an 
rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.171555] INFO: task kworker/5:4:1006 blocked for more than 122 
seconds.
[  247.171557]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.171558] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  247.171559] task:kworker/5:4     state:D stack:26456 pid:1006 
tgid:1006  ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.171566] Workqueue: dio/sda iomap_dio_complete_work
[  247.171569] Call Trace:
[  247.171570]  <TASK>
[  247.171572]  __schedule+0x1179/0x44c0
[  247.171576]  ? kick_pool_pick+0x24c/0x640
[  247.171582]  ? __pfx___schedule+0x10/0x10
[  247.171586]  ? lock_release+0xc9/0x290
[  247.171591]  schedule+0xd2/0x260
[  247.171595]  schedule_preempt_disabled+0x11/0x20
[  247.171597]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.171603]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.171611]  ? xfs_dec_freecounter+0xec/0x580
[  247.171616]  down_write_nested+0x1b5/0x1e0
[  247.171619]  ? __pfx_down_write_nested+0x10/0x10
[  247.171623]  ? xfs_ilock+0x2a0/0x3d0
[  247.171627]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.171631]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.171635]  ? __lock_acquire+0x607/0x1f70
[  247.171640]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.171657]  iomap_dio_complete+0x136/0xaa0
[  247.171661]  ? process_scheduled_works+0xaee/0x1b30
[  247.171663]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.171667]  iomap_dio_complete_work+0x51/0x80
[  247.171670]  process_scheduled_works+0xc08/0x1b30
[  247.171684]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.171687]  ? lock_acquire+0x185/0x2e0
[  247.171691]  ? lock_is_held_type+0x9e/0x120
[  247.171695]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.171699]  worker_thread+0x4ad/0xb40
[  247.171705]  ? __pfx_worker_thread+0x10/0x10
[  247.171709]  kthread+0x351/0x450
[  247.171711]  ? __pfx_kthread+0x10/0x10
[  247.171714]  ret_from_fork+0x563/0x800
[  247.171718]  ? __pfx_ret_from_fork+0x10/0x10
[  247.171721]  ? __pfx_kthread+0x10/0x10
[  247.171723]  ? __switch_to+0x58e/0xea0
[  247.171726]  ? __pfx_kthread+0x10/0x10
[  247.171729]  ret_from_fork_asm+0x1a/0x30
[  247.171737]  </TASK>
[  247.171902] INFO: task kworker/5:4:1006 <writer> blocked on an 
rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.171905] INFO: task kworker/5:6:1009 blocked for more than 122 
seconds.
[  247.171906]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.171908] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  247.171908] task:kworker/5:6     state:D stack:25000 pid:1009 
tgid:1009  ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.171915] Workqueue: dio/sda iomap_dio_complete_work
[  247.171919] Call Trace:
[  247.171920]  <TASK>
[  247.171922]  __schedule+0x1179/0x44c0
[  247.171929]  ? __pfx___schedule+0x10/0x10
[  247.171933]  ? lock_release+0xc9/0x290
[  247.171935]  ? _raw_spin_unlock_irq+0x23/0x40
[  247.171940]  schedule+0xd2/0x260
[  247.171944]  schedule_preempt_disabled+0x11/0x20
[  247.171947]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.171952]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.171960]  ? xfs_dec_freecounter+0xec/0x580
[  247.171965]  down_write_nested+0x1b5/0x1e0
[  247.171968]  ? __pfx_down_write_nested+0x10/0x10
[  247.171973]  ? xfs_ilock+0x2a0/0x3d0
[  247.171976]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.171980]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.171984]  ? __lock_acquire+0x607/0x1f70
[  247.171989]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.171994]  iomap_dio_complete+0x136/0xaa0
[  247.171997]  ? process_scheduled_works+0xaee/0x1b30
[  247.172000]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.172004]  iomap_dio_complete_work+0x51/0x80
[  247.172007]  process_scheduled_works+0xc08/0x1b30
[  247.172021]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.172024]  ? lock_acquire+0x185/0x2e0
[  247.172028]  ? lock_is_held_type+0x9e/0x120
[  247.172032]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.172036]  worker_thread+0x4ad/0xb40
[  247.172041]  ? __kthread_parkme+0x99/0x180
[  247.172045]  ? __pfx_worker_thread+0x10/0x10
[  247.172062]  kthread+0x351/0x450
[  247.172065]  ? __pfx_kthread+0x10/0x10
[  247.172068]  ret_from_fork+0x563/0x800
[  247.172072]  ? __pfx_ret_from_fork+0x10/0x10
[  247.172075]  ? __pfx_kthread+0x10/0x10
[  247.172077]  ? __switch_to+0x58e/0xea0
[  247.172081]  ? __pfx_kthread+0x10/0x10
[  247.172084]  ret_from_fork_asm+0x1a/0x30
[  247.172092]  </TASK>
[  247.172277] INFO: task kworker/5:6:1009 <writer> blocked on an 
rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.172282] INFO: task kworker/7:3:1019 blocked for more than 122 
seconds.
[  247.172283]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.172285] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  247.172286] task:kworker/7:3     state:D stack:25912 pid:1019 
tgid:1019  ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.172292] Workqueue: dio/sda iomap_dio_complete_work
[  247.172295] Call Trace:
[  247.172296]  <TASK>
[  247.172299]  __schedule+0x1179/0x44c0
[  247.172302]  ? kick_pool_pick+0x24c/0x640
[  247.172308]  ? __pfx___schedule+0x10/0x10
[  247.172312]  ? lock_release+0xc9/0x290
[  247.172317]  schedule+0xd2/0x260
[  247.172321]  xlog_grant_head_wait+0x112/0x9b0
[  247.172327]  xlog_grant_head_check+0x377/0x500
[  247.172331]  ? __pfx_xlog_grant_head_check+0x10/0x10
[  247.172337]  xfs_log_reserve+0x291/0x6f0
[  247.172343]  ? __pfx_xfs_log_reserve+0x10/0x10
[  247.172346]  ? __percpu_counter_compare+0xa8/0xf0
[  247.172352]  ? xfs_dec_freecounter+0xec/0x580
[  247.172357]  xfs_trans_alloc+0x205/0x900
[  247.172363]  xfs_reflink_end_atomic_cow+0x197/0x640
[  247.172367]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.172372]  ? __lock_acquire+0x607/0x1f70
[  247.172378]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.172383]  iomap_dio_complete+0x136/0xaa0
[  247.172386]  ? process_scheduled_works+0xaee/0x1b30
[  247.172388]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.172392]  iomap_dio_complete_work+0x51/0x80
[  247.172396]  process_scheduled_works+0xc08/0x1b30
[  247.172411]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.172415]  ? lock_acquire+0x185/0x2e0
[  247.172419]  ? lock_is_held_type+0x9e/0x120
[  247.172424]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.172428]  worker_thread+0x4ad/0xb40
[  247.172434]  ? __pfx_worker_thread+0x10/0x10
[  247.172437]  kthread+0x351/0x450
[  247.172439]  ? __pfx_kthread+0x10/0x10
[  247.172442]  ret_from_fork+0x563/0x800
[  247.172445]  ? __pfx_ret_from_fork+0x10/0x10
[  247.172448]  ? __pfx_kthread+0x10/0x10
[  247.172450]  ? __switch_to+0x58e/0xea0
[  247.172454]  ? __pfx_kthread+0x10/0x10
[  247.172457]  ret_from_fork_asm+0x1a/0x30
[  247.172465]  </TASK>
[  247.172471] INFO: task kworker/61:4:1051 blocked for more than 122 
seconds.
[  247.172472]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.172473] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  247.172474] task:kworker/61:4    state:D stack:26016 pid:1051 
tgid:1051  ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.172481] Workqueue: dio/sda iomap_dio_complete_work
[  247.172484] Call Trace:
[  247.172485]  <TASK>
[  247.172487]  __schedule+0x1179/0x44c0
[  247.172490]  ? kick_pool_pick+0x24c/0x640
[  247.172496]  ? __pfx___schedule+0x10/0x10
[  247.172500]  ? lock_release+0xc9/0x290
[  247.172504]  schedule+0xd2/0x260
[  247.172508]  schedule_preempt_disabled+0x11/0x20
[  247.172511]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.172517]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.172526]  ? xfs_dec_freecounter+0xec/0x580
[  247.172531]  down_write_nested+0x1b5/0x1e0
[  247.172534]  ? __pfx_down_write_nested+0x10/0x10
[  247.172539]  ? xfs_ilock+0x2a0/0x3d0
[  247.172543]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.172546]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.172550]  ? __lock_acquire+0x607/0x1f70
[  247.172555]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.172560]  iomap_dio_complete+0x136/0xaa0
[  247.172563]  ? process_scheduled_works+0xaee/0x1b30
[  247.172565]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.172569]  iomap_dio_complete_work+0x51/0x80
[  247.172572]  process_scheduled_works+0xc08/0x1b30
[  247.172586]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.172589]  ? lock_acquire+0x185/0x2e0
[  247.172594]  ? lock_is_held_type+0x9e/0x120
[  247.172597]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.172601]  worker_thread+0x4ad/0xb40
[  247.172607]  ? __pfx_worker_thread+0x10/0x10
[  247.172610]  kthread+0x351/0x450
[  247.172612]  ? __pfx_kthread+0x10/0x10
[  247.172615]  ret_from_fork+0x563/0x800
[  247.172618]  ? __pfx_ret_from_fork+0x10/0x10
[  247.172621]  ? __pfx_kthread+0x10/0x10
[  247.172623]  ? __switch_to+0x58e/0xea0
[  247.172626]  ? __pfx_kthread+0x10/0x10
[  247.172629]  ret_from_fork_asm+0x1a/0x30
[  247.172637]  </TASK>
[  247.172804] INFO: task kworker/61:4:1051 <writer> blocked on an 
rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.172810] INFO: task kworker/34:2:1071 blocked for more than 122 
seconds.
[  247.172812]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.172813] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  247.172814] task:kworker/34:2    state:D stack:26360 pid:1071 
tgid:1071  ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.172820] Workqueue: dio/sda iomap_dio_complete_work
[  247.172823] Call Trace:
[  247.172824]  <TASK>
[  247.172827]  __schedule+0x1179/0x44c0
[  247.172830]  ? kick_pool_pick+0x24c/0x640
[  247.172836]  ? __pfx___schedule+0x10/0x10
[  247.172840]  ? lock_release+0xc9/0x290
[  247.172844]  schedule+0xd2/0x260
[  247.172848]  schedule_preempt_disabled+0x11/0x20
[  247.172851]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.172857]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.172865]  ? xfs_dec_freecounter+0xec/0x580
[  247.172869]  down_write_nested+0x1b5/0x1e0
[  247.172872]  ? __pfx_down_write_nested+0x10/0x10
[  247.172876]  ? xfs_ilock+0x2a0/0x3d0
[  247.172880]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.172884]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.172888]  ? __lock_acquire+0x607/0x1f70
[  247.172893]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.172898]  iomap_dio_complete+0x136/0xaa0
[  247.172901]  ? process_scheduled_works+0xaee/0x1b30
[  247.172903]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.172907]  iomap_dio_complete_work+0x51/0x80
[  247.172910]  process_scheduled_works+0xc08/0x1b30
[  247.172924]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.172927]  ? lock_acquire+0x185/0x2e0
[  247.172931]  ? lock_is_held_type+0x9e/0x120
[  247.172935]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.172939]  worker_thread+0x4ad/0xb40
[  247.172944]  ? __pfx_worker_thread+0x10/0x10
[  247.172947]  kthread+0x351/0x450
[  247.172949]  ? __pfx_kthread+0x10/0x10
[  247.172952]  ret_from_fork+0x563/0x800
[  247.172956]  ? __pfx_ret_from_fork+0x10/0x10
[  247.172959]  ? __pfx_kthread+0x10/0x10
[  247.172961]  ? __switch_to+0x58e/0xea0
[  247.172964]  ? __pfx_kthread+0x10/0x10
[  247.172967]  ret_from_fork_asm+0x1a/0x30
[  247.172975]  </TASK>
[  247.173163] INFO: task kworker/34:2:1071 <writer> blocked on an 
rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.173167] INFO: task kworker/6:5:1080 blocked for more than 122 
seconds.
[  247.173169]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.173170] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  247.173171] task:kworker/6:5     state:D stack:26424 pid:1080 
tgid:1080  ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.173178] Workqueue: dio/sda iomap_dio_complete_work
[  247.173181] Call Trace:
[  247.173181]  <TASK>
[  247.173184]  __schedule+0x1179/0x44c0
[  247.173187]  ? kick_pool_pick+0x24c/0x640
[  247.173193]  ? __pfx___schedule+0x10/0x10
[  247.173197]  ? lock_release+0xc9/0x290
[  247.173201]  schedule+0xd2/0x260
[  247.173205]  xlog_grant_head_wait+0x112/0x9b0
[  247.173211]  xlog_grant_head_check+0x377/0x500
[  247.173214]  ? __pfx_xlog_grant_head_check+0x10/0x10
[  247.173221]  xfs_log_reserve+0x291/0x6f0
[  247.173225]  ? __pfx_xfs_log_reserve+0x10/0x10
[  247.173228]  ? __percpu_counter_compare+0xa8/0xf0
[  247.173233]  ? xfs_dec_freecounter+0xec/0x580
[  247.173237]  xfs_trans_alloc+0x205/0x900
[  247.173244]  xfs_reflink_end_atomic_cow+0x197/0x640
[  247.173248]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.173252]  ? __lock_acquire+0x607/0x1f70
[  247.173257]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.173262]  iomap_dio_complete+0x136/0xaa0
[  247.173265]  ? process_scheduled_works+0xaee/0x1b30
[  247.173268]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.173272]  iomap_dio_complete_work+0x51/0x80
[  247.173275]  process_scheduled_works+0xc08/0x1b30
[  247.173288]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.173291]  ? lock_acquire+0x185/0x2e0
[  247.173295]  ? lock_is_held_type+0x9e/0x120
[  247.173299]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.173303]  worker_thread+0x4ad/0xb40
[  247.173309]  ? __pfx_worker_thread+0x10/0x10
[  247.173312]  kthread+0x351/0x450
[  247.173314]  ? __pfx_kthread+0x10/0x10
[  247.173317]  ret_from_fork+0x563/0x800
[  247.173320]  ? __pfx_ret_from_fork+0x10/0x10
[  247.173323]  ? __pfx_kthread+0x10/0x10
[  247.173325]  ? __switch_to+0x58e/0xea0
[  247.173328]  ? __pfx_kthread+0x10/0x10
[  247.173331]  ret_from_fork_asm+0x1a/0x30
[  247.173339]  </TASK>
[  247.173340] Future hung task reports are suppressed, see sysctl 
kernel.hung_task_warnings
[  247.173778]
[  247.173778] Showing all locks held in the system:
[  247.173782] locks held by pr/ttyS0/15: 2, last CPU#41:
[  247.173784]  #0: ffffffff892141f8 (console_srcu){....}-{0:0}, at: 
console_srcu_read_lock+0x46/0x60
[  247.173792]  #1: ffffffff8c4893d8 (&port_lock_key){-...}-{3:3}, at: 
univ8250_console_device_lock+0x56/0xa0
[  247.173839] locks held by khungtaskd/356: 1, last CPU#40:
[  247.173841]  #0: ffffffff89221460 (rcu_read_lock){....}-{1:3}, at: 
debug_show_all_locks+0x36/0x1c0
[  247.173858] locks held by syslogd/464: 1, last CPU#59:
[  247.173860]  #0: ffff888105817cc8 (&u->iolock){+.+.}-{4:4}, at: 
__unix_dgram_recvmsg+0x1ad/0xc30
[  247.173872] locks held by kworker/34:1/568: 4, on CPU#34:
[  247.173874]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.173880]  #1: ffff88812a6a7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.173885]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.173891]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.173915] locks held by kworker/48:12/800: 4, on CPU#48:
[  247.173917]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.173922]  #1: ffff88814a0dfd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.173927]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.173932]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.173938] locks held by kworker/48:13/807: 4, on CPU#48:
[  247.173940]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.173945]  #1: ffff88814a12fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.173950]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.173955]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.173972] locks held by kworker/52:22/893: 4, on CPU#52:
[  247.173973]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.173979]  #1: ffff88814a4ffd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.173983]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.173988]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174009] locks held by kworker/5:4/1006: 4, on CPU#5:
[  247.174011]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174016]  #1: ffff88814b3efd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174021]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174026]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174032] locks held by kworker/5:6/1009: 4, on CPU#5:
[  247.174033]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174039]  #1: ffff88814b48fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174043]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174273]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174281] locks held by kworker/7:3/1019: 3, on CPU#7:
[  247.174282]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174288]  #1: ffff88814b687d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174293]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174302] locks held by kworker/61:4/1051: 4, on CPU#61:
[  247.174304]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174309]  #1: ffff88814bb0fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174314]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174319]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174327] locks held by kworker/34:2/1071: 4, on CPU#34:
[  247.174328]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174334]  #1: ffff88814bc57d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174338]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174343]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174349] locks held by kworker/6:5/1080: 3, on CPU#6:
[  247.174351]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174356]  #1: ffff88814bda7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174361]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174371] locks held by kworker/34:19/1123: 4, on CPU#34:
[  247.174373]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174378]  #1: ffff88814c187d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174383]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174387]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174395] locks held by kworker/7:10/1144: 3, on CPU#7:
[  247.174397]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174402]  #1: ffff88814c497d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174407]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174428] locks held by kworker/59:3/1308: 3, on CPU#59:
[  247.174430]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174435]  #1: ffff88813524fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174440]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174445] locks held by kworker/59:4/1309: 3, on CPU#59:
[  247.174446]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
Jobs: 14 (f=14):[  247.174452]  #1: ffff8881036b7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
  [W(14)][10.5%][[  247.174457]  #2: ffff88813af38628 
(sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174462] locks held by kworker/59:5/1310: 4, on CPU#59:
[  247.174463]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174469]  #1: ffff888135e6fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174473]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174478]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174498] locks held by kworker/28:2/1449: 4, on CPU#28:
[  247.174499]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174505]  #1: ffff888148d87d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174510]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174515]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174521] locks held by kworker/28:4/1454: 4, on CPU#28:
[  247.174522]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174527]  #1: ffff888148c7fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174532]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174537]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174542] locks held by kworker/28:6/1457: 3, on CPU#28:
[  247.174544]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174549]  #1: ffff888148b5fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174554]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174559] locks held by kworker/28:8/1461: 4, on CPU#28:
[  247.174561]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174566]  #1: ffff888148c5fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174571]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174576]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174588] locks held by kworker/28:9/1537: 4, on CPU#28:
[  247.174590]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174595]  #1: ffff888147537d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174600]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174605]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174612] locks held by kworker/28:13/1549: 4, on CPU#28:
[  247.174613]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174618]  #1: ffff88814770fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174623]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174628]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174635] locks held by kworker/60:10/1577: 4, on CPU#60:
[  247.174636]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174642]  #1: ffff8881446ffd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174656]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174661]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174666] locks held by kworker/60:11/1578: 4, on CPU#60:
[  247.174667]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174673]  #1: ffff888144707d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174677]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174682]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174699] locks held by kworker/59:10/1722: 3, on CPU#59:
[  247.174701]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174706]  #1: ffff888143dffd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174711]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174717] locks held by kworker/59:11/1726: 4, on CPU#59:
[  247.174719]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174724]  #1: ffff888143e2fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174729]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174733]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174739] locks held by kworker/59:12/1727: 4, on CPU#59:
[  247.174740]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174746]  #1: ffff888143e37d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174751]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174755]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174761] locks held by kworker/59:14/1729: 3, on CPU#59:
[  247.174762]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174767]  #1: ffff888143e4fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174772]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174780] locks held by kworker/59:18/1772: 4, on CPU#59:
[  247.174782]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174787]  #1: ffff888147097d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174792]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174797]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174804] locks held by kworker/59:20/1800: 4, on CPU#59:
[  247.174805]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174810]  #1: ffff8881471cfd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174815]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174820]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174828] locks held by kworker/25:7/1821: 4, on CPU#25:
[  247.174829]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174835]  #1: ffff888146b9fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174840]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174845]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174850] locks held by kworker/25:13/1834: 2, last CPU#25:
[  247.174852]  #0: ffff88810028cd40 
((wq_completion)events_power_efficient){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174857]  #1: ffff88814642fd30 
((work_completion)(&(&gc_work->dwork)->work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174863] locks held by kworker/48:36/1852: 3, on CPU#48:
[  247.174865]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174870]  #1: ffff88813f75fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174875]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174880] locks held by kworker/48:38/1854: 3, on CPU#48:
[  247.174881]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174887]  #1: ffff88812bfffd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174892]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174897] locks held by kworker/48:41/1859: 4, on CPU#48:
[  247.174899]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174904]  #1: ffff88814654fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174909]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174914]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174919] locks held by kworker/60:26/1863: 4, on CPU#60:
[  247.174920]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174926]  #1: ffff888146577d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174930]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174935]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174945] locks held by kworker/5:10/1933: 4, on CPU#5:
[  247.174947]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174952]  #1: ffff888146367d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174957]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174962]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174967] locks held by kworker/5:13/1936: 3, on CPU#5:
[  247.174968]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174974]  #1: ffff88814637fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174979]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.174984] locks held by kworker/5:14/1937: 3, on CPU#5:
[  247.174985]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.174990]  #1: ffff88814638fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.174995]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175000] locks held by kworker/5:15/1938: 3, on CPU#5:
[  247.175002]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175007]  #1: ffff888146397d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175012]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175020] locks held by kworker/14:20/1967: 4, on CPU#14:
[  247.175022]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175027]  #1: ffff888142a0fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175032]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175037]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175071] locks held by kworker/62:21/2093: 4, on CPU#62:
[  247.175073]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175079]  #1: ffff8881454b7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175084]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175089]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175108] locks held by kworker/61:26/2190: 4, on CPU#61:
[  247.175110]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175115]  #1: ffff888145307d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175120]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175125]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175131] locks held by kworker/61:32/2196: 4, on CPU#61:
[  247.175133]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175138]  #1: ffff888145347d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175143]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175148]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175158] locks held by kworker/61:65/2234: 4, on CPU#61:
[  247.175160]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175165]  #1: ffff888144c87d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175170]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175175]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175181] locks held by kworker/61:71/2241: 4, on CPU#61:
[  247.175182]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175188]  #1: ffff888144ccfd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175193]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175198]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175204] locks held by kworker/61:73/2244: 4, on CPU#61:
[  247.175205]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175211]  #1: ffff888144cefd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175216]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175221]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175227] locks held by kworker/61:75/2246: 3, on CPU#61:
[  247.175228]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175234]  #1: ffff888144d07d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175239]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175244] locks held by kworker/61:77/2248: 3, on CPU#61:
[  247.175246]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175251]  #1: ffff888144d17d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175257]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175262] locks held by kworker/61:78/2249: 4, on CPU#61:
[  247.175264]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175270]  #1: ffff888144d27d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175275]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175280]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175286] locks held by kworker/61:80/2251: 3, on CPU#61:
[  247.175288]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175294]  #1: ffff888144d37d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175299]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175304] locks held by kworker/61:81/2252: 3, on CPU#61:
[  247.175306]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175311]  #1: ffff888144d3fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175317]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175329] locks held by kworker/35:38/2302: 4, on CPU#35:
[  247.175331]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175336]  #1: ffff8881448c7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175341]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175346]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175362] locks held by kworker/34:36/2368: 4, on CPU#34:
[  247.175364]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175370]  #1: ffff888145967d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175374]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175379]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175385] locks held by kworker/34:37/2369: 4, on CPU#34:
[  247.175386]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175392]  #1: ffff888145977d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175397]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175402]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175408] locks held by kworker/34:44/2376: 4, on CPU#34:
[  247.175410]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175415]  #1: ffff8881459c7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175420]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175425]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175448] locks held by kworker/7:27/2494: 3, on CPU#7:
[  247.175450]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175455]  #1: ffff8881432ffd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175461]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175466] locks held by kworker/6:28/2495: 4, on CPU#6:
[  247.175468]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175473]  #1: ffff888143307d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175479]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175484]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175490] locks held by kworker/7:31/2502: 3, on CPU#7:
[  247.175491]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175497]  #1: ffff88814335fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175502]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175507] locks held by kworker/6:37/2509: 3, on CPU#6:
[  247.175509]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175515]  #1: ffff8881433afd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175519]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175525] locks held by kworker/7:34/2513: 4, on CPU#7:
[  247.175526]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175532]  #1: ffff8881433d7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175537]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175542]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175547] locks held by kworker/6:44/2519: 3, on CPU#6:
[  247.175549]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175554]  #1: ffff888142c0fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175559]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175566] locks held by kworker/59:21/2537: 4, on CPU#59:
[  247.175568]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175574]  #1: ffff888142ce7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175579]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175584]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175590] locks held by kworker/59:22/2538: 3, on CPU#59:
[  247.175591]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175597]  #1: ffff888142cefd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175601]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175607] locks held by kworker/59:26/2542: 4, on CPU#59:
[  247.175608]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175614]  #1: ffff88813fe87d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175619]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175624]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175629] locks held by kworker/59:27/2543: 3, on CPU#59:
[  247.175631]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175636]  #1: ffff88813fe77d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175641]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175657] locks held by kworker/59:28/2544: 3, on CPU#59:
[  247.175659]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175664]  #1: ffff88813fcd7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175669]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175680] locks held by kworker/60:35/2583: 3, on CPU#60:
[  247.175682]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175687]  #1: ffff888142f8fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175692]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175700] locks held by kworker/26:7/2606: 4, on CPU#26:
[  247.175701]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175707]  #1: ffff888142477d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175711]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175716]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175727] locks held by kworker/14:27/2649: 3, on CPU#14:
[  247.175729]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175734]  #1: ffff888142617d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175739]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175747] locks held by kworker/14:39/2673: 4, on CPU#14:
[  247.175749]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175754]  #1: ffff888142727d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175759]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175764]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175770] locks held by kworker/14:49/2687: 4, on CPU#14:
[  247.175772]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175777]  #1: ffff8881427bfd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175782]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175787]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175792] locks held by kworker/14:50/2688: 4, on CPU#14:
[  247.175794]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175799]  #1: ffff8881427c7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175804]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175808]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175814] locks held by kworker/5:20/2691: 4, on CPU#5:
[  247.175815]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175820]  #1: ffff8881427e7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175825]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175830]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175836] locks held by kworker/5:22/2695: 4, on CPU#5:
[  247.175837]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175842]  #1: ffff888146ea7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175847]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175852]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175872] locks held by kworker/60:46/2815: 4, on CPU#60:
[  247.175874]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175879]  #1: ffff88813995fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175884]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175889]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175896] locks held by kworker/59:30/2827: 4, on CPU#59:
[  247.175898]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175903]  #1: ffff888138a77d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175908]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175913]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175918] locks held by kworker/59:31/2829: 3, on CPU#59:
[  247.175920]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175925]  #1: ffff888145a7fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175930]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175935] locks held by kworker/59:33/2831: 4, on CPU#59:
[  247.175936]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175942]  #1: ffff88813ceffd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175946]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175951]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175957] locks held by kworker/59:35/2833: 3, on CPU#59:
[  247.175958]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175963]  #1: ffff88812c9afd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175968]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.175980] locks held by kworker/28:16/2887: 3, on CPU#28:
[  247.175981]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.175987]  #1: ffff88814d967d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.175992]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176000] locks held by kworker/48:50/2910: 4, on CPU#48:
[  247.176002]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.176007]  #1: ffff88814db57d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.176012]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176017]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176035] locks held by kworker/26:38/3006: 4, on CPU#26:
[  247.176036]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.176042]  #1: ffff888148f3fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.176058]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176065]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176081] locks held by kworker/59:43/3082: 4, on CPU#59:
[  247.176082]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.176088]  #1: ffff8881350b7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.176093]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176098]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176103] locks held by kworker/59:44/3084: 3, on CPU#59:
[  247.176105]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.176110]  #1: ffff888145d17d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.176115]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176120] locks held by kworker/59:45/3085: 3, on CPU#59:
[  247.176122]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.176127]  #1: ffff88814d76fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.176132]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176137] locks held by kworker/59:47/3087: 4, on CPU#59:
[  247.176139]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.176144]  #1: ffff88814d49fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.176149]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176154]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176165] locks held by kworker/28:19/3123: 3, on CPU#28:
[  247.176166]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.176171]  #1: ffff888146a97d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.176176]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176191] locks held by kworker/34:49/3179: 4, on CPU#34:
[  247.176192]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.176198]  #1: ffff88814e587d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.176203]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176208]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176214] locks held by kworker/34:50/3180: 4, on CPU#34:
[  247.176215]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.176221]  #1: ffff88814e58fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.176225]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176230]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176237] locks held by kworker/34:54/3184: 4, on CPU#34:
[  247.176238]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.176244]  #1: ffff888142dc7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.176249]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176254]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176259] locks held by kworker/34:55/3185: 4, on CPU#34:
[  247.176260]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.176266]  #1: ffff888142dcfd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.176271]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176276]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176300] locks held by fio/3278: 2, on CPU#6:
[  247.176302]  #0: ffff8881247bae90 
(&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176309]  #1: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176315] locks held by fio/3279: 2, on CPU#62:
[  247.176317]  #0: ffff8881247bae90 
(&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176322]  #1: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176327] locks held by fio/3280: 2, on CPU#7:
[  247.176329]  #0: ffff8881247bae90 
(&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176334]  #1: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176340] locks held by fio/3281: 2, on CPU#28:
[  247.176341]  #0: ffff8881247bae90 
(&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176346]  #1: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176351] locks held by fio/3282: 2, on CPU#60:
[  247.176353]  #0: ffff8881247bae90 
(&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176358]  #1: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176363] locks held by fio/3283: 2, on CPU#34:
[  247.176364]  #0: ffff8881247bae90 
(&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176369]  #1: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176375] locks held by fio/3284: 3, on CPU#26:
[  247.176376]  #0: ffff8881247bae90 
(&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176381]  #1: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176386]  #2: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x28c/0x4d0
[  247.176391] locks held by fio/3285: 2, on CPU#14:
[  247.176393]  #0: ffff8881247bae90 
(&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176398]  #1: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176403] locks held by fio/3286: 3, on CPU#26:
[  247.176405]  #0: ffff8881247bae90 
(&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176410]  #1: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176415]  #2: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x28c/0x4d0
[  247.176421] locks held by fio/3287: 2, on CPU#48:
[  247.176422]  #0: ffff8881247bae90 
(&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176428]  #1: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176433] locks held by fio/3288: 2, on CPU#5:
[  247.176434]  #0: ffff8881247bae90 
(&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176440]  #1: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176445] locks held by fio/3289: 3, on CPU#26:
[  247.176446]  #0: ffff8881247bae90 
(&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176452]  #1: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176457]  #2: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x28c/0x4d0
[  247.176462] locks held by fio/3290: 2, on CPU#61:
[  247.176463]  #0: ffff8881247bae90 
(&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176469]  #1: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176474] locks held by fio/3291: 3, on CPU#26:
[  247.176476]  #0: ffff8881247bae90 
(&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176481]  #1: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176486]  #2: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x28c/0x4d0
[  247.176505] locks held by kworker/62:53/3393: 3, on CPU#62:
[  247.176506]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.176512]  #1: ffff88814759fd30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.176517]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176523] locks held by kworker/62:54/3394: 4, on CPU#62:
[  247.176524]  #0: ffff888135aeb140 
((wq_completion)dio/sda#9){+.+.}-{0:0}, at: 
process_scheduled_works+0xff2/0x1b30
[  247.176530]  #1: ffff88814f5d7d30 
((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: 
process_scheduled_works+0xb9f/0x1b30
[  247.176535]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: 
xfs_trans_alloc+0xa7/0x900
[  247.176540]  #3: ffff8881247bac80 
(&xfs_nondir_ilock_class){++++}-{4:4}, at: 
xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176546]
[  247.176547] =============================================
[  247.176547]

Re: [bug report] fstests generic/774 hang again

[Dave Chinner]

On Thu, Sep 17, 2026 at 09:28:15AM +0100, John Garry wrote:
> On 9/16/26 23:23, Dave Chinner wrote:
> > On Wed, Sep 16, 2026 at 11:23:16AM +0100, John Garry wrote:
> > > On 15/09/2026 16:41, John Garry wrote:
> > > > On 9/15/26 15:50, Darrick J. Wong wrote:
> > > > > > thanks
> > > > > > 
> > > > > > About the kernel code, I am wondering if using IOMAP_DIO_FORCE_WAIT for
> > > > > > CoW-based atomics could help avoid this issue as we seem to be
> > > > > > bogged down
> > > > > > in lock contention.
> > > > > Which lock is being contended, anyway?  It looks like the ILOCK?
> > > > 
> > > > Yeah, I think so Re. ilock. I had some perf data illustrating this from
> > > > last year, which I can't seem to find, so I will re-generate it.
> > > 
> > > Here is perf call graph snippet when running fio with 72x threads issuing
> > > 4KB atomic writes on 1MB file:
> > > 
> > > --50.46%--xfs_file_dio_write_atomic
> > > |
> > > |--37.19%--iomap_dio_rw
> > > |          |
> > > |      --37.14%--__iomap_dio_rw
> > > |           |
> > > |       --36.50%--iomap_iter
> > > |            |
> > > |             --36.49%--xfs_atomic_write_cow_iomap_next
> > > |                  |
> > > |                  |--27.34%--xfs_ilock
> > > |                  |          |
> > > |                  |      --27.34%--down_write
> > > |                  |           |
> > > |                  |       --27.28%--rwsem_down_write_slowpath
> > > |                  |            |
> > > |                  |            |--24.65%--osq_lock
> > > |                  |            |
> > > |                  |             --2.35%--rwsem_spin_on_owner
> > > |                  |
> > > |                  |--7.62%--xfs_trans_alloc_inode
> > > |                  |          |
> > > |                  |      --7.59%--xfs_ilock
> > 
> > Where's the other half of the lock contention?
> 
> I was expecting xfs_reflink_end_atomic_cow() -> xfs_ilock() to be a point of
> contention as well, but it does not show up. I suppose that it because all
> those threads at completion stage are sleeping.
> 
> However, there is this which I omitted:
> 
> --13.21%--xfs_file_write_checks
>    |
>       --13.18%--kiocb_modified
>          |
>             --13.17%--file_update_time_flags
>                |
>                   --13.16%--xfs_vn_update_time
>                      |
>                         --12.99%--xfs_ilock
>                            |
>                               --12.98%--down_write

Fair enough, I'd expect that from the all the threads blocked on the
lock there in the hung task output.

....

> JFYI, I have another log for a similar hang which we see one thread deadlock
> for log space, below.

Looks very similar. So, same questiosn about log size, etc.

-Dave.
-- 
Dave Chinner
dgc@kernel.org

Re: [bug report] fstests generic/774 hang again

[Shin'ichiro Kawasaki]

On Sep 16, 2026 / 11:23, John Garry wrote:
[...]
> We could try something like this:
> 
> ----8<------
> 
> [PATCH] xfs: serialize CoW-based atomic writes
> 
> We have had reports of system hangs when testing Cow-based atomic writes
> for large block sizes.
> 
> Analysis has shown large contention on the the per-inode ilock,
> superficially in creating the mapping in
> xfs_atomic_write_cow_iomap_begin().
> 
> Serialize atomic writes to reduce this contention. In some cases this may
> reduce performance, but high performance has not been a requirement so far
> for CoW-based atomic writes.
> 
> Signed-off-by: John Garry <john.garry@linux.dev>
[...]
> ---->8------
> 
> This is far from ideal, but I think that it should stops the hangs - having
> the kernel hang is intolerable.

Thanks. I applied the patch on top of xfs/for-next branch commit 0ca15a1a1151,
then repeated the test case 100 times on top of the kernel. I did not observe
the hang. This patch looks avoiding the hang.

Re: [bug report] fstests generic/774 hang again

[Shin'ichiro Kawasaki]

On Sep 15, 2026 / 20:48, Shin'ichiro Kawasaki wrote:
> On Sep 15, 2026 / 11:00, John Garry wrote:
> > On 9/15/26 10:34, Shin'ichiro Kawasaki wrote:
> > > I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB
> > > TCMU fileio devices. Actually I once reported this hang symptom in last October,
> > > and Darrick and John kindly took some actions [1]. Since then, the hang
> > > disappeared and I have not observed the hang almost one year. However, my test
> > > system started reporting the hang again last week.
> > > 
> > > [1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t
> > 
> > This was the series to resolve, I think:
> > https://lore.kernel.org/fstests/176279908967.605950.2192923313361120314.stgit@frogsfrogsfrogs/T/#t
> > 
> > As an experiment, can you reduce the file size further, like:
> > 
> > --- a/tests/generic/774
> > +++ b/tests/generic/774
> > @@ -29,7 +29,7 @@ aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))")
> >  fsbsize=$(_get_block_size $SCRATCH_MNT)
> > 
> >  threads=$(_min "$(($(nproc) * 2 * LOAD_FACTOR))" "100")
> > -filesize=$((aw_bsize * threads * 10))
> > +filesize=$((aw_bsize * threads))
> >  depth=$threads
> >  aw_io_size=$((filesize / threads))
> >  aw_io_inc=$aw_io_size
> 
> Thanks for the response. I repeated the test case 100 times with the change
> above, and I did not observe the hang. The change looks avoiding the hang.
> 
> > 
> > > 
> > > FYI, here I attache the kernel message [2].
> > 
> > Is mainline baseline hanging, i.e. v7.3-rc2?
> 
> I will try this out tomorrow.

I observed the hang with the mainline v7.3-rc2 kernel also:

[  516.678848] [   T5360] run fstests generic/774 at 2026-09-16 11:01:26
[  516.808855] [   T2710] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[  519.197067] [   T2825] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[  519.325438] [   T6378] XFS (sdh): Mounting V5 Filesystem f004c5a8-b5fd-4f05-9ea1-977c32c9fc76
[  519.342073] [   T6378] XFS (sdh): Ending clean mount
[  519.704393] [   T6429] XFS (sdh): Unmounting Filesystem f004c5a8-b5fd-4f05-9ea1-977c32c9fc76
[  520.341237] [   T2904] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[  520.461538] [   T6477] XFS (sdh): Mounting V5 Filesystem dcd73d62-bc80-42a1-9893-7815befb6fd1
[  520.480790] [   T6477] XFS (sdh): Ending clean mount
[  862.484216] [    T169] INFO: task kworker/0:0:9 blocked for more than 122 seconds.
[  862.486738] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  862.488497] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.491457] [    T169] task:kworker/0:0     state:D stack:0     pid:9     tgid:9     ppid:2      task_flags:0x4248060 flags:0x00080000
[  862.493590] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  862.494696] [    T169] Call Trace:
[  862.496681] [    T169]  <TASK>
[  862.497741] [    T169]  __schedule+0xc86/0x1b10
[  862.499020] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  862.500434] [    T169]  ? __pfx___schedule+0x10/0x10
[  862.501999] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.503638] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  862.505095] [    T169]  ? lock_acquire+0x11a/0x140
[  862.506540] [    T169]  schedule+0xe5/0x2e0
[  862.507546] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.508858] [    T169]  schedule_preempt_disabled+0x16/0x30
[  862.510463] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  862.511857] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  862.513526] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  862.514807] [    T169]  ? lock_acquire+0x11a/0x140
[  862.516272] [    T169]  down_write_nested+0x1c0/0x1f0
[  862.517706] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  862.519465] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  862.522491] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  862.525468] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  862.528537] [    T169]  ? lock_acquire+0x11a/0x140
[  862.529870] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  862.531028] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  862.533879] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  862.536645] [    T169]  iomap_dio_complete+0x140/0xb20
[  862.537895] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  862.539424] [    T169]  iomap_dio_complete_work+0x58/0x90
[  862.540711] [    T169]  process_one_work+0x947/0x1760
[  862.542425] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  862.543881] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  862.545586] [    T169]  worker_thread+0x601/0xff0
[  862.547505] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.548953] [    T169]  kthread+0x361/0x460
[  862.550636] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.552067] [    T169]  ret_from_fork+0x560/0x860
[  862.598716] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  862.599886] [    T169]  ? __switch_to+0x473/0xd50
[  862.601344] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.602570] [    T169]  ret_from_fork_asm+0x1a/0x30
[  862.604391] [    T169]  </TASK>
[  862.605720] [    T169] INFO: task kworker/0:0:9 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  862.608060] [    T169] INFO: task kworker/10:0:95 blocked for more than 123 seconds.
[  862.610611] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  862.612670] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.615074] [    T169] task:kworker/10:0    state:D stack:0     pid:95    tgid:95    ppid:2      task_flags:0x4248060 flags:0x00080000
[  862.617510] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  862.618863] [    T169] Call Trace:
[  862.619984] [    T169]  <TASK>
[  862.621425] [    T169]  __schedule+0xc86/0x1b10
[  862.622635] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  862.623900] [    T169]  ? __pfx___schedule+0x10/0x10
[  862.625384] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.626539] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  862.627851] [    T169]  ? rcu_is_watching+0x69/0x90
[  862.629309] [    T169]  ? lock_acquire+0x11a/0x140
[  862.630501] [    T169]  schedule+0xe5/0x2e0
[  862.631786] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.633410] [    T169]  schedule_preempt_disabled+0x16/0x30
[  862.634482] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  862.635829] [    T169]  ? __kasan_slab_alloc+0x7f/0x90
[  862.637088] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  862.638545] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  862.639768] [    T169]  ? lock_acquire+0x11a/0x140
[  862.641378] [    T169]  down_write_nested+0x1c0/0x1f0
[  862.642616] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  862.644347] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  862.646648] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  862.649368] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  862.651640] [    T169]  ? update_load_avg+0xb65/0x2c70
[  862.652721] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  862.653736] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  862.656196] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  862.658537] [    T169]  iomap_dio_complete+0x140/0xb20
[  862.659475] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  862.660451] [    T169]  iomap_dio_complete_work+0x58/0x90
[  862.661510] [    T169]  process_one_work+0x947/0x1760
[  862.662631] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  862.663652] [    T169]  ? __try_to_del_timer_sync+0xd3/0x130
[  862.664757] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  862.665845] [    T169]  worker_thread+0x601/0xff0
[  862.666850] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.667903] [    T169]  kthread+0x361/0x460
[  862.669351] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.670628] [    T169]  ret_from_fork+0x560/0x860
[  862.671393] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  862.672180] [    T169]  ? __switch_to+0x473/0xd50
[  862.673196] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.674012] [    T169]  ret_from_fork_asm+0x1a/0x30
[  862.675087] [    T169]  </TASK>
[  862.675904] [    T169] INFO: task kworker/10:0:95 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  862.677484] [    T169] INFO: task kworker/13:0:107 blocked for more than 123 seconds.
[  862.678751] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  862.679934] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.681454] [    T169] task:kworker/13:0    state:D stack:0     pid:107   tgid:107   ppid:2      task_flags:0x4248060 flags:0x00080000
[  862.683148] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  862.684174] [    T169] Call Trace:
[  862.685025] [    T169]  <TASK>
[  862.685742] [    T169]  __schedule+0xc86/0x1b10
[  862.686690] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  862.687637] [    T169]  ? __pfx___schedule+0x10/0x10
[  862.688543] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.689410] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  862.690380] [    T169]  ? lock_acquire+0x11a/0x140
[  862.691570] [    T169]  schedule+0xe5/0x2e0
[  862.692351] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.693301] [    T169]  schedule_preempt_disabled+0x16/0x30
[  862.694317] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  862.695571] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  862.696636] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  862.697620] [    T169]  ? lock_acquire+0x11a/0x140
[  862.698603] [    T169]  down_write_nested+0x1c0/0x1f0
[  862.699567] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  862.700843] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  862.702991] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  862.705330] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  862.707139] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  862.708286] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  862.709873] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  862.711607] [    T169]  iomap_dio_complete+0x140/0xb20
[  862.712622] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  862.713758] [    T169]  iomap_dio_complete_work+0x58/0x90
[  862.714854] [    T169]  process_one_work+0x947/0x1760
[  862.715809] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  862.716900] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  862.718142] [    T169]  worker_thread+0x601/0xff0
[  862.719263] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.720613] [    T169]  kthread+0x361/0x460
[  862.721568] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.722681] [    T169]  ret_from_fork+0x560/0x860
[  862.723567] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  862.724692] [    T169]  ? __switch_to+0x473/0xd50
[  862.725643] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.726494] [    T169]  ret_from_fork_asm+0x1a/0x30
[  862.727727] [    T169]  </TASK>
[  862.728590] [    T169] INFO: task kworker/13:0:107 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  862.730308] [    T169] INFO: task kworker/18:0:131 blocked for more than 123 seconds.
[  862.731761] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  862.732954] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.734431] [    T169] task:kworker/18:0    state:D stack:0     pid:131   tgid:131   ppid:2      task_flags:0x4248060 flags:0x00080000
[  862.735987] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  862.737180] [    T169] Call Trace:
[  862.737922] [    T169]  <TASK>
[  862.738784] [    T169]  __schedule+0xc86/0x1b10
[  862.739699] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  862.740958] [    T169]  ? __pfx___schedule+0x10/0x10
[  862.742223] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.743837] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  862.745556] [    T169]  ? lock_acquire+0x11a/0x140
[  862.747573] [    T169]  schedule+0xe5/0x2e0
[  862.749001] [    T169]  xlog_grant_head_wait+0x175/0xac0 [xfs]
[  862.751614] [    T169]  xlog_grant_head_check+0x312/0x3f0 [xfs]
[  862.754331] [    T169]  ? __pfx_xlog_grant_head_check+0x10/0x10 [xfs]
[  862.757018] [    T169]  xfs_log_reserve+0x2b2/0x7a0 [xfs]
[  862.759361] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  862.760571] [    T169]  ? __pfx_xfs_log_reserve+0x10/0x10 [xfs]
[  862.763313] [    T169]  ? xfs_dec_freecounter+0x81/0x520 [xfs]
[  862.765660] [    T169]  xfs_trans_alloc+0x1a4/0x820 [xfs]
[  862.767935] [    T169]  xfs_reflink_end_atomic_cow+0x1b1/0x560 [xfs]
[  862.770604] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  862.773762] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  862.774868] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  862.777326] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  862.779795] [    T169]  iomap_dio_complete+0x140/0xb20
[  862.780989] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  862.782460] [    T169]  iomap_dio_complete_work+0x58/0x90
[  862.783667] [    T169]  process_one_work+0x947/0x1760
[  862.784759] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  862.786260] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  862.787550] [    T169]  worker_thread+0x601/0xff0
[  862.788602] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.789757] [    T169]  kthread+0x361/0x460
[  862.790931] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.791965] [    T169]  ret_from_fork+0x560/0x860
[  862.793177] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  862.794419] [    T169]  ? __switch_to+0x473/0xd50
[  862.795543] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.796560] [    T169]  ret_from_fork_asm+0x1a/0x30
[  862.797661] [    T169]  </TASK>
[  862.798604] [    T169] INFO: task kworker/21:0:149 blocked for more than 123 seconds.
[  862.800419] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  862.801700] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.803425] [    T169] task:kworker/21:0    state:D stack:0     pid:149   tgid:149   ppid:2      task_flags:0x4248060 flags:0x00080000
[  862.805354] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  862.806351] [    T169] Call Trace:
[  862.807571] [    T169]  <TASK>
[  862.808374] [    T169]  __schedule+0xc86/0x1b10
[  862.809575] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  862.810650] [    T169]  ? __pfx___schedule+0x10/0x10
[  862.811595] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.812602] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  862.813639] [    T169]  ? rcu_is_watching+0x69/0x90
[  862.814471] [    T169]  ? lock_acquire+0x11a/0x140
[  862.815458] [    T169]  schedule+0xe5/0x2e0
[  862.816473] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.817491] [    T169]  schedule_preempt_disabled+0x16/0x30
[  862.818527] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  862.819664] [    T169]  ? __kasan_slab_alloc+0x7f/0x90
[  862.820690] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  862.821863] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  862.823035] [    T169]  ? lock_acquire+0x11a/0x140
[  862.882304] [    T169]  down_write_nested+0x1c0/0x1f0
[  862.882992] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  862.884022] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  862.885942] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  862.888021] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  862.890047] [    T169]  ? update_load_avg+0xb65/0x2c70
[  862.890723] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  862.891859] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  862.893613] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  862.895520] [    T169]  iomap_dio_complete+0x140/0xb20
[  862.896394] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  862.897155] [    T169]  iomap_dio_complete_work+0x58/0x90
[  862.898492] [    T169]  process_one_work+0x947/0x1760
[  862.899478] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  862.900184] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  862.901237] [    T169]  worker_thread+0x601/0xff0
[  862.901968] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.903572] [    T169]  kthread+0x361/0x460
[  862.904365] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.905281] [    T169]  ret_from_fork+0x560/0x860
[  862.906378] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  862.907247] [    T169]  ? __switch_to+0x473/0xd50
[  862.908264] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.909171] [    T169]  ret_from_fork_asm+0x1a/0x30
[  862.910168] [    T169]  </TASK>
[  862.910906] [    T169] INFO: task kworker/21:0:149 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  862.912626] [    T169] INFO: task kworker/10:1:182 blocked for more than 123 seconds.
[  862.913741] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  862.914683] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.915828] [    T169] task:kworker/10:1    state:D stack:0     pid:182   tgid:182   ppid:2      task_flags:0x4248060 flags:0x00080000
[  862.917294] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  862.918366] [    T169] Call Trace:
[  862.918970] [    T169]  <TASK>
[  862.919592] [    T169]  __schedule+0xc86/0x1b10
[  862.920361] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  862.921220] [    T169]  ? __pfx___schedule+0x10/0x10
[  862.922009] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.922876] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  862.923772] [    T169]  ? lock_acquire+0x11a/0x140
[  862.924586] [    T169]  schedule+0xe5/0x2e0
[  862.925449] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.926194] [    T169]  schedule_preempt_disabled+0x16/0x30
[  862.927011] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  862.927988] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  862.928985] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  862.930050] [    T169]  ? lock_acquire+0x11a/0x140
[  862.930852] [    T169]  down_write_nested+0x1c0/0x1f0
[  862.931837] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  862.932707] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  862.934501] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  862.936477] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  862.938135] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  862.939145] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  862.940733] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  862.942374] [    T169]  iomap_dio_complete+0x140/0xb20
[  862.943029] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  862.943836] [    T169]  iomap_dio_complete_work+0x58/0x90
[  862.944511] [    T169]  process_one_work+0x947/0x1760
[  862.945264] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  862.945902] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  862.946883] [    T169]  worker_thread+0x601/0xff0
[  862.947501] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.948411] [    T169]  ? __kthread_parkme+0xbd/0x210
[  862.949411] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.950258] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.950981] [    T169]  kthread+0x361/0x460
[  862.951742] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.952475] [    T169]  ret_from_fork+0x560/0x860
[  862.953085] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  862.953768] [    T169]  ? __switch_to+0x473/0xd50
[  862.954462] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.955329] [    T169]  ret_from_fork_asm+0x1a/0x30
[  862.956135] [    T169]  </TASK>
[  862.956743] [    T169] INFO: task kworker/10:1:182 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  862.958496] [    T169] INFO: task kworker/4:1:192 blocked for more than 123 seconds.
[  862.959780] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  862.960649] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.961646] [    T169] task:kworker/4:1     state:D stack:0     pid:192   tgid:192   ppid:2      task_flags:0x4248060 flags:0x00080000
[  862.962802] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  862.963656] [    T169] Call Trace:
[  862.964239] [    T169]  <TASK>
[  862.964876] [    T169]  __schedule+0xc86/0x1b10
[  862.965650] [    T169]  ? __pfx___schedule+0x10/0x10
[  862.966357] [    T169]  ? rwsem_optimistic_spin+0x1f0/0x480
[  862.967116] [    T169]  ? wq_worker_sleeping+0x65/0x280
[  862.967818] [    T169]  ? lock_acquire+0x11a/0x140
[  862.968455] [    T169]  schedule+0xe5/0x2e0
[  862.969236] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.969956] [    T169]  schedule_preempt_disabled+0x16/0x30
[  862.970745] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  862.971466] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  862.972318] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  862.973018] [    T169]  ? lock_acquire+0x11a/0x140
[  862.973929] [    T169]  down_write_nested+0x1c0/0x1f0
[  862.974668] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  862.975517] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  862.977132] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  862.978753] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  862.980434] [    T169]  ? lock_acquire+0x11a/0x140
[  862.981088] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  862.981891] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  862.983313] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  862.984866] [    T169]  iomap_dio_complete+0x140/0xb20
[  862.985470] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  862.986145] [    T169]  iomap_dio_complete_work+0x58/0x90
[  862.987165] [    T169]  process_one_work+0x947/0x1760
[  862.987782] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  862.988614] [    T169]  ? __try_to_del_timer_sync+0xd3/0x130
[  862.989478] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  862.990329] [    T169]  worker_thread+0x601/0xff0
[  862.991073] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.991879] [    T169]  kthread+0x361/0x460
[  862.992366] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.992943] [    T169]  ret_from_fork+0x560/0x860
[  862.993616] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  862.994429] [    T169]  ? __switch_to+0x473/0xd50
[  862.995061] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.995764] [    T169]  ret_from_fork_asm+0x1a/0x30
[  862.996514] [    T169]  </TASK>
[  862.997177] [    T169] INFO: task kworker/4:1:192 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  862.998650] [    T169] INFO: task kworker/13:1:203 blocked for more than 123 seconds.
[  862.999569] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  863.000412] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  863.001516] [    T169] task:kworker/13:1    state:D stack:0     pid:203   tgid:203   ppid:2      task_flags:0x4248060 flags:0x00080000
[  863.002727] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  863.003433] [    T169] Call Trace:
[  863.003999] [    T169]  <TASK>
[  863.004720] [    T169]  __schedule+0xc86/0x1b10
[  863.005341] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  863.005926] [    T169]  ? __pfx___schedule+0x10/0x10
[  863.006767] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  863.007384] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  863.008172] [    T169]  ? rcu_is_watching+0x69/0x90
[  863.008798] [    T169]  ? lock_acquire+0x11a/0x140
[  863.009533] [    T169]  schedule+0xe5/0x2e0
[  863.010175] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  863.010753] [    T169]  schedule_preempt_disabled+0x16/0x30
[  863.011389] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  863.012000] [    T169]  ? __kasan_slab_alloc+0x7f/0x90
[  863.012699] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  863.013432] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  863.014201] [    T169]  ? lock_acquire+0x11a/0x140
[  863.014754] [    T169]  down_write_nested+0x1c0/0x1f0
[  863.015577] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  863.016221] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  863.017869] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  863.019128] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  863.020502] [    T169]  ? update_load_avg+0xb65/0x2c70
[  863.021124] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  863.021694] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  863.022983] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  863.024392] [    T169]  iomap_dio_complete+0x140/0xb20
[  863.024936] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  863.025750] [    T169]  iomap_dio_complete_work+0x58/0x90
[  863.026539] [    T169]  process_one_work+0x947/0x1760
[  863.027125] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  863.027748] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  863.028469] [    T169]  worker_thread+0x601/0xff0
[  863.029048] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  863.029602] [    T169]  ? __kthread_parkme+0xbd/0x210
[  863.030330] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  863.031083] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  863.031679] [    T169]  kthread+0x361/0x460
[  863.032432] [    T169]  ? __pfx_kthread+0x10/0x10
[  863.033138] [    T169]  ret_from_fork+0x560/0x860
[  863.033773] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  863.034384] [    T169]  ? __switch_to+0x473/0xd50
[  863.034950] [    T169]  ? __pfx_kthread+0x10/0x10
[  863.046680] [    T169]  ret_from_fork_asm+0x1a/0x30
[  863.047257] [    T169]  </TASK>
[  863.047797] [    T169] INFO: task kworker/13:1:203 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  863.048786] [    T169] INFO: task kworker/7:1:267 blocked for more than 123 seconds.
[  863.049651] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  863.050339] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  863.051394] [    T169] task:kworker/7:1     state:D stack:0     pid:267   tgid:267   ppid:2      task_flags:0x4248060 flags:0x00080000
[  863.052474] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  863.053233] [    T169] Call Trace:
[  863.053879] [    T169]  <TASK>
[  863.054380] [    T169]  __schedule+0xc86/0x1b10
[  863.055017] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  863.055705] [    T169]  ? __pfx___schedule+0x10/0x10
[  863.056191] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  863.056776] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  863.057382] [    T169]  ? rcu_is_watching+0x69/0x90
[  863.057888] [    T169]  ? lock_acquire+0x11a/0x140
[  863.058510] [    T169]  schedule+0xe5/0x2e0
[  863.059152] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  863.059634] [    T169]  schedule_preempt_disabled+0x16/0x30
[  863.060191] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  863.060660] [    T169]  ? __kasan_slab_alloc+0x7f/0x90
[  863.061305] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  863.061975] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  863.062852] [    T169]  ? lock_acquire+0x11a/0x140
[  863.063494] [    T169]  down_write_nested+0x1c0/0x1f0
[  863.063987] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  863.064651] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  863.065744] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  863.066867] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  863.068171] [    T169]  ? lock_acquire+0x11a/0x140
[  863.068608] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  863.069209] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  863.070280] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  863.071520] [    T169]  iomap_dio_complete+0x140/0xb20
[  863.072007] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  863.072614] [    T169]  iomap_dio_complete_work+0x58/0x90
[  863.073410] [    T169]  process_one_work+0x947/0x1760
[  863.073943] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  863.074647] [    T169]  ? __try_to_del_timer_sync+0xd3/0x130
[  863.075263] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  863.075717] [    T169]  worker_thread+0x601/0xff0
[  863.076457] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  863.076864] [    T169]  kthread+0x361/0x460
[  863.077206] [    T169]  ? __pfx_kthread+0x10/0x10
[  863.077635] [    T169]  ret_from_fork+0x560/0x860
[  863.077979] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  863.078467] [    T169]  ? __switch_to+0x473/0xd50
[  863.079013] [    T169]  ? __pfx_kthread+0x10/0x10
[  863.079724] [    T169]  ret_from_fork_asm+0x1a/0x30
[  863.080357] [    T169]  </TASK>
[  863.080782] [    T169] INFO: task kworker/7:1:267 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  863.081659] [    T169] INFO: task kworker/18:1:273 blocked for more than 123 seconds.
[  863.082544] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  863.083261] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  863.084177] [    T169] task:kworker/18:1    state:D stack:0     pid:273   tgid:273   ppid:2      task_flags:0x4248060 flags:0x00080000
[  863.085081] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  863.085709] [    T169] Call Trace:
[  863.086190] [    T169]  <TASK>
[  863.086664] [    T169]  __schedule+0xc86/0x1b10
[  863.087350] [    T169]  ? __pfx___schedule+0x10/0x10
[  863.088018] [    T169]  ? rwsem_optimistic_spin+0x1f0/0x480
[  863.088839] [    T169]  ? wq_worker_sleeping+0x65/0x280
[  863.089320] [    T169]  ? lock_acquire+0x11a/0x140
[  863.089716] [    T169]  schedule+0xe5/0x2e0
[  863.090107] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  863.090475] [    T169]  schedule_preempt_disabled+0x16/0x30
[  863.090998] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  863.091593] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  863.092182] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  863.092592] [    T169]  ? lock_acquire+0x11a/0x140
[  863.092928] [    T169]  down_write_nested+0x1c0/0x1f0
[  863.093339] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  863.093878] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  863.094891] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  863.095987] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  863.097206] [    T169]  ? lock_acquire+0x11a/0x140
[  863.097778] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  863.098326] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  863.099445] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  863.100619] [    T169]  iomap_dio_complete+0x140/0xb20
[  863.101449] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  863.102004] [    T169]  iomap_dio_complete_work+0x58/0x90
[  863.102595] [    T169]  process_one_work+0x947/0x1760
[  863.103268] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  863.103864] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  863.104356] [    T169]  worker_thread+0x601/0xff0
[  863.104893] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  863.105480] [    T169]  kthread+0x361/0x460
[  863.105995] [    T169]  ? __pfx_kthread+0x10/0x10
[  863.106607] [    T169]  ret_from_fork+0x560/0x860
[  863.107270] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  863.107794] [    T169]  ? __switch_to+0x473/0xd50
[  863.108317] [    T169]  ? __pfx_kthread+0x10/0x10
[  863.108847] [    T169]  ret_from_fork_asm+0x1a/0x30
[  863.109354] [    T169]  </TASK>
[  863.109815] [    T169] INFO: task kworker/18:1:273 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  863.110690] [    T169] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
[  863.111555] [    T169] INFO: lockdep is turned off.

Re: [bug report] fstests generic/774 hang again

[Dave Chinner]

On Tue, Sep 15, 2026 at 06:34:50PM +0900, Shin'ichiro Kawasaki wrote:
> I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB
> TCMU fileio devices. Actually I once reported this hang symptom in last October,
> and Darrick and John kindly took some actions [1]. Since then, the hang
> disappeared and I have not observed the hang almost one year. However, my test
> system started reporting the hang again last week.
> 
> [1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t
> 
> FYI, here I attache the kernel message [2]. I observed the hang last week with
> the kernel on the xfs/for-next branch at the commit 0ca15a1a1151 ("xfs: remove
> an extra cast in xfs_file_compat_ioctl"). Today, I tried again with the kernel
> at current xfs/for-next branch tip 4266ffdfd7cb ("xfs: remove an extra cast in
> xfs_file_compat_ioctl"), and observed the hang again. The hang can be recreated
> in stable manner by repeating the test case 20 times or so.
> 
> I also tried other test cases in atmoicwrites group, and observed no failure.
> 
>  generic/765        [not run] write atomic not supported by this block device
>  generic/767         11s
>  generic/768         13s
>  generic/769         13s
>  generic/770         32s
>  generic/773        [not run] write atomic not supported by this block device
>  generic/774        (skipped)
>  generic/775         293s
>  generic/776        [notrun] write atomic not supported by this block device
>  generic/778         48s
>  xfs/838            [not run] External volumes not in use, skipped this test
>  xfs/839            [not run] XFS error injection requires CONFIG_XFS_DEBUG
>  xfs/840            [not run] write atomic not supported by this block device
> 
> Action for fix will be appreciated. If I can do anything on my test node, please
> let me know.
.....

A bunch of threads waiting on the ILOCK here:

 down_write_nested+0x1c0/0x1f0
 xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
 xfs_dio_write_end_io+0x4b7/0x650 [xfs]
 iomap_dio_complete+0x140/0xb20
 iomap_dio_complete_work+0x58/0x90
 process_one_work+0x947/0x1760

And the holder:

 schedule+0xe5/0x2e0
 xlog_grant_head_wait+0x175/0xac0 [xfs]
 xlog_grant_head_check+0x312/0x3f0 [xfs]
 xfs_log_regrant+0x380/0x7d0 [xfs]
 xfs_trans_roll+0x2d9/0x420 [xfs]
 xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
 xfs_defer_finish_noroll+0x460/0xe70 [xfs]
 xfs_trans_commit+0xfc/0x180 [xfs]
 xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
 xfs_dio_write_end_io+0x4b7/0x650 [xfs]
 iomap_dio_complete+0x140/0xb20
 iomap_dio_complete_work+0x58/0x90
 process_one_work+0x947/0x1760

is waiting on log space whilst holding the ILOCK.

This looks to me like the test runs out of log space because of all
the IO completions holding transaction reservations waiting on the
ILOCK, whilst the ILOCK holder can't get enough log space to regrant
on transaction roll to continue the transaction.

Oh:

STATIC void
xfs_calc_default_atomic_ioend_reservation(
        struct xfs_mount        *mp,
        struct xfs_trans_resv   *resp)
{
        /* Pick a default that will scale reasonably for the log size. */
        resp->tr_atomic_ioend = resp->tr_itruncate;
}

Which means each reservation is probably holding hundreds of KB to
MB of journal reservation. If the journal is 64MB in size, then a
few dozen o these might be all that is needed to consume all the
reservation space.

Hmmm - there are at least 30 tasks stuck in
xfs_reflink_end_atomic_cow(), another 30 stuck in
xfs_vn_update_time() holding reservations, and significant number
xfs_atomic_write_cow_iomap_begin()->xfs_trans_alloc_inode() holding
tr_write reservations.

IOWs, smells of the journal being run out of reservation space, and
no new space being able to be freed because all the dirty inodes
in the journal that need to be written back to free up space are
locked waiting for journal space to come free....

How big is the journal in the filesystem being tested? If you
increase the size of the journal, does it go away? Can you get a
dump of the transaction reservation sizes for the filesystem in
question (the xfs_db logres command can do this, IIRC) and then run
the math on the reservations held from the dump of all the blocked
tasks holding locks to see if this matches the size of the journal
in the fs?

Cheers,

Dave.
-- 
Dave Chinner
dgc@kernel.org

Re: [bug report] fstests generic/774 hang again

[Shin'ichiro Kawasaki]

On Sep 17, 2026 / 07:45, Dave Chinner wrote:
[...]
> 
> IOWs, smells of the journal being run out of reservation space, and
> no new space being able to be freed because all the dirty inodes
> in the journal that need to be written back to free up space are
> locked waiting for journal space to come free....
> 
> How big is the journal in the filesystem being tested? If you
> increase the size of the journal, does it go away? Can you get a
> dump of the transaction reservation sizes for the filesystem in
> question (the xfs_db logres command can do this, IIRC) and then run
> the math on the reservations held from the dump of all the blocked
> tasks holding locks to see if this matches the size of the journal
> in the fs?

Dave, thanks for the comments. I recreated the hang, and gathered relevant
numbers [3]. Here, /dev/sdh is the SCRATCH_DEV. My interpretations are
as follows (If I misunderstand anything correction will be appreciated):

 - The log section size is 4096 * 16384 = 64MiB.
 - The type 26 logres has 327552 size and logcount 5, then each blocking task
   consumes about 1.6MiB
 - My test node has 24 nproc, and the test case creases 2 * nproc * LOAD_FACTOR
   threads. LOAD_FACTOR is 1, then it creates 48 fio threads.

Assuming all fio threads are blocked, and each thread consumes 16MiB in the
log section, 1.6MiB * 48 = 76.8Mib will be consumed. This is beyond the log
section size 64MiB. The sysfs attributes reserve_grant_head_bytes has the
number close to 64MiB, and I guess it implies the log section is almost full.
Based on these, Dave's guess look correct.

As suggested, now I'm trying to increase the journal/log section size.
I added the mkfs options below:

  MKFS_OPTIONS="-l size=128m"

With this, I repeated the test case 30 times, and did not observe the hang.
It looks like the log section size increase avoids the hang. I will keep it on
running to see if the hang is observed after 100 times run.



[3]

$ sudo xfs_info /dev/sdh
meta-data=/dev/sdh               isize=512    agcount=4, agsize=524288 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=1
         =                       exchange=1   metadir=0
data     =                       bsize=4096   blocks=2097152, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1, parent=1
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
         =                       rgcount=0    rgsize=0 extents
         =                       zoned=0      start=0 reserved=0
$ sudo xfs_db -c logres -r /dev/sdh
type 0 logres 193528 logcount 5 flags 0x4
type 1 logres 327552 logcount 5 flags 0x4
type 2 logres 428160 logcount 5 flags 0x4
type 3 logres 247204 logcount 5 flags 0x4
type 4 logres 230820 logcount 5 flags 0x4
type 5 logres 312612 logcount 6 flags 0x4
type 6 logres 311460 logcount 5 flags 0x4
type 7 logres 285696 logcount 2 flags 0x4
type 8 logres 311460 logcount 6 flags 0x4
type 9 logres 303992 logcount 2 flags 0x4
type 10 logres 2168 logcount 0 flags 0x0
type 11 logres 82176 logcount 2 flags 0x4
type 12 logres 116856 logcount 2 flags 0x4
type 13 logres 760 logcount 0 flags 0x0
type 14 logres 326784 logcount 1 flags 0x4
type 15 logres 23288 logcount 3 flags 0x4
type 16 logres 21760 logcount 0 flags 0x0
type 17 logres 164480 logcount 3 flags 0x4
type 18 logres 640 logcount 0 flags 0x0
type 19 logres 111864 logcount 2 flags 0x4
type 20 logres 4224 logcount 0 flags 0x0
type 21 logres 6512 logcount 0 flags 0x0
type 22 logres 232 logcount 1 flags 0x0
type 23 logres 197751 logcount 5 flags 0x4
type 24 logres 640 logcount 1 flags 0x0
type 25 logres 760 logcount 0 flags 0x0
type 26 logres 327552 logcount 5 flags 0x4
minlogsize logres 428160 logcount 5
$ grep . /sys/fs/xfs/sdh/log/*
/sys/fs/xfs/sdh/log/log_head_lsn:1:103190
/sys/fs/xfs/sdh/log/log_tail_lsn:1:101012
/sys/fs/xfs/sdh/log/reserve_grant_head_bytes:66053588
/sys/fs/xfs/sdh/log/write_grant_head_bytes:65718696

Re: [bug report] fstests generic/774 hang again

[Shin'ichiro Kawasaki]

On Sep 17, 2026 / 15:52, Shin'ichiro Kawasaki wrote:
[...]
> As suggested, now I'm trying to increase the journal/log section size.
> I added the mkfs options below:
> 
>   MKFS_OPTIONS="-l size=128m"
> 
> With this, I repeated the test case 30 times, and did not observe the hang.
> It looks like the log section size increase avoids the hang. I will keep it on
> running to see if the hang is observed after 100 times run.

The test case did not hang all through the 100 times repeated runs. Then
expanding the log section from 64MiB to 128MiB avoided the hang.

Re: [bug report] fstests generic/774 hang again

[Dave Chinner]

On Thu, Sep 17, 2026 at 07:45:11PM +0900, Shin'ichiro Kawasaki wrote:
> On Sep 17, 2026 / 15:52, Shin'ichiro Kawasaki wrote:
> [...]
> > As suggested, now I'm trying to increase the journal/log section size.
> > I added the mkfs options below:
> > 
> >   MKFS_OPTIONS="-l size=128m"
> > 
> > With this, I repeated the test case 30 times, and did not observe the hang.
> > It looks like the log section size increase avoids the hang. I will keep it on
> > running to see if the hang is observed after 100 times run.
> 
> The test case did not hang all through the 100 times repeated runs. Then
> expanding the log section from 64MiB to 128MiB avoided the hang.

Ok, so the fix for the moment is to make the software COW path for
atomic writes use IOLOCK_EXCL, rather than IOLOCK_SHARED. That will
result in all the IO submission blocking on the IOLOCK with no other
resources held rather than blocking on the ILOCK whilst holding an
unreleasable transaction reservation.

Cheers,

Dave.
-- 
Dave Chinner
dgc@kernel.org

Re: [bug report] fstests generic/774 hang again

[Shin'ichiro Kawasaki]

On Sep 18, 2026 / 07:24, Dave Chinner wrote:
> On Thu, Sep 17, 2026 at 07:45:11PM +0900, Shin'ichiro Kawasaki wrote:
> > On Sep 17, 2026 / 15:52, Shin'ichiro Kawasaki wrote:
> > [...]
> > > As suggested, now I'm trying to increase the journal/log section size.
> > > I added the mkfs options below:
> > > 
> > >   MKFS_OPTIONS="-l size=128m"
> > > 
> > > With this, I repeated the test case 30 times, and did not observe the hang.
> > > It looks like the log section size increase avoids the hang. I will keep it on
> > > running to see if the hang is observed after 100 times run.
> > 
> > The test case did not hang all through the 100 times repeated runs. Then
> > expanding the log section from 64MiB to 128MiB avoided the hang.
> 
> Ok, so the fix for the moment is to make the software COW path for
> atomic writes use IOLOCK_EXCL, rather than IOLOCK_SHARED. That will
> result in all the IO submission blocking on the IOLOCK with no other
> resources held rather than blocking on the ILOCK whilst holding an
> unreleasable transaction reservation.

Thanks for the explanation and the fix suggestion. Based on the suggestion,
I cooked a fix trial patch below [4]. I applied this patch on top of
the xfs-linux/for-next branch kernel at the git hash 0ca15a1a115. I repeated
the test case generic/774 100 times on the kernel, and observed no hang.
Looks working good. I will do further testing to confirm no regression.
Meanwhile, comments on the patch will be appreciated.


[4] fix trial patch

-----------------------------------------------------------------------------

From 2d2cbd8e3424953880f6ffc8e2ed5b4f844b0822 Mon Sep 17 00:00:00 2001
From: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Date: Fri, 18 Sep 2026 00:00:00 +0900
Subject: [PATCH] xfs: serialise software COW atomic write submission on the
 IOLOCK

When fstests generic/774 is repeated on a 8GiB device with a 64MiB
journal size, the test case hangs. Dozens of IO completion workers are
blocked on the ILOCK in xfs_reflink_end_atomic_cow():

  down_write_nested+0x1c0/0x1f0
  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
  iomap_dio_complete+0x140/0xb20
  iomap_dio_complete_work+0x58/0x90
  process_one_work+0x947/0x1760

while the ILOCK holder is waiting for journal space:

  xlog_grant_head_wait+0x175/0xac0 [xfs]
  xlog_grant_head_check+0x312/0x3f0 [xfs]
  xfs_log_regrant+0x380/0x7d0 [xfs]
  xfs_trans_roll+0x2d9/0x420 [xfs]
  xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
  xfs_defer_finish_noroll+0x460/0xe70 [xfs]
  xfs_trans_commit+0xfc/0x180 [xfs]
  xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
  xfs_dio_write_end_io+0x4b7/0x650 [xfs]

Each atomic write software COW submitter allocates around 1.6MiB of
journal resource under ILOCK. When a few dozen of atomic writes are
submitted, it is enough to run out of the 64MiB journal size. This
caused the deadlock between ILOCK and the journal resource.

To avoid the deadlock, take the IOLOCK exclusively at the atomic write
software COW submission. This serializes the submission in the IO path,
so concurrent atomic writes block on the IOLOCK holding no other
resources.

Fixes: 9baeac3ab1f8 ("xfs: add xfs_file_dio_write_atomic()")
Closes: https://lore.kernel.org/linux-xfs/aqkP9CTUvlC2YCIb@shinmob/
Suggested-by: Dave Chinner <dgc@kernel.org>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
 fs/xfs/xfs_file.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 426a67b813a..8d10ae22884 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -829,7 +829,7 @@ xfs_file_dio_write_atomic(
 	struct kiocb		*iocb,
 	struct iov_iter		*from)
 {
-	unsigned int		iolock = XFS_IOLOCK_SHARED;
+	unsigned int		iolock;
 	ssize_t			ret, ocount = iov_iter_count(from);
 	unsigned int		dio_flags = 0;
 	const struct iomap_ops	*dops;
@@ -844,6 +844,17 @@ xfs_file_dio_write_atomic(
 		dops = &xfs_direct_write_iomap_ops;
 
 retry:
+	/*
+	 * Concurrent atomic write COW submissions under ILOCK can run out
+	 * journal reservation resource and can deadlock with the IO
+	 * completions. To avoid the deadlock, serialize submissions of
+	 * atomic write COW by taking IOLOCK exclusively.
+	 */
+	if (dops == &xfs_atomic_write_cow_iomap_ops)
+		iolock = XFS_IOLOCK_EXCL;
+	else
+		iolock = XFS_IOLOCK_SHARED;
+
 	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
 	if (ret)
 		return ret;
@@ -853,7 +864,8 @@ xfs_file_dio_write_atomic(
 		goto out_unlock;
 
 	/* Demote similar to xfs_file_dio_write_aligned() */
-	if (iolock == XFS_IOLOCK_EXCL) {
+	if (iolock == XFS_IOLOCK_EXCL &&
+	    dops != &xfs_atomic_write_cow_iomap_ops) {
 		xfs_ilock_demote(ip, XFS_IOLOCK_EXCL);
 		iolock = XFS_IOLOCK_SHARED;
 	}
-- 
2.55.0

Re: [bug report] fstests generic/774 hang again

[Dave Chinner]

On Sat, Sep 19, 2026 at 08:53:39PM +0900, Shin'ichiro Kawasaki wrote:
> On Sep 18, 2026 / 07:24, Dave Chinner wrote:
> > On Thu, Sep 17, 2026 at 07:45:11PM +0900, Shin'ichiro Kawasaki wrote:
> > > On Sep 17, 2026 / 15:52, Shin'ichiro Kawasaki wrote:
> > > [...]
> > > > As suggested, now I'm trying to increase the journal/log section size.
> > > > I added the mkfs options below:
> > > > 
> > > >   MKFS_OPTIONS="-l size=128m"
> > > > 
> > > > With this, I repeated the test case 30 times, and did not observe the hang.
> > > > It looks like the log section size increase avoids the hang. I will keep it on
> > > > running to see if the hang is observed after 100 times run.
> > > 
> > > The test case did not hang all through the 100 times repeated runs. Then
> > > expanding the log section from 64MiB to 128MiB avoided the hang.
> > 
> > Ok, so the fix for the moment is to make the software COW path for
> > atomic writes use IOLOCK_EXCL, rather than IOLOCK_SHARED. That will
> > result in all the IO submission blocking on the IOLOCK with no other
> > resources held rather than blocking on the ILOCK whilst holding an
> > unreleasable transaction reservation.
> 
> Thanks for the explanation and the fix suggestion. Based on the suggestion,
> I cooked a fix trial patch below [4]. I applied this patch on top of
> the xfs-linux/for-next branch kernel at the git hash 0ca15a1a115. I repeated
> the test case generic/774 100 times on the kernel, and observed no hang.
> Looks working good. I will do further testing to confirm no regression.
> Meanwhile, comments on the patch will be appreciated.
> 
> 
> [4] fix trial patch
> 
> -----------------------------------------------------------------------------
> 
> From 2d2cbd8e3424953880f6ffc8e2ed5b4f844b0822 Mon Sep 17 00:00:00 2001
> From: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
> Date: Fri, 18 Sep 2026 00:00:00 +0900
> Subject: [PATCH] xfs: serialise software COW atomic write submission on the
>  IOLOCK
> 
> When fstests generic/774 is repeated on a 8GiB device with a 64MiB
> journal size, the test case hangs. Dozens of IO completion workers are
> blocked on the ILOCK in xfs_reflink_end_atomic_cow():
> 
>   down_write_nested+0x1c0/0x1f0
>   xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
>   xfs_dio_write_end_io+0x4b7/0x650 [xfs]
>   iomap_dio_complete+0x140/0xb20
>   iomap_dio_complete_work+0x58/0x90
>   process_one_work+0x947/0x1760
> 
> while the ILOCK holder is waiting for journal space:
> 
>   xlog_grant_head_wait+0x175/0xac0 [xfs]
>   xlog_grant_head_check+0x312/0x3f0 [xfs]
>   xfs_log_regrant+0x380/0x7d0 [xfs]
>   xfs_trans_roll+0x2d9/0x420 [xfs]
>   xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
>   xfs_defer_finish_noroll+0x460/0xe70 [xfs]
>   xfs_trans_commit+0xfc/0x180 [xfs]
>   xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
>   xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> 
> Each atomic write software COW submitter allocates around 1.6MiB of
> journal resource under ILOCK. When a few dozen of atomic writes are
> submitted, it is enough to run out of the 64MiB journal size. This
> caused the deadlock between ILOCK and the journal resource.
> 
> To avoid the deadlock, take the IOLOCK exclusively at the atomic write
> software COW submission. This serializes the submission in the IO path,
> so concurrent atomic writes block on the IOLOCK holding no other
> resources.
> 
> Fixes: 9baeac3ab1f8 ("xfs: add xfs_file_dio_write_atomic()")
> Closes: https://lore.kernel.org/linux-xfs/aqkP9CTUvlC2YCIb@shinmob/
> Suggested-by: Dave Chinner <dgc@kernel.org>
> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
> ---
>  fs/xfs/xfs_file.c | 16 ++++++++++++++--
>  1 file changed, 14 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
> index 426a67b813a..8d10ae22884 100644
> --- a/fs/xfs/xfs_file.c
> +++ b/fs/xfs/xfs_file.c
> @@ -829,7 +829,7 @@ xfs_file_dio_write_atomic(
>  	struct kiocb		*iocb,
>  	struct iov_iter		*from)
>  {
> -	unsigned int		iolock = XFS_IOLOCK_SHARED;
> +	unsigned int		iolock;
>  	ssize_t			ret, ocount = iov_iter_count(from);
>  	unsigned int		dio_flags = 0;
>  	const struct iomap_ops	*dops;
> @@ -844,6 +844,17 @@ xfs_file_dio_write_atomic(
>  		dops = &xfs_direct_write_iomap_ops;
>  
>  retry:
> +	/*
> +	 * Concurrent atomic write COW submissions under ILOCK can run out
> +	 * journal reservation resource and can deadlock with the IO
> +	 * completions. To avoid the deadlock, serialize submissions of
> +	 * atomic write COW by taking IOLOCK exclusively.
> +	 */
> +	if (dops == &xfs_atomic_write_cow_iomap_ops)
> +		iolock = XFS_IOLOCK_EXCL;
> +	else
> +		iolock = XFS_IOLOCK_SHARED;
> +
>  	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
>  	if (ret)
>  		return ret;

Urk, that's pretty nasty.

> @@ -853,7 +864,8 @@ xfs_file_dio_write_atomic(
>  		goto out_unlock;
>  
>  	/* Demote similar to xfs_file_dio_write_aligned() */
> -	if (iolock == XFS_IOLOCK_EXCL) {
> +	if (iolock == XFS_IOLOCK_EXCL &&
> +	    dops != &xfs_atomic_write_cow_iomap_ops) {
>  		xfs_ilock_demote(ip, XFS_IOLOCK_EXCL);
>  		iolock = XFS_IOLOCK_SHARED;
>  	}

And at this point, we now have several atomic write cow ops specific
operations in this function (locking, the retry loop, etc).

This feels much more like there should be a separate function for
the software cow path, and the fast path simply calls it on
ENOPROTOOPT from the dio submission. That gets rid of all the
conditionals and looping from the fast path. i.e

	if (ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
		return xfs_file_dio_write_atomic_cow();

	/* do normal DIO write */

	if (error == -ENOPROTOOPT)
		return xfs_file_dio_write_atomic_cow();
	return error;


That essentially makes xfs_file_dio_write_atomic() and
xfs_file_dio_write_aligned() the same code, except for the above two
checks, hence they could easily be collapsed back into a common
implementation is:

	if ((iocb->ki_flags & IOCB_ATOMIC) &&
	    ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
		return xfs_file_dio_write_atomic_cow();

	/* do normal DIO write */

	if (error == -ENOPROTOOPT)
		return xfs_file_dio_write_atomic_cow();
	return error;

That seems like a much more natural breakdown that the current
duplication of the DIO write submission code...

-Dave.
-- 
Dave Chinner
dgc@kernel.org

Re: [bug report] fstests generic/774 hang again

[Shin'ichiro Kawasaki]

On Sep 22, 2026 / 08:05, Dave Chinner wrote:
> On Sat, Sep 19, 2026 at 08:53:39PM +0900, Shin'ichiro Kawasaki wrote:
[...]
> > diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
> > index 426a67b813a..8d10ae22884 100644
> > --- a/fs/xfs/xfs_file.c
> > +++ b/fs/xfs/xfs_file.c
> > @@ -829,7 +829,7 @@ xfs_file_dio_write_atomic(
> >  	struct kiocb		*iocb,
> >  	struct iov_iter		*from)
> >  {
> > -	unsigned int		iolock = XFS_IOLOCK_SHARED;
> > +	unsigned int		iolock;
> >  	ssize_t			ret, ocount = iov_iter_count(from);
> >  	unsigned int		dio_flags = 0;
> >  	const struct iomap_ops	*dops;
> > @@ -844,6 +844,17 @@ xfs_file_dio_write_atomic(
> >  		dops = &xfs_direct_write_iomap_ops;
> >  
> >  retry:
> > +	/*
> > +	 * Concurrent atomic write COW submissions under ILOCK can run out
> > +	 * journal reservation resource and can deadlock with the IO
> > +	 * completions. To avoid the deadlock, serialize submissions of
> > +	 * atomic write COW by taking IOLOCK exclusively.
> > +	 */
> > +	if (dops == &xfs_atomic_write_cow_iomap_ops)
> > +		iolock = XFS_IOLOCK_EXCL;
> > +	else
> > +		iolock = XFS_IOLOCK_SHARED;
> > +
> >  	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
> >  	if (ret)
> >  		return ret;
> 
> Urk, that's pretty nasty.
> 
> > @@ -853,7 +864,8 @@ xfs_file_dio_write_atomic(
> >  		goto out_unlock;
> >  
> >  	/* Demote similar to xfs_file_dio_write_aligned() */
> > -	if (iolock == XFS_IOLOCK_EXCL) {
> > +	if (iolock == XFS_IOLOCK_EXCL &&
> > +	    dops != &xfs_atomic_write_cow_iomap_ops) {
> >  		xfs_ilock_demote(ip, XFS_IOLOCK_EXCL);
> >  		iolock = XFS_IOLOCK_SHARED;
> >  	}
> 
> And at this point, we now have several atomic write cow ops specific
> operations in this function (locking, the retry loop, etc).
> 
> This feels much more like there should be a separate function for
> the software cow path, and the fast path simply calls it on
> ENOPROTOOPT from the dio submission. That gets rid of all the
> conditionals and looping from the fast path. i.e
> 
> 	if (ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
> 		return xfs_file_dio_write_atomic_cow();
> 
> 	/* do normal DIO write */
> 
> 	if (error == -ENOPROTOOPT)
> 		return xfs_file_dio_write_atomic_cow();
> 	return error;
> 
> 
> That essentially makes xfs_file_dio_write_atomic() and
> xfs_file_dio_write_aligned() the same code, except for the above two
> checks, hence they could easily be collapsed back into a common
> implementation is:
> 
> 	if ((iocb->ki_flags & IOCB_ATOMIC) &&
> 	    ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
> 		return xfs_file_dio_write_atomic_cow();
> 
> 	/* do normal DIO write */
> 
> 	if (error == -ENOPROTOOPT)
> 		return xfs_file_dio_write_atomic_cow();
> 	return error;
> 
> That seems like a much more natural breakdown that the current
> duplication of the DIO write submission code...

Thanks for the comment. I'll try to factor out the duplication based
on your idea.

On the other hand, Darrick suggested antoher solution approach to allocate
enough space before taking ILOCK by increasing tr_logcount. I wonder which
way is the better: "allocate enough space" or "take exclusive IOLOCK". I'll
do some experiment for the "allocate enough space" approach before working
on the dupliaction clean up.

Re: [bug report] fstests generic/774 hang again

[Dave Chinner]

On Fri, Sep 25, 2026 at 10:32:44AM +0900, Shin'ichiro Kawasaki wrote:
> On Sep 22, 2026 / 08:05, Dave Chinner wrote:
> > On Sat, Sep 19, 2026 at 08:53:39PM +0900, Shin'ichiro Kawasaki wrote:
> [...]
> > > diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
> > > index 426a67b813a..8d10ae22884 100644
> > > --- a/fs/xfs/xfs_file.c
> > > +++ b/fs/xfs/xfs_file.c
> > > @@ -829,7 +829,7 @@ xfs_file_dio_write_atomic(
> > >  	struct kiocb		*iocb,
> > >  	struct iov_iter		*from)
> > >  {
> > > -	unsigned int		iolock = XFS_IOLOCK_SHARED;
> > > +	unsigned int		iolock;
> > >  	ssize_t			ret, ocount = iov_iter_count(from);
> > >  	unsigned int		dio_flags = 0;
> > >  	const struct iomap_ops	*dops;
> > > @@ -844,6 +844,17 @@ xfs_file_dio_write_atomic(
> > >  		dops = &xfs_direct_write_iomap_ops;
> > >  
> > >  retry:
> > > +	/*
> > > +	 * Concurrent atomic write COW submissions under ILOCK can run out
> > > +	 * journal reservation resource and can deadlock with the IO
> > > +	 * completions. To avoid the deadlock, serialize submissions of
> > > +	 * atomic write COW by taking IOLOCK exclusively.
> > > +	 */
> > > +	if (dops == &xfs_atomic_write_cow_iomap_ops)
> > > +		iolock = XFS_IOLOCK_EXCL;
> > > +	else
> > > +		iolock = XFS_IOLOCK_SHARED;
> > > +
> > >  	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
> > >  	if (ret)
> > >  		return ret;
> > 
> > Urk, that's pretty nasty.
> > 
> > > @@ -853,7 +864,8 @@ xfs_file_dio_write_atomic(
> > >  		goto out_unlock;
> > >  
> > >  	/* Demote similar to xfs_file_dio_write_aligned() */
> > > -	if (iolock == XFS_IOLOCK_EXCL) {
> > > +	if (iolock == XFS_IOLOCK_EXCL &&
> > > +	    dops != &xfs_atomic_write_cow_iomap_ops) {
> > >  		xfs_ilock_demote(ip, XFS_IOLOCK_EXCL);
> > >  		iolock = XFS_IOLOCK_SHARED;
> > >  	}
> > 
> > And at this point, we now have several atomic write cow ops specific
> > operations in this function (locking, the retry loop, etc).
> > 
> > This feels much more like there should be a separate function for
> > the software cow path, and the fast path simply calls it on
> > ENOPROTOOPT from the dio submission. That gets rid of all the
> > conditionals and looping from the fast path. i.e
> > 
> > 	if (ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
> > 		return xfs_file_dio_write_atomic_cow();
> > 
> > 	/* do normal DIO write */
> > 
> > 	if (error == -ENOPROTOOPT)
> > 		return xfs_file_dio_write_atomic_cow();
> > 	return error;
> > 
> > 
> > That essentially makes xfs_file_dio_write_atomic() and
> > xfs_file_dio_write_aligned() the same code, except for the above two
> > checks, hence they could easily be collapsed back into a common
> > implementation is:
> > 
> > 	if ((iocb->ki_flags & IOCB_ATOMIC) &&
> > 	    ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
> > 		return xfs_file_dio_write_atomic_cow();
> > 
> > 	/* do normal DIO write */
> > 
> > 	if (error == -ENOPROTOOPT)
> > 		return xfs_file_dio_write_atomic_cow();
> > 	return error;
> > 
> > That seems like a much more natural breakdown that the current
> > duplication of the DIO write submission code...
> 
> Thanks for the comment. I'll try to factor out the duplication based
> on your idea.
> 
> On the other hand, Darrick suggested antoher solution approach to allocate
> enough space before taking ILOCK by increasing tr_logcount.

No, that doesn't fix anything. tr_logcount is an optimisation to
minimise the number of blocking regrants a -typical- rolling
transaction will take. It trades off an increase in initial write
grant reservation space (i.e. they use more log space) to enable
more transaction rolls without needing to refresh the write grant
for the next operation in the transaction.

IOWs, it will reduce the number of concurrent transactions that can
be running at any given time, but if the deferops intent processing
can still run out of pre-reservation space and block waiting for a
write grant.

Hence increasing tr_logcount just kicks the can down
the road, making it slightly harder to trigger the deadlock at the
cost of lowering modification concurrency in the filesystem.


> I wonder which
> way is the better: "allocate enough space" or "take exclusive IOLOCK". I'll
> do some experiment for the "allocate enough space" approach before working
> on the dupliaction clean up.

"allocate enough space" if not a fix - it's a bandaid. "take
exclusive IOLOCK" reflects the fact that COW operations are
inherently single threaded due to their reliance on holding the
ILOCK_EXCL for long periods of time on both IO submission and IO
completion.

Cheers,

Dave.
-- 
Dave Chinner
dgc@kernel.org

Re: [bug report] fstests generic/774 hang again

[Darrick J. Wong]

On Mon, Sep 28, 2026 at 07:30:24AM +1000, Dave Chinner wrote:
> On Fri, Sep 25, 2026 at 10:32:44AM +0900, Shin'ichiro Kawasaki wrote:
> > On Sep 22, 2026 / 08:05, Dave Chinner wrote:
> > > On Sat, Sep 19, 2026 at 08:53:39PM +0900, Shin'ichiro Kawasaki wrote:
> > [...]
> > > > diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
> > > > index 426a67b813a..8d10ae22884 100644
> > > > --- a/fs/xfs/xfs_file.c
> > > > +++ b/fs/xfs/xfs_file.c
> > > > @@ -829,7 +829,7 @@ xfs_file_dio_write_atomic(
> > > >  	struct kiocb		*iocb,
> > > >  	struct iov_iter		*from)
> > > >  {
> > > > -	unsigned int		iolock = XFS_IOLOCK_SHARED;
> > > > +	unsigned int		iolock;
> > > >  	ssize_t			ret, ocount = iov_iter_count(from);
> > > >  	unsigned int		dio_flags = 0;
> > > >  	const struct iomap_ops	*dops;
> > > > @@ -844,6 +844,17 @@ xfs_file_dio_write_atomic(
> > > >  		dops = &xfs_direct_write_iomap_ops;
> > > >  
> > > >  retry:
> > > > +	/*
> > > > +	 * Concurrent atomic write COW submissions under ILOCK can run out
> > > > +	 * journal reservation resource and can deadlock with the IO
> > > > +	 * completions. To avoid the deadlock, serialize submissions of
> > > > +	 * atomic write COW by taking IOLOCK exclusively.
> > > > +	 */
> > > > +	if (dops == &xfs_atomic_write_cow_iomap_ops)
> > > > +		iolock = XFS_IOLOCK_EXCL;
> > > > +	else
> > > > +		iolock = XFS_IOLOCK_SHARED;
> > > > +
> > > >  	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
> > > >  	if (ret)
> > > >  		return ret;
> > > 
> > > Urk, that's pretty nasty.
> > > 
> > > > @@ -853,7 +864,8 @@ xfs_file_dio_write_atomic(
> > > >  		goto out_unlock;
> > > >  
> > > >  	/* Demote similar to xfs_file_dio_write_aligned() */
> > > > -	if (iolock == XFS_IOLOCK_EXCL) {
> > > > +	if (iolock == XFS_IOLOCK_EXCL &&
> > > > +	    dops != &xfs_atomic_write_cow_iomap_ops) {
> > > >  		xfs_ilock_demote(ip, XFS_IOLOCK_EXCL);
> > > >  		iolock = XFS_IOLOCK_SHARED;
> > > >  	}
> > > 
> > > And at this point, we now have several atomic write cow ops specific
> > > operations in this function (locking, the retry loop, etc).
> > > 
> > > This feels much more like there should be a separate function for
> > > the software cow path, and the fast path simply calls it on
> > > ENOPROTOOPT from the dio submission. That gets rid of all the
> > > conditionals and looping from the fast path. i.e
> > > 
> > > 	if (ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
> > > 		return xfs_file_dio_write_atomic_cow();
> > > 
> > > 	/* do normal DIO write */
> > > 
> > > 	if (error == -ENOPROTOOPT)
> > > 		return xfs_file_dio_write_atomic_cow();
> > > 	return error;
> > > 
> > > 
> > > That essentially makes xfs_file_dio_write_atomic() and
> > > xfs_file_dio_write_aligned() the same code, except for the above two
> > > checks, hence they could easily be collapsed back into a common
> > > implementation is:
> > > 
> > > 	if ((iocb->ki_flags & IOCB_ATOMIC) &&
> > > 	    ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
> > > 		return xfs_file_dio_write_atomic_cow();
> > > 
> > > 	/* do normal DIO write */
> > > 
> > > 	if (error == -ENOPROTOOPT)
> > > 		return xfs_file_dio_write_atomic_cow();
> > > 	return error;
> > > 
> > > That seems like a much more natural breakdown that the current
> > > duplication of the DIO write submission code...
> > 
> > Thanks for the comment. I'll try to factor out the duplication based
> > on your idea.
> > 
> > On the other hand, Darrick suggested antoher solution approach to allocate
> > enough space before taking ILOCK by increasing tr_logcount.
> 
> No, that doesn't fix anything. tr_logcount is an optimisation to
> minimise the number of blocking regrants a -typical- rolling
> transaction will take. It trades off an increase in initial write
> grant reservation space (i.e. they use more log space) to enable
> more transaction rolls without needing to refresh the write grant
> for the next operation in the transaction.
> 
> IOWs, it will reduce the number of concurrent transactions that can
> be running at any given time, but if the deferops intent processing
> can still run out of pre-reservation space and block waiting for a
> write grant.
> 
> Hence increasing tr_logcount just kicks the can down
> the road, making it slightly harder to trigger the deadlock at the
> cost of lowering modification concurrency in the filesystem.
> 
> 
> > I wonder which
> > way is the better: "allocate enough space" or "take exclusive IOLOCK". I'll
> > do some experiment for the "allocate enough space" approach before working
> > on the dupliaction clean up.
> 
> "allocate enough space" if not a fix - it's a bandaid. "take
> exclusive IOLOCK" reflects the fact that COW operations are
> inherently single threaded due to their reliance on holding the
> ILOCK_EXCL for long periods of time on both IO submission and IO
> completion.

Wellp, I'll let you and your LLM toy figure out the correct fix for
this, then.

--D

Re: [bug report] fstests generic/774 hang again

[Darrick J. Wong]

On Thu, Sep 17, 2026 at 07:45:04AM +1000, Dave Chinner wrote:
> On Tue, Sep 15, 2026 at 06:34:50PM +0900, Shin'ichiro Kawasaki wrote:
> > I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB
> > TCMU fileio devices. Actually I once reported this hang symptom in last October,
> > and Darrick and John kindly took some actions [1]. Since then, the hang
> > disappeared and I have not observed the hang almost one year. However, my test
> > system started reporting the hang again last week.
> > 
> > [1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t
> > 
> > FYI, here I attache the kernel message [2]. I observed the hang last week with
> > the kernel on the xfs/for-next branch at the commit 0ca15a1a1151 ("xfs: remove
> > an extra cast in xfs_file_compat_ioctl"). Today, I tried again with the kernel
> > at current xfs/for-next branch tip 4266ffdfd7cb ("xfs: remove an extra cast in
> > xfs_file_compat_ioctl"), and observed the hang again. The hang can be recreated
> > in stable manner by repeating the test case 20 times or so.
> > 
> > I also tried other test cases in atmoicwrites group, and observed no failure.
> > 
> >  generic/765        [not run] write atomic not supported by this block device
> >  generic/767         11s
> >  generic/768         13s
> >  generic/769         13s
> >  generic/770         32s
> >  generic/773        [not run] write atomic not supported by this block device
> >  generic/774        (skipped)
> >  generic/775         293s
> >  generic/776        [notrun] write atomic not supported by this block device
> >  generic/778         48s
> >  xfs/838            [not run] External volumes not in use, skipped this test
> >  xfs/839            [not run] XFS error injection requires CONFIG_XFS_DEBUG
> >  xfs/840            [not run] write atomic not supported by this block device
> > 
> > Action for fix will be appreciated. If I can do anything on my test node, please
> > let me know.
> .....
> 
> A bunch of threads waiting on the ILOCK here:
> 
>  down_write_nested+0x1c0/0x1f0
>  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
>  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
>  iomap_dio_complete+0x140/0xb20
>  iomap_dio_complete_work+0x58/0x90
>  process_one_work+0x947/0x1760
> 
> And the holder:
> 
>  schedule+0xe5/0x2e0
>  xlog_grant_head_wait+0x175/0xac0 [xfs]
>  xlog_grant_head_check+0x312/0x3f0 [xfs]
>  xfs_log_regrant+0x380/0x7d0 [xfs]
>  xfs_trans_roll+0x2d9/0x420 [xfs]
>  xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
>  xfs_defer_finish_noroll+0x460/0xe70 [xfs]
>  xfs_trans_commit+0xfc/0x180 [xfs]
>  xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
>  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
>  iomap_dio_complete+0x140/0xb20
>  iomap_dio_complete_work+0x58/0x90
>  process_one_work+0x947/0x1760
> 
> is waiting on log space whilst holding the ILOCK.
> 
> This looks to me like the test runs out of log space because of all
> the IO completions holding transaction reservations waiting on the
> ILOCK, whilst the ILOCK holder can't get enough log space to regrant
> on transaction roll to continue the transaction.

Agreed.

Why are we calling xlog_grant_head_check from within
xfs_reflink_end_atomic_cow?  I think the reason for doing that is
because we've exhausted t_cnt in the ticket (i.e. we've already rolled
more than tr_logcount times).

Oh.  tr_atomic_ioend.tr_logcount is 5 on a rmap+reflink filesystem,
which it inherits from tr_itruncate.  However, tr_itruncate is only
intended to remove two extents from a file, so it only needs 5 rolls.
Coincidentally we calculate 5 rolls for each atomic extent remapping
operation.

For atomic write ioends, what if we increased tr_logcount to 5x the
number of remappings that would have to occur to finish the write?
That would preallocate all the permanent reservation we'd need before we
take the ILOCK, which avoids the situation of needing to obtain more log
space while holding ILOCK.

The downside is that you'd have to limit the software awu_max even
further, perhaps to 40% of the log size divided by
(tr_logres*tr_logcount).  However, we'd still be able to handle
concurrent atomic writes to different parts of the file, at least until
fragmentation got bad.

<shrug>

--D

> 
> Oh:
> 
> STATIC void
> xfs_calc_default_atomic_ioend_reservation(
>         struct xfs_mount        *mp,
>         struct xfs_trans_resv   *resp)
> {
>         /* Pick a default that will scale reasonably for the log size. */
>         resp->tr_atomic_ioend = resp->tr_itruncate;
> }
> 
> Which means each reservation is probably holding hundreds of KB to
> MB of journal reservation. If the journal is 64MB in size, then a
> few dozen o these might be all that is needed to consume all the
> reservation space.
> 
> Hmmm - there are at least 30 tasks stuck in
> xfs_reflink_end_atomic_cow(), another 30 stuck in
> xfs_vn_update_time() holding reservations, and significant number
> xfs_atomic_write_cow_iomap_begin()->xfs_trans_alloc_inode() holding
> tr_write reservations.
> 
> IOWs, smells of the journal being run out of reservation space, and
> no new space being able to be freed because all the dirty inodes
> in the journal that need to be written back to free up space are
> locked waiting for journal space to come free....
> 
> How big is the journal in the filesystem being tested? If you
> increase the size of the journal, does it go away? Can you get a
> dump of the transaction reservation sizes for the filesystem in
> question (the xfs_db logres command can do this, IIRC) and then run
> the math on the reservations held from the dump of all the blocked
> tasks holding locks to see if this matches the size of the journal
> in the fs?
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> dgc@kernel.org
> 

Re: [bug report] fstests generic/774 hang again

[Shin'ichiro Kawasaki]

On Sep 21, 2026 / 17:28, Darrick J. Wong wrote:
> On Thu, Sep 17, 2026 at 07:45:04AM +1000, Dave Chinner wrote:
[...]
> > A bunch of threads waiting on the ILOCK here:
> > 
> >  down_write_nested+0x1c0/0x1f0
> >  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> >  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> >  iomap_dio_complete+0x140/0xb20
> >  iomap_dio_complete_work+0x58/0x90
> >  process_one_work+0x947/0x1760
> > 
> > And the holder:
> > 
> >  schedule+0xe5/0x2e0
> >  xlog_grant_head_wait+0x175/0xac0 [xfs]
> >  xlog_grant_head_check+0x312/0x3f0 [xfs]
> >  xfs_log_regrant+0x380/0x7d0 [xfs]
> >  xfs_trans_roll+0x2d9/0x420 [xfs]
> >  xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
> >  xfs_defer_finish_noroll+0x460/0xe70 [xfs]
> >  xfs_trans_commit+0xfc/0x180 [xfs]
> >  xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
> >  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> >  iomap_dio_complete+0x140/0xb20
> >  iomap_dio_complete_work+0x58/0x90
> >  process_one_work+0x947/0x1760
> > 
> > is waiting on log space whilst holding the ILOCK.
> > 
> > This looks to me like the test runs out of log space because of all
> > the IO completions holding transaction reservations waiting on the
> > ILOCK, whilst the ILOCK holder can't get enough log space to regrant
> > on transaction roll to continue the transaction.
> 
> Agreed.
> 
> Why are we calling xlog_grant_head_check from within
> xfs_reflink_end_atomic_cow?  I think the reason for doing that is
> because we've exhausted t_cnt in the ticket (i.e. we've already rolled
> more than tr_logcount times).
> 
> Oh.  tr_atomic_ioend.tr_logcount is 5 on a rmap+reflink filesystem,
> which it inherits from tr_itruncate.  However, tr_itruncate is only
> intended to remove two extents from a file, so it only needs 5 rolls.
> Coincidentally we calculate 5 rolls for each atomic extent remapping
> operation.
> 
> For atomic write ioends, what if we increased tr_logcount to 5x the
> number of remappings that would have to occur to finish the write?
> That would preallocate all the permanent reservation we'd need before we
> take the ILOCK, which avoids the situation of needing to obtain more log
> space while holding ILOCK.
> 
> The downside is that you'd have to limit the software awu_max even
> further, perhaps to 40% of the log size divided by
> (tr_logres*tr_logcount).  However, we'd still be able to handle
> concurrent atomic writes to different parts of the file, at least until
> fragmentation got bad.

Darrick, thanks for the idea. I wanted to try out the idea, so I cooled
the experimental patch below. Is this what you meant? It is a very rough
patch, and does not reflect the "limit the software awu_max even further"
part, it is just for experiment.

Anyway, I applied this patch on top of the kernel xfs/for-next branch git
hash 0ca15a1a1151, and repeated the test case g774. It did not hang after
100 times repeat, so it looks working.


From c90ec246b0b9e14097b0ad1d1ef5661261dd61db Mon Sep 17 00:00:00 2001
From: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Date: Thu, 24 Sep 2026 19:08:32 +0900
Subject: [PATCH] xfs: try out Darrick's idea for g774 hang

[experimental patch]

Darrick's idea from
https://lore.kernel.org/linux-xfs/20260922002814.GK2705364@frogsfrogsfrogs/

The deadlock of g774 caused by grant space while holding ILOCK. To avoid the
deadlock, allocate enough space before taking the ILOCK by increasing
tr_logcount. 5 rolls are required for each extent, then multiply tr_logcount
by 5.
---
 fs/xfs/xfs_reflink.c | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/fs/xfs/xfs_reflink.c b/fs/xfs/xfs_reflink.c
index 48013613663..9d3da713a01 100644
--- a/fs/xfs/xfs_reflink.c
+++ b/fs/xfs/xfs_reflink.c
@@ -1020,6 +1020,28 @@ xfs_reflink_end_cow(
 	return error;
 }
 
+/*
+ * Number of transaction rolls needed to remap a single extent during an atomic
+ * write ioend.
+ */
+#define XFS_ATOMIC_IOEND_ROLLS	5
+
+/*
+ * Compute the number of log operations to reserve for an atomic write ioend
+ * that remaps @blockcount blocks with a @logres byte reservation.
+ */
+STATIC int
+xfs_calc_atomic_write_ioend_logcount(
+	struct xfs_mount	*mp,
+	xfs_extlen_t		blockcount,
+	unsigned int		logres)
+{
+	if (blockcount == 0 || logres == 0)
+		return M_RES(mp)->tr_atomic_ioend.tr_logcount;
+
+	return blockcount * XFS_ATOMIC_IOEND_ROLLS;
+}
+
 /*
  * Fully remap all of the file's data fork at once, which is the critical part
  * in achieving atomic behaviour.
@@ -1036,6 +1058,7 @@ xfs_reflink_end_atomic_cow(
 	xfs_fileoff_t			end_fsb;
 	int				error = 0;
 	struct xfs_mount		*mp = ip->i_mount;
+	struct xfs_trans_res		resv = M_RES(mp)->tr_atomic_ioend;
 	struct xfs_trans		*tp;
 	unsigned int			resblks;
 
@@ -1051,7 +1074,15 @@ xfs_reflink_end_atomic_cow(
 	resblks = (end_fsb - offset_fsb) *
 			XFS_NEXTENTADD_SPACE_RES(mp, 1, XFS_DATA_FORK);
 
-	error = xfs_trans_alloc(mp, &M_RES(mp)->tr_atomic_ioend, resblks, 0,
+	/*
+	 * Reserve enough log operations to remap every block of this write
+	 * before we take ILOCK, so that no roll below has to wait for grant
+	 * space while holding ILOCK.
+	 */
+	resv.tr_logcount = xfs_calc_atomic_write_ioend_logcount(mp,
+			end_fsb - offset_fsb, resv.tr_logres);
+
+	error = xfs_trans_alloc(mp, &resv, resblks, 0,
 			XFS_TRANS_RESERVE, &tp);
 	if (error)
 		return error;
-- 
2.55.0

Re: [bug report] fstests generic/774 hang again

[Darrick J. Wong]

On Fri, Sep 25, 2026 at 10:38:07AM +0900, Shin'ichiro Kawasaki wrote:
> On Sep 21, 2026 / 17:28, Darrick J. Wong wrote:
> > On Thu, Sep 17, 2026 at 07:45:04AM +1000, Dave Chinner wrote:
> [...]
> > > A bunch of threads waiting on the ILOCK here:
> > > 
> > >  down_write_nested+0x1c0/0x1f0
> > >  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
> > >  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> > >  iomap_dio_complete+0x140/0xb20
> > >  iomap_dio_complete_work+0x58/0x90
> > >  process_one_work+0x947/0x1760
> > > 
> > > And the holder:
> > > 
> > >  schedule+0xe5/0x2e0
> > >  xlog_grant_head_wait+0x175/0xac0 [xfs]
> > >  xlog_grant_head_check+0x312/0x3f0 [xfs]
> > >  xfs_log_regrant+0x380/0x7d0 [xfs]
> > >  xfs_trans_roll+0x2d9/0x420 [xfs]
> > >  xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
> > >  xfs_defer_finish_noroll+0x460/0xe70 [xfs]
> > >  xfs_trans_commit+0xfc/0x180 [xfs]
> > >  xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
> > >  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
> > >  iomap_dio_complete+0x140/0xb20
> > >  iomap_dio_complete_work+0x58/0x90
> > >  process_one_work+0x947/0x1760
> > > 
> > > is waiting on log space whilst holding the ILOCK.
> > > 
> > > This looks to me like the test runs out of log space because of all
> > > the IO completions holding transaction reservations waiting on the
> > > ILOCK, whilst the ILOCK holder can't get enough log space to regrant
> > > on transaction roll to continue the transaction.
> > 
> > Agreed.
> > 
> > Why are we calling xlog_grant_head_check from within
> > xfs_reflink_end_atomic_cow?  I think the reason for doing that is
> > because we've exhausted t_cnt in the ticket (i.e. we've already rolled
> > more than tr_logcount times).
> > 
> > Oh.  tr_atomic_ioend.tr_logcount is 5 on a rmap+reflink filesystem,
> > which it inherits from tr_itruncate.  However, tr_itruncate is only
> > intended to remove two extents from a file, so it only needs 5 rolls.
> > Coincidentally we calculate 5 rolls for each atomic extent remapping
> > operation.
> > 
> > For atomic write ioends, what if we increased tr_logcount to 5x the
> > number of remappings that would have to occur to finish the write?
> > That would preallocate all the permanent reservation we'd need before we
> > take the ILOCK, which avoids the situation of needing to obtain more log
> > space while holding ILOCK.
> > 
> > The downside is that you'd have to limit the software awu_max even
> > further, perhaps to 40% of the log size divided by
> > (tr_logres*tr_logcount).  However, we'd still be able to handle
> > concurrent atomic writes to different parts of the file, at least until
> > fragmentation got bad.
> 
> Darrick, thanks for the idea. I wanted to try out the idea, so I cooled
> the experimental patch below. Is this what you meant? It is a very rough
> patch, and does not reflect the "limit the software awu_max even further"
> part, it is just for experiment.
> 
> Anyway, I applied this patch on top of the kernel xfs/for-next branch git
> hash 0ca15a1a1151, and repeated the test case g774. It did not hang after
> 100 times repeat, so it looks working.
> 
> 
> From c90ec246b0b9e14097b0ad1d1ef5661261dd61db Mon Sep 17 00:00:00 2001
> From: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
> Date: Thu, 24 Sep 2026 19:08:32 +0900
> Subject: [PATCH] xfs: try out Darrick's idea for g774 hang
> 
> [experimental patch]
> 
> Darrick's idea from
> https://lore.kernel.org/linux-xfs/20260922002814.GK2705364@frogsfrogsfrogs/
> 
> The deadlock of g774 caused by grant space while holding ILOCK. To avoid the
> deadlock, allocate enough space before taking the ILOCK by increasing
> tr_logcount. 5 rolls are required for each extent, then multiply tr_logcount
> by 5.
> ---
>  fs/xfs/xfs_reflink.c | 33 ++++++++++++++++++++++++++++++++-
>  1 file changed, 32 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/xfs/xfs_reflink.c b/fs/xfs/xfs_reflink.c
> index 48013613663..9d3da713a01 100644
> --- a/fs/xfs/xfs_reflink.c
> +++ b/fs/xfs/xfs_reflink.c
> @@ -1020,6 +1020,28 @@ xfs_reflink_end_cow(
>  	return error;
>  }
>  
> +/*
> + * Number of transaction rolls needed to remap a single extent during an atomic
> + * write ioend.
> + */
> +#define XFS_ATOMIC_IOEND_ROLLS	5
> +
> +/*
> + * Compute the number of log operations to reserve for an atomic write ioend
> + * that remaps @blockcount blocks with a @logres byte reservation.
> + */
> +STATIC int
> +xfs_calc_atomic_write_ioend_logcount(
> +	struct xfs_mount	*mp,
> +	xfs_extlen_t		blockcount,
> +	unsigned int		logres)
> +{
> +	if (blockcount == 0 || logres == 0)
> +		return M_RES(mp)->tr_atomic_ioend.tr_logcount;
> +
> +	return blockcount * XFS_ATOMIC_IOEND_ROLLS;
> +}
> +
>  /*
>   * Fully remap all of the file's data fork at once, which is the critical part
>   * in achieving atomic behaviour.
> @@ -1036,6 +1058,7 @@ xfs_reflink_end_atomic_cow(
>  	xfs_fileoff_t			end_fsb;
>  	int				error = 0;
>  	struct xfs_mount		*mp = ip->i_mount;
> +	struct xfs_trans_res		resv = M_RES(mp)->tr_atomic_ioend;
>  	struct xfs_trans		*tp;
>  	unsigned int			resblks;
>  
> @@ -1051,7 +1074,15 @@ xfs_reflink_end_atomic_cow(
>  	resblks = (end_fsb - offset_fsb) *
>  			XFS_NEXTENTADD_SPACE_RES(mp, 1, XFS_DATA_FORK);
>  
> -	error = xfs_trans_alloc(mp, &M_RES(mp)->tr_atomic_ioend, resblks, 0,
> +	/*
> +	 * Reserve enough log operations to remap every block of this write
> +	 * before we take ILOCK, so that no roll below has to wait for grant
> +	 * space while holding ILOCK.
> +	 */
> +	resv.tr_logcount = xfs_calc_atomic_write_ioend_logcount(mp,
> +			end_fsb - offset_fsb, resv.tr_logres);
> +
> +	error = xfs_trans_alloc(mp, &resv, resblks, 0,
>  			XFS_TRANS_RESERVE, &tp);
>  	if (error)
>  		return error;

Yeah, I think you're on the right track with the dynamic tr_logcount
changes but I wonder if you want to constrain the max software atomic
write size so that we don't end up with gigantic chains that consume
significant portions of the log for a single write?

Admittedly I see the multi-megabyte atomic write limits and marvel at
how far we've drifted from the original requirements.

--D

diff --git a/fs/xfs/libxfs/xfs_trans_resv.c b/fs/xfs/libxfs/xfs_trans_resv.c
index 1c20a7c27aa1bf..2e84544cd22a0a 100644
--- a/fs/xfs/libxfs/xfs_trans_resv.c
+++ b/fs/xfs/libxfs/xfs_trans_resv.c
@@ -1473,6 +1473,10 @@ xfs_calc_max_atomic_write_fsblocks(
        struct xfs_mount                *mp)
 {
        const struct xfs_trans_res      *resv = &M_RES(mp)->tr_atomic_ioend;
+       const unsigned int              max_log_bytes =
+               XFS_FSB_TO_B(mp, mp->m_sb.sb_logblocks * 10 / 4);
+       const unsigned int              log_bytes_per_aw_block =
+               (resv->tr_logres * resv->tr_logcount * 5) / 4;
        unsigned int                    per_intent = 0;
        unsigned int                    step_size = 0;
        unsigned int                    ret = 0;
@@ -1483,6 +1487,9 @@ xfs_calc_max_atomic_write_fsblocks(
 
                if (resv->tr_logres >= step_size)
                        ret = (resv->tr_logres - step_size) / per_intent;
+
+               if (ret > 0 && log_bytes_per_aw_block > max_log_bytes / ret)
+                       ret = max_log_bytes / log_bytes_per_aw_block;
        }
 
        trace_xfs_calc_max_atomic_write_fsblocks(mp, per_intent, step_size,