Re: all cel/for-rc (1197eb59 lockd: fix nlm_close

Linux XFS filesystem development
 help / color / mirror / Atom feed

* Re: all cel/for-rc (1197eb59 lockd: fix nlm_close_files) results
       [not found]               ` <608CA2C9-464C-464E-B4AD-CF1B8C494A5A@oracle.com>
@ 2022-07-25 13:22                 ` Bruce Fields
  2022-07-25 22:03                   ` Dave Chinner
  0 siblings, 1 reply; 4+ messages in thread
From: Bruce Fields @ 2022-07-25 13:22 UTC (permalink / raw)
  To: Chuck Lever III; +Cc: linux-xfs, linux-nfs

Adding linux-xfs in case anyone has an idea....  I'm seeing this on an
NFS server.  I believe the client was in generic/187 or generic/460.

On Mon, Jul 25, 2022 at 01:14:27PM +0000, Chuck Lever III wrote:
> 
> 
> > On Jul 25, 2022, at 8:16 AM, Bruce Fields <bfields@fieldses.org> wrote:
> > 
> > And again:
> > 
> > [ 3987.996455] INFO: task kworker/u8:2:40 blocked for more than 120
> > seconds.
> > [ 3987.998415]       Not tainted 5.19.0-rc6-00014-g51fd2eb52c0c #3560
> > [ 3988.000005] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> > disables this message.
> > [ 3988.002131] task:kworker/u8:2    state:D stack:24784 pid:   40 ppid:
> > 2 flags:0x00004000
> > [ 3988.004628] Workqueue: xfs-cil/vdf xlog_cil_push_work
> > [ 3988.005841] Call Trace:
> > [ 3988.005968]  <TASK>
> > [ 3988.006052]  __schedule+0xb73/0x2260
> > [ 3988.006251]  ? io_schedule_timeout+0x150/0x150
> > [ 3988.006511]  schedule+0xe0/0x200
> > [ 3988.006676]  xlog_wait_on_iclog+0x435/0x600
> > [ 3988.006914]  ? xfs_log_mount+0x550/0x550
> > [ 3988.007119]  ? mark_held_locks+0x9e/0xe0
> > [ 3988.007311]  ? wake_up_q+0xf0/0xf0
> > [ 3988.007470]  ? rwlock_bug.part.0+0x90/0x90
> > [ 3988.007704]  ? kmem_cache_free.part.0+0x99/0x1a0
> > [ 3988.007977]  xlog_cil_push_work+0xcf4/0x14f0
> > [ 3988.008206]  ? xlog_cil_write_commit_record+0x310/0x310
> > [ 3988.008565]  ? lock_downgrade+0x6a0/0x6a0
> > [ 3988.008796]  ? lock_is_held_type+0xd8/0x130
> > [ 3988.009030]  process_one_work+0x7ee/0x12d0
> > [ 3988.009238]  ? lock_downgrade+0x6a0/0x6a0
> > [ 3988.009432]  ? pwq_dec_nr_in_flight+0x230/0x230
> > [ 3988.009667]  ? rwlock_bug.part.0+0x90/0x90
> > [ 3988.009896]  worker_thread+0xfc/0x1270
> > [ 3988.010259]  ? process_one_work+0x12d0/0x12d0
> > [ 3988.010591]  kthread+0x298/0x340
> > [ 3988.010754]  ? kthread_complete_and_exit+0x20/0x20
> > [ 3988.011018]  ret_from_fork+0x22/0x30
> > [ 3988.011201]  </TASK>
> > [ 3988.011298] INFO: task nfsd:23110 blocked for more than 120 seconds.
> > [ 3988.011746]       Not tainted 5.19.0-rc6-00014-g51fd2eb52c0c #3560
> > [ 3988.012232] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> > disables this message.
> > [ 3988.012828] task:nfsd            state:D stack:24400 pid:23110 ppid:
> > 2 flags:0x00004000
> > [ 3988.013426] Call Trace:
> > [ 3988.013521]  <TASK>
> > [ 3988.013636]  __schedule+0xb73/0x2260
> > [ 3988.013850]  ? io_schedule_timeout+0x150/0x150
> > [ 3988.014160]  ? reacquire_held_locks+0x4e0/0x4e0
> > [ 3988.014409]  ? do_raw_spin_lock+0x11e/0x240
> > [ 3988.014732]  schedule+0xe0/0x200
> > [ 3988.014905]  io_schedule+0xbf/0x130
> > [ 3988.015152]  folio_wait_bit_common+0x305/0x610
> > [ 3988.015387]  ? replace_page_cache_page+0xd30/0xd30
> > [ 3988.015754]  ? folio_unlock+0x70/0x70
> > [ 3988.016016]  ? lock_is_held_type+0xd8/0x130
> > [ 3988.016385]  folio_wait_writeback+0x40/0x230
> > [ 3988.016604]  __filemap_fdatawait_range+0x10a/0x290
> > [ 3988.017149]  ? file_check_and_advance_wb_err+0x270/0x270
> > [ 3988.017448]  ? lock_release+0x380/0x750
> > [ 3988.017708]  file_write_and_wait_range+0xb4/0xf0
> > [ 3988.018028]  xfs_file_fsync+0x146/0x830
> > [ 3988.018279]  ? xfs_file_read_iter+0x450/0x450
> > [ 3988.018569]  ? lock_is_held_type+0xd8/0x130
> > [ 3988.018841]  nfsd_commit+0x304/0x550 [nfsd]
> > [ 3988.019161]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
> > [ 3988.019394]  ? percpu_counter_add_batch+0x77/0x130
> > [ 3988.019736]  nfsd4_proc_compound+0xcbd/0x2070 [nfsd]
> > [ 3988.020121]  nfsd_dispatch+0x4b9/0xb90 [nfsd]
> > [ 3988.020409]  svc_process_common+0xb23/0x1ac0 [sunrpc]
> > [ 3988.020781]  ? svc_create_pooled+0x4d0/0x4d0 [sunrpc]
> > [ 3988.021165]  ? nfsd_svc+0xc60/0xc60 [nfsd]
> > [ 3988.021381]  ? svc_sock_secure_port+0x2a/0x40 [sunrpc]
> > [ 3988.021746]  ? svc_recv+0x119f/0x2890 [sunrpc]
> > [ 3988.022015]  svc_process+0x32d/0x4a0 [sunrpc]
> > [ 3988.022336]  nfsd+0x2d6/0x570 [nfsd]
> > [ 3988.022564]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
> > [ 3988.022874]  kthread+0x298/0x340
> > [ 3988.023018]  ? kthread_complete_and_exit+0x20/0x20
> > [ 3988.023332]  ret_from_fork+0x22/0x30
> > [ 3988.023513]  </TASK>
> > ...
> 
> All three call traces you've sent me show an NFSv4 COMMIT
> waiting in folio_wait_bit_common(). Should you report this
> to the XFS folks? Or maybe a little bisecting might help.

Reproducing is a little slow and not 100% reliable.  I'm not sure I'll
get to it.

--b.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: all cel/for-rc (1197eb59 lockd: fix nlm_close_files) results
  2022-07-25 13:22                 ` all cel/for-rc (1197eb59 lockd: fix nlm_close_files) results Bruce Fields
@ 2022-07-25 22:03                   ` Dave Chinner
  2022-07-25 22:06                     ` Dave Chinner
  0 siblings, 1 reply; 4+ messages in thread
From: Dave Chinner @ 2022-07-25 22:03 UTC (permalink / raw)
  To: Bruce Fields; +Cc: Chuck Lever III, linux-xfs, linux-nfs

On Mon, Jul 25, 2022 at 09:22:52AM -0400, Bruce Fields wrote:
> Adding linux-xfs in case anyone has an idea....  I'm seeing this on an
> NFS server.  I believe the client was in generic/187 or generic/460.
> 
> On Mon, Jul 25, 2022 at 01:14:27PM +0000, Chuck Lever III wrote:
> > 
> > 
> > > On Jul 25, 2022, at 8:16 AM, Bruce Fields <bfields@fieldses.org> wrote:
> > > 
> > > And again:
> > > 
> > > [ 3987.996455] INFO: task kworker/u8:2:40 blocked for more than 120
> > > seconds.
> > > [ 3987.998415]       Not tainted 5.19.0-rc6-00014-g51fd2eb52c0c #3560
> > > [ 3988.000005] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> > > disables this message.
> > > [ 3988.002131] task:kworker/u8:2    state:D stack:24784 pid:   40 ppid:
> > > 2 flags:0x00004000
> > > [ 3988.004628] Workqueue: xfs-cil/vdf xlog_cil_push_work
> > > [ 3988.005841] Call Trace:
> > > [ 3988.005968]  <TASK>
> > > [ 3988.006052]  __schedule+0xb73/0x2260
> > > [ 3988.006251]  ? io_schedule_timeout+0x150/0x150
> > > [ 3988.006511]  schedule+0xe0/0x200
> > > [ 3988.006676]  xlog_wait_on_iclog+0x435/0x600
> > > [ 3988.006914]  ? xfs_log_mount+0x550/0x550
> > > [ 3988.007119]  ? mark_held_locks+0x9e/0xe0
> > > [ 3988.007311]  ? wake_up_q+0xf0/0xf0
> > > [ 3988.007470]  ? rwlock_bug.part.0+0x90/0x90
> > > [ 3988.007704]  ? kmem_cache_free.part.0+0x99/0x1a0
> > > [ 3988.007977]  xlog_cil_push_work+0xcf4/0x14f0
> > > [ 3988.008206]  ? xlog_cil_write_commit_record+0x310/0x310
> > > [ 3988.008565]  ? lock_downgrade+0x6a0/0x6a0
> > > [ 3988.008796]  ? lock_is_held_type+0xd8/0x130
> > > [ 3988.009030]  process_one_work+0x7ee/0x12d0
> > > [ 3988.009238]  ? lock_downgrade+0x6a0/0x6a0
> > > [ 3988.009432]  ? pwq_dec_nr_in_flight+0x230/0x230
> > > [ 3988.009667]  ? rwlock_bug.part.0+0x90/0x90
> > > [ 3988.009896]  worker_thread+0xfc/0x1270
> > > [ 3988.010259]  ? process_one_work+0x12d0/0x12d0
> > > [ 3988.010591]  kthread+0x298/0x340
> > > [ 3988.010754]  ? kthread_complete_and_exit+0x20/0x20
> > > [ 3988.011018]  ret_from_fork+0x22/0x30
> > > [ 3988.011201]  </TASK>
> > > [ 3988.011298] INFO: task nfsd:23110 blocked for more than 120 seconds.
> > > [ 3988.011746]       Not tainted 5.19.0-rc6-00014-g51fd2eb52c0c #3560
> > > [ 3988.012232] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> > > disables this message.
> > > [ 3988.012828] task:nfsd            state:D stack:24400 pid:23110 ppid:
> > > 2 flags:0x00004000
> > > [ 3988.013426] Call Trace:
> > > [ 3988.013521]  <TASK>
> > > [ 3988.013636]  __schedule+0xb73/0x2260
> > > [ 3988.013850]  ? io_schedule_timeout+0x150/0x150
> > > [ 3988.014160]  ? reacquire_held_locks+0x4e0/0x4e0
> > > [ 3988.014409]  ? do_raw_spin_lock+0x11e/0x240
> > > [ 3988.014732]  schedule+0xe0/0x200
> > > [ 3988.014905]  io_schedule+0xbf/0x130
> > > [ 3988.015152]  folio_wait_bit_common+0x305/0x610
> > > [ 3988.015387]  ? replace_page_cache_page+0xd30/0xd30
> > > [ 3988.015754]  ? folio_unlock+0x70/0x70
> > > [ 3988.016016]  ? lock_is_held_type+0xd8/0x130
> > > [ 3988.016385]  folio_wait_writeback+0x40/0x230
> > > [ 3988.016604]  __filemap_fdatawait_range+0x10a/0x290
> > > [ 3988.017149]  ? file_check_and_advance_wb_err+0x270/0x270
> > > [ 3988.017448]  ? lock_release+0x380/0x750
> > > [ 3988.017708]  file_write_and_wait_range+0xb4/0xf0
> > > [ 3988.018028]  xfs_file_fsync+0x146/0x830
> > > [ 3988.018279]  ? xfs_file_read_iter+0x450/0x450
> > > [ 3988.018569]  ? lock_is_held_type+0xd8/0x130
> > > [ 3988.018841]  nfsd_commit+0x304/0x550 [nfsd]
> > > [ 3988.019161]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
> > > [ 3988.019394]  ? percpu_counter_add_batch+0x77/0x130
> > > [ 3988.019736]  nfsd4_proc_compound+0xcbd/0x2070 [nfsd]
> > > [ 3988.020121]  nfsd_dispatch+0x4b9/0xb90 [nfsd]
> > > [ 3988.020409]  svc_process_common+0xb23/0x1ac0 [sunrpc]
> > > [ 3988.020781]  ? svc_create_pooled+0x4d0/0x4d0 [sunrpc]
> > > [ 3988.021165]  ? nfsd_svc+0xc60/0xc60 [nfsd]
> > > [ 3988.021381]  ? svc_sock_secure_port+0x2a/0x40 [sunrpc]
> > > [ 3988.021746]  ? svc_recv+0x119f/0x2890 [sunrpc]
> > > [ 3988.022015]  svc_process+0x32d/0x4a0 [sunrpc]
> > > [ 3988.022336]  nfsd+0x2d6/0x570 [nfsd]
> > > [ 3988.022564]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
> > > [ 3988.022874]  kthread+0x298/0x340
> > > [ 3988.023018]  ? kthread_complete_and_exit+0x20/0x20
> > > [ 3988.023332]  ret_from_fork+0x22/0x30
> > > [ 3988.023513]  </TASK>
> > > ...
> > 
> > All three call traces you've sent me show an NFSv4 COMMIT
> > waiting in folio_wait_bit_common(). Should you report this
> > to the XFS folks? Or maybe a little bisecting might help.
> 
> Reproducing is a little slow and not 100% reliable.  I'm not sure I'll
> get to it.

Both tasks are waiting on IO completion, and they are waiting on
independent IOs which means it's likely not a filesystem layer
problem being tripped over. i.e. it's likely an issue lower in the
storage stack (or hardware) that is causing this.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: all cel/for-rc (1197eb59 lockd: fix nlm_close_files) results
  2022-07-25 22:03                   ` Dave Chinner
@ 2022-07-25 22:06                     ` Dave Chinner
  2022-09-03 15:55                       ` J. Bruce Fields
  0 siblings, 1 reply; 4+ messages in thread
From: Dave Chinner @ 2022-07-25 22:06 UTC (permalink / raw)
  To: Bruce Fields; +Cc: Chuck Lever III, linux-xfs, linux-nfs

On Tue, Jul 26, 2022 at 08:03:20AM +1000, Dave Chinner wrote:
> On Mon, Jul 25, 2022 at 09:22:52AM -0400, Bruce Fields wrote:
> > Adding linux-xfs in case anyone has an idea....  I'm seeing this on an
> > NFS server.  I believe the client was in generic/187 or generic/460.
> > 
> > On Mon, Jul 25, 2022 at 01:14:27PM +0000, Chuck Lever III wrote:
> > > 
> > > 
> > > > On Jul 25, 2022, at 8:16 AM, Bruce Fields <bfields@fieldses.org> wrote:
> > > > 
> > > > And again:
> > > > 
> > > > [ 3987.996455] INFO: task kworker/u8:2:40 blocked for more than 120
> > > > seconds.
> > > > [ 3987.998415]       Not tainted 5.19.0-rc6-00014-g51fd2eb52c0c #3560
> > > > [ 3988.000005] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> > > > disables this message.
> > > > [ 3988.002131] task:kworker/u8:2    state:D stack:24784 pid:   40 ppid:
> > > > 2 flags:0x00004000
> > > > [ 3988.004628] Workqueue: xfs-cil/vdf xlog_cil_push_work
> > > > [ 3988.005841] Call Trace:
> > > > [ 3988.005968]  <TASK>
> > > > [ 3988.006052]  __schedule+0xb73/0x2260
> > > > [ 3988.006251]  ? io_schedule_timeout+0x150/0x150
> > > > [ 3988.006511]  schedule+0xe0/0x200
> > > > [ 3988.006676]  xlog_wait_on_iclog+0x435/0x600
> > > > [ 3988.006914]  ? xfs_log_mount+0x550/0x550
> > > > [ 3988.007119]  ? mark_held_locks+0x9e/0xe0
> > > > [ 3988.007311]  ? wake_up_q+0xf0/0xf0
> > > > [ 3988.007470]  ? rwlock_bug.part.0+0x90/0x90
> > > > [ 3988.007704]  ? kmem_cache_free.part.0+0x99/0x1a0
> > > > [ 3988.007977]  xlog_cil_push_work+0xcf4/0x14f0
> > > > [ 3988.008206]  ? xlog_cil_write_commit_record+0x310/0x310
> > > > [ 3988.008565]  ? lock_downgrade+0x6a0/0x6a0
> > > > [ 3988.008796]  ? lock_is_held_type+0xd8/0x130
> > > > [ 3988.009030]  process_one_work+0x7ee/0x12d0
> > > > [ 3988.009238]  ? lock_downgrade+0x6a0/0x6a0
> > > > [ 3988.009432]  ? pwq_dec_nr_in_flight+0x230/0x230
> > > > [ 3988.009667]  ? rwlock_bug.part.0+0x90/0x90
> > > > [ 3988.009896]  worker_thread+0xfc/0x1270
> > > > [ 3988.010259]  ? process_one_work+0x12d0/0x12d0
> > > > [ 3988.010591]  kthread+0x298/0x340
> > > > [ 3988.010754]  ? kthread_complete_and_exit+0x20/0x20
> > > > [ 3988.011018]  ret_from_fork+0x22/0x30
> > > > [ 3988.011201]  </TASK>
> > > > [ 3988.011298] INFO: task nfsd:23110 blocked for more than 120 seconds.
> > > > [ 3988.011746]       Not tainted 5.19.0-rc6-00014-g51fd2eb52c0c #3560
> > > > [ 3988.012232] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> > > > disables this message.
> > > > [ 3988.012828] task:nfsd            state:D stack:24400 pid:23110 ppid:
> > > > 2 flags:0x00004000
> > > > [ 3988.013426] Call Trace:
> > > > [ 3988.013521]  <TASK>
> > > > [ 3988.013636]  __schedule+0xb73/0x2260
> > > > [ 3988.013850]  ? io_schedule_timeout+0x150/0x150
> > > > [ 3988.014160]  ? reacquire_held_locks+0x4e0/0x4e0
> > > > [ 3988.014409]  ? do_raw_spin_lock+0x11e/0x240
> > > > [ 3988.014732]  schedule+0xe0/0x200
> > > > [ 3988.014905]  io_schedule+0xbf/0x130
> > > > [ 3988.015152]  folio_wait_bit_common+0x305/0x610
> > > > [ 3988.015387]  ? replace_page_cache_page+0xd30/0xd30
> > > > [ 3988.015754]  ? folio_unlock+0x70/0x70
> > > > [ 3988.016016]  ? lock_is_held_type+0xd8/0x130
> > > > [ 3988.016385]  folio_wait_writeback+0x40/0x230
> > > > [ 3988.016604]  __filemap_fdatawait_range+0x10a/0x290
> > > > [ 3988.017149]  ? file_check_and_advance_wb_err+0x270/0x270
> > > > [ 3988.017448]  ? lock_release+0x380/0x750
> > > > [ 3988.017708]  file_write_and_wait_range+0xb4/0xf0
> > > > [ 3988.018028]  xfs_file_fsync+0x146/0x830
> > > > [ 3988.018279]  ? xfs_file_read_iter+0x450/0x450
> > > > [ 3988.018569]  ? lock_is_held_type+0xd8/0x130
> > > > [ 3988.018841]  nfsd_commit+0x304/0x550 [nfsd]
> > > > [ 3988.019161]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
> > > > [ 3988.019394]  ? percpu_counter_add_batch+0x77/0x130
> > > > [ 3988.019736]  nfsd4_proc_compound+0xcbd/0x2070 [nfsd]
> > > > [ 3988.020121]  nfsd_dispatch+0x4b9/0xb90 [nfsd]
> > > > [ 3988.020409]  svc_process_common+0xb23/0x1ac0 [sunrpc]
> > > > [ 3988.020781]  ? svc_create_pooled+0x4d0/0x4d0 [sunrpc]
> > > > [ 3988.021165]  ? nfsd_svc+0xc60/0xc60 [nfsd]
> > > > [ 3988.021381]  ? svc_sock_secure_port+0x2a/0x40 [sunrpc]
> > > > [ 3988.021746]  ? svc_recv+0x119f/0x2890 [sunrpc]
> > > > [ 3988.022015]  svc_process+0x32d/0x4a0 [sunrpc]
> > > > [ 3988.022336]  nfsd+0x2d6/0x570 [nfsd]
> > > > [ 3988.022564]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
> > > > [ 3988.022874]  kthread+0x298/0x340
> > > > [ 3988.023018]  ? kthread_complete_and_exit+0x20/0x20
> > > > [ 3988.023332]  ret_from_fork+0x22/0x30
> > > > [ 3988.023513]  </TASK>
> > > > ...
> > > 
> > > All three call traces you've sent me show an NFSv4 COMMIT
> > > waiting in folio_wait_bit_common(). Should you report this
> > > to the XFS folks? Or maybe a little bisecting might help.
> > 
> > Reproducing is a little slow and not 100% reliable.  I'm not sure I'll
> > get to it.
> 
> Both tasks are waiting on IO completion, and they are waiting on
> independent IOs which means it's likely not a filesystem layer
> problem being tripped over. i.e. it's likely an issue lower in the
> storage stack (or hardware) that is causing this.

Hit send too soon - sysrq-w output when it's all jammed up will tell
us everything that is actually blocked, and likely point more
closely at where things have gone wrong...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: all cel/for-rc (1197eb59 lockd: fix nlm_close_files) results
  2022-07-25 22:06                     ` Dave Chinner
@ 2022-09-03 15:55                       ` J. Bruce Fields
  0 siblings, 0 replies; 4+ messages in thread
From: J. Bruce Fields @ 2022-09-03 15:55 UTC (permalink / raw)
  To: Dave Chinner; +Cc: Chuck Lever III, linux-xfs, linux-nfs

Sorry for being slow to get back to this.  Here's dmesg from the first "blocked" message, this time including a sysrq-w dump at the end.

This is Chuck's for-next (bb49f295d682, which is based on 6.0-rc3).

The client is hung in generic/590 with "server not responding, still trying"
messages.

For what it's worth, the server is a VM, and the exported xfs filesystem
is on a virtio disk (which in turn is backed by a file on an xfs
filesystem on an ssd).  The host seems to be behaving normally.

--b.

[ 5438.560575] INFO: task nfsd:23105 blocked for more than 120 seconds.
[ 5438.562574]       Not tainted 6.0.0-rc3-00008-gbb49f295d682 #3631
[ 5438.563612] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5438.564823] task:nfsd            state:D stack:24336 pid:23105 ppid:     2 flags:0x00004000
[ 5438.566358] Call Trace:
[ 5438.566682]  <TASK>
[ 5438.566874]  __schedule+0xb73/0x2260
[ 5438.567335]  ? io_schedule_timeout+0x160/0x160
[ 5438.567897]  ? reacquire_held_locks+0x4e0/0x4e0
[ 5438.568587]  ? do_raw_spin_lock+0x11e/0x250
[ 5438.569239]  schedule+0x133/0x230
[ 5438.569624]  io_schedule+0xc3/0x140
[ 5438.570040]  folio_wait_bit_common+0x2e4/0x610
[ 5438.570618]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[ 5438.571742]  ? folio_unlock+0x70/0x70
[ 5438.572480]  ? lock_is_held_type+0xd8/0x130
[ 5438.573034]  folio_wait_writeback+0x40/0x230
[ 5438.573580]  __filemap_fdatawait_range+0x10a/0x2a0
[ 5438.574161]  ? filemap_fdatawrite+0x170/0x170
[ 5438.574632]  ? lock_is_held_type+0xd8/0x130
[ 5438.575040]  file_write_and_wait_range+0xbc/0xf0
[ 5438.575295]  xfs_file_fsync+0x146/0x830
[ 5438.575572]  ? xfs_dio_write_end_io+0x760/0x760
[ 5438.575802]  ? lock_is_held_type+0xd8/0x130
[ 5438.576166]  nfsd_commit+0x304/0x560 [nfsd]
[ 5438.576578]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[ 5438.576823]  ? percpu_counter_add_batch+0x77/0x130
[ 5438.577312]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[ 5438.577832]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[ 5438.578332]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[ 5438.578838]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[ 5438.579273]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[ 5438.579574]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[ 5438.580039]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[ 5438.580482]  svc_process+0x32d/0x4a0 [sunrpc]
[ 5438.581265]  nfsd+0x2d6/0x570 [nfsd]
[ 5438.581936]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[ 5438.582530]  kthread+0x298/0x340
[ 5438.582817]  ? kthread_complete_and_exit+0x20/0x20
[ 5438.583090]  ret_from_fork+0x22/0x30
[ 5438.583334]  </TASK>
[ 5438.583453] INFO: task nfsd:23106 blocked for more than 120 seconds.
[ 5438.583957]       Not tainted 6.0.0-rc3-00008-gbb49f295d682 #3631
[ 5438.584552] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5438.585390] task:nfsd            state:D stack:24448 pid:23106 ppid:     2 flags:0x00004000
[ 5438.586255] Call Trace:
[ 5438.586484]  <TASK>
[ 5438.586582]  __schedule+0xb73/0x2260
[ 5438.586911]  ? io_schedule_timeout+0x160/0x160
[ 5438.587320]  ? sbitmap_get_shallow+0x189/0x5f0
[ 5438.587656]  schedule+0x133/0x230
[ 5438.587855]  io_schedule+0xc3/0x140
[ 5438.588131]  blk_mq_get_tag+0x38c/0xa80
[ 5438.588374]  ? blk_mq_get_tags+0x240/0x240
[ 5438.588591]  ? lock_is_held_type+0xd8/0x130
[ 5438.588817]  ? sched_debug_start+0x180/0x180
[ 5438.589153]  ? __bio_split_to_limits+0x712/0x1080
[ 5438.589421]  __blk_mq_alloc_requests+0x492/0xc50
[ 5438.589687]  ? reacquire_held_locks+0x4e0/0x4e0
[ 5438.589984]  ? blk_mq_timeout_work+0x380/0x380
[ 5438.590265]  blk_mq_submit_bio+0x73c/0x17d0
[ 5438.590565]  ? blk_mq_try_issue_list_directly+0x9a0/0x9a0
[ 5438.590914]  submit_bio_noacct_nocheck+0x5b1/0x820
[ 5438.591289]  ? should_fail_request+0x70/0x70
[ 5438.591558]  ? submit_bio_noacct+0xa50/0x1140
[ 5438.591826]  ? iomap_page_create+0x8b/0x3d0
[ 5438.592122]  iomap_do_writepage+0x1210/0x2450
[ 5438.592409]  ? iomap_write_end+0x9c0/0x9c0
[ 5438.592671]  write_cache_pages+0x3d8/0xed0
[ 5438.592971]  ? iomap_write_end+0x9c0/0x9c0
[ 5438.593196]  ? folio_wait_stable+0xc0/0xc0
[ 5438.593413]  ? lock_is_held_type+0xd8/0x130
[ 5438.593649]  ? do_raw_spin_lock+0x11e/0x250
[ 5438.593879]  iomap_writepages+0x42/0xb0
[ 5438.594085]  xfs_vm_writepages+0x109/0x160
[ 5438.594303]  ? xfs_vm_read_folio+0x20/0x20
[ 5438.594529]  ? nfsd_setuser+0x5fb/0x9c0 [nfsd]
[ 5438.594805]  do_writepages+0x15c/0x590
[ 5438.595002]  ? lock_is_held_type+0xd8/0x130
[ 5438.595230]  ? page_writeback_cpu_online+0x20/0x20
[ 5438.595494]  ? lock_release+0x380/0x760
[ 5438.595696]  ? __lock_acquire+0xc67/0x69f0
[ 5438.595920]  ? lock_is_held_type+0xd8/0x130
[ 5438.596170]  __filemap_fdatawrite_range+0x13e/0x170
[ 5438.596471]  ? rcu_read_lock_sched_held+0x3f/0x70
[ 5438.596734]  ? delete_from_page_cache_batch+0xa30/0xa30
[ 5438.597042]  ? lock_is_held_type+0xd8/0x130
[ 5438.597316]  file_write_and_wait_range+0x8e/0xf0
[ 5438.597587]  xfs_file_fsync+0x146/0x830
[ 5438.597790]  ? xfs_dio_write_end_io+0x760/0x760
[ 5438.598045]  ? lock_is_held_type+0xd8/0x130
[ 5438.598273]  nfsd_commit+0x304/0x560 [nfsd]
[ 5438.598527]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[ 5438.598783]  ? percpu_counter_add_batch+0x77/0x130
[ 5438.599058]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[ 5438.599375]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[ 5438.599640]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[ 5438.599980]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[ 5438.600370]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[ 5438.600656]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[ 5438.601049]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[ 5438.601382]  svc_process+0x32d/0x4a0 [sunrpc]
[ 5438.601713]  nfsd+0x2d6/0x570 [nfsd]
[ 5438.601974]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[ 5438.602284]  kthread+0x298/0x340
[ 5438.602437]  ? kthread_complete_and_exit+0x20/0x20
[ 5438.602698]  ret_from_fork+0x22/0x30
[ 5438.602879]  </TASK>
[ 5438.602960] INFO: task nfsd:23107 blocked for more than 120 seconds.
[ 5438.603304]       Not tainted 6.0.0-rc3-00008-gbb49f295d682 #3631
[ 5438.603631] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5438.604119] task:nfsd            state:D stack:24608 pid:23107 ppid:     2 flags:0x00004000
[ 5438.604644] Call Trace:
[ 5438.604737]  <TASK>
[ 5438.604812]  __schedule+0xb73/0x2260
[ 5438.605013]  ? io_schedule_timeout+0x160/0x160
[ 5438.605237]  ? reacquire_held_locks+0x4e0/0x4e0
[ 5438.605485]  ? do_raw_spin_lock+0x11e/0x250
[ 5438.605718]  schedule+0x133/0x230
[ 5438.605870]  io_schedule+0xc3/0x140
[ 5438.606036]  folio_wait_bit_common+0x2e4/0x610
[ 5438.606268]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[ 5438.606583]  ? folio_unlock+0x70/0x70
[ 5438.606756]  ? lock_is_held_type+0xd8/0x130
[ 5438.607031]  folio_wait_writeback+0x40/0x230
[ 5438.607248]  __filemap_fdatawait_range+0x10a/0x2a0
[ 5438.607497]  ? filemap_fdatawrite+0x170/0x170
[ 5438.607716]  ? lock_is_held_type+0xd8/0x130
[ 5438.607942]  file_write_and_wait_range+0xbc/0xf0
[ 5438.608261]  xfs_file_fsync+0x146/0x830
[ 5438.608462]  ? xfs_dio_write_end_io+0x760/0x760
[ 5438.608692]  ? lock_is_held_type+0xd8/0x130
[ 5438.608903]  nfsd_commit+0x304/0x560 [nfsd]
[ 5438.609142]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[ 5438.609380]  ? percpu_counter_add_batch+0x77/0x130
[ 5438.609649]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[ 5438.609942]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[ 5438.610191]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[ 5438.610493]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[ 5438.610788]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[ 5438.611010]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[ 5438.611310]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[ 5438.611567]  svc_process+0x32d/0x4a0 [sunrpc]
[ 5438.611823]  nfsd+0x2d6/0x570 [nfsd]
[ 5438.612018]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[ 5438.612412]  kthread+0x298/0x340
[ 5438.612559]  ? kthread_complete_and_exit+0x20/0x20
[ 5438.612807]  ret_from_fork+0x22/0x30
[ 5438.612994]  </TASK>
[ 5438.613070] INFO: task nfsd:23108 blocked for more than 120 seconds.
[ 5438.613423]       Not tainted 6.0.0-rc3-00008-gbb49f295d682 #3631
[ 5438.613759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5438.614198] task:nfsd            state:D stack:24432 pid:23108 ppid:     2 flags:0x00004000
[ 5438.614675] Call Trace:
[ 5438.614767]  <TASK>
[ 5438.614841]  __schedule+0xb73/0x2260
[ 5438.615019]  ? io_schedule_timeout+0x160/0x160
[ 5438.615242]  ? reacquire_held_locks+0x4e0/0x4e0
[ 5438.615472]  ? do_raw_spin_lock+0x11e/0x250
[ 5438.615683]  schedule+0x133/0x230
[ 5438.615835]  io_schedule+0xc3/0x140
[ 5438.616001]  folio_wait_bit_common+0x2e4/0x610
[ 5438.616325]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[ 5438.616642]  ? folio_unlock+0x70/0x70
[ 5438.616838]  ? lock_is_held_type+0xd8/0x130
[ 5438.617077]  folio_wait_writeback+0x40/0x230
[ 5438.617313]  __filemap_fdatawait_range+0x10a/0x2a0
[ 5438.617679]  ? filemap_fdatawrite+0x170/0x170
[ 5438.617929]  ? lock_is_held_type+0xd8/0x130
[ 5438.618173]  file_write_and_wait_range+0xbc/0xf0
[ 5438.618412]  xfs_file_fsync+0x146/0x830
[ 5438.618598]  ? xfs_dio_write_end_io+0x760/0x760
[ 5438.618827]  ? lock_is_held_type+0xd8/0x130
[ 5438.619043]  nfsd_commit+0x304/0x560 [nfsd]
[ 5438.619278]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[ 5438.619515]  ? percpu_counter_add_batch+0x77/0x130
[ 5438.619764]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[ 5438.620108]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[ 5438.620404]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[ 5438.620708]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[ 5438.621007]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[ 5438.621227]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[ 5438.621547]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[ 5438.621804]  svc_process+0x32d/0x4a0 [sunrpc]
[ 5438.622063]  nfsd+0x2d6/0x570 [nfsd]
[ 5438.622257]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[ 5438.622550]  kthread+0x298/0x340
[ 5438.622695]  ? kthread_complete_and_exit+0x20/0x20
[ 5438.622943]  ret_from_fork+0x22/0x30
[ 5438.623150]  </TASK>
[ 5438.623227] INFO: task nfsd:23109 blocked for more than 120 seconds.
[ 5438.623569]       Not tainted 6.0.0-rc3-00008-gbb49f295d682 #3631
[ 5438.623896] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5438.624438] task:nfsd            state:D stack:24000 pid:23109 ppid:     2 flags:0x00004000
[ 5438.624917] Call Trace:
[ 5438.625013]  <TASK>
[ 5438.625088]  __schedule+0xb73/0x2260
[ 5438.625264]  ? io_schedule_timeout+0x160/0x160
[ 5438.625507]  ? reacquire_held_locks+0x4e0/0x4e0
[ 5438.625736]  ? do_raw_spin_lock+0x11e/0x250
[ 5438.625947]  schedule+0x133/0x230
[ 5438.626103]  io_schedule+0xc3/0x140
[ 5438.626265]  folio_wait_bit_common+0x2e4/0x610
[ 5438.626496]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[ 5438.626811]  ? folio_unlock+0x70/0x70
[ 5438.626987]  ? lock_is_held_type+0xd8/0x130
[ 5438.627199]  folio_wait_writeback+0x40/0x230
[ 5438.627416]  __filemap_fdatawait_range+0x10a/0x2a0
[ 5438.627665]  ? filemap_fdatawrite+0x170/0x170
[ 5438.627884]  ? lock_is_held_type+0xd8/0x130
[ 5438.628164]  file_write_and_wait_range+0xbc/0xf0
[ 5438.628451]  xfs_file_fsync+0x146/0x830
[ 5438.628639]  ? xfs_dio_write_end_io+0x760/0x760
[ 5438.628869]  ? lock_is_held_type+0xd8/0x130
[ 5438.629084]  nfsd_commit+0x304/0x560 [nfsd]
[ 5438.629318]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[ 5438.629576]  ? percpu_counter_add_batch+0x77/0x130
[ 5438.629825]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[ 5438.630121]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[ 5438.630367]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[ 5438.630668]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[ 5438.630966]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[ 5438.631185]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[ 5438.631484]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[ 5438.631742]  svc_process+0x32d/0x4a0 [sunrpc]
[ 5438.632001]  nfsd+0x2d6/0x570 [nfsd]
[ 5438.632288]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[ 5438.632585]  kthread+0x298/0x340
[ 5438.632730]  ? kthread_complete_and_exit+0x20/0x20
[ 5438.632983]  ret_from_fork+0x22/0x30
[ 5438.633165]  </TASK>
[ 5438.633240] INFO: task nfsd:23110 blocked for more than 120 seconds.
[ 5438.633604]       Not tainted 6.0.0-rc3-00008-gbb49f295d682 #3631
[ 5438.633930] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5438.634368] task:nfsd            state:D stack:23952 pid:23110 ppid:     2 flags:0x00004000
[ 5438.634846] Call Trace:
[ 5438.634937]  <TASK>
[ 5438.635014]  __schedule+0xb73/0x2260
[ 5438.635190]  ? io_schedule_timeout+0x160/0x160
[ 5438.635413]  ? reacquire_held_locks+0x4e0/0x4e0
[ 5438.635642]  ? do_raw_spin_lock+0x11e/0x250
[ 5438.635853]  schedule+0x133/0x230
[ 5438.636008]  io_schedule+0xc3/0x140
[ 5438.636220]  folio_wait_bit_common+0x2e4/0x610
[ 5438.636500]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[ 5438.636817]  ? folio_unlock+0x70/0x70
[ 5438.637047]  ? lock_is_held_type+0xd8/0x130
[ 5438.637260]  folio_wait_writeback+0x40/0x230
[ 5438.637497]  __filemap_fdatawait_range+0x10a/0x2a0
[ 5438.637837]  ? filemap_fdatawrite+0x170/0x170
[ 5438.638071]  ? lock_is_held_type+0xd8/0x130
[ 5438.638297]  file_write_and_wait_range+0xbc/0xf0
[ 5438.638533]  xfs_file_fsync+0x146/0x830
[ 5438.638719]  ? xfs_dio_write_end_io+0x760/0x760
[ 5438.638948]  ? lock_is_held_type+0xd8/0x130
[ 5438.639164]  nfsd_commit+0x304/0x560 [nfsd]
[ 5438.639398]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[ 5438.639634]  ? percpu_counter_add_batch+0x77/0x130
[ 5438.639883]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[ 5438.640260]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[ 5438.640523]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[ 5438.640825]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[ 5438.641123]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[ 5438.641343]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[ 5438.641664]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[ 5438.641922]  svc_process+0x32d/0x4a0 [sunrpc]
[ 5438.642181]  nfsd+0x2d6/0x570 [nfsd]
[ 5438.642374]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[ 5438.642667]  kthread+0x298/0x340
[ 5438.642812]  ? kthread_complete_and_exit+0x20/0x20
[ 5438.643063]  ret_from_fork+0x22/0x30
[ 5438.643245]  </TASK>
[ 5438.643321] INFO: task nfsd:23111 blocked for more than 120 seconds.
[ 5438.643663]       Not tainted 6.0.0-rc3-00008-gbb49f295d682 #3631
[ 5438.643991] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5438.644520] task:nfsd            state:D stack:23808 pid:23111 ppid:     2 flags:0x00004000
[ 5438.645004] Call Trace:
[ 5438.645096]  <TASK>
[ 5438.645171]  __schedule+0xb73/0x2260
[ 5438.645346]  ? io_schedule_timeout+0x160/0x160
[ 5438.645590]  ? reacquire_held_locks+0x4e0/0x4e0
[ 5438.645819]  ? do_raw_spin_lock+0x11e/0x250
[ 5438.646033]  schedule+0x133/0x230
[ 5438.646186]  io_schedule+0xc3/0x140
[ 5438.646348]  folio_wait_bit_common+0x2e4/0x610
[ 5438.646578]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[ 5438.646893]  ? folio_unlock+0x70/0x70
[ 5438.647087]  ? lock_is_held_type+0xd8/0x130
[ 5438.647302]  folio_wait_writeback+0x40/0x230
[ 5438.647519]  __filemap_fdatawait_range+0x10a/0x2a0
[ 5438.647767]  ? filemap_fdatawrite+0x170/0x170
[ 5438.648048]  ? lock_is_held_type+0xd8/0x130
[ 5438.648348]  file_write_and_wait_range+0xbc/0xf0
[ 5438.648587]  xfs_file_fsync+0x146/0x830
[ 5438.648773]  ? xfs_dio_write_end_io+0x760/0x760
[ 5438.649007]  ? lock_is_held_type+0xd8/0x130
[ 5438.649218]  nfsd_commit+0x304/0x560 [nfsd]
[ 5438.649464]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[ 5438.649710]  ? percpu_counter_add_batch+0x77/0x130
[ 5438.649961]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[ 5438.650254]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[ 5438.650499]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[ 5438.650799]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[ 5438.651096]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[ 5438.651316]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[ 5438.651615]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[ 5438.651873]  svc_process+0x32d/0x4a0 [sunrpc]
[ 5438.652181]  nfsd+0x2d6/0x570 [nfsd]
[ 5438.652423]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[ 5438.652719]  kthread+0x298/0x340
[ 5438.652865]  ? kthread_complete_and_exit+0x20/0x20
[ 5438.653117]  ret_from_fork+0x22/0x30
[ 5438.653299]  </TASK>
[ 5438.653375] INFO: task nfsd:23112 blocked for more than 120 seconds.
[ 5438.653738]       Not tainted 6.0.0-rc3-00008-gbb49f295d682 #3631
[ 5438.654067] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5438.654502] task:nfsd            state:D stack:21968 pid:23112 ppid:     2 flags:0x00004000
[ 5438.654982] Call Trace:
[ 5438.655073]  <TASK>
[ 5438.655148]  __schedule+0xb73/0x2260
[ 5438.655323]  ? io_schedule_timeout+0x160/0x160
[ 5438.655546]  ? reacquire_held_locks+0x4e0/0x4e0
[ 5438.655776]  ? do_raw_spin_lock+0x11e/0x250
[ 5438.655989]  schedule+0x133/0x230
[ 5438.656190]  io_schedule+0xc3/0x140
[ 5438.656404]  folio_wait_bit_common+0x2e4/0x610
[ 5438.656637]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[ 5438.656956]  ? folio_unlock+0x70/0x70
[ 5438.657130]  ? lock_is_held_type+0xd8/0x130
[ 5438.657342]  folio_wait_writeback+0x40/0x230
[ 5438.657580]  __filemap_fdatawait_range+0x10a/0x2a0
[ 5438.657828]  ? filemap_fdatawrite+0x170/0x170
[ 5438.658157]  ? lock_is_held_type+0xd8/0x130
[ 5438.658505]  file_write_and_wait_range+0xbc/0xf0
[ 5438.658771]  xfs_file_fsync+0x146/0x830
[ 5438.659083]  ? xfs_dio_write_end_io+0x760/0x760
[ 5438.659527]  ? lock_is_held_type+0xd8/0x130
[ 5438.659851]  nfsd_commit+0x304/0x560 [nfsd]
[ 5438.660339]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[ 5438.660686]  ? percpu_counter_add_batch+0x77/0x130
[ 5438.661067]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[ 5438.661360]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[ 5438.661770]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[ 5438.662440]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[ 5438.662807]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[ 5438.663154]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[ 5438.663467]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[ 5438.663908]  svc_process+0x32d/0x4a0 [sunrpc]
[ 5438.664250]  nfsd+0x2d6/0x570 [nfsd]
[ 5438.664453]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[ 5438.664852]  kthread+0x298/0x340
[ 5438.665093]  ? kthread_complete_and_exit+0x20/0x20
[ 5438.665370]  ret_from_fork+0x22/0x30
[ 5438.665695]  </TASK>
[ 5438.665785] 
               Showing all locks held in the system:
[ 5438.666128] 1 lock held by rcu_tasks_kthre/10:
[ 5438.666346]  #0: ffffffff850cd970 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x2e/0xd70
[ 5438.667112] 1 lock held by rcu_tasks_rude_/11:
[ 5438.667454]  #0: ffffffff850cd6d0 (rcu_tasks_rude.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x2e/0xd70
[ 5438.668212] 1 lock held by rcu_tasks_trace/12:
[ 5438.668583]  #0: ffffffff850cd3d0 (rcu_tasks_trace.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x2e/0xd70
[ 5438.669222] 1 lock held by khungtaskd/38:
[ 5438.669413]  #0: ffffffff850ce440 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x53/0x260
[ 5438.669969] 1 lock held by systemd-journal/2571:
[ 5438.670207]  #0: ffff888016f6e548 (mapping.invalidate_lock#2){++++}-{3:3}, at: filemap_fault+0x336/0x1f60
[ 5438.670770] 3 locks held by kworker/u8:1/24268:
[ 5438.671041]  #0: ffff88800869b138 ((wq_completion)writeback){+.+.}-{0:0}, at: process_one_work+0x70a/0x12e0
[ 5438.671791]  #1: ffff88801db27dd0 ((work_completion)(&(&wb->dwork)->work)){+.+.}-{0:0}, at: process_one_work+0x738/0x12e0
[ 5438.672638]  #2: ffff8880125f80e0 (&type->s_umount_key#48){++++}-{3:3}, at: trylock_super+0x16/0xc0
[ 5438.673237] 2 locks held by kworker/2:0/24284:
[ 5438.673480]  #0: ffff88800d5b9938 ((wq_completion)xfs-sync/vdf){+.+.}-{0:0}, at: process_one_work+0x70a/0x12e0
[ 5438.674077]  #1: ffff888047297dd0 ((work_completion)(&(&log->l_work)->work)){+.+.}-{0:0}, at: process_one_work+0x738/0x12e0

[ 5438.674759] =============================================

[ 5559.392647] INFO: task nfsd:23105 blocked for more than 241 seconds.
[ 5559.394498]       Not tainted 6.0.0-rc3-00008-gbb49f295d682 #3631
[ 5559.399159] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5559.403011] task:nfsd            state:D stack:24336 pid:23105 ppid:     2 flags:0x00004000
[ 5559.403552] Call Trace:
[ 5559.403655]  <TASK>
[ 5559.403738]  __schedule+0xb73/0x2260
[ 5559.403956]  ? io_schedule_timeout+0x160/0x160
[ 5559.404780]  ? reacquire_held_locks+0x4e0/0x4e0
[ 5559.405026]  ? do_raw_spin_lock+0x11e/0x250
[ 5559.405239]  schedule+0x133/0x230
[ 5559.405393]  io_schedule+0xc3/0x140
[ 5559.405567]  folio_wait_bit_common+0x2e4/0x610
[ 5559.405808]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[ 5559.406126]  ? folio_unlock+0x70/0x70
[ 5559.406299]  ? lock_is_held_type+0xd8/0x130
[ 5559.406512]  folio_wait_writeback+0x40/0x230
[ 5559.406729]  __filemap_fdatawait_range+0x10a/0x2a0
[ 5559.406979]  ? filemap_fdatawrite+0x170/0x170
[ 5559.407199]  ? lock_is_held_type+0xd8/0x130
[ 5559.407424]  file_write_and_wait_range+0xbc/0xf0
[ 5559.407660]  xfs_file_fsync+0x146/0x830
[ 5559.407847]  ? xfs_dio_write_end_io+0x760/0x760
[ 5559.408377]  ? lock_is_held_type+0xd8/0x130
[ 5559.408957]  nfsd_commit+0x304/0x560 [nfsd]
[ 5559.409826]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[ 5559.410890]  ? percpu_counter_add_batch+0x77/0x130
[ 5559.411417]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[ 5559.412568]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[ 5559.413689]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[ 5559.414040]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[ 5559.414336]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[ 5559.414556]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[ 5559.414856]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[ 5559.415138]  svc_process+0x32d/0x4a0 [sunrpc]
[ 5559.415395]  nfsd+0x2d6/0x570 [nfsd]
[ 5559.415588]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[ 5559.415881]  kthread+0x298/0x340
[ 5559.416085]  ? kthread_complete_and_exit+0x20/0x20
[ 5559.416645]  ret_from_fork+0x22/0x30
[ 5559.417928]  </TASK>
[ 5559.418011] INFO: task nfsd:23106 blocked for more than 241 seconds.
[ 5559.418355]       Not tainted 6.0.0-rc3-00008-gbb49f295d682 #3631
[ 5559.418684] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5559.419126] task:nfsd            state:D stack:24448 pid:23106 ppid:     2 flags:0x00004000
[ 5559.419606] Call Trace:
[ 5559.419698]  <TASK>
[ 5559.419774]  __schedule+0xb73/0x2260
[ 5559.419991]  ? io_schedule_timeout+0x160/0x160
[ 5559.420682]  ? sbitmap_get_shallow+0x189/0x5f0
[ 5559.421926]  schedule+0x133/0x230
[ 5559.422088]  io_schedule+0xc3/0x140
[ 5559.422251]  blk_mq_get_tag+0x38c/0xa80
[ 5559.422443]  ? blk_mq_get_tags+0x240/0x240
[ 5559.422643]  ? lock_is_held_type+0xd8/0x130
[ 5559.422854]  ? sched_debug_start+0x180/0x180
[ 5559.423075]  ? __bio_split_to_limits+0x712/0x1080
[ 5559.423322]  __blk_mq_alloc_requests+0x492/0xc50
[ 5559.423568]  ? reacquire_held_locks+0x4e0/0x4e0
[ 5559.423799]  ? blk_mq_timeout_work+0x380/0x380
[ 5559.424081]  blk_mq_submit_bio+0x73c/0x17d0
[ 5559.424704]  ? blk_mq_try_issue_list_directly+0x9a0/0x9a0
[ 5559.425763]  submit_bio_noacct_nocheck+0x5b1/0x820
[ 5559.426942]  ? should_fail_request+0x70/0x70
[ 5559.427220]  ? submit_bio_noacct+0xa50/0x1140
[ 5559.427444]  ? iomap_page_create+0x8b/0x3d0
[ 5559.427655]  iomap_do_writepage+0x1210/0x2450
[ 5559.427889]  ? iomap_write_end+0x9c0/0x9c0
[ 5559.428190]  write_cache_pages+0x3d8/0xed0
[ 5559.428396]  ? iomap_write_end+0x9c0/0x9c0
[ 5559.428603]  ? folio_wait_stable+0xc0/0xc0
[ 5559.428803]  ? lock_is_held_type+0xd8/0x130
[ 5559.429026]  ? do_raw_spin_lock+0x11e/0x250
[ 5559.429239]  iomap_writepages+0x42/0xb0
[ 5559.429425]  xfs_vm_writepages+0x109/0x160
[ 5559.429645]  ? xfs_vm_read_folio+0x20/0x20
[ 5559.430036]  ? nfsd_setuser+0x5fb/0x9c0 [nfsd]
[ 5559.430609]  do_writepages+0x15c/0x590
[ 5559.431891]  ? lock_is_held_type+0xd8/0x130
[ 5559.433503]  ? page_writeback_cpu_online+0x20/0x20
[ 5559.434063]  ? lock_release+0x380/0x760
[ 5559.434264]  ? __lock_acquire+0xc67/0x69f0
[ 5559.434479]  ? lock_is_held_type+0xd8/0x130
[ 5559.434696]  __filemap_fdatawrite_range+0x13e/0x170
[ 5559.434966]  ? rcu_read_lock_sched_held+0x3f/0x70
[ 5559.435207]  ? delete_from_page_cache_batch+0xa30/0xa30
[ 5559.435486]  ? lock_is_held_type+0xd8/0x130
[ 5559.435694]  file_write_and_wait_range+0x8e/0xf0
[ 5559.435963]  xfs_file_fsync+0x146/0x830
[ 5559.436917]  ? xfs_dio_write_end_io+0x760/0x760
[ 5559.438010]  ? lock_is_held_type+0xd8/0x130
[ 5559.438482]  nfsd_commit+0x304/0x560 [nfsd]
[ 5559.438731]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[ 5559.438989]  ? percpu_counter_add_batch+0x77/0x130
[ 5559.439239]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[ 5559.439532]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[ 5559.439778]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[ 5559.440184]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[ 5559.440491]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[ 5559.440711]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[ 5559.441160]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[ 5559.441428]  svc_process+0x32d/0x4a0 [sunrpc]
[ 5559.441729]  nfsd+0x2d6/0x570 [nfsd]
[ 5559.441952]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[ 5559.442247]  kthread+0x298/0x340
[ 5559.442392]  ? kthread_complete_and_exit+0x20/0x20
[ 5559.442641]  ret_from_fork+0x22/0x30
[ 5559.442823]  </TASK>
[ 5559.442927] 
               Showing all locks held in the system:
[ 5559.443180] 1 lock held by rcu_tasks_kthre/10:
[ 5559.443402]  #0: ffffffff850cd970 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x2e/0xd70
[ 5559.444017] 1 lock held by rcu_tasks_rude_/11:
[ 5559.444332]  #0: ffffffff850cd6d0 (rcu_tasks_rude.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x2e/0xd70
[ 5559.444965] 1 lock held by rcu_tasks_trace/12:
[ 5559.445186]  #0: ffffffff850cd3d0 (rcu_tasks_trace.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x2e/0xd70
[ 5559.446062] 1 lock held by khungtaskd/38:
[ 5559.446298]  #0: ffffffff850ce440 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x53/0x260
[ 5559.446875] 2 locks held by kswapd0/74:
[ 5559.447109] 3 locks held by kworker/u8:1/24268:
[ 5559.447347]  #0: ffff88800869b138 ((wq_completion)writeback){+.+.}-{0:0}, at: process_one_work+0x70a/0x12e0
[ 5559.447940]  #1: ffff88801db27dd0 ((work_completion)(&(&wb->dwork)->work)){+.+.}-{0:0}, at: process_one_work+0x738/0x12e0
[ 5559.448672]  #2: ffff8880125f80e0 (&type->s_umount_key#48){++++}-{3:3}, at: trylock_super+0x16/0xc0
[ 5559.449242] 2 locks held by kworker/2:0/24284:
[ 5559.449463]  #0: ffff88800d5b9938 ((wq_completion)xfs-sync/vdf){+.+.}-{0:0}, at: process_one_work+0x70a/0x12e0
[ 5559.450078]  #1: ffff888047297dd0 ((work_completion)(&(&log->l_work)->work)){+.+.}-{0:0}, at: process_one_work+0x738/0x12e0

[ 5559.450768] =============================================

[ 8795.664246] clocksource: timekeeping watchdog on CPU2: acpi_pm wd-wd read-back delay of 70679ns
[ 8795.666604] clocksource: wd-tsc-wd read-back delay of 232431ns, clock-skew test skipped!
[ 8797.644284] clocksource: timekeeping watchdog on CPU2: acpi_pm wd-wd read-back delay of 62857ns
[ 8797.646577] clocksource: wd-tsc-wd read-back delay of 241930ns, clock-skew test skipped!
[21025.695599] clocksource: timekeeping watchdog on CPU2: acpi_pm wd-wd read-back delay of 240533ns
[21025.698830] clocksource: wd-tsc-wd read-back delay of 170692ns, clock-skew test skipped!
[26373.720180] clocksource: timekeeping watchdog on CPU2: acpi_pm wd-wd read-back delay of 50844ns
[26373.722764] clocksource: wd-tsc-wd read-back delay of 119568ns, clock-skew test skipped!
[35820.390214] sysrq: Show Blocked State
[35820.390588] task:nfsd            state:D stack:24336 pid:23105 ppid:     2 flags:0x00004000
[35820.391159] Call Trace:
[35820.391268]  <TASK>
[35820.391352]  __schedule+0xb73/0x2260
[35820.391552]  ? io_schedule_timeout+0x160/0x160
[35820.391801]  ? reacquire_held_locks+0x4e0/0x4e0
[35820.392032]  ? do_raw_spin_lock+0x11e/0x250
[35820.392247]  schedule+0x133/0x230
[35820.392401]  io_schedule+0xc3/0x140
[35820.392563]  folio_wait_bit_common+0x2e4/0x610
[35820.392796]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[35820.393154]  ? folio_unlock+0x70/0x70
[35820.394717]  ? lock_is_held_type+0xd8/0x130
[35820.394940]  folio_wait_writeback+0x40/0x230
[35820.395159]  __filemap_fdatawait_range+0x10a/0x2a0
[35820.395413]  ? filemap_fdatawrite+0x170/0x170
[35820.395634]  ? lock_is_held_type+0xd8/0x130
[35820.395861]  file_write_and_wait_range+0xbc/0xf0
[35820.396099]  xfs_file_fsync+0x146/0x830
[35820.396290]  ? xfs_dio_write_end_io+0x760/0x760
[35820.396521]  ? lock_is_held_type+0xd8/0x130
[35820.396734]  nfsd_commit+0x304/0x560 [nfsd]
[35820.397027]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[35820.397318]  ? percpu_counter_add_batch+0x77/0x130
[35820.397580]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[35820.397876]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[35820.398122]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[35820.398457]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[35820.398752]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[35820.398971]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[35820.399276]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[35820.399535]  svc_process+0x32d/0x4a0 [sunrpc]
[35820.399792]  nfsd+0x2d6/0x570 [nfsd]
[35820.399985]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[35820.400282]  kthread+0x298/0x340
[35820.400429]  ? kthread_complete_and_exit+0x20/0x20
[35820.400677]  ret_from_fork+0x22/0x30
[35820.400901]  </TASK>
[35820.400980] task:nfsd            state:D stack:24448 pid:23106 ppid:     2 flags:0x00004000
[35820.401543] Call Trace:
[35820.401637]  <TASK>
[35820.401712]  __schedule+0xb73/0x2260
[35820.401889]  ? io_schedule_timeout+0x160/0x160
[35820.402112]  ? sbitmap_get_shallow+0x189/0x5f0
[35820.402581]  schedule+0x133/0x230
[35820.402738]  io_schedule+0xc3/0x140
[35820.402901]  blk_mq_get_tag+0x38c/0xa80
[35820.403093]  ? blk_mq_get_tags+0x240/0x240
[35820.403319]  ? lock_is_held_type+0xd8/0x130
[35820.403529]  ? sched_debug_start+0x180/0x180
[35820.403744]  ? __bio_split_to_limits+0x712/0x1080
[35820.403989]  __blk_mq_alloc_requests+0x492/0xc50
[35820.404249]  ? reacquire_held_locks+0x4e0/0x4e0
[35820.404489]  ? blk_mq_timeout_work+0x380/0x380
[35820.404721]  blk_mq_submit_bio+0x73c/0x17d0
[35820.404981]  ? blk_mq_try_issue_list_directly+0x9a0/0x9a0
[35820.405409]  submit_bio_noacct_nocheck+0x5b1/0x820
[35820.405663]  ? should_fail_request+0x70/0x70
[35820.405876]  ? submit_bio_noacct+0xa50/0x1140
[35820.406100]  ? iomap_page_create+0x8b/0x3d0
[35820.406357]  iomap_do_writepage+0x1210/0x2450
[35820.406592]  ? iomap_write_end+0x9c0/0x9c0
[35820.406799]  write_cache_pages+0x3d8/0xed0
[35820.407002]  ? iomap_write_end+0x9c0/0x9c0
[35820.407209]  ? folio_wait_stable+0xc0/0xc0
[35820.407412]  ? lock_is_held_type+0xd8/0x130
[35820.407632]  ? do_raw_spin_lock+0x11e/0x250
[35820.407845]  iomap_writepages+0x42/0xb0
[35820.408031]  xfs_vm_writepages+0x109/0x160
[35820.408233]  ? xfs_vm_read_folio+0x20/0x20
[35820.408444]  ? nfsd_setuser+0x5fb/0x9c0 [nfsd]
[35820.408695]  do_writepages+0x15c/0x590
[35820.408924]  ? lock_is_held_type+0xd8/0x130
[35820.409148]  ? page_writeback_cpu_online+0x20/0x20
[35820.409451]  ? lock_release+0x380/0x760
[35820.409640]  ? __lock_acquire+0xc67/0x69f0
[35820.409849]  ? lock_is_held_type+0xd8/0x130
[35820.410060]  __filemap_fdatawrite_range+0x13e/0x170
[35820.410377]  ? rcu_read_lock_sched_held+0x3f/0x70
[35820.410617]  ? delete_from_page_cache_batch+0xa30/0xa30
[35820.410896]  ? lock_is_held_type+0xd8/0x130
[35820.411104]  file_write_and_wait_range+0x8e/0xf0
[35820.411345]  xfs_file_fsync+0x146/0x830
[35820.411531]  ? xfs_dio_write_end_io+0x760/0x760
[35820.411760]  ? lock_is_held_type+0xd8/0x130
[35820.411972]  nfsd_commit+0x304/0x560 [nfsd]
[35820.412206]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[35820.412447]  ? percpu_counter_add_batch+0x77/0x130
[35820.412696]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[35820.413035]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[35820.413349]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[35820.413653]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[35820.413947]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[35820.414166]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[35820.414489]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[35820.414747]  svc_process+0x32d/0x4a0 [sunrpc]
[35820.415003]  nfsd+0x2d6/0x570 [nfsd]
[35820.415197]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[35820.415494]  kthread+0x298/0x340
[35820.415639]  ? kthread_complete_and_exit+0x20/0x20
[35820.415887]  ret_from_fork+0x22/0x30
[35820.416070]  </TASK>
[35820.416146] task:nfsd            state:D stack:24608 pid:23107 ppid:     2 flags:0x00004000
[35820.416628] Call Trace:
[35820.416720]  <TASK>
[35820.416795]  __schedule+0xb73/0x2260
[35820.417017]  ? io_schedule_timeout+0x160/0x160
[35820.417303]  ? reacquire_held_locks+0x4e0/0x4e0
[35820.417567]  ? do_raw_spin_lock+0x11e/0x250
[35820.417778]  schedule+0x133/0x230
[35820.417931]  io_schedule+0xc3/0x140
[35820.418093]  folio_wait_bit_common+0x2e4/0x610
[35820.418347]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[35820.418661]  ? folio_unlock+0x70/0x70
[35820.418834]  ? lock_is_held_type+0xd8/0x130
[35820.419047]  folio_wait_writeback+0x40/0x230
[35820.419267]  __filemap_fdatawait_range+0x10a/0x2a0
[35820.419515]  ? filemap_fdatawrite+0x170/0x170
[35820.419735]  ? lock_is_held_type+0xd8/0x130
[35820.419961]  file_write_and_wait_range+0xbc/0xf0
[35820.420198]  xfs_file_fsync+0x146/0x830
[35820.420387]  ? xfs_dio_write_end_io+0x760/0x760
[35820.420616]  ? lock_is_held_type+0xd8/0x130
[35820.420829]  nfsd_commit+0x304/0x560 [nfsd]
[35820.421123]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[35820.421419]  ? percpu_counter_add_batch+0x77/0x130
[35820.421671]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[35820.421966]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[35820.422212]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[35820.422538]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[35820.422832]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[35820.423051]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[35820.423353]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[35820.423611]  svc_process+0x32d/0x4a0 [sunrpc]
[35820.423891]  nfsd+0x2d6/0x570 [nfsd]
[35820.424084]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[35820.424400]  kthread+0x298/0x340
[35820.424545]  ? kthread_complete_and_exit+0x20/0x20
[35820.424793]  ret_from_fork+0x22/0x30
[35820.425012]  </TASK>
[35820.425089] task:nfsd            state:D stack:24432 pid:23108 ppid:     2 flags:0x00004000
[35820.425658] Call Trace:
[35820.425753]  <TASK>
[35820.425829]  __schedule+0xb73/0x2260
[35820.426006]  ? io_schedule_timeout+0x160/0x160
[35820.426250]  ? reacquire_held_locks+0x4e0/0x4e0
[35820.426489]  ? do_raw_spin_lock+0x11e/0x250
[35820.426700]  schedule+0x133/0x230
[35820.426853]  io_schedule+0xc3/0x140
[35820.427015]  folio_wait_bit_common+0x2e4/0x610
[35820.427310]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[35820.427705]  ? folio_unlock+0x70/0x70
[35820.427879]  ? lock_is_held_type+0xd8/0x130
[35820.428092]  folio_wait_writeback+0x40/0x230
[35820.428335]  __filemap_fdatawait_range+0x10a/0x2a0
[35820.428584]  ? filemap_fdatawrite+0x170/0x170
[35820.428803]  ? lock_is_held_type+0xd8/0x130
[35820.429075]  file_write_and_wait_range+0xbc/0xf0
[35820.429393]  xfs_file_fsync+0x146/0x830
[35820.429582]  ? xfs_dio_write_end_io+0x760/0x760
[35820.429811]  ? lock_is_held_type+0xd8/0x130
[35820.430023]  nfsd_commit+0x304/0x560 [nfsd]
[35820.430300]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[35820.430538]  ? percpu_counter_add_batch+0x77/0x130
[35820.430787]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[35820.431080]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[35820.431345]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[35820.431647]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[35820.431941]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[35820.432160]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[35820.432463]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[35820.432721]  svc_process+0x32d/0x4a0 [sunrpc]
[35820.433025]  nfsd+0x2d6/0x570 [nfsd]
[35820.433254]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[35820.433582]  kthread+0x298/0x340
[35820.433727]  ? kthread_complete_and_exit+0x20/0x20
[35820.433975]  ret_from_fork+0x22/0x30
[35820.434158]  </TASK>
[35820.434248] task:nfsd            state:D stack:24000 pid:23109 ppid:     2 flags:0x00004000
[35820.434735] Call Trace:
[35820.434826]  <TASK>
[35820.434901]  __schedule+0xb73/0x2260
[35820.435077]  ? io_schedule_timeout+0x160/0x160
[35820.435343]  ? reacquire_held_locks+0x4e0/0x4e0
[35820.435573]  ? do_raw_spin_lock+0x11e/0x250
[35820.435784]  schedule+0x133/0x230
[35820.435937]  io_schedule+0xc3/0x140
[35820.436099]  folio_wait_bit_common+0x2e4/0x610
[35820.436334]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[35820.436649]  ? folio_unlock+0x70/0x70
[35820.436821]  ? lock_is_held_type+0xd8/0x130
[35820.437074]  folio_wait_writeback+0x40/0x230
[35820.437373]  __filemap_fdatawait_range+0x10a/0x2a0
[35820.437630]  ? filemap_fdatawrite+0x170/0x170
[35820.437856]  ? lock_is_held_type+0xd8/0x130
[35820.438088]  file_write_and_wait_range+0xbc/0xf0
[35820.438341]  xfs_file_fsync+0x146/0x830
[35820.438527]  ? xfs_dio_write_end_io+0x760/0x760
[35820.438757]  ? lock_is_held_type+0xd8/0x130
[35820.438968]  nfsd_commit+0x304/0x560 [nfsd]
[35820.439202]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[35820.439443]  ? percpu_counter_add_batch+0x77/0x130
[35820.439692]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[35820.439985]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[35820.440267]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[35820.440569]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[35820.440899]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[35820.441140]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[35820.441497]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[35820.441757]  svc_process+0x32d/0x4a0 [sunrpc]
[35820.442015]  nfsd+0x2d6/0x570 [nfsd]
[35820.442209]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[35820.442527]  kthread+0x298/0x340
[35820.442672]  ? kthread_complete_and_exit+0x20/0x20
[35820.443203]  ret_from_fork+0x22/0x30
[35820.443430]  </TASK>
[35820.443510] task:nfsd            state:D stack:23952 pid:23110 ppid:     2 flags:0x00004000
[35820.444272] Call Trace:
[35820.444376]  <TASK>
[35820.444455]  __schedule+0xb73/0x2260
[35820.444641]  ? io_schedule_timeout+0x160/0x160
[35820.444895]  ? reacquire_held_locks+0x4e0/0x4e0
[35820.445395]  ? do_raw_spin_lock+0x11e/0x250
[35820.445620]  schedule+0x133/0x230
[35820.445782]  io_schedule+0xc3/0x140
[35820.445952]  folio_wait_bit_common+0x2e4/0x610
[35820.446421]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[35820.446767]  ? folio_unlock+0x70/0x70
[35820.446965]  ? lock_is_held_type+0xd8/0x130
[35820.447190]  folio_wait_writeback+0x40/0x230
[35820.447426]  __filemap_fdatawait_range+0x10a/0x2a0
[35820.447688]  ? filemap_fdatawrite+0x170/0x170
[35820.448152]  ? lock_is_held_type+0xd8/0x130
[35820.448416]  file_write_and_wait_range+0xbc/0xf0
[35820.448660]  xfs_file_fsync+0x146/0x830
[35820.448879]  ? xfs_dio_write_end_io+0x760/0x760
[35820.449142]  ? lock_is_held_type+0xd8/0x130
[35820.449422]  nfsd_commit+0x304/0x560 [nfsd]
[35820.450015]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[35820.450416]  ? percpu_counter_add_batch+0x77/0x130
[35820.450893]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[35820.451261]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[35820.451518]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[35820.451826]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[35820.452121]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[35820.452367]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[35820.452667]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[35820.452963]  svc_process+0x32d/0x4a0 [sunrpc]
[35820.453248]  nfsd+0x2d6/0x570 [nfsd]
[35820.453504]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[35820.453798]  kthread+0x298/0x340
[35820.453943]  ? kthread_complete_and_exit+0x20/0x20
[35820.454191]  ret_from_fork+0x22/0x30
[35820.454398]  </TASK>
[35820.454475] task:nfsd            state:D stack:23808 pid:23111 ppid:     2 flags:0x00004000
[35820.454952] Call Trace:
[35820.455043]  <TASK>
[35820.455118]  __schedule+0xb73/0x2260
[35820.455297]  ? io_schedule_timeout+0x160/0x160
[35820.455520]  ? reacquire_held_locks+0x4e0/0x4e0
[35820.455749]  ? do_raw_spin_lock+0x11e/0x250
[35820.455959]  schedule+0x133/0x230
[35820.456112]  io_schedule+0xc3/0x140
[35820.456277]  folio_wait_bit_common+0x2e4/0x610
[35820.456508]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[35820.456823]  ? folio_unlock+0x70/0x70
[35820.457036]  ? lock_is_held_type+0xd8/0x130
[35820.457338]  folio_wait_writeback+0x40/0x230
[35820.457565]  __filemap_fdatawait_range+0x10a/0x2a0
[35820.457814]  ? filemap_fdatawrite+0x170/0x170
[35820.458033]  ? lock_is_held_type+0xd8/0x130
[35820.458284]  file_write_and_wait_range+0xbc/0xf0
[35820.458522]  xfs_file_fsync+0x146/0x830
[35820.458708]  ? xfs_dio_write_end_io+0x760/0x760
[35820.458938]  ? lock_is_held_type+0xd8/0x130
[35820.459150]  nfsd_commit+0x304/0x560 [nfsd]
[35820.459387]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[35820.459624]  ? percpu_counter_add_batch+0x77/0x130
[35820.459872]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[35820.460166]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[35820.460415]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[35820.460718]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[35820.461054]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[35820.461391]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[35820.461696]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[35820.461955]  svc_process+0x32d/0x4a0 [sunrpc]
[35820.462214]  nfsd+0x2d6/0x570 [nfsd]
[35820.462434]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[35820.462727]  kthread+0x298/0x340
[35820.462872]  ? kthread_complete_and_exit+0x20/0x20
[35820.463120]  ret_from_fork+0x22/0x30
[35820.463306]  </TASK>
[35820.463382] task:nfsd            state:D stack:21968 pid:23112 ppid:     2 flags:0x00004000
[35820.463860] Call Trace:
[35820.463951]  <TASK>
[35820.464026]  __schedule+0xb73/0x2260
[35820.464202]  ? io_schedule_timeout+0x160/0x160
[35820.464429]  ? reacquire_held_locks+0x4e0/0x4e0
[35820.464658]  ? do_raw_spin_lock+0x11e/0x250
[35820.464909]  schedule+0x133/0x230
[35820.465064]  io_schedule+0xc3/0x140
[35820.465370]  folio_wait_bit_common+0x2e4/0x610
[35820.465604]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[35820.465919]  ? folio_unlock+0x70/0x70
[35820.466093]  ? lock_is_held_type+0xd8/0x130
[35820.466330]  folio_wait_writeback+0x40/0x230
[35820.466548]  __filemap_fdatawait_range+0x10a/0x2a0
[35820.466796]  ? filemap_fdatawrite+0x170/0x170
[35820.467015]  ? lock_is_held_type+0xd8/0x130
[35820.467246]  file_write_and_wait_range+0xbc/0xf0
[35820.467483]  xfs_file_fsync+0x146/0x830
[35820.467669]  ? xfs_dio_write_end_io+0x760/0x760
[35820.467899]  ? lock_is_held_type+0xd8/0x130
[35820.468110]  nfsd_commit+0x304/0x560 [nfsd]
[35820.468347]  ? nfsd_write+0x4f0/0x4f0 [nfsd]
[35820.468584]  ? percpu_counter_add_batch+0x77/0x130
[35820.468854]  nfsd4_proc_compound+0xd6f/0x26d0 [nfsd]
[35820.469196]  nfsd_dispatch+0x4b9/0xba0 [nfsd]
[35820.469540]  svc_process_common+0xc14/0x1aa0 [sunrpc]
[35820.469845]  ? svc_create_pooled+0x5c0/0x5c0 [sunrpc]
[35820.470140]  ? nfsd_svc+0xc70/0xc70 [nfsd]
[35820.470404]  ? svc_sock_secure_port+0x2a/0x50 [sunrpc]
[35820.470703]  ? svc_recv+0xc6b/0x28c0 [sunrpc]
[35820.470962]  svc_process+0x32d/0x4a0 [sunrpc]
[35820.471223]  nfsd+0x2d6/0x570 [nfsd]
[35820.471417]  ? nfsd_shutdown_threads+0x290/0x290 [nfsd]
[35820.471710]  kthread+0x298/0x340
[35820.471856]  ? kthread_complete_and_exit+0x20/0x20
[35820.472104]  ret_from_fork+0x22/0x30
[35820.472291]  </TASK>
[35820.472368] task:kworker/u8:1    state:D stack:25008 pid:24268 ppid:     2 flags:0x00004000
[35820.472877] Workqueue: writeback wb_workfn (flush-253:80)
[35820.473197] Call Trace:
[35820.473357]  <TASK>
[35820.473435]  __schedule+0xb73/0x2260
[35820.473612]  ? io_schedule_timeout+0x160/0x160
[35820.473838]  ? blk_start_plug_nr_ios+0x280/0x280
[35820.474078]  schedule+0x133/0x230
[35820.474364]  io_schedule+0xc3/0x140
[35820.474551]  folio_wait_bit_common+0x2e4/0x610
[35820.474790]  ? perf_trace_mm_filemap_op_page_cache+0x780/0x780
[35820.475114]  ? folio_unlock+0x70/0x70
[35820.475305]  write_cache_pages+0x71c/0xed0
[35820.475515]  ? iomap_write_end+0x9c0/0x9c0
[35820.475736]  ? folio_wait_stable+0xc0/0xc0
[35820.475938]  ? lock_is_held_type+0xd8/0x130
[35820.476157]  ? do_raw_spin_lock+0x11e/0x250
[35820.476393]  iomap_writepages+0x42/0xb0
[35820.476579]  xfs_vm_writepages+0x109/0x160
[35820.476779]  ? xfs_vm_read_folio+0x20/0x20
[35820.477040]  do_writepages+0x15c/0x590
[35820.477329]  ? page_writeback_cpu_online+0x20/0x20
[35820.477583]  ? reacquire_held_locks+0x201/0x4e0
[35820.477817]  ? writeback_sb_inodes+0x2dc/0xf90
[35820.478049]  ? lock_is_held_type+0xd8/0x130
[35820.478288]  ? find_held_lock+0x2d/0x110
[35820.478487]  ? lock_is_held_type+0xd8/0x130
[35820.478713]  __writeback_single_inode+0xd9/0xbb0
[35820.478954]  writeback_sb_inodes+0x4b1/0xf90
[35820.479183]  ? sync_inode_metadata+0x90/0x90
[35820.479418]  ? down_read_trylock+0x1b0/0x360
[35820.479629]  ? trylock_super+0x16/0xc0
[35820.479820]  __writeback_inodes_wb+0xb4/0x200
[35820.480046]  wb_writeback+0x4e9/0x870
[35820.480240]  ? __writeback_inodes_wb+0x200/0x200
[35820.480490]  ? get_nr_dirty_inodes+0x18/0x1b0
[35820.480708]  ? lockdep_hardirqs_on_prepare+0x17b/0x410
[35820.481014]  ? _raw_spin_unlock_irq+0x24/0x50
[35820.481378]  wb_workfn+0x782/0xc60
[35820.481552]  ? inode_wait_for_writeback+0x30/0x30
[35820.481798]  ? lock_acquire+0x1b1/0x520
[35820.481986]  ? lock_acquire+0x1c1/0x520
[35820.482176]  ? lock_downgrade+0x6a0/0x6a0
[35820.482415]  ? lock_is_held_type+0xd8/0x130
[35820.482629]  process_one_work+0x7f2/0x12e0
[35820.482836]  ? lock_downgrade+0x6a0/0x6a0
[35820.483030]  ? pwq_dec_nr_in_flight+0x230/0x230
[35820.483289]  ? rwlock_bug.part.0+0x90/0x90
[35820.483495]  worker_thread+0xfc/0x1270
[35820.483683]  ? __kthread_parkme+0xc1/0x1f0
[35820.483886]  ? process_one_work+0x12e0/0x12e0
[35820.484104]  kthread+0x298/0x340
[35820.484252]  ? kthread_complete_and_exit+0x20/0x20
[35820.484501]  ret_from_fork+0x22/0x30
[35820.484684]  </TASK>
[35820.484760] task:kworker/2:0     state:D stack:26528 pid:24284 ppid:     2 flags:0x00004000
[35820.485363] Workqueue: xfs-sync/vdf xfs_log_worker
[35820.485611] Call Trace:
[35820.485703]  <TASK>
[35820.485778]  __schedule+0xb73/0x2260
[35820.485954]  ? io_schedule_timeout+0x160/0x160
[35820.486180]  ? _raw_spin_unlock_irq+0x24/0x50
[35820.486425]  schedule+0x133/0x230
[35820.486579]  io_schedule+0xc3/0x140
[35820.486741]  blk_mq_get_tag+0x38c/0xa80
[35820.486932]  ? blk_mq_get_tags+0x240/0x240
[35820.487134]  ? sched_debug_start+0x180/0x180
[35820.487359]  __blk_mq_alloc_requests+0x492/0xc50
[35820.487605]  ? reacquire_held_locks+0x4e0/0x4e0
[35820.487832]  ? lockdep_hardirqs_on_prepare+0x410/0x410
[35820.488098]  ? blk_mq_timeout_work+0x380/0x380
[35820.488332]  blk_mq_submit_bio+0x73c/0x17d0
[35820.488546]  ? blk_mq_try_issue_list_directly+0x9a0/0x9a0
[35820.488829]  ? find_held_lock+0x2d/0x110
[35820.489071]  submit_bio_noacct_nocheck+0x5b1/0x820
[35820.489411]  ? should_fail_request+0x70/0x70
[35820.489625]  ? submit_bio_noacct+0xa50/0x1140
[35820.489854]  xlog_state_release_iclog+0x39e/0x770
[35820.490092]  ? xlog_state_switch_iclogs+0x43e/0x630
[35820.490371]  xfs_log_force+0x522/0xb30
[35820.490554]  xfs_log_worker+0x86/0x2b0
[35820.490735]  process_one_work+0x7f2/0x12e0
[35820.490941]  ? lock_downgrade+0x6a0/0x6a0
[35820.491145]  ? pwq_dec_nr_in_flight+0x230/0x230
[35820.491429]  ? rwlock_bug.part.0+0x90/0x90
[35820.491636]  worker_thread+0xfc/0x1270
[35820.491824]  ? __kthread_parkme+0xc1/0x1f0
[35820.492027]  ? process_one_work+0x12e0/0x12e0
[35820.492249]  kthread+0x298/0x340
[35820.492394]  ? kthread_complete_and_exit+0x20/0x20
[35820.492642]  ret_from_fork+0x22/0x30
[35820.492824]  </TASK>


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2022-09-03 15:55 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <20220724125542.89A6FA0482@home.fieldses.org>
     [not found] ` <20220724125848.GA14741@fieldses.org>
     [not found]   ` <FFC9022A-B2AB-4455-870B-6801B327C5B7@oracle.com>
     [not found]     ` <20220724163401.GA17655@fieldses.org>
     [not found]       ` <20220724163702.GB17655@fieldses.org>
     [not found]         ` <0CFBAAC2-1ECF-48E1-A687-75176C2F1A17@oracle.com>
     [not found]           ` <20220725021029.GA789@fieldses.org>
     [not found]             ` <20220725121600.GA17056@fieldses.org>
     [not found]               ` <608CA2C9-464C-464E-B4AD-CF1B8C494A5A@oracle.com>
2022-07-25 13:22                 ` all cel/for-rc (1197eb59 lockd: fix nlm_close_files) results Bruce Fields
2022-07-25 22:03                   ` Dave Chinner
2022-07-25 22:06                     ` Dave Chinner
2022-09-03 15:55                       ` J. Bruce Fields

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox