Filesystem lockup during backup

["Torbjörn Jansson" <torbjorn@jansson.tech>]

Hello.

i have a btrfs filesystem on two 18tb disks that i use as backup destination for my proxmox cluster.
the filesystem is using btrfs raid1 mirroring and is exported over nfs to the other nodes.

because this is used primarily for backups there are periods of heavy writes (several backups running at the same time) and when this happens it is very likely the filesystem and nfsd locks up completely.
this then starts a chain reaction due to the default hard mount blocking processes then eventually ceph also becomes unhappy and then the vms goes down.

below is the hung task output from dmesg on the computer with the disks.

any idea whats going on and what i can do about it?



[1560204.654347] INFO: task nfsd:5136 blocked for more than 122 seconds.
[1560204.654351]       Tainted: P           O       6.14.11-2-pve #1
[1560204.654353] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1560204.654355] task:nfsd            state:D stack:0     pid:5136  tgid:5136  ppid:2      task_flags:0x200040 flags:0x00004000
[1560204.654361] Call Trace:
[1560204.654363]  <TASK>
[1560204.654366]  __schedule+0x466/0x1400
[1560204.654376]  schedule+0x29/0x130
[1560204.654381]  io_schedule+0x4c/0x80
[1560204.654387]  folio_wait_bit_common+0x122/0x2e0
[1560204.654393]  ? __pfx_wake_page_function+0x10/0x10
[1560204.654400]  __folio_lock+0x17/0x30
[1560204.654404]  extent_write_cache_pages+0x36e/0x7f0 [btrfs]
[1560204.654559]  btrfs_writepages+0x75/0x130 [btrfs]
[1560204.654703]  do_writepages+0xde/0x280
[1560204.654710]  ? __pfx_ip_finish_output+0x10/0x10
[1560204.654715]  ? wbc_attach_and_unlock_inode+0xd1/0x130
[1560204.654721]  filemap_fdatawrite_wbc+0x58/0x80
[1560204.654726]  ? __ip_queue_xmit+0x19b/0x4e0
[1560204.654731]  __filemap_fdatawrite_range+0x6d/0xa0
[1560204.654744]  filemap_fdatawrite_range+0x13/0x30
[1560204.654748]  btrfs_fdatawrite_range+0x28/0x70 [btrfs]
[1560204.654889]  start_ordered_ops.constprop.0+0x4e/0x90 [btrfs]
[1560204.655029]  btrfs_sync_file+0xa9/0x610 [btrfs]
[1560204.655159]  ? list_lru_del_obj+0xad/0xe0
[1560204.655168]  vfs_fsync_range+0x42/0xa0
[1560204.655174]  nfsd_commit+0x9f/0x180 [nfsd]
[1560204.655275]  nfsd4_commit+0x60/0xa0 [nfsd]
[1560204.655367]  nfsd4_proc_compound+0x3ad/0x760 [nfsd]
[1560204.655427]  nfsd_dispatch+0xce/0x220 [nfsd]
[1560204.655486]  svc_process_common+0x464/0x6f0 [sunrpc]
[1560204.655553]  ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[1560204.655611]  svc_process+0x136/0x1f0 [sunrpc]
[1560204.655675]  svc_recv+0x7bb/0x9a0 [sunrpc]
[1560204.655741]  ? __pfx_nfsd+0x10/0x10 [nfsd]
[1560204.655798]  nfsd+0x90/0xf0 [nfsd]
[1560204.655852]  kthread+0xf9/0x230
[1560204.655855]  ? __pfx_kthread+0x10/0x10
[1560204.655858]  ret_from_fork+0x44/0x70
[1560204.655862]  ? __pfx_kthread+0x10/0x10
[1560204.655864]  ret_from_fork_asm+0x1a/0x30
[1560204.655871]  </TASK>