Re: [RFC PATCH v4 0/9] NFSv4/flexfiles: Add support for striped layouts

[Mike Snitzer <snitzer@kernel.org>]

On Wed, Sep 24, 2025 at 04:20:41PM +0000, Jonathan Curley wrote:
> This patch series introduces support for striped layouts:
> 
> The first 2 patches are simple preparation changes. There should be
> no logical impact to the code.
> 
> The 3rd patch refactors the nfs4_ff_layout_mirror struct to have an
> array of a new nfs4_ff_layout_ds_stripe type. The
> nfs4_ff_layout_ds_stripe has all the contents of ff_data_server4 per
> the flexfile rfc. I called it ds_stripe because ds was already taken
> by the deviceid side of the code.
> 
> The patches 4-8 update various paths to be dss_id aware. Most of this
> consists of either adding a new parameter to the function or adding a
> loop. Depending on which is appropriate.
> 
> The final patch 9 updates the layout creation path to populate the
> array and turns the feature on.
> 
> v1:
>  - Fixes function parameter 'dss_id' not described in
>    'nfs4_ff_layout_prepare_ds'
> 
> v2:
>  - Fixes layout stat error reporting path for commit to properly
>    calculate dss_id.
> 
> v3:
>  - Fixes do_div dividend to be u64.
> 
> v4:
>  - Use regular division operators for u32 commit path math.
>  - Fix mirror null check in ff_rw_layout_has_available_ds.
> 
> Jonathan Curley (9):
>   NFSv4/flexfiles: Remove cred local variable dependency
>   NFSv4/flexfiles: Use ds_commit_idx when marking a write commit
>   NFSv4/flexfiles: Add data structure support for striped layouts
>   NFSv4/flexfiles: Update low level helper functions to be DS stripe
>     aware.
>   NFSv4/flexfiles: Read path updates for striped layouts
>   NFSv4/flexfiles: Commit path updates for striped layouts
>   NFSv4/flexfiles: Write path updates for striped layouts
>   NFSv4/flexfiles: Update layout stats & error paths for striped layouts
>   NFSv4/flexfiles: Add support for striped layouts
> 
>  fs/nfs/flexfilelayout/flexfilelayout.c    | 778 +++++++++++++++-------
>  fs/nfs/flexfilelayout/flexfilelayout.h    |  64 +-
>  fs/nfs/flexfilelayout/flexfilelayoutdev.c | 105 +--
>  fs/nfs/write.c                            |   2 +-
>  4 files changed, 635 insertions(+), 314 deletions(-)
> 
> -- 
> 2.34.1
> 
> 

Hi Jonathan,

Testing the latest 'nfs-for-6.18-1' tag (now merged into Linus' tree),
with your flexfiles striped layout changes, using NFS LOCALIO results
in NFSD shutdown hanging due to nfsd refcount.

I'm using 4.2 flexfiles client that connects to local system's DS via
NFS v3 over LOCALIO, and then simply issuing very brief IO with dd:

dd if=/dev/zero of=/mnt/hs_test/dd_thisisa.test bs=47008 count=2 oflag=direct
dd if=/mnt/hs_test/dd_thisisa.test of=/dev/null bs=47008 count=2 iflag=direct

followed by umount of the filesystem, and then attempt to stop all
NFS/NFSD services so that all related kernel modules may be unloaded:

umount /mnt/hs_test
systemctl stop rpc-statd.service
systemctl stop rpc-statd-notify.service
systemctl stop var-lib-nfs-rpc_pipefs.mount
systemctl stop proc-fs-nfsd.mount
systemctl stop nfs-server.service
 # ^ this hangs below...

systemctl stop rpcbind
systemctl stop rpcbind.socket
systemctl stop nfsdcld.service
systemctl stop nfs-client.target

/var/log/messages shows:

Oct  6 18:08:20 plsm121c-06 systemd[1]: mnt-hs_test.mount: Deactivated successfully.
Oct  6 18:08:20 plsm121c-06 systemd[1]: Stopping NFS status monitor for NFSv2/3 locking....
Oct  6 18:08:20 plsm121c-06 systemd[1]: rpc-statd.service: Deactivated successfully.
Oct  6 18:08:20 plsm121c-06 systemd[1]: Stopped NFS status monitor for NFSv2/3 locking..
Oct  6 18:08:20 plsm121c-06 systemd[1]: rpc-statd-notify.service: Deactivated successfully.
Oct  6 18:08:20 plsm121c-06 systemd[1]: Stopped Notify NFS peers of a restart.
Oct  6 18:08:20 plsm121c-06 rpc.idmapd[8982]: exiting on signal 15
Oct  6 18:08:20 plsm121c-06 systemd[1]: Stopping NFSv4 ID-name mapping service...
Oct  6 18:08:20 plsm121c-06 systemd[1]: Stopping NFSv4 Client Tracking Daemon...
Oct  6 18:08:20 plsm121c-06 systemd[1]: nfs-idmapd.service: Deactivated successfully.
Oct  6 18:08:20 plsm121c-06 systemd[1]: Stopped NFSv4 ID-name mapping service.
Oct  6 18:08:20 plsm121c-06 systemd[1]: nfsdcld.service: Deactivated successfully.
Oct  6 18:08:20 plsm121c-06 systemd[1]: Stopped NFSv4 Client Tracking Daemon.
Oct  6 18:08:20 plsm121c-06 systemd[1]: Stopped target rpc_pipefs.target.
Oct  6 18:08:20 plsm121c-06 systemd[1]: Unmounting RPC Pipe File System...
Oct  6 18:08:20 plsm121c-06 systemd[1]: var-lib-nfs-rpc_pipefs.mount: Deactivated successfully.
Oct  6 18:08:20 plsm121c-06 systemd[1]: Unmounted RPC Pipe File System.
Oct  6 18:08:20 plsm121c-06 systemd[1]: Stopping NFS server and services...
Oct  6 18:08:22 plsm121c-06 rpc.mountd[8988]: v4.2 client detached: 0x68a8fa0468e404e3 from "192.168.0.105:853"

Oct  6 18:09:50 plsm121c-06 systemd[1]: nfs-server.service: Stopping timed out. Terminating.
Oct  6 18:10:00 plsm121c-06 systemd[1]: Starting system activity accounting tool...
Oct  6 18:10:00 plsm121c-06 systemd[1]: sysstat-collect.service: Deactivated successfully.
Oct  6 18:10:00 plsm121c-06 systemd[1]: Finished system activity accounting tool.

Oct  6 18:11:21 plsm121c-06 systemd[1]: nfs-server.service: State 'stop-sigterm' timed out. Killing.
Oct  6 18:11:21 plsm121c-06 systemd[1]: nfs-server.service: Killing process 9669 (rpc.nfsd) with signal SIGKILL.
Oct  6 18:11:26 plsm121c-06 kernel: INFO: task rpc.nfsd:9669 blocked for more than 122 seconds.
Oct  6 18:11:26 plsm121c-06 kernel:      Not tainted 6.12.24.23.hs.snitm+ #44
Oct  6 18:11:26 plsm121c-06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct  6 18:11:26 plsm121c-06 kernel: task:rpc.nfsd        state:D stack:0     pid:9669  tgid:9669  ppid:1      flags:0x00004006
Oct  6 18:11:26 plsm121c-06 kernel: Call Trace:
Oct  6 18:11:26 plsm121c-06 kernel: <TASK>
Oct  6 18:11:26 plsm121c-06 kernel: __schedule+0x26d/0x530
Oct  6 18:11:26 plsm121c-06 kernel: schedule+0x27/0xa0
Oct  6 18:11:26 plsm121c-06 kernel: schedule_timeout+0x14e/0x160
Oct  6 18:11:26 plsm121c-06 kernel: ? svc_destroy+0xce/0x160 [sunrpc]
Oct  6 18:11:26 plsm121c-06 kernel: ? lockd_put+0x5f/0x90 [lockd]
Oct  6 18:11:26 plsm121c-06 kernel: __wait_for_common+0x8f/0x1d0
Oct  6 18:11:26 plsm121c-06 kernel: ? __pfx_schedule_timeout+0x10/0x10
Oct  6 18:11:26 plsm121c-06 kernel: nfsd_destroy_serv+0x138/0x1a0 [nfsd]
Oct  6 18:11:26 plsm121c-06 kernel: nfsd_svc+0xe0/0x170 [nfsd]
Oct  6 18:11:26 plsm121c-06 kernel: write_threads+0xc3/0x190 [nfsd]
Oct  6 18:11:26 plsm121c-06 kernel: ? simple_transaction_get+0xc2/0xe0
Oct  6 18:11:26 plsm121c-06 kernel: ? __pfx_write_threads+0x10/0x10 [nfsd]
Oct  6 18:11:26 plsm121c-06 kernel: nfsctl_transaction_write+0x47/0x80 [nfsd]
Oct  6 18:11:26 plsm121c-06 kernel: vfs_write+0xfa/0x420
Oct  6 18:11:26 plsm121c-06 kernel: ? syscall_exit_work+0xf3/0x120
Oct  6 18:11:26 plsm121c-06 kernel: ? syscall_exit_to_user_mode+0x32/0x1b0
Oct  6 18:11:26 plsm121c-06 kernel: ksys_write+0x63/0xe0
Oct  6 18:11:26 plsm121c-06 kernel: do_syscall_64+0x7d/0x160
Oct  6 18:11:26 plsm121c-06 kernel: ? __x64_sys_close+0x3c/0x80
Oct  6 18:11:26 plsm121c-06 kernel: ? kmem_cache_free+0x347/0x450
Oct  6 18:11:26 plsm121c-06 kernel: ? syscall_exit_work+0xf3/0x120
Oct  6 18:11:26 plsm121c-06 kernel: ? syscall_exit_to_user_mode+0x32/0x1b0
Oct  6 18:11:26 plsm121c-06 kernel: ? do_syscall_64+0x89/0x160
Oct  6 18:11:26 plsm121c-06 kernel: ? list_lru_add+0x142/0x190
Oct  6 18:11:26 plsm121c-06 kernel: ? task_lookup_next_fdget_rcu+0x91/0xd0
Oct  6 18:11:26 plsm121c-06 kernel: ? __pfx_proc_fd_instantiate+0x10/0x10
Oct  6 18:11:26 plsm121c-06 kernel: ? proc_readfd_common+0xf5/0x1f0
Oct  6 18:11:26 plsm121c-06 kernel: ? atime_needs_update+0x61/0x120
Oct  6 18:11:26 plsm121c-06 kernel: ? touch_atime+0x1e/0x100
Oct  6 18:11:26 plsm121c-06 kernel: ? iterate_dir+0x18f/0x220
Oct  6 18:11:26 plsm121c-06 kernel: ? __x64_sys_getdents64+0xf7/0x120
Oct  6 18:11:26 plsm121c-06 kernel: ? syscall_exit_work+0xf3/0x120
Oct  6 18:11:26 plsm121c-06 kernel: ? syscall_exit_to_user_mode+0x32/0x1b0
Oct  6 18:11:26 plsm121c-06 kernel: ? do_syscall_64+0x89/0x160
Oct  6 18:11:26 plsm121c-06 kernel: ? syscall_exit_work+0xf3/0x120
Oct  6 18:11:26 plsm121c-06 kernel: ? syscall_exit_to_user_mode+0x32/0x1b0
Oct  6 18:11:26 plsm121c-06 kernel: ? do_syscall_64+0x89/0x160
Oct  6 18:11:26 plsm121c-06 kernel: ? syscall_exit_work+0xf3/0x120
Oct  6 18:11:26 plsm121c-06 kernel: ? syscall_exit_to_user_mode+0x32/0x1b0
Oct  6 18:11:26 plsm121c-06 kernel: ? do_syscall_64+0x89/0x160
Oct  6 18:11:26 plsm121c-06 kernel: ? do_user_addr_fault+0x341/0x6b0
Oct  6 18:11:26 plsm121c-06 kernel: ? exc_page_fault+0x70/0x160
Oct  6 18:11:26 plsm121c-06 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Oct  6 18:11:26 plsm121c-06 kernel: RIP: 0033:0x7fc17ecfd617
Oct  6 18:11:26 plsm121c-06 kernel: RSP: 002b:00007ffc321ff1c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
Oct  6 18:11:26 plsm121c-06 kernel: RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fc17ecfd617
Oct  6 18:11:26 plsm121c-06 kernel: RDX: 0000000000000002 RSI: 000056109cba2c20 RDI: 0000000000000003
Oct  6 18:11:26 plsm121c-06 kernel: RBP: 0000000000000003 R08: 0000000000000000 R09: 00007ffc321ff060
Oct  6 18:11:26 plsm121c-06 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000056109cba2c20
Oct  6 18:11:26 plsm121c-06 kernel: R13: 00007fc17ee086c8 R14: 00007ffc321ff290 R15: 00000000ffffffff
Oct  6 18:11:26 plsm121c-06 kernel: </TASK>

I started to try to bisect but if I only apply the first 3 commits in
your series:

fec80afc41af NFSv4/flexfiles: Remove cred local variable dependency
eb71428e1a7f NFSv4/flexfiles: Use ds_commit_idx when marking a write commit
d442670c0f63 NFSv4/flexfiles: Add data structure support for striped layouts

and rerun the test I get a kernel panic (which I don't yet have a
crashdump for, kdump appears misconfigured on my system).

Mike