Re: Current state of nfsdv4 recovery.

[Jeff Layton <jlayton@kernel.org>]

On Fri, 2026-03-06 at 11:46 -0800, Matthew Schumacher wrote:
> Hello List,
> 
> I am building an HA nfsv4 cluster and need to have my locking migrate 
> when a node fails or goes into standby for maint.  There is a lot of 
> documentation around the net, but some of it seems old, like this: 
> https://wiki.linux-nfs.org/wiki/index.php/Nfsd4_server_recovery. I think 
> I have what I need worked out, but I'm still losing locks on recovery.  
> Can someone help me understand how this currently works or point me to 
> the up to date documentation?
> 
> This is what I'm currently doing:
> 
> Moving from nodeA to nodeB
> 
> NodeA: Remove nfsd ip address
> NodeA: /usr/sbin/exportfs -au
> NodeA: /usr/sbin/rpc.nfsd 0
> NodeA: killall --ns $$ -q -v -w rpc.mountd
> NodeA: killall --ns $$ -q -v -w nfsdcld
> NodeA: killall --ns $$ -q -v -w rpc.idmapd
> NodeA: killall --ns $$ -q -v -w rpcbind
> NodeA: umount /var/lib/nfs/rpc_pipefs
> NodeA: zpool export (which umounts /var/lib/nfs)
> 
> NodeB: zpool import (which mounts /var/lib/nfs)
> NodeB: mount /proc/fs/nfs if not previously mounted
> NodeB: mount -t rpc_pipefs -o nodev sunrpc /var/lib/nfs/rpc_pipefs
> NodeB: /sbin/rpcbind -l -w
> NodeB: /usr/sbin/rpc.idmapd
> NodeB: /usr/sbin/nfsdcld -d
> NodeB: /usr/sbin/exportfs -av
> NodeB: /usr/sbin/rpc.mountd
> NodeB: /usr/sbin/rpc.nfsd -s -V 4.2 -U -t 8
> NodeB: start nfsd ip address
> 
>  From what I understand, nfsdcld writes the lock information to 
> /var/lib/nfs/nfsdcltrack/main.sqlite which is moved to the other side 
> and everything is started, but I still lose locks when I migrate.
> 
> Mar  6 08:39:14 nfsha user.warn kernel: [37675.787509] NFS: 
> 10.255.255.3: lost 1 locks
> 
> I think the problem is that kernel (6.12.74) isn't getting the locks 
> reported to it when nfsdcld starts.  Here are some logs:
> 
> Mar  6 08:38:46 nfsha daemon.warn rpc.idmapd[14113]: Setting log level to 0
> Mar  6 08:38:46 nfsha daemon.warn rpc.idmapd[14113]: libnfsidmap: Unable 
> to determine the NFSv4 domain; Using 'localdomain' as the NFSv4 domain 
> which means UIDs will be mapped to the 'Nobody-User' user defined in 
> /etc/idmapd.conf
> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: 
> sqlite_startup_query_grace: current_epoch=2 recovery_epoch=1
> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: 
> sqlite_check_db_health: returning 0
> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: attaching 
> /var/lib/nfs/nfsdcltrack/main.sqlite
> Mar  6 08:38:46 nfsha daemon.notice rpc.mountd[14125]: Version 2.8.5 
> starting
> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: detaching database
> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: 
> sqlite_copy_cltrack_records: returning 0
> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: sqlite_prepare_dbh: 
> num_cltrack_records = 2
> Mar  6 08:38:47 nfsha user.info kernel: [37649.053531] NFSD: Using UMH 
> upcall client tracking operations.
> Mar  6 08:38:47 nfsha user.info kernel: [37649.055262] NFSD: Using UMH 
> upcall client tracking operations.

The kernel failed to recognize the cld upcall for some reason, so it
fell back to running the usermodehelper upcall.

> Mar  6 08:38:47 nfsha user.info kernel: [37649.055264] NFSD: starting 
> 90-second grace period (net f0000000)
> Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: sqlite_prepare_dbh: 
> num_legacy_records = 0
> Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: cld_pipe_init: init 
> pipe handlers
> Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: cld_pipe_open: opening 
> upcall pipe /var/lib/nfs/rpc_pipefs/nfsd/cld
> Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: cld_pipe_open: open of 
> /var/lib/nfs/rpc_pipefs/nfsd/cld failed: No such file or directory
> Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: main: Starting event 
> dispatch handler.
> Mar  6 08:38:47 nfsha daemon.info IPaddr(failover-ip)[14153]: INFO: eval 
> ifconfig eth3:0 10.255.255.3 netmask 255.255.255.0 broadcast 10.255.255.255
> Mar  6 08:38:47 nfsha daemon.notice rpc.mountd[14125]: v4.2 client 
> attached: 0x7bdfeba269ab0316 from "10.255.255.2:47380"
> Mar  6 08:39:13 nfsha daemon.notice rpc.mountd[14125]: v4.2 client 
> attached: 0x7bdfeba369ab0316 from "10.255.255.1:49220"
> Mar  6 08:39:14 nfsha user.warn kernel: [37675.787509] NFS: 
> 10.255.255.3: lost 1 locks
> 
> I think the problem is that nfsdcld can't pass the locks to the kernel 
> using /var/lib/nfs/rpc_pipefs/nfsd/cld as it's missing.  I'm not sure 
> why, /var/lib/nfs/rpc_pipefs/ has everything else:
> 
> find /var/lib/nfs/rpc_pipefs/
> /var/lib/nfs/rpc_pipefs/
> /var/lib/nfs/rpc_pipefs/gssd
> /var/lib/nfs/rpc_pipefs/gssd/clntXX
> /var/lib/nfs/rpc_pipefs/gssd/clntXX/gssd
> /var/lib/nfs/rpc_pipefs/gssd/clntXX/info
> /var/lib/nfs/rpc_pipefs/nfsd
> /var/lib/nfs/rpc_pipefs/cache
> /var/lib/nfs/rpc_pipefs/nfsd4_cb
> /var/lib/nfs/rpc_pipefs/statd
> /var/lib/nfs/rpc_pipefs/portmap
> /var/lib/nfs/rpc_pipefs/nfs
> /var/lib/nfs/rpc_pipefs/nfs/blocklayout
> /var/lib/nfs/rpc_pipefs/nfs/clnt1
> /var/lib/nfs/rpc_pipefs/nfs/clnt1/info
> /var/lib/nfs/rpc_pipefs/nfs/clnt0
> /var/lib/nfs/rpc_pipefs/nfs/clnt0/info
> /var/lib/nfs/rpc_pipefs/mount
> /var/lib/nfs/rpc_pipefs/lockd
> 

That's very odd. You might want to look in dmesg and see if there are
any warnings in there about creating the pipes in rpc_pipefs.

> Anyone able to point me in the right direction?  I feel like I'm pretty 
> close to getting this to work.

-- 
Jeff Layton <jlayton@kernel.org>