Current state of nfsdv4 recovery.

Current state of nfsdv4 recovery.

[Matthew Schumacher]

Hello List,

I am building an HA nfsv4 cluster and need to have my locking migrate 
when a node fails or goes into standby for maint.  There is a lot of 
documentation around the net, but some of it seems old, like this: 
https://wiki.linux-nfs.org/wiki/index.php/Nfsd4_server_recovery. I think 
I have what I need worked out, but I'm still losing locks on recovery.  
Can someone help me understand how this currently works or point me to 
the up to date documentation?

This is what I'm currently doing:

Moving from nodeA to nodeB

NodeA: Remove nfsd ip address
NodeA: /usr/sbin/exportfs -au
NodeA: /usr/sbin/rpc.nfsd 0
NodeA: killall --ns $$ -q -v -w rpc.mountd
NodeA: killall --ns $$ -q -v -w nfsdcld
NodeA: killall --ns $$ -q -v -w rpc.idmapd
NodeA: killall --ns $$ -q -v -w rpcbind
NodeA: umount /var/lib/nfs/rpc_pipefs
NodeA: zpool export (which umounts /var/lib/nfs)

NodeB: zpool import (which mounts /var/lib/nfs)
NodeB: mount /proc/fs/nfs if not previously mounted
NodeB: mount -t rpc_pipefs -o nodev sunrpc /var/lib/nfs/rpc_pipefs
NodeB: /sbin/rpcbind -l -w
NodeB: /usr/sbin/rpc.idmapd
NodeB: /usr/sbin/nfsdcld -d
NodeB: /usr/sbin/exportfs -av
NodeB: /usr/sbin/rpc.mountd
NodeB: /usr/sbin/rpc.nfsd -s -V 4.2 -U -t 8
NodeB: start nfsd ip address

 From what I understand, nfsdcld writes the lock information to 
/var/lib/nfs/nfsdcltrack/main.sqlite which is moved to the other side 
and everything is started, but I still lose locks when I migrate.

Mar  6 08:39:14 nfsha user.warn kernel: [37675.787509] NFS: 
10.255.255.3: lost 1 locks

I think the problem is that kernel (6.12.74) isn't getting the locks 
reported to it when nfsdcld starts.  Here are some logs:

Mar  6 08:38:46 nfsha daemon.warn rpc.idmapd[14113]: Setting log level to 0
Mar  6 08:38:46 nfsha daemon.warn rpc.idmapd[14113]: libnfsidmap: Unable 
to determine the NFSv4 domain; Using 'localdomain' as the NFSv4 domain 
which means UIDs will be mapped to the 'Nobody-User' user defined in 
/etc/idmapd.conf
Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: 
sqlite_startup_query_grace: current_epoch=2 recovery_epoch=1
Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: 
sqlite_check_db_health: returning 0
Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: attaching 
/var/lib/nfs/nfsdcltrack/main.sqlite
Mar  6 08:38:46 nfsha daemon.notice rpc.mountd[14125]: Version 2.8.5 
starting
Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: detaching database
Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: 
sqlite_copy_cltrack_records: returning 0
Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: sqlite_prepare_dbh: 
num_cltrack_records = 2
Mar  6 08:38:47 nfsha user.info kernel: [37649.053531] NFSD: Using UMH 
upcall client tracking operations.
Mar  6 08:38:47 nfsha user.info kernel: [37649.055262] NFSD: Using UMH 
upcall client tracking operations.
Mar  6 08:38:47 nfsha user.info kernel: [37649.055264] NFSD: starting 
90-second grace period (net f0000000)
Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: sqlite_prepare_dbh: 
num_legacy_records = 0
Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: cld_pipe_init: init 
pipe handlers
Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: cld_pipe_open: opening 
upcall pipe /var/lib/nfs/rpc_pipefs/nfsd/cld
Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: cld_pipe_open: open of 
/var/lib/nfs/rpc_pipefs/nfsd/cld failed: No such file or directory
Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: main: Starting event 
dispatch handler.
Mar  6 08:38:47 nfsha daemon.info IPaddr(failover-ip)[14153]: INFO: eval 
ifconfig eth3:0 10.255.255.3 netmask 255.255.255.0 broadcast 10.255.255.255
Mar  6 08:38:47 nfsha daemon.notice rpc.mountd[14125]: v4.2 client 
attached: 0x7bdfeba269ab0316 from "10.255.255.2:47380"
Mar  6 08:39:13 nfsha daemon.notice rpc.mountd[14125]: v4.2 client 
attached: 0x7bdfeba369ab0316 from "10.255.255.1:49220"
Mar  6 08:39:14 nfsha user.warn kernel: [37675.787509] NFS: 
10.255.255.3: lost 1 locks

I think the problem is that nfsdcld can't pass the locks to the kernel 
using /var/lib/nfs/rpc_pipefs/nfsd/cld as it's missing.  I'm not sure 
why, /var/lib/nfs/rpc_pipefs/ has everything else:

find /var/lib/nfs/rpc_pipefs/
/var/lib/nfs/rpc_pipefs/
/var/lib/nfs/rpc_pipefs/gssd
/var/lib/nfs/rpc_pipefs/gssd/clntXX
/var/lib/nfs/rpc_pipefs/gssd/clntXX/gssd
/var/lib/nfs/rpc_pipefs/gssd/clntXX/info
/var/lib/nfs/rpc_pipefs/nfsd
/var/lib/nfs/rpc_pipefs/cache
/var/lib/nfs/rpc_pipefs/nfsd4_cb
/var/lib/nfs/rpc_pipefs/statd
/var/lib/nfs/rpc_pipefs/portmap
/var/lib/nfs/rpc_pipefs/nfs
/var/lib/nfs/rpc_pipefs/nfs/blocklayout
/var/lib/nfs/rpc_pipefs/nfs/clnt1
/var/lib/nfs/rpc_pipefs/nfs/clnt1/info
/var/lib/nfs/rpc_pipefs/nfs/clnt0
/var/lib/nfs/rpc_pipefs/nfs/clnt0/info
/var/lib/nfs/rpc_pipefs/mount
/var/lib/nfs/rpc_pipefs/lockd

Anyone able to point me in the right direction?  I feel like I'm pretty 
close to getting this to work.

Re: Current state of nfsdv4 recovery.

[Jeff Layton]

On Fri, 2026-03-06 at 11:46 -0800, Matthew Schumacher wrote:
> Hello List,
> 
> I am building an HA nfsv4 cluster and need to have my locking migrate 
> when a node fails or goes into standby for maint.  There is a lot of 
> documentation around the net, but some of it seems old, like this: 
> https://wiki.linux-nfs.org/wiki/index.php/Nfsd4_server_recovery. I think 
> I have what I need worked out, but I'm still losing locks on recovery.  
> Can someone help me understand how this currently works or point me to 
> the up to date documentation?
> 
> This is what I'm currently doing:
> 
> Moving from nodeA to nodeB
> 
> NodeA: Remove nfsd ip address
> NodeA: /usr/sbin/exportfs -au
> NodeA: /usr/sbin/rpc.nfsd 0
> NodeA: killall --ns $$ -q -v -w rpc.mountd
> NodeA: killall --ns $$ -q -v -w nfsdcld
> NodeA: killall --ns $$ -q -v -w rpc.idmapd
> NodeA: killall --ns $$ -q -v -w rpcbind
> NodeA: umount /var/lib/nfs/rpc_pipefs
> NodeA: zpool export (which umounts /var/lib/nfs)
> 
> NodeB: zpool import (which mounts /var/lib/nfs)
> NodeB: mount /proc/fs/nfs if not previously mounted
> NodeB: mount -t rpc_pipefs -o nodev sunrpc /var/lib/nfs/rpc_pipefs
> NodeB: /sbin/rpcbind -l -w
> NodeB: /usr/sbin/rpc.idmapd
> NodeB: /usr/sbin/nfsdcld -d
> NodeB: /usr/sbin/exportfs -av
> NodeB: /usr/sbin/rpc.mountd
> NodeB: /usr/sbin/rpc.nfsd -s -V 4.2 -U -t 8
> NodeB: start nfsd ip address
> 
>  From what I understand, nfsdcld writes the lock information to 
> /var/lib/nfs/nfsdcltrack/main.sqlite which is moved to the other side 
> and everything is started, but I still lose locks when I migrate.
> 
> Mar  6 08:39:14 nfsha user.warn kernel: [37675.787509] NFS: 
> 10.255.255.3: lost 1 locks
> 
> I think the problem is that kernel (6.12.74) isn't getting the locks 
> reported to it when nfsdcld starts.  Here are some logs:
> 
> Mar  6 08:38:46 nfsha daemon.warn rpc.idmapd[14113]: Setting log level to 0
> Mar  6 08:38:46 nfsha daemon.warn rpc.idmapd[14113]: libnfsidmap: Unable 
> to determine the NFSv4 domain; Using 'localdomain' as the NFSv4 domain 
> which means UIDs will be mapped to the 'Nobody-User' user defined in 
> /etc/idmapd.conf
> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: 
> sqlite_startup_query_grace: current_epoch=2 recovery_epoch=1
> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: 
> sqlite_check_db_health: returning 0
> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: attaching 
> /var/lib/nfs/nfsdcltrack/main.sqlite
> Mar  6 08:38:46 nfsha daemon.notice rpc.mountd[14125]: Version 2.8.5 
> starting
> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: detaching database
> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: 
> sqlite_copy_cltrack_records: returning 0
> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: sqlite_prepare_dbh: 
> num_cltrack_records = 2
> Mar  6 08:38:47 nfsha user.info kernel: [37649.053531] NFSD: Using UMH 
> upcall client tracking operations.
> Mar  6 08:38:47 nfsha user.info kernel: [37649.055262] NFSD: Using UMH 
> upcall client tracking operations.

The kernel failed to recognize the cld upcall for some reason, so it
fell back to running the usermodehelper upcall.

> Mar  6 08:38:47 nfsha user.info kernel: [37649.055264] NFSD: starting 
> 90-second grace period (net f0000000)
> Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: sqlite_prepare_dbh: 
> num_legacy_records = 0
> Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: cld_pipe_init: init 
> pipe handlers
> Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: cld_pipe_open: opening 
> upcall pipe /var/lib/nfs/rpc_pipefs/nfsd/cld
> Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: cld_pipe_open: open of 
> /var/lib/nfs/rpc_pipefs/nfsd/cld failed: No such file or directory
> Mar  6 08:38:47 nfsha daemon.info nfsdcld[14116]: main: Starting event 
> dispatch handler.
> Mar  6 08:38:47 nfsha daemon.info IPaddr(failover-ip)[14153]: INFO: eval 
> ifconfig eth3:0 10.255.255.3 netmask 255.255.255.0 broadcast 10.255.255.255
> Mar  6 08:38:47 nfsha daemon.notice rpc.mountd[14125]: v4.2 client 
> attached: 0x7bdfeba269ab0316 from "10.255.255.2:47380"
> Mar  6 08:39:13 nfsha daemon.notice rpc.mountd[14125]: v4.2 client 
> attached: 0x7bdfeba369ab0316 from "10.255.255.1:49220"
> Mar  6 08:39:14 nfsha user.warn kernel: [37675.787509] NFS: 
> 10.255.255.3: lost 1 locks
> 
> I think the problem is that nfsdcld can't pass the locks to the kernel 
> using /var/lib/nfs/rpc_pipefs/nfsd/cld as it's missing.  I'm not sure 
> why, /var/lib/nfs/rpc_pipefs/ has everything else:
> 
> find /var/lib/nfs/rpc_pipefs/
> /var/lib/nfs/rpc_pipefs/
> /var/lib/nfs/rpc_pipefs/gssd
> /var/lib/nfs/rpc_pipefs/gssd/clntXX
> /var/lib/nfs/rpc_pipefs/gssd/clntXX/gssd
> /var/lib/nfs/rpc_pipefs/gssd/clntXX/info
> /var/lib/nfs/rpc_pipefs/nfsd
> /var/lib/nfs/rpc_pipefs/cache
> /var/lib/nfs/rpc_pipefs/nfsd4_cb
> /var/lib/nfs/rpc_pipefs/statd
> /var/lib/nfs/rpc_pipefs/portmap
> /var/lib/nfs/rpc_pipefs/nfs
> /var/lib/nfs/rpc_pipefs/nfs/blocklayout
> /var/lib/nfs/rpc_pipefs/nfs/clnt1
> /var/lib/nfs/rpc_pipefs/nfs/clnt1/info
> /var/lib/nfs/rpc_pipefs/nfs/clnt0
> /var/lib/nfs/rpc_pipefs/nfs/clnt0/info
> /var/lib/nfs/rpc_pipefs/mount
> /var/lib/nfs/rpc_pipefs/lockd
> 

That's very odd. You might want to look in dmesg and see if there are
any warnings in there about creating the pipes in rpc_pipefs.

> Anyone able to point me in the right direction?  I feel like I'm pretty 
> close to getting this to work.

-- 
Jeff Layton <jlayton@kernel.org>

Re: Current state of nfsdv4 recovery.

[Matthew Schumacher]

On 3/6/26 12:49 PM, Jeff Layton wrote:
> On Fri, 2026-03-06 at 11:46 -0800, Matthew Schumacher wrote:
<snip>
>>
>> Moving from nodeA to nodeB
>>
>> NodeA: Remove nfsd ip address
>> NodeA: /usr/sbin/exportfs -au
>> NodeA: /usr/sbin/rpc.nfsd 0
>> NodeA: killall --ns $$ -q -v -w rpc.mountd
>> NodeA: killall --ns $$ -q -v -w nfsdcld
>> NodeA: killall --ns $$ -q -v -w rpc.idmapd
>> NodeA: killall --ns $$ -q -v -w rpcbind
>> NodeA: umount /var/lib/nfs/rpc_pipefs
>> NodeA: zpool export (which umounts /var/lib/nfs)
>>
>> NodeB: zpool import (which mounts /var/lib/nfs)
>> NodeB: mount /proc/fs/nfs if not previously mounted
>> NodeB: mount -t rpc_pipefs -o nodev sunrpc /var/lib/nfs/rpc_pipefs
>> NodeB: /sbin/rpcbind -l -w
>> NodeB: /usr/sbin/rpc.idmapd
>> NodeB: /usr/sbin/nfsdcld -d
>> NodeB: /usr/sbin/exportfs -av
>> NodeB: /usr/sbin/rpc.mountd
>> NodeB: /usr/sbin/rpc.nfsd -s -V 4.2 -U -t 8
>> NodeB: start nfsd ip address
>>
>>   From what I understand, nfsdcld writes the lock information to
>> /var/lib/nfs/nfsdcltrack/main.sqlite which is moved to the other side
>> and everything is started, but I still lose locks when I migrate.
>>
>> Mar  6 08:39:14 nfsha user.warn kernel: [37675.787509] NFS:
>> 10.255.255.3: lost 1 locks
>>
>> I think the problem is that kernel (6.12.74) isn't getting the locks
>> reported to it when nfsdcld starts.  Here are some logs:
>>
>> Mar  6 08:38:46 nfsha daemon.warn rpc.idmapd[14113]: Setting log level to 0
>> Mar  6 08:38:46 nfsha daemon.warn rpc.idmapd[14113]: libnfsidmap: Unable
>> to determine the NFSv4 domain; Using 'localdomain' as the NFSv4 domain
>> which means UIDs will be mapped to the 'Nobody-User' user defined in
>> /etc/idmapd.conf
>> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]:
>> sqlite_startup_query_grace: current_epoch=2 recovery_epoch=1
>> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]:
>> sqlite_check_db_health: returning 0
>> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: attaching
>> /var/lib/nfs/nfsdcltrack/main.sqlite
>> Mar  6 08:38:46 nfsha daemon.notice rpc.mountd[14125]: Version 2.8.5
>> starting
>> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: detaching database
>> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]:
>> sqlite_copy_cltrack_records: returning 0
>> Mar  6 08:38:46 nfsha daemon.info nfsdcld[14116]: sqlite_prepare_dbh:
>> num_cltrack_records = 2
>> Mar  6 08:38:47 nfsha user.info kernel: [37649.053531] NFSD: Using UMH
>> upcall client tracking operations.
>> Mar  6 08:38:47 nfsha user.info kernel: [37649.055262] NFSD: Using UMH
>> upcall client tracking operations.
> 
> The kernel failed to recognize the cld upcall for some reason, so it
> fell back to running the usermodehelper upcall.
<snip>

Thanks for the reply, I figured out why /var/lib/nfs/rpc_pipefs/nfsd/cld 
was missing, I had CONFIG_NFSD_LEGACY_CLIENT_TRACKING enabled in the 
kernel.  Now that that is fixed, nfsv4 recovery seems to be working 
because I see it notifying the nfs clients and ending the grace period, 
but I'm still losing the lock.  Any idea why?  Or how to debug?

Mar  6 14:53:31 nfsha daemon.warn rpc.idmapd[3243]: Setting log level to 0
Mar  6 14:53:31 nfsha daemon.warn rpc.idmapd[3243]: libnfsidmap: Unable 
to determine the NFSv4 domain; Using 'localdomain' as the NFSv4 domain 
which means UIDs will be mapped to the 'Nobody-User' user defined in 
/etc/idmapd.conf
Mar  6 14:53:31 nfsha daemon.notice rpc.mountd[3252]: Version 2.8.5 starting
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: 
sqlite_startup_query_grace: current_epoch=5 recovery_epoch=0
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: sqlite_check_db_health: 
returning 0
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: cld_pipe_init: init 
pipe handlers
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: cld_pipe_open: opening 
upcall pipe /var/lib/nfs/rpc_pipefs/nfsd/cld
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: cld_pipe_open: open of 
/var/lib/nfs/rpc_pipefs/nfsd/cld failed: No such file or directory
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: main: Starting event 
dispatch handler.
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: cld_inotify_cb: called 
for EV_READ
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: cld_pipe_open: opening 
upcall pipe /var/lib/nfs/rpc_pipefs/nfsd/cld
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: cld_get_version: 
version = 2.
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: Doing downcall with 
status 0
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: cld_gracestart: 
updating grace epochs
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: sqlite_grace_start: 
current_epoch=6 recovery_epoch=5
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: cld_gracestart: sending 
client records to the kernel
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: Sending client Linux 
NFSv4.2 virttest-2
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: Sending client Linux 
NFSv4.2 virttest-1
Mar  6 14:53:31 nfsha daemon.info nfsdcld[3254]: Doing downcall with 
status 0
Mar  6 14:53:31 nfsha user.info kernel: [ 1831.162648] NFSD: Using 
nfsdcld client tracking operations.
Mar  6 14:53:31 nfsha user.info kernel: [ 1831.162651] NFSD: starting 
90-second grace period (net f0000000)
Mar  6 14:53:31 nfsha daemon.info IPaddr(failover-ip)[3283]: INFO: Using 
calculated netmask for 10.255.255.3: 255.255.255.0
Mar  6 14:53:31 nfsha daemon.info IPaddr(failover-ip)[3283]: INFO: eval 
ifconfig eth3:0 10.255.255.3 netmask 255.255.255.0 broadcast 10.255.255.255
Mar  6 14:53:34 nfsha daemon.notice rpc.mountd[3252]: v4.2 client 
attached: 0x2baa617369ab5aeb from "10.255.255.1:51662"
Mar  6 14:53:34 nfsha daemon.info nfsdcld[3254]: cld_create: create 
client record.
Mar  6 14:53:34 nfsha daemon.info nfsdcld[3254]: 
sqlite_insert_client_and_princhash: returning 0
Mar  6 14:53:34 nfsha daemon.info nfsdcld[3254]: Doing downcall with 
status 0
Mar  6 14:53:42 nfsha daemon.notice rpc.mountd[3252]: v4.2 client 
attached: 0x2baa617469ab5aeb from "10.255.255.2:53976"
Mar  6 14:53:42 nfsha daemon.info nfsdcld[3254]: cld_create: create 
client record.
Mar  6 14:53:42 nfsha daemon.info nfsdcld[3254]: 
sqlite_insert_client_and_princhash: returning 0
Mar  6 14:53:42 nfsha daemon.info nfsdcld[3254]: Doing downcall with 
status 0
Mar  6 14:53:42 nfsha daemon.info nfsdcld[3254]: cld_gracedone: grace done.
Mar  6 14:53:42 nfsha user.info kernel: [ 1842.038046] NFSD: all clients 
done reclaiming, ending NFSv4 grace period (net f0000000)
Mar  6 14:53:42 nfsha daemon.info nfsdcld[3254]: sqlite_grace_done: 
current_epoch=6 recovery_epoch=0
Mar  6 14:53:42 nfsha daemon.info nfsdcld[3254]: Doing downcall with 
status 0
Mar  6 14:53:48 nfsha user.warn kernel: [ 1847.336008] NFS: 
10.255.255.3: lost 1 locks

I'm mounting with:

mount -t nfs -o 
rw,sync,vers=4.2,hard,noresvport,proto=tcp,timeo=600,retrans=2 
10.255.255.3:/dev/datastore-stub /datastore

Re: Current state of nfsdv4 recovery.

[Matthew Schumacher]

On 3/6/26 12:49 PM, Jeff Layton wrote:
> On Fri, 2026-03-06 at 11:46 -0800, Matthew Schumacher wrote:
>> Hello List,
>>
>> I am building an HA nfsv4 cluster and need to have my locking migrate
>> when a node fails or goes into standby for maint.  There is a lot of
>> documentation around the net, but some of it seems old, like this:
>> https://wiki.linux-nfs.org/wiki/index.php/Nfsd4_server_recovery. I think
>> I have what I need worked out, but I'm still losing locks on recovery.
>> Can someone help me understand how this currently works or point me to
>> the up to date documentation?
>>
>> This is what I'm currently doing:
>>
>> Moving from nodeA to nodeB
>>
>> NodeA: Remove nfsd ip address
>> NodeA: /usr/sbin/exportfs -au
>> NodeA: /usr/sbin/rpc.nfsd 0
>> NodeA: killall --ns $$ -q -v -w rpc.mountd
>> NodeA: killall --ns $$ -q -v -w nfsdcld
>> NodeA: killall --ns $$ -q -v -w rpc.idmapd
>> NodeA: killall --ns $$ -q -v -w rpcbind
>> NodeA: umount /var/lib/nfs/rpc_pipefs
>> NodeA: zpool export (which umounts /var/lib/nfs)
>>
>> NodeB: zpool import (which mounts /var/lib/nfs)
>> NodeB: mount /proc/fs/nfs if not previously mounted
>> NodeB: mount -t rpc_pipefs -o nodev sunrpc /var/lib/nfs/rpc_pipefs
>> NodeB: /sbin/rpcbind -l -w
>> NodeB: /usr/sbin/rpc.idmapd
>> NodeB: /usr/sbin/nfsdcld -d
>> NodeB: /usr/sbin/exportfs -av
>> NodeB: /usr/sbin/rpc.mountd
>> NodeB: /usr/sbin/rpc.nfsd -s -V 4.2 -U -t 8
>> NodeB: start nfsd ip address
>>
>>   From what I understand, nfsdcld writes the lock information to
>> /var/lib/nfs/nfsdcltrack/main.sqlite which is moved to the other side
>> and everything is started, but I still lose locks when I migrate.
>>
>> Mar  6 08:39:14 nfsha user.warn kernel: [37675.787509] NFS:
>> 10.255.255.3: lost 1 locks
>>
>> I think the problem is that kernel (6.12.74) isn't getting the locks
>> reported to it when nfsdcld starts.  Here are some logs:
>>
<snip old logs>
>>
>> I think the problem is that nfsdcld can't pass the locks to the kernel
>> using /var/lib/nfs/rpc_pipefs/nfsd/cld as it's missing.  I'm not sure
>> why, /var/lib/nfs/rpc_pipefs/ has everything else:
>>
<snip old pipefs>
>>
> 
> That's very odd. You might want to look in dmesg and see if there are
> any warnings in there about creating the pipes in rpc_pipefs.
> 


Hello Jeff, others...
I did get nfsdcld to talk to the kernel, but I still lose my sessions 
when I move NFS from one node to another.  I wonder if my understanding 
is correct:

If I have /var/lib/nfs mounted, and nfsdcld running, then it should 
persist my sessions and locks to the main.sqlite database, then when I 
shut down nfsdcld, umount /var/lib/nfs, then mount /var/lib/nfs on a 
different node and start nfsdcld it should tell the kernel about my 
sessions and locks and everything keep working.

Is that correct?

Right now I see my client happily working with a session:

Mar  9 11:39:46 nodeA user.warn kernel: [245803.517797] encode_sequence: 
sessionid=1773078573:732586373:19:0 seqid=1939 slotid=0 max_slotid=0 
cache_this=0

Then when I move the NFS server to another node and assume the same IP 
address, the clients session is reset:

Mar  9 11:45:55 nodeA user.warn kernel: [246172.155392] 
nfs41_sequence_process ERROR: -10052 Reset session
Mar  9 11:45:55 nodeA user.warn kernel: [246172.155394] nfs4_free_slot: 
slotid 0 highest_used_slotid 4294967295
Mar  9 11:45:55 nodeA user.warn kernel: [246172.155396] 
nfs41_sequence_process: Error -10052 free the slot
Mar  9 11:45:55 nodeA user.warn kernel: [246172.155397] 
nfs41_sequence_call_done ERROR -10052
Mar  9 11:45:55 nodeA user.warn kernel: [246172.155397] 
nfs4_schedule_lease_recovery: scheduling lease recovery for server 
10.255.255.3
Mar  9 11:45:55 nodeA user.warn kernel: [246172.155411] 
nfs41_sequence_call_done rpc_cred 000000009fd26b60
Mar  9 11:45:55 nodeA user.warn kernel: [246172.155413] 
nfs4_schedule_state_renewal: requeueing work. Lease period = 5
Mar  9 11:45:55 nodeA user.warn kernel: [246172.155570] NFS: Got error 
-10052 from the server on DESTROY_SESSION. Session has been destroyed 
regardless...
Mar  9 11:45:55 nodeA user.warn kernel: [246172.155572] --> 
nfs4_proc_create_session clp=00000000ea3d2c57 session=00000000348f3ed3
Mar  9 11:45:55 nodeA user.warn kernel: [246172.155574] 
nfs4_init_channel_attrs: Fore Channel : max_rqst_sz=1049620 
max_resp_sz=1049480 max_ops=8 max_reqs=64
Mar  9 11:45:55 nodeA user.warn kernel: [246172.155576] 
nfs4_init_channel_attrs: Back Channel : max_rqst_sz=4096 
max_resp_sz=4096 max_resp_sz_cached=0 max_ops=2 max_reqs=16
Mar  9 11:45:55 nodeA user.warn kernel: [246172.155732] 
nfs4_reset_session: session reset failed with status -10022 for server 
10.255.255.3!
Mar  9 11:45:55 nodeA user.warn kernel: [246172.155734] 
nfs4_handle_reclaim_lease_error: handled error -10022 for server 
10.255.255.3
Mar  9 11:45:55 nodeA user.warn kernel: [246172.155946] 
_nfs4_proc_exchange_id: server_scope mismatch detected
Mar  9 11:45:55 nodeA user.warn kernel: [246172.193390] NFS: 
10.255.255.3: lost 1 locks


On the server side I see:

Mar  9 11:45:05 nodeB daemon.notice rpc.mountd[26917]: Version 2.8.5 
starting
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: 
sqlite_startup_query_grace: current_epoch=6 recovery_epoch=0
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: 
sqlite_check_db_health: returning 0
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: cld_pipe_init: init 
pipe handlers
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: cld_pipe_open: opening 
upcall pipe /var/lib/nfs/rpc_pipefs/nfsd/cld
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: cld_pipe_open: open of 
/var/lib/nfs/rpc_pipefs/nfsd/cld failed: No such file or directory
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: main: Starting event 
dispatch handler.
Mar  9 11:45:05 nodeB user.warn kernel: [246116.696423] set_max_drc 
nfsd_drc_max_mem 259694592
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: cld_inotify_cb: called 
for EV_READ
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: cld_pipe_open: opening 
upcall pipe /var/lib/nfs/rpc_pipefs/nfsd/cld
Mar  9 11:45:05 nodeB user.warn kernel: [246116.697573] nfsd: creating 
service
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: cld_get_version: 
version = 2.
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: Doing downcall with 
status 0
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: cld_gracestart: 
updating grace epochs
Mar  9 11:45:05 nodeB user.warn kernel: [246116.801997] 
alloc_cld_upcall: allocated xid 0
Mar  9 11:45:05 nodeB user.warn kernel: [246116.802105] 
nfsd4_cld_get_version: userspace returned version 2
Mar  9 11:45:05 nodeB user.warn kernel: [246116.802107] 
alloc_cld_upcall: allocated xid 1
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: sqlite_grace_start: 
current_epoch=7 recovery_epoch=6
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: cld_gracestart: 
sending client records to the kernel
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: Sending client Linux 
NFSv4.2 nodeA
Mar  9 11:45:05 nodeB daemon.info nfsdcld[26920]: Doing downcall with 
status 0
Mar  9 11:45:05 nodeB user.info kernel: [246117.076010] NFSD: Using 
nfsdcld client tracking operations.
Mar  9 11:45:05 nodeB user.info kernel: [246117.076012] NFSD: starting 
90-second grace period (net f0000000)
Mar  9 11:45:55 nodeB user.warn kernel: [246167.001121] 
__find_in_sessionid_hashtbl: 1773078573:732586373:19:0
Mar  9 11:45:55 nodeB user.warn kernel: [246167.001124] 
__find_in_sessionid_hashtbl: session not found
Mar  9 11:45:55 nodeB user.warn kernel: [246167.001460] 
nfsd4_destroy_session: 1773078573:732586373:19:0
Mar  9 11:45:55 nodeB user.warn kernel: [246167.001463] 
__find_in_sessionid_hashtbl: 1773078573:732586373:19:0
Mar  9 11:45:55 nodeB user.warn kernel: [246167.001464] 
__find_in_sessionid_hashtbl: session not found
Mar  9 11:45:55 nodeB daemon.notice rpc.mountd[26917]: v4.2 client 
attached: 0x5f2a885f69af1531 from "10.255.255.1:39802"
Mar  9 11:45:55 nodeB user.warn kernel: [246167.001785] 
nfsd4_exchange_id rqstp=00000000c4f6faf6 exid=00000000ac47a836 
clname.len=32 clname.data=00000000c870e87d ip_addr=10.255.255.1 flags 
103, spa_how 0
Mar  9 11:45:55 nodeB user.warn kernel: [246167.001797] 
nfsd4_exchange_id seqid 0 flags 20001
Mar  9 11:45:55 nodeB daemon.info nfsdcld[26920]: cld_create: create 
client record.
Mar  9 11:45:55 nodeB user.warn kernel: [246167.003796] 
__find_in_sessionid_hashtbl: 1773081905:1596622943:23:0
Mar  9 11:45:55 nodeB user.warn kernel: [246167.003798] nfsd4_sequence: 
slotid 0
Mar  9 11:45:55 nodeB user.warn kernel: [246167.004116] found domain 
10.255.255.0/24
Mar  9 11:45:55 nodeB user.warn kernel: [246167.004118] found fsidtype 1
Mar  9 11:45:55 nodeB user.warn kernel: [246167.004119] found fsid length 4
Mar  9 11:45:55 nodeB user.warn kernel: [246167.004120] Path seems to be </>
Mar  9 11:45:55 nodeB user.warn kernel: [246167.004121] Found the path /
Mar  9 11:45:55 nodeB user.warn kernel: [246167.004190] nfsd: 
fh_compose(exp 00:12/1 /, ino=1)
Mar  9 11:45:55 nodeB user.warn kernel: [246167.004209] --> 
nfsd4_store_cache_entry slot 00000000640dc198
Mar  9 11:45:55 nodeB user.warn kernel: [246167.004464] 
__find_in_sessionid_hashtbl: 1773081905:1596622943:23:0
Mar  9 11:45:55 nodeB user.warn kernel: [246167.004466] nfsd4_sequence: 
slotid 0
Mar  9 11:45:55 nodeB user.warn kernel: [246167.004468] 
alloc_cld_upcall: allocated xid 2
Mar  9 11:45:55 nodeB daemon.info nfsdcld[26920]: 
sqlite_insert_client_and_princhash: returning 0
Mar  9 11:45:55 nodeB daemon.info nfsdcld[26920]: Doing downcall with 
status 0
Mar  9 11:45:55 nodeB daemon.info nfsdcld[26920]: cld_gracedone: grace done.

When the session is reset and the lock lost my VM no longer performs I/O 
until it's restarted.

Should my locks and sessions be moving over or am I missing something?

Thanks!

Re: Current state of nfsdv4 recovery.

[Matthew Schumacher]

On 3/9/26 11:57 AM, Matthew Schumacher wrote:
> On 3/6/26 12:49 PM, Jeff Layton wrote:
>> On Fri, 2026-03-06 at 11:46 -0800, Matthew Schumacher wrote:
>>> Hello List,
>>>
>>> I am building an HA nfsv4 cluster and need to have my locking migrate
>>> when a node fails or goes into standby for maint.  There is a lot of
>>> documentation around the net, but some of it seems old, like this:
>>> https://wiki.linux-nfs.org/wiki/index.php/Nfsd4_server_recovery. I 
>>> think
>>> I have what I need worked out, but I'm still losing locks on recovery.
>>> Can someone help me understand how this currently works or point me to
>>> the up to date documentation?
>>>
>>> This is what I'm currently doing:
>>>
>>> Moving from nodeA to nodeB
>>>
>>> NodeA: Remove nfsd ip address
>>> NodeA: /usr/sbin/exportfs -au
>>> NodeA: /usr/sbin/rpc.nfsd 0
>>> NodeA: killall --ns $$ -q -v -w rpc.mountd
>>> NodeA: killall --ns $$ -q -v -w nfsdcld
>>> NodeA: killall --ns $$ -q -v -w rpc.idmapd
>>> NodeA: killall --ns $$ -q -v -w rpcbind
>>> NodeA: umount /var/lib/nfs/rpc_pipefs
>>> NodeA: zpool export (which umounts /var/lib/nfs)
>>>
>>> NodeB: zpool import (which mounts /var/lib/nfs)
>>> NodeB: mount /proc/fs/nfs if not previously mounted
>>> NodeB: mount -t rpc_pipefs -o nodev sunrpc /var/lib/nfs/rpc_pipefs
>>> NodeB: /sbin/rpcbind -l -w
>>> NodeB: /usr/sbin/rpc.idmapd
>>> NodeB: /usr/sbin/nfsdcld -d
>>> NodeB: /usr/sbin/exportfs -av
>>> NodeB: /usr/sbin/rpc.mountd
>>> NodeB: /usr/sbin/rpc.nfsd -s -V 4.2 -U -t 8
>>> NodeB: start nfsd ip address
>>>
>>>   From what I understand, nfsdcld writes the lock information to
>>> /var/lib/nfs/nfsdcltrack/main.sqlite which is moved to the other side
>>> and everything is started, but I still lose locks when I migrate.
>>>
>>> Mar  6 08:39:14 nfsha user.warn kernel: [37675.787509] NFS:
>>> 10.255.255.3: lost 1 locks
>>>
>>> I think the problem is that kernel (6.12.74) isn't getting the locks
>>> reported to it when nfsdcld starts.  Here are some logs:
>>>
> <snip old logs>
>>>
>>> I think the problem is that nfsdcld can't pass the locks to the kernel
>>> using /var/lib/nfs/rpc_pipefs/nfsd/cld as it's missing.  I'm not sure
>>> why, /var/lib/nfs/rpc_pipefs/ has everything else:
>>>
> <snip old pipefs>
>>>
>>
>> That's very odd. You might want to look in dmesg and see if there are
>> any warnings in there about creating the pipes in rpc_pipefs.
>>
>
>
> Hello Jeff, others...
> I did get nfsdcld to talk to the kernel, but I still lose my sessions 
> when I move NFS from one node to another.  I wonder if my 
> understanding is correct:
>
> If I have /var/lib/nfs mounted, and nfsdcld running, then it should 
> persist my sessions and locks to the main.sqlite database, then when I 
> shut down nfsdcld, umount /var/lib/nfs, then mount /var/lib/nfs on a 
> different node and start nfsdcld it should tell the kernel about my 
> sessions and locks and everything keep working.
>
> Is that correct?
>
> Right now I see my client happily working with a session:
>
> Mar  9 11:39:46 nodeA user.warn kernel: [245803.517797] 
> encode_sequence: sessionid=1773078573:732586373:19:0 seqid=1939 
> slotid=0 max_slotid=0 cache_this=0
>
> Then when I move the NFS server to another node and assume the same IP 
> address, the clients session is reset:
>
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.155392] 
> nfs41_sequence_process ERROR: -10052 Reset session
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.155394] 
> nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.155396] 
> nfs41_sequence_process: Error -10052 free the slot
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.155397] 
> nfs41_sequence_call_done ERROR -10052
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.155397] 
> nfs4_schedule_lease_recovery: scheduling lease recovery for server 
> 10.255.255.3
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.155411] 
> nfs41_sequence_call_done rpc_cred 000000009fd26b60
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.155413] 
> nfs4_schedule_state_renewal: requeueing work. Lease period = 5
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.155570] NFS: Got error 
> -10052 from the server on DESTROY_SESSION. Session has been destroyed 
> regardless...
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.155572] --> 
> nfs4_proc_create_session clp=00000000ea3d2c57 session=00000000348f3ed3
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.155574] 
> nfs4_init_channel_attrs: Fore Channel : max_rqst_sz=1049620 
> max_resp_sz=1049480 max_ops=8 max_reqs=64
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.155576] 
> nfs4_init_channel_attrs: Back Channel : max_rqst_sz=4096 
> max_resp_sz=4096 max_resp_sz_cached=0 max_ops=2 max_reqs=16
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.155732] 
> nfs4_reset_session: session reset failed with status -10022 for server 
> 10.255.255.3!
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.155734] 
> nfs4_handle_reclaim_lease_error: handled error -10022 for server 
> 10.255.255.3
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.155946] 
> _nfs4_proc_exchange_id: server_scope mismatch detected
> Mar  9 11:45:55 nodeA user.warn kernel: [246172.193390] NFS: 
> 10.255.255.3: lost 1 locks
>
<snip>

Replying to the list...

It appears my assumption is correct because now I see my locks and 
sessions maintained when moving from nodeA to nodeB... the issue was this:

_nfs4_proc_exchange_id: server_scope mismatch detected

The scope must be the same on all nodes in a cluster and defaults to the 
node hostname.

My cluster is happy doing nfsv4 now.