[PATCH] nfslocalio: use correct wait address in nfs_uuid_put()

[PATCH] nfslocalio: use correct wait address in nfs_uuid_put()

[NeilBrown]

This wait_var_event_spinlock() in nfs_uuid_put() is waiting for the
wakeup signalled at the end of nfs_close_local_fh().  That
wake_up_var_locked() uses &nfl->nfs_uuid, so the waiter must use the
same address, else nfs_uuid_put() could wait indefinitely causing
various problems.

Fixes: 21fb44034695 ("nfs_localio: protect race between nfs_uuid_put() and nfs_close_local_fh()")
Reported-by: Mike Snitzer <snitzer@kernel.org>
Signed-off-by: NeilBrown <neil@brown.name>
---
 fs/nfs_common/nfslocalio.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/nfs_common/nfslocalio.c b/fs/nfs_common/nfslocalio.c
index 05c7c16e37ab..bc8dcfb256a3 100644
--- a/fs/nfs_common/nfslocalio.c
+++ b/fs/nfs_common/nfslocalio.c
@@ -177,7 +177,7 @@ static bool nfs_uuid_put(nfs_uuid_t *nfs_uuid)
 			/* nfs_close_local_fh() is doing the
 			 * close and we must wait. until it unlinks
 			 */
-			wait_var_event_spinlock(nfl,
+			wait_var_event_spinlock(&nfl->nfs_uuid,
 						list_first_entry_or_null(
 							&nfs_uuid->files,
 							struct nfs_file_localio,
-- 
2.49.0

Re: nfslocalio: use correct wait address in nfs_uuid_put()

[Mike Snitzer]

On Mon, Jul 14, 2025 at 01:56:13PM +1000, NeilBrown wrote:
> 
> This wait_var_event_spinlock() in nfs_uuid_put() is waiting for the
> wakeup signalled at the end of nfs_close_local_fh().  That
> wake_up_var_locked() uses &nfl->nfs_uuid, so the waiter must use the
> same address, else nfs_uuid_put() could wait indefinitely causing
> various problems.
> 
> Fixes: 21fb44034695 ("nfs_localio: protect race between nfs_uuid_put() and nfs_close_local_fh()")
> Reported-by: Mike Snitzer <snitzer@kernel.org>
> Signed-off-by: NeilBrown <neil@brown.name>
> ---
>  fs/nfs_common/nfslocalio.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/fs/nfs_common/nfslocalio.c b/fs/nfs_common/nfslocalio.c
> index 05c7c16e37ab..bc8dcfb256a3 100644
> --- a/fs/nfs_common/nfslocalio.c
> +++ b/fs/nfs_common/nfslocalio.c
> @@ -177,7 +177,7 @@ static bool nfs_uuid_put(nfs_uuid_t *nfs_uuid)
>  			/* nfs_close_local_fh() is doing the
>  			 * close and we must wait. until it unlinks
>  			 */
> -			wait_var_event_spinlock(nfl,
> +			wait_var_event_spinlock(&nfl->nfs_uuid,
>  						list_first_entry_or_null(
>  							&nfs_uuid->files,
>  							struct nfs_file_localio,
> -- 
> 2.49.0
> 
> 

Makes sense:

Acked-by: Mike Snitzer <snitzer@kernel.org>

And I _will_ try to get this tested at the scale I was able to test
late last week.  Will let you and others know (hopefully within the
next 24h).

Mike

Re: nfslocalio: use correct wait address in nfs_uuid_put()

[Mike Snitzer]

On Mon, Jul 14, 2025 at 10:31:43AM -0400, Mike Snitzer wrote:
> On Mon, Jul 14, 2025 at 01:56:13PM +1000, NeilBrown wrote:
> > 
> > This wait_var_event_spinlock() in nfs_uuid_put() is waiting for the
> > wakeup signalled at the end of nfs_close_local_fh().  That
> > wake_up_var_locked() uses &nfl->nfs_uuid, so the waiter must use the
> > same address, else nfs_uuid_put() could wait indefinitely causing
> > various problems.
> > 
> > Fixes: 21fb44034695 ("nfs_localio: protect race between nfs_uuid_put() and nfs_close_local_fh()")
> > Reported-by: Mike Snitzer <snitzer@kernel.org>
> > Signed-off-by: NeilBrown <neil@brown.name>
> > ---
> >  fs/nfs_common/nfslocalio.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/fs/nfs_common/nfslocalio.c b/fs/nfs_common/nfslocalio.c
> > index 05c7c16e37ab..bc8dcfb256a3 100644
> > --- a/fs/nfs_common/nfslocalio.c
> > +++ b/fs/nfs_common/nfslocalio.c
> > @@ -177,7 +177,7 @@ static bool nfs_uuid_put(nfs_uuid_t *nfs_uuid)
> >  			/* nfs_close_local_fh() is doing the
> >  			 * close and we must wait. until it unlinks
> >  			 */
> > -			wait_var_event_spinlock(nfl,
> > +			wait_var_event_spinlock(&nfl->nfs_uuid,
> >  						list_first_entry_or_null(
> >  							&nfs_uuid->files,
> >  							struct nfs_file_localio,
> > -- 
> > 2.49.0
> > 
> > 
> 
> Makes sense:
> 
> Acked-by: Mike Snitzer <snitzer@kernel.org>
> 
> And I _will_ try to get this tested at the scale I was able to test
> late last week.  Will let you and others know (hopefully within the
> next 24h).

Unfortunately this fix doesn't resolve the hang I easily experience
simply by running fio like I reported here (followed by attempted NFSD
shutdown):
https://lore.kernel.org/linux-nfs/aG0pJXVtApZ9C5vy@kernel.org/

After fio runs, 'systemctl stop proc-fs-nfsd.mount' hangs:

[  737.997724] INFO: task rpc.nfsd:9947 blocked for more than 122 seconds.
[  737.997984]       Not tainted 6.12.24.9.hs.snitm+ #16
[  737.998155] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  737.998304] task:rpc.nfsd        state:D stack:0     pid:9947  tgid:9947  ppid:1      flags:0x00004006
[  737.998453] Call Trace:
[  737.998597]  <TASK>
[  737.998781]  __schedule+0x26d/0x530
[  737.998925]  schedule+0x27/0xa0
[  737.999057]  schedule_timeout+0x14e/0x160
[  737.999193]  ? svc_destroy+0xce/0x160 [sunrpc]
[  737.999378]  ? lockd_put+0x5f/0x90 [lockd]
[  737.999518]  __wait_for_common+0x8f/0x1d0
[  737.999667]  ? __pfx_schedule_timeout+0x10/0x10
[  737.999801]  nfsd_destroy_serv+0x13f/0x1a0 [nfsd]
[  737.999976]  nfsd_svc+0xe0/0x170 [nfsd]
[  738.000136]  write_threads+0xc3/0x190 [nfsd]
[  738.000293]  ? simple_transaction_get+0xc2/0xe0
[  738.000421]  ? __pfx_write_threads+0x10/0x10 [nfsd]
[  738.000578]  nfsctl_transaction_write+0x47/0x80 [nfsd]
[  738.000780]  vfs_write+0xfa/0x420
[  738.000906]  ksys_write+0x63/0xe0
[  738.001030]  do_syscall_64+0x7d/0x160
[  738.001152]  ? do_user_addr_fault+0x341/0x6b0
[  738.001278]  ? exc_page_fault+0x70/0x160
[  738.001400]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  738.001526] RIP: 0033:0x7fd836efd617
[  738.001661] RSP: 002b:00007ffee6d1ff18 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[  738.001788] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fd836efd617
[  738.001914] RDX: 0000000000000002 RSI: 0000558592fc1c20 RDI: 0000000000000003
[  738.002041] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007ffee6d1fdb0
[  738.002169] R10: 0000000000000000 R11: 0000000000000246 R12: 0000558592fc1c20
[  738.002298] R13: 00007fd83711a6c8 R14: 00007ffee6d1ffe0 R15: 00000000ffffffff
[  738.002429]  </TASK>

So I stand by my _strong_ suggestion to revert your LOCALIO changes
in time for 6.16-rc7 (changes that were merged for 6.16), series here:
https://lore.kernel.org/linux-nfs/20250714031359.10192-1-snitzer@kernel.org/

At this point they'd even be difficult to justify landing during the
6.17 merge window let alone 6.16 final.

Just the messenger here.. sorry.

Mike