[syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes

[syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    50c8770a42fa Add linux-next specific files for 20250702
git tree:       linux-next
console+strace: https://syzkaller.appspot.com/x/log.txt?x=162e7982580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=70c16e4e191115d4
dashboard link: https://syzkaller.appspot.com/bug?extid=169de184e9defe7fe709
compiler:       Debian clang version 20.1.6 (++20250514063057+1e4d39e07757-1~exp1~20250514183223.118), Debian LLD 20.1.6
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=1247d770580000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=17ffe48c580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/3d4ef6bedc5b/disk-50c8770a.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/15b7565dc0ef/vmlinux-50c8770a.xz
kernel image: https://storage.googleapis.com/syzbot-assets/3b397342a62b/bzImage-50c8770a.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+169de184e9defe7fe709@syzkaller.appspotmail.com

============================================
WARNING: possible recursive locking detected
6.16.0-rc4-next-20250702-syzkaller #0 Not tainted
--------------------------------------------
syz-executor309/5837 is trying to acquire lock:
ffff88807f5bc8c8 (&sb->s_type->i_mutex_key#15){+.+.}-{4:4}, at: inode_lock include/linux/fs.h:869 [inline]
ffff88807f5bc8c8 (&sb->s_type->i_mutex_key#15){+.+.}-{4:4}, at: rpc_close_pipes+0x10a/0x730 net/sunrpc/rpc_pipe.c:178

but task is already holding lock:
ffff88807f5b91c8 (&sb->s_type->i_mutex_key#15){+.+.}-{4:4}, at: inode_lock include/linux/fs.h:869 [inline]
ffff88807f5b91c8 (&sb->s_type->i_mutex_key#15){+.+.}-{4:4}, at: __simple_recursive_removal+0x190/0x510 fs/libfs.c:627

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(&sb->s_type->i_mutex_key#15);
  lock(&sb->s_type->i_mutex_key#15);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

4 locks held by syz-executor309/5837:
 #0: ffff888033ee40e0 (&type->s_umount_key#42){+.+.}-{4:4}, at: __super_lock fs/super.c:57 [inline]
 #0: ffff888033ee40e0 (&type->s_umount_key#42){+.+.}-{4:4}, at: __super_lock_excl fs/super.c:72 [inline]
 #0: ffff888033ee40e0 (&type->s_umount_key#42){+.+.}-{4:4}, at: deactivate_super+0xa9/0xe0 fs/super.c:506
 #1: ffff8881446f60a0 (&sn->pipefs_sb_lock){+.+.}-{4:4}, at: rpc_kill_sb+0x77/0x190 net/sunrpc/rpc_pipe.c:1196
 #2: ffffffff8f8e2df0 ((rpc_pipefs_notifier_list).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x54/0x90 kernel/notifier.c:379
 #3: ffff88807f5b91c8 (&sb->s_type->i_mutex_key#15){+.+.}-{4:4}, at: inode_lock include/linux/fs.h:869 [inline]
 #3: ffff88807f5b91c8 (&sb->s_type->i_mutex_key#15){+.+.}-{4:4}, at: __simple_recursive_removal+0x190/0x510 fs/libfs.c:627

stack backtrace:
CPU: 0 UID: 0 PID: 5837 Comm: syz-executor309 Not tainted 6.16.0-rc4-next-20250702-syzkaller #0 PREEMPT(full) 
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/07/2025
Call Trace:
 <TASK>
 dump_stack_lvl+0x189/0x250 lib/dump_stack.c:120
 print_deadlock_bug+0x28b/0x2a0 kernel/locking/lockdep.c:3044
 check_deadlock kernel/locking/lockdep.c:3096 [inline]
 validate_chain+0x1a3f/0x2140 kernel/locking/lockdep.c:3898
 __lock_acquire+0xab9/0xd20 kernel/locking/lockdep.c:5240
 lock_acquire+0x120/0x360 kernel/locking/lockdep.c:5871
 down_write+0x96/0x1f0 kernel/locking/rwsem.c:1577
 inode_lock include/linux/fs.h:869 [inline]
 rpc_close_pipes+0x10a/0x730 net/sunrpc/rpc_pipe.c:178
 __simple_recursive_removal+0x208/0x510 fs/libfs.c:631
 rpc_unlink+0x56/0x80 net/sunrpc/rpc_pipe.c:696
 rpc_pipefs_event+0xc0/0x170 fs/nfs/blocklayout/rpc_pipefs.c:179
 notifier_call_chain+0x1b3/0x3e0 kernel/notifier.c:85
 blocking_notifier_call_chain+0x6a/0x90 kernel/notifier.c:380
 rpc_kill_sb+0xd0/0x190 net/sunrpc/rpc_pipe.c:1204
 deactivate_locked_super+0xb9/0x130 fs/super.c:474
 cleanup_mnt+0x425/0x4c0 fs/namespace.c:1417
 task_work_run+0x1d1/0x260 kernel/task_work.c:227
 ptrace_notify+0x281/0x2c0 kernel/signal.c:2520
 ptrace_report_syscall include/linux/ptrace.h:415 [inline]
 ptrace_report_syscall_exit include/linux/ptrace.h:477 [inline]
 syscall_exit_work+0xc6/0x1d0 kernel/entry/syscall-common.c:111
 syscall_exit_to_user_mode_work include/linux/entry-common.h:173 [inline]
 syscall_exit_to_user_mode include/linux/entry-common.h:210 [inline]
 do_syscall_64+0x2ad/0x3b0 arch/x86/entry/syscall_64.c:100
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f8f7dbc82e9
Code: 48 83 c4 28 c3 e8 37 17 00 00 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffe287838a8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
RAX: ffffffffffffffec RBX: 0000200000000000 RCX: 00007f8f7dbc82e9


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

Re: [syzbot] Re: [syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes

[syzbot]

For archival purposes, forwarding an incoming command email to
linux-kernel@vger.kernel.org.

***

Subject: Re: [syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes
Author: lizhi.xu@windriver.com

#syz test

diff --git a/net/sunrpc/rpc_pipe.c b/net/sunrpc/rpc_pipe.c
index 0bd1df2ebb47..ae5643e0bc43 100644
--- a/net/sunrpc/rpc_pipe.c
+++ b/net/sunrpc/rpc_pipe.c
@@ -693,7 +693,7 @@ void
 rpc_unlink(struct rpc_pipe *pipe)
 {
 	if (pipe->dentry) {
-		simple_recursive_removal(pipe->dentry, rpc_close_pipes);
+		locked_recursive_removal(pipe->dentry, rpc_close_pipes);
 		pipe->dentry = NULL;
 	}
 }

Re: [syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes

[Hillf Danton]

> Date: Wed, 02 Jul 2025 10:41:32 -0700
> syzbot found the following issue on:
> 
> HEAD commit:    50c8770a42fa Add linux-next specific files for 20250702
> git tree:       linux-next
> console+strace: https://syzkaller.appspot.com/x/log.txt?x=162e7982580000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=70c16e4e191115d4
> dashboard link: https://syzkaller.appspot.com/bug?extid=169de184e9defe7fe709
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=1247d770580000
> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=17ffe48c580000

#syz test

--- x/net/sunrpc/rpc_pipe.c
+++ y/net/sunrpc/rpc_pipe.c
@@ -175,7 +175,7 @@ rpc_close_pipes(struct dentry *dentry)
 	int need_release;
 	LIST_HEAD(free_list);
 
-	inode_lock(inode);
+	inode_lock_nested(inode, 1);
 	spin_lock(&pipe->lock);
 	need_release = pipe->nreaders != 0 || pipe->nwriters != 0;
 	pipe->nreaders = 0;
--

Re: [syzbot] Re: [syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes

[Al Viro]

On Wed, Jul 02, 2025 at 06:17:26PM -0700, syzbot wrote:
> For archival purposes, forwarding an incoming command email to
> linux-kernel@vger.kernel.org.
> 
> ***
> 
> Subject: Re: [syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes
> Author: lizhi.xu@windriver.com
> 
> #syz test
> 
> diff --git a/net/sunrpc/rpc_pipe.c b/net/sunrpc/rpc_pipe.c
> index 0bd1df2ebb47..ae5643e0bc43 100644
> --- a/net/sunrpc/rpc_pipe.c
> +++ b/net/sunrpc/rpc_pipe.c
> @@ -693,7 +693,7 @@ void
>  rpc_unlink(struct rpc_pipe *pipe)
>  {
>  	if (pipe->dentry) {
> -		simple_recursive_removal(pipe->dentry, rpc_close_pipes);
> +		locked_recursive_removal(pipe->dentry, rpc_close_pipes);
>  		pipe->dentry = NULL;
>  	}
>  }

Excuse me, but... which caller of rpc_unlink() is holding any directories locked?
IOW, this patch is an LLM-grade nonsense.  If it really *is* chatbot-generated,
that's a lovely demonstration of the reasons why generative AI have no business
sending patches of any kind.

Note that report clearly refers to rpc_close_pipes() as one of the locations
involved in a possible deadlock.  The difference between simple_recursive_removal()
and locked_recursive_removal() is that the latter is to be called when the caller
is already holding ->i_rwsem on the victim's parent.  *IF* that was the case,
the deadlock report would point to that caller vs. simple_recursive_removal().

rpc_close_pipes() is a *callback* passed to simple_recursive_removal(); if it
turned around and called rpc_unlink() we would have a serious problem indeed,
and it's very easy to see that nothing of that sort is happening.

The worst part is, that patch is likely to make lockdep STFU - by failing to
lock the parent.  You really need to reason; "throw random shit at the bot
until the warning goes away" is an actively harmful strategy.

As for the warning, it is a false positive caused by lockdep annotations, as
the original report suggested.  Replace inode_lock(inode) in rpc_close_pipes()
with inode_lock_nested(inode, I_MUTEX_CHILD) and try to reproduce that.

For syzbot maintainers - git blame is useful.  I would really appreciate the
original report...

Re: [syzbot] Re: [syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes

[Al Viro]

On Thu, Jul 03, 2025 at 02:50:33AM +0100, Al Viro wrote:

> As for the warning, it is a false positive caused by lockdep annotations, as
> the original report suggested.  Replace inode_lock(inode) in rpc_close_pipes()
> with inode_lock_nested(inode, I_MUTEX_CHILD) and try to reproduce that.

Nope...  I_MUTEX_CHILD is outside of I_MUTEX_NORMAL, not the other way round.

OK, so the right annotations would be I_MUTEX_CHILD in __simple_recursive_removal(),
with I_MUTEX_NORMAL in rpc_close_pipes() and, for callers of locked_recursive_removal(),
I_MUTEX_PARENT.

The nesting order is PARENT, then CHILD, then NORMAL.  Sure, we could leave
annotations in simple_recursive_removal() as-is and switch rpc_close_pipes() to
e.g. I_MUTEX_XATTR, but that's too brittle for words.

The ordering is "ancestors before descendents", and it is satisfied here; the
question is how to map it to lockdep classes ;-/

Re: [syzbot] Re: [syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes

[syzbot]

For archival purposes, forwarding an incoming command email to
linux-kernel@vger.kernel.org.

***

Subject: Re: [syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes
Author: lizhi.xu@windriver.com

#syz test

diff --git a/fs/libfs.c b/fs/libfs.c
index 3bc6c3750b47..0e3e33c4f159 100644
--- a/fs/libfs.c
+++ b/fs/libfs.c
@@ -611,7 +611,7 @@ static void __simple_recursive_removal(struct dentry *dentry,
 		struct dentry *victim = NULL, *child;
 		struct inode *inode = this->d_inode;
 
-		inode_lock(inode);
+		inode_lock_nested(inode, I_MUTEX_CHILD);
 		if (d_is_dir(this))
 			inode->i_flags |= S_DEAD;
 		while ((child = find_next_child(this, victim)) == NULL) {
@@ -624,7 +624,7 @@ static void __simple_recursive_removal(struct dentry *dentry,
 			this = this->d_parent;
 			inode = this->d_inode;
 			if (!locked || victim != dentry)
-				inode_lock(inode);
+				inode_lock_nested(inode, I_MUTEX_CHILD);
 			if (simple_positive(victim)) {
 				d_invalidate(victim);	// avoid lost mounts
 				if (callback)

Re: [syzbot] Re: [syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes

[Al Viro]

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.git work.rpc_pipe

Re: [syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-by: syzbot+169de184e9defe7fe709@syzkaller.appspotmail.com
Tested-by: syzbot+169de184e9defe7fe709@syzkaller.appspotmail.com

Tested on:

commit:         50c8770a Add linux-next specific files for 20250702
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=1791748c580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=76d012e863976d4c
dashboard link: https://syzkaller.appspot.com/bug?extid=169de184e9defe7fe709
compiler:       Debian clang version 20.1.7 (++20250616065708+6146a88f6049-1~exp1~20250616065826.132), Debian LLD 20.1.7
patch:          https://syzkaller.appspot.com/x/patch.diff?x=151db48c580000

Note: testing is done by a robot and is best-effort only.

Re: [syzbot] Re: [syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes

[syzbot]

For archival purposes, forwarding an incoming command email to
linux-kernel@vger.kernel.org.

***

Subject: Re: [syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes
Author: lizhi.xu@windriver.com

#syz test

diff --git a/net/sunrpc/rpc_pipe.c b/net/sunrpc/rpc_pipe.c
index 0bd1df2ebb47..e30270e5883a 100644
--- a/net/sunrpc/rpc_pipe.c
+++ b/net/sunrpc/rpc_pipe.c
@@ -175,7 +175,7 @@ rpc_close_pipes(struct dentry *dentry)
 	int need_release;
 	LIST_HEAD(free_list);
 
-	inode_lock(inode);
+	inode_lock_nested(inode, I_MUTEX_CHILD);
 	spin_lock(&pipe->lock);
 	need_release = pipe->nreaders != 0 || pipe->nwriters != 0;
 	pipe->nreaders = 0;

Re: [syzbot] [nfs?] [net?] possible deadlock in rpc_close_pipes

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-by: syzbot+169de184e9defe7fe709@syzkaller.appspotmail.com
Tested-by: syzbot+169de184e9defe7fe709@syzkaller.appspotmail.com

Tested on:

commit:         350db61f rpc_create_client_dir(): return 0 or -E...
git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.git work.rpc_pipe
console output: https://syzkaller.appspot.com/x/log.txt?x=15970c8c580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=af650e21e0ca5e40
dashboard link: https://syzkaller.appspot.com/bug?extid=169de184e9defe7fe709
compiler:       Debian clang version 20.1.7 (++20250616065708+6146a88f6049-1~exp1~20250616065826.132), Debian LLD 20.1.7

Note: no patches were applied.
Note: testing is done by a robot and is best-effort only.