[bug report] smc: possible deadlock in smc_switch_to_fallback

[bug report] smc: possible deadlock in smc_switch_to_fallback

[江世昊]

Hi developers:

We may found a bug in smc module which can lead to deadlock recently.

HEAD commit: 819837584309 6.12.0-rc5
git tree: upstream
kernel config: https://drive.google.com/file/d/1-9pltE-1CMgGgNFu9o5l0BlCHk3Rnzb_/view?usp=sharing
console output: https://drive.google.com/file/d/1uH0RDdftVIq_Boa6YWowLj3WujqVBuL7/view?usp=sharing
syz repro: https://drive.google.com/file/d/1WUjiYSMebSZ7fR0uxrGx-kDLrf1v_Nra/view?usp=sharing
C reproducer: https://drive.google.com/file/d/1_lB5_uacVnfDNE8VpuiY-NxEtDdzJ0Z8/view?usp=sharing

Environment:
Ubuntu 22.04 on Linux 5.15
QEMU emulator version 6.2.0
qemu-system-x86_64 \
-m 2G \
-smp 2 \
-kernel /home/wd/bzImage \
-append "console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=/home/wd/bullseye.img,format=raw \
-net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 \
-net nic,model=e1000 \
-enable-kvm \
-nographic \
-pidfile vm.pid \
2>&1 | tee vm.log

If you fix this issue, please add the following tag to the commit:
Reported-by: Shihao Jiang<sh.jiang@zju.edu.cn>

======================================================
WARNING: possible circular locking dependency detected
6.12.0-rc5 #1 Tainted: G        W         
------------------------------------------------------
syz-executor351/9413 is trying to acquire lock:
ffff88804bd68aa8 (&smc->clcsock_release_lock){+.+.}-{3:3}, at: smc_switch_to_fallback+0x2d/0xa00 net/smc/af_smc.c:902

but task is already holding lock:
ffff88804bd68258 (sk_lock-AF_INET6){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1611 [inline]
ffff88804bd68258 (sk_lock-AF_INET6){+.+.}-{0:0}, at: smc_setsockopt+0x323/0xc10 net/smc/af_smc.c:3077

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (sk_lock-AF_INET6){+.+.}-{0:0}:
       lock_sock_nested+0x3a/0xf0 net/core/sock.c:3611
       lock_sock include/net/sock.h:1611 [inline]
       sockopt_lock_sock+0x54/0x70 net/core/sock.c:1125
       do_ipv6_setsockopt+0x2509/0x48b0 net/ipv6/ipv6_sockglue.c:567
       ipv6_setsockopt+0xcb/0x170 net/ipv6/ipv6_sockglue.c:993
       rawv6_setsockopt+0xd7/0x700 net/ipv6/raw.c:1054
       do_sock_setsockopt+0x227/0x480 net/socket.c:2334
       __sys_setsockopt+0x1a6/0x270 net/socket.c:2357
       __do_sys_setsockopt net/socket.c:2366 [inline]
       __se_sys_setsockopt net/socket.c:2363 [inline]
       __x64_sys_setsockopt+0xbd/0x160 net/socket.c:2363
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xcb/0x250 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #1 (rtnl_mutex){+.+.}-{3:3}:
       __mutex_lock_common kernel/locking/mutex.c:608 [inline]
       __mutex_lock+0x147/0x930 kernel/locking/mutex.c:752
       do_ip_setsockopt+0x18a1/0x3780 net/ipv4/ip_sockglue.c:1077
       ip_setsockopt+0x59/0xf0 net/ipv4/ip_sockglue.c:1417
       tcp_setsockopt+0xa9/0x110 net/ipv4/tcp.c:4029
       smc_setsockopt+0x1b1/0xc10 net/smc/af_smc.c:3064
       do_sock_setsockopt+0x227/0x480 net/socket.c:2334
       __sys_setsockopt+0x1a6/0x270 net/socket.c:2357
       __do_sys_setsockopt net/socket.c:2366 [inline]
       __se_sys_setsockopt net/socket.c:2363 [inline]
       __x64_sys_setsockopt+0xbd/0x160 net/socket.c:2363
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xcb/0x250 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #0 (&smc->clcsock_release_lock){+.+.}-{3:3}:
       check_prev_add kernel/locking/lockdep.c:3161 [inline]
       check_prevs_add kernel/locking/lockdep.c:3280 [inline]
       validate_chain kernel/locking/lockdep.c:3904 [inline]
       __lock_acquire+0x2425/0x3b90 kernel/locking/lockdep.c:5202
       lock_acquire.part.0+0x11b/0x370 kernel/locking/lockdep.c:5825
       __mutex_lock_common kernel/locking/mutex.c:608 [inline]
       __mutex_lock+0x147/0x930 kernel/locking/mutex.c:752
       smc_switch_to_fallback+0x2d/0xa00 net/smc/af_smc.c:902
       smc_setsockopt+0x809/0xc10 net/smc/af_smc.c:3087
       do_sock_setsockopt+0x227/0x480 net/socket.c:2334
       __sys_setsockopt+0x1a6/0x270 net/socket.c:2357
       __do_sys_setsockopt net/socket.c:2366 [inline]
       __se_sys_setsockopt net/socket.c:2363 [inline]
       __x64_sys_setsockopt+0xbd/0x160 net/socket.c:2363
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xcb/0x250 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

other info that might help us debug this:

Chain exists of:
  &smc->clcsock_release_lock --> rtnl_mutex --> sk_lock-AF_INET6

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(sk_lock-AF_INET6);
                               lock(rtnl_mutex);
                               lock(sk_lock-AF_INET6);
  lock(&smc->clcsock_release_lock);

 *** DEADLOCK ***

1 lock held by syz-executor351/9413:
 #0: ffff88804bd68258 (sk_lock-AF_INET6){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1611 [inline]
 #0: ffff88804bd68258 (sk_lock-AF_INET6){+.+.}-{0:0}, at: smc_setsockopt+0x323/0xc10 net/smc/af_smc.c:3077

stack backtrace:
CPU: 1 UID: 0 PID: 9413 Comm: syz-executor351 Tainted: G        W          6.12.0-rc5 #1
Tainted: [W]=WARN
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:94 [inline]
 dump_stack_lvl+0x116/0x1b0 lib/dump_stack.c:120
 print_circular_bug+0x406/0x5c0 kernel/locking/lockdep.c:2074
 check_noncircular+0x2f7/0x3e0 kernel/locking/lockdep.c:2206
 check_prev_add kernel/locking/lockdep.c:3161 [inline]
 check_prevs_add kernel/locking/lockdep.c:3280 [inline]
 validate_chain kernel/locking/lockdep.c:3904 [inline]
 __lock_acquire+0x2425/0x3b90 kernel/locking/lockdep.c:5202
 lock_acquire.part.0+0x11b/0x370 kernel/locking/lockdep.c:5825
 __mutex_lock_common kernel/locking/mutex.c:608 [inline]
 __mutex_lock+0x147/0x930 kernel/locking/mutex.c:752
 smc_switch_to_fallback+0x2d/0xa00 net/smc/af_smc.c:902
 smc_setsockopt+0x809/0xc10 net/smc/af_smc.c:3087
 do_sock_setsockopt+0x227/0x480 net/socket.c:2334
 __sys_setsockopt+0x1a6/0x270 net/socket.c:2357
 __do_sys_setsockopt net/socket.c:2366 [inline]
 __se_sys_setsockopt net/socket.c:2363 [inline]
 __x64_sys_setsockopt+0xbd/0x160 net/socket.c:2363
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xcb/0x250 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f8d96fc304d
Code: 28 c3 e8 46 1e 00 00 66 0f 1f 44 00 00 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffef30e0218 EFLAGS: 00000246 ORIG_RAX: 0000000000000036
RAX: ffffffffffffffda RBX: 00007ffef30e0418 RCX: 00007f8d96fc304d
RDX: 0000000000000017 RSI: 0000000000000029 RDI: 0000000000000005
RBP: 0000000000000001 R08: 0000000000000004 R09: 00007ffef30e0418
R10: 00000000200000c0 R11: 0000000000000246 R12: 0000000000000001
R13: 00007ffef30e0408 R14: 00007f8d97040530 R15: 0000000000000001
 </TASK>

======================================================

Best Regards, 
Shihao

Re: [bug report] smc: possible deadlock in smc_switch_to_fallback

[Greg KH]

On Thu, Nov 28, 2024 at 05:12:23PM +0800, 江世昊 wrote:
> Hi developers:
> 
> We may found a bug in smc module which can lead to deadlock recently.
> 
> HEAD commit: 819837584309 6.12.0-rc5
> git tree: upstream
> kernel config: https://drive.google.com/file/d/1-9pltE-1CMgGgNFu9o5l0BlCHk3Rnzb_/view?usp=sharing
> console output: https://drive.google.com/file/d/1uH0RDdftVIq_Boa6YWowLj3WujqVBuL7/view?usp=sharing
> syz repro: https://drive.google.com/file/d/1WUjiYSMebSZ7fR0uxrGx-kDLrf1v_Nra/view?usp=sharing
> C reproducer: https://drive.google.com/file/d/1_lB5_uacVnfDNE8VpuiY-NxEtDdzJ0Z8/view?usp=sharing
> 
> Environment:
> Ubuntu 22.04 on Linux 5.15
> QEMU emulator version 6.2.0
> qemu-system-x86_64 \
> -m 2G \
> -smp 2 \
> -kernel /home/wd/bzImage \
> -append "console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
> -drive file=/home/wd/bullseye.img,format=raw \
> -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 \
> -net nic,model=e1000 \
> -enable-kvm \
> -nographic \
> -pidfile vm.pid \
> 2>&1 | tee vm.log
> 
> If you fix this issue, please add the following tag to the commit:
> Reported-by: Shihao Jiang<sh.jiang@zju.edu.cn>
> 
> ======================================================
> WARNING: possible circular locking dependency detected
> 6.12.0-rc5 #1 Tainted: G        W         
> ------------------------------------------------------
> syz-executor351/9413 is trying to acquire lock:
> ffff88804bd68aa8 (&smc->clcsock_release_lock){+.+.}-{3:3}, at: smc_switch_to_fallback+0x2d/0xa00 net/smc/af_smc.c:902
> 
> but task is already holding lock:
> ffff88804bd68258 (sk_lock-AF_INET6){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1611 [inline]
> ffff88804bd68258 (sk_lock-AF_INET6){+.+.}-{0:0}, at: smc_setsockopt+0x323/0xc10 net/smc/af_smc.c:3077
> 
> which lock already depends on the new lock.

Please submit a fix for this issue as you have a test-case for it.  It's
just a "normal" lockdep warning, not a "security" issue.

thanks,

greg k-h