Re: Stalls when starting a VSOCK listening socket: soft lockups, RCU stalls, timeout

Matthieu Baerts
2026-03-06T21:40:47Z

06 Mar 2026 17:57:26 Matthieu Baerts <matttbe@kernel.org>:

(...)

> After having sent this email, I re-checked on my side, and I was able to
> reproduce this issue with the technique described above: using the
> docker image with "build" argument, then max 50 boot iterations with "vm
> auto normal" argument. I then used 'git bisect' between v6.18 and
> v6.19-rc1 to find the guilty commit, and got:
>
>   653fda7ae73d ("sched/mmcid: Switch over to the new mechanism")
>
> Reverting it on top of v6.19-rc1 fixes the issue.
>
> Unfortunatelly, reverting it on top of Linus' tree causes some
> conflicts. I did my best to resolve them, and with this patch attached
> below -- also available in [1] -- I no longer have the issue. I don't
> know if it is correct -- some quick tests don't show any issues -- nor
> if Jiri should test it. I guess the final fix will be different from
> this simple revert.

As probably expected, even if this revert fixed the boot issues, it
caused a regression during the tests execution, see below.

[  493.608357][    C3] rcu: 3-....: (26000 ticks this GP) idle=d54c/1/0x4000000000000000 softirq=214151/214154 fqs=6500
[  493.609867][    C3] rcu: (t=26003 jiffies g=392697 q=1127 ncpus=4)
[  493.610566][    C3] CPU: 3 UID: 0 PID: 4961 Comm: sleep Tainted: G                 N  7.0.0-rc2+ #1 PREEMPT(full)
[  493.610575][    C3] Tainted: [N]=TEST
[  493.610577][    C3] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[  493.610583][    C3] RIP: 0010:sched_mm_cid_after_execve (kernel/sched/sched.h:4026 (discriminator 2))
[  493.610596][    C3] Code: 05 00 00 4c 89 e8 48 c1 f8 06 4c 89 4c 24 08 49 8d bc c1 10 0b 00 00 e8 99 4a 90 00 4c 8b 4c 24 08 f0 4d 0f ab a9 10 0b 00 00 <0f> 82 b7 00 00 00 48 8b 54 24 10 44 8b 44 24 1c 48 b8 00 00 00 00
All code
========
0: 05 00 00 4c 89       add    $0x894c0000,%eax
5: e8 48 c1 f8 06       call   0x6f8c152
a: 4c 89 4c 24 08       mov    %r9,0x8(%rsp)
f: 49 8d bc c1 10 0b 00 lea    0xb10(%r9,%rax,8),%rdi
16: 00
17: e8 99 4a 90 00       call   0x904ab5
1c: 4c 8b 4c 24 08       mov    0x8(%rsp),%r9
21: f0 4d 0f ab a9 10 0b lock bts %r13,0xb10(%r9)
28: 00 00
2a:* 0f 82 b7 00 00 00   jb     0xe7 <-- trapping instruction
30: 48 8b 54 24 10       mov    0x10(%rsp),%rdx
35: 44 8b 44 24 1c       mov    0x1c(%rsp),%r8d
3a: 48                   rex.W
3b: b8 00 00 00 00       mov    $0x0,%eax

Code starting with the faulting instruction
===========================================
0: 0f 82 b7 00 00 00    jb     0xbd
6: 48 8b 54 24 10       mov    0x10(%rsp),%rdx
b: 44 8b 44 24 1c       mov    0x1c(%rsp),%r8d
10: 48                   rex.W
11: b8 00 00 00 00       mov    $0x0,%eax
[  493.610600][    C3] RSP: 0018:ffffc90002957e00 EFLAGS: 00000247
[  493.610605][    C3] RAX: 0000000000000001 RBX: 0000000000000001 RCX: 0000000000000001
[  493.610609][    C3] RDX: 0000000000000001 RSI: 0000000000000008 RDI: ffff8881000aba10
[  493.610612][    C3] RBP: dffffc0000000000 R08: ffffffff8185fe97 R09: ffff8881000aaf00
[  493.610615][    C3] R10: ffffed1020015743 R11: 0000000000000000 R12: ffffed1021dda7a8
[  493.610618][    C3] R13: 0000000000000000 R14: ffff8881000aaf00 R15: ffff88810eed3800
[  493.610663][    C3] FS:  0000000000000000(0000) GS:ffff8881cc110000(0000) knlGS:0000000000000000
[  493.610680][    C3] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  493.610684][    C3] CR2: 00007ffef3a7dd89 CR3: 00000001204a1005 CR4: 0000000000370ef0
[  493.610688][    C3] Call Trace:
[  493.610691][    C3]  <TASK>
[  493.610703][    C3]  bprm_execve (include/linux/rseq.h:140)
[  493.610717][    C3]  do_execveat_common.isra.0 (fs/exec.c:1846)
[  493.610731][    C3]  __x64_sys_execve (include/linux/fs.h:2539)
[  493.610740][    C3]  do_syscall_64 (arch/x86/entry/syscall_64.c:63 (discriminator 1))
[  493.610750][    C3]  ? exc_page_fault (arch/x86/mm/fault.c:1480 (discriminator 3))
[  493.610760][    C3]  entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
[  493.610767][    C3] RIP: 0033:0x7fb401448140
[  493.610780][    C3] Code: Unable to access opcode bytes at 0x7fb401448116.

Code starting with the faulting instruction
===========================================
[  493.610782][    C3] RSP: 002b:00007ffef3a7da70 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[  493.610787][    C3] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 0000000000000000
[  493.610790][    C3] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[  493.610793][    C3] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[  493.610795][    C3] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  493.610796][    C3] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  493.610815][    C3]  </TASK>
main_loop_s: timed out
[  503.574195][   T17] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 3-.... } 35898 jiffies s: 8761 root: 0x8/.
[  503.575291][   T17] rcu: blocking rcu_node structures (internal RCU debug):
[  503.576282][   T17] Sending NMI from CPU 1 to CPUs 3:
[  503.577258][    C3] NMI backtrace for cpu 3
[  503.577269][    C3] CPU: 3 UID: 0 PID: 4961 Comm: sleep Tainted: G                 N  7.0.0-rc2+ #1 PREEMPT(full)
[  503.577277][    C3] Tainted: [N]=TEST
[  503.577279][    C3] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[  503.577282][    C3] RIP: 0010:sched_mm_cid_after_execve (kernel/sched/sched.h:4022)
[  503.577295][    C3] Code: 34 2e 40 38 f0 7c 09 40 84 f6 0f 85 6e 01 00 00 8b 35 d6 72 d5 02 49 8d be 10 0b 00 00 e8 b6 cc f9 00 49 89 c5 41 80 3c 24 00 <0f> 85 5f 01 00 00 4d 8b b7 40 05 00 00 41 39 dd 0f 83 1e fd ff ff
All code
========
0: 34 2e                xor    $0x2e,%al
2: 40 38 f0             cmp    %sil,%al
5: 7c 09                jl     0x10
7: 40 84 f6             test   %sil,%sil
a: 0f 85 6e 01 00 00    jne    0x17e
10: 8b 35 d6 72 d5 02    mov    0x2d572d6(%rip),%esi        # 0x2d572ec
16: 49 8d be 10 0b 00 00 lea    0xb10(%r14),%rdi
1d: e8 b6 cc f9 00       call   0xf9ccd8
22: 49 89 c5             mov    %rax,%r13
25: 41 80 3c 24 00       cmpb   $0x0,(%r12)
2a:* 0f 85 5f 01 00 00    jne    0x18f <-- trapping instruction
30: 4d 8b b7 40 05 00 00 mov    0x540(%r15),%r14
37: 41 39 dd             cmp    %ebx,%r13d
3a: 0f 83 1e fd ff ff    jae    0xfffffffffffffd5e

Code starting with the faulting instruction
===========================================
0: 0f 85 5f 01 00 00    jne    0x165
6: 4d 8b b7 40 05 00 00 mov    0x540(%r15),%r14
d: 41 39 dd             cmp    %ebx,%r13d
10: 0f 83 1e fd ff ff    jae    0xfffffffffffffd34
[  503.577300][    C3] RSP: 0018:ffffc90002957e00 EFLAGS: 00000246
[  503.577305][    C3] RAX: 0000000000000001 RBX: 0000000000000001 RCX: 0000000000000001
[  503.577308][    C3] RDX: 0000000000000001 RSI: dffffc0000000000 RDI: ffff8881000aba10
[  503.577311][    C3] RBP: dffffc0000000000 R08: ffffffff8185fe97 R09: ffff8881000aaf00
[  503.577314][    C3] R10: ffffed1020015743 R11: 0000000000000000 R12: ffffed1021dda7a8
[  503.577316][    C3] R13: 0000000000000001 R14: ffff8881000aaf00 R15: ffff88810eed3800
[  503.577335][    C3] FS:  0000000000000000(0000) GS:ffff8881cc110000(0000) knlGS:0000000000000000
[  503.577350][    C3] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  503.577353][    C3] CR2: 00007fb401448116 CR3: 00000001204a1005 CR4: 0000000000370ef0
[  503.577356][    C3] Call Trace:
[  503.577360][    C3]  <TASK>
[  503.577368][    C3]  bprm_execve (include/linux/rseq.h:140)
[  503.577377][    C3]  do_execveat_common.isra.0 (fs/exec.c:1846)
[  503.577385][    C3]  __x64_sys_execve (include/linux/fs.h:2539)
[  503.577392][    C3]  do_syscall_64 (arch/x86/entry/syscall_64.c:63 (discriminator 1))
[  503.577401][    C3]  ? exc_page_fault (arch/x86/mm/fault.c:1480 (discriminator 3))
[  503.577408][    C3]  entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
[  503.577415][    C3] RIP: 0033:0x7fb401448140
[  503.577427][    C3] Code: Unable to access opcode bytes at 0x7fb401448116.

Code starting with the faulting instruction
===========================================
[  503.577430][    C3] RSP: 002b:00007ffef3a7da70 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[  503.577435][    C3] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 0000000000000000
[  503.577437][    C3] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[  503.577439][    C3] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[  503.577442][    C3] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  503.577444][    C3] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  503.577453][    C3]  </TASK>
[  530.927646][    C3] watchdog: BUG: soft lockup - CPU#3 stuck for 60s! [sleep:4961]
[  530.927660][    C3] Modules linked in: nft_tproxy nf_tproxy_ipv6 nf_tproxy_ipv4 nft_socket nf_socket_ipv4 nf_socket_ipv6 nf_tables sch_netem tcp_diag mptcp_diag inet_diag mptcp_token_test mptcp_crypto_test kunit
[  530.927693][    C3] irq event stamp: 141062
[  530.927696][    C3] hardirqs last  enabled at (141061): irqentry_exit (kernel/entry/common.c:243)
[  530.927711][    C3] hardirqs last disabled at (141062): sysvec_apic_timer_interrupt (arch/x86/include/asm/hardirq.h:81)
[  530.927715][    C3] softirqs last  enabled at (140962): handle_softirqs (kernel/softirq.c:469 (discriminator 2))
[  530.927720][    C3] softirqs last disabled at (140949): __irq_exit_rcu (kernel/softirq.c:657)
[  530.927727][    C3] CPU: 3 UID: 0 PID: 4961 Comm: sleep Tainted: G                 N  7.0.0-rc2+ #1 PREEMPT(full)
[  530.927732][    C3] Tainted: [N]=TEST
[  530.927734][    C3] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[  530.927736][    C3] RIP: 0010:sched_mm_cid_after_execve (kernel/sched/sched.h:4045 (discriminator 6))
[  530.927742][    C3] Code: 02 83 04 85 c0 0f 84 6b 02 00 00 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 cc cc cc cc 48 c7 c7 40 f5 cd 83 e8 bd df 19 02 <49> 8d be 00 01 00 00 48 89 f8 48 c1 e8 03 80 3c 28 00 0f 85 65 02
All code
========
0: 02 83 04 85 c0 0f    add    0xfc08504(%rbx),%al
6: 84 6b 02             test   %ch,0x2(%rbx)
9: 00 00                add    %al,(%rax)
b: 48 83 c4 30          add    $0x30,%rsp
f: 5b                   pop    %rbx
10: 5d                   pop    %rbp
11: 41 5c                pop    %r12
13: 41 5d                pop    %r13
15: 41 5e                pop    %r14
17: 41 5f                pop    %r15
19: c3                   ret
1a: cc                   int3
1b: cc                   int3
1c: cc                   int3
1d: cc                   int3
1e: 48 c7 c7 40 f5 cd 83 mov    $0xffffffff83cdf540,%rdi
25: e8 bd df 19 02       call   0x219dfe7
2a:* 49 8d be 00 01 00 00 lea    0x100(%r14),%rdi <-- trapping instruction
31: 48 89 f8             mov    %rdi,%rax
34: 48 c1 e8 03          shr    $0x3,%rax
38: 80 3c 28 00          cmpb   $0x0,(%rax,%rbp,1)
3c: 0f                   .byte 0xf
3d: 85 65 02             test   %esp,0x2(%rbp)

Code starting with the faulting instruction
===========================================
0: 49 8d be 00 01 00 00 lea    0x100(%r14),%rdi
7: 48 89 f8             mov    %rdi,%rax
a: 48 c1 e8 03          shr    $0x3,%rax
e: 80 3c 28 00          cmpb   $0x0,(%rax,%rbp,1)
12: 0f                   .byte 0xf
13: 85 65 02             test   %esp,0x2(%rbp)
[  530.927744][    C3] RSP: 0018:ffffc90002957e00 EFLAGS: 00000202
[  530.927747][    C3] RAX: 0000000000000003 RBX: 0000000000000001 RCX: 0000000000000001
[  530.927749][    C3] RDX: 0000000000000001 RSI: ffffffff83cdf540 RDI: ffffffff83eec920
[  530.927751][    C3] RBP: dffffc0000000000 R08: ffffffff8185fd4a R09: 0000000000000000
[  530.927752][    C3] R10: 0000000000000003 R11: 0000000000000000 R12: ffffed1021dda7a8
[  530.927753][    C3] R13: 0000000000000000 R14: ffff8881000aaf00 R15: ffff88810eed3800
[  530.927766][    C3] FS:  0000000000000000(0000) GS:ffff8881cc110000(0000) knlGS:0000000000000000
[  530.927776][    C3] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  530.927778][    C3] CR2: 00007fb401448116 CR3: 00000001204a1005 CR4: 0000000000370ef0
[  530.927780][    C3] Call Trace:
[  530.927784][    C3]  <TASK>
[  530.927792][    C3]  bprm_execve (include/linux/rseq.h:140)
[  530.927801][    C3]  do_execveat_common.isra.0 (fs/exec.c:1846)
[  530.927807][    C3]  __x64_sys_execve (include/linux/fs.h:2539)
[  530.927812][    C3]  do_syscall_64 (arch/x86/entry/syscall_64.c:63 (discriminator 1))
[  530.927817][    C3]  ? exc_page_fault (arch/x86/mm/fault.c:1480 (discriminator 3))
[  530.927821][    C3]  entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
[  530.927826][    C3] RIP: 0033:0x7fb401448140
[  530.927835][    C3] Code: Unable to access opcode bytes at 0x7fb401448116.

Code starting with the faulting instruction
===========================================
[  530.927837][    C3] RSP: 002b:00007ffef3a7da70 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[  530.927839][    C3] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 0000000000000000
[  530.927840][    C3] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[  530.927842][    C3] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[  530.927843][    C3] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  530.927844][    C3] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  530.927854][    C3]  </TASK>
[  530.927857][    C3] Kernel panic - not syncing: softlockup: hung tasks
[  530.958801][    C3] CPU: 3 UID: 0 PID: 4961 Comm: sleep Tainted: G             L   N  7.0.0-rc2+ #1 PREEMPT(full)
[  530.960019][    C3] Tainted: [L]=SOFTLOCKUP, [N]=TEST
[  530.960652][    C3] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[  530.961437][    C3] Call Trace:
[  530.961931][    C3]  <IRQ>
[  530.962208][    C3]  dump_stack_lvl (lib/dump_stack.c:122)
[  530.962740][    C3]  vpanic (kernel/panic.c:651)
[  530.963120][    C3]  panic (kernel/panic.c:787)
[  530.963502][    C3]  ? __pfx_panic (kernel/panic.c:783)
[  530.964024][    C3]  ? add_taint (arch/x86/include/asm/bitops.h:60)
[  530.964459][    C3]  watchdog_timer_fn.cold (kernel/watchdog.c:871)
[  530.965081][    C3]  ? __pfx_watchdog_timer_fn (kernel/watchdog.c:774)
[  530.965629][    C3]  __run_hrtimer (kernel/time/hrtimer.c:1785)
[  530.966417][    C3]  __hrtimer_run_queues (include/linux/timerqueue.h:25)
[  530.966930][    C3]  ? __pfx___hrtimer_run_queues (kernel/time/hrtimer.c:1819)
[  530.967470][    C3]  ? ktime_get_update_offsets_now (kernel/time/timekeeping.c:381)
[  530.968251][    C3]  hrtimer_interrupt (kernel/time/hrtimer.c:1914)
[  530.968990][    C3]  __sysvec_apic_timer_interrupt (arch/x86/include/asm/jump_label.h:37)
[  530.969783][    C3]  sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1056 (discriminator 47))
[  530.970343][    C3]  </IRQ>
[  530.970723][    C3]  <TASK>
[  530.971036][    C3]  asm_sysvec_apic_timer_interrupt (arch/x86/include/asm/idtentry.h:697)
[  530.971947][    C3] RIP: 0010:sched_mm_cid_after_execve (kernel/sched/sched.h:4045 (discriminator 6))
[  530.972671][    C3] Code: 02 83 04 85 c0 0f 84 6b 02 00 00 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 cc cc cc cc 48 c7 c7 40 f5 cd 83 e8 bd df 19 02 <49> 8d be 00 01 00 00 48 89 f8 48 c1 e8 03 80 3c 28 00 0f 85 65 02
All code
========
0: 02 83 04 85 c0 0f    add    0xfc08504(%rbx),%al
6: 84 6b 02             test   %ch,0x2(%rbx)
9: 00 00                add    %al,(%rax)
b: 48 83 c4 30          add    $0x30,%rsp
f: 5b                   pop    %rbx
10: 5d                   pop    %rbp
11: 41 5c                pop    %r12
13: 41 5d                pop    %r13
15: 41 5e                pop    %r14
17: 41 5f                pop    %r15
19: c3                   ret
1a: cc                   int3
1b: cc                   int3
1c: cc                   int3
1d: cc                   int3
1e: 48 c7 c7 40 f5 cd 83 mov    $0xffffffff83cdf540,%rdi
25: e8 bd df 19 02       call   0x219dfe7
2a:* 49 8d be 00 01 00 00 lea    0x100(%r14),%rdi <-- trapping instruction
31: 48 89 f8             mov    %rdi,%rax
34: 48 c1 e8 03          shr    $0x3,%rax
38: 80 3c 28 00          cmpb   $0x0,(%rax,%rbp,1)
3c: 0f                   .byte 0xf
3d: 85 65 02             test   %esp,0x2(%rbp)

Code starting with the faulting instruction
===========================================
0: 49 8d be 00 01 00 00 lea    0x100(%r14),%rdi
7: 48 89 f8             mov    %rdi,%rax
a: 48 c1 e8 03          shr    $0x3,%rax
e: 80 3c 28 00          cmpb   $0x0,(%rax,%rbp,1)
12: 0f                   .byte 0xf
13: 85 65 02             test   %esp,0x2(%rbp)
[  530.974394][    C3] RSP: 0018:ffffc90002957e00 EFLAGS: 00000202
[  530.975308][    C3] RAX: 0000000000000003 RBX: 0000000000000001 RCX: 0000000000000001
[  530.976147][    C3] RDX: 0000000000000001 RSI: ffffffff83cdf540 RDI: ffffffff83eec920
[  530.977030][    C3] RBP: dffffc0000000000 R08: ffffffff8185fd4a R09: 0000000000000000
[  530.977910][    C3] R10: 0000000000000003 R11: 0000000000000000 R12: ffffed1021dda7a8
[  530.978926][    C3] R13: 0000000000000000 R14: ffff8881000aaf00 R15: ffff88810eed3800
[  530.979595][    C3]  ? sched_mm_cid_after_execve (arch/x86/include/asm/bitops.h:136 (discriminator 1))
[  530.980252][    C3]  bprm_execve (include/linux/rseq.h:140)
[  530.980941][    C3]  do_execveat_common.isra.0 (fs/exec.c:1846)
[  530.981424][    C3]  __x64_sys_execve (include/linux/fs.h:2539)
[  530.981941][    C3]  do_syscall_64 (arch/x86/entry/syscall_64.c:63 (discriminator 1))
[  530.982406][    C3]  ? exc_page_fault (arch/x86/mm/fault.c:1480 (discriminator 3))
[  530.983097][    C3]  entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
[  530.983809][    C3] RIP: 0033:0x7fb401448140
[  530.984409][    C3] Code: Unable to access opcode bytes at 0x7fb401448116.

Code starting with the faulting instruction
===========================================
[  530.985240][    C3] RSP: 002b:00007ffef3a7da70 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[  530.986149][    C3] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 0000000000000000
[  530.986945][    C3] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[  530.987807][    C3] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[  530.988490][    C3] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  530.989334][    C3] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  530.990257][    C3]  </TASK>
[  530.991655][    C3] Kernel Offset: disabled

More details:

https://github.com/multipath-tcp/mptcp_net-next/actions/runs/22775627304#summary-66068034344

Cheers,
Matt