Re: Stalls when starting a VSOCK listening socket: soft lockups, RCU stalls, timeout

Matthieu Baerts
2026-03-06T11:06:59Z

Hi Thomas,

Thank you for looking into this!

On 06/03/2026 10:57, Thomas Gleixner wrote:
> On Fri, Mar 06 2026 at 06:48, Jiri Slaby wrote:
>> On 05. 03. 26, 20:25, Thomas Gleixner wrote:
>>> Is there simple way to reproduce?
>>
>> Unfortunately not at all. To date, I even cannot reproduce locally, it
>> reproduces exclusively in opensuse build service (and github CI as per
>> Matthieu's report). I have a project in there with packages which fail
>> more often than others:
>> https://build.opensuse.org/project/monitor/home:jirislaby:softlockup
>> But it's all green ATM.
>>
>> Builds of Go 1.24 and tests of rust 1.90 fail the most. The former even
>> takes only ~ 8 minutes, so it's not that intensive build at all. So the
>> reasons are unknown to me. At least, Go apparently uses threads for
>> building (unlike gcc/clang with forks/processes). Dunno about rust.
>
> I tried with tons of test cases which stress test mmcid with threads and
> failed.

On my side, I didn't manage to reproduce it locally either.

> Can you provide me your .config, source version, VM setup (Number of
> CPUs, memory etc.)?

My CI ran into this issue 2 days ago, with and without a debug kernel
config. The kernel being tested was on top of 'net-next', which was on
top of this commit from Linus' tree: fbdfa8da05b6 ("selftests:
tc-testing: fix list_categories() crash on list type").

- Config without debug:

https://github.com/user-attachments/files/25791728/config-run-22657946888-normal-join.gz

- Config with debug:

https://github.com/user-attachments/files/25791960/config-run-22657946888-debug-nojoin.gz

- Just in case, stacktraces available there:

https://github.com/multipath-tcp/mptcp_net-next/actions/runs/22657946888

My tests are being executed in VMs I don't control using a kernel v6.14
on Azure with 4 vCPUs, 16GB of RAM, and KVM nested support. From more
details about what's in it:

https://github.com/actions/runner-images/blob/ubuntu24/20260302.42/images/ubuntu/Ubuntu2404-Readme.md

From there, a docker container is started, from which QEMU 10.1.0
(Debian 1:10.1.0+ds-5ubuntu2.2) is launched with 4 vCPU and 5GB of RAM
using this command:

/usr/bin/qemu-system-x86_64 \
-name mptcpdev \
-m 5120M \
-smp 4 \
-chardev socket,id=charvirtfs5,path=/tmp/virtmevrwrzu5k \
-device vhost-user-fs-device,chardev=charvirtfs5,tag=ROOTFS \
-object memory-backend-memfd,id=mem,size=5120M,share=on \
-numa node,memdev=mem \
-machine accel=kvm:tcg \
-M microvm,accel=kvm,pcie=on,rtc=on \
-cpu host,topoext=on \
-parallel none \
-net none \
-echr 1 \
-chardev file,path=/proc/self/fd/2,id=dmesg \
-device virtio-serial-device \
-device virtconsole,chardev=dmesg \
-chardev stdio,id=console,signal=off,mux=on \
-serial chardev:console \
-mon chardev=console \
-vga none \
-display none \
-device vhost-vsock-device,guest-cid=3 \
-kernel
/home/runner/work/mptcp_net-next/mptcp_net-next/.virtme/build/arch/x86/boot/bzImage
\
-append 'virtme_hostname=mptcpdev nr_open=1048576
virtme_link_mods=/home/runner/work/mptcp_net-next/mptcp_net-next/.virtme/build/.virtme_mods/lib/modules/0.0.0
virtme_rw_overlay0=/tmp console=hvc0 earlyprintk=serial,ttyS0,115200
virtme_console=ttyS0 psmouse.proto=exps
virtme.vsockexec=`/tmp/virtme-console/3.sh`
virtme_chdir=home/runner/work/mptcp_net-next/mptcp_net-next
virtme_root_user=1 rootfstype=virtiofs root=ROOTFS raid=noautodetect rw
debug nokaslr mitigations=off softlockup_panic=1 nmi_watchdog=1
hung_task_panic=1 panic=-1 oops=panic
init=/usr/local/lib/python3.13/dist-packages/virtme/guest/bin/virtme-ng-init'
\
-gdb tcp::1234 \
-qmp tcp::3636,server,nowait \
-no-reboot

It is possible to locally launch the same command using the same QEMU
version (but not the same host kernel) with the help of Docker:

$ cd <kernel source code>
# docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --rm \
-it --privileged mptcp/mptcp-upstream-virtme-docker:latest \
manual normal

This will build a new kernel in O=.virtme/build, launch it and give you
access to a prompt.

After that, you can do also use the "auto" mode with the last built
image to boot the VM, only print "OK", stop and retry if there were no
errors:

$ cd <kernel source code>
$ echo 'echo OK' > .virtme-exec-run
# i=1; \
while docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --rm \
-it --privileged mptcp/mptcp-upstream-virtme-docker:latest \
vm auto normal; do \
echo "== Attempt: $i: OK =="; \
i=$((i+1)); \
done; \
echo "== Failure after $i attempts =="

> I tried to find it on that github page Matthiue mentioned but I'm
> probably too stupid to navigate this clicky interface.

I'm sorry about that, I understand, the interface is not very clear. Do
not hesitate to tell me if you need anything else from me.

Cheers,
Matt
--