public inbox for virtualization@lists.linux-foundation.org
 help / color / mirror / Atom feed
From: Jiri Slaby <jirislaby@kernel.org>
To: Thomas Gleixner <tglx@kernel.org>, Peter Zijlstra <peterz@infradead.org>
Cc: "Matthieu Baerts" <matttbe@kernel.org>,
	"Stefan Hajnoczi" <stefanha@redhat.com>,
	"Stefano Garzarella" <sgarzare@redhat.com>,
	kvm@vger.kernel.org, virtualization@lists.linux.dev,
	Netdev <netdev@vger.kernel.org>,
	rcu@vger.kernel.org, "MPTCP Linux" <mptcp@lists.linux.dev>,
	"Linux Kernel" <linux-kernel@vger.kernel.org>,
	"Shinichiro Kawasaki" <shinichiro.kawasaki@wdc.com>,
	"Paul E. McKenney" <paulmck@kernel.org>,
	"Dave Hansen" <dave.hansen@linux.intel.com>,
	"luto@kernel.org" <luto@kernel.org>,
	"Michal Koutný" <MKoutny@suse.com>,
	"Waiman Long" <longman@redhat.com>
Subject: Re: Stalls when starting a VSOCK listening socket: soft lockups, RCU stalls, timeout
Date: Fri, 6 Mar 2026 11:16:28 +0100	[thread overview]
Message-ID: <babe39d9-aeb4-47ae-83fb-eae6193fb3aa@kernel.org> (raw)
In-Reply-To: <87qzpx2sck.ffs@tglx>

On 06. 03. 26, 10:57, Thomas Gleixner wrote:
> On Fri, Mar 06 2026 at 06:48, Jiri Slaby wrote:
>> On 05. 03. 26, 20:25, Thomas Gleixner wrote:
>>> Is there simple way to reproduce?
>>
>> Unfortunately not at all. To date, I even cannot reproduce locally, it
>> reproduces exclusively in opensuse build service (and github CI as per
>> Matthieu's report). I have a project in there with packages which fail
>> more often than others:
>>     https://build.opensuse.org/project/monitor/home:jirislaby:softlockup
>> But it's all green ATM.
>>
>> Builds of Go 1.24 and tests of rust 1.90 fail the most. The former even
>> takes only ~ 8 minutes, so it's not that intensive build at all. So the
>> reasons are unknown to me. At least, Go apparently uses threads for
>> building (unlike gcc/clang with forks/processes). Dunno about rust.
> 
> I tried with tons of test cases which stress test mmcid with threads and
> failed.

Me too, with many artificial pthread or fork or combined loads/bombs 
with loops and yields.

I was once successful to see the failure in a local build of Go: using 
"osc build --vm-type=kvm" which is what the build service (see below) calls.

It's extremely hard to hit it locally. So there is likely some rather 
small race window or whatnot.

> Can you provide me your .config

Sure, it's the standard openSUSE kernel, i.e.:
https://github.com/openSUSE/kernel-source/blob/9c1596772e0/config/x86_64/default

source version,

It happens with 6.19+, the current failures are with the commit above 
which is 6.19.5.

I added 7.0-rc2 as well now to:
   https://build.opensuse.org/project/monitor/home:jirislaby:softlockup

Well, it already failed for Go:
  
https://build.opensuse.org/package/live_build_log/home:jirislaby:softlockup/go1.24/617/x86_64

So at least it is consistent, and not stable tree related ;).


If that helps, I would be likely able to "bisect" the 4 your mm_cid 
patches if they can be reverted on the top of 6.19 easily. (By letting 
the kernel run in the build service.)

> VM setup (Number of CPUs, memory etc.)?

For example, the currently failing build:
https://build.opensuse.org/package/live_build_log/home:jirislaby:softlockup/rust1.90:test/openSUSE_Factory/x86_64

says:
[   10s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none 
-cpu host -M pc,accel=kvm,usb=off,dump-guest-core=off,vmport=off 
-sandbox on -bios /usr/share/qemu/qboot.rom -object 
rng-random,filename=/dev/random,id=rng0 -device virtio-rng-pci,rng=rng0 
-object iothread,id=io0 -run-with user=qemu -net none -kernel 
/var/cache/obs/worker/root_4/.mount/boot/kernel -initrd 
/var/cache/obs/worker/root_4/.mount/boot/initrd -append 
root=/dev/disk/by-id/virtio-0 rootfstype=ext4 rootflags=noatime 
elevator=noop nmi_watchdog=0 rw ia32_emulation=1 oops=panic panic=1 
quiet console=hvc0 init=/.build/build -m 40960 -drive 
file=/var/cache/obs/worker/root_4/root,format=raw,if=none,id=disk,cache=unsafe,aio=io_uring 
-device virtio-blk-pci,iothread=io0,drive=disk,serial=0 -drive 
file=/var/cache/obs/worker/root_4/swap,format=raw,if=none,id=swap,cache=unsafe,aio=io_uring 
-device virtio-blk-pci,iothread=io0,drive=swap,serial=1 -device 
virtio-serial,max_ports=2 -device virtconsole,chardev=virtiocon0 
-chardev stdio,mux=on,id=virtiocon0 -mon chardev=virtiocon0 -chardev 
socket,id=monitor,server=on,wait=off,path=/var/cache/obs/worker/root_4/root.qemu/monitor 
-mon chardev=monitor,mode=readline -smp 12



The with-7.0-rc2 Go fail above runs:

[    4s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none 
-cpu host -M pc,accel=kvm,usb=off,dump-guest-core=off,vmport=off 
-sandbox on -bios /usr/share/qemu/qboot.rom -object 
rng-random,filename=/dev/random,id=rng0 -device virtio-rng-pci,rng=rng0 
-object iothread,id=io0 -run-with user=qemu -net none -kernel 
/var/cache/obs/worker/root_12/.mount/boot/kernel -initrd 
/var/cache/obs/worker/root_12/.mount/boot/initrd -append 
root=/dev/disk/by-id/virtio-0 rootfstype=ext4 rootflags=noatime 
elevator=noop nmi_watchdog=0 rw ia32_emulation=1 oops=panic panic=1 
quiet console=hvc0 init=/.build/build -m 16384 -drive 
file=/var/cache/obs/worker/root_12/root,format=raw,if=none,id=disk,cache=unsafe,aio=io_uring 
-device virtio-blk-pci,iothread=io0,drive=disk,serial=0 -drive 
file=/var/cache/obs/worker/root_12/swap,format=raw,if=none,id=swap,cache=unsafe,aio=io_uring 
-device virtio-blk-pci,iothread=io0,drive=swap,serial=1 -device 
virtio-serial,max_ports=2 -device virtconsole,chardev=virtiocon0 
-chardev stdio,mux=on,id=virtiocon0 -mon chardev=virtiocon0 -chardev 
socket,id=monitor,server=on,wait=off,path=/var/cache/obs/worker/root_12/root.qemu/monitor 
-mon chardev=monitor,mode=readline -smp 4



> I tried to find it on that github page Matthiue mentioned but I'm
> probably too stupid to navigate this clicky interface.

I haven't looked into the details of the github failure yet...

thanks,
-- 
js
suse labs

  reply	other threads:[~2026-03-06 10:16 UTC|newest]

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-02-06 11:54 Stalls when starting a VSOCK listening socket: soft lockups, RCU stalls, timeout Matthieu Baerts
2026-02-06 16:38 ` Stefano Garzarella
2026-02-06 17:13   ` Matthieu Baerts
2026-02-26 10:37 ` Jiri Slaby
2026-03-02  5:28   ` Jiri Slaby
2026-03-02 11:46     ` Peter Zijlstra
2026-03-02 14:30       ` Waiman Long
2026-03-05  7:00       ` Jiri Slaby
2026-03-05 11:53         ` Jiri Slaby
2026-03-05 12:20           ` Jiri Slaby
2026-03-05 16:16             ` Thomas Gleixner
2026-03-05 17:33               ` Jiri Slaby
2026-03-05 19:25                 ` Thomas Gleixner
2026-03-06  5:48                   ` Jiri Slaby
2026-03-06  9:57                     ` Thomas Gleixner
2026-03-06 10:16                       ` Jiri Slaby [this message]
2026-03-06 16:28                         ` Thomas Gleixner
2026-03-06 11:06                       ` Matthieu Baerts
2026-03-06 16:57                         ` Matthieu Baerts
2026-03-06 18:31                           ` Jiri Slaby
2026-03-06 18:44                             ` Matthieu Baerts
2026-03-06 21:40                           ` Matthieu Baerts
2026-03-06 15:24                       ` Peter Zijlstra
2026-03-07  9:01                         ` Thomas Gleixner
2026-03-07 22:29                           ` Thomas Gleixner
2026-03-08  9:15                             ` Thomas Gleixner
2026-03-08 16:55                               ` Jiri Slaby
2026-03-08 16:58                               ` Thomas Gleixner
2026-03-08 17:23                                 ` Matthieu Baerts
2026-03-09  8:43                                   ` Thomas Gleixner
2026-03-09 12:23                                     ` Matthieu Baerts
2026-03-10  8:09                                       ` Thomas Gleixner
2026-03-10  8:20                                         ` Thomas Gleixner
2026-03-10  8:56                                         ` Jiri Slaby
2026-03-10  9:00                                           ` Jiri Slaby
2026-03-10 10:03                                             ` Thomas Gleixner
2026-03-10 10:06                                               ` Thomas Gleixner
2026-03-10 11:24                                                 ` Matthieu Baerts
2026-03-10 11:54                                                   ` Peter Zijlstra
2026-03-10 12:28                                                     ` Thomas Gleixner
2026-03-10 13:40                                                       ` Matthieu Baerts
2026-03-10 13:47                                                         ` Thomas Gleixner
2026-03-10 15:51                                                           ` Matthieu Baerts
2026-03-03 13:23   ` Matthieu Baerts
2026-03-05  6:46     ` Jiri Slaby

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=babe39d9-aeb4-47ae-83fb-eae6193fb3aa@kernel.org \
    --to=jirislaby@kernel.org \
    --cc=MKoutny@suse.com \
    --cc=dave.hansen@linux.intel.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=longman@redhat.com \
    --cc=luto@kernel.org \
    --cc=matttbe@kernel.org \
    --cc=mptcp@lists.linux.dev \
    --cc=netdev@vger.kernel.org \
    --cc=paulmck@kernel.org \
    --cc=peterz@infradead.org \
    --cc=rcu@vger.kernel.org \
    --cc=sgarzare@redhat.com \
    --cc=shinichiro.kawasaki@wdc.com \
    --cc=stefanha@redhat.com \
    --cc=tglx@kernel.org \
    --cc=virtualization@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox