All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jiri Slaby <jirislaby@kernel.org>
To: Thomas Gleixner <tglx@kernel.org>, Peter Zijlstra <peterz@infradead.org>
Cc: "Matthieu Baerts" <matttbe@kernel.org>,
	"Stefan Hajnoczi" <stefanha@redhat.com>,
	"Stefano Garzarella" <sgarzare@redhat.com>,
	kvm@vger.kernel.org, virtualization@lists.linux.dev,
	Netdev <netdev@vger.kernel.org>,
	rcu@vger.kernel.org, "MPTCP Linux" <mptcp@lists.linux.dev>,
	"Linux Kernel" <linux-kernel@vger.kernel.org>,
	"Shinichiro Kawasaki" <shinichiro.kawasaki@wdc.com>,
	"Paul E. McKenney" <paulmck@kernel.org>,
	"Dave Hansen" <dave.hansen@linux.intel.com>,
	"luto@kernel.org" <luto@kernel.org>,
	"Michal Koutný" <MKoutny@suse.com>,
	"Waiman Long" <longman@redhat.com>
Subject: Re: Stalls when starting a VSOCK listening socket: soft lockups, RCU stalls, timeout
Date: Fri, 6 Mar 2026 11:16:28 +0100	[thread overview]
Message-ID: <babe39d9-aeb4-47ae-83fb-eae6193fb3aa@kernel.org> (raw)
In-Reply-To: <87qzpx2sck.ffs@tglx>

On 06. 03. 26, 10:57, Thomas Gleixner wrote:
> On Fri, Mar 06 2026 at 06:48, Jiri Slaby wrote:
>> On 05. 03. 26, 20:25, Thomas Gleixner wrote:
>>> Is there simple way to reproduce?
>>
>> Unfortunately not at all. To date, I even cannot reproduce locally, it
>> reproduces exclusively in opensuse build service (and github CI as per
>> Matthieu's report). I have a project in there with packages which fail
>> more often than others:
>>     https://build.opensuse.org/project/monitor/home:jirislaby:softlockup
>> But it's all green ATM.
>>
>> Builds of Go 1.24 and tests of rust 1.90 fail the most. The former even
>> takes only ~ 8 minutes, so it's not that intensive build at all. So the
>> reasons are unknown to me. At least, Go apparently uses threads for
>> building (unlike gcc/clang with forks/processes). Dunno about rust.
> 
> I tried with tons of test cases which stress test mmcid with threads and
> failed.

Me too, with many artificial pthread or fork or combined loads/bombs 
with loops and yields.

I was once successful to see the failure in a local build of Go: using 
"osc build --vm-type=kvm" which is what the build service (see below) calls.

It's extremely hard to hit it locally. So there is likely some rather 
small race window or whatnot.

> Can you provide me your .config

Sure, it's the standard openSUSE kernel, i.e.:
https://github.com/openSUSE/kernel-source/blob/9c1596772e0/config/x86_64/default

source version,

It happens with 6.19+, the current failures are with the commit above 
which is 6.19.5.

I added 7.0-rc2 as well now to:
   https://build.opensuse.org/project/monitor/home:jirislaby:softlockup

Well, it already failed for Go:
  
https://build.opensuse.org/package/live_build_log/home:jirislaby:softlockup/go1.24/617/x86_64

So at least it is consistent, and not stable tree related ;).


If that helps, I would be likely able to "bisect" the 4 your mm_cid 
patches if they can be reverted on the top of 6.19 easily. (By letting 
the kernel run in the build service.)

> VM setup (Number of CPUs, memory etc.)?

For example, the currently failing build:
https://build.opensuse.org/package/live_build_log/home:jirislaby:softlockup/rust1.90:test/openSUSE_Factory/x86_64

says:
[   10s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none 
-cpu host -M pc,accel=kvm,usb=off,dump-guest-core=off,vmport=off 
-sandbox on -bios /usr/share/qemu/qboot.rom -object 
rng-random,filename=/dev/random,id=rng0 -device virtio-rng-pci,rng=rng0 
-object iothread,id=io0 -run-with user=qemu -net none -kernel 
/var/cache/obs/worker/root_4/.mount/boot/kernel -initrd 
/var/cache/obs/worker/root_4/.mount/boot/initrd -append 
root=/dev/disk/by-id/virtio-0 rootfstype=ext4 rootflags=noatime 
elevator=noop nmi_watchdog=0 rw ia32_emulation=1 oops=panic panic=1 
quiet console=hvc0 init=/.build/build -m 40960 -drive 
file=/var/cache/obs/worker/root_4/root,format=raw,if=none,id=disk,cache=unsafe,aio=io_uring 
-device virtio-blk-pci,iothread=io0,drive=disk,serial=0 -drive 
file=/var/cache/obs/worker/root_4/swap,format=raw,if=none,id=swap,cache=unsafe,aio=io_uring 
-device virtio-blk-pci,iothread=io0,drive=swap,serial=1 -device 
virtio-serial,max_ports=2 -device virtconsole,chardev=virtiocon0 
-chardev stdio,mux=on,id=virtiocon0 -mon chardev=virtiocon0 -chardev 
socket,id=monitor,server=on,wait=off,path=/var/cache/obs/worker/root_4/root.qemu/monitor 
-mon chardev=monitor,mode=readline -smp 12



The with-7.0-rc2 Go fail above runs:

[    4s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none 
-cpu host -M pc,accel=kvm,usb=off,dump-guest-core=off,vmport=off 
-sandbox on -bios /usr/share/qemu/qboot.rom -object 
rng-random,filename=/dev/random,id=rng0 -device virtio-rng-pci,rng=rng0 
-object iothread,id=io0 -run-with user=qemu -net none -kernel 
/var/cache/obs/worker/root_12/.mount/boot/kernel -initrd 
/var/cache/obs/worker/root_12/.mount/boot/initrd -append 
root=/dev/disk/by-id/virtio-0 rootfstype=ext4 rootflags=noatime 
elevator=noop nmi_watchdog=0 rw ia32_emulation=1 oops=panic panic=1 
quiet console=hvc0 init=/.build/build -m 16384 -drive 
file=/var/cache/obs/worker/root_12/root,format=raw,if=none,id=disk,cache=unsafe,aio=io_uring 
-device virtio-blk-pci,iothread=io0,drive=disk,serial=0 -drive 
file=/var/cache/obs/worker/root_12/swap,format=raw,if=none,id=swap,cache=unsafe,aio=io_uring 
-device virtio-blk-pci,iothread=io0,drive=swap,serial=1 -device 
virtio-serial,max_ports=2 -device virtconsole,chardev=virtiocon0 
-chardev stdio,mux=on,id=virtiocon0 -mon chardev=virtiocon0 -chardev 
socket,id=monitor,server=on,wait=off,path=/var/cache/obs/worker/root_12/root.qemu/monitor 
-mon chardev=monitor,mode=readline -smp 4



> I tried to find it on that github page Matthiue mentioned but I'm
> probably too stupid to navigate this clicky interface.

I haven't looked into the details of the github failure yet...

thanks,
-- 
js
suse labs

  reply	other threads:[~2026-03-06 10:16 UTC|newest]

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-02-06 11:54 Stalls when starting a VSOCK listening socket: soft lockups, RCU stalls, timeout Matthieu Baerts
2026-02-06 16:38 ` Stefano Garzarella
2026-02-06 17:13   ` Matthieu Baerts
2026-02-26 10:37 ` Jiri Slaby
2026-03-02  5:28   ` Jiri Slaby
2026-03-02 11:46     ` Peter Zijlstra
2026-03-02 14:30       ` Waiman Long
2026-03-05  7:00       ` Jiri Slaby
2026-03-05 11:53         ` Jiri Slaby
2026-03-05 12:20           ` Jiri Slaby
2026-03-05 16:16             ` Thomas Gleixner
2026-03-05 17:33               ` Jiri Slaby
2026-03-05 19:25                 ` Thomas Gleixner
2026-03-06  5:48                   ` Jiri Slaby
2026-03-06  9:57                     ` Thomas Gleixner
2026-03-06 10:16                       ` Jiri Slaby [this message]
2026-03-06 16:28                         ` Thomas Gleixner
2026-03-06 11:06                       ` Matthieu Baerts
2026-03-06 16:57                         ` Matthieu Baerts
2026-03-06 18:31                           ` Jiri Slaby
2026-03-06 18:44                             ` Matthieu Baerts
2026-03-06 21:40                           ` Matthieu Baerts
2026-03-06 15:24                       ` Peter Zijlstra
2026-03-07  9:01                         ` Thomas Gleixner
2026-03-07 22:29                           ` Thomas Gleixner
2026-03-08  9:15                             ` Thomas Gleixner
2026-03-08 16:55                               ` Jiri Slaby
2026-03-08 16:58                               ` Thomas Gleixner
2026-03-08 17:23                                 ` Matthieu Baerts
2026-03-09  8:43                                   ` Thomas Gleixner
2026-03-09 12:23                                     ` Matthieu Baerts
2026-03-10  8:09                                       ` Thomas Gleixner
2026-03-10  8:20                                         ` Thomas Gleixner
2026-03-10  8:56                                         ` Jiri Slaby
2026-03-10  9:00                                           ` Jiri Slaby
2026-03-10 10:03                                             ` Thomas Gleixner
2026-03-10 10:06                                               ` Thomas Gleixner
2026-03-10 11:24                                                 ` Matthieu Baerts
2026-03-10 11:54                                                   ` Peter Zijlstra
2026-03-10 12:28                                                     ` Thomas Gleixner
2026-03-10 13:40                                                       ` Matthieu Baerts
2026-03-10 13:47                                                         ` Thomas Gleixner
2026-03-10 15:51                                                           ` Matthieu Baerts
2026-03-03 13:23   ` Matthieu Baerts
2026-03-05  6:46     ` Jiri Slaby

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=babe39d9-aeb4-47ae-83fb-eae6193fb3aa@kernel.org \
    --to=jirislaby@kernel.org \
    --cc=MKoutny@suse.com \
    --cc=dave.hansen@linux.intel.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=longman@redhat.com \
    --cc=luto@kernel.org \
    --cc=matttbe@kernel.org \
    --cc=mptcp@lists.linux.dev \
    --cc=netdev@vger.kernel.org \
    --cc=paulmck@kernel.org \
    --cc=peterz@infradead.org \
    --cc=rcu@vger.kernel.org \
    --cc=sgarzare@redhat.com \
    --cc=shinichiro.kawasaki@wdc.com \
    --cc=stefanha@redhat.com \
    --cc=tglx@kernel.org \
    --cc=virtualization@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.