Re: BUG: soft lockup - CPU#0 stuck for 26s! with nested KVM on 5.19.x

[Sean Christopherson <seanjc@google.com>]

On Thu, Sep 08, 2022, František Šumšal wrote:
> On 9/7/22 17:23, Sean Christopherson wrote:
> > On Wed, Sep 07, 2022, František Šumšal wrote:
> > > On 9/7/22 17:08, Sean Christopherson wrote:
> > > > On Wed, Sep 07, 2022, František Šumšal wrote:
> > > > > Hello!
> > > > > 
> > > > > In our Arch Linux part of the upstream systemd CI I recently noticed an
> > > > > uptrend in CPU soft lockups when running one of our tests. This test runs
> > > > > several systemd-nspawn containers in succession and sometimes the underlying
> > > > > VM locks up due to a CPU soft lockup
> > > > 
> > > > By "underlying VM", do you mean L1 or L2?  Where
> > > > 
> > > >       L0 == Bare Metal
> > > >       L1 == Arch Linux (KVM, 5.19.5-arch1-1/5.19.7-arch1-1)
> > > >       L2 == Arch Linux (nested KVM or QEMU TCG, 5.19.5-arch1-1/5.19.7-arch1-1)
> > > 
> > > I mean L2.
> > 
> > Is there anything interesting in the L1 or L0 logs?  A failure in a lower level
> > can manifest as a soft lockup and/or stall in the VM, e.g. because a vCPU isn't
> > run by the host for whatever reason.
> 
> There's nothing (quite literally) in the L0 logs, the host is silent when running the VM/tests.
> As for L1, there doesn't seem to be anything interesting as well. Here are the L1 and L2 logs
> for reference: https://mrc0mmand.fedorapeople.org/kernel-kvm-soft-lockup/2022-09-07-logs/
> 
> > 
> > Does the bug repro with an older version of QEMU?  If it's difficult to roll back
> > the QEMU version, then we can punt on this question for now.
> 
> > 
> > Is it possible to run the nspawn tests in L1?  If the bug repros there, that would
> > greatly shrink the size of the haystack.
> 
> I've fiddled around with the test and managed to trim it down enough so it's easy to run in both
> L1 and L2, and after couple of hours I managed to reproduce it in both layers. That also somewhat
> answers the QEMU question, since L0 uses QEMU 6.2.0 to run L1, and L1 uses QEMU 7.0.0 to run L2.
> In both cases I used TCG emulation, since with it the issue appears to reproduce slightly more
> often (or maybe I was just unlucky with KVM).
> 
> https://mrc0mmand.fedorapeople.org/kernel-kvm-soft-lockup/2022-09-07-logs-no-L2/L1_console.log
> 
> As in the previous case, there's nothing of interest in the L0 logs.
> 
> This also raises a question - is this issue still KVM-related, since in the last case there's
> just L0 baremetal and L1 QEMU/TCG without KVM involved?

Ya, unless there's a latent bug in KVM that's present in your L0 kernel, which is
extremely unlikely given that the bug repros with 4.18 and 5.17 as the bare metal
kernel, this isn't a KVM problem.

The mm, ext4, and scheduler subsystems are all likely candidates.  I'm not familiar
enough with their gory details to point fingers though.

Do you think it's possible to bisect the L1 kernel using the QEMU/TCG configuration?
That'd probably be the least awful way to get a root cause.