Re: storm-of-soft-lockups: spinlocks running on all cores, preventing forward progress (4.14- to 5.0+)

[Zygo Blaxell <ce3g8jdj@umail.furryterror.org>]

[-- Attachment #1: Type: text/plain, Size: 2174 bytes --]

On Thu, May 16, 2019 at 09:57:01AM +0300, Nikolay Borisov wrote:
> 
> 
> On 16.05.19 г. 0:36 ч., Zygo Blaxell wrote:
> > "Storm-of-soft-lockups" is a failure mode where btrfs puts all of the
> > CPU cores in kernel functions that are unable to make forward progress,
> > but also unwilling to release their respective CPU cores.  This is
> > usually accompanied by a lot of CPU usage (detectable as either kvm CPU
> > usage or just a lot of CPU fan noise) though I don't know if all cores
> > are spinning or only some of them.
> > 
> > The kernel console presents a continual stream of "BUG: soft lockup"
> > warnings for some days.  None of the call traces change during this time.
> > The only way out is to reboot.
> > 
> > You can reproduce this by writing a bunch of data to a filesystem while
> > bees is running on all cores.  It takes a few days to occur naturally.
> > It can probably be sped up by just doing a bunch of random LOGICAL_INO
> > ioctls in a tight loop on each core.
> > 
> > Here's an instance on a 4-CPU VM where CPU#0 is running btrfs-transaction
> > (btrfs_try_tree_write_lock) and CPU#1-3 are running the LOGICAL_INO
> > ioctl (btrfs_tree_read_lock_atomic):
> 
> 
> Provide output of all sleeping threads when this occur via
>  echo w > /proc/sysrq-trigger.

The machine is dead in this state--it doesn't respond to pings or serial
port input, and can't run a shell.  The serial console doesn't respond
to BREAK-w either.  Those per-CPU stack traces every 22 seconds are all
I get, and also the only indication the system is not completely stopped.
The per-CPU stack traces do continue for days, and never report any
processes running other than those four.

> Also do you have this patch on the affected machine:
> 
> 38e3eebff643 ("btrfs: honor path->skip_locking in backref code") can you
> try and test with it applied ?

I have that patch applied already from when I was collecting deadlock
fixes earlier this year.

I have observations of storm-of-soft-lockups going back to at least
4.14 (it is #5 out of the 6 most common ways 4.14.y kernels fail).
So it is not a new bug.

> <SNIP>
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]