From: Jan Kara <jack@suse.cz>
To: John Garry <john.g.garry@oracle.com>
Cc: linux-kernel@vger.kernel.org, viro@zeniv.linux.org.uk,
brauner@kernel.org, jack@suse.cz, linux-fsdevel@vger.kernel.org
Subject: Re: v6.12-rc workqueue lockups
Date: Wed, 23 Oct 2024 22:13:14 +0200 [thread overview]
Message-ID: <20241023201314.hqfewgr6lej6f7df@quack3> (raw)
In-Reply-To: <63d6ceeb-a22f-4dee-bc9d-8687ce4c7355@oracle.com>
Hi!
On Wed 23-10-24 11:19:24, John Garry wrote:
> I have been seeing lockups reliably occur on v6.12-rc1, 3, 4 and linus'
> master branch:
>
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 22 09:07:15 ...
> kernel:watchdog: BUG: soft lockup - CPU#12 stuck for 26s! [khugepaged:154]
>
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 22 09:08:07 ...
> kernel:BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck
> for 44s!
>
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 22 09:08:07 ...
> kernel:BUG: workqueue lockup - pool cpus=4 node=0 flags=0x0 nice=0 stuck
> for 35s!
>
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 22 09:08:07 ...
> kernel:BUG: workqueue lockup - pool cpus=10 node=0 flags=0x0 nice=0 stuck
> for 33s!
>
> This is while doing some server MySQL performance testing. v6.11 has no such
> issue.
>
> I added some debug, and we seem to be spending a lot of time in FS
> writeback, specifically wb_workfn() -> wb_do_writeback() - ring any bells?
Thanks for report. This doesn't remind me of anything but checking the
writeback changes we have merged 532980cb1bf ("inode: port __I_SYNC to var
event") which could have odd consequences if we made mistake somewhere...
It probably won't be easy to revert but you could try whether the problem
reproduces before / after this commit.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
next prev parent reply other threads:[~2024-10-23 20:13 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-10-23 10:19 v6.12-rc workqueue lockups John Garry
2024-10-23 20:13 ` Jan Kara [this message]
2024-10-24 3:18 ` Dave Chinner
2024-10-23 20:39 ` Jan Kara
2024-10-24 9:35 ` John Garry
2024-10-24 21:13 ` Dave Chinner
2024-10-24 22:23 ` John Garry
2024-10-28 5:35 ` Dave Chinner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20241023201314.hqfewgr6lej6f7df@quack3 \
--to=jack@suse.cz \
--cc=brauner@kernel.org \
--cc=john.g.garry@oracle.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox