From: Jens Axboe <axboe@kernel.dk>
To: Dexuan Cui <decui@microsoft.com>,
"'ming.lei@redhat.com'" <ming.lei@redhat.com>,
'Christoph Hellwig' <hch@lst.de>,
"'linux-block@vger.kernel.org'" <linux-block@vger.kernel.org>
Cc: Long Li <longli@microsoft.com>,
"Michael Kelley (LINUX)" <mikelley@microsoft.com>,
"'linux-kernel@vger.kernel.org'" <linux-kernel@vger.kernel.org>
Subject: Re: Random high CPU utilization in blk-mq with the none scheduler
Date: Fri, 10 Dec 2021 19:04:47 -0700 [thread overview]
Message-ID: <b80bfe9a-bece-1f32-3d2a-fb4d94b1fa8c@kernel.dk> (raw)
In-Reply-To: <BYAPR21MB1270DCE17A0FE017AF3272F1BF729@BYAPR21MB1270.namprd21.prod.outlook.com>
On 12/10/21 6:29 PM, Dexuan Cui wrote:
>> From: Dexuan Cui
>> Sent: Thursday, December 9, 2021 7:30 PM
>>
>> Hi all,
>> I found a random high CPU utilization issue with some database benchmark
>> program running on a 192-CPU virtual machine (VM). Originally the issue
>> was found with RHEL 8.4 and Ubuntu 20.04, and further tests show that the
>> issue also reproduces with the latest upstream stable kernel v5.15.7, but
>> *not* with v5.16-rc1. It looks like someone resolved the issue in v5.16-rc1
>> recently?
>
> I did git-bisect on the linux-block tree's for-5.16/block branch and this patch
> resolves the random high CPU utilization issue (I'm not sure how):
> dc5fc361d891 ("block: attempt direct issue of plug list")
> https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?h=for-5.16/block&id=dc5fc361d891e089dfd9c0a975dc78041036b906
>
> Do you think if it's easy to backport it to earlier versions like 5.10?
> It looks like there are a lot of prerequisite patches.
It's more likely the real fix is avoiding the repeated plug list scan,
which I guess makes sense. That is this commit:
commit d38a9c04c0d5637a828269dccb9703d42d40d42b
Author: Jens Axboe <axboe@kernel.dk>
Date: Thu Oct 14 07:24:07 2021 -0600
block: only check previous entry for plug merge attempt
If that's the case, try 5.15.x again and do:
echo 2 > /sys/block/<dev>/queue/nomerges
for each drive you are using in the IO test, and see if that gets
rid of the excess CPU usage.
--
Jens Axboe
next prev parent reply other threads:[~2021-12-11 2:05 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-10 3:30 Random high CPU utilization in blk-mq with the none scheduler Dexuan Cui
2021-12-11 1:29 ` Dexuan Cui
2021-12-11 2:04 ` Jens Axboe [this message]
2021-12-11 3:10 ` Dexuan Cui
2021-12-11 3:15 ` Jens Axboe
2021-12-11 3:44 ` Dexuan Cui
2021-12-11 7:09 ` Dexuan Cui
2021-12-11 14:21 ` Jens Axboe
2021-12-11 18:54 ` Dexuan Cui
2021-12-13 18:43 ` Jens Axboe
2021-12-14 0:43 ` Dexuan Cui
2021-12-13 3:23 ` Ming Lei
2021-12-13 4:20 ` Dexuan Cui
2021-12-13 7:38 ` Ming Lei
2021-12-14 0:31 ` Dexuan Cui
2021-12-14 0:53 ` Ming Lei
2021-12-14 3:09 ` Dexuan Cui
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b80bfe9a-bece-1f32-3d2a-fb4d94b1fa8c@kernel.dk \
--to=axboe@kernel.dk \
--cc=decui@microsoft.com \
--cc=hch@lst.de \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=longli@microsoft.com \
--cc=mikelley@microsoft.com \
--cc=ming.lei@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox