From: Jens Axboe <axboe@kernel.dk>
To: Dexuan Cui <decui@microsoft.com>,
"'ming.lei@redhat.com'" <ming.lei@redhat.com>,
'Christoph Hellwig' <hch@lst.de>,
"'linux-block@vger.kernel.org'" <linux-block@vger.kernel.org>
Cc: Long Li <longli@microsoft.com>,
"Michael Kelley (LINUX)" <mikelley@microsoft.com>,
"'linux-kernel@vger.kernel.org'" <linux-kernel@vger.kernel.org>
Subject: Re: Random high CPU utilization in blk-mq with the none scheduler
Date: Fri, 10 Dec 2021 19:04:47 -0700 [thread overview]
Message-ID: <b80bfe9a-bece-1f32-3d2a-fb4d94b1fa8c@kernel.dk> (raw)
In-Reply-To: <BYAPR21MB1270DCE17A0FE017AF3272F1BF729@BYAPR21MB1270.namprd21.prod.outlook.com>
On 12/10/21 6:29 PM, Dexuan Cui wrote:
>> From: Dexuan Cui
>> Sent: Thursday, December 9, 2021 7:30 PM
>>
>> Hi all,
>> I found a random high CPU utilization issue with some database benchmark
>> program running on a 192-CPU virtual machine (VM). Originally the issue
>> was found with RHEL 8.4 and Ubuntu 20.04, and further tests show that the
>> issue also reproduces with the latest upstream stable kernel v5.15.7, but
>> *not* with v5.16-rc1. It looks like someone resolved the issue in v5.16-rc1
>> recently?
>
> I did git-bisect on the linux-block tree's for-5.16/block branch and this patch
> resolves the random high CPU utilization issue (I'm not sure how):
> dc5fc361d891 ("block: attempt direct issue of plug list")
> https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?h=for-5.16/block&id=dc5fc361d891e089dfd9c0a975dc78041036b906
>
> Do you think if it's easy to backport it to earlier versions like 5.10?
> It looks like there are a lot of prerequisite patches.
It's more likely the real fix is avoiding the repeated plug list scan,
which I guess makes sense. That is this commit:
commit d38a9c04c0d5637a828269dccb9703d42d40d42b
Author: Jens Axboe <axboe@kernel.dk>
Date: Thu Oct 14 07:24:07 2021 -0600
block: only check previous entry for plug merge attempt
If that's the case, try 5.15.x again and do:
echo 2 > /sys/block/<dev>/queue/nomerges
for each drive you are using in the IO test, and see if that gets
rid of the excess CPU usage.
--
Jens Axboe
next prev parent reply other threads:[~2021-12-11 2:05 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-10 3:30 Random high CPU utilization in blk-mq with the none scheduler Dexuan Cui
2021-12-11 1:29 ` Dexuan Cui
2021-12-11 2:04 ` Jens Axboe [this message]
2021-12-11 3:10 ` Dexuan Cui
2021-12-11 3:15 ` Jens Axboe
2021-12-11 3:44 ` Dexuan Cui
2021-12-11 7:09 ` Dexuan Cui
2021-12-11 14:21 ` Jens Axboe
2021-12-11 18:54 ` Dexuan Cui
2021-12-13 18:43 ` Jens Axboe
2021-12-14 0:43 ` Dexuan Cui
2021-12-13 3:23 ` Ming Lei
2021-12-13 4:20 ` Dexuan Cui
2021-12-13 7:38 ` Ming Lei
2021-12-14 0:31 ` Dexuan Cui
2021-12-14 0:53 ` [dm-devel] " Ming Lei
2021-12-14 0:53 ` Ming Lei
2021-12-14 3:09 ` [dm-devel] " Dexuan Cui
2021-12-14 3:09 ` Dexuan Cui
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b80bfe9a-bece-1f32-3d2a-fb4d94b1fa8c@kernel.dk \
--to=axboe@kernel.dk \
--cc=decui@microsoft.com \
--cc=hch@lst.de \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=longli@microsoft.com \
--cc=mikelley@microsoft.com \
--cc=ming.lei@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.