[PATCH]block: Observing higher CPU utilization during random IO testing

[PATCH]block: Observing higher CPU utilization during random IO testing

[wenxiong]

From: Wen Xiong <wenxiong@linux.ibm.com>

Hi All,

Our performance team observed the higher CPU utilization in RHEL10 compared
to RHEL9.8, observed the similar issue in upstream kernel(v7.1-rc6) as well
when running FIO random IO tests. Random IO tests are more CPU intensive
than sequential IO tests due to several factors: more context switching,
interrupt Handling, cache Inefficiency etc.

Given commit 060406c61c7c ("block: add plug while submitting IO")
causes performance regression. This patch reverts it.

Below is performance comparison with the latest upstream kernel.

Iotype  qd   nj    rmix    mpstat busy        mpstat busy
                         with inner plug    without inner plug
Randrw  1    20    100       53%                 24%
Randrw  1    40    100       70%                 24%
Randrw  1    20    70        40%                 24%
Randrw  1    40    70        60%                 26%
Randrw  1    20    0         14%                 6%
Randrw  1    40    0         20%                 7%

Signed-off-by: Wen Xiong <wenxiong@linux.ibm.com>
Suggested-by: Ming Lei <tom.leiming@gmail.com>
---
 block/blk-core.c | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/block/blk-core.c b/block/blk-core.c
index 17450058ea6d..ee87edf166b6 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -626,11 +626,6 @@ static inline blk_status_t blk_check_zone_append(struct request_queue *q,
 
 static void __submit_bio(struct bio *bio)
 {
-	/* If plug is not used, add new plug here to cache nsecs time. */
-	struct blk_plug plug;
-
-	blk_start_plug(&plug);
-
 	if (!bdev_test_flag(bio->bi_bdev, BD_HAS_SUBMIT_BIO)) {
 		blk_mq_submit_bio(bio);
 	} else if (likely(bio_queue_enter(bio) == 0)) {
@@ -645,8 +640,6 @@ static void __submit_bio(struct bio *bio)
 		}
 		blk_queue_exit(disk->queue);
 	}
-
-	blk_finish_plug(&plug);
 }
 
 /*
-- 
2.52.0

Re: [PATCH]block: Observing higher CPU utilization during random IO testing

[Ming Lei]

On Thu, Jun 04, 2026 at 12:27:09PM -0400, wenxiong@linux.ibm.com wrote:
> From: Wen Xiong <wenxiong@linux.ibm.com>
> 
> Hi All,
> 
> Our performance team observed the higher CPU utilization in RHEL10 compared
> to RHEL9.8, observed the similar issue in upstream kernel(v7.1-rc6) as well
> when running FIO random IO tests. Random IO tests are more CPU intensive
> than sequential IO tests due to several factors: more context switching,
> interrupt Handling, cache Inefficiency etc.
> 
> Given commit 060406c61c7c ("block: add plug while submitting IO")
> causes performance regression. This patch reverts it.

One thing is that plugging should be applied in outer io code path, also
it doesn't make sense to apply this timestamp optimization for single IO batch.

> 
> Below is performance comparison with the latest upstream kernel.
> 
> Iotype  qd   nj    rmix    mpstat busy        mpstat busy
>                          with inner plug    without inner plug
> Randrw  1    20    100       53%                 24%
> Randrw  1    40    100       70%                 24%
> Randrw  1    20    70        40%                 24%
> Randrw  1    40    70        60%                 26%
> Randrw  1    20    0         14%                 6%
> Randrw  1    40    0         20%                 7%
> 
> Signed-off-by: Wen Xiong <wenxiong@linux.ibm.com>
> Suggested-by: Ming Lei <tom.leiming@gmail.com>

With commit log update, this patch looks fine:

Reviewed-by: Ming Lei <tom.leiming@gmail.com>


Thanks,
Ming

Re: [PATCH]block: Observing higher CPU utilization during random IO testing

[Jens Axboe]

On 6/4/26 10:27 AM, wenxiong@linux.ibm.com wrote:
> From: Wen Xiong <wenxiong@linux.ibm.com>
> 
> Hi All,
> 
> Our performance team observed the higher CPU utilization in RHEL10 compared
> to RHEL9.8, observed the similar issue in upstream kernel(v7.1-rc6) as well
> when running FIO random IO tests. Random IO tests are more CPU intensive
> than sequential IO tests due to several factors: more context switching,
> interrupt Handling, cache Inefficiency etc.
> 
> Given commit 060406c61c7c ("block: add plug while submitting IO")
> causes performance regression. This patch reverts it.
> 
> Below is performance comparison with the latest upstream kernel.
> 
> Iotype  qd   nj    rmix    mpstat busy        mpstat busy
>                          with inner plug    without inner plug
> Randrw  1    20    100       53%                 24%
> Randrw  1    40    100       70%                 24%
> Randrw  1    20    70        40%                 24%
> Randrw  1    40    70        60%                 26%
> Randrw  1    20    0         14%                 6%
> Randrw  1    40    0         20%                 7%

I'm fine with the actual change, but the commit title and commit message
are pretty abysmal. If you were browsing git log and came across
something that said

"block: Observing higher CPU utilization during random IO testing"

then what would you think the change did? You would have no idea. That
title is a bug report title, it's not a git commit title. A good git
commit message should tell you WHY a change is being made, and the title
should be a short summary of that change.

I'll rewrite these, but please keep this in mind for future submissions.
Should not be necessary for me to completely rewrite your git commit
message.

You also identify 060406c61c7c as the commit that this is fixing. This
goes into a Fixes line before your signed-off-by. And you would
presumably want this to go into stable as well, yet that's not tagged.
Please read the documentation on how to submit patches.

-- 
Jens Axboe

Re: [PATCH]block: Observing higher CPU utilization during random IO testing

[Jens Axboe]

On 6/8/26 9:15 AM, Jens Axboe wrote:
> On 6/4/26 10:27 AM, wenxiong@linux.ibm.com wrote:
>> From: Wen Xiong <wenxiong@linux.ibm.com>
>>
>> Hi All,
>>
>> Our performance team observed the higher CPU utilization in RHEL10 compared
>> to RHEL9.8, observed the similar issue in upstream kernel(v7.1-rc6) as well
>> when running FIO random IO tests. Random IO tests are more CPU intensive
>> than sequential IO tests due to several factors: more context switching,
>> interrupt Handling, cache Inefficiency etc.
>>
>> Given commit 060406c61c7c ("block: add plug while submitting IO")
>> causes performance regression. This patch reverts it.
>>
>> Below is performance comparison with the latest upstream kernel.
>>
>> Iotype  qd   nj    rmix    mpstat busy        mpstat busy
>>                          with inner plug    without inner plug
>> Randrw  1    20    100       53%                 24%
>> Randrw  1    40    100       70%                 24%
>> Randrw  1    20    70        40%                 24%
>> Randrw  1    40    70        60%                 26%
>> Randrw  1    20    0         14%                 6%
>> Randrw  1    40    0         20%                 7%
> 
> I'm fine with the actual change, but the commit title and commit message
> are pretty abysmal. If you were browsing git log and came across
> something that said
> 
> "block: Observing higher CPU utilization during random IO testing"
> 
> then what would you think the change did? You would have no idea. That
> title is a bug report title, it's not a git commit title. A good git
> commit message should tell you WHY a change is being made, and the title
> should be a short summary of that change.
> 
> I'll rewrite these, but please keep this in mind for future submissions.
> Should not be necessary for me to completely rewrite your git commit
> message.
> 
> You also identify 060406c61c7c as the commit that this is fixing. This
> goes into a Fixes line before your signed-off-by. And you would
> presumably want this to go into stable as well, yet that's not tagged.
> Please read the documentation on how to submit patches.

Patch also doesn't apply without fuzz. Please submit a v2 that's against
for-7.2/block and heeds the above advice to turn this into something
that's closer to a good patch submission.

-- 
Jens Axboe