From: Wen Xiong <wenxiong@linux.ibm.com>
To: yukuai@fygo.io
Cc: Jens Axboe <axboe@kernel.dk>,
linux-block@vger.kernel.org, tom.leiming@gmail.com,
jmoyer@redhat.com, Gjoyce <gjoyce@linux.ibm.com>,
wenxiong@us.ibm.com
Subject: Re: Observing higher CPU utilization during random IO fio testing
Date: Fri, 29 May 2026 12:13:49 -0500 [thread overview]
Message-ID: <f7021a9b85cf1aac489d1a07b4fdaaf9@linux.ibm.com> (raw)
In-Reply-To: <043e357f-5b37-4e05-9433-271504fc1d30@fygo.io>
On 2026-05-25 00:28, Yu Kuai wrote:
> 在 2026/5/22 5:52, Jens Axboe 写道:
> Yes, perf data will be helpful. And please show your test in details
> and I'll
> check if I can reproduce it.
Hi Yu Kuai,
Have you reproduced the issue yet?
Below is some perf data we took while running random read test:
Test:
FIO random read with qdepth=1 nj=20, we saw higher CPU utilization in
this testcase.
Perf record:
start fio run on one session and kickoff the script in another session
while test is running
Perf report:
With blk_start_plug/blk_finish_plug before calling __submit_bio() in
blk-core.c:
Top.txt
2.41% fio [kernel.kallsyms]
[k] cpupri_set
1.16% fio [kernel.kallsyms]
[k] queued_spin_lock_slowpath
0.75% fio [kernel.kallsyms]
[k] sbitmap_find_bit
0.47% fio [kernel.kallsyms]
[k] set_next_task_rt
0.41% fio [kernel.kallsyms]
[k] pull_rt_task
0.34% fio [kernel.kallsyms]
[k] enqueue_pushable_task
…
0.02% fio [kernel.kallsyms]
[k] __blk_flush_plug
0.01% fio [kernel.kallsyms]
[k] blk_add_rq_to_plug
0.01% fio [kernel.kallsyms]
[k] blk_mq_flush_plug_list
0.00% fio [kernel.kallsyms]
[k] blk_attempt_plug_merge
Callgraph.txt
2.41% fio [kernel.kallsyms]
[k] cpupri_set
|
---cpupri_set
|
|--1.15%--__enqueue_rt_entity
| enqueue_task_rt
| enqueue_task
| ttwu_do_activate
Perf report
Without blk_start_plug and blk_finish_plug before calling
__submit_bio():
Top.txt
0.67% fio [kernel.kallsyms]
[k] queued_spin_lock_slowpath
0.64% fio [kernel.kallsyms]
[k] sched_balance_newidle
0.47% fio [kernel.kallsyms]
[k] _raw_spin_lock
0.39% fio [kernel.kallsyms]
[k] sbitmap_find_bit
0.35% fio [kernel.kallsyms]
[k] cpupri_set
0.28% fio [kernel.kallsyms]
[k] work_grab_pending
0.24% fio [kernel.kallsyms]
[k] lookup_ioctx
0.23% fio [kernel.kallsyms]
[k] __schedule
…
…
0.00% fio [kernel.kallsyms]
[k] blk_attempt_plug_merge
Call graph.txt:
0.35% fio [kernel.kallsyms]
[k] cpupri_set
|
---cpupri_set
|
|--0.17%--arch_local_irq_restore.part.0
| |
| |--0.14%--finish_task_switch.isra.0
| | __schedule
| | |
| | |--0.13%--schedule
| | | |
| | | |--0.07%--read_events
…..
|--0.13%--__enqueue_rt_entity
| enqueue_task_rt
| enqueue_task
| ttwu_do_activate
From above perf data, looks like
1. High time spent in cpupri_set(): tasks being enqueued/dequeued
frequently, more IO scheduling.
2. Call more plug routines.
If you need full perf data report, I can email/attach your full report.
Thanks for your help!
Wendy
next prev parent reply other threads:[~2026-05-29 17:14 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-05-21 19:44 Observing higher CPU utilization during random IO fio testing Wen Xiong
2026-05-21 21:52 ` Jens Axboe
2026-05-25 5:28 ` Yu Kuai
2026-05-26 15:28 ` Wen Xiong
2026-05-29 17:13 ` Wen Xiong [this message]
2026-05-31 11:45 ` Yu Kuai
2026-05-30 1:10 ` Ming Lei
2026-05-31 11:56 ` Yu Kuai
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f7021a9b85cf1aac489d1a07b4fdaaf9@linux.ibm.com \
--to=wenxiong@linux.ibm.com \
--cc=axboe@kernel.dk \
--cc=gjoyce@linux.ibm.com \
--cc=jmoyer@redhat.com \
--cc=linux-block@vger.kernel.org \
--cc=tom.leiming@gmail.com \
--cc=wenxiong@us.ibm.com \
--cc=yukuai@fygo.io \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox