From: Ali Gholami Rudi <aligrudi@gmail.com>
To: Yu Kuai <yukuai1@huaweicloud.com>
Cc: Xiao Ni <xni@redhat.com>,
linux-raid@vger.kernel.org, song@kernel.org,
"yukuai (C)" <yukuai3@huawei.com>
Subject: Re: Unacceptably Poor RAID1 Performance with Many CPU Cores
Date: Mon, 19 Jun 2023 08:49:45 +0330 [thread overview]
Message-ID: <20231906084945@tare.nit> (raw)
In-Reply-To: <20231906000051@laper.mirepesht>
Hi,
Yu Kuai <yukuai1@huaweicloud.com> wrote:
> Can you try to test with --bitmap=none, and --assume-clean(or echo
> frozen to sync_action), let's see if spin_lock from wait_barrier() and
> from md_bitmap_startwrite is bypassed, how performance will be.
I did not notice that it was syncing disks in the background.
It is much better now:
READ: IOPS=1748K BW=6830MiB/s
WRITE: IOPS= 749K BW=2928MiB/s
Perf's output:
+ 98.04% 0.00% fio [unknown] [k] 0xffffffffffffffff
+ 97.86% 0.00% fio fio [.] 0x000055ed33e52117
+ 94.64% 0.03% fio libc.so.6 [.] syscall
+ 94.54% 0.09% fio [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 94.44% 0.03% fio [kernel.kallsyms] [k] do_syscall_64
+ 94.31% 0.00% fio fio [.] 0x000055ed33e4fceb
+ 94.31% 0.07% fio fio [.] td_io_queue
+ 94.15% 0.03% fio fio [.] td_io_commit
+ 93.77% 0.05% fio libaio.so.1.0.2 [.] io_submit
+ 93.77% 0.00% fio fio [.] 0x000055ed33e74e5e
+ 92.03% 0.04% fio [kernel.kallsyms] [k] __x64_sys_io_submit
- 91.86% 0.05% fio [kernel.kallsyms] [k] io_submit_one
- 91.81% io_submit_one
- 52.43% aio_read
- 51.94% ext4_file_read_iter
- 36.03% iomap_dio_rw
- 36.02% __iomap_dio_rw
- 21.56% iomap_dio_bio_iter
- 17.72% submit_bio_noacct_nocheck
- 17.09% __submit_bio
- 14.81% md_handle_request
- 8.34% raid10_make_request
- 6.96% raid10_read_request
- 2.76% regular_request_wait
2.58% wait_barrier
0.76% read_balance
- 1.07% asm_common_interrupt
- 1.07% common_interrupt
- 1.06% __common_interrupt
+ 1.06% handle_edge_irq
2.22% md_submit_bio
0.51% blk_mq_submit_bio
- 10.68% iomap_iter
- 10.64% ext4_iomap_begin
- 4.07% ext4_map_blocks
- 3.80% ext4_es_lookup_extent
1.77% _raw_read_lock
1.68% _raw_read_unlock
3.51% ext4_set_iomap
+ 0.54% asm_common_interrupt
- 1.21% blk_finish_plug
- 1.20% __blk_flush_plug
- 1.19% blk_mq_flush_plug_list
- 1.15% nvme_queue_rqs
+ 1.01% nvme_prep_rq.part.0
- 6.04% down_read
- 1.73% asm_common_interrupt
- 1.73% common_interrupt
- 1.72% __common_interrupt
- 1.71% handle_edge_irq
- 1.58% handle_irq_event
- 1.57% __handle_irq_event_percpu
- nvme_irq
- 1.46% blk_mq_end_request_batch
0.80% raid10_end_write_request
0.55% raid10_end_read_request
- 5.43% up_read
+ 0.99% asm_common_interrupt
- 3.99% touch_atime
- 3.87% atime_needs_update
+ 0.68% asm_common_interrupt
- 39.00% aio_write
- 38.78% ext4_file_write_iter
- 28.16% iomap_dio_rw
- 28.15% __iomap_dio_rw
- 14.49% iomap_dio_bio_iter
- 12.65% submit_bio_noacct_nocheck
- 12.61% __submit_bio
- 11.54% md_handle_request
- 7.91% raid10_make_request
3.24% raid10_write_one_disk
- 1.21% regular_request_wait
1.13% wait_barrier
1.14% wait_blocked_dev
+ 0.57% asm_common_interrupt
1.04% md_submit_bio
- 8.05% blk_finish_plug
- 8.04% __blk_flush_plug
- 6.42% raid10_unplug
- 4.19% __wake_up_common_lock
- 3.60% _raw_spin_lock_irqsave
3.40% native_queued_spin_lock_slowpath
0.59% _raw_spin_unlock_irqrestore
- 1.00% submit_bio_noacct_nocheck
+ 0.93% blk_mq_submit_bio
- 1.59% blk_mq_flush_plug_list
- 1.02% blk_mq_sched_insert_requests
+ 0.98% blk_mq_try_issue_list_directly
- 4.53% iomap_iter
- 4.51% ext4_iomap_overwrite_begin
- 4.51% ext4_iomap_begin
- 1.69% ext4_map_blocks
- 1.58% ext4_es_lookup_extent
0.72% _raw_read_lock
0.71% _raw_read_unlock
1.54% ext4_set_iomap
- 2.83% up_read
+ 0.92% asm_common_interrupt
- 2.26% down_read
+ 0.57% asm_common_interrupt
- 1.81% ext4_map_blocks
- 1.66% ext4_es_lookup_extent
0.81% _raw_read_lock
0.70% _raw_read_unlock
- 1.67% file_modified_flags
1.43% inode_needs_update_time.part.0
+ 64.19% 0.02% fio [kernel.kallsyms] [k] iomap_dio_rw
+ 64.17% 2.41% fio [kernel.kallsyms] [k] __iomap_dio_rw
+ 52.43% 0.03% fio [kernel.kallsyms] [k] aio_read
+ 51.94% 0.04% fio [kernel.kallsyms] [k] ext4_file_read_iter
Thanks,
Ali
next prev parent reply other threads:[~2023-06-19 5:36 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-15 7:54 Unacceptably Poor RAID1 Performance with Many CPU Cores Ali Gholami Rudi
2023-06-15 9:16 ` Xiao Ni
2023-06-15 17:08 ` Ali Gholami Rudi
2023-06-15 17:36 ` Ali Gholami Rudi
2023-06-16 1:53 ` Xiao Ni
2023-06-16 5:20 ` Ali Gholami Rudi
2023-06-15 14:02 ` Yu Kuai
2023-06-16 2:14 ` Xiao Ni
2023-06-16 2:34 ` Yu Kuai
2023-06-16 5:52 ` Ali Gholami Rudi
[not found] ` <20231606091224@laper.mirepesht>
2023-06-16 7:31 ` Ali Gholami Rudi
2023-06-16 7:42 ` Yu Kuai
2023-06-16 8:21 ` Ali Gholami Rudi
2023-06-16 8:34 ` Yu Kuai
2023-06-16 8:52 ` Ali Gholami Rudi
2023-06-16 9:17 ` Yu Kuai
2023-06-16 11:51 ` Ali Gholami Rudi
2023-06-16 12:27 ` Yu Kuai
2023-06-18 20:30 ` Ali Gholami Rudi
2023-06-19 1:22 ` Yu Kuai
2023-06-19 5:19 ` Ali Gholami Rudi [this message]
2023-06-19 6:53 ` Yu Kuai
2023-06-21 8:05 ` Xiao Ni
2023-06-21 8:26 ` Yu Kuai
2023-06-21 8:55 ` Xiao Ni
2023-07-01 11:17 ` Ali Gholami Rudi
2023-07-03 12:39 ` Yu Kuai
2023-07-05 7:59 ` Ali Gholami Rudi
2023-06-21 19:34 ` Wols Lists
2023-06-23 0:52 ` Xiao Ni
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20231906084945@tare.nit \
--to=aligrudi@gmail.com \
--cc=linux-raid@vger.kernel.org \
--cc=song@kernel.org \
--cc=xni@redhat.com \
--cc=yukuai1@huaweicloud.com \
--cc=yukuai3@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).