From: Ali Gholami Rudi <aligrudi@gmail.com>
To: Yu Kuai <yukuai1@huaweicloud.com>
Cc: Xiao Ni <xni@redhat.com>,
linux-raid@vger.kernel.org, song@kernel.org,
"yukuai (C)" <yukuai3@huawei.com>
Subject: Re: Unacceptably Poor RAID1 Performance with Many CPU Cores
Date: Fri, 16 Jun 2023 15:21:06 +0330 [thread overview]
Message-ID: <20231606152106@laper.mirepesht> (raw)
In-Reply-To: <20231606122233@laper.mirepesht>
Ali Gholami Rudi <aligrudi@gmail.com> wrote:
> Yu Kuai <yukuai1@huaweicloud.com> wrote:
> > > diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> > > index 4fcfcb350d2b..52f0c24128ff 100644
> > > --- a/drivers/md/raid10.c
> > > +++ b/drivers/md/raid10.c
> > > @@ -905,7 +905,7 @@ static void flush_pending_writes(struct r10conf *conf)
> > > /* flush any pending bitmap writes to disk
> > > * before proceeding w/ I/O */
> > > md_bitmap_unplug(conf->mddev->bitmap);
> > > - wake_up(&conf->wait_barrier);
> > > + wake_up_barrier(conf);
> > >
> > > while (bio) { /* submit pending writes */
> > > struct bio *next = bio->bi_next;
> >
> > Thanks for the testing, sorry that I missed one place... Can you try to
> > change wake_up() to wake_up_barrier() from raid10_unplug() and test
> > again?
>
> OK. I replaced the second occurrence of wake_up().
>
> > > Without the patch:
> > > READ: IOPS=2033k BW=8329MB/s
> > > WRITE: IOPS= 871k BW=3569MB/s
> > >
> > > With the patch:
> > > READ: IOPS=2027K BW=7920MiB/s
> > > WRITE: IOPS= 869K BW=3394MiB/s
>
> With the second patch:
> READ: IOPS=3642K BW=13900MiB/s
> WRITE: IOPS=1561K BW= 6097MiB/s
>
> That is impressive. Great job.
Perf's output:
+ 93.79% 0.09% fio [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 92.89% 0.05% fio [kernel.kallsyms] [k] do_syscall_64
+ 86.59% 0.07% fio [kernel.kallsyms] [k] __x64_sys_io_submit
- 85.61% 0.10% fio [kernel.kallsyms] [k] io_submit_one
- 85.51% io_submit_one
- 47.98% aio_read
- 46.18% blkdev_read_iter
- 44.90% __blkdev_direct_IO_async
- 41.68% submit_bio_noacct_nocheck
- 41.50% __submit_bio
- 18.76% md_handle_request
- 18.71% raid10_make_request
- 18.54% raid10_read_request
16.54% read_balance
1.80% wait_barrier_nolock
- 14.18% raid10_end_read_request
- 8.16% raid_end_bio_io
7.44% allow_barrier
- 8.40% brd_submit_bio
2.49% __radix_tree_lookup
- 1.39% bio_iov_iter_get_pages
- 1.04% iov_iter_get_pages
- __iov_iter_get_pages_alloc
- 0.93% get_user_pages_fast
0.79% internal_get_user_pages_fast
- 1.17% bio_alloc_bioset
0.59% mempool_alloc
0.93% filemap_write_and_wait_range
0.65% security_file_permission
- 35.21% aio_write
- 34.53% blkdev_write_iter
- 18.25% __generic_file_write_iter
- 17.84% generic_file_direct_write
- 17.35% __blkdev_direct_IO_async
- 16.34% submit_bio_noacct_nocheck
- 16.31% __submit_bio
- 16.28% md_handle_request
- 16.26% raid10_make_request
9.26% raid10_write_one_disk
2.11% wait_blocked_dev
0.58% wait_barrier_nolock
- 16.02% blk_finish_plug
- 16.01% flush_plug_callbacks
- 16.00% raid10_unplug
- 15.89% submit_bio_noacct_nocheck
- 15.84% __submit_bio
- 8.66% raid10_end_write_request
- 3.55% raid_end_bio_io
3.54% allow_barrier
0.72% find_bio_disk.isra.0
- 7.04% brd_submit_bio
1.38% __radix_tree_lookup
0.61% kmem_cache_alloc
+ 84.41% 0.99% fio fio [.] thread_main
+ 83.79% 0.00% fio [unknown] [.] 0xffffffffffffffff
+ 83.60% 0.00% fio fio [.] run_threads
+ 83.32% 0.00% fio fio [.] do_io (inlined)
+ 81.12% 0.43% fio libc-2.31.so [.] syscall
+ 76.23% 0.69% fio fio [.] td_io_queue
+ 76.16% 4.66% fio [kernel.kallsyms] [k] submit_bio_noacct_nocheck
+ 75.63% 0.25% fio fio [.] fio_libaio_commit
+ 75.57% 0.17% fio fio [.] td_io_commit
+ 75.54% 0.00% fio fio [.] io_u_submit (inlined)
+ 75.33% 0.17% fio libaio.so.1.0.1 [.] io_submit
+ 73.66% 0.07% fio [kernel.kallsyms] [k] __submit_bio
+ 67.30% 5.07% fio [kernel.kallsyms] [k] __blkdev_direct_IO_async
+ 48.02% 0.03% fio [kernel.kallsyms] [k] aio_read
+ 46.22% 0.05% fio [kernel.kallsyms] [k] blkdev_read_iter
+ 35.71% 3.88% fio [kernel.kallsyms] [k] raid10_make_request
+ 35.23% 0.02% fio [kernel.kallsyms] [k] aio_write
+ 35.08% 0.06% fio [kernel.kallsyms] [k] md_handle_request
+ 34.55% 0.02% fio [kernel.kallsyms] [k] blkdev_write_iter
+ 20.16% 1.65% fio [kernel.kallsyms] [k] raid10_read_request
+ 18.27% 0.01% fio [kernel.kallsyms] [k] __generic_file_write_iter
+ 18.02% 3.63% fio [kernel.kallsyms] [k] brd_submit_bio
+ 17.86% 0.02% fio [kernel.kallsyms] [k] generic_file_direct_write
+ 17.08% 11.16% fio [kernel.kallsyms] [k] read_balance
+ 16.24% 0.33% fio [kernel.kallsyms] [k] raid10_unplug
+ 16.02% 0.01% fio [kernel.kallsyms] [k] blk_finish_plug
+ 16.02% 0.01% fio [kernel.kallsyms] [k] flush_plug_callbacks
+ 14.25% 0.26% fio [kernel.kallsyms] [k] raid10_end_read_request
+ 12.77% 1.98% fio [kernel.kallsyms] [k] allow_barrier
+ 11.74% 0.40% fio [kernel.kallsyms] [k] raid_end_bio_io
+ 10.21% 1.99% fio [kernel.kallsyms] [k] raid10_write_one_disk
+ 8.85% 1.52% fio [kernel.kallsyms] [k] raid10_end_write_request
8.06% 6.43% fio [kernel.kallsyms] [k] wait_barrier_nolock
..
Thanks,
Ali
next prev parent reply other threads:[~2023-06-16 11:52 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-15 7:54 Unacceptably Poor RAID1 Performance with Many CPU Cores Ali Gholami Rudi
2023-06-15 9:16 ` Xiao Ni
2023-06-15 17:08 ` Ali Gholami Rudi
2023-06-15 17:36 ` Ali Gholami Rudi
2023-06-16 1:53 ` Xiao Ni
2023-06-16 5:20 ` Ali Gholami Rudi
2023-06-15 14:02 ` Yu Kuai
2023-06-16 2:14 ` Xiao Ni
2023-06-16 2:34 ` Yu Kuai
2023-06-16 5:52 ` Ali Gholami Rudi
[not found] ` <20231606091224@laper.mirepesht>
2023-06-16 7:31 ` Ali Gholami Rudi
2023-06-16 7:42 ` Yu Kuai
2023-06-16 8:21 ` Ali Gholami Rudi
2023-06-16 8:34 ` Yu Kuai
2023-06-16 8:52 ` Ali Gholami Rudi
2023-06-16 9:17 ` Yu Kuai
2023-06-16 11:51 ` Ali Gholami Rudi [this message]
2023-06-16 12:27 ` Yu Kuai
2023-06-18 20:30 ` Ali Gholami Rudi
2023-06-19 1:22 ` Yu Kuai
2023-06-19 5:19 ` Ali Gholami Rudi
2023-06-19 6:53 ` Yu Kuai
2023-06-21 8:05 ` Xiao Ni
2023-06-21 8:26 ` Yu Kuai
2023-06-21 8:55 ` Xiao Ni
2023-07-01 11:17 ` Ali Gholami Rudi
2023-07-03 12:39 ` Yu Kuai
2023-07-05 7:59 ` Ali Gholami Rudi
2023-06-21 19:34 ` Wols Lists
2023-06-23 0:52 ` Xiao Ni
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20231606152106@laper.mirepesht \
--to=aligrudi@gmail.com \
--cc=linux-raid@vger.kernel.org \
--cc=song@kernel.org \
--cc=xni@redhat.com \
--cc=yukuai1@huaweicloud.com \
--cc=yukuai3@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).