Re: Unacceptably Poor RAID1 Performance with Many CPU Cores

[Ali Gholami Rudi <aligrudi@gmail.com>]

Hi,

Yu Kuai <yukuai1@huaweicloud.com> wrote:
> > And this is perf's output for raid10:
> > 
> > +   97.33%     0.04%  fio      [kernel.kallsyms]       [k] entry_SYSCALL_64_after_hwframe
> > +   96.96%     0.02%  fio      [kernel.kallsyms]       [k] do_syscall_64
> > +   94.43%     0.03%  fio      [kernel.kallsyms]       [k] __x64_sys_io_submit
> > -   93.71%     0.04%  fio      [kernel.kallsyms]       [k] io_submit_one
> >     - 93.67% io_submit_one
> >        - 76.03% aio_write
> >           - 75.53% blkdev_write_iter
> >              - 68.95% blk_finish_plug
> >                 - flush_plug_callbacks
> >                    - 68.93% raid10_unplug
> >                       - 64.31% __wake_up_common_lock
> >                          - 64.17% _raw_spin_lock_irqsave
> >                               native_queued_spin_lock_slowpath
> 
> This is unexpected, can you check if your kernel contain following
> patch?
> 
> commit 460af1f9d9e62acce4a21f9bd00b5bcd5963bcd4
> Author: Yu Kuai <yukuai3@huawei.com>
> Date:   Mon May 29 21:11:06 2023 +0800
> 
>      md/raid1-10: limit the number of plugged bio
> 
> If so, can you retest with following patch?
> 
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index d0de8c9fb3cf..6fdd99c3e59a 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -911,7 +911,7 @@ static void flush_pending_writes(struct r10conf *conf)
> 
>                  blk_start_plug(&plug);
>                  raid1_prepare_flush_writes(conf->mddev->bitmap);
> -               wake_up(&conf->wait_barrier);
> +               wake_up_barrier(&conf->wait_barrier);
> 
>                  while (bio) { /* submit pending writes */
>                          struct bio *next = bio->bi_next;

No, this patch was not present.  I applied this one:

diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 4fcfcb350d2b..52f0c24128ff 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -905,7 +905,7 @@ static void flush_pending_writes(struct r10conf *conf)
 		/* flush any pending bitmap writes to disk
 		 * before proceeding w/ I/O */
 		md_bitmap_unplug(conf->mddev->bitmap);
-		wake_up(&conf->wait_barrier);
+		wake_up_barrier(conf);
 
 		while (bio) { /* submit pending writes */
 			struct bio *next = bio->bi_next;

I get almost the same result as before nevertheless:

Without the patch:
READ:  IOPS=2033k BW=8329MB/s
WRITE: IOPS= 871k BW=3569MB/s

With the patch:
READ:  IOPS=2027K BW=7920MiB/s
WRITE: IOPS= 869K BW=3394MiB/s

Perf:

+   96.23%     0.04%  fio      [kernel.kallsyms]       [k] entry_SYSCALL_64_after_hwframe
+   95.86%     0.02%  fio      [kernel.kallsyms]       [k] do_syscall_64
+   94.30%     0.03%  fio      [kernel.kallsyms]       [k] __x64_sys_io_submit
-   93.63%     0.04%  fio      [kernel.kallsyms]       [k] io_submit_one
   - 93.58% io_submit_one
      - 76.44% aio_write
         - 75.97% blkdev_write_iter
            - 70.17% blk_finish_plug
               - flush_plug_callbacks
                  - 70.15% raid10_unplug
                     - 66.12% __wake_up_common_lock
                        - 65.97% _raw_spin_lock_irqsave
                             65.57% native_queued_spin_lock_slowpath
                     - 3.85% submit_bio_noacct_nocheck
                        - 3.84% __submit_bio
                           - 2.09% raid10_end_write_request
                              - 0.83% raid_end_bio_io
                                   0.82% allow_barrier
                             1.70% brd_submit_bio
            + 5.59% __generic_file_write_iter
      + 15.71% aio_read
+   88.38%     0.71%  fio      fio                     [.] thread_main
+   87.89%     0.00%  fio      [unknown]               [k] 0xffffffffffffffff
+   87.81%     0.00%  fio      fio                     [.] run_threads
+   87.54%     0.00%  fio      fio                     [.] do_io (inlined)
+   86.79%     0.31%  fio      libc-2.31.so            [.] syscall
+   86.19%     0.54%  fio      fio                     [.] td_io_queue
+   85.79%     0.18%  fio      fio                     [.] fio_libaio_commit
+   85.76%     0.14%  fio      fio                     [.] td_io_commit
+   85.69%     0.14%  fio      libaio.so.1.0.1         [.] io_submit
+   85.66%     0.00%  fio      fio                     [.] io_u_submit (inlined)
+   76.45%     0.01%  fio      [kernel.kallsyms]       [k] aio_write
..