Re: Unacceptably Poor RAID1 Performance with Many CPU Cores

[Ali Gholami Rudi <aligrudi@gmail.com>]

Ali Gholami Rudi <aligrudi@gmail.com> wrote:
> Xiao Ni <xni@redhat.com> wrote:
> > On Thu, Jun 15, 2023 at 10:06 PM Yu Kuai <yukuai1@huaweicloud.com> wrote:
> > > This looks familiar... Perhaps can you try to test with raid10 with
> > > latest mainline kernel? I used to optimize spin_lock for raid10, and I
> > > don't do this for raid1 yet... I can try to do the same thing for raid1
> > > if it's valuable.
> 
> I do get improvements with raid10:
> 
> Without RAID (writing to /dev/ram0)
> READ:  IOPS=15.8M BW=60.3GiB/s
> WRITE: IOPS= 6.8M BW=27.7GiB/s
> 
> RAID1 (writing to /dev/md/test)
> READ:  IOPS=518K BW=2028MiB/s
> WRITE: IOPS=222K BW= 912MiB/s
> 
> RAID10 (writing to /dev/md/test)
> READ:  IOPS=2033k BW=8329MB/s
> WRITE: IOPS= 871k BW=3569MB/s
> 
> raid10 is about four times faster than raid1.

And this is perf's output for raid10:

+   97.33%     0.04%  fio      [kernel.kallsyms]       [k] entry_SYSCALL_64_after_hwframe
+   96.96%     0.02%  fio      [kernel.kallsyms]       [k] do_syscall_64
+   94.43%     0.03%  fio      [kernel.kallsyms]       [k] __x64_sys_io_submit
-   93.71%     0.04%  fio      [kernel.kallsyms]       [k] io_submit_one
   - 93.67% io_submit_one
      - 76.03% aio_write
         - 75.53% blkdev_write_iter
            - 68.95% blk_finish_plug
               - flush_plug_callbacks
                  - 68.93% raid10_unplug
                     - 64.31% __wake_up_common_lock
                        - 64.17% _raw_spin_lock_irqsave
                             native_queued_spin_lock_slowpath
                     - 4.43% submit_bio_noacct_nocheck
                        - 4.42% __submit_bio
                           - 2.28% raid10_end_write_request
                              - 0.82% raid_end_bio_io
                                   0.82% allow_barrier
                             2.09% brd_submit_bio
            - 6.41% __generic_file_write_iter
               - 6.08% generic_file_direct_write
                  - 5.64% __blkdev_direct_IO_async
                     - 4.72% submit_bio_noacct_nocheck
                        - 4.69% __submit_bio
                           - 4.67% md_handle_request
                              - 4.66% raid10_make_request
                                   2.59% raid10_write_one_disk
      - 16.14% aio_read
         - 15.07% blkdev_read_iter
            - 14.16% __blkdev_direct_IO_async
               - 11.36% submit_bio_noacct_nocheck
                  - 11.17% __submit_bio
                     - 5.89% md_handle_request
                        - 5.84% raid10_make_request
                           + 4.18% raid10_read_request
                     - 3.74% raid10_end_read_request
                        - 2.04% raid_end_bio_io
                             1.46% allow_barrier
                          0.55% mempool_free
                       1.39% brd_submit_bio
               - 1.33% bio_iov_iter_get_pages
                  - 1.00% iov_iter_get_pages
                     - __iov_iter_get_pages_alloc
                        - 0.85% get_user_pages_fast
                             0.75% internal_get_user_pages_fast
                 0.93% bio_alloc_bioset
              0.65% filemap_write_and_wait_range
+   88.31%     0.86%  fio      fio                     [.] thread_main
+   87.69%     0.00%  fio      [unknown]               [k] 0xffffffffffffffff
+   87.60%     0.00%  fio      fio                     [.] run_threads
+   87.31%     0.00%  fio      fio                     [.] do_io (inlined)
+   86.60%     0.32%  fio      libc-2.31.so            [.] syscall
+   85.87%     0.52%  fio      fio                     [.] td_io_queue
+   85.49%     0.18%  fio      fio                     [.] fio_libaio_commit
+   85.45%     0.14%  fio      fio                     [.] td_io_commit
+   85.37%     0.11%  fio      libaio.so.1.0.1         [.] io_submit
+   85.35%     0.00%  fio      fio                     [.] io_u_submit (inlined)
+   76.04%     0.01%  fio      [kernel.kallsyms]       [k] aio_write
+   75.54%     0.01%  fio      [kernel.kallsyms]       [k] blkdev_write_iter
+   68.96%     0.00%  fio      [kernel.kallsyms]       [k] blk_finish_plug
+   68.95%     0.00%  fio      [kernel.kallsyms]       [k] flush_plug_callbacks
+   68.94%     0.13%  fio      [kernel.kallsyms]       [k] raid10_unplug
+   64.41%     0.03%  fio      [kernel.kallsyms]       [k] _raw_spin_lock_irqsave
+   64.32%     0.01%  fio      [kernel.kallsyms]       [k] __wake_up_common_lock
+   64.05%    63.85%  fio      [kernel.kallsyms]       [k] native_queued_spin_lock_slowpath
+   21.05%     1.51%  fio      [kernel.kallsyms]       [k] submit_bio_noacct_nocheck
+   20.97%     1.18%  fio      [kernel.kallsyms]       [k] __blkdev_direct_IO_async
+   20.29%     0.03%  fio      [kernel.kallsyms]       [k] __submit_bio
+   16.15%     0.02%  fio      [kernel.kallsyms]       [k] aio_read
..

Thanks,
Ali