Re: Unacceptably Poor RAID1 Performance with Many CPU Cores

[Ali Gholami Rudi <aligrudi@gmail.com>]

Ali Gholami Rudi <aligrudi@gmail.com> wrote:
> Xiao Ni <xni@redhat.com> wrote:
> > Because it can be reproduced easily in your environment. Can you try
> > with the latest upstream kernel? If the problem doesn't exist with
> > latest upstream kernel. You can use git bisect to find which patch can
> > fix this problem.
> 
> I just tried the upstream.  I get almost the same result with 1G ramdisks.
> 
> Without RAID (writing to /dev/ram0)
> READ:  IOPS=15.8M BW=60.3GiB/s
> WRITE: IOPS= 6.8M BW=27.7GiB/s
> 
> RAID1 (writing to /dev/md/test)
> READ:  IOPS=518K BW=2028MiB/s
> WRITE: IOPS=222K BW= 912MiB/s

And this is perf's output:

+   98.73%     0.01%  fio      [kernel.kallsyms]       [k] entry_SYSCALL_64_after_hwframe
+   98.63%     0.01%  fio      [kernel.kallsyms]       [k] do_syscall_64
+   97.28%     0.01%  fio      [kernel.kallsyms]       [k] __x64_sys_io_submit
-   97.09%     0.01%  fio      [kernel.kallsyms]       [k] io_submit_one
   - 97.08% io_submit_one
      - 53.58% aio_write
         - 53.42% blkdev_write_iter
            - 35.28% blk_finish_plug
               - flush_plug_callbacks
                  - 35.27% raid1_unplug
                     - flush_bio_list
                        - 17.88% submit_bio_noacct_nocheck
                           - 17.88% __submit_bio
                              - 17.61% raid1_end_write_request
                                 - 17.47% raid_end_bio_io
                                    - 17.41% __wake_up_common_lock
                                       - 17.38% _raw_spin_lock_irqsave
                                            native_queued_spin_lock_slowpath
                        - 17.35% __wake_up_common_lock
                           - 17.31% _raw_spin_lock_irqsave
                                native_queued_spin_lock_slowpath
            + 18.07% __generic_file_write_iter
      - 43.00% aio_read
         - 42.64% blkdev_read_iter
            - 42.37% __blkdev_direct_IO_async
               - 41.40% submit_bio_noacct_nocheck
                  - 41.34% __submit_bio
                     - 40.68% raid1_end_read_request
                        - 40.55% raid_end_bio_io
                           - 40.35% __wake_up_common_lock
                              - 40.28% _raw_spin_lock_irqsave
                                   native_queued_spin_lock_slowpath
+   95.19%     0.32%  fio      fio                     [.] thread_main
+   95.08%     0.00%  fio      [unknown]               [.] 0xffffffffffffffff
+   95.03%     0.00%  fio      fio                     [.] run_threads
+   94.77%     0.00%  fio      fio                     [.] do_io (inlined)
+   94.65%     0.16%  fio      fio                     [.] td_io_queue
+   94.65%     0.11%  fio      libc-2.31.so            [.] syscall
+   94.54%     0.07%  fio      fio                     [.] fio_libaio_commit
+   94.53%     0.05%  fio      fio                     [.] td_io_commit
+   94.50%     0.00%  fio      fio                     [.] io_u_submit (inlined)
+   94.47%     0.04%  fio      libaio.so.1.0.1         [.] io_submit
+   92.48%     0.02%  fio      [kernel.kallsyms]       [k] _raw_spin_lock_irqsave
+   92.48%     0.00%  fio      [kernel.kallsyms]       [k] __wake_up_common_lock
+   92.46%    92.32%  fio      [kernel.kallsyms]       [k] native_queued_spin_lock_slowpath
+   76.85%     0.03%  fio      [kernel.kallsyms]       [k] submit_bio_noacct_nocheck
+   76.76%     0.00%  fio      [kernel.kallsyms]       [k] __submit_bio
+   60.25%     0.06%  fio      [kernel.kallsyms]       [k] __blkdev_direct_IO_async
+   58.12%     0.11%  fio      [kernel.kallsyms]       [k] raid_end_bio_io
..

Thanks,
Ali