Re: Unacceptably Poor RAID1 Performance with Many CPU Cores

[Yu Kuai <yukuai1@huaweicloud.com>]

Hi,

在 2023/06/15 15:54, Ali Gholami Rudi 写道:
> Perf output:
> 
> Samples: 1M of event 'cycles', Event count (approx.): 1158425235997
>    Children      Self  Command  Shared Object           Symbol
> +   97.98%     0.01%  fio      fio                     [.] fio_libaio_commit
> +   97.95%     0.01%  fio      libaio.so.1.0.1         [.] io_submit
> +   97.85%     0.01%  fio      [kernel.kallsyms]       [k] __x64_sys_io_submit
> -   97.82%     0.01%  fio      [kernel.kallsyms]       [k] io_submit_one
>     - 97.81% io_submit_one
>        - 54.62% aio_write
>           - 54.60% blkdev_write_iter
>              - 36.30% blk_finish_plug
>                 - flush_plug_callbacks
>                    - 36.29% raid1_unplug
>                       - flush_bio_list
>                          - 18.44% submit_bio_noacct
>                             - 18.40% brd_submit_bio
>                                - 18.13% raid1_end_write_request
>                                   - 17.94% raid_end_bio_io
>                                      - 17.82% __wake_up_common_lock
>                                         + 17.79% _raw_spin_lock_irqsave
>                          - 17.79% __wake_up_common_lock
>                             + 17.76% _raw_spin_lock_irqsave
>              + 18.29% __generic_file_write_iter
>        - 43.12% aio_read
>           - 43.07% blkdev_read_iter
>              - generic_file_read_iter
>                 - 43.04% blkdev_direct_IO
>                    - 42.95% submit_bio_noacct
>                       - 42.23% brd_submit_bio
>                          - 41.91% raid1_end_read_request
>                             - 41.70% raid_end_bio_io
>                                - 41.43% __wake_up_common_lock
>                                   + 41.36% _raw_spin_lock_irqsave
>                       - 0.68% md_submit_bio
>                            0.61% md_handle_request
> +   94.90%     0.00%  fio      [kernel.kallsyms]       [k] __wake_up_common_lock
> +   94.86%     0.22%  fio      [kernel.kallsyms]       [k] _raw_spin_lock_irqsave
> +   94.64%    94.64%  fio      [kernel.kallsyms]       [k] native_queued_spin_lock_slowpath
> +   79.63%     0.02%  fio      [kernel.kallsyms]       [k] submit_bio_noacct

This looks familiar... Perhaps can you try to test with raid10 with
latest mainline kernel? I used to optimize spin_lock for raid10, and I
don't do this for raid1 yet... I can try to do the same thing for raid1
if it's valuable.

> 
> 
> FIO configuration file:
> 
> [global]
> name=random reads and writes
> ioengine=libaio
> direct=1
> readwrite=randrw
> rwmixread=70
> iodepth=64
> buffered=0
> #filename=/dev/ram0
> filename=/dev/dm/test
> size=1G
> runtime=30
> time_based
> randrepeat=0
> norandommap
> refill_buffers
> ramp_time=10
> bs=4k
> numjobs=400

400 is too aggressive, I think spin_lock from fast path is probably
causing the problem, same as I met before for raid10...

Thanks,
Kuai

> group_reporting=1
> [job1]
> 
> .
>