RE: [PATCH V3 8/8] scsi: megaraid: improve scsi_mq performance via .host_tagset

Kashyap Desai
2018-02-28T14:58:48Z

Ming -

Quick testing on my setup - Performance slightly degraded (4-5% drop)for
megaraid_sas driver with this patch. (From 1610K IOPS it goes to 1544K)
I confirm that after applying this patch, we have #queue = #numa node.

ls -l
/sys/devices/pci0000:80/0000:80:02.0/0000:83:00.0/host10/target10:2:23/10:
2:23:0/block/sdy/mq
total 0
drwxr-xr-x. 18 root root 0 Feb 28 09:53 0
drwxr-xr-x. 18 root root 0 Feb 28 09:53 1

I would suggest to skip megaraid_sas driver changes using shared_tagset
until and unless there is obvious gain. If overall interface of using
shared_tagset is commit in kernel tree, we will investigate (megaraid_sas
driver) in future about real benefit of using it.

Without patch -

4.64% [megaraid_sas] [k] complete_cmd_fusion
3.23% [kernel] [k] irq_entries_start
3.18% [kernel] [k] _raw_spin_lock
3.06% [kernel] [k] syscall_return_via_sysret
2.74% [kernel] [k] bt_iter
2.55% [kernel] [k] scsi_queue_rq
2.21% [megaraid_sas] [k] megasas_build_io_fusion
1.80% [megaraid_sas] [k] megasas_queue_command
1.59% [kernel] [k] __audit_syscall_exit
1.55% [kernel] [k] _raw_spin_lock_irqsave
1.38% [megaraid_sas] [k] megasas_build_and_issue_cmd_fusion
1.34% [kernel] [k] do_io_submit
1.33% [kernel] [k] gup_pgd_range
1.26% [kernel] [k] scsi_softirq_done
1.20% fio [.] __fio_gettime
1.20% [kernel] [k] switch_mm_irqs_off
1.00% [megaraid_sas] [k] megasas_build_ldio_fusion
0.97% fio [.] get_io_u
0.89% [kernel] [k] lookup_ioctx
0.80% [kernel] [k] scsi_dec_host_busy
0.78% [kernel] [k] blkdev_direct_IO
0.78% [megaraid_sas] [k] MR_GetPhyParams
0.73% [kernel] [k] aio_read_events
0.70% [megaraid_sas] [k] MR_BuildRaidContext
0.64% [kernel] [k] blk_mq_complete_request
0.64% fio [.] thread_main
0.63% [kernel] [k] blk_queue_split
0.63% [kernel] [k] blk_mq_get_request
0.61% [kernel] [k] read_tsc
0.59% [kernel] [k] kmem_cache_a

With patch -

4.36% [megaraid_sas] [k] complete_cmd_fusion
3.24% [kernel] [k] irq_entries_start
3.00% [kernel] [k] syscall_return_via_sysret
2.41% [kernel] [k] scsi_queue_rq
2.41% [kernel] [k] _raw_spin_lock
2.22% [megaraid_sas] [k] megasas_build_io_fusion
1.92% [kernel] [k] bt_iter
1.74% [megaraid_sas] [k] megasas_queue_command
1.48% [kernel] [k] gup_pgd_range
1.44% [kernel] [k] __audit_syscall_exit
1.33% [megaraid_sas] [k] megasas_build_and_issue_cmd_fusion
1.29% [kernel] [k] _raw_spin_lock_irqsave
1.25% fio [.] get_io_u
1.24% fio [.] __fio_gettime
1.22% [kernel] [k] do_io_submit
1.18% [megaraid_sas] [k] megasas_build_ldio_fusion
1.02% [kernel] [k] blk_mq_get_request
0.91% [kernel] [k] lookup_ioctx
0.91% [kernel] [k] scsi_softirq_done
0.88% [kernel] [k] scsi_dec_host_busy
0.87% [kernel] [k] blkdev_direct_IO
0.77% [megaraid_sas] [k] MR_BuildRaidContext
0.76% [megaraid_sas] [k] MR_GetPhyParams
0.73% [kernel] [k] __fget
0.70% [kernel] [k] switch_mm_irqs_off
0.70% fio [.] thread_main
0.69% [kernel] [k] aio_read_events
0.68% [kernel] [k] note_interrupt
0.65% [kernel] [k] do_syscal

Kashyap

> -----Original Message-----
> From: Ming Lei [mailto:ming.lei@redhat.com]
> Sent: Tuesday, February 27, 2018 3:38 PM
> To: Jens Axboe; linux-block@vger.kernel.org; Christoph Hellwig; Mike
Snitzer
> Cc: linux-scsi@vger.kernel.org; Hannes Reinecke; Arun Easi; Omar
Sandoval;
> Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace;
Kashyap
> Desai; Peter Rivera; Laurence Oberman; Ming Lei
> Subject: [PATCH V3 8/8] scsi: megaraid: improve scsi_mq performance via
> .host_tagset
>
> It is observed on null_blk that IOPS can be improved much by simply
making
> hw queue per NUMA node, so this patch applies the introduced
.host_tagset
> for improving performance.
>
> In reality, .can_queue is quite big, and NUMA node number is often
small, so
> each hw queue's depth should be high enough to saturate device.
>
> Cc: Arun Easi <arun.easi@cavium.com>
> Cc: Omar Sandoval <osandov@fb.com>,
> Cc: "Martin K. Petersen" <martin.petersen@oracle.com>,
> Cc: James Bottomley <james.bottomley@hansenpartnership.com>,
> Cc: Christoph Hellwig <hch@lst.de>,
> Cc: Don Brace <don.brace@microsemi.com>
> Cc: Kashyap Desai <kashyap.desai@broadcom.com>
> Cc: Peter Rivera <peter.rivera@broadcom.com>
> Cc: Laurence Oberman <loberman@redhat.com>
> Cc: Hannes Reinecke <hare@suse.de>
> Cc: Mike Snitzer <snitzer@redhat.com>
> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> ---
> drivers/scsi/megaraid/megaraid_sas_base.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/drivers/scsi/megaraid/megaraid_sas_base.c
> b/drivers/scsi/megaraid/megaraid_sas_base.c
> index 065956cb2aeb..0b46f97cbfdb 100644
> --- a/drivers/scsi/megaraid/megaraid_sas_base.c
> +++ b/drivers/scsi/megaraid/megaraid_sas_base.c
> @@ -3177,6 +3177,7 @@ static struct scsi_host_template megasas_template
> = {
> .use_clustering = ENABLE_CLUSTERING,
> .change_queue_depth = scsi_change_queue_depth,
> .no_write_same = 1,
> + .host_tagset = 1,
> };
>
> /**
> @@ -5947,6 +5948,8 @@ static int megasas_start_aen(struct
> megasas_instance *instance) static int megasas_io_attach(struct
> megasas_instance *instance) {
> struct Scsi_Host *host = instance->host;
> + /* 256 tags should be high enough to saturate device */
> + int max_queues = DIV_ROUND_UP(host->can_queue, 256);
>
> /*
> * Export parameters required by SCSI mid-layer @@ -5987,6 +5990,9
> @@ static int megasas_io_attach(struct megasas_instance *instance)
> host->max_lun = MEGASAS_MAX_LUN;
> host->max_cmd_len = 16;
>
> + /* per NUMA node hw queue */
> + host->nr_hw_queues = min_t(int, nr_node_ids, max_queues);
> +
> /*
> * Notify the mid-layer about the new controller
> */
> --
> 2.9.5