iSER multi-queue configuration

iSER multi-queue configuration

[Ming Lin]

Hi,

I use mlx5 to setup iSER and have scsi mq enabled.

root@xxx:~# cat /sys/module/scsi_mod/parameters/use_blk_mq
Y

But it seems there is only 1 HW queue that maps to 48 SW queues.

root@xxx:/sys/block/sdb/mq# ls
0

root@xxx:/sys/block/sdb/mq# ls 0/
active  cpu1   cpu11  cpu13  cpu15  cpu17  cpu19  cpu20  cpu22  cpu24
cpu26  cpu28  cpu3   cpu31  cpu33  cpu35  cpu37  cpu39  cpu40  cpu42
cpu44  cpu46  cpu5  cpu7  cpu9      dispatched  pending  run
cpu0    cpu10  cpu12  cpu14  cpu16  cpu18  cpu2   cpu21  cpu23  cpu25
cpu27  cpu29  cpu30  cpu32  cpu34  cpu36  cpu38  cpu4   cpu41  cpu43
cpu45  cpu47  cpu6  cpu8  cpu_list  io_poll     queued   tags

How to configure it that I can have 48 HW queues?
That is 1 HW queue map to 1 SW queue.

Thanks.

Re: iSER multi-queue configuration

[Ming Lin]

On Fri, 2016-04-29 at 14:11 -0700, Ming Lin wrote:
> Hi,
> 
> I use mlx5 to setup iSER and have scsi mq enabled.
> 
> root@xxx:~# cat /sys/module/scsi_mod/parameters/use_blk_mq
> Y
> 
> But it seems there is only 1 HW queue that maps to 48 SW queues.
> 
> root@xxx:/sys/block/sdb/mq# ls
> 0
> 
> root@xxx:/sys/block/sdb/mq# ls 0/
> active  cpu1   cpu11  cpu13  cpu15  cpu17  cpu19  cpu20  cpu22  cpu24
> cpu26  cpu28  cpu3   cpu31  cpu33  cpu35  cpu37  cpu39  cpu40  cpu42
> cpu44  cpu46  cpu5  cpu7  cpu9      dispatched  pending  run
> cpu0    cpu10  cpu12  cpu14  cpu16  cpu18  cpu2   cpu21  cpu23  cpu25
> cpu27  cpu29  cpu30  cpu32  cpu34  cpu36  cpu38  cpu4   cpu41  cpu43
> cpu45  cpu47  cpu6  cpu8  cpu_list  io_poll     queued   tags
> 
> How to configure it that I can have 48 HW queues?
> That is 1 HW queue map to 1 SW queue.

Hi Sagi,

I'm looking at why there is only 1 HW queue for iser.

2185 int scsi_mq_setup_tags(struct Scsi_Host *shost)
2186 {
...
2199         shost->tag_set.nr_hw_queues = shost->nr_hw_queues ? : 1;


It seems because iser code does not set shost->nr_hw_queues at all.

Why?

Re: iSER multi-queue configuration

[Sagi Grimberg]

> Hi Sagi,

Hi Ming,

> I'm looking at why there is only 1 HW queue for iser.
>
> 2185 int scsi_mq_setup_tags(struct Scsi_Host *shost)
> 2186 {
> ...
> 2199         shost->tag_set.nr_hw_queues = shost->nr_hw_queues ? : 1;
>
>
> It seems because iser code does not set shost->nr_hw_queues at all.
>
> Why?

The iscsi driver has not yet been converted to scsi_mq and iser as
an iscsi transport still does not use multi-queue.

In iscsi, a hw queue maps to an iscsi session. In order to support
multi-queue we need to have iscsi open multiple sessions and map
a session to a queue.

I had some preliminary works on this (joint with Mike Christie) but I
didn't have time thus far to implement a full conversion to scsi-mq.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iSER multi-queue configuration

[Ming Lin]

On Sun, 2016-05-01 at 05:53 +0300, Sagi Grimberg wrote:
> > Hi Sagi,
> 
> Hi Ming,
> 
> > I'm looking at why there is only 1 HW queue for iser.
> > 
> > 2185 int scsi_mq_setup_tags(struct Scsi_Host *shost)
> > 2186 {
> > ...
> > 2199         shost->tag_set.nr_hw_queues = shost->nr_hw_queues ? :
> > 1;
> > 
> > 
> > It seems because iser code does not set shost->nr_hw_queues at all.
> > 
> > Why?
> 
> The iscsi driver has not yet been converted to scsi_mq and iser as
> an iscsi transport still does not use multi-queue.

I'm going to compare performance between NVMe over RDMA and SCSI over
RDMA. SRP supports scsi_mq, so I'd use SRP to do the apples-to-apples
comparison.

> 
> In iscsi, a hw queue maps to an iscsi session. In order to support
> multi-queue we need to have iscsi open multiple sessions and map
> a session to a queue.
> 
> I had some preliminary works on this (joint with Mike Christie) but I
> didn't have time thus far to implement a full conversion to scsi-mq.

I'd happy to test it once you done.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: iSER multi-queue configuration

[Steve Wise]

> -----Original Message-----
> From: linux-rdma-owner@vger.kernel.org [mailto:linux-rdma-
> owner@vger.kernel.org] On Behalf Of Ming Lin
> Sent: Sunday, May 01, 2016 1:07 AM
> To: Sagi Grimberg; linux-rdma@vger.kernel.org
> Cc: Bart Van Assche; target-devel
> Subject: Re: iSER multi-queue configuration
> 
> On Sun, 2016-05-01 at 05:53 +0300, Sagi Grimberg wrote:
> > > Hi Sagi,
> >
> > Hi Ming,
> >
> > > I'm looking at why there is only 1 HW queue for iser.
> > >
> > > 2185 int scsi_mq_setup_tags(struct Scsi_Host *shost)
> > > 2186 {
> > > ...
> > > 2199         shost->tag_set.nr_hw_queues = shost->nr_hw_queues ? :
> > > 1;
> > >
> > >
> > > It seems because iser code does not set shost->nr_hw_queues at all.
> > >
> > > Why?
> >
> > The iscsi driver has not yet been converted to scsi_mq and iser as
> > an iscsi transport still does not use multi-queue.
> 
> I'm going to compare performance between NVMe over RDMA and SCSI over
> RDMA. SRP supports scsi_mq, so I'd use SRP to do the apples-to-apples
> comparison.
> 
> >
> > In iscsi, a hw queue maps to an iscsi session. In order to support
> > multi-queue we need to have iscsi open multiple sessions and map
> > a session to a queue.
> >
> > I had some preliminary works on this (joint with Mike Christie) but I
> > didn't have time thus far to implement a full conversion to scsi-mq.
> 
> I'd happy to test it once you done.

Ditto.  SRP as a comparison is only useful for IB/RoCE transports.  iSER will allow iWARP to play in the game as well...

Steve.