Re: About scsi device queue depth

Hannes Reinecke
2021-01-12T07:24:20Z

On 1/11/21 5:21 PM, John Garry wrote:
> Hi,
>
> I was looking at some IOMMU issue on a LSI RAID 3008 card, and noticed
> that performance there is not what I get on other SAS HBAs - it's lower.
>
> After some debugging and fiddling with sdev queue depth in mpt3sas
> driver, I am finding that performance changes appreciably with sdev
> queue depth:
>
> sdev qdepth    fio number jobs*     1    10    20
> 16                    1590    1654    1660
> 32                    1545    1646    1654
> 64                    1436    1085    1070
> 254 (default)                1436    1070    1050
>
> fio queue depth is 40, and I'm using 12x SAS SSDs.
>
> I got comparable disparity in results for fio queue depth = 128 and num
> jobs = 1:
>
> sdev qdepth    fio number jobs*     1
> 16                    1640
> 32                    1618
> 64                    1577
> 254 (default)                1437
>
> IO sched = none.
>
> That driver also sets queue depth tracking = 1, but never seems to kick in.
>
> So it seems to me that the block layer is merging more bios per request,
> as averge sg count per request goes up from 1 - > upto 6 or more. As I
> see, when queue depth lowers the only thing that is really changing is
> that we fail more often in getting the budget in
> scsi_mq_get_budget()->scsi_dev_queue_ready().
>
> So initial sdev queue depth comes from cmd_per_lun by default or
> manually setting in the driver via scsi_change_queue_depth(). It seems
> to me that some drivers are not setting this optimally, as above.
>
> Thoughts on guidance for setting sdev queue depth? Could blk-mq changed
> this behavior?
>
First of all: are these 'real' SAS SSDs?
The peak at 32 seems very ATA-ish, and I wouldn't put it past the LSI
folks to optimize for that case :-)
Can you get a more detailed picture by changing the queue depth more
finegrained?
(Will get you nicer graphs to boot :-)

Cheers,

Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer