Re: Blk-mq/scsi-mq Tuning

[Hannes Reinecke <hare@suse.de>]

On 10/30/2015 02:25 PM, Chad Dupuis wrote:
> 
> 
> On Fri, 30 Oct 2015, Hannes Reinecke wrote:
> 
>> On 10/28/2015 09:11 PM, Chad Dupuis wrote:
>>> Hi Folks,
>>>
>>> We¹ve begun to explore blk-mq and scsi-mq and wanted to know if there
>>> were
>>> any best practices in terms of block layer settings.  We¹re looking
>>> specifically at the FCoE and iSCSI protocols.
>>>
>>> A little background on the queues in our hardware first: we have a per
>>> connection transmit queue and multiple, global receive queues.  The
>>> transmit queues are not pegged to a particular CPU.  The receive queues
>>> are pegged to the first N CPUs where N is the number of receive queues.
>>> We set the nr_hw_queues in the scsi_host_template to N as well.
>>>
>> Weelll ... I think you'll run into issues here.
>> The whole point of the multiqueue implementation is that you can tag the
>> submission _and_ completion queue to a single CPU, thereby eliminating
>> locking.
>> If you only peg the completion queue to a CPU you'll still have
>> contention on the submission queue, needing to take locks etc.
>>
>> Plus you will _inevitably_ incur cache misses, as the completion will
>> basically never occur on the same CPU which did the submissoin.
>> Hence the context needs to be bounced to the CPU holding the completion
>> queue, or you'll need to do a IPI to inform the submitting CPU.
>> But if you do that you're essentially doing single-queue submission,
>> so I doubt we're seeing that great improvements.
> 
> This was why I was asking if there was a blk-mq API to be able to set
> CPU affinity for the hardware context queues so I could steer the
> submissions to the CPUs that my receive queues are on (even if they are
> allowed to float).
> 
But what would that achieve?
Each of the hardware context queues would still having to use the
same submission queue, so you'd have to have some serialisation
with spinlocks et.al. during submission. Which is what blk-mq
tries to avoid.
Am I wrong?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html