Re: Blk-mq/scsi-mq Tuning

[Bart Van Assche <bart.vanassche@sandisk.com>]

On 10/28/2015 01:11 PM, Chad Dupuis wrote:
> We¹ve begun to explore blk-mq and scsi-mq and wanted to know if there were
> any best practices in terms of block layer settings.  We¹re looking
> specifically at the FCoE and iSCSI protocols.
>
> A little background on the queues in our hardware first: we have a per
> connection transmit queue and multiple, global receive queues.  The
> transmit queues are not pegged to a particular CPU.  The receive queues
> are pegged to the first N CPUs where N is the number of receive queues.
> We set the nr_hw_queues in the scsi_host_template to N as well.
>
> In our initial testing we¹re not seeing the performance scale as we would
> expect so we wanted to see if there some Œknobs¹ if you will that we could
> try tuning to try to increase the performance.  Also, one question we did
> have is there an official API to be able to set the CPU affinity of the
> hw_ctx_queues?

(added Jens to CC-list)

Hello Chad,

It's great news that you are looking into adding scsi-mq support for 
FCoE and iSCSI initiator HBA's. If you do not see the performance scale 
as expected that probably means that lock contention occurs in the code 
that submits requests to the SCSI request queues. Have you already tried 
to measure L3 cache misses with perf (e.g. perf record -ag -e 
LLC-store-misses sleep 10 && perf report) ? If a single function is 
responsible for more than 10% of the L3 cache misses usually that means 
that that function is causing a bottleneck.

As far as I know an official API for setting the CPU affinity of the 
hw_ctx queues is not yet available. The approach of the SRP initiator 
driver (ib_srp) is that it assumes that the HCA supports MSI-X and that 
MSI-X interrupts have been spread evenly over processors. The ib_srp 
driver selects an MSI-X interrupt for each hw_ctx queue via the 
comp_vector member of struct ib_cq_init_attr. The script I am using 
myself is available at 
http://thread.gmane.org/gmane.linux.kernel.device-mapper.devel/21312/focus=98409. 
I hope one day that script will be superfluous :-)

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html