[Question] on blk_mq_map_queues()

[Question] on blk_mq_map_queues()

[John Garry]

Hi All,

I have a question on the core-to-hw queue mapping in 
blk_mq_map_queues(). In looking at the latest version, we map the first 
$nr_queues cores to separate queues, and then map sibling threaded-cores 
to the same queue. First off, I am just curious why we always map the 
first batch of cores to separate queues.

As I understand for blk mq, we try to map cores which are physically 
close to the same hw queue (in the case of more cores than hw queues). 
So, by this rationale, we would not map the first $nr_queues cores to 
each separate queue, but would rather map core [0, $nr_cores/$nr_queues) 
to q0, [$nr_cores/$nr_queues, 2*$nr_cores/$nr_queues) to q1, and so on 
(assuming $nr_queues is evenly divisible into $nr_cores, for simplicity).

So what is the full desired mapping behaviour?

The specific problem I see is that for my 64-core system (no 
hyperthreading) and 16 hw queues, I see cores per 4-core cluster being 
mapped to different queues, when I would expect them to be mapped to 
same queue.

I know that we can add our own per-driver mapping function to solve, but 
I would expect that that generic mapper would cover a generic platform.

Thanks in advance,
John

Re: [Question] on blk_mq_map_queues()

[Christoph Hellwig]

On Thu, Oct 05, 2017 at 02:28:43PM +0100, John Garry wrote:
> I know that we can add our own per-driver mapping function to solve, but I 
> would expect that that generic mapper would cover a generic platform.

Why aren't you using blk_mq_pci_map_queues?

What kind of hardware are we talking about?

Re: [Question] on blk_mq_map_queues()

[John Garry]

On 05/10/2017 16:59, Christoph Hellwig wrote:
> On Thu, Oct 05, 2017 at 02:28:43PM +0100, John Garry wrote:
>> I know that we can add our own per-driver mapping function to solve, but I
>> would expect that that generic mapper would cover a generic platform.
>
> Why aren't you using blk_mq_pci_map_queues?
>
> What kind of hardware are we talking about?
>
> .
>

It's a HiSilicon hip07 (D05) platform. For this platform, the integrated 
SAS controller is a platform device. This controller supports 16 hw queues.

In terms of core profile, the board has 2 sockets, each socket has 2 cpu 
die, and each die has 4 clusters of 4 cores (ARM A72), so a total of 64 
cores.

Cheers,
John

Re: [Question] on blk_mq_map_queues()

[Christoph Hellwig]

On Thu, Oct 05, 2017 at 05:06:54PM +0100, John Garry wrote:
> It's a HiSilicon hip07 (D05) platform. For this platform, the integrated 
> SAS controller is a platform device. This controller supports 16 hw queues.

That's v1 or v2 in drivers/scsi/hisi_sas?

Seems like you need to implement the equivalent of the 
pci_alloc_irq_vectors vector spreading for platform devices to get
your vectors properly assigned, and the have a blk_mq_of_map_queues
equivalent to blk_mq_pci_map_queues to map the queues based on the
interrupt assignment.

Re: [Question] on blk_mq_map_queues()

[John Garry]

On 05/10/2017 17:27, Christoph Hellwig wrote:
> On Thu, Oct 05, 2017 at 05:06:54PM +0100, John Garry wrote:
>> It's a HiSilicon hip07 (D05) platform. For this platform, the integrated
>> SAS controller is a platform device. This controller supports 16 hw queues.
>
> That's v1 or v2 in drivers/scsi/hisi_sas?

That's v2 drivers/scsi/hisi_sas.

>
> Seems like you need to implement the equivalent of the
> pci_alloc_irq_vectors vector spreading for platform devices to get
> your vectors properly assigned, and the have a blk_mq_of_map_queues
> equivalent to blk_mq_pci_map_queues to map the queues based on the
> interrupt assignment.
>
> .

Right, I'll have a look.

But I am still interested in patch "blk-mq: map all HWQ also in 
hyperthreaded system" and what it is the intended mapping behaviour of 
blk_mq_of_map_queues() supposed to be. As I said originally, it does not 
always seem to follow the general principle of assigning close cores to 
the same queue.

Cheers,
John