Re: Another (ESP?) scsi blk-mq problem on sparc64

[Jens Axboe <axboe@kernel.dk>]

[-- Attachment #1: Type: text/plain, Size: 2010 bytes --]

On 2014-11-14 15:59, Meelis Roos wrote:
>>>> The second oops is in blk_mq_map_queue() which is a trivial
>>>> two level cpu lookup.  I wonder if there's something odd about
>>>> cpu numbers on these big old sparc systems?
>>>
>>> CPU numbers are sparse - they are determined by hardware slot number and
>>> some models only fill every other mainboard slot, and first slots can be
>>> free. I have first board offline and currently have CPUs numbered
>>> 10,11,14,15 online.
>>>
>>> Here is debug with Jens's patch:
>>
>>> [  133.971050] CPU 11: synchronized TICK with master CPU (last diff -1 cycles, maxerr 516 cycles)
>>> [  133.975491] CPU 14: synchronized TICK with master CPU (last diff -3 cycles, maxerr 531 cycles)
>>> [  133.979943] CPU 15: synchronized TICK with master CPU (last diff -3 cycles, maxerr 531 cycles)
>>> [  133.980146] Brought up 4 CPUs
>>
>> So this looks like this might be the issue. On a scsi-mq disabled boot,
>> you have 4 CPUs, but how are they numbered?
>
> The numbers are always the same.

I would hope so, my question was really on what CPU numbers you see. But 
I guess that 10, 11, 14, and 15?

> But everything seems to be mapped to queue 0?

As it should, scsi-mq only supports a single hw queue for now.

>> We might need Christophs debug patch on top this to fully know...
>
> Applied it too, dmesg is below. Yes it does spam the log a lot, and over
> 9600bps console its' somewhat slow :)
>
> There is another detail to note  -this server contains a faulty disk as
> sdc that times out spinup. I left it in the server because it helped to
> pinpoint and fix a previous error in esp scsi driver. This can be a
> factor here too - the error handling details.

It could be. So we have tons of mappings from CPU10 to queue 0, but then 
we see this:

> [  256.236742] cpu: 10
> [  256.236749] queue: 809119744

and it turns to crap. This is pretty weird. Try with this debug patch - 
get rid of the other ones first. It should reduce your noise level too.

-- 
Jens Axboe


[-- Attachment #2: debug.patch --]
[-- Type: text/x-patch, Size: 1154 bytes --]

diff --git a/block/blk-mq-cpumap.c b/block/blk-mq-cpumap.c
index 1065d7c65fa1..9200e2aee746 100644
--- a/block/blk-mq-cpumap.c
+++ b/block/blk-mq-cpumap.c
@@ -81,6 +81,9 @@ int blk_mq_update_queue_map(unsigned int *map, unsigned int nr_queues)
 			map[i] = map[first_sibling];
 	}
 
+	for (i = 0; i < queue; i++)
+		printk(KERN_ERR "cpumap %d -> %d\n", i, map[i]);
+
 	free_cpumask_var(cpus);
 	return 0;
 }
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 68929bad9a6a..1678da3505ea 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1265,12 +1265,25 @@ run_queue:
 	blk_mq_put_ctx(data.ctx);
 }
 
+static int did_warn;
+
 /*
  * Default mapping to a software queue, since we use one per CPU.
  */
 struct blk_mq_hw_ctx *blk_mq_map_queue(struct request_queue *q, const int cpu)
 {
-	return q->queue_hw_ctx[q->mq_map[cpu]];
+	int i;
+
+	i = q->mq_map[cpu];
+	if (!i || did_warn)
+		return q->queue_hw_ctx[0];
+
+	printk(KERN_ERR "blk-mq: cpu %u got queue %u\n", cpu, i);
+	for_each_online_cpu(i)
+		printk(KERN_ERR "  cpu%d -> queue index %u\n", i, q->mq_map[i]);
+
+	did_warn = 1;
+	return q->queue_hw_ctx[0];
 }
 EXPORT_SYMBOL(blk_mq_map_queue);