From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:55848 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751326AbeDFIyF (ORCPT ); Fri, 6 Apr 2018 04:54:05 -0400 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w368n5tJ016133 for ; Fri, 6 Apr 2018 04:54:04 -0400 Received: from e06smtp15.uk.ibm.com (e06smtp15.uk.ibm.com [195.75.94.111]) by mx0b-001b2d01.pphosted.com with ESMTP id 2h65mygdr5-1 (version=TLSv1.2 cipher=AES256-SHA256 bits=256 verify=NOT) for ; Fri, 06 Apr 2018 04:54:04 -0400 Received: from localhost by e06smtp15.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 6 Apr 2018 09:54:01 +0100 Subject: Re: [PATCH] blk-mq: only run mapped hw queues in blk_mq_run_hw_queues() From: Christian Borntraeger To: Ming Lei Cc: Jens Axboe , linux-block@vger.kernel.org, Christoph Hellwig , Stefan Haberland , Christoph Hellwig References: <20180329104806.GB17537@ming.t460p> <20180329114313.GC17537@ming.t460p> <20180330025340.GB12412@ming.t460p> <20180405160503.GA20818@ming.t460p> <20180405161142.GA20972@ming.t460p> <3a72f42f-db90-6092-5e1b-0579d2095daa@de.ibm.com> <20180406084106.GA8940@ming.t460p> Date: Fri, 6 Apr 2018 10:53:57 +0200 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Message-Id: <8f47f9d7-a470-39f5-7ca8-1b26bef0cc0c@de.ibm.com> Sender: linux-block-owner@vger.kernel.org List-Id: linux-block@vger.kernel.org On 04/06/2018 10:51 AM, Christian Borntraeger wrote: > > > On 04/06/2018 10:41 AM, Ming Lei wrote: >> On Thu, Apr 05, 2018 at 07:39:56PM +0200, Christian Borntraeger wrote: >>> >>> >>> On 04/05/2018 06:11 PM, Ming Lei wrote: >>>>> >>>>> Could you please apply the following patch and provide the dmesg boot log? >>>> >>>> And please post out the 'lscpu' log together from the test machine too. >>> >>> attached. >>> >>> As I said before this seems to go way with CONFIG_NR_CPUS=64 or smaller. >>> We have 282 nr_cpu_ids here (max 141CPUs on that z13 with SMT2) but only 8 Cores >>> == 16 threads. >> >> OK, thanks! >> >> The most weird thing is that hctx->next_cpu is computed as 512 since >> nr_cpu_id is 282, and hctx->next_cpu should have pointed to one of >> possible CPU. >> >> Looks like it is a s390 specific issue, since I can setup one queue >> which has same mapping with yours: >> >> - nr_cpu_id is 282 >> - CPU 0~15 is online >> - 64 queues null_blk >> - still run all hw queues in .complete handler >> >> But can't reproduce this issue at all. >> >> So please test the following patch, which may tell us why hctx->next_cpu >> is computed wrong: > > I see things like > > [ 8.196907] wrong next_cpu 512, blk_mq_map_swqueue, first_and > [ 8.196910] wrong next_cpu 512, blk_mq_map_swqueue, first_and > [ 8.196912] wrong next_cpu 512, blk_mq_map_swqueue, first_and > [ 8.196913] wrong next_cpu 512, blk_mq_map_swqueue, first_and > [ 8.196914] wrong next_cpu 512, blk_mq_map_swqueue, first_and > [ 8.196915] wrong next_cpu 512, blk_mq_map_swqueue, first_and > [ 8.196916] wrong next_cpu 512, blk_mq_map_swqueue, first_and > [ 8.196916] wrong next_cpu 512, blk_mq_map_swqueue, first_and > [ 8.196917] wrong next_cpu 512, blk_mq_map_swqueue, first_and > [ 8.196918] wrong next_cpu 512, blk_mq_map_swqueue, first_and There are more # dmesg | grep "wrong next" | cut -d "]" -f 2- | uniq -c 10 wrong next_cpu 512, blk_mq_map_swqueue, first_and 72 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and 1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and 1 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and 1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and 1 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and 1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and 1 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and 1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and 1 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and 1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and 1 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and 1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and 7 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and 1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and 1 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and 1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and 10 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and > > which is exactly what happens if the find and and operation fails (returns size of bitmap). > > FWIW, I added a dump stack for the case when we run unbound before I tested your patch: > > Apr 06 10:47:41 s38lp39 kernel: CPU: 15 PID: 86 Comm: ksoftirqd/15 Not tainted 4.16.0-07249-g864f9fc031e4-dirty #2 > Apr 06 10:47:41 s38lp39 kernel: Hardware name: IBM 2964 NC9 704 (LPAR) > Apr 06 10:47:41 s38lp39 kernel: Call Trace: > Apr 06 10:47:41 s38lp39 kernel: ([<0000000000113946>] show_stack+0x56/0x80) > Apr 06 10:47:41 s38lp39 kernel: [<00000000009d8132>] dump_stack+0x82/0xb0 > Apr 06 10:47:41 s38lp39 kernel: [<00000000006a05de>] blk_mq_hctx_next_cpu+0x12e/0x138 > Apr 06 10:47:41 s38lp39 kernel: [<00000000006a084c>] __blk_mq_delay_run_hw_queue+0x94/0xd8 > Apr 06 10:47:41 s38lp39 kernel: [<00000000006a097a>] blk_mq_run_hw_queue+0x82/0x180 > Apr 06 10:47:41 s38lp39 kernel: [<00000000006a0ae0>] blk_mq_run_hw_queues+0x68/0x88 > Apr 06 10:47:41 s38lp39 kernel: [<000000000069fc4e>] __blk_mq_complete_request+0x11e/0x1d8 > Apr 06 10:47:41 s38lp39 kernel: [<000000000069fd94>] blk_mq_complete_request+0x8c/0xc8 > Apr 06 10:47:41 s38lp39 kernel: [<0000000000824c50>] dasd_block_tasklet+0x158/0x490 > Apr 06 10:47:41 s38lp39 kernel: [<000000000014a952>] tasklet_action_common.isra.5+0x7a/0x100 > Apr 06 10:47:41 s38lp39 kernel: [<00000000009f8248>] __do_softirq+0x98/0x368 > Apr 06 10:47:41 s38lp39 kernel: [<000000000014a322>] run_ksoftirqd+0x4a/0x68 > Apr 06 10:47:41 s38lp39 kernel: [<000000000016dc20>] smpboot_thread_fn+0x108/0x1b0 > Apr 06 10:47:41 s38lp39 kernel: [<0000000000168e70>] kthread+0x148/0x160 > Apr 06 10:47:41 s38lp39 kernel: [<00000000009f727a>] kernel_thread_starter+0x6/0xc > Apr 06 10:47:41 s38lp39 kernel: [<00000000009f7274>] kernel_thread_starter+0x0/0xc >