From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751526AbdKUJud (ORCPT ); Tue, 21 Nov 2017 04:50:33 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:55098 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751232AbdKUJu3 (ORCPT ); Tue, 21 Nov 2017 04:50:29 -0500 Subject: Re: 4.14: WARNING: CPU: 4 PID: 2895 at block/blk-mq.c:1144 with virtio-blk From: Christian Borntraeger To: Jens Axboe , Bart Van Assche , "virtualization@lists.linux-foundation.org" , "linux-block@vger.kernel.org" , "mst@redhat.com" , "jasowang@redhat.com" , "linux-kernel@vger.kernel.org" References: <9c5eec5d-f542-4d76-6933-6fe31203ce09@de.ibm.com> <1511205644.2396.32.camel@wdc.com> <04526c98-ffc5-1eca-3aa8-50f9212c4323@de.ibm.com> <5c9f2228-0a8b-8225-7038-e6cb3f31ca0b@kernel.dk> <2e44dbd3-2f90-c267-560c-91d1d4b0e892@de.ibm.com> <823b9dd5-7781-5a72-03ff-bc931433fc19@kernel.dk> Date: Tue, 21 Nov 2017 10:50:18 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 x-cbid: 17112109-0020-0000-0000-000003CED24C X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17112109-0021-0000-0000-000042641D19 Message-Id: <15f232d2-2aaa-df7c-57e8-2f710e051e84@de.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-11-21_03:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1711210136 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 11/21/2017 09:35 AM, Christian Borntraeger wrote: > > > On 11/20/2017 09:52 PM, Jens Axboe wrote: >> On 11/20/2017 01:49 PM, Christian Borntraeger wrote: >>> >>> >>> On 11/20/2017 08:42 PM, Jens Axboe wrote: >>>> On 11/20/2017 12:29 PM, Christian Borntraeger wrote: >>>>> >>>>> >>>>> On 11/20/2017 08:20 PM, Bart Van Assche wrote: >>>>>> On Fri, 2017-11-17 at 15:42 +0100, Christian Borntraeger wrote: >>>>>>> This is >>>>>>> >>>>>>> b7a71e66d (Jens Axboe 2017-08-01 09:28:24 -0600 1141) * are mapped to it. >>>>>>> b7a71e66d (Jens Axboe 2017-08-01 09:28:24 -0600 1142) */ >>>>>>> 6a83e74d2 (Bart Van Assche 2016-11-02 10:09:51 -0600 1143) WARN_ON(!cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask) && >>>>>>> 6a83e74d2 (Bart Van Assche 2016-11-02 10:09:51 -0600 1144) cpu_online(hctx->next_cpu)); >>>>>>> 6a83e74d2 (Bart Van Assche 2016-11-02 10:09:51 -0600 1145) >>>>>>> b7a71e66d (Jens Axboe 2017-08-01 09:28:24 -0600 1146) /* >>>>>> >>>>>> Did you really try to figure out when the code that reported the warning >>>>>> was introduced? I think that warning was introduced through the following >>>>>> commit: >>>>> >>>>> This was more a cut'n'paste to show which warning triggered since line numbers are somewhat volatile. >>>>> >>>>>> >>>>>> commit fd1270d5df6a005e1248e87042159a799cc4b2c9 >>>>>> Date: Wed Apr 16 09:23:48 2014 -0600 >>>>>> >>>>>> blk-mq: don't use preempt_count() to check for right CPU >>>>>> >>>>>> UP or CONFIG_PREEMPT_NONE will return 0, and what we really >>>>>> want to check is whether or not we are on the right CPU. >>>>>> So don't make PREEMPT part of this, just test the CPU in >>>>>> the mask directly. >>>>>> >>>>>> Anyway, I think that warning is appropriate and useful. So the next step >>>>>> is to figure out what work item was involved and why that work item got >>>>>> executed on the wrong CPU. >>>>> >>>>> It seems to be related to virtio-blk (is triggered by fio on such disks). Your comment basically >>>>> says: "no this is not a known issue" then :-) >>>>> I will try to take a dump to find out the work item >>>> >>>> blk-mq does not attempt to freeze/sync existing work if a CPU goes away, >>>> and we reconfigure the mappings. So I don't think the above is unexpected, >>>> if you are doing CPU hot unplug while running a fio job. >>> >>> I did a cpu hot plug (adding a CPU) and I started fio AFTER that. >> >> OK, that's different, we should not be triggering a warning for that. >> What does your machine/virtblk topology look like in terms of CPUS, >> nr of queues for virtblk, etc? > > FWIW, 4.11 does work, 4.12 and later is broken. In fact: 4.12 is fine, 4.12.14 is broken.